[PATCH] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[PATCH] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

From: Josef Johansson <josef@oderland.se>


PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
    
'commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
functions")' introduced functions pci_msi_update_mask() and 
pci_msix_write_vector_ctrl() that were missing checks for
pci_msi_ignore_mask that existed in 'commit 446a98b19fd6 ("PCI/MSI: Use
new mask/unmask functions")'. This patch adds them back since it was
causing softlocks in amdgpu drivers under Xen.

As explained in 'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
on Xen PV")', when running as Xen PV guest, masking MSI-X is a 
responsibility of the hypervisor.

Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
functions")
Suggested-by: Jason Andryuk <jandryuk@gmail.com>
Signed-off-by: Josef Johansson <josef@oderland.se>
---

This patch solves a major issue with booting 5.15-rc1 under Xen
with amdgpu drivers. Specifically Lenovo P14s Gen 1, AMD 4750U.

The softlock below takes about ~2-3 minutes to release, and will
lock again if I switch between X and console, when staying in
X I can do things without it softlock.

I have to note that this is my first commit and PCI/MSI area is
not my area of expertise. I tried to mimic what was
obviously missing between the aforementioned commits. There may be
better ways to solve this problem, or other places to put the checks.
Should desc->msi_attrib.is_virtual be checked? It is not checked in
'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X on Xen PV")'

I should also note that my original problem with *flip done timeout*
inside drm during suspend/resume is not solved, but with this patch
at least the kernel boots properly.

The kernel is much more stable not running inside Xen, and much 
more stable running pci=nomsi (under Xen). Are we missing something
more regarding masking?

The error that occurs is:

kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 6 PID: 3754 at
drivers/gpu/drm/amd/amdgpu/../display/amdgp
u_dm/amdgpu_dm.c:8630 amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
 fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
 xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G   W 5.15.0-1.fc32.qubes.x86_64 #1
kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
2020
kernel: RIP: e030:amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
kernel: Code: 8b 45 b0 48 c7 c7 4b fc 90 c0 4c 89 55 88 8b b0 f0 03 00 00 e8 6d
cb ca ff 4c 8b 55 88 0f b6 55 ab 49 8b 72 08 e9 2b fa ff ff <0f> 0b e9 fa fe ff
ff e8 40 2f 6e c1 0f 1f 44 00 00 55 b9 27 00 00
kernel: RSP: e02b:ffffc90042d93638 EFLAGS: 00010002
kernel: RAX: ffff888110840210 RBX: 00000000000083c1 RCX: 0000000000000466
kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff888110840170
kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000)
knlGS:000000000000
0000
kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
kernel: Call Trace:
kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
kernel:  fb_pan_display+0x83/0x100
kernel:  fb_set_var+0x200/0x3b0
kernel:  fbcon_blank+0x186/0x280
kernel:  do_unblank_screen+0xaa/0x150
kernel:  complete_change_console+0x54/0x120
kernel:  vt_ioctl+0x31d/0x5f0
kernel:  tty_ioctl+0x312/0x780
kernel:  __x64_sys_ioctl+0x83/0xb0
kernel:  do_syscall_64+0x3b/0x90
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
kernel: RIP: 0033:0x718c6e33217b
kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX:
0000000000000010
kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
kernel: ---[ end trace 2c3e3998803422cb ]---
kernel: ------------[ cut here ]------------
kernel: WARNING: CPU: 6 PID: 3754 at
drivers/gpu/drm/amd/amdgpu/../display/amdgp
u_dm/amdgpu_dm.c:8217 prepare_flip_isr+0x64/0x70 [amdgpu]
kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
 fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
 xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G W 5.15.0-1.fc32.qubes.x86_64 #1
kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
2020
kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00 66 90 c3 8b 97 f0 03 00 00 48
c7 c6 18 72 8d c0 48 c7 c7 90 5b a7 c0 e9 7e 6e 13 c1 0f 0b <0f> 0b eb b4 0f 1f
84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41
kernel: RSP: e02b:ffffc90042d93630 EFLAGS: 00010086
kernel: RAX: 0000000000000001 RBX: 00000000000083c1 RCX: 0000000000000466
kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff88810cb2e000
kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000) knlGS:000000000000
0000
kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
kernel: Call Trace:
kernel:  amdgpu_dm_commit_planes+0x8bd/0x9c0 [amdgpu]
kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
kernel:  fb_pan_display+0x83/0x100
kernel:  fb_set_var+0x200/0x3b0
kernel:  fbcon_blank+0x186/0x280
kernel:  do_unblank_screen+0xaa/0x150
kernel:  complete_change_console+0x54/0x120
kernel:  vt_ioctl+0x31d/0x5f0
kernel:  tty_ioctl+0x312/0x780
kernel:  __x64_sys_ioctl+0x83/0xb0
kernel:  do_syscall_64+0x3b/0x90
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
kernel: RIP: 0033:0x718c6e33217b
kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
kernel: ---[ end trace 2c3e3998803422cc ]---

It is discussed briefly at:
https://lore.kernel.org/linux-pci/CAKf6xpvGyCKVHsvauP54=0j10fxis4XiiqBNWH+1cpkbtt_QJw@mail.gmail.com/


diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 0099a00af361..355b791e382f 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 {
 	void __iomem *desc_addr = pci_msix_desc_addr(desc);
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 }

--
2.31.1

Re: [PATCH] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Bjorn Helgaas]

[+cc Marc]

On Mon, Oct 18, 2021 at 08:22:32AM +0200, Josef Johansson wrote:
> From: Josef Johansson <josef@oderland.se>
> 
> 
> PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
>     
> 'commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions")' introduced functions pci_msi_update_mask() and 
> pci_msix_write_vector_ctrl() that were missing checks for
> pci_msi_ignore_mask that existed in 'commit 446a98b19fd6 ("PCI/MSI: Use
> new mask/unmask functions")'. This patch adds them back since it was
> causing softlocks in amdgpu drivers under Xen.
> 
> As explained in 'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
> on Xen PV")', when running as Xen PV guest, masking MSI-X is a 
> responsibility of the hypervisor.
> 
> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions")
> Suggested-by: Jason Andryuk <jandryuk@gmail.com>
> Signed-off-by: Josef Johansson <josef@oderland.se>

fcacdfbef5a1 appeared in v5.15-rc1, and we should try to get this
fixed before v5.15.

I could merge it but would like an ack from Thomas since he wrote
fcacdfbef5a1.  He merged fcacdfbef5a1, so I wouldn't complain if the
fix followed the same path.

If I merged it (or if you were to repost it), I would drop the single
quotes around the commit citations and write the commit log in
imperative mood, as you did for the subject
(https://chris.beams.io/posts/git-commit/)

> ---
> 
> This patch solves a major issue with booting 5.15-rc1 under Xen
> with amdgpu drivers. Specifically Lenovo P14s Gen 1, AMD 4750U.
> 
> The softlock below takes about ~2-3 minutes to release, and will
> lock again if I switch between X and console, when staying in
> X I can do things without it softlock.

I don't actually see a softlock mentioned below; am I missing
something?  It's nice to include a couple lines of dmesg log to help
people connect the issue with the fix, but most of the below is not
relevant and can be easily found from the Link: tag.  Also, some of
the lines seem to be wrapped.  They're more useful when not wrapped
because grep can find them.

> I have to note that this is my first commit and PCI/MSI area is
> not my area of expertise. I tried to mimic what was
> obviously missing between the aforementioned commits. There may be
> better ways to solve this problem, or other places to put the checks.
> Should desc->msi_attrib.is_virtual be checked? It is not checked in
> 'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X on Xen PV")'
> 
> I should also note that my original problem with *flip done timeout*
> inside drm during suspend/resume is not solved, but with this patch
> at least the kernel boots properly.
> 
> The kernel is much more stable not running inside Xen, and much 
> more stable running pci=nomsi (under Xen). Are we missing something
> more regarding masking?

It does sound like something else is broken as well, but I have no
idea what that would be.

> The error that occurs is:
> 
> kernel: ------------[ cut here ]------------
> kernel: WARNING: CPU: 6 PID: 3754 at
> drivers/gpu/drm/amd/amdgpu/../display/amdgp
> u_dm/amdgpu_dm.c:8630 amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
> kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
> intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
> deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
> nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
> sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
>  fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
> sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
> rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
> sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
> hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
>  xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G   W 5.15.0-1.fc32.qubes.x86_64 #1
> kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
> 2020
> kernel: RIP: e030:amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
> kernel: Code: 8b 45 b0 48 c7 c7 4b fc 90 c0 4c 89 55 88 8b b0 f0 03 00 00 e8 6d
> cb ca ff 4c 8b 55 88 0f b6 55 ab 49 8b 72 08 e9 2b fa ff ff <0f> 0b e9 fa fe ff
> ff e8 40 2f 6e c1 0f 1f 44 00 00 55 b9 27 00 00
> kernel: RSP: e02b:ffffc90042d93638 EFLAGS: 00010002
> kernel: RAX: ffff888110840210 RBX: 00000000000083c1 RCX: 0000000000000466
> kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff888110840170
> kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
> kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
> kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
> kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000)
> knlGS:000000000000
> 0000
> kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
> kernel: Call Trace:
> kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
> kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
> kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
> kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
> kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
> kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
> kernel:  fb_pan_display+0x83/0x100
> kernel:  fb_set_var+0x200/0x3b0
> kernel:  fbcon_blank+0x186/0x280
> kernel:  do_unblank_screen+0xaa/0x150
> kernel:  complete_change_console+0x54/0x120
> kernel:  vt_ioctl+0x31d/0x5f0
> kernel:  tty_ioctl+0x312/0x780
> kernel:  __x64_sys_ioctl+0x83/0xb0
> kernel:  do_syscall_64+0x3b/0x90
> kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
> kernel: RIP: 0033:0x718c6e33217b
> kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
> ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
> 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
> kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX:
> 0000000000000010
> kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
> kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
> kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
> kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
> kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
> kernel: ---[ end trace 2c3e3998803422cb ]---
> kernel: ------------[ cut here ]------------
> kernel: WARNING: CPU: 6 PID: 3754 at
> drivers/gpu/drm/amd/amdgpu/../display/amdgp
> u_dm/amdgpu_dm.c:8217 prepare_flip_isr+0x64/0x70 [amdgpu]
> kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
> intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
> deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
> nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
> sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
>  fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
> sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
> rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
> sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
> hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
>  xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G W 5.15.0-1.fc32.qubes.x86_64 #1
> kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
> 2020
> kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
> kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00 66 90 c3 8b 97 f0 03 00 00 48
> c7 c6 18 72 8d c0 48 c7 c7 90 5b a7 c0 e9 7e 6e 13 c1 0f 0b <0f> 0b eb b4 0f 1f
> 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41
> kernel: RSP: e02b:ffffc90042d93630 EFLAGS: 00010086
> kernel: RAX: 0000000000000001 RBX: 00000000000083c1 RCX: 0000000000000466
> kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff88810cb2e000
> kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
> kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
> kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
> kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000) knlGS:000000000000
> 0000
> kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
> kernel: Call Trace:
> kernel:  amdgpu_dm_commit_planes+0x8bd/0x9c0 [amdgpu]
> kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
> kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
> kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
> kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
> kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
> kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
> kernel:  fb_pan_display+0x83/0x100
> kernel:  fb_set_var+0x200/0x3b0
> kernel:  fbcon_blank+0x186/0x280
> kernel:  do_unblank_screen+0xaa/0x150
> kernel:  complete_change_console+0x54/0x120
> kernel:  vt_ioctl+0x31d/0x5f0
> kernel:  tty_ioctl+0x312/0x780
> kernel:  __x64_sys_ioctl+0x83/0xb0
> kernel:  do_syscall_64+0x3b/0x90
> kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
> kernel: RIP: 0033:0x718c6e33217b
> kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
> ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
> 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
> kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
> kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
> kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
> kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
> kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
> kernel: ---[ end trace 2c3e3998803422cc ]---
> 
> It is discussed briefly at:
> https://lore.kernel.org/linux-pci/CAKf6xpvGyCKVHsvauP54=0j10fxis4XiiqBNWH+1cpkbtt_QJw@mail.gmail.com/
> 
> 
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 0099a00af361..355b791e382f 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>  	unsigned long flags;
>  
> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +		return;
> +
>  	raw_spin_lock_irqsave(lock, flags);
>  	desc->msi_mask &= ~clear;
>  	desc->msi_mask |= set;
> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>  {
>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>  
> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +		return;
> +
>  	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  }
> 
> --
> 2.31.1
> 

Re: [PATCH] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

On 10/19/21 21:57, Bjorn Helgaas wrote:
> [+cc Marc]
>
> On Mon, Oct 18, 2021 at 08:22:32AM +0200, Josef Johansson wrote:
>> From: Josef Johansson <josef@oderland.se>
>>
>>
>> PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
>>     
>> 'commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>> functions")' introduced functions pci_msi_update_mask() and 
>> pci_msix_write_vector_ctrl() that were missing checks for
>> pci_msi_ignore_mask that existed in 'commit 446a98b19fd6 ("PCI/MSI: Use
>> new mask/unmask functions")'. This patch adds them back since it was
>> causing softlocks in amdgpu drivers under Xen.
>>
>> As explained in 'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
>> on Xen PV")', when running as Xen PV guest, masking MSI-X is a 
>> responsibility of the hypervisor.
>>
>> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>> functions")
>> Suggested-by: Jason Andryuk <jandryuk@gmail.com>
>> Signed-off-by: Josef Johansson <josef@oderland.se>
> fcacdfbef5a1 appeared in v5.15-rc1, and we should try to get this
> fixed before v5.15.
>
> I could merge it but would like an ack from Thomas since he wrote
> fcacdfbef5a1.  He merged fcacdfbef5a1, so I wouldn't complain if the
> fix followed the same path.
>
> If I merged it (or if you were to repost it), I would drop the single
> quotes around the commit citations and write the commit log in
> imperative mood, as you did for the subject
> (https://chris.beams.io/posts/git-commit/)

I'll make an effort to do a better commit log. Thanks for reviewing the
entry!

What is the time frame for v5.15?

>> ---
>>
>> This patch solves a major issue with booting 5.15-rc1 under Xen
>> with amdgpu drivers. Specifically Lenovo P14s Gen 1, AMD 4750U.
>>
>> The softlock below takes about ~2-3 minutes to release, and will
>> lock again if I switch between X and console, when staying in
>> X I can do things without it softlock.
> I don't actually see a softlock mentioned below; am I missing
> something?  It's nice to include a couple lines of dmesg log to help
> people connect the issue with the fix, but most of the below is not
> relevant and can be easily found from the Link: tag.  Also, some of
> the lines seem to be wrapped.  They're more useful when not wrapped
> because grep can find them.

Sorry for my lack of words here. I used deadlock when I first described
it, but since

it was released after a while, I thought softlock would be more fitting.

I'll dig a bit to try to get a better dmesg around the stacktrace. Sorry
about the wrapping,

I'm trying hard to keep those 80 chars ;-)

>> I have to note that this is my first commit and PCI/MSI area is
>> not my area of expertise. I tried to mimic what was
>> obviously missing between the aforementioned commits. There may be
>> better ways to solve this problem, or other places to put the checks.
>> Should desc->msi_attrib.is_virtual be checked? It is not checked in
>> 'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X on Xen PV")'
>>
>> I should also note that my original problem with *flip done timeout*
>> inside drm during suspend/resume is not solved, but with this patch
>> at least the kernel boots properly.
>>
>> The kernel is much more stable not running inside Xen, and much 
>> more stable running pci=nomsi (under Xen). Are we missing something
>> more regarding masking?
> It does sound like something else is broken as well, but I have no
> idea what that would be.

Should I take a stab at describing the issue better at hand or should we
focus on getting this

specific patch out of the way?

I have 'solved' my current problems a bit by kernel flags right now, but
I would be happy to

share my story so far.

>> The error that occurs is:
>>
>> kernel: ------------[ cut here ]------------
>> kernel: WARNING: CPU: 6 PID: 3754 at
>> drivers/gpu/drm/amd/amdgpu/../display/amdgp
>> u_dm/amdgpu_dm.c:8630 amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
>> kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
>> intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
>> deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
>> nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
>> sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
>>  fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
>> sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
>> rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
>> sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
>> hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
>>  xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>> kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G   W 5.15.0-1.fc32.qubes.x86_64 #1
>> kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
>> 2020
>> kernel: RIP: e030:amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]
>> kernel: Code: 8b 45 b0 48 c7 c7 4b fc 90 c0 4c 89 55 88 8b b0 f0 03 00 00 e8 6d
>> cb ca ff 4c 8b 55 88 0f b6 55 ab 49 8b 72 08 e9 2b fa ff ff <0f> 0b e9 fa fe ff
>> ff e8 40 2f 6e c1 0f 1f 44 00 00 55 b9 27 00 00
>> kernel: RSP: e02b:ffffc90042d93638 EFLAGS: 00010002
>> kernel: RAX: ffff888110840210 RBX: 00000000000083c1 RCX: 0000000000000466
>> kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff888110840170
>> kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
>> kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
>> kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
>> kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000)
>> knlGS:000000000000
>> 0000
>> kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
>> kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
>> kernel: Call Trace:
>> kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
>> kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
>> kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
>> kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
>> kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
>> kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
>> kernel:  fb_pan_display+0x83/0x100
>> kernel:  fb_set_var+0x200/0x3b0
>> kernel:  fbcon_blank+0x186/0x280
>> kernel:  do_unblank_screen+0xaa/0x150
>> kernel:  complete_change_console+0x54/0x120
>> kernel:  vt_ioctl+0x31d/0x5f0
>> kernel:  tty_ioctl+0x312/0x780
>> kernel:  __x64_sys_ioctl+0x83/0xb0
>> kernel:  do_syscall_64+0x3b/0x90
>> kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
>> kernel: RIP: 0033:0x718c6e33217b
>> kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
>> ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
>> 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
>> kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX:
>> 0000000000000010
>> kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
>> kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
>> kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
>> kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
>> kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
>> kernel: ---[ end trace 2c3e3998803422cb ]---
>> kernel: ------------[ cut here ]------------
>> kernel: WARNING: CPU: 6 PID: 3754 at
>> drivers/gpu/drm/amd/amdgpu/../display/amdgp
>> u_dm/amdgpu_dm.c:8217 prepare_flip_isr+0x64/0x70 [amdgpu]
>> kernel: Modules linked in: nf_tables nfnetlink vfat fat intel_rapl_msr wmi_bmof
>> intel_rapl_common pcspkr joydev uvcvideo k10temp sp5100_tco videobuf2_vmalloc vi
>> deobuf2_memops i2c_piix4 videobuf2_v4l2 videobuf2_common videodev mc iwlwifi thi
>> nkpad_acpi platform_profile ipmi_devintf ledtrig_audio ucsi_acpi cfg80211 ipmi_m
>> sghandler r8169 snd typec_ucsi soundcore typec rfkill wmi video amd_pmc i2c_scmi
>>  fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt tru
>> sted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul iommu_v2 crc32_pclmul c
>> rc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ccp cec gha
>> sh_clmulni_intel sdhci_pci xhci_pci xhci_pci_renesas serio_raw cqhci drm sdhci x
>> hci_hcd mmc_core nvme ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
>>  xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>> kernel: CPU: 6 PID: 3754 Comm: Xorg Tainted: G W 5.15.0-1.fc32.qubes.x86_64 #1
>> kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET61W(1.30) 12/21/
>> 2020
>> kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
>> kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00 66 90 c3 8b 97 f0 03 00 00 48
>> c7 c6 18 72 8d c0 48 c7 c7 90 5b a7 c0 e9 7e 6e 13 c1 0f 0b <0f> 0b eb b4 0f 1f
>> 84 00 00 00 00 00 0f 1f 44 00 00 41 57 41 56 41
>> kernel: RSP: e02b:ffffc90042d93630 EFLAGS: 00010086
>> kernel: RAX: 0000000000000001 RBX: 00000000000083c1 RCX: 0000000000000466
>> kernel: RDX: 0000000000000001 RSI: 0000000000000204 RDI: ffff88810cb2e000
>> kernel: RBP: ffffc90042d936f8 R08: 0000000000000002 R09: 0000000000000001
>> kernel: R10: 0000000000000000 R11: ffff88810cb2e118 R12: ffff888110840210
>> kernel: R13: ffff88810cb2e000 R14: ffff888103d50400 R15: ffff888103bb2c00
>> kernel: FS:  0000718c6de4da40(0000) GS:ffff888140780000(0000) knlGS:000000000000
>> 0000
>> kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
>> kernel: CR2: 0000726ada294000 CR3: 0000000103f0e000 CR4: 0000000000050660
>> kernel: Call Trace:
>> kernel:  amdgpu_dm_commit_planes+0x8bd/0x9c0 [amdgpu]
>> kernel:  amdgpu_dm_atomic_commit_tail+0xc3e/0x1360 [amdgpu]
>> kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
>> kernel:  drm_atomic_helper_commit+0x113/0x140 [drm_kms_helper]
>> kernel:  drm_client_modeset_commit_atomic+0x1fc/0x240 [drm]
>> kernel:  drm_client_modeset_commit_locked+0x53/0x80 [drm]
>> kernel:  drm_fb_helper_pan_display+0xdc/0x210 [drm_kms_helper]
>> kernel:  fb_pan_display+0x83/0x100
>> kernel:  fb_set_var+0x200/0x3b0
>> kernel:  fbcon_blank+0x186/0x280
>> kernel:  do_unblank_screen+0xaa/0x150
>> kernel:  complete_change_console+0x54/0x120
>> kernel:  vt_ioctl+0x31d/0x5f0
>> kernel:  tty_ioctl+0x312/0x780
>> kernel:  __x64_sys_ioctl+0x83/0xb0
>> kernel:  do_syscall_64+0x3b/0x90
>> kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
>> kernel: RIP: 0033:0x718c6e33217b
>> kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
>> ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
>> 73 01 c3 48 8b 0d ed ac 0c 00 f7 d8 64 89 01 48
>> kernel: RSP: 002b:00007ffd5c6157c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
>> kernel: RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 0000718c6e33217b
>> kernel: RDX: 0000000000000001 RSI: 0000000000005605 RDI: 0000000000000014
>> kernel: RBP: 000057b9b2aa93f4 R08: 0000000000000000 R09: 0000000000000001
>> kernel: R10: fffffffffffff9ce R11: 0000000000000246 R12: 000057b9b2aa94b0
>> kernel: R13: 000057b9b2aa94a0 R14: 000057b9b2aa93f0 R15: 00007ffd5c615844
>> kernel: ---[ end trace 2c3e3998803422cc ]---
>>
>> It is discussed briefly at:
>> https://lore.kernel.org/linux-pci/CAKf6xpvGyCKVHsvauP54=0j10fxis4XiiqBNWH+1cpkbtt_QJw@mail.gmail.com/
>>
>>
>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>> index 0099a00af361..355b791e382f 100644
>> --- a/drivers/pci/msi.c
>> +++ b/drivers/pci/msi.c
>> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>>  	unsigned long flags;
>>  
>> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>> +		return;
>> +
>>  	raw_spin_lock_irqsave(lock, flags);
>>  	desc->msi_mask &= ~clear;
>>  	desc->msi_mask |= set;
>> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>>  {
>>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>>  
>> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>> +		return;
>> +
>>  	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>  }
>>
>> --
>> 2.31.1
>>

Re: [PATCH] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Bjorn Helgaas]

On Tue, Oct 19, 2021 at 10:15:05PM +0200, Josef Johansson wrote:
> On 10/19/21 21:57, Bjorn Helgaas wrote:
> > On Mon, Oct 18, 2021 at 08:22:32AM +0200, Josef Johansson wrote:

> I'll make an effort to do a better commit log. Thanks for reviewing the
> entry!
> 
> What is the time frame for v5.15?

Soon.  v5.15-rc7 should happen Oct 24, final release likely Oct 31.
Ideally a fix would be in before the 24th.

> >> This patch solves a major issue with booting 5.15-rc1 under Xen
> >> with amdgpu drivers. Specifically Lenovo P14s Gen 1, AMD 4750U.
> >>
> >> The softlock below takes about ~2-3 minutes to release, and will
> >> lock again if I switch between X and console, when staying in
> >> X I can do things without it softlock.
> >
> > I don't actually see a softlock mentioned below; am I missing
> > something?  It's nice to include a couple lines of dmesg log to help
> > people connect the issue with the fix, but most of the below is not
> > relevant and can be easily found from the Link: tag.  Also, some of
> > the lines seem to be wrapped.  They're more useful when not wrapped
> > because grep can find them.
> 
> Sorry for my lack of words here. I used deadlock when I first
> described it, but since it was released after a while, I thought
> softlock would be more fitting.

I looked up the WARN_ON at amdgpu_dm.c:8630 but I didn't see anything
related to deadlock or soft lockup or any kind of delay:

  WARN_ON(acrtc_attach->pflip_status != AMDGPU_FLIP_NONE);

DRM folks might have an idea.

> I'll dig a bit to try to get a better dmesg around the stacktrace. Sorry
> about the wrapping,
> 
> I'm trying hard to keep those 80 chars ;-)

I appreciate that.  Commit logs should be 75 chars because git log
indents by 4.  Quotes of dmesg and the like should be unwrapped.

> >> I should also note that my original problem with *flip done timeout*
> >> inside drm during suspend/resume is not solved, but with this patch
> >> at least the kernel boots properly.
> >>
> >> The kernel is much more stable not running inside Xen, and much 
> >> more stable running pci=nomsi (under Xen). Are we missing something
> >> more regarding masking?
> > It does sound like something else is broken as well, but I have no
> > idea what that would be.
> 
> Should I take a stab at describing the issue better at hand or
> should we focus on getting this specific patch out of the way?

Unless the other issues seem related, we should dispose of this one
by itself ASAP.

> I have 'solved' my current problems a bit by kernel flags right now,
> but I would be happy to share my story so far.

Description of your workaround would be quite useful.  Probably not to
*me*, but to people who know the area (I'd start with the DRM folks).

> >> The error that occurs is:
> >>
> >> kernel: ------------[ cut here ]------------
> >> kernel: WARNING: CPU: 6 PID: 3754 at
> >> drivers/gpu/drm/amd/amdgpu/../display/amdgp
> >> u_dm/amdgpu_dm.c:8630 amdgpu_dm_commit_planes+0x9b4/0x9c0 [amdgpu]

[PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

From: Josef Johansson <josef@oderland.se>


PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
    
commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
functions") introduce functions pci_msi_update_mask() and 
pci_msix_write_vector_ctrl() that is missing checks for
pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
new mask/unmask functions"). Add them back since it is
causing severe lockups in amdgpu drivers under Xen during boot.

As explained in commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
on Xen PV"), when running as Xen PV guest, masking MSI-X is a 
responsibility of the hypervisor.

Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
functions")
Suggested-by: Jason Andryuk <jandryuk@gmail.com>
Signed-off-by: Josef Johansson <josef@oderland.se>
---

v1
* commit log is not in the correct mood
* commit log has single quotes around the commits
* the information with what led up to the patch is lacking.

v2
* correct the mood
* correct the quotes
* add much more information.

Here I describe the current patch and what led up to it,
more on what led up to it below the first attached dmesg.

This patch solves a major issue with booting 5.15-rc1 under Xen
with amdgpu drivers. Specifically Lenovo P14s Gen 1, AMD 4750U.

During boot around when I unluck the disk the first entry of
'Fence fallback timer expired' occur, the laptop is mostly useless.

After a while the WARN_ON trace shows up and the boot process starts
again, until it again gets stuck. This pattern repeats until X boots,
after that point it's kind of ok to work in.
If I try to switch to console the same process happens.

My solution to this was at first bisecting and finding 
commit 446a98b19fd6 ("PCI/MSI: Use new mask/unmask functions") series,
reverting this commit made the boot fast again with no lockups.

Later on I tried to apply pci=nomsi as a kernel argument and that
worked fine as well, letting me compile the kernel without the revert.

I have to note that this is my first commit and PCI/MSI area is
not my area of expertise. I tried to mimic what was
obviously missing between the aforementioned commits. There may be
better ways to solve this problem, or other places to put the checks.
Should desc->msi_attrib.is_virtual be checked? It is not checked in
'commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X on Xen PV")'

If there's anything I can try out, I'd be happy to assist.

The important bits from dmesg, with debug-configs on.

Oct 11 22:32:00 dom0 kernel: Linux version 5.15.0-1.fc32.qubes.x86_64
(user@compiler) (gcc (GCC) 10.3.1 20210422 (Red Hat 10.3.1-1), GNU ld
version 2.34-6.fc32) #14 SMP Mon Oct 11 20:12:00 UTC 2021

Oct 11 22:32:00 dom0 kernel: Command line: placeholder
root=/dev/mapper/qubes_dom0-root ro
rd.luks.uuid=luks-c8f1f8e3-a5e7-4697-b01e-b104f5a0eedb
rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap
plymouth.ignore-serial-consoles rd.driver.pre=btrfs
rd.driver.blacklist=pcspkr

[snip]

Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: [drm] fb0: amdgpu
frame buffer device
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring gfx uses
VM inv eng 0 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.0.0 uses VM inv eng 1 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.1.0 uses VM inv eng 4 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.2.0 uses VM inv eng 5 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.3.0 uses VM inv eng 6 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.0.1 uses VM inv eng 7 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.1.1 uses VM inv eng 8 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.2.1 uses VM inv eng 9 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring
comp_1.3.1 uses VM inv eng 10 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring kiq_2.1.0
uses VM inv eng 11 on hub 0
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring sdma0
uses VM inv eng 0 on hub 1
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring vcn_dec
uses VM inv eng 1 on hub 1
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring vcn_enc0
uses VM inv eng 4 on hub 1
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring vcn_enc1
uses VM inv eng 5 on hub 1
Oct 11 22:32:04 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: ring jpeg_dec
uses VM inv eng 6 on hub 1
Oct 11 22:32:04 dom0 kernel: [drm] Initialized amdgpu 3.42.0 20150101
for 0000:07:00.0 on minor 0

[snip]

Oct 11 22:32:06 dom0 kernel: [drm] Fence fallback timer expired on ring gfx
Oct 11 22:32:07 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.0.0
Oct 11 22:32:07 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.1.0
Oct 11 22:32:08 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.2.0
Oct 11 22:32:08 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.3.0
Oct 11 22:32:09 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.0.1
Oct 11 22:32:09 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.1.1
Oct 11 22:32:10 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.2.1
Oct 11 22:32:10 dom0 kernel: [drm] Fence fallback timer expired on ring
comp_1.3.1
Oct 11 22:32:11 dom0 kernel: [drm] Fence fallback timer expired on ring
sdma0
Oct 11 22:32:11 dom0 kernel: [drm] Fence fallback timer expired on ring
vcn_dec
Oct 11 22:32:12 dom0 kernel: [drm] Fence fallback timer expired on ring
vcn_enc0
Oct 11 22:32:12 dom0 kernel: amdgpu 0000:07:00.0: [drm] *ERROR* Sending
link address failed with -5
Oct 11 22:32:12 dom0 kernel: [drm] Fence fallback timer expired on ring
vcn_enc1
Oct 11 22:32:13 dom0 kernel: [drm] Fence fallback timer expired on ring
jpeg_dec
Oct 11 22:32:14 dom0 kernel: [drm:drm_atomic_helper_wait_for_flip_done
[drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out
[snip]

Oct 11 22:32:37 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:32:37 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CRTC:67:crtc-0] commit wait timed out
Oct 11 22:32:47 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:32:47 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CONNECTOR:78:eDP-1] commit wait timed out
Oct 11 22:32:57 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:32:57 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[PLANE:55:plane-3] commit wait timed out
Oct 11 22:32:57 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:32:57 dom0 kernel: WARNING: CPU: 5 PID: 1425 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8689
amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:32:57 dom0 kernel: Modules linked in: intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops
joydev videobuf2_v4l2 k10temp videobuf2_common videodev sp5100_tco mc
i2c_piix4 iwlwifi cfg80211 thinkpad_acpi platform_profile ipmi_devintf
ledtrig_audio ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi
rfkill typec wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool
dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder
hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel
gpu_sched i2c_algo_bit drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm
cqhci drm_kms_helper serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas
xhci_hcd drm nvme mmc_core ehci_pci ehci_hcd nvme_core
xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc
xen_gntdev xen_evtchn uinput
Oct 11 22:32:57 dom0 kernel: CPU: 5 PID: 1425 Comm: setfont Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:32:57 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:32:57 dom0 kernel: RIP:
e030:amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:32:57 dom0 kernel: Code: 8b 45 b0 48 c7 c7 a2 bc 8c c0 4c 89
55 88 8b b0 80 05 00 00 e8 e5 04 be ff 4c 8b 55 88 0f b6 55 ab 49 8b 72
08 e9 53 fa ff ff <0f> 0b e9 e2 fe ff ff e8 78 e4 82 c1 0f 1f 84 00 00
00 00 00 0f 1f
Oct 11 22:32:57 dom0 kernel: RSP: e02b:ffffc900403e7698 EFLAGS: 00010002
Oct 11 22:32:57 dom0 kernel: RAX: ffff888110500210 RBX: 00000000000010ac
RCX: 00000000000021e8
Oct 11 22:32:57 dom0 kernel: RDX: ffff88813f418e40 RSI: ffff888110500450
RDI: ffff88810ddf0f30
Oct 11 22:32:57 dom0 kernel: RBP: ffffc900403e7758 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:32:57 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:32:57 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810dee8cc0
R15: ffff88810ddada00
Oct 11 22:32:57 dom0 kernel: FS:  00007005ac91c580(0000)
GS:ffff88813f400000(0000) knlGS:0000000000000000
Oct 11 22:32:57 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:32:57 dom0 kernel: CR2: 000073c449584d48 CR3: 0000000104974000
CR4: 0000000000050660
Oct 11 22:32:57 dom0 kernel: Call Trace:
Oct 11 22:32:57 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:32:57 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:32:57 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:32:57 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:32:57 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:32:57 dom0 kernel:  drm_fb_helper_pan_display+0x9b/0x110
[drm_kms_helper]
Oct 11 22:32:57 dom0 kernel:  fb_pan_display+0x8f/0x110
Oct 11 22:32:57 dom0 kernel:  bit_update_start+0x1a/0x40
Oct 11 22:32:57 dom0 kernel:  fbcon_switch+0x357/0x500
Oct 11 22:32:57 dom0 kernel:  redraw_screen+0xe9/0x230
Oct 11 22:32:57 dom0 kernel:  fbcon_do_set_font+0x170/0x190
Oct 11 22:32:57 dom0 kernel:  con_font_op+0x156/0x250
Oct 11 22:32:57 dom0 kernel:  ? do_anonymous_page+0x1ec/0x3b0
Oct 11 22:32:57 dom0 kernel:  ? _copy_from_user+0x45/0x80
Oct 11 22:32:57 dom0 kernel:  vt_k_ioctl+0x1b7/0x630
Oct 11 22:32:57 dom0 kernel:  vt_ioctl+0x7d/0x660
Oct 11 22:32:57 dom0 kernel:  tty_ioctl+0x354/0x810
Oct 11 22:32:57 dom0 kernel:  ? __lock_release+0x181/0x2d0
Oct 11 22:32:57 dom0 kernel:  ? ktime_get_coarse_real_ts64+0xe/0x50
Oct 11 22:32:57 dom0 kernel:  ?
lockdep_hardirqs_on_prepare.part.0+0xbf/0x140
Oct 11 22:32:57 dom0 kernel:  ?
seqcount_lockdep_reader_access.constprop.0+0x84/0x90
Oct 11 22:32:57 dom0 kernel:  __x64_sys_ioctl+0x83/0xb0
Oct 11 22:32:57 dom0 kernel:  do_syscall_64+0x3b/0x90
Oct 11 22:32:57 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:32:57 dom0 kernel: RIP: 0033:0x7005ac84917b
Oct 11 22:32:57 dom0 kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7
00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8
10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7
d8 64 89 01 48
Oct 11 22:32:57 dom0 kernel: RSP: 002b:00007ffc7dee5a78 EFLAGS: 00000246
ORIG_RAX: 0000000000000010
Oct 11 22:32:57 dom0 kernel: RAX: ffffffffffffffda RBX: 0000000000000010
RCX: 00007005ac84917b
Oct 11 22:32:57 dom0 kernel: RDX: 00007ffc7dee5aa0 RSI: 0000000000004b72
RDI: 0000000000000003
Oct 11 22:32:57 dom0 kernel: RBP: 0000000000000200 R08: 0000000000000010
R09: 00007005ac914a40
Oct 11 22:32:57 dom0 kernel: R10: 0000000000000072 R11: 0000000000000246
R12: 0000000000000008
Oct 11 22:32:57 dom0 kernel: R13: 0000558e251606a0 R14: 0000000000000003
R15: 00007ffc7dee5aa0
Oct 11 22:32:57 dom0 kernel: irq event stamp: 13242
Oct 11 22:32:57 dom0 kernel: hardirqs last  enabled at (13241):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:32:57 dom0 kernel: hardirqs last disabled at (13242):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:32:57 dom0 kernel: softirqs last  enabled at (12464):
[<ffffffff8103b972>] fpu_clone+0x72/0x200
Oct 11 22:32:57 dom0 kernel: softirqs last disabled at (12462):
[<ffffffff8103b905>] fpu_clone+0x5/0x200
Oct 11 22:32:57 dom0 kernel: ---[ end trace 6fec6583c02534af ]---
Oct 11 22:32:57 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:32:57 dom0 kernel: WARNING: CPU: 5 PID: 1425 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8276
prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:32:57 dom0 kernel: Modules linked in: intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops
joydev videobuf2_v4l2 k10temp videobuf2_common videodev sp5100_tco mc
i2c_piix4 iwlwifi cfg80211 thinkpad_acpi platform_profile ipmi_devintf
ledtrig_audio ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi
rfkill typec wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool
dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder
hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel
gpu_sched i2c_algo_bit drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm
cqhci drm_kms_helper serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas
xhci_hcd drm nvme mmc_core ehci_pci ehci_hcd nvme_core
xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc
xen_gntdev xen_evtchn uinput
Oct 11 22:32:57 dom0 kernel: CPU: 5 PID: 1425 Comm: setfont Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:32:57 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:32:57 dom0 kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:32:57 dom0 kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00
66 90 c3 8b 97 80 05 00 00 48 c7 c6 d8 65 89 c0 48 c7 c7 d8 fa a1 c0 e9
0e ed 1f c1 0f 0b <0f> 0b eb b4 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
41 57 41 56 41
Oct 11 22:32:57 dom0 kernel: RSP: e02b:ffffc900403e7690 EFLAGS: 00010082
Oct 11 22:32:57 dom0 kernel: RAX: 0000000000000001 RBX: 00000000000010ac
RCX: 00000000000021e8
Oct 11 22:32:57 dom0 kernel: RDX: ffff88813f418e40 RSI: ffff888110500450
RDI: ffff88810d35e800
Oct 11 22:32:57 dom0 kernel: RBP: ffffc900403e7758 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:32:57 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:32:57 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810dee8cc0
R15: ffff88810ddada00
Oct 11 22:32:57 dom0 kernel: FS:  00007005ac91c580(0000)
GS:ffff88813f400000(0000) knlGS:0000000000000000
Oct 11 22:32:57 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:32:57 dom0 kernel: CR2: 000073c449584d48 CR3: 0000000104974000
CR4: 0000000000050660
Oct 11 22:32:57 dom0 kernel: Call Trace:
Oct 11 22:32:57 dom0 kernel:  amdgpu_dm_commit_planes+0x87d/0x9a0 [amdgpu]
Oct 11 22:32:57 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:32:57 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:32:57 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:32:57 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:32:57 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:32:57 dom0 kernel:  drm_fb_helper_pan_display+0x9b/0x110
[drm_kms_helper]
Oct 11 22:32:57 dom0 kernel:  fb_pan_display+0x8f/0x110
Oct 11 22:32:57 dom0 kernel:  bit_update_start+0x1a/0x40
Oct 11 22:32:57 dom0 kernel:  fbcon_switch+0x357/0x500
Oct 11 22:32:57 dom0 kernel:  redraw_screen+0xe9/0x230
Oct 11 22:32:57 dom0 kernel:  fbcon_do_set_font+0x170/0x190
Oct 11 22:32:57 dom0 kernel:  con_font_op+0x156/0x250
Oct 11 22:32:57 dom0 kernel:  ? do_anonymous_page+0x1ec/0x3b0
Oct 11 22:32:57 dom0 kernel:  ? _copy_from_user+0x45/0x80
Oct 11 22:32:57 dom0 kernel:  vt_k_ioctl+0x1b7/0x630
Oct 11 22:32:57 dom0 kernel:  vt_ioctl+0x7d/0x660
Oct 11 22:32:57 dom0 kernel:  tty_ioctl+0x354/0x810
Oct 11 22:32:57 dom0 kernel:  ? __lock_release+0x181/0x2d0
Oct 11 22:32:57 dom0 kernel:  ? ktime_get_coarse_real_ts64+0xe/0x50
Oct 11 22:32:57 dom0 kernel:  ?
lockdep_hardirqs_on_prepare.part.0+0xbf/0x140
Oct 11 22:32:57 dom0 kernel:  ?
seqcount_lockdep_reader_access.constprop.0+0x84/0x90
Oct 11 22:32:57 dom0 kernel:  __x64_sys_ioctl+0x83/0xb0
Oct 11 22:32:57 dom0 kernel:  do_syscall_64+0x3b/0x90
Oct 11 22:32:57 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:32:57 dom0 kernel: RIP: 0033:0x7005ac84917b
Oct 11 22:32:57 dom0 kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7
00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8
10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7
d8 64 89 01 48
Oct 11 22:32:57 dom0 kernel: RSP: 002b:00007ffc7dee5a78 EFLAGS: 00000246
ORIG_RAX: 0000000000000010
Oct 11 22:32:57 dom0 kernel: RAX: ffffffffffffffda RBX: 0000000000000010
RCX: 00007005ac84917b
Oct 11 22:32:57 dom0 kernel: RDX: 00007ffc7dee5aa0 RSI: 0000000000004b72
RDI: 0000000000000003
Oct 11 22:32:57 dom0 kernel: RBP: 0000000000000200 R08: 0000000000000010
R09: 00007005ac914a40
Oct 11 22:32:57 dom0 kernel: R10: 0000000000000072 R11: 0000000000000246
R12: 0000000000000008
Oct 11 22:32:57 dom0 kernel: R13: 0000558e251606a0 R14: 0000000000000003
R15: 00007ffc7dee5aa0
Oct 11 22:32:57 dom0 kernel: irq event stamp: 13242
Oct 11 22:32:57 dom0 kernel: hardirqs last  enabled at (13241):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:32:57 dom0 kernel: hardirqs last disabled at (13242):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:32:57 dom0 kernel: softirqs last  enabled at (12464):
[<ffffffff8103b972>] fpu_clone+0x72/0x200
Oct 11 22:32:57 dom0 kernel: softirqs last disabled at (12462):
[<ffffffff8103b905>] fpu_clone+0x5/0x200
Oct 11 22:32:57 dom0 kernel: ---[ end trace 6fec6583c02534b0 ]---
Oct 11 22:33:07 dom0 kernel: [drm:drm_atomic_helper_wait_for_flip_done
[drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out
Oct 11 22:33:18 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:33:18 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CRTC:67:crtc-0] commit wait timed out
Oct 11 22:33:28 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:33:28 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CONNECTOR:78:eDP-1] commit wait timed out
Oct 11 22:33:38 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:33:38 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[PLANE:55:plane-3] commit wait timed out
Oct 11 22:33:38 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:33:38 dom0 kernel: WARNING: CPU: 5 PID: 1427 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8689
amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:33:38 dom0 kernel: Modules linked in: intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops
joydev videobuf2_v4l2 k10temp videobuf2_common videodev sp5100_tco mc
i2c_piix4 iwlwifi cfg80211 thinkpad_acpi platform_profile ipmi_devintf
ledtrig_audio ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi
rfkill typec wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool
dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder
hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel
gpu_sched i2c_algo_bit drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm
cqhci drm_kms_helper serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas
xhci_hcd drm nvme mmc_core ehci_pci ehci_hcd nvme_core
xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc
xen_gntdev xen_evtchn uinput
Oct 11 22:33:38 dom0 kernel: CPU: 5 PID: 1427 Comm: setfont Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:33:38 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:33:38 dom0 kernel: RIP:
e030:amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:33:38 dom0 kernel: Code: 8b 45 b0 48 c7 c7 a2 bc 8c c0 4c 89
55 88 8b b0 80 05 00 00 e8 e5 04 be ff 4c 8b 55 88 0f b6 55 ab 49 8b 72
08 e9 53 fa ff ff <0f> 0b e9 e2 fe ff ff e8 78 e4 82 c1 0f 1f 84 00 00
00 00 00 0f 1f
Oct 11 22:33:38 dom0 kernel: RSP: e02b:ffffc90041613698 EFLAGS: 00010002
Oct 11 22:33:38 dom0 kernel: RAX: ffff888110500210 RBX: 0000000000001a46
RCX: 00000000000021e8
Oct 11 22:33:38 dom0 kernel: RDX: ffff88813f418e40 RSI: ffff888110500450
RDI: ffff888106330f30
Oct 11 22:33:38 dom0 kernel: RBP: ffffc90041613758 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:33:38 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:33:38 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810c39c0c0
R15: ffff88810de78c00
Oct 11 22:33:38 dom0 kernel: FS:  0000723fc3d77580(0000)
GS:ffff88813f400000(0000) knlGS:0000000000000000
Oct 11 22:33:38 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:33:38 dom0 kernel: CR2: 000070eb0f040520 CR3: 000000010aeaa000
CR4: 0000000000050660
Oct 11 22:33:38 dom0 kernel: Call Trace:
Oct 11 22:33:38 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:33:38 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:33:38 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:33:38 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:33:38 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:33:38 dom0 kernel:  drm_fb_helper_pan_display+0x9b/0x110
[drm_kms_helper]
Oct 11 22:33:38 dom0 kernel:  fb_pan_display+0x8f/0x110
Oct 11 22:33:38 dom0 kernel:  bit_update_start+0x1a/0x40
Oct 11 22:33:38 dom0 kernel:  fbcon_switch+0x357/0x500
Oct 11 22:33:38 dom0 kernel:  redraw_screen+0xe9/0x230
Oct 11 22:33:38 dom0 kernel:  fbcon_do_set_font+0x170/0x190
Oct 11 22:33:38 dom0 kernel:  con_font_op+0x156/0x250
Oct 11 22:33:38 dom0 kernel:  ? do_anonymous_page+0x1ec/0x3b0
Oct 11 22:33:38 dom0 kernel:  ? _copy_from_user+0x45/0x80
Oct 11 22:33:38 dom0 kernel:  vt_k_ioctl+0x1b7/0x630
Oct 11 22:33:38 dom0 kernel:  vt_ioctl+0x7d/0x660
Oct 11 22:33:38 dom0 kernel:  tty_ioctl+0x354/0x810
Oct 11 22:33:38 dom0 kernel:  ? __lock_release+0x181/0x2d0
Oct 11 22:33:38 dom0 kernel:  ? ktime_get_coarse_real_ts64+0xe/0x50
Oct 11 22:33:38 dom0 kernel:  ?
lockdep_hardirqs_on_prepare.part.0+0xbf/0x140
Oct 11 22:33:38 dom0 kernel:  ?
seqcount_lockdep_reader_access.constprop.0+0x84/0x90
Oct 11 22:33:38 dom0 kernel:  __x64_sys_ioctl+0x83/0xb0
Oct 11 22:33:38 dom0 kernel:  do_syscall_64+0x3b/0x90
Oct 11 22:33:38 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:33:38 dom0 kernel: RIP: 0033:0x723fc3ca417b
Oct 11 22:33:38 dom0 kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7
00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8
10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7
d8 64 89 01 48
Oct 11 22:33:38 dom0 kernel: RSP: 002b:00007ffc0f3b06a8 EFLAGS: 00000246
ORIG_RAX: 0000000000000010
Oct 11 22:33:38 dom0 kernel: RAX: ffffffffffffffda RBX: 0000000000000010
RCX: 0000723fc3ca417b
Oct 11 22:33:38 dom0 kernel: RDX: 00007ffc0f3b06d0 RSI: 0000000000004b72
RDI: 0000000000000003
Oct 11 22:33:38 dom0 kernel: RBP: 0000000000000200 R08: 0000000000000010
R09: 0000723fc3d6fa40
Oct 11 22:33:38 dom0 kernel: R10: 0000000000000072 R11: 0000000000000246
R12: 0000000000000008
Oct 11 22:33:38 dom0 kernel: R13: 00005750656256a0 R14: 0000000000000003
R15: 00007ffc0f3b06d0
Oct 11 22:33:38 dom0 kernel: irq event stamp: 9426
Oct 11 22:33:38 dom0 kernel: hardirqs last  enabled at (9425):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:33:38 dom0 kernel: hardirqs last disabled at (9426):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:33:38 dom0 kernel: softirqs last  enabled at (8654):
[<ffffffff8103b972>] fpu_clone+0x72/0x200
Oct 11 22:33:38 dom0 kernel: softirqs last disabled at (8652):
[<ffffffff8103b905>] fpu_clone+0x5/0x200
Oct 11 22:33:38 dom0 kernel: ---[ end trace 6fec6583c02534b1 ]---
Oct 11 22:33:38 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:33:38 dom0 kernel: WARNING: CPU: 5 PID: 1427 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8276
prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:33:38 dom0 kernel: Modules linked in: intel_rapl_msr wmi_bmof
intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops
joydev videobuf2_v4l2 k10temp videobuf2_common videodev sp5100_tco mc
i2c_piix4 iwlwifi cfg80211 thinkpad_acpi platform_profile ipmi_devintf
ledtrig_audio ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi
rfkill typec wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool
dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder
hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel
gpu_sched i2c_algo_bit drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm
cqhci drm_kms_helper serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas
xhci_hcd drm nvme mmc_core ehci_pci ehci_hcd nvme_core
xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc
xen_gntdev xen_evtchn uinput
Oct 11 22:33:38 dom0 kernel: CPU: 5 PID: 1427 Comm: setfont Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:33:38 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:33:38 dom0 kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:33:38 dom0 kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00
66 90 c3 8b 97 80 05 00 00 48 c7 c6 d8 65 89 c0 48 c7 c7 d8 fa a1 c0 e9
0e ed 1f c1 0f 0b <0f> 0b eb b4 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
41 57 41 56 41
Oct 11 22:33:38 dom0 kernel: RSP: e02b:ffffc90041613690 EFLAGS: 00010086
Oct 11 22:33:38 dom0 kernel: RAX: 0000000000000001 RBX: 0000000000001a46
RCX: 00000000000021e8
Oct 11 22:33:38 dom0 kernel: RDX: ffff88813f418e40 RSI: ffff888110500450
RDI: ffff88810d35e800
Oct 11 22:33:38 dom0 kernel: RBP: ffffc90041613758 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:33:38 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:33:38 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810c39c0c0
R15: ffff88810de78c00
Oct 11 22:33:38 dom0 kernel: FS:  0000723fc3d77580(0000)
GS:ffff88813f400000(0000) knlGS:0000000000000000
Oct 11 22:33:38 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:33:38 dom0 kernel: CR2: 000070eb0f040520 CR3: 000000010aeaa000
CR4: 0000000000050660
Oct 11 22:33:38 dom0 kernel: Call Trace:
Oct 11 22:33:38 dom0 kernel:  amdgpu_dm_commit_planes+0x87d/0x9a0 [amdgpu]
Oct 11 22:33:38 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:33:38 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:33:38 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:33:38 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:33:38 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:33:38 dom0 kernel:  drm_fb_helper_pan_display+0x9b/0x110
[drm_kms_helper]
Oct 11 22:33:38 dom0 kernel:  fb_pan_display+0x8f/0x110
Oct 11 22:33:38 dom0 kernel:  bit_update_start+0x1a/0x40
Oct 11 22:33:38 dom0 kernel:  fbcon_switch+0x357/0x500
Oct 11 22:33:38 dom0 kernel:  redraw_screen+0xe9/0x230
Oct 11 22:33:38 dom0 kernel:  fbcon_do_set_font+0x170/0x190
Oct 11 22:33:38 dom0 kernel:  con_font_op+0x156/0x250
Oct 11 22:33:38 dom0 kernel:  ? do_anonymous_page+0x1ec/0x3b0
Oct 11 22:33:38 dom0 kernel:  ? _copy_from_user+0x45/0x80
Oct 11 22:33:38 dom0 kernel:  vt_k_ioctl+0x1b7/0x630
Oct 11 22:33:38 dom0 kernel:  vt_ioctl+0x7d/0x660
Oct 11 22:33:38 dom0 kernel:  tty_ioctl+0x354/0x810
Oct 11 22:33:38 dom0 kernel:  ? __lock_release+0x181/0x2d0
Oct 11 22:33:38 dom0 kernel:  ? ktime_get_coarse_real_ts64+0xe/0x50
Oct 11 22:33:38 dom0 kernel:  ?
lockdep_hardirqs_on_prepare.part.0+0xbf/0x140
Oct 11 22:33:38 dom0 kernel:  ?
seqcount_lockdep_reader_access.constprop.0+0x84/0x90
Oct 11 22:33:38 dom0 kernel:  __x64_sys_ioctl+0x83/0xb0
Oct 11 22:33:38 dom0 kernel:  do_syscall_64+0x3b/0x90
Oct 11 22:33:38 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:33:38 dom0 kernel: RIP: 0033:0x723fc3ca417b
Oct 11 22:33:38 dom0 kernel: Code: 0f 1e fa 48 8b 05 1d ad 0c 00 64 c7
00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8
10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d ed ac 0c 00 f7
d8 64 89 01 48
Oct 11 22:33:38 dom0 kernel: RSP: 002b:00007ffc0f3b06a8 EFLAGS: 00000246
ORIG_RAX: 0000000000000010
Oct 11 22:33:38 dom0 kernel: RAX: ffffffffffffffda RBX: 0000000000000010
RCX: 0000723fc3ca417b
Oct 11 22:33:38 dom0 kernel: RDX: 00007ffc0f3b06d0 RSI: 0000000000004b72
RDI: 0000000000000003
Oct 11 22:33:38 dom0 kernel: RBP: 0000000000000200 R08: 0000000000000010
R09: 0000723fc3d6fa40
Oct 11 22:33:38 dom0 kernel: R10: 0000000000000072 R11: 0000000000000246
R12: 0000000000000008
Oct 11 22:33:38 dom0 kernel: R13: 00005750656256a0 R14: 0000000000000003
R15: 00007ffc0f3b06d0
Oct 11 22:33:38 dom0 kernel: irq event stamp: 9426
Oct 11 22:33:38 dom0 kernel: hardirqs last  enabled at (9425):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:33:38 dom0 kernel: hardirqs last disabled at (9426):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:33:38 dom0 kernel: softirqs last  enabled at (8654):
[<ffffffff8103b972>] fpu_clone+0x72/0x200
Oct 11 22:33:38 dom0 kernel: softirqs last disabled at (8652):
[<ffffffff8103b905>] fpu_clone+0x5/0x200
Oct 11 22:33:38 dom0 kernel: ---[ end trace 6fec6583c02534b2 ]---
Oct 11 22:33:48 dom0 kernel: [drm:drm_atomic_helper_wait_for_flip_done
[drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out
Oct 11 22:33:49 dom0 kernel: EXT4-fs (nvme0n1p2): mounted filesystem
with ordered data mode. Opts: discard. Quota mode: none.
Oct 11 22:33:56 dom0 kernel: [drm] Fence fallback timer expired on ring
sdma0
Oct 11 22:33:59 dom0 kernel: usb 6-1: USB disconnect, device number 2
Oct 11 22:34:06 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:34:06 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CRTC:67:crtc-0] commit wait timed out
Oct 11 22:34:17 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:34:17 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CONNECTOR:78:eDP-1] commit wait timed out
Oct 11 22:34:27 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:34:27 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[PLANE:55:plane-3] commit wait timed out
Oct 11 22:34:27 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:34:27 dom0 kernel: WARNING: CPU: 7 PID: 2636 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8689
amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:34:27 dom0 kernel: Modules linked in: nf_tables nfnetlink vfat
fat intel_rapl_msr wmi_bmof intel_rapl_common pcspkr uvcvideo
videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 k10temp
videobuf2_common videodev sp5100_tco mc i2c_piix4 iwlwifi cfg80211
thinkpad_acpi platform_profile ipmi_devintf ledtrig_audio
ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi rfkill typec
wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data
dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu
crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit
drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm cqhci drm_kms_helper
serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas xhci_hcd drm nvme
mmc_core ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
Oct 11 22:34:27 dom0 kernel: CPU: 7 PID: 2636 Comm: Xorg Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:34:27 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:34:27 dom0 kernel: RIP:
e030:amdgpu_dm_commit_planes+0x98c/0x9a0 [amdgpu]
Oct 11 22:34:27 dom0 kernel: Code: 8b 45 b0 48 c7 c7 a2 bc 8c c0 4c 89
55 88 8b b0 80 05 00 00 e8 e5 04 be ff 4c 8b 55 88 0f b6 55 ab 49 8b 72
08 e9 53 fa ff ff <0f> 0b e9 e2 fe ff ff e8 78 e4 82 c1 0f 1f 84 00 00
00 00 00 0f 1f
Oct 11 22:34:27 dom0 kernel: RSP: e02b:ffffc90042d8f960 EFLAGS: 00010002
Oct 11 22:34:27 dom0 kernel: RAX: ffff888110500210 RBX: 00000000000025ad
RCX: 00000000000021e8
Oct 11 22:34:27 dom0 kernel: RDX: ffff88813f818e40 RSI: ffff888110500450
RDI: ffff888104cec3f0
Oct 11 22:34:27 dom0 kernel: RBP: ffffc90042d8fa20 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:34:27 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:34:27 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810a677cc0
R15: ffff88810a6d5c00
Oct 11 22:34:27 dom0 kernel: FS:  00007f896429fa40(0000)
GS:ffff88813f800000(0000) knlGS:0000000000000000
Oct 11 22:34:27 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:34:27 dom0 kernel: CR2: 00007cc75690e000 CR3: 00000001062e6000
CR4: 0000000000050660
Oct 11 22:34:27 dom0 kernel: Call Trace:
Oct 11 22:34:27 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:34:27 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:34:27 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:34:27 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:34:27 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:34:27 dom0 kernel:  drm_client_modeset_commit+0x24/0x40 [drm]
Oct 11 22:34:27 dom0 kernel:  drm_fb_helper_lastclose+0x45/0x80
[drm_kms_helper]
Oct 11 22:34:27 dom0 kernel:  amdgpu_driver_lastclose_kms+0xa/0x10 [amdgpu]
Oct 11 22:34:27 dom0 kernel:  drm_release+0xe1/0x110 [drm]
Oct 11 22:34:27 dom0 kernel:  __fput+0x9d/0x260
Oct 11 22:34:27 dom0 kernel:  task_work_run+0x5c/0x90
Oct 11 22:34:27 dom0 kernel:  exit_to_user_mode_loop+0x1ce/0x1e0
Oct 11 22:34:27 dom0 kernel:  exit_to_user_mode_prepare+0xe3/0x150
Oct 11 22:34:27 dom0 kernel:  syscall_exit_to_user_mode+0x27/0x60
Oct 11 22:34:27 dom0 kernel:  do_syscall_64+0x48/0x90
Oct 11 22:34:27 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:34:27 dom0 kernel: RIP: 0033:0x7f896486ba17
Oct 11 22:34:27 dom0 kernel: Code: 00 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8
03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c
e8 f3 fb ff ff
Oct 11 22:34:27 dom0 kernel: RSP: 002b:00007fff9083b528 EFLAGS: 00000246
ORIG_RAX: 0000000000000003
Oct 11 22:34:27 dom0 kernel: RAX: 0000000000000000 RBX: 00005f57edbd09f0
RCX: 00007f896486ba17
Oct 11 22:34:27 dom0 kernel: RDX: 00005f57edbb1010 RSI: 00005f57edbd0b60
RDI: 0000000000000014
Oct 11 22:34:27 dom0 kernel: RBP: 0000000000000014 R08: 0000000000000000
R09: 00007f896484fa40
Oct 11 22:34:27 dom0 kernel: R10: 00005f57eba9c302 R11: 0000000000000246
R12: 00005f57edbd0b60
Oct 11 22:34:27 dom0 kernel: R13: 00005f57edbd0a30 R14: 0000000000000000
R15: 0000000000000000
Oct 11 22:34:27 dom0 kernel: irq event stamp: 51614
Oct 11 22:34:27 dom0 kernel: hardirqs last  enabled at (51613):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:34:27 dom0 kernel: hardirqs last disabled at (51614):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:34:27 dom0 kernel: softirqs last  enabled at (51264):
[<ffffffff820002f9>] __do_softirq+0x2f9/0x428
Oct 11 22:34:27 dom0 kernel: softirqs last disabled at (51257):
[<ffffffff810f50c0>] __irq_exit_rcu+0xd0/0x100
Oct 11 22:34:27 dom0 kernel: ---[ end trace 6fec6583c02534b3 ]---
Oct 11 22:34:27 dom0 kernel: ------------[ cut here ]------------
Oct 11 22:34:27 dom0 kernel: WARNING: CPU: 7 PID: 2636 at
drivers/gpu/drm/amd/amdgpu/../display/amdgpu_dm/amdgpu_dm.c:8276
prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:34:27 dom0 kernel: Modules linked in: nf_tables nfnetlink vfat
fat intel_rapl_msr wmi_bmof intel_rapl_common pcspkr uvcvideo
videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 k10temp
videobuf2_common videodev sp5100_tco mc i2c_piix4 iwlwifi cfg80211
thinkpad_acpi platform_profile ipmi_devintf ledtrig_audio
ipmi_msghandler snd r8169 ucsi_acpi soundcore typec_ucsi rfkill typec
wmi video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data
dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu
crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit
drm_ttm_helper sdhci_pci ghash_clmulni_intel ttm cqhci drm_kms_helper
serio_raw cec sdhci ccp xhci_pci xhci_pci_renesas xhci_hcd drm nvme
mmc_core ehci_pci ehci_hcd nvme_core xen_acpi_processor xen_privcmd
xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
Oct 11 22:34:27 dom0 kernel: CPU: 7 PID: 2636 Comm: Xorg Tainted:
G        W        --------- ---  5.15.0-1.fc32.qubes.x86_64 #14
Oct 11 22:34:27 dom0 kernel: Hardware name: LENOVO
20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 11 22:34:27 dom0 kernel: RIP: e030:prepare_flip_isr+0x64/0x70 [amdgpu]
Oct 11 22:34:27 dom0 kernel: Code: 00 48 c7 80 38 01 00 00 00 00 00 00
66 90 c3 8b 97 80 05 00 00 48 c7 c6 d8 65 89 c0 48 c7 c7 d8 fa a1 c0 e9
0e ed 1f c1 0f 0b <0f> 0b eb b4 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00
41 57 41 56 41
Oct 11 22:34:27 dom0 kernel: RSP: e02b:ffffc90042d8f958 EFLAGS: 00010082
Oct 11 22:34:27 dom0 kernel: RAX: 0000000000000001 RBX: 00000000000025ad
RCX: 00000000000021e8
Oct 11 22:34:27 dom0 kernel: RDX: ffff88813f818e40 RSI: ffff888110500450
RDI: ffff88810d35e800
Oct 11 22:34:27 dom0 kernel: RBP: ffffc90042d8fa20 R08: 0000000000000002
R09: 00000000000c04d8
Oct 11 22:34:27 dom0 kernel: R10: 0000000000000000 R11: 0000000000080000
R12: ffff888110500210
Oct 11 22:34:27 dom0 kernel: R13: ffff88810d35e800 R14: ffff88810a677cc0
R15: ffff88810a6d5c00
Oct 11 22:34:27 dom0 kernel: FS:  00007f896429fa40(0000)
GS:ffff88813f800000(0000) knlGS:0000000000000000
Oct 11 22:34:27 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0:
0000000080050033
Oct 11 22:34:27 dom0 kernel: CR2: 00007cc75690e000 CR3: 00000001062e6000
CR4: 0000000000050660
Oct 11 22:34:27 dom0 kernel: Call Trace:
Oct 11 22:34:27 dom0 kernel:  amdgpu_dm_commit_planes+0x87d/0x9a0 [amdgpu]
Oct 11 22:34:27 dom0 kernel:  amdgpu_dm_atomic_commit_tail+0xbb2/0x1200
[amdgpu]
Oct 11 22:34:27 dom0 kernel:  commit_tail+0x94/0x130 [drm_kms_helper]
Oct 11 22:34:27 dom0 kernel:  drm_atomic_helper_commit+0x136/0x160
[drm_kms_helper]
Oct 11 22:34:27 dom0 kernel: 
drm_client_modeset_commit_atomic+0x249/0x290 [drm]
Oct 11 22:34:27 dom0 kernel:  drm_client_modeset_commit_locked+0x58/0x80
[drm]
Oct 11 22:34:27 dom0 kernel:  drm_client_modeset_commit+0x24/0x40 [drm]
Oct 11 22:34:27 dom0 kernel:  drm_fb_helper_lastclose+0x45/0x80
[drm_kms_helper]
Oct 11 22:34:27 dom0 kernel:  amdgpu_driver_lastclose_kms+0xa/0x10 [amdgpu]
Oct 11 22:34:27 dom0 kernel:  drm_release+0xe1/0x110 [drm]
Oct 11 22:34:27 dom0 kernel:  __fput+0x9d/0x260
Oct 11 22:34:27 dom0 kernel:  task_work_run+0x5c/0x90
Oct 11 22:34:27 dom0 kernel:  exit_to_user_mode_loop+0x1ce/0x1e0
Oct 11 22:34:27 dom0 kernel:  exit_to_user_mode_prepare+0xe3/0x150
Oct 11 22:34:27 dom0 kernel:  syscall_exit_to_user_mode+0x27/0x60
Oct 11 22:34:27 dom0 kernel:  do_syscall_64+0x48/0x90
Oct 11 22:34:27 dom0 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xae
Oct 11 22:34:27 dom0 kernel: RIP: 0033:0x7f896486ba17
Oct 11 22:34:27 dom0 kernel: Code: 00 00 f7 d8 64 89 02 48 c7 c0 ff ff
ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8
03 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c
e8 f3 fb ff ff
Oct 11 22:34:27 dom0 kernel: RSP: 002b:00007fff9083b528 EFLAGS: 00000246
ORIG_RAX: 0000000000000003
Oct 11 22:34:27 dom0 kernel: RAX: 0000000000000000 RBX: 00005f57edbd09f0
RCX: 00007f896486ba17
Oct 11 22:34:27 dom0 kernel: RDX: 00005f57edbb1010 RSI: 00005f57edbd0b60
RDI: 0000000000000014
Oct 11 22:34:27 dom0 kernel: RBP: 0000000000000014 R08: 0000000000000000
R09: 00007f896484fa40
Oct 11 22:34:27 dom0 kernel: R10: 00005f57eba9c302 R11: 0000000000000246
R12: 00005f57edbd0b60
Oct 11 22:34:27 dom0 kernel: R13: 00005f57edbd0a30 R14: 0000000000000000
R15: 0000000000000000
Oct 11 22:34:27 dom0 kernel: irq event stamp: 51614
Oct 11 22:34:27 dom0 kernel: hardirqs last  enabled at (51613):
[<ffffffff81df81c7>] _raw_spin_unlock_irqrestore+0x37/0x40
Oct 11 22:34:27 dom0 kernel: hardirqs last disabled at (51614):
[<ffffffff81df7fc5>] _raw_spin_lock_irqsave+0x75/0x90
Oct 11 22:34:27 dom0 kernel: softirqs last  enabled at (51264):
[<ffffffff820002f9>] __do_softirq+0x2f9/0x428
Oct 11 22:34:27 dom0 kernel: softirqs last disabled at (51257):
[<ffffffff810f50c0>] __irq_exit_rcu+0xd0/0x100
Oct 11 22:34:27 dom0 kernel: ---[ end trace 6fec6583c02534b4 ]---
Oct 11 22:34:37 dom0 kernel: [drm:drm_atomic_helper_wait_for_flip_done
[drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out
Oct 11 22:34:38 dom0 kernel: [drm] Fence fallback timer expired on ring
sdma0
Oct 11 22:34:48 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:34:48 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CRTC:67:crtc-0] commit wait timed out
Oct 11 22:34:58 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:34:58 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[CONNECTOR:78:eDP-1] commit wait timed out
Oct 11 22:35:08 dom0 kernel: [drm:drm_crtc_commit_wait [drm]] *ERROR*
flip_done timed out
Oct 11 22:35:08 dom0 kernel:
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR*
[PLANE:55:plane-3] commit wait timed out
Oct 11 22:35:08 dom0 kernel: ------------[ cut here ]------------

Full log with drm.debug=1 can be be found at the original issue
https://gitlab.freedesktop.org/drm/amd/-/issues/1715
https://gitlab.freedesktop.org/drm/amd/uploads/c91af638cb8fb5f3c25130f1edabb47e/dmesg


My original goal was to solve suspend/resume, that is why I'm testing
out v5.15. During that investigation I have found that:
* v5.15-rc2+ with pci=nomsi is quite resilient to crashing amdgpu drivers
* Compiling with CONFIG_HSA_AMD=n makes a difference, I don't believe Xen
  has proper support for HSA anyhow.
* Compiling with CONFIG_AMD_PMC=y makes a difference, apparently qiurks
  are enabled when not compiling as a module.

I did a last honest try with v5.15-rc5 with this patch applied to get more
data and I did find some more. The original problem show itself, but since
amdgpu actually describes their firmware commands now, it became less cryptic.

After this message the screen will be blank, I can still get passed the xscreensaver
but the screen is dead. With pci=nomsi applied I can after suspending the laptop
once more, get the screen back in order. This also occurs without starting X.

Oct 17 22:49:11 dom0 kernel: nvme 0000:01:00.0: saving config space at offset 0x38 (reading 0x0)
Oct 17 22:49:11 dom0 kernel: nvme 0000:01:00.0: saving config space at offset 0x3c (reading 0x1ff)
Oct 17 22:49:11 dom0 kernel: usb 4-4: reset full-speed USB device number 3 using xhci_hcd
Oct 17 22:49:11 dom0 kernel: nvme nvme0: 8/0/0 default/read/poll queues
Oct 17 22:49:11 dom0 kernel: usb 1-2: reset high-speed USB device number 2 using xhci_hcd
Oct 17 22:49:11 dom0 kernel: psmouse serio1: synaptics: queried max coordinates: x [..5678], y [..4694]
Oct 17 22:49:11 dom0 kernel: psmouse serio1: synaptics: queried min coordinates: x [1266..], y [1162..]
Oct 17 22:49:11 dom0 kernel: [drm] psp gfx command SETUP_TMR(0x5) failed and response status is (0x0)
Oct 17 22:49:11 dom0 kernel: [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed!
Oct 17 22:49:11 dom0 kernel: [drm:psp_resume [amdgpu]] *ERROR* PSP resume failed
Oct 17 22:49:11 dom0 kernel: [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* resume of IP block <psp> failed -22
Oct 17 22:49:11 dom0 kernel: amdgpu 0000:07:00.0: amdgpu: amdgpu_device_ip_resume failed (-22).
Oct 17 22:49:11 dom0 kernel: PM: dpm_run_callback(): pci_pm_resume+0x0/0xe0 returns -22
Oct 17 22:49:11 dom0 kernel: amdgpu 0000:07:00.0: PM: failed to resume async: error -22
Oct 17 22:49:11 dom0 kernel: PM: resume devices took 2.422 seconds
Oct 17 22:49:11 dom0 kernel: OOM killer enabled.
Oct 17 22:49:11 dom0 kernel: Restarting tasks ... done.
Oct 17 22:49:11 dom0 kernel: PM: suspend exit

Since I also had _a lot_ of debug-flags applied I also got this neat deadlock

Oct 17 22:52:07 dom0 kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 17 22:52:07 dom0 kernel: ACPI: \_SB_.PLTF.C002: Found 3 idle states
Oct 17 22:52:07 dom0 kernel: ACPI: FW issue: working around C-state latencies out of order
Oct 17 22:52:07 dom0 kernel: CPU2 is up
Oct 17 22:52:07 dom0 kernel: ------------[ cut here ]------------
Oct 17 22:52:07 dom0 kernel: installing Xen timer for CPU 3
Oct 17 22:52:07 dom0 kernel: 
Oct 17 22:52:07 dom0 kernel: ======================================================
Oct 17 22:52:07 dom0 kernel: WARNING: possible circular locking dependency detected
Oct 17 22:52:07 dom0 kernel: 5.15.0-0.rc5.0.fc32.qubes.x86_64 #1 Tainted: G        W        --------- --- 
Oct 17 22:52:07 dom0 kernel: ------------------------------------------------------
Oct 17 22:52:07 dom0 kernel: kworker/2:0/11917 is trying to acquire lock:
Oct 17 22:52:07 dom0 kernel: ffffffff82962858 ((console_sem).lock){-...}-{2:2}, at: down_trylock+0xf/0x30
Oct 17 22:52:07 dom0 kernel: 
                             but task is already holding lock:
Oct 17 22:52:07 dom0 kernel: ffff8881406ab558 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x1e/0x80
Oct 17 22:52:07 dom0 kernel: 
                             which lock already depends on the new lock.
Oct 17 22:52:07 dom0 kernel: 
                             the existing dependency chain (in reverse order) is:
Oct 17 22:52:07 dom0 kernel: 
                             -> #2 (&rq->__lock){-.-.}-{2:2}:
Oct 17 22:52:07 dom0 kernel:        __lock_acquire+0x3a0/0x6b0
Oct 17 22:52:07 dom0 kernel:        lock_acquire+0xf5/0x300
Oct 17 22:52:07 dom0 kernel:        _raw_spin_lock_nested+0x2a/0x40
Oct 17 22:52:07 dom0 kernel:        raw_spin_rq_lock_nested+0x1e/0x80
Oct 17 22:52:07 dom0 kernel:        task_fork_fair+0x39/0x180
Oct 17 22:52:07 dom0 kernel:        sched_fork+0x115/0x290
Oct 17 22:52:07 dom0 kernel:        copy_process+0xd2e/0x2b80
Oct 17 22:52:07 dom0 kernel:        kernel_clone+0xa4/0x300
Oct 17 22:52:07 dom0 kernel:        kernel_thread+0x55/0x70
Oct 17 22:52:07 dom0 kernel:        rest_init+0x1e/0x280
Oct 17 22:52:07 dom0 kernel:        start_kernel+0x65d/0x69b
Oct 17 22:52:07 dom0 kernel:        xen_start_kernel+0x5fb/0x61c
Oct 17 22:52:07 dom0 kernel:        reset_early_page_tables+0x0/0x9d
Oct 17 22:52:07 dom0 kernel: 
                             -> #1 (&p->pi_lock){-.-.}-{2:2}:
Oct 17 22:52:07 dom0 kernel:        __lock_acquire+0x3a0/0x6b0
Oct 17 22:52:07 dom0 kernel:        lock_acquire+0xf5/0x300
Oct 17 22:52:07 dom0 kernel:        _raw_spin_lock_irqsave+0x48/0x60
Oct 17 22:52:07 dom0 kernel:        try_to_wake_up+0x53/0x5f0
Oct 17 22:52:07 dom0 kernel:        up+0x40/0x60
Oct 17 22:52:07 dom0 kernel:        __up_console_sem+0x56/0x70
Oct 17 22:52:07 dom0 kernel:        console_unlock+0x2ae/0x3c0
Oct 17 22:52:07 dom0 kernel:        vprintk_emit+0x141/0x160
Oct 17 22:52:07 dom0 kernel:        _printk+0x68/0x7f
Oct 17 22:52:07 dom0 kernel:        acpi_register_gsi_xen.cold+0x61/0x81
Oct 17 22:52:07 dom0 kernel:        acpi_pci_irq_enable+0xdd/0x240
Oct 17 22:52:07 dom0 kernel:        do_pci_enable_device+0x8a/0x110
Oct 17 22:52:07 dom0 kernel:        pci_enable_device_flags+0xcf/0x100
Oct 17 22:52:07 dom0 kernel:        nvme_pci_enable+0x28/0x1e0 [nvme]
Oct 17 22:52:07 dom0 kernel:        nvme_reset_work+0x61/0x490 [nvme]
Oct 17 22:52:07 dom0 kernel:        process_one_work+0x294/0x590
Oct 17 22:52:07 dom0 kernel:        worker_thread+0x49/0x310
Oct 17 22:52:07 dom0 kernel:        kthread+0x120/0x140
Oct 17 22:52:07 dom0 kernel:        ret_from_fork+0x22/0x30
Oct 17 22:52:07 dom0 kernel: 
                             -> #0 ((console_sem).lock){-...}-{2:2}:
Oct 17 22:52:07 dom0 kernel:        check_prev_add+0x8f/0xbf0
Oct 17 22:52:07 dom0 kernel:        validate_chain+0x38a/0x420
Oct 17 22:52:07 dom0 kernel:        __lock_acquire+0x3a0/0x6b0
Oct 17 22:52:07 dom0 kernel:        lock_acquire+0xf5/0x300
Oct 17 22:52:07 dom0 kernel:        _raw_spin_lock_irqsave+0x48/0x60
Oct 17 22:52:07 dom0 kernel:        down_trylock+0xf/0x30
Oct 17 22:52:07 dom0 kernel:        __down_trylock_console_sem+0x32/0xa0
Oct 17 22:52:07 dom0 kernel:        console_trylock_spinning+0x13/0x1e0
Oct 17 22:52:07 dom0 kernel:        vprintk_emit+0xa8/0x160
Oct 17 22:52:07 dom0 kernel:        _printk+0x68/0x7f
Oct 17 22:52:07 dom0 kernel:        __warn_printk+0x51/0x93
Oct 17 22:52:07 dom0 kernel:        __update_blocked_fair+0x4f4/0x510
Oct 17 22:52:07 dom0 kernel:        update_blocked_averages+0xe3/0x280
Oct 17 22:52:07 dom0 kernel:        newidle_balance+0x160/0x600
Oct 17 22:52:07 dom0 kernel:        pick_next_task_fair+0x39/0x3f0
Oct 17 22:52:07 dom0 kernel:        pick_next_task+0x4c/0xbb0
Oct 17 22:52:07 dom0 kernel:        __schedule+0x135/0x600
Oct 17 22:52:07 dom0 kernel:        schedule+0x59/0xc0
Oct 17 22:52:07 dom0 kernel:        worker_thread+0xb3/0x310
Oct 17 22:52:07 dom0 kernel:        kthread+0x120/0x140
Oct 17 22:52:07 dom0 kernel:        ret_from_fork+0x22/0x30
Oct 17 22:52:07 dom0 kernel: 
                             other info that might help us debug this:
Oct 17 22:52:07 dom0 kernel: Chain exists of:
                               (console_sem).lock --> &p->pi_lock --> &rq->__lock
Oct 17 22:52:07 dom0 kernel:  Possible unsafe locking scenario:
Oct 17 22:52:07 dom0 kernel:        CPU0                    CPU1
Oct 17 22:52:07 dom0 kernel:        ----                    ----
Oct 17 22:52:07 dom0 kernel:   lock(&rq->__lock);
Oct 17 22:52:07 dom0 kernel:                                lock(&p->pi_lock);
Oct 17 22:52:07 dom0 kernel:                                lock(&rq->__lock);
Oct 17 22:52:07 dom0 kernel:   lock((console_sem).lock);
Oct 17 22:52:07 dom0 kernel: 
                              *** DEADLOCK ***
Oct 17 22:52:07 dom0 kernel: 1 lock held by kworker/2:0/11917:
Oct 17 22:52:07 dom0 kernel:  #0: ffff8881406ab558 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x1e/0x80
Oct 17 22:52:07 dom0 kernel: 
                             stack backtrace:
Oct 17 22:52:07 dom0 kernel: CPU: 2 PID: 11917 Comm: kworker/2:0 Tainted: G        W        --------- ---  5.15.0-0.rc5.0.fc32.qubes.x86_64 #1
Oct 17 22:52:07 dom0 kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 17 22:52:07 dom0 kernel: Workqueue:  0x0 (events)
Oct 17 22:52:07 dom0 kernel: Call Trace:
Oct 17 22:52:07 dom0 kernel:  dump_stack_lvl+0x57/0x72
Oct 17 22:52:07 dom0 kernel:  check_noncircular+0x10a/0x120
Oct 17 22:52:07 dom0 kernel:  check_prev_add+0x8f/0xbf0
Oct 17 22:52:07 dom0 kernel:  ? add_chain_cache+0x10d/0x2d0
Oct 17 22:52:07 dom0 kernel:  ? load_balance+0x2b0/0x7f0
Oct 17 22:52:07 dom0 kernel:  validate_chain+0x38a/0x420
Oct 17 22:52:07 dom0 kernel:  __lock_acquire+0x3a0/0x6b0
Oct 17 22:52:07 dom0 kernel:  lock_acquire+0xf5/0x300
Oct 17 22:52:07 dom0 kernel:  ? down_trylock+0xf/0x30
Oct 17 22:52:07 dom0 kernel:  ? vprintk_emit+0xa8/0x160
Oct 17 22:52:07 dom0 kernel:  _raw_spin_lock_irqsave+0x48/0x60
Oct 17 22:52:07 dom0 kernel:  ? down_trylock+0xf/0x30
Oct 17 22:52:07 dom0 kernel:  down_trylock+0xf/0x30
Oct 17 22:52:07 dom0 kernel:  ? vprintk_emit+0xa8/0x160
Oct 17 22:52:07 dom0 kernel:  __down_trylock_console_sem+0x32/0xa0
Oct 17 22:52:07 dom0 kernel:  console_trylock_spinning+0x13/0x1e0
Oct 17 22:52:07 dom0 kernel:  vprintk_emit+0xa8/0x160
Oct 17 22:52:07 dom0 kernel:  _printk+0x68/0x7f
Oct 17 22:52:07 dom0 kernel:  ? lock_is_held_type+0xa5/0x120
Oct 17 22:52:07 dom0 kernel:  __warn_printk+0x51/0x93
Oct 17 22:52:07 dom0 kernel:  ? lock_is_held_type+0xa5/0x120
Oct 17 22:52:07 dom0 kernel:  __update_blocked_fair+0x4f4/0x510
Oct 17 22:52:07 dom0 kernel:  update_blocked_averages+0xe3/0x280
Oct 17 22:52:07 dom0 kernel:  newidle_balance+0x160/0x600
Oct 17 22:52:07 dom0 kernel:  pick_next_task_fair+0x39/0x3f0
Oct 17 22:52:07 dom0 kernel:  pick_next_task+0x4c/0xbb0
Oct 17 22:52:07 dom0 kernel:  ? dequeue_task_fair+0xd1/0x4a0
Oct 17 22:52:07 dom0 kernel:  __schedule+0x135/0x600
Oct 17 22:52:07 dom0 kernel:  schedule+0x59/0xc0
Oct 17 22:52:07 dom0 kernel:  worker_thread+0xb3/0x310
Oct 17 22:52:07 dom0 kernel:  ? process_one_work+0x590/0x590
Oct 17 22:52:07 dom0 kernel:  kthread+0x120/0x140
Oct 17 22:52:07 dom0 kernel:  ? set_kthread_struct+0x40/0x40
Oct 17 22:52:07 dom0 kernel:  ret_from_fork+0x22/0x30
Oct 17 22:52:07 dom0 kernel: cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
Oct 17 22:52:07 dom0 kernel: WARNING: CPU: 2 PID: 11917 at kernel/sched/fair.c:3339 __update_blocked_fair+0x4f4/0x510
Oct 17 22:52:07 dom0 kernel: Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common uvcvideo pcspkr videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 videobuf2_common sp5100_tco k10temp i2c_piix4 videodev mc iwlwifi cfg80211 ipmi_devintf ipmi_msghandler thinkpad_acpi platform_profile ledtrig_audio rfkill snd r8169 soundcore video ucsi_acpi i2c_scmi typec_ucsi wmi typec fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel gpu_sched i2c_algo_bit serio_raw nvme ehci_pci drm_ttm_helper sdhci_pci ttm cqhci ehci_hcd drm_kms_helper sdhci nvme_core xhci_pci cec xhci_pci_renesas ccp mmc_core drm xhci_hcd xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
Oct 17 22:52:07 dom0 kernel: CPU: 2 PID: 11917 Comm: kworker/2:0 Tainted: G        W        --------- ---  5.15.0-0.rc5.0.fc32.qubes.x86_64 #1
Oct 17 22:52:07 dom0 kernel: Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Oct 17 22:52:07 dom0 kernel: Workqueue:  0x0 (events)
Oct 17 22:52:07 dom0 kernel: RIP: e030:__update_blocked_fair+0x4f4/0x510
Oct 17 22:52:07 dom0 kernel: Code: 01 00 00 48 89 90 08 0a 00 00 e9 e6 fc ff ff 45 31 ff e9 70 ff ff ff 48 c7 c7 c8 dc 5f 82 c6 05 a7 5f ab 01 01 e8 73 1c c4 00 <0f> 0b 41 8b 86 78 01 00 00 e9 a5 fc ff ff 66 66 2e 0f 1f 84 00 00
Oct 17 22:52:07 dom0 kernel: RSP: e02b:ffffc90040627cc0 EFLAGS: 00010082
Oct 17 22:52:07 dom0 kernel: RAX: 0000000000000000 RBX: ffff8881406abdb8 RCX: ffff888140698dd8
Oct 17 22:52:07 dom0 kernel: RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff888140698dd0
Oct 17 22:52:07 dom0 kernel: RBP: ffff8881406ab780 R08: 0000000000000001 R09: 0000000000000000
Oct 17 22:52:07 dom0 kernel: R10: 0000000000000000 R11: ffffffffffffffff R12: 0000000000000010
Oct 17 22:52:07 dom0 kernel: R13: ffff8881406abf38 R14: ffff8881406ab600 R15: 0000011e9f271699
Oct 17 22:52:07 dom0 kernel: FS:  0000000000000000(0000) GS:ffff888140680000(0000) knlGS:0000000000000000
Oct 17 22:52:07 dom0 kernel: CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 17 22:52:07 dom0 kernel: CR2: 00006300c5f02360 CR3: 0000000002826000 CR4: 0000000000050660
Oct 17 22:52:07 dom0 kernel: Call Trace:
Oct 17 22:52:07 dom0 kernel:  update_blocked_averages+0xe3/0x280
Oct 17 22:52:07 dom0 kernel:  newidle_balance+0x160/0x600
Oct 17 22:52:07 dom0 kernel:  pick_next_task_fair+0x39/0x3f0
Oct 17 22:52:07 dom0 kernel:  pick_next_task+0x4c/0xbb0
Oct 17 22:52:07 dom0 kernel:  ? dequeue_task_fair+0xd1/0x4a0
Oct 17 22:52:07 dom0 kernel:  __schedule+0x135/0x600
Oct 17 22:52:07 dom0 kernel:  schedule+0x59/0xc0
Oct 17 22:52:07 dom0 kernel:  worker_thread+0xb3/0x310
Oct 17 22:52:07 dom0 kernel:  ? process_one_work+0x590/0x590
Oct 17 22:52:07 dom0 kernel:  kthread+0x120/0x140
Oct 17 22:52:07 dom0 kernel:  ? set_kthread_struct+0x40/0x40
Oct 17 22:52:07 dom0 kernel:  ret_from_fork+0x22/0x30
Oct 17 22:52:07 dom0 kernel: irq event stamp: 526
Oct 17 22:52:07 dom0 kernel: hardirqs last  enabled at (525): [<ffffffff81dfd104>] _raw_spin_unlock_irq+0x24/0x40
Oct 17 22:52:07 dom0 kernel: hardirqs last disabled at (526): [<ffffffff81df513a>] __schedule+0x3aa/0x600
Oct 17 22:52:07 dom0 kernel: softirqs last  enabled at (420): [<ffffffff811e69f1>] css_free_rwork_fn+0x71/0x350
Oct 17 22:52:07 dom0 kernel: softirqs last disabled at (418): [<ffffffff811e69d6>] css_free_rwork_fn+0x56/0x350
Oct 17 22:52:07 dom0 kernel: ---[ end trace 276648458889a290 ]---
Oct 17 22:52:07 dom0 kernel: cpu 3 spinlock event irq 79
Oct 17 22:52:07 dom0 kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 17 22:52:07 dom0 kernel: ACPI: \_SB_.PLTF.C003: Found 3 idle states
Oct 17 22:52:07 dom0 kernel: ACPI: FW issue: working around C-state latencies out of order
Oct 17 22:52:07 dom0 kernel: CPU3 is up
Oct 17 22:52:07 dom0 kernel: installing Xen timer for CPU 4
Oct 17 22:52:07 dom0 kernel: cpu 4 spinlock event irq 85
Oct 17 22:52:07 dom0 kernel: [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
Oct 17 22:52:07 dom0 kernel: ACPI: \_SB_.PLTF.C004: Found 3 idle states
Oct 17 22:52:07 dom0 kernel: ACPI: FW issue: working around C-state latencies out of order
 
Found in full:
https://gitlab.freedesktop.org/drm/amd/uploads/478771dfe18ff303f072fcac9a3da16b/setup-tmr-failed.dmesg


When booting without Xen things just work in v5.15. Here's a fresh lspci and dmesg
from a working boot without Xen:

https://gitlab.freedesktop.org/drm/amd/uploads/81059ddc533de1c7cf21fa98b76f217d/without-xen.lspci
https://gitlab.freedesktop.org/drm/amd/uploads/f3df205cbd9e9e8b769c35f374f20f9d/without-xen.dmesg

Previous mailing list discussion:
https://lore.kernel.org/linux-pci/CAKf6xpvGyCKVHsvauP54=0j10fxis4XiiqBNWH+1cpkbtt_QJw@mail.gmail.com/


diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 0099a00af361..355b791e382f 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 {
 	void __iomem *desc_addr = pci_msix_desc_addr(desc);
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 }

--
2.31.1

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Marc Zyngier]

On Tue, 19 Oct 2021 22:48:19 +0100,
Josef Johansson <josef@oderland.se> wrote:
> 
> From: Josef Johansson <josef@oderland.se>
> 
> 
> PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
>     
> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions") introduce functions pci_msi_update_mask() and 
> pci_msix_write_vector_ctrl() that is missing checks for
> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> new mask/unmask functions"). Add them back since it is
> causing severe lockups in amdgpu drivers under Xen during boot.
> 
> As explained in commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
> on Xen PV"), when running as Xen PV guest, masking MSI-X is a 
> responsibility of the hypervisor.
> 
> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions")
> Suggested-by: Jason Andryuk <jandryuk@gmail.com>
> Signed-off-by: Josef Johansson <josef@oderland.se>
>

[...]

>
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 0099a00af361..355b791e382f 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>  	unsigned long flags;
>  
> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +		return;
> +

I'd rather be consistent, and keep the check outside of
pci_msi_update_mask(), just like we do in __pci_msi_mask_desc().
Something like this instead:

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 0099a00af361..6c69eab304ce 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -420,7 +420,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	pci_msi_update_mask(entry, 0, 0);
+	if (!(pci_msi_ignore_mask || desc->msi_attrib.is_virtual))
+		pci_msi_update_mask(entry, 0, 0);
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);

But the commit message talks about MSI-X, and the above is MSI
only. Is Xen messing with the former, the latter, or both?

>  	raw_spin_lock_irqsave(lock, flags);
>  	desc->msi_mask &= ~clear;
>  	desc->msi_mask |= set;
> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>  {
>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>  
> +	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +		return;
> +
>  	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  }

I have similar reservations for this one.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Jason Andryuk]

Hi, Marc,

Adding Juergen and Boris since this involves Xen.

On Wed, Oct 20, 2021 at 8:51 AM Marc Zyngier <maz@kernel.org> wrote:
>
> On Tue, 19 Oct 2021 22:48:19 +0100,
> Josef Johansson <josef@oderland.se> wrote:
> >
> > From: Josef Johansson <josef@oderland.se>
> >
> >
> > PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
> >
> > commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> > functions") introduce functions pci_msi_update_mask() and
> > pci_msix_write_vector_ctrl() that is missing checks for
> > pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> > new mask/unmask functions"). Add them back since it is
> > causing severe lockups in amdgpu drivers under Xen during boot.
> >
> > As explained in commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
> > on Xen PV"), when running as Xen PV guest, masking MSI-X is a
> > responsibility of the hypervisor.
> >
> > Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> > functions")
> > Suggested-by: Jason Andryuk <jandryuk@gmail.com>
> > Signed-off-by: Josef Johansson <josef@oderland.se>
> >
>
> [...]
>
> >
> > diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> > index 0099a00af361..355b791e382f 100644
> > --- a/drivers/pci/msi.c
> > +++ b/drivers/pci/msi.c
> > @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
> >       raw_spinlock_t *lock = &desc->dev->msi_lock;
> >       unsigned long flags;
> >
> > +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> > +             return;
> > +
>
> I'd rather be consistent, and keep the check outside of
> pci_msi_update_mask(), just like we do in __pci_msi_mask_desc().
> Something like this instead:
>
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 0099a00af361..6c69eab304ce 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -420,7 +420,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
>         arch_restore_msi_irqs(dev);
>
>         pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
> -       pci_msi_update_mask(entry, 0, 0);
> +       if (!(pci_msi_ignore_mask || desc->msi_attrib.is_virtual))
> +               pci_msi_update_mask(entry, 0, 0);
>         control &= ~PCI_MSI_FLAGS_QSIZE;
>         control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
>         pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
>
> But the commit message talks about MSI-X, and the above is MSI
> only. Is Xen messing with the former, the latter, or both?

My understanding is pci_msi_ignore_mask covers both MSI and MSI-X for Xen.

> >       raw_spin_lock_irqsave(lock, flags);
> >       desc->msi_mask &= ~clear;
> >       desc->msi_mask |= set;
> > @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
> >  {
> >       void __iomem *desc_addr = pci_msix_desc_addr(desc);
> >
> > +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> > +             return;
> > +
> >       writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
> >  }
>
> I have similar reservations for this one.

The problem here is some of the changes in commit 446a98b19fd6
("PCI/MSI: Use new mask/unmask functions") bypass the checks in
__pci_msi_mask_desc/__pci_msi_unmask_desc.  I've wondered if it would
be cleaner to push all the `if (pci_msi_ignore_mask)` checks down to
the place of the writes.  That keeps dropping the write local to the
write and leaves the higher level code consistent between the regular
and Xen PV cases.  I don't know where checking
desc->msi_attrib.is_virtual is appropriate.

Regards,
Jason

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

On 10/20/21 16:03, Jason Andryuk wrote:
> Hi, Marc,
>
> Adding Juergen and Boris since this involves Xen.
>
> On Wed, Oct 20, 2021 at 8:51 AM Marc Zyngier <maz@kernel.org> wrote:
>> On Tue, 19 Oct 2021 22:48:19 +0100,
>> Josef Johansson <josef@oderland.se> wrote:
>>> From: Josef Johansson <josef@oderland.se>
>>>
>>>
>>> PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
>>>
>>> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>>> functions") introduce functions pci_msi_update_mask() and
>>> pci_msix_write_vector_ctrl() that is missing checks for
>>> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
>>> new mask/unmask functions"). Add them back since it is
>>> causing severe lockups in amdgpu drivers under Xen during boot.
>>>
>>> As explained in commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
>>> on Xen PV"), when running as Xen PV guest, masking MSI-X is a
>>> responsibility of the hypervisor.
>>>
>>> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>>> functions")
>>> Suggested-by: Jason Andryuk <jandryuk@gmail.com>
>>> Signed-off-by: Josef Johansson <josef@oderland.se>
>>>
>> [...]
>>
>>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>>> index 0099a00af361..355b791e382f 100644
>>> --- a/drivers/pci/msi.c
>>> +++ b/drivers/pci/msi.c
>>> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>>>       raw_spinlock_t *lock = &desc->dev->msi_lock;
>>>       unsigned long flags;
>>>
>>> +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>> +             return;
>>> +
>> I'd rather be consistent, and keep the check outside of
>> pci_msi_update_mask(), just like we do in __pci_msi_mask_desc().
>> Something like this instead:
>>
>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>> index 0099a00af361..6c69eab304ce 100644
>> --- a/drivers/pci/msi.c
>> +++ b/drivers/pci/msi.c
>> @@ -420,7 +420,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
>>         arch_restore_msi_irqs(dev);
>>
>>         pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
>> -       pci_msi_update_mask(entry, 0, 0);
>> +       if (!(pci_msi_ignore_mask || desc->msi_attrib.is_virtual))
>> +               pci_msi_update_mask(entry, 0, 0);
>>         control &= ~PCI_MSI_FLAGS_QSIZE;
>>         control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
>>         pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
>>
>> But the commit message talks about MSI-X, and the above is MSI
>> only. Is Xen messing with the former, the latter, or both?
> My understanding is pci_msi_ignore_mask covers both MSI and MSI-X for Xen.

Please let me know if I should go ahead and try it out and send in a v3
of the patch.

I'm watching for further discussion right now, just to be clear.
>>>       raw_spin_lock_irqsave(lock, flags);
>>>       desc->msi_mask &= ~clear;
>>>       desc->msi_mask |= set;
>>> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>>>  {
>>>       void __iomem *desc_addr = pci_msix_desc_addr(desc);
>>>
>>> +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>> +             return;
>>> +
>>>       writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>>  }
>> I have similar reservations for this one.
> The problem here is some of the changes in commit 446a98b19fd6
> ("PCI/MSI: Use new mask/unmask functions") bypass the checks in
> __pci_msi_mask_desc/__pci_msi_unmask_desc.  I've wondered if it would
> be cleaner to push all the `if (pci_msi_ignore_mask)` checks down to
> the place of the writes.  That keeps dropping the write local to the
> write and leaves the higher level code consistent between the regular
> and Xen PV cases.  I don't know where checking
> desc->msi_attrib.is_virtual is appropriate.
>
> Regards,
> Jason

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

On 10/21/21 10:25, Josef Johansson wrote:
> On 10/20/21 16:03, Jason Andryuk wrote:
>> Hi, Marc,
>>
>> Adding Juergen and Boris since this involves Xen.
>>
>> On Wed, Oct 20, 2021 at 8:51 AM Marc Zyngier <maz@kernel.org> wrote:
>>> On Tue, 19 Oct 2021 22:48:19 +0100,
>>> Josef Johansson <josef@oderland.se> wrote:
>>>> From: Josef Johansson <josef@oderland.se>
>>>>
>>>>
>>>> PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV
>>>>
>>>> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>>>> functions") introduce functions pci_msi_update_mask() and
>>>> pci_msix_write_vector_ctrl() that is missing checks for
>>>> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
>>>> new mask/unmask functions"). Add them back since it is
>>>> causing severe lockups in amdgpu drivers under Xen during boot.
>>>>
>>>> As explained in commit 1a519dc7a73c ("PCI/MSI: Skip masking MSI-X
>>>> on Xen PV"), when running as Xen PV guest, masking MSI-X is a
>>>> responsibility of the hypervisor.
>>>>
>>>> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>>>> functions")
>>>> Suggested-by: Jason Andryuk <jandryuk@gmail.com>
>>>> Signed-off-by: Josef Johansson <josef@oderland.se>
>>>>
>>> [...]
>>>
>>>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>>>> index 0099a00af361..355b791e382f 100644
>>>> --- a/drivers/pci/msi.c
>>>> +++ b/drivers/pci/msi.c
>>>> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>>>>       raw_spinlock_t *lock = &desc->dev->msi_lock;
>>>>       unsigned long flags;
>>>>
>>>> +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>>> +             return;
>>>> +
>>> I'd rather be consistent, and keep the check outside of
>>> pci_msi_update_mask(), just like we do in __pci_msi_mask_desc().
>>> Something like this instead:
>>>
>>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
>>> index 0099a00af361..6c69eab304ce 100644
>>> --- a/drivers/pci/msi.c
>>> +++ b/drivers/pci/msi.c
>>> @@ -420,7 +420,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
>>>         arch_restore_msi_irqs(dev);
>>>
>>>         pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
>>> -       pci_msi_update_mask(entry, 0, 0);
>>> +       if (!(pci_msi_ignore_mask || desc->msi_attrib.is_virtual))
>>> +               pci_msi_update_mask(entry, 0, 0);
>>>         control &= ~PCI_MSI_FLAGS_QSIZE;
>>>         control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
>>>         pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
>>>
>>> But the commit message talks about MSI-X, and the above is MSI
>>> only. Is Xen messing with the former, the latter, or both?
>> My understanding is pci_msi_ignore_mask covers both MSI and MSI-X for Xen.
> Please let me know if I should go ahead and try it out and send in a v3
> of the patch.
>
> I'm watching for further discussion right now, just to be clear.

Hi,

I ended up with this patch, I also masked pci_set_mask and
pci_set_unmask, even though patching __pci_restore_msi_state and
__pci_restore_msi_state solved this problem, I found that it did not
properly make the system be able to survive flip_done timeout related
problems during suspend/resume. Would this be something you had in mind
Marc? I will make one more try with just patching
__pci_restore_msi_state and __pci_restore_msix_state just to make sure.
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c index
4b4792940e86..0b2225066778 100644 --- a/drivers/pci/msi.c +++
b/drivers/pci/msi.c @@ -420,7 +420,8 @@ static void
__pci_restore_msi_state(struct pci_dev *dev) arch_restore_msi_irqs(dev);
pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control); -
pci_msi_update_mask(entry, 0, 0); + if (!(pci_msi_ignore_mask ||
entry->msi_attrib.is_virtual)) + pci_msi_update_mask(entry, 0, 0);
control &= ~PCI_MSI_FLAGS_QSIZE; control |= (entry->msi_attrib.multiple
<< 4) | PCI_MSI_FLAGS_ENABLE; pci_write_config_word(dev, dev->msi_cap +
PCI_MSI_FLAGS, control); @@ -450,8 +451,9 @@ static void
__pci_restore_msix_state(struct pci_dev *dev) PCI_MSIX_FLAGS_ENABLE |
PCI_MSIX_FLAGS_MASKALL); arch_restore_msi_irqs(dev); -
for_each_pci_msi_entry(entry, dev) - pci_msix_write_vector_ctrl(entry,
entry->msix_ctrl); + if (!(pci_msi_ignore_mask ||
entry->msi_attrib.is_virtual)) + for_each_pci_msi_entry(entry, dev) +
pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0); } @@ -546,7
+548,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
return -ENOMEM; /* All MSIs are unmasked by default; mask them all */ -
pci_msi_mask(entry, msi_multi_mask(entry)); + if (!pci_msi_ignore_mask)
+ pci_msi_mask(entry, msi_multi_mask(entry));
list_add_tail(&entry->list, dev_to_msi_list(&dev->dev)); @@ -577,7
+580,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
return 0; err: - pci_msi_unmask(entry, msi_multi_mask(entry)); + if
(!pci_msi_ignore_mask) + pci_msi_unmask(entry, msi_multi_mask(entry));
free_msi_irqs(dev); return ret; } @@ -865,7 +868,8 @@ static void
pci_msi_shutdown(struct pci_dev *dev) dev->msi_enabled = 0; /* Return
the device with MSI unmasked as initial states */ - pci_msi_unmask(desc,
msi_multi_mask(desc)); + if (!pci_msi_ignore_mask) +
pci_msi_unmask(desc, msi_multi_mask(desc)); /* Restore dev->irq to its
default pin-assertion IRQ */ dev->irq = desc->msi_attrib.default_irq; @@
-950,8 +954,9 @@ static void pci_msix_shutdown(struct pci_dev *dev) } /*
Return the device with MSI-X masked as initial states */ -
for_each_pci_msi_entry(entry, dev) - pci_msix_mask(entry); + if
(!pci_msi_ignore_mask) + for_each_pci_msi_entry(entry, dev) +
pci_msix_mask(entry); pci_msix_clear_and_set_ctrl(dev,
PCI_MSIX_FLAGS_ENABLE, 0); pci_intx_for_msi(dev, 1);


>>>>       raw_spin_lock_irqsave(lock, flags);
>>>>       desc->msi_mask &= ~clear;
>>>>       desc->msi_mask |= set;
>>>> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>>>>  {
>>>>       void __iomem *desc_addr = pci_msix_desc_addr(desc);
>>>>
>>>> +     if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>>> +             return;
>>>> +
>>>>       writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>>>  }
>>> I have similar reservations for this one.
>> The problem here is some of the changes in commit 446a98b19fd6
>> ("PCI/MSI: Use new mask/unmask functions") bypass the checks in
>> __pci_msi_mask_desc/__pci_msi_unmask_desc.  I've wondered if it would
>> be cleaner to push all the `if (pci_msi_ignore_mask)` checks down to
>> the place of the writes.  That keeps dropping the write local to the
>> write and leaves the higher level code consistent between the regular
>> and Xen PV cases.  I don't know where checking
>> desc->msi_attrib.is_virtual is appropriate.

This makes sense the patch would be like so, I'm testing this out now
hoping it will

perform as good. Now the check is performed in four places

* pci_msi_update_mask

* pci_msix_mask

* pci_msix_unmask

* msix_mask_all

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4b4792940e86..6fa60ad9cba2 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -186,6 +189,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 
 static inline void pci_msix_mask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	desc->msix_ctrl |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 	/* Flush write to device */
@@ -194,15 +200,15 @@ static inline void pci_msix_mask(struct msi_desc *desc)
 
 static inline void pci_msix_unmask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	desc->msix_ctrl &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 }
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_mask(desc);
 	else if (desc->msi_attrib.maskbit)
@@ -211,9 +217,6 @@ static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_unmask(desc);
 	else if (desc->msi_attrib.maskbit)



That leaves me with a though, will this set masked, and should be checked as well?

void __pci_write_msi_msg(struct msi_desc *entry, struct msi_msg *msg)
{
        struct pci_dev *dev = msi_desc_to_pci_dev(entry);

        if (dev->current_state != PCI_D0 || pci_dev_is_disconnected(dev)) {
                /* Don't touch the hardware now */
        } else if (entry->msi_attrib.is_msix) {
                void __iomem *base = pci_msix_desc_addr(entry);
                u32 ctrl = entry->msix_ctrl;
                bool unmasked = !(ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT);

                if (entry->msi_attrib.is_virtual)
                        goto skip;

                /*
                 * The specification mandates that the entry is masked
                 * when the message is modified:
                 *
                 * "If software changes the Address or Data value of an
                 * entry while the entry is unmasked, the result is
                 * undefined."
                 */
                if (unmasked)
>>>                     pci_msix_write_vector_ctrl(entry, ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT);

>> Regards,
>> Jason

[PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Jason Andryuk]

commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
functions") introduce functions pci_msi_update_mask() and
pci_msix_write_vector_ctrl() that is missing checks for
pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
new mask/unmask functions").  The checks are in place at the high level
__pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
directly to the helpers.

Push the pci_msi_ignore_mask check down to the functions that make
the actual writes.  This keeps the logic local to the writes that need
to be bypassed.

With Xen PV, the hypervisor is responsible for masking and unmasking the
interrupts, which pci_msi_ignore_mask is used to indicate.

This change avoids lockups in amdgpu drivers under Xen during boot.

Fixes: commit 446a98b19fd6 ("PCI/MSI: Use new mask/unmask functions")
Reported-by: Josef Johansson <josef@oderland.se>
Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
---
 drivers/pci/msi.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4b4792940e86..478536bafc39 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (pci_msi_ignore_mask)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 {
 	void __iomem *desc_addr = pci_msix_desc_addr(desc);
 
+	if (pci_msi_ignore_mask)
+		return;
+
 	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 }
 
@@ -200,7 +206,7 @@ static inline void pci_msix_unmask(struct msi_desc *desc)
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+	if (desc->msi_attrib.is_virtual)
 		return;
 
 	if (desc->msi_attrib.is_msix)
@@ -211,7 +217,7 @@ static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+	if (desc->msi_attrib.is_virtual)
 		return;
 
 	if (desc->msi_attrib.is_msix)
-- 
2.30.2

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Jason Andryuk]

On Sun, Oct 24, 2021 at 2:55 PM Josef Johansson <josef@oderland.se> wrote:

> I ended up with this patch, I also masked pci_set_mask and
> pci_set_unmask, even though patching __pci_restore_msi_state and
> __pci_restore_msi_state solved this problem, I found that it did not
> properly make the system be able to survive flip_done timeout related
> problems during suspend/resume. Would this be something you had in mind
> Marc? I will make one more try with just patching
> __pci_restore_msi_state and __pci_restore_msix_state just to make sure.
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c index
> 4b4792940e86..0b2225066778 100644 --- a/drivers/pci/msi.c +++
> b/drivers/pci/msi.c @@ -420,7 +420,8 @@ static void
> __pci_restore_msi_state(struct pci_dev *dev) arch_restore_msi_irqs(dev);
> pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control); -
> pci_msi_update_mask(entry, 0, 0); + if (!(pci_msi_ignore_mask ||
> entry->msi_attrib.is_virtual)) + pci_msi_update_mask(entry, 0, 0);
> control &= ~PCI_MSI_FLAGS_QSIZE; control |= (entry->msi_attrib.multiple

This patch was mangled.

> This makes sense the patch would be like so, I'm testing this out now
> hoping it will
>
> perform as good. Now the check is performed in four places

Close.  I'll reply with my compiled, but untested patch of what I was thinking.

> That leaves me with a though, will this set masked, and should be checked as well?
>
> void __pci_write_msi_msg(struct msi_desc *entry, struct msi_msg *msg)
> {
>         struct pci_dev *dev = msi_desc_to_pci_dev(entry);
>
>         if (dev->current_state != PCI_D0 || pci_dev_is_disconnected(dev)) {
>                 /* Don't touch the hardware now */
>         } else if (entry->msi_attrib.is_msix) {
>                 void __iomem *base = pci_msix_desc_addr(entry);
>                 u32 ctrl = entry->msix_ctrl;
>                 bool unmasked = !(ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT);
>
>                 if (entry->msi_attrib.is_virtual)
>                         goto skip;
>
>                 /*
>                  * The specification mandates that the entry is masked
>                  * when the message is modified:
>                  *
>                  * "If software changes the Address or Data value of an
>                  * entry while the entry is unmasked, the result is
>                  * undefined."
>                  */
>                 if (unmasked)
> >>>                     pci_msix_write_vector_ctrl(entry, ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT);

My patch adds a check in pci_msix_write_vector_ctrl(), but the comment
above means PV Xen's behavior may be incorrect if Linux is calling
this function and modifying the message.

Regards,
Jason

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 891 bytes --]

On Sun, 2021-10-24 at 21:25 -0400, Jason Andryuk wrote:
> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions") introduce functions pci_msi_update_mask() and
> pci_msix_write_vector_ctrl() that is missing checks for
> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> new mask/unmask functions").  The checks are in place at the high level
> __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
> directly to the helpers.
> 
> Push the pci_msi_ignore_mask check down to the functions that make
> the actual writes.  This keeps the logic local to the writes that need
> to be bypassed.
> 
> With Xen PV, the hypervisor is responsible for masking and unmasking the
> interrupts, which pci_msi_ignore_mask is used to indicate.

This isn't just for Xen PV; Xen HVM guests let Xen unmask the MSI for
them too.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Roger Pau Monné]

On Mon, Oct 25, 2021 at 08:44:52AM +0100, David Woodhouse wrote:
> On Sun, 2021-10-24 at 21:25 -0400, Jason Andryuk wrote:
> > commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> > functions") introduce functions pci_msi_update_mask() and
> > pci_msix_write_vector_ctrl() that is missing checks for
> > pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> > new mask/unmask functions").  The checks are in place at the high level
> > __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
> > directly to the helpers.
> > 
> > Push the pci_msi_ignore_mask check down to the functions that make
> > the actual writes.  This keeps the logic local to the writes that need
> > to be bypassed.
> > 
> > With Xen PV, the hypervisor is responsible for masking and unmasking the
> > interrupts, which pci_msi_ignore_mask is used to indicate.
> 
> This isn't just for Xen PV; Xen HVM guests let Xen unmask the MSI for
> them too.

It's kind of optional for HVM guests, as it depends on
XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
guests, thus dropping any benefits from having hardware assisted APIC
virtualization or posted interrupts support.

AFAICT pci_msi_ignore_mask is not (correctly) set for HVM guest when
MSI interrupts are routed over event channels.

Regards, Roger.

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> It's kind of optional for HVM guests, as it depends on
> XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> guests, thus dropping any benefits from having hardware assisted APIC
> virtualization or posted interrupts support.

Indeed. After implementing PIRQ support for Xen guests running under
KVM, I spent a "happy" couple of days swearing at it because it
actually *worked* if something would just *unmask* the f***ing MSI, but
the guest inexplicably (to me) didn't do that.

Took me a while to work out that Xen itself is *snooping* on the MSI
table writes even while they are *masked*, to capture the magic MSI
message (with vector==0) which means it's actually a PIRQ# in the
destination ID bits, and then magically unmask the MSI when the guest
binds that PIRQ to an event channel.

I did not enjoy implementing that part.

FWIW the *guest* could potentlaly be smarter here and elect not to use
PIRQs when hardware assisted vAPIC is present. Aren't there some bits
in the CPUID that Xen advertises, which indicate that? 


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Jason Andryuk]

On Sun, Oct 24, 2021 at 9:26 PM Jason Andryuk <jandryuk@gmail.com> wrote:
>
> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> functions") introduce functions pci_msi_update_mask() and
> pci_msix_write_vector_ctrl() that is missing checks for
> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> new mask/unmask functions").  The checks are in place at the high level
> __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
> directly to the helpers.
>
> Push the pci_msi_ignore_mask check down to the functions that make
> the actual writes.  This keeps the logic local to the writes that need
> to be bypassed.
>
> With Xen PV, the hypervisor is responsible for masking and unmasking the
> interrupts, which pci_msi_ignore_mask is used to indicate.
>
> This change avoids lockups in amdgpu drivers under Xen during boot.
>
> Fixes: commit 446a98b19fd6 ("PCI/MSI: Use new mask/unmask functions")
> Reported-by: Josef Johansson <josef@oderland.se>
> Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
> ---

I should have written that this is untested.  If this is the desired
approach, Josef should test that it solves his boot hangs.

Regards,
Jason

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Jason Andryuk]

On Mon, Oct 25, 2021 at 3:44 AM David Woodhouse <dwmw2@infradead.org> wrote:
>
> On Sun, 2021-10-24 at 21:25 -0400, Jason Andryuk wrote:
> > commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
> > functions") introduce functions pci_msi_update_mask() and
> > pci_msix_write_vector_ctrl() that is missing checks for
> > pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
> > new mask/unmask functions").  The checks are in place at the high level
> > __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
> > directly to the helpers.
> >
> > Push the pci_msi_ignore_mask check down to the functions that make
> > the actual writes.  This keeps the logic local to the writes that need
> > to be bypassed.
> >
> > With Xen PV, the hypervisor is responsible for masking and unmasking the
> > interrupts, which pci_msi_ignore_mask is used to indicate.
>
> This isn't just for Xen PV; Xen HVM guests let Xen unmask the MSI for
> them too.

Ah, that makes sense that Xen handles both.  I was repeating another
commit message's statement.  Oh, it looks like pci_msi_ignore_mask is
PV-specific.

Regards,
Jason

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Roger Pau Monné]

On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > It's kind of optional for HVM guests, as it depends on
> > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > guests, thus dropping any benefits from having hardware assisted APIC
> > virtualization or posted interrupts support.
> 
> Indeed. After implementing PIRQ support for Xen guests running under
> KVM, I spent a "happy" couple of days swearing at it because it
> actually *worked* if something would just *unmask* the f***ing MSI, but
> the guest inexplicably (to me) didn't do that.
> 
> Took me a while to work out that Xen itself is *snooping* on the MSI
> table writes even while they are *masked*, to capture the magic MSI
> message (with vector==0) which means it's actually a PIRQ# in the
> destination ID bits, and then magically unmask the MSI when the guest
> binds that PIRQ to an event channel.
> 
> I did not enjoy implementing that part.

I can see that. It's even better because none of this is actually
documented.

> FWIW the *guest* could potentlaly be smarter here and elect not to use
> PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> in the CPUID that Xen advertises, which indicate that? 

Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
specifications gives us much benefit. We avoid a couple of vm exits
for sure, but the cost of having all those modifications in guest
OSes is not worth it.

Roger.

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]

On Mon, 2021-10-25 at 14:58 +0200, Roger Pau Monné wrote:
> On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> > On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > > It's kind of optional for HVM guests, as it depends on
> > > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > > guests, thus dropping any benefits from having hardware assisted APIC
> > > virtualization or posted interrupts support.
> > 
> > Indeed. After implementing PIRQ support for Xen guests running under
> > KVM, I spent a "happy" couple of days swearing at it because it
> > actually *worked* if something would just *unmask* the f***ing MSI, but
> > the guest inexplicably (to me) didn't do that.
> > 
> > Took me a while to work out that Xen itself is *snooping* on the MSI
> > table writes even while they are *masked*, to capture the magic MSI
> > message (with vector==0) which means it's actually a PIRQ# in the
> > destination ID bits, and then magically unmask the MSI when the guest
> > binds that PIRQ to an event channel.
> > 
> > I did not enjoy implementing that part.
> 
> I can see that. It's even better because none of this is actually
> documented.

Indeed. I still haven't worked out if/how Xen actually *masks* the
corresponding MSI-X again. It can't do so when the evtchn is masked,
since that's just a bit in the shinfo page. So while the evtchn is
masked, the MSI can still be screaming into the void?

Perhaps it does so when the PIRQ is unbound from the evtchn?

> > FWIW the *guest* could potentlaly be smarter here and elect not to use
> > PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> > in the CPUID that Xen advertises, which indicate that? 
> 
> Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
> XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
> specifications gives us much benefit. We avoid a couple of vm exits
> for sure, but the cost of having all those modifications in guest
> OSes is not worth it.

These days with posted interrupts, it doesn't even save us any vmexits;
it's all that additional guest complexity just to give us *more*
vmexits than we would have had :)

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Roger Pau Monné]

On Mon, Oct 25, 2021 at 02:02:38PM +0100, David Woodhouse wrote:
> On Mon, 2021-10-25 at 14:58 +0200, Roger Pau Monné wrote:
> > On Mon, Oct 25, 2021 at 12:53:31PM +0100, David Woodhouse wrote:
> > > On Mon, 2021-10-25 at 13:43 +0200, Roger Pau Monné wrote:
> > > > It's kind of optional for HVM guests, as it depends on
> > > > XENFEAT_hvm_pirqs, which sadly gets unconditionally set for HVM
> > > > guests, thus dropping any benefits from having hardware assisted APIC
> > > > virtualization or posted interrupts support.
> > > 
> > > Indeed. After implementing PIRQ support for Xen guests running under
> > > KVM, I spent a "happy" couple of days swearing at it because it
> > > actually *worked* if something would just *unmask* the f***ing MSI, but
> > > the guest inexplicably (to me) didn't do that.
> > > 
> > > Took me a while to work out that Xen itself is *snooping* on the MSI
> > > table writes even while they are *masked*, to capture the magic MSI
> > > message (with vector==0) which means it's actually a PIRQ# in the
> > > destination ID bits, and then magically unmask the MSI when the guest
> > > binds that PIRQ to an event channel.
> > > 
> > > I did not enjoy implementing that part.
> > 
> > I can see that. It's even better because none of this is actually
> > documented.
> 
> Indeed. I still haven't worked out if/how Xen actually *masks* the
> corresponding MSI-X again. It can't do so when the evtchn is masked,
> since that's just a bit in the shinfo page. So while the evtchn is
> masked, the MSI can still be screaming into the void?

I think so, it's quite weird because as a side effect of this mangling
Xen is transforming an edge triggered interrupt to a level triggered
one, as masked event channels belonging to MSI vectors will get set to
pending.

So it's not entirely screaming into the void because it will get
(wrongly) set as pending when masked.

> Perhaps it does so when the PIRQ is unbound from the evtchn?
> 
> > > FWIW the *guest* could potentlaly be smarter here and elect not to use
> > > PIRQs when hardware assisted vAPIC is present. Aren't there some bits
> > > in the CPUID that Xen advertises, which indicate that? 
> > 
> > Yes, it's in leaf 0x40000x04. FWIW, I would also be fine with removing
> > XENFEAT_hvm_pirqs, as I don't think diverging from the hardware
> > specifications gives us much benefit. We avoid a couple of vm exits
> > for sure, but the cost of having all those modifications in guest
> > OSes is not worth it.
> 
> These days with posted interrupts, it doesn't even save us any vmexits;
> it's all that additional guest complexity just to give us *more*
> vmexits than we would have had :)

Oh indeed. I was thinking about hardware without APIC hw
virtualization or posted interrupts.

Roger.

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Josef Johansson]

On 10/25/21 14:27, Jason Andryuk wrote:
> On Sun, Oct 24, 2021 at 9:26 PM Jason Andryuk <jandryuk@gmail.com> wrote:
>> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>> functions") introduce functions pci_msi_update_mask() and
>> pci_msix_write_vector_ctrl() that is missing checks for
>> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
>> new mask/unmask functions").  The checks are in place at the high level
>> __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
>> directly to the helpers.
>>
>> Push the pci_msi_ignore_mask check down to the functions that make
>> the actual writes.  This keeps the logic local to the writes that need
>> to be bypassed.
>>
>> With Xen PV, the hypervisor is responsible for masking and unmasking the
>> interrupts, which pci_msi_ignore_mask is used to indicate.
>>
>> This change avoids lockups in amdgpu drivers under Xen during boot.
>>
>> Fixes: commit 446a98b19fd6 ("PCI/MSI: Use new mask/unmask functions")
>> Reported-by: Josef Johansson <josef@oderland.se>
>> Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
>> ---
> I should have written that this is untested.  If this is the desired
> approach, Josef should test that it solves his boot hangs.
>
> Regards,
> Jason

I've tested this today, both the above patch, but also my own below
where I'm patching inside __pci_write_msi_msg,
which is the outcome of the patch above.

I found that both solves the boot hang, but both have severe effects
on suspends/resume later on. The below patch without patching
__pci_write_msi_msg seems to be ideal, solves boot problems but not
causing too much others. There seems to me that there's undocumented
dragons here. Doing more test later today.

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4b4792940e86..e97eea1bc93a 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -186,6 +189,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 
 static inline void pci_msix_mask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	desc->msix_ctrl |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 	/* Flush write to device */
@@ -194,15 +200,15 @@ static inline void pci_msix_mask(struct msi_desc *desc)
 
 static inline void pci_msix_unmask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+		return;
+
 	desc->msix_ctrl &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 }
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_mask(desc);
 	else if (desc->msi_attrib.maskbit)
@@ -211,9 +217,6 @@ static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_unmask(desc);
 	else if (desc->msi_attrib.maskbit)
@@ -307,7 +310,7 @@ void __pci_write_msi_msg(struct msi_desc *entry, struct msi_msg *msg)
 		 * entry while the entry is unmasked, the result is
 		 * undefined."
 		 */
-		if (unmasked)
+		if (unmasked && !pci_msi_ignore_mask)
 			pci_msix_write_vector_ctrl(entry, ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT);
 
 		writel(msg->address_lo, base + PCI_MSIX_ENTRY_LOWER_ADDR);
@@ -450,8 +453,9 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 				PCI_MSIX_FLAGS_ENABLE | PCI_MSIX_FLAGS_MASKALL);
 
 	arch_restore_msi_irqs(dev);
-	for_each_pci_msi_entry(entry, dev)
-		pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
+	if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
+		for_each_pci_msi_entry(entry, dev)
+			pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
 
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0);
 }

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

On 10/25/21 03:25, Jason Andryuk wrote:
> On Sun, Oct 24, 2021 at 2:55 PM Josef Johansson <josef@oderland.se> wrote:
>
>> I ended up with this patch, I also masked pci_set_mask and
>> pci_set_unmask, even though patching __pci_restore_msi_state and
>> __pci_restore_msi_state solved this problem, I found that it did not
>> properly make the system be able to survive flip_done timeout related
>> problems during suspend/resume. Would this be something you had in mind
>> Marc? I will make one more try with just patching
>> __pci_restore_msi_state and __pci_restore_msix_state just to make sure.
>> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c index
>> 4b4792940e86..0b2225066778 100644 --- a/drivers/pci/msi.c +++
>> b/drivers/pci/msi.c @@ -420,7 +420,8 @@ static void
>> __pci_restore_msi_state(struct pci_dev *dev) arch_restore_msi_irqs(dev);
>> pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control); -
>> pci_msi_update_mask(entry, 0, 0); + if (!(pci_msi_ignore_mask ||
>> entry->msi_attrib.is_virtual)) + pci_msi_update_mask(entry, 0, 0);
>> control &= ~PCI_MSI_FLAGS_QSIZE; control |= (entry->msi_attrib.multiple
> This patch was mangled.
Thunderbird dislikes me plenty. Let's hope this turns out better.

I ended up with this patch, I also masked pci_set_mask and
pci_set_unmask, even though patching __pci_restore_msi_state and
__pci_restore_msi_state solved this problem, I found that it did not
properly make the system be able to survive flip_done timeout related
problems during suspend/resume. Would this be something you had in mind
Marc? I will make one more try with just patching
__pci_restore_msi_state and __pci_restore_msix_state just to make sure.


diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4b4792940e86..0b2225066778 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -420,7 +420,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	pci_msi_update_mask(entry, 0, 0);
+	if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
+		pci_msi_update_mask(entry, 0, 0);
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -450,8 +451,9 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 				PCI_MSIX_FLAGS_ENABLE | PCI_MSIX_FLAGS_MASKALL);
 
 	arch_restore_msi_irqs(dev);
-	for_each_pci_msi_entry(entry, dev)
-		pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
+	if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
+		for_each_pci_msi_entry(entry, dev)
+			pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
 
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0);
 }
@@ -546,7 +548,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
 		return -ENOMEM;
 
 	/* All MSIs are unmasked by default; mask them all */
-	pci_msi_mask(entry, msi_multi_mask(entry));
+	if (!pci_msi_ignore_mask)
+		pci_msi_mask(entry, msi_multi_mask(entry));
 
 	list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
 
@@ -577,7 +580,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
 	return 0;
 
 err:
-	pci_msi_unmask(entry, msi_multi_mask(entry));
+	if (!pci_msi_ignore_mask)
+		pci_msi_unmask(entry, msi_multi_mask(entry));
 	free_msi_irqs(dev);
 	return ret;
 }
@@ -865,7 +868,8 @@ static void pci_msi_shutdown(struct pci_dev *dev)
 	dev->msi_enabled = 0;
 
 	/* Return the device with MSI unmasked as initial states */
-	pci_msi_unmask(desc, msi_multi_mask(desc));
+	if (!pci_msi_ignore_mask)
+		pci_msi_unmask(desc, msi_multi_mask(desc));
 
 	/* Restore dev->irq to its default pin-assertion IRQ */
 	dev->irq = desc->msi_attrib.default_irq;
@@ -950,8 +954,9 @@ static void pci_msix_shutdown(struct pci_dev *dev)
 	}
 
 	/* Return the device with MSI-X masked as initial states */
-	for_each_pci_msi_entry(entry, dev)
-		pci_msix_mask(entry);
+	if (!pci_msi_ignore_mask)
+		for_each_pci_msi_entry(entry, dev)
+			pci_msix_mask(entry);
 
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
 	pci_intx_for_msi(dev, 1);



>> This makes sense the patch would be like so, I'm testing this out now
>> hoping it will
>>
>> perform as good. Now the check is performed in four places
> Close.  I'll reply with my compiled, but untested patch of what I was thinking.
>> That leaves me with a though, will this set masked, and should be checked as well?
>>
>> void __pci_write_msi_msg(struct msi_desc *entry, struct msi_msg *msg)
>> {
>>         struct pci_dev *dev = msi_desc_to_pci_dev(entry);
>>
>>         if (dev->current_state != PCI_D0 || pci_dev_is_disconnected(dev)) {
>>                 /* Don't touch the hardware now */
>>         } else if (entry->msi_attrib.is_msix) {
>>                 void __iomem *base = pci_msix_desc_addr(entry);
>>                 u32 ctrl = entry->msix_ctrl;
>>                 bool unmasked = !(ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT);
>>
>>                 if (entry->msi_attrib.is_virtual)
>>                         goto skip;
>>
>>                 /*
>>                  * The specification mandates that the entry is masked
>>                  * when the message is modified:
>>                  *
>>                  * "If software changes the Address or Data value of an
>>                  * entry while the entry is unmasked, the result is
>>                  * undefined."
>>                  */
>>                 if (unmasked)
>>>>>                     pci_msix_write_vector_ctrl(entry, ctrl | PCI_MSIX_ENTRY_CTRL_MASKBIT);
> My patch adds a check in pci_msix_write_vector_ctrl(), but the comment
> above means PV Xen's behavior may be incorrect if Linux is calling
> this function and modifying the message.
>
> Regards,
> Jason
Turns out it seems to mess things up. I'm compiling this patch right now
with config flags below ( for anyone trying the same ). It should
perform ok I hope.

CONFIG_AMD_PMC=y
#CONFIG_HSA_AMD is not set
#CONFIG_DRM_AMD_SECURE_DISPLAY is not set
#CONFIG_CRYPTO_DEV_CCP is not set

Moving checks pci_msix_mask/pci_msix_unmask to ensure that init/shutdown
gets the checks as well. Avoiding
pci_msix_write_vector_ctrl/__pci_write_msi_msg
since it seems to have odd effects, like the comment in __pci_write_msi_msg
tells us. Just applying checks in __pci_restore_msi_state and
__pci_restore_msix_state
did not do the trick.

diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 4b4792940e86..acf14a4708e6 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -186,6 +186,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
 
 static inline void pci_msix_mask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask)
+		return;
+
 	desc->msix_ctrl |= PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 	/* Flush write to device */
@@ -194,13 +197,16 @@ static inline void pci_msix_mask(struct msi_desc *desc)
 
 static inline void pci_msix_unmask(struct msi_desc *desc)
 {
+	if (pci_msi_ignore_mask)
+		return;
+
 	desc->msix_ctrl &= ~PCI_MSIX_ENTRY_CTRL_MASKBIT;
 	pci_msix_write_vector_ctrl(desc, desc->msix_ctrl);
 }
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+	if (desc->msi_attrib.is_virtual)
 		return;
 
 	if (desc->msi_attrib.is_msix)
@@ -211,7 +217,7 @@ static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
+	if (desc->msi_attrib.is_virtual)
 		return;
 
 	if (desc->msi_attrib.is_msix)
@@ -420,7 +426,8 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	pci_msi_update_mask(entry, 0, 0);
+	if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
+		pci_msi_update_mask(entry, 0, 0);
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -450,8 +457,9 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 				PCI_MSIX_FLAGS_ENABLE | PCI_MSIX_FLAGS_MASKALL);
 
 	arch_restore_msi_irqs(dev);
-	for_each_pci_msi_entry(entry, dev)
-		pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
+	if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
+		for_each_pci_msi_entry(entry, dev)
+			pci_msix_write_vector_ctrl(entry, entry->msix_ctrl);
 
 	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_MASKALL, 0);
 }

Please let me know if I should submit any of the two, or make changes to them.

Regards
- Josef

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Josef Johansson]

On 10/25/21 18:46, Josef Johansson wrote:
> On 10/25/21 14:27, Jason Andryuk wrote:
>> On Sun, Oct 24, 2021 at 9:26 PM Jason Andryuk <jandryuk@gmail.com> wrote:
>>> commit fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask
>>> functions") introduce functions pci_msi_update_mask() and
>>> pci_msix_write_vector_ctrl() that is missing checks for
>>> pci_msi_ignore_mask that exists in commit 446a98b19fd6 ("PCI/MSI: Use
>>> new mask/unmask functions").  The checks are in place at the high level
>>> __pci_msi_mask_desc()/__pci_msi_unmask_desc(), but some functions call
>>> directly to the helpers.
>>>
>>> Push the pci_msi_ignore_mask check down to the functions that make
>>> the actual writes.  This keeps the logic local to the writes that need
>>> to be bypassed.
>>>
>>> With Xen PV, the hypervisor is responsible for masking and unmasking the
>>> interrupts, which pci_msi_ignore_mask is used to indicate.
>>>
>>> This change avoids lockups in amdgpu drivers under Xen during boot.
>>>
>>> Fixes: commit 446a98b19fd6 ("PCI/MSI: Use new mask/unmask functions")
>>> Reported-by: Josef Johansson <josef@oderland.se>
>>> Signed-off-by: Jason Andryuk <jandryuk@gmail.com>
>>> ---
>> I should have written that this is untested.  If this is the desired
>> approach, Josef should test that it solves his boot hangs.
>>
>> Regards,
>> Jason
> I've tested this today, both the above patch, but also my own below
> where I'm patching inside __pci_write_msi_msg,
> which is the outcome of the patch above.
I tested a lot of kernels today. To create a good baseline I compiled
without any of our patches here
and with my config flags set.

CONFIG_AMD_PMC=y
# CONFIG_HSA_AMD is not set
# CONFIG_CRYPTO_DEV_CCP is not set

The kernel stopped as before, and hung.

Test number 2 was to boot with amdgpu.msi=0.
This still resulted in a bad boot since all the xhcd drivers complained.
We can be sure that it's not amdgpu per se.

Test number 3 was with Jason's patch. It worked, but suspend/resume is
not working well.
Generally it's not behaving like other kernels do, which makes it
actually change the behavior.
Now with test 4 I tried that thought, maybe this is still a good change?
I'm deprived of a good baseline in all this, so it's very hard to
navigate between all the variables.

Test number 4 was with Jason's patch plus the amdgpu-patch below.
It worked, even suspend/resume, 2 times, but then it all crashed and
burn with quite interesting stacktraces. Are amdgpu doing it wrong here
or is it just me nitpicking?

index cc2e0c9cfe0a..f125597eb991 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c
@@ -279,17 +279,8 @@ static bool amdgpu_msi_ok(struct amdgpu_device *adev)
 
 static void amdgpu_restore_msix(struct amdgpu_device *adev)
 {
-	u16 ctrl;
-
-	pci_read_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, &ctrl);
-	if (!(ctrl & PCI_MSIX_FLAGS_ENABLE))
-		return;
-
-	/* VF FLR */
-	ctrl &= ~PCI_MSIX_FLAGS_ENABLE;
-	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
-	ctrl |= PCI_MSIX_FLAGS_ENABLE;
-	pci_write_config_word(adev->pdev, adev->pdev->msix_cap + PCI_MSIX_FLAGS, ctrl);
+	// checks if msix is enabled also
+	pci_restore_msi_state(adev->pdev);
 }
 
 /**

During the tests I fiddled with dpm settings, and it does have an effect
one the graphical output during suspend/resume. So maybe there's
hardware problems at play here as well.

I also looked through the code before and after Thomas' changes, and I
can't see that
this patch should make any functional difference compared to before the
MSI series of patches.
It's even such that is_virtual should be checked withing vector_ctrl. I
find Jason's patch
quite nice since it really places the checks on few places making it
easier not to slip.
Compared to my attempt that even failed because I forgot one more place
to put the checks.

With that said I would really like some more tests on this with
different chipsets, on Xen.
Any takers?

What I'm seeing is that there's no readl() in pci_msix_unmask(), it was
one in the code path before.
I'm very much unsure if there should be one there though.

We can really do a better job at the documentation for
pci_msi_ignore_mask, at least in msi.c,
maybe that should be a different patch adding some comments such that
driver folks really see
the benefits of using the built in restore functions e.g.

This became so much bigger project than I thought, thanks all for
chiming in on it.

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

On Mon, 2021-10-25 at 21:21 +0200, Josef Johansson wrote:
> +       if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))

Is it just me, or is that a lot easier to read if you write it as the
tautologically-identical (!pci_msi_ignore_mask && !entry->…is_virtual)?


> @@ -546,7 +548,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
>                 return -ENOMEM;

>         /* All MSIs are unmasked by default; mask them all *
> -       pci_msi_mask(entry, msi_multi_mask(entry))
> +       if (!pci_msi_ignore_mask)
> +               pci_msi_mask(entry, msi_multi_mask(entry));
>
>         list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));


Hm, I thought that older kernels *did* do this part (or at least the
later ones in pci_msi*_hutdown). I was watching it when I did the Xen
hosting implementation I mentioned before; even a hack to unmask them
all when the VM was started, wasn't working because the guest would
*mask* all MSI-X, just never unmask them again.


I wonder if we should rename 'pci_msi_ignore_mask' to something with
Xen in its name because Xen is the only user of this abomination (which
fundamentally seems to require that the virtual hardware use MSI
entries even while they're masked, so hopefully nobody else would
*ever* do such a thing), and the required behaviour is very Xen-
specific.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5174 bytes --]

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[Josef Johansson]

On 10/27/21 08:24, David Woodhouse wrote:
> On Mon, 2021-10-25 at 21:21 +0200, Josef Johansson wrote:
>> +       if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
> Is it just me, or is that a lot easier to read if you write it as the
> tautologically-identical (!pci_msi_ignore_mask && !entry->…is_virtual)?
Sure, the less parentheses the better.
>
>> @@ -546,7 +548,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
>>                 return -ENOMEM;
>>         /* All MSIs are unmasked by default; mask them all *
>> -       pci_msi_mask(entry, msi_multi_mask(entry))
>> +       if (!pci_msi_ignore_mask)
>> +               pci_msi_mask(entry, msi_multi_mask(entry));
>>
>>         list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
>
> Hm, I thought that older kernels *did* do this part (or at least the
> later ones in pci_msi*_hutdown). I was watching it when I did the Xen
> hosting implementation I mentioned before; even a hack to unmask them
> all when the VM was started, wasn't working because the guest would
> *mask* all MSI-X, just never unmask them again.
When you're saying *did* here, do you mean that they mask even though
pci_msi_ignore_mask = 0?

As I was looking through pre Thomas' changes and post, it seems that we
did indeed
check for pci_msi_ignore_mask in msi_capability_init.
>
> I wonder if we should rename 'pci_msi_ignore_mask' to something with
> Xen in its name because Xen is the only user of this abomination (which
> fundamentally seems to require that the virtual hardware use MSI
> entries even while they're masked, so hopefully nobody else would
> *ever* do such a thing), and the required behaviour is very Xen-
> specific.
Second that, i.e. pci_msi_masked_by_xen.

Re: [PATCH v2] PCI/MSI: Re-add checks for skip masking MSI-X on Xen PV

[David Woodhouse]

On 27 October 2021 09:13:36 BST, Josef Johansson <josef@oderland.se> wrote:
>On 10/27/21 08:24, David Woodhouse wrote:
>> On Mon, 2021-10-25 at 21:21 +0200, Josef Johansson wrote:
>>> +       if (!(pci_msi_ignore_mask || entry->msi_attrib.is_virtual))
>> Is it just me, or is that a lot easier to read if you write it as the
>> tautologically-identical (!pci_msi_ignore_mask && !entry->…is_virtual)?
>Sure, the less parentheses the better.
>>
>>> @@ -546,7 +548,8 @@ static int msi_capability_init(struct pci_dev *dev, int nvec,
>>>                 return -ENOMEM;
>>>         /* All MSIs are unmasked by default; mask them all *
>>> -       pci_msi_mask(entry, msi_multi_mask(entry))
>>> +       if (!pci_msi_ignore_mask)
>>> +               pci_msi_mask(entry, msi_multi_mask(entry));
>>>
>>>         list_add_tail(&entry->list, dev_to_msi_list(&dev->dev));
>>
>> Hm, I thought that older kernels *did* do this part (or at least the
>> later ones in pci_msi*_hutdown). I was watching it when I did the Xen
>> hosting implementation I mentioned before; even a hack to unmask them
>> all when the VM was started, wasn't working because the guest would
>> *mask* all MSI-X, just never unmask them again.
>When you're saying *did* here, do you mean that they mask even though
>pci_msi_ignore_mask = 0?
>
>As I was looking through pre Thomas' changes and post, it seems that we
>did indeed
>check for pci_msi_ignore_mask in msi_capability_init.


Ah, maybe not so old. When my VMM part didn't work with standard ancient distro test images I turned to a relatively current git HEAD.

So I was probably just unfortunate, and masking MSI under Xen at setup time was a temporary aberration; on older kernels the hack of enabling each vector at startup might have worked?

I'll disable my eventual VMM-side fix and retest different guest kernels to be sure (and to make sure I have the full permutation set for regression testing because regardless of how insane Xen's behaviour is, I need to faithfully emulate it for every Linux kernel behaviour that existed).


-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH] PCI/MSI: Fix masking MSI/MSI-X on Xen PV

[Thomas Gleixner]

On Sun, Oct 24 2021 at 21:25, Jason Andryuk wrote:
> diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
> index 4b4792940e86..478536bafc39 100644
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask(struct msi_desc *desc, u32 clear, u32 s
>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>  	unsigned long flags;
>  
> +	if (pci_msi_ignore_mask)
> +		return;
> +
>  	raw_spin_lock_irqsave(lock, flags);
>  	desc->msi_mask &= ~clear;
>  	desc->msi_mask |= set;
> @@ -181,6 +184,9 @@ static void pci_msix_write_vector_ctrl(struct msi_desc *desc, u32 ctrl)
>  {
>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>  
> +	if (pci_msi_ignore_mask)
> +		return;
> +
>  	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  }
>  
> @@ -200,7 +206,7 @@ static inline void pci_msix_unmask(struct msi_desc *desc)
>  
>  static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
>  {
> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +	if (desc->msi_attrib.is_virtual)
>  		return;
>  
>  	if (desc->msi_attrib.is_msix)
> @@ -211,7 +217,7 @@ static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
>  
>  static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
>  {
> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> +	if (desc->msi_attrib.is_virtual)
>  		return;
>  
>  	if (desc->msi_attrib.is_msix)

No, really. This is horrible and incomplete. The right thing to do is to
move the check back into the low level accessors and remove it from the
call sites simply because the low level accessors can be reached not
only from the mask/unmask functions. But the above also fails to respect
msi_attrib.maskbit... I'll send out a proper fix in a few.

Thanks,

        tglx

[PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Thomas Gleixner]

The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
low level accessors into the higher level mask/unmask functions.

This missed the fact that these accessors can be invoked from other places
as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
also violates the virtual MSIX and the msi_attrib.maskbit protections.

Instead of sprinkling checks all over the place, lift them back into the
low level accessor functions. To avoid checking three different conditions
combine them into one property of msi_desc::msi_attrib.

Reported-by: Josef Johansson <josef@oderland.se>
Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask functions")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Andryuk <jandryuk@gmail.com>
Cc: Marc Zyngier <maz@kernel.org>
Cc: Bjorn Helgaas <helgaas@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: xen-devel <xen-devel@lists.xenproject.org>
Cc: Juergen Gross <jgross@suse.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: David Woodhouse <dwmw2@infradead.org>
---
 drivers/pci/msi.c   |   26 ++++++++++++++------------
 include/linux/msi.h |    2 +-
 2 files changed, 15 insertions(+), 13 deletions(-)

--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (!desc->msi_attrib.can_mask)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -181,7 +184,8 @@ static void pci_msix_write_vector_ctrl(s
 {
 	void __iomem *desc_addr = pci_msix_desc_addr(desc);
 
-	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
+	if (desc->msi_attrib.can_mask)
+		writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 }
 
 static inline void pci_msix_mask(struct msi_desc *desc)
@@ -200,23 +204,17 @@ static inline void pci_msix_unmask(struc
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_mask(desc);
-	else if (desc->msi_attrib.maskbit)
+	else
 		pci_msi_mask(desc, mask);
 }
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_unmask(desc);
-	else if (desc->msi_attrib.maskbit)
+	else
 		pci_msi_unmask(desc, mask);
 }
 
@@ -484,7 +482,8 @@ msi_setup_entry(struct pci_dev *dev, int
 	entry->msi_attrib.is_64		= !!(control & PCI_MSI_FLAGS_64BIT);
 	entry->msi_attrib.is_virtual    = 0;
 	entry->msi_attrib.entry_nr	= 0;
-	entry->msi_attrib.maskbit	= !!(control & PCI_MSI_FLAGS_MASKBIT);
+	entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
+					  !!(control & PCI_MSI_FLAGS_MASKBIT);
 	entry->msi_attrib.default_irq	= dev->irq;	/* Save IOAPIC IRQ */
 	entry->msi_attrib.multi_cap	= (control & PCI_MSI_FLAGS_QMASK) >> 1;
 	entry->msi_attrib.multiple	= ilog2(__roundup_pow_of_two(nvec));
@@ -495,7 +494,7 @@ msi_setup_entry(struct pci_dev *dev, int
 		entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
 
 	/* Save the initial mask status */
-	if (entry->msi_attrib.maskbit)
+	if (entry->msi_attrib.can_mask)
 		pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
 
 out:
@@ -638,10 +637,13 @@ static int msix_setup_entries(struct pci
 		entry->msi_attrib.is_virtual =
 			entry->msi_attrib.entry_nr >= vec_count;
 
+		entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
+						  !entry->msi_attrib.is_virtual;
+
 		entry->msi_attrib.default_irq	= dev->irq;
 		entry->mask_base		= base;
 
-		if (!entry->msi_attrib.is_virtual) {
+		if (!entry->msi_attrib.can_mask) {
 			addr = pci_msix_desc_addr(entry);
 			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 		}
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -148,7 +148,7 @@ struct msi_desc {
 				u8	is_msix		: 1;
 				u8	multiple	: 3;
 				u8	multi_cap	: 3;
-				u8	maskbit		: 1;
+				u8	can_mask	: 1;
 				u8	is_64		: 1;
 				u8	is_virtual	: 1;
 				u16	entry_nr;

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 10/27/21 11:50, Thomas Gleixner wrote:
> The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
> low level accessors into the higher level mask/unmask functions.
>
> This missed the fact that these accessors can be invoked from other places
> as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
> also violates the virtual MSIX and the msi_attrib.maskbit protections.
>
> Instead of sprinkling checks all over the place, lift them back into the
> low level accessor functions. To avoid checking three different conditions
> combine them into one property of msi_desc::msi_attrib.
>
> Reported-by: Josef Johansson <josef@oderland.se>
> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask functions")
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: Jason Andryuk <jandryuk@gmail.com>
> Cc: Marc Zyngier <maz@kernel.org>
> Cc: Bjorn Helgaas <helgaas@kernel.org>
> Cc: linux-pci@vger.kernel.org
> Cc: xen-devel <xen-devel@lists.xenproject.org>
> Cc: Juergen Gross <jgross@suse.com>
> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
> Cc: David Woodhouse <dwmw2@infradead.org>
> ---
>  drivers/pci/msi.c   |   26 ++++++++++++++------------
>  include/linux/msi.h |    2 +-
>  2 files changed, 15 insertions(+), 13 deletions(-)
>
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask
>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>  	unsigned long flags;
>  
> +	if (!desc->msi_attrib.can_mask)
> +		return;
> +
>  	raw_spin_lock_irqsave(lock, flags);
>  	desc->msi_mask &= ~clear;
>  	desc->msi_mask |= set;
> @@ -181,7 +184,8 @@ static void pci_msix_write_vector_ctrl(s
>  {
>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>  
> -	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
> +	if (desc->msi_attrib.can_mask)
> +		writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  }
>  
>  static inline void pci_msix_mask(struct msi_desc *desc)
> @@ -200,23 +204,17 @@ static inline void pci_msix_unmask(struc
>  
>  static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
>  {
> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> -		return;
> -
>  	if (desc->msi_attrib.is_msix)
>  		pci_msix_mask(desc);
> -	else if (desc->msi_attrib.maskbit)
> +	else
>  		pci_msi_mask(desc, mask);
>  }
>  
>  static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
>  {
> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
> -		return;
> -
>  	if (desc->msi_attrib.is_msix)
>  		pci_msix_unmask(desc);
> -	else if (desc->msi_attrib.maskbit)
> +	else
>  		pci_msi_unmask(desc, mask);
>  }
>  
> @@ -484,7 +482,8 @@ msi_setup_entry(struct pci_dev *dev, int
>  	entry->msi_attrib.is_64		= !!(control & PCI_MSI_FLAGS_64BIT);
>  	entry->msi_attrib.is_virtual    = 0;
>  	entry->msi_attrib.entry_nr	= 0;
> -	entry->msi_attrib.maskbit	= !!(control & PCI_MSI_FLAGS_MASKBIT);
> +	entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
> +					  !!(control & PCI_MSI_FLAGS_MASKBIT);
>  	entry->msi_attrib.default_irq	= dev->irq;	/* Save IOAPIC IRQ */
>  	entry->msi_attrib.multi_cap	= (control & PCI_MSI_FLAGS_QMASK) >> 1;
>  	entry->msi_attrib.multiple	= ilog2(__roundup_pow_of_two(nvec));
> @@ -495,7 +494,7 @@ msi_setup_entry(struct pci_dev *dev, int
>  		entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
>  
>  	/* Save the initial mask status */
> -	if (entry->msi_attrib.maskbit)
> +	if (entry->msi_attrib.can_mask)
>  		pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
>  
>  out:
> @@ -638,10 +637,13 @@ static int msix_setup_entries(struct pci
>  		entry->msi_attrib.is_virtual =
>  			entry->msi_attrib.entry_nr >= vec_count;
>  
> +		entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
> +						  !entry->msi_attrib.is_virtual;
> +
>  		entry->msi_attrib.default_irq	= dev->irq;
>  		entry->mask_base		= base;
>  
> -		if (!entry->msi_attrib.is_virtual) {
> +		if (!entry->msi_attrib.can_mask) {
>  			addr = pci_msix_desc_addr(entry);
>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  		}
> --- a/include/linux/msi.h
> +++ b/include/linux/msi.h
> @@ -148,7 +148,7 @@ struct msi_desc {
>  				u8	is_msix		: 1;
>  				u8	multiple	: 3;
>  				u8	multi_cap	: 3;
> -				u8	maskbit		: 1;
> +				u8	can_mask	: 1;
>  				u8	is_64		: 1;
>  				u8	is_virtual	: 1;
>  				u16	entry_nr;
Thanks,
I'll test this out ASAP.

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[David Woodhouse]

On 27 October 2021 10:50:15 BST, Thomas Gleixner <tglx@linutronix.de> wrote:
>The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
>low level accessors into the higher level mask/unmask functions.
>
>This missed the fact that these accessors can be invoked from other places
>as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
>also violates the virtual MSIX and the msi_attrib.maskbit protections.

Not just PV. It's Xen HVM guests too.

I'll also give it a spin on both Xen and not-Xen. Thanks.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 10/27/21 11:54, Josef Johansson wrote:
> On 10/27/21 11:50, Thomas Gleixner wrote:
>> The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
>> low level accessors into the higher level mask/unmask functions.
>>
>> This missed the fact that these accessors can be invoked from other places
>> as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
>> also violates the virtual MSIX and the msi_attrib.maskbit protections.
>>
>> Instead of sprinkling checks all over the place, lift them back into the
>> low level accessor functions. To avoid checking three different conditions
>> combine them into one property of msi_desc::msi_attrib.
>>
>> Reported-by: Josef Johansson <josef@oderland.se>
>> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask functions")
>> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>> Cc: Jason Andryuk <jandryuk@gmail.com>
>> Cc: Marc Zyngier <maz@kernel.org>
>> Cc: Bjorn Helgaas <helgaas@kernel.org>
>> Cc: linux-pci@vger.kernel.org
>> Cc: xen-devel <xen-devel@lists.xenproject.org>
>> Cc: Juergen Gross <jgross@suse.com>
>> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
>> Cc: David Woodhouse <dwmw2@infradead.org>
>> ---
>>  drivers/pci/msi.c   |   26 ++++++++++++++------------
>>  include/linux/msi.h |    2 +-
>>  2 files changed, 15 insertions(+), 13 deletions(-)
>>
>> --- a/drivers/pci/msi.c
>> +++ b/drivers/pci/msi.c
>> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask
>>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>>  	unsigned long flags;
>>  
>> +	if (!desc->msi_attrib.can_mask)
>> +		return;
>> +
>>  	raw_spin_lock_irqsave(lock, flags);
>>  	desc->msi_mask &= ~clear;
>>  	desc->msi_mask |= set;
>> @@ -181,7 +184,8 @@ static void pci_msix_write_vector_ctrl(s
>>  {
>>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>>  
>> -	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>> +	if (desc->msi_attrib.can_mask)
>> +		writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>  }
>>  
>>  static inline void pci_msix_mask(struct msi_desc *desc)
>> @@ -200,23 +204,17 @@ static inline void pci_msix_unmask(struc
>>  
>>  static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
>>  {
>> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>> -		return;
>> -
>>  	if (desc->msi_attrib.is_msix)
>>  		pci_msix_mask(desc);
>> -	else if (desc->msi_attrib.maskbit)
>> +	else
>>  		pci_msi_mask(desc, mask);
>>  }
>>  
>>  static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
>>  {
>> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>> -		return;
>> -
>>  	if (desc->msi_attrib.is_msix)
>>  		pci_msix_unmask(desc);
>> -	else if (desc->msi_attrib.maskbit)
>> +	else
>>  		pci_msi_unmask(desc, mask);
>>  }
>>  
>> @@ -484,7 +482,8 @@ msi_setup_entry(struct pci_dev *dev, int
>>  	entry->msi_attrib.is_64		= !!(control & PCI_MSI_FLAGS_64BIT);
>>  	entry->msi_attrib.is_virtual    = 0;
>>  	entry->msi_attrib.entry_nr	= 0;
>> -	entry->msi_attrib.maskbit	= !!(control & PCI_MSI_FLAGS_MASKBIT);
>> +	entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
>> +					  !!(control & PCI_MSI_FLAGS_MASKBIT);
>>  	entry->msi_attrib.default_irq	= dev->irq;	/* Save IOAPIC IRQ */
>>  	entry->msi_attrib.multi_cap	= (control & PCI_MSI_FLAGS_QMASK) >> 1;
>>  	entry->msi_attrib.multiple	= ilog2(__roundup_pow_of_two(nvec));
>> @@ -495,7 +494,7 @@ msi_setup_entry(struct pci_dev *dev, int
>>  		entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
>>  
>>  	/* Save the initial mask status */
>> -	if (entry->msi_attrib.maskbit)
>> +	if (entry->msi_attrib.can_mask)
>>  		pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
>>  
>>  out:
>> @@ -638,10 +637,13 @@ static int msix_setup_entries(struct pci
>>  		entry->msi_attrib.is_virtual =
>>  			entry->msi_attrib.entry_nr >= vec_count;
>>  
>> +		entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
>> +						  !entry->msi_attrib.is_virtual;
>> +
>>  		entry->msi_attrib.default_irq	= dev->irq;
>>  		entry->mask_base		= base;
>>  
>> -		if (!entry->msi_attrib.is_virtual) {
>> +		if (!entry->msi_attrib.can_mask) {
>>  			addr = pci_msix_desc_addr(entry);
>>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>  		}
>> --- a/include/linux/msi.h
>> +++ b/include/linux/msi.h
>> @@ -148,7 +148,7 @@ struct msi_desc {
>>  				u8	is_msix		: 1;
>>  				u8	multiple	: 3;
>>  				u8	multi_cap	: 3;
>> -				u8	maskbit		: 1;
>> +				u8	can_mask	: 1;
>>  				u8	is_64		: 1;
>>  				u8	is_virtual	: 1;
>>  				u16	entry_nr;
> Thanks,
> I'll test this out ASAP.
I'm adding

diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
index 6a5ecee6e567..28d509452958 100644
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -529,10 +529,10 @@ static bool msi_check_reservation_mode(struct irq_domain *domain,
 
 	/*
 	 * Checking the first MSI descriptor is sufficient. MSIX supports
-	 * masking and MSI does so when the maskbit is set.
+	 * masking and MSI does so when the can_mask is set.
 	 */
 	desc = first_msi_entry(dev);
-	return desc->msi_attrib.is_msix || desc->msi_attrib.maskbit;
+	return desc->msi_attrib.is_msix || desc->msi_attrib.can_mask;
 }
 
 int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 10/27/21 14:01, Josef Johansson wrote:
> On 10/27/21 11:54, Josef Johansson wrote:
>> On 10/27/21 11:50, Thomas Gleixner wrote:
>>> The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
>>> low level accessors into the higher level mask/unmask functions.
>>>
>>> This missed the fact that these accessors can be invoked from other places
>>> as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
>>> also violates the virtual MSIX and the msi_attrib.maskbit protections.
>>>
>>> Instead of sprinkling checks all over the place, lift them back into the
>>> low level accessor functions. To avoid checking three different conditions
>>> combine them into one property of msi_desc::msi_attrib.
>>>
>>> Reported-by: Josef Johansson <josef@oderland.se>
>>> Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask functions")
>>> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
>>> Cc: Jason Andryuk <jandryuk@gmail.com>
>>> Cc: Marc Zyngier <maz@kernel.org>
>>> Cc: Bjorn Helgaas <helgaas@kernel.org>
>>> Cc: linux-pci@vger.kernel.org
>>> Cc: xen-devel <xen-devel@lists.xenproject.org>
>>> Cc: Juergen Gross <jgross@suse.com>
>>> Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
>>> Cc: David Woodhouse <dwmw2@infradead.org>
>>> ---
>>>  drivers/pci/msi.c   |   26 ++++++++++++++------------
>>>  include/linux/msi.h |    2 +-
>>>  2 files changed, 15 insertions(+), 13 deletions(-)
>>>
>>> --- a/drivers/pci/msi.c
>>> +++ b/drivers/pci/msi.c
>>> @@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask
>>>  	raw_spinlock_t *lock = &desc->dev->msi_lock;
>>>  	unsigned long flags;
>>>  
>>> +	if (!desc->msi_attrib.can_mask)
>>> +		return;
>>> +
>>>  	raw_spin_lock_irqsave(lock, flags);
>>>  	desc->msi_mask &= ~clear;
>>>  	desc->msi_mask |= set;
>>> @@ -181,7 +184,8 @@ static void pci_msix_write_vector_ctrl(s
>>>  {
>>>  	void __iomem *desc_addr = pci_msix_desc_addr(desc);
>>>  
>>> -	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>> +	if (desc->msi_attrib.can_mask)
>>> +		writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>>  }
>>>  
>>>  static inline void pci_msix_mask(struct msi_desc *desc)
>>> @@ -200,23 +204,17 @@ static inline void pci_msix_unmask(struc
>>>  
>>>  static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
>>>  {
>>> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>> -		return;
>>> -
>>>  	if (desc->msi_attrib.is_msix)
>>>  		pci_msix_mask(desc);
>>> -	else if (desc->msi_attrib.maskbit)
>>> +	else
>>>  		pci_msi_mask(desc, mask);
>>>  }
>>>  
>>>  static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
>>>  {
>>> -	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
>>> -		return;
>>> -
>>>  	if (desc->msi_attrib.is_msix)
>>>  		pci_msix_unmask(desc);
>>> -	else if (desc->msi_attrib.maskbit)
>>> +	else
>>>  		pci_msi_unmask(desc, mask);
>>>  }
>>>  
>>> @@ -484,7 +482,8 @@ msi_setup_entry(struct pci_dev *dev, int
>>>  	entry->msi_attrib.is_64		= !!(control & PCI_MSI_FLAGS_64BIT);
>>>  	entry->msi_attrib.is_virtual    = 0;
>>>  	entry->msi_attrib.entry_nr	= 0;
>>> -	entry->msi_attrib.maskbit	= !!(control & PCI_MSI_FLAGS_MASKBIT);
>>> +	entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
>>> +					  !!(control & PCI_MSI_FLAGS_MASKBIT);
>>>  	entry->msi_attrib.default_irq	= dev->irq;	/* Save IOAPIC IRQ */
>>>  	entry->msi_attrib.multi_cap	= (control & PCI_MSI_FLAGS_QMASK) >> 1;
>>>  	entry->msi_attrib.multiple	= ilog2(__roundup_pow_of_two(nvec));
>>> @@ -495,7 +494,7 @@ msi_setup_entry(struct pci_dev *dev, int
>>>  		entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
>>>  
>>>  	/* Save the initial mask status */
>>> -	if (entry->msi_attrib.maskbit)
>>> +	if (entry->msi_attrib.can_mask)
>>>  		pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
>>>  
>>>  out:
>>> @@ -638,10 +637,13 @@ static int msix_setup_entries(struct pci
>>>  		entry->msi_attrib.is_virtual =
>>>  			entry->msi_attrib.entry_nr >= vec_count;
>>>  
>>> +		entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
>>> +						  !entry->msi_attrib.is_virtual;
>>> +
>>>  		entry->msi_attrib.default_irq	= dev->irq;
>>>  		entry->mask_base		= base;
>>>  
>>> -		if (!entry->msi_attrib.is_virtual) {
>>> +		if (!entry->msi_attrib.can_mask) {
>>>  			addr = pci_msix_desc_addr(entry);
>>>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>>  		}
>>> --- a/include/linux/msi.h
>>> +++ b/include/linux/msi.h
>>> @@ -148,7 +148,7 @@ struct msi_desc {
>>>  				u8	is_msix		: 1;
>>>  				u8	multiple	: 3;
>>>  				u8	multi_cap	: 3;
>>> -				u8	maskbit		: 1;
>>> +				u8	can_mask	: 1;
>>>  				u8	is_64		: 1;
>>>  				u8	is_virtual	: 1;
>>>  				u16	entry_nr;
>> Thanks,
>> I'll test this out ASAP.
> I'm adding
>
> diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
> index 6a5ecee6e567..28d509452958 100644
> --- a/kernel/irq/msi.c
> +++ b/kernel/irq/msi.c
> @@ -529,10 +529,10 @@ static bool msi_check_reservation_mode(struct irq_domain *domain,
>  
>  	/*
>  	 * Checking the first MSI descriptor is sufficient. MSIX supports
> -	 * masking and MSI does so when the maskbit is set.
> +	 * masking and MSI does so when the can_mask is set.
>  	 */
>  	desc = first_msi_entry(dev);
> -	return desc->msi_attrib.is_msix || desc->msi_attrib.maskbit;
> +	return desc->msi_attrib.is_msix || desc->msi_attrib.can_mask;
>  }
>  
>  int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
>
Hi Thomas,

With the above added the kernel boots fine and I can even suspend it twice.
Which is with my laptop, a good sign.

You can add Tested-By: josef@oderland.se.

Thanks again!

When I suspend I get errors from Xen, including stacktraces below
if anyone has any clue, if this might be related. I get one each time I
suspend
and the third time amdgpu gives up.

rtc_cmos 00:01: registered as rtc0
rtc_cmos 00:01: setting system clock to 2021-10-27T15:04:35 UTC (1635347075)
rtc_cmos 00:01: no alarms, y3k, 114 bytes nvram
device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
efifb: probing for efifb
efifb: cannot reserve video memory at 0x60000000
------------[ cut here ]------------
ioremap on RAM at 0x0000000060000000 - 0x00000000607e8fff
WARNING: CPU: 7 PID: 1 at arch/x86/mm/ioremap.c:210 __ioremap_caller+0x332/0x350
Modules linked in:
CPU: 7 PID: 1 Comm: swapper/0 Not tainted 5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
RIP: e030:__ioremap_caller+0x332/0x350
Code: e8 c3 ca ff ff 49 09 c6 e9 32 fe ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 35 f2 5d 82 c6 05 e8 7b a9 01 01 e8 48 39 be 00 <0f> 0b 45 31 e4 e9 ac fe ff ff e8 ff f5 c3 00 66 66 2e 0f 1f 84 00
RSP: e02b:ffffc9004007bb00 EFLAGS: 00010286
RAX: 0000000000000000 RBX: 00000000007e9000 RCX: ffffffff82915ca8
RDX: c0000000ffffdfff RSI: 0000000000000000 RDI: ffffffff82865ca0
RBP: 0000000060000000 R08: 0000000000000000 R09: ffffc9004007b948
R10: ffffc9004007b940 R11: ffffffff82945ce8 R12: 0000000000000001
R13: 00000000007e9000 R14: 00000000007e9000 R15: ffffffff81c8f772
FS:  0000000000000000(0000) GS:ffff8881407c0000(0000) knlGS:0000000000000000
CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000002810000 CR4: 0000000000050660
Call Trace:
 efifb_probe.cold+0x2e6/0x688
 platform_probe+0x3f/0x90
 call_driver_probe+0x24/0xc0
 really_probe+0x1e7/0x310
 __driver_probe_device+0xfe/0x180
 driver_probe_device+0x1e/0x90
 __device_attach_driver+0x72/0xe0
 ? driver_allows_async_probing+0x50/0x50
 ? driver_allows_async_probing+0x50/0x50
 bus_for_each_drv+0x8f/0xd0
 __device_attach+0xe9/0x1f0
 bus_probe_device+0x8e/0xa0
 device_add+0x3fb/0x630
 platform_device_add+0x102/0x230
 sysfb_init+0xea/0x141
 ? firmware_map_add_early+0xb8/0xb8
 do_one_initcall+0x57/0x200
 do_initcalls+0x109/0x166
 kernel_init_freeable+0x23c/0x2bd
 ? rest_init+0xc0/0xc0
 kernel_init+0x16/0x120
 ret_from_fork+0x22/0x30
---[ end trace b068d3cd1b7f5f49 ]---
efifb: abort, cannot remap video memory 0x7e9000 @ 0x60000000
efi-framebuffer: probe of efi-framebuffer.0 failed with error -5
--
printk: Suspending console(s) (use no_console_suspend to debug)
[drm] free PSP TMR buffer
PM: suspend devices took 0.428 seconds
ACPI: EC: interrupt blocked
ACPI: PM: Preparing to enter system sleep state S3
ACPI: EC: event blocked
ACPI: EC: EC stopped
ACPI: PM: Saving platform NVS memory
Disabling non-boot CPUs ...
------------[ cut here ]------------
WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522 switch_mm_irqs_off+0x3c5/0x400
Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
RIP: e030:switch_mm_irqs_off+0x3c5/0x400
Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
Call Trace:
 switch_mm+0x1c/0x30
 idle_task_exit+0x55/0x60
 play_dead_common+0xa/0x20
 xen_pv_play_dead+0xa/0x60
 do_idle+0xd1/0xe0
 cpu_startup_entry+0x19/0x20
 asm_cpu_bringup_and_idle+0x5/0x1000
---[ end trace b068d3cd1b7f5f4b ]---
smpboot: CPU 1 is now offline
smpboot: CPU 2 is now offline
smpboot: CPU 3 is now offline
smpboot: CPU 4 is now offline
smpboot: CPU 5 is now offline
smpboot: CPU 6 is now offline
smpboot: CPU 7 is now offline
ACPI: PM: Low-level resume complete
ACPI: EC: EC started
ACPI: PM: Restoring platform NVS memory
xen_acpi_processor: Uploading Xen processor PM info
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU11
--
CPU2 is up
installing Xen timer for CPU 3
cpu 3 spinlock event irq 79
[Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
ACPI: \_SB_.PLTF.C003: Found 3 idle states
ACPI: FW issue: working around C-state latencies out of order
CPU3 is up
------------[ cut here ]------------
cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
installing Xen timer for CPU 4
WARNING: CPU: 3 PID: 455 at kernel/sched/fair.c:3339 __update_blocked_fair+0x49b/0x4b0
Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
CPU: 3 PID: 455 Comm: kworker/3:2 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
Workqueue:  0x0 (events)
RIP: e030:__update_blocked_fair+0x49b/0x4b0
Code: 6b fd ff ff 49 8b 96 48 01 00 00 48 89 90 50 09 00 00 e9 ff fc ff ff 48 c7 c7 10 7a 5e 82 c6 05 f3 35 9e 01 01 e8 1f f3 b2 00 <0f> 0b 41 8b 86 38 01 00 00 e9 c6 fc ff ff 0f 1f 80 00 00 00 00 0f
RSP: e02b:ffffc900410d7ce0 EFLAGS: 00010082
RAX: 0000000000000000 RBX: 0000000000000018 RCX: ffff8881406d8a08
RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8881406d8a00
RBP: ffff8881406e9800 R08: 0000000000000048 R09: ffffc900410d7c78
R10: 0000000000000049 R11: 000000002d2d2d2d R12: ffff8881406e9f80
R13: ffff8881406e9e40 R14: ffff8881406e96c0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8881406c0000(0000) knlGS:0000000000000000
CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000782e51820000 CR3: 0000000002810000 CR4: 0000000000050660
Call Trace:
 update_blocked_averages+0xa8/0x180
 newidle_balance+0x175/0x380
 pick_next_task_fair+0x39/0x3e0
 pick_next_task+0x4c/0xbd0
 ? dequeue_task_fair+0xba/0x390
 __schedule+0x13a/0x570
 schedule+0x44/0xa0
 worker_thread+0xc0/0x320
 ? process_one_work+0x390/0x390
 kthread+0x10f/0x130
 ? set_kthread_struct+0x40/0x40
 ret_from_fork+0x22/0x30
---[ end trace b068d3cd1b7f5f4c ]---
cpu 4 spinlock event irq 85
[Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
ACPI: \_SB_.PLTF.C004: Found 3 idle states
ACPI: FW issue: working around C-state latencies out of order
CPU4 is up
installing Xen timer for CPU 5
cpu 5 spinlock event irq 91
[Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
ACPI: \_SB_.PLTF.C005: Found 3 idle states
ACPI: FW issue: working around C-state latencies out of order
CPU5 is up

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Thomas Gleixner]

Josef!

On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
> On 10/27/21 14:01, Josef Johansson wrote:
>> diff --git a/kernel/irq/msi.c b/kernel/irq/msi.c
>> index 6a5ecee6e567..28d509452958 100644
>> --- a/kernel/irq/msi.c
>> +++ b/kernel/irq/msi.c
>> @@ -529,10 +529,10 @@ static bool msi_check_reservation_mode(struct irq_domain *domain,
>>  
>>  	/*
>>  	 * Checking the first MSI descriptor is sufficient. MSIX supports
>> -	 * masking and MSI does so when the maskbit is set.
>> +	 * masking and MSI does so when the can_mask is set.
>>  	 */
>>  	desc = first_msi_entry(dev);
>> -	return desc->msi_attrib.is_msix || desc->msi_attrib.maskbit;
>> +	return desc->msi_attrib.is_msix || desc->msi_attrib.can_mask;
>>  }
>>  
>>  int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,
>>
> Hi Thomas,
>
> With the above added the kernel boots fine and I can even suspend it twice.
> Which is with my laptop, a good sign.
>
> You can add Tested-By: josef@oderland.se.

Thank you for fixing my quick hack in vacation mode. I'll send out a v2
in a minute.

Thanks,

        tglx
 

[PATCH v2] PCI/MSI: Move non-mask check back into low level accessors

[Thomas Gleixner]

From: Thomas Gleixner <tglx@linutronix.de>

The recent rework of PCI/MSI[X] masking moved the non-mask checks from the
low level accessors into the higher level mask/unmask functions.

This missed the fact that these accessors can be invoked from other places
as well. The missing checks break XEN-PV which sets pci_msi_ignore_mask and
also violates the virtual MSIX and the msi_attrib.maskbit protections.

Instead of sprinkling checks all over the place, lift them back into the
low level accessor functions. To avoid checking three different conditions
combine them into one property of msi_desc::msi_attrib.

[ josef: Fixed the missed conversion in the core code ]

Fixes: fcacdfbef5a1 ("PCI/MSI: Provide a new set of mask and unmask functions")
Reported-by: Josef Johansson <josef@oderland.se>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Tested-by: Josef Johansson <josef@oderland.se>
Cc: stable@vger.kernel.org
---
V2: Added the missing conversion in the core code - Josef
---
 drivers/pci/msi.c   |   26 ++++++++++++++------------
 include/linux/msi.h |    2 +-
 kernel/irq/msi.c    |    4 ++--
 3 files changed, 17 insertions(+), 15 deletions(-)

--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -148,6 +148,9 @@ static noinline void pci_msi_update_mask
 	raw_spinlock_t *lock = &desc->dev->msi_lock;
 	unsigned long flags;
 
+	if (!desc->msi_attrib.can_mask)
+		return;
+
 	raw_spin_lock_irqsave(lock, flags);
 	desc->msi_mask &= ~clear;
 	desc->msi_mask |= set;
@@ -181,7 +184,8 @@ static void pci_msix_write_vector_ctrl(s
 {
 	void __iomem *desc_addr = pci_msix_desc_addr(desc);
 
-	writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
+	if (desc->msi_attrib.can_mask)
+		writel(ctrl, desc_addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 }
 
 static inline void pci_msix_mask(struct msi_desc *desc)
@@ -200,23 +204,17 @@ static inline void pci_msix_unmask(struc
 
 static void __pci_msi_mask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_mask(desc);
-	else if (desc->msi_attrib.maskbit)
+	else
 		pci_msi_mask(desc, mask);
 }
 
 static void __pci_msi_unmask_desc(struct msi_desc *desc, u32 mask)
 {
-	if (pci_msi_ignore_mask || desc->msi_attrib.is_virtual)
-		return;
-
 	if (desc->msi_attrib.is_msix)
 		pci_msix_unmask(desc);
-	else if (desc->msi_attrib.maskbit)
+	else
 		pci_msi_unmask(desc, mask);
 }
 
@@ -484,7 +482,8 @@ msi_setup_entry(struct pci_dev *dev, int
 	entry->msi_attrib.is_64		= !!(control & PCI_MSI_FLAGS_64BIT);
 	entry->msi_attrib.is_virtual    = 0;
 	entry->msi_attrib.entry_nr	= 0;
-	entry->msi_attrib.maskbit	= !!(control & PCI_MSI_FLAGS_MASKBIT);
+	entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
+					  !!(control & PCI_MSI_FLAGS_MASKBIT);
 	entry->msi_attrib.default_irq	= dev->irq;	/* Save IOAPIC IRQ */
 	entry->msi_attrib.multi_cap	= (control & PCI_MSI_FLAGS_QMASK) >> 1;
 	entry->msi_attrib.multiple	= ilog2(__roundup_pow_of_two(nvec));
@@ -495,7 +494,7 @@ msi_setup_entry(struct pci_dev *dev, int
 		entry->mask_pos = dev->msi_cap + PCI_MSI_MASK_32;
 
 	/* Save the initial mask status */
-	if (entry->msi_attrib.maskbit)
+	if (entry->msi_attrib.can_mask)
 		pci_read_config_dword(dev, entry->mask_pos, &entry->msi_mask);
 
 out:
@@ -638,10 +637,13 @@ static int msix_setup_entries(struct pci
 		entry->msi_attrib.is_virtual =
 			entry->msi_attrib.entry_nr >= vec_count;
 
+		entry->msi_attrib.can_mask	= !pci_msi_ignore_mask &&
+						  !entry->msi_attrib.is_virtual;
+
 		entry->msi_attrib.default_irq	= dev->irq;
 		entry->mask_base		= base;
 
-		if (!entry->msi_attrib.is_virtual) {
+		if (!entry->msi_attrib.can_mask) {
 			addr = pci_msix_desc_addr(entry);
 			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
 		}
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -148,7 +148,7 @@ struct msi_desc {
 				u8	is_msix		: 1;
 				u8	multiple	: 3;
 				u8	multi_cap	: 3;
-				u8	maskbit		: 1;
+				u8	can_mask	: 1;
 				u8	is_64		: 1;
 				u8	is_virtual	: 1;
 				u16	entry_nr;
--- a/kernel/irq/msi.c
+++ b/kernel/irq/msi.c
@@ -529,10 +529,10 @@ static bool msi_check_reservation_mode(s
 
 	/*
 	 * Checking the first MSI descriptor is sufficient. MSIX supports
-	 * masking and MSI does so when the maskbit is set.
+	 * masking and MSI does so when the can_mask attribute is set.
 	 */
 	desc = first_msi_entry(dev);
-	return desc->msi_attrib.is_msix || desc->msi_attrib.maskbit;
+	return desc->msi_attrib.is_msix || desc->msi_attrib.can_mask;
 }
 
 int __msi_domain_alloc_irqs(struct irq_domain *domain, struct device *dev,

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Thomas Gleixner]

On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:

CC+: EFIFB and scheduler folks

> On 10/27/21 14:01, Josef Johansson wrote:
> When I suspend I get errors from Xen, including stacktraces below
> if anyone has any clue, if this might be related. I get one each time I
> suspend
> and the third time amdgpu gives up.
>
> rtc_cmos 00:01: registered as rtc0
> rtc_cmos 00:01: setting system clock to 2021-10-27T15:04:35 UTC (1635347075)
> rtc_cmos 00:01: no alarms, y3k, 114 bytes nvram
> device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
> device-mapper: uevent: version 1.0.3
> device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
> efifb: probing for efifb
> efifb: cannot reserve video memory at 0x60000000
> ------------[ cut here ]------------
> ioremap on RAM at 0x0000000060000000 - 0x00000000607e8fff
> WARNING: CPU: 7 PID: 1 at arch/x86/mm/ioremap.c:210 __ioremap_caller+0x332/0x350

That's this warning:

	/*
	 * Don't allow anybody to remap normal RAM that we're using..
	 */
	if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
		WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
			  &phys_addr, &last_addr);
		return NULL;
	}


> Modules linked in:
> CPU: 7 PID: 1 Comm: swapper/0 Not tainted 5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> RIP: e030:__ioremap_caller+0x332/0x350
> Code: e8 c3 ca ff ff 49 09 c6 e9 32 fe ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 35 f2 5d 82 c6 05 e8 7b a9 01 01 e8 48 39 be 00 <0f> 0b 45 31 e4 e9 ac fe ff ff e8 ff f5 c3 00 66 66 2e 0f 1f 84 00
> RSP: e02b:ffffc9004007bb00 EFLAGS: 00010286
> RAX: 0000000000000000 RBX: 00000000007e9000 RCX: ffffffff82915ca8
> RDX: c0000000ffffdfff RSI: 0000000000000000 RDI: ffffffff82865ca0
> RBP: 0000000060000000 R08: 0000000000000000 R09: ffffc9004007b948
> R10: ffffc9004007b940 R11: ffffffff82945ce8 R12: 0000000000000001
> R13: 00000000007e9000 R14: 00000000007e9000 R15: ffffffff81c8f772
> FS:  0000000000000000(0000) GS:ffff8881407c0000(0000) knlGS:0000000000000000
> CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000000000000000 CR3: 0000000002810000 CR4: 0000000000050660
> Call Trace:
>  efifb_probe.cold+0x2e6/0x688

Why is this probing EFIFB at resume? Josef is that hibernate or suspend
to RAM?

>  platform_probe+0x3f/0x90
>  call_driver_probe+0x24/0xc0
>  really_probe+0x1e7/0x310
>  __driver_probe_device+0xfe/0x180
>  driver_probe_device+0x1e/0x90
>  __device_attach_driver+0x72/0xe0
>  ? driver_allows_async_probing+0x50/0x50
>  ? driver_allows_async_probing+0x50/0x50
>  bus_for_each_drv+0x8f/0xd0
>  __device_attach+0xe9/0x1f0
>  bus_probe_device+0x8e/0xa0
>  device_add+0x3fb/0x630
>  platform_device_add+0x102/0x230
>  sysfb_init+0xea/0x141
>  ? firmware_map_add_early+0xb8/0xb8
>  do_one_initcall+0x57/0x200
>  do_initcalls+0x109/0x166
>  kernel_init_freeable+0x23c/0x2bd
>  ? rest_init+0xc0/0xc0
>  kernel_init+0x16/0x120
>  ret_from_fork+0x22/0x30
> ---[ end trace b068d3cd1b7f5f49 ]---
> efifb: abort, cannot remap video memory 0x7e9000 @ 0x60000000
> efi-framebuffer: probe of efi-framebuffer.0 failed with error -5
> --
> printk: Suspending console(s) (use no_console_suspend to debug)
> [drm] free PSP TMR buffer
> PM: suspend devices took 0.428 seconds
> ACPI: EC: interrupt blocked
> ACPI: PM: Preparing to enter system sleep state S3
> ACPI: EC: event blocked
> ACPI: EC: EC stopped
> ACPI: PM: Saving platform NVS memory
> Disabling non-boot CPUs ...
> ------------[ cut here ]------------
> WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522  switch_mm_irqs_off+0x3c5/0x400

	if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {

> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> RIP: e030:switch_mm_irqs_off+0x3c5/0x400
> Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
> RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
> RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
> RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
> RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
> R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
> FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
> CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
> CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
> Call Trace:
>  switch_mm+0x1c/0x30
>  idle_task_exit+0x55/0x60
>  play_dead_common+0xa/0x20
>  xen_pv_play_dead+0xa/0x60

So this is when bringing the non boot CPUs down and the switch_mm() code
discovers inconsistency between CR3 and the expected value.

Would probably be interesting to print the actual values, but XEN folks
might have an idea.

>  do_idle+0xd1/0xe0
>  cpu_startup_entry+0x19/0x20
>  asm_cpu_bringup_and_idle+0x5/0x1000
> ---[ end trace b068d3cd1b7f5f4b ]---
> smpboot: CPU 1 is now offline
> smpboot: CPU 2 is now offline
> smpboot: CPU 3 is now offline
> smpboot: CPU 4 is now offline
> smpboot: CPU 5 is now offline
> smpboot: CPU 6 is now offline
> smpboot: CPU 7 is now offline
> ACPI: PM: Low-level resume complete
> ACPI: EC: EC started
> ACPI: PM: Restoring platform NVS memory
> xen_acpi_processor: Uploading Xen processor PM info
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU11
> --
> CPU2 is up
> installing Xen timer for CPU 3
> cpu 3 spinlock event irq 79
> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> ACPI: \_SB_.PLTF.C003: Found 3 idle states
> ACPI: FW issue: working around C-state latencies out of order
> CPU3 is up
> ------------[ cut here ]------------
> cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
> installing Xen timer for CPU 4
> WARNING: CPU: 3 PID: 455 at kernel/sched/fair.c:3339  __update_blocked_fair+0x49b/0x4b0

	/*
	 * _avg must be null when _sum are null because _avg = _sum / divider
	 * Make sure that rounding and/or propagation of PELT values never
	 * break this.
	 */
	SCHED_WARN_ON(cfs_rq->avg.load_avg ||
		      cfs_rq->avg.util_avg ||
		      cfs_rq->avg.runnable_avg);

PeterZ, does that ring any bell?

> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> CPU: 3 PID: 455 Comm: kworker/3:2 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> Workqueue:  0x0 (events)
> RIP: e030:__update_blocked_fair+0x49b/0x4b0
> Code: 6b fd ff ff 49 8b 96 48 01 00 00 48 89 90 50 09 00 00 e9 ff fc ff ff 48 c7 c7 10 7a 5e 82 c6 05 f3 35 9e 01 01 e8 1f f3 b2 00 <0f> 0b 41 8b 86 38 01 00 00 e9 c6 fc ff ff 0f 1f 80 00 00 00 00 0f
> RSP: e02b:ffffc900410d7ce0 EFLAGS: 00010082
> RAX: 0000000000000000 RBX: 0000000000000018 RCX: ffff8881406d8a08
> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8881406d8a00
> RBP: ffff8881406e9800 R08: 0000000000000048 R09: ffffc900410d7c78
> R10: 0000000000000049 R11: 000000002d2d2d2d R12: ffff8881406e9f80
> R13: ffff8881406e9e40 R14: ffff8881406e96c0 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff8881406c0000(0000) knlGS:0000000000000000
> CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000782e51820000 CR3: 0000000002810000 CR4: 0000000000050660
> Call Trace:
>  update_blocked_averages+0xa8/0x180
>  newidle_balance+0x175/0x380
>  pick_next_task_fair+0x39/0x3e0
>  pick_next_task+0x4c/0xbd0
>  ? dequeue_task_fair+0xba/0x390
>  __schedule+0x13a/0x570
>  schedule+0x44/0xa0
>  worker_thread+0xc0/0x320
>  ? process_one_work+0x390/0x390
>  kthread+0x10f/0x130
>  ? set_kthread_struct+0x40/0x40
>  ret_from_fork+0x22/0x30
> ---[ end trace b068d3cd1b7f5f4c ]---
> cpu 4 spinlock event irq 85
> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> ACPI: \_SB_.PLTF.C004: Found 3 idle states
> ACPI: FW issue: working around C-state latencies out of order
> CPU4 is up
> installing Xen timer for CPU 5
> cpu 5 spinlock event irq 91
> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> ACPI: \_SB_.PLTF.C005: Found 3 idle states
> ACPI: FW issue: working around C-state latencies out of order
> CPU5 is up

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 11/4/21 00:45, Thomas Gleixner wrote:
> On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
>
> CC+: EFIFB and scheduler folks
>
>> On 10/27/21 14:01, Josef Johansson wrote:
>> When I suspend I get errors from Xen, including stacktraces below
>> if anyone has any clue, if this might be related. I get one each time I
>> suspend
>> and the third time amdgpu gives up.
>>
>> rtc_cmos 00:01: registered as rtc0
>> rtc_cmos 00:01: setting system clock to 2021-10-27T15:04:35 UTC (1635347075)
>> rtc_cmos 00:01: no alarms, y3k, 114 bytes nvram
>> device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
>> device-mapper: uevent: version 1.0.3
>> device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
>> efifb: probing for efifb
>> efifb: cannot reserve video memory at 0x60000000
>> ------------[ cut here ]------------
>> ioremap on RAM at 0x0000000060000000 - 0x00000000607e8fff
>> WARNING: CPU: 7 PID: 1 at arch/x86/mm/ioremap.c:210 __ioremap_caller+0x332/0x350
> That's this warning:
>
> 	/*
> 	 * Don't allow anybody to remap normal RAM that we're using..
> 	 */
> 	if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
> 		WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
> 			  &phys_addr, &last_addr);
> 		return NULL;
> 	}
>
>
>> Modules linked in:
>> CPU: 7 PID: 1 Comm: swapper/0 Not tainted 5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
>> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
>> RIP: e030:__ioremap_caller+0x332/0x350
>> Code: e8 c3 ca ff ff 49 09 c6 e9 32 fe ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 35 f2 5d 82 c6 05 e8 7b a9 01 01 e8 48 39 be 00 <0f> 0b 45 31 e4 e9 ac fe ff ff e8 ff f5 c3 00 66 66 2e 0f 1f 84 00
>> RSP: e02b:ffffc9004007bb00 EFLAGS: 00010286
>> RAX: 0000000000000000 RBX: 00000000007e9000 RCX: ffffffff82915ca8
>> RDX: c0000000ffffdfff RSI: 0000000000000000 RDI: ffffffff82865ca0
>> RBP: 0000000060000000 R08: 0000000000000000 R09: ffffc9004007b948
>> R10: ffffc9004007b940 R11: ffffffff82945ce8 R12: 0000000000000001
>> R13: 00000000007e9000 R14: 00000000007e9000 R15: ffffffff81c8f772
>> FS:  0000000000000000(0000) GS:ffff8881407c0000(0000) knlGS:0000000000000000
>> CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000000000000000 CR3: 0000000002810000 CR4: 0000000000050660
>> Call Trace:
>>  efifb_probe.cold+0x2e6/0x688
> Why is this probing EFIFB at resume? Josef is that hibernate or suspend
> to RAM?
This is actually on boot, might be a totally ignore able warning though.
Would be nice to actually get rid of!
>
>>  platform_probe+0x3f/0x90
>>  call_driver_probe+0x24/0xc0
>>  really_probe+0x1e7/0x310
>>  __driver_probe_device+0xfe/0x180
>>  driver_probe_device+0x1e/0x90
>>  __device_attach_driver+0x72/0xe0
>>  ? driver_allows_async_probing+0x50/0x50
>>  ? driver_allows_async_probing+0x50/0x50
>>  bus_for_each_drv+0x8f/0xd0
>>  __device_attach+0xe9/0x1f0
>>  bus_probe_device+0x8e/0xa0
>>  device_add+0x3fb/0x630
>>  platform_device_add+0x102/0x230
>>  sysfb_init+0xea/0x141
>>  ? firmware_map_add_early+0xb8/0xb8
>>  do_one_initcall+0x57/0x200
>>  do_initcalls+0x109/0x166
>>  kernel_init_freeable+0x23c/0x2bd
>>  ? rest_init+0xc0/0xc0
>>  kernel_init+0x16/0x120
>>  ret_from_fork+0x22/0x30
>> ---[ end trace b068d3cd1b7f5f49 ]---
>> efifb: abort, cannot remap video memory 0x7e9000 @ 0x60000000
>> efi-framebuffer: probe of efi-framebuffer.0 failed with error -5
>> --
>> printk: Suspending console(s) (use no_console_suspend to debug)
>> [drm] free PSP TMR buffer
>> PM: suspend devices took 0.428 seconds
>> ACPI: EC: interrupt blocked
>> ACPI: PM: Preparing to enter system sleep state S3
>> ACPI: EC: event blocked
>> ACPI: EC: EC stopped
>> ACPI: PM: Saving platform NVS memory
>> Disabling non-boot CPUs ...
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522  switch_mm_irqs_off+0x3c5/0x400
> 	if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {
>
>> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>> CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
>> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
>> RIP: e030:switch_mm_irqs_off+0x3c5/0x400
>> Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
>> RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
>> RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
>> RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
>> RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
>> R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
>> R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
>> FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
>> CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
>> CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
>> Call Trace:
>>  switch_mm+0x1c/0x30
>>  idle_task_exit+0x55/0x60
>>  play_dead_common+0xa/0x20
>>  xen_pv_play_dead+0xa/0x60
> So this is when bringing the non boot CPUs down and the switch_mm() code
> discovers inconsistency between CR3 and the expected value.
>
> Would probably be interesting to print the actual values, but XEN folks
> might have an idea.
My guess here after digging through the code is that XEN is just (as the
comment above this warn
says), just doing a load_cr3(swapper_pg_dir) without going through
switch_mm first.
I could add a BUG_ON somewhere, or maybe printk, but I'm very unsure to
where I should put them.
Assistance here would be great.
>
>>  do_idle+0xd1/0xe0
>>  cpu_startup_entry+0x19/0x20
>>  asm_cpu_bringup_and_idle+0x5/0x1000
>> ---[ end trace b068d3cd1b7f5f4b ]---
>> smpboot: CPU 1 is now offline
>> smpboot: CPU 2 is now offline
>> smpboot: CPU 3 is now offline
>> smpboot: CPU 4 is now offline
>> smpboot: CPU 5 is now offline
>> smpboot: CPU 6 is now offline
>> smpboot: CPU 7 is now offline
>> ACPI: PM: Low-level resume complete
>> ACPI: EC: EC started
>> ACPI: PM: Restoring platform NVS memory
>> xen_acpi_processor: Uploading Xen processor PM info
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU11
>> --
>> CPU2 is up
>> installing Xen timer for CPU 3
>> cpu 3 spinlock event irq 79
>> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
>> ACPI: \_SB_.PLTF.C003: Found 3 idle states
>> ACPI: FW issue: working around C-state latencies out of order
>> CPU3 is up
>> ------------[ cut here ]------------
>> cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
>> installing Xen timer for CPU 4
>> WARNING: CPU: 3 PID: 455 at kernel/sched/fair.c:3339  __update_blocked_fair+0x49b/0x4b0
> 	/*
> 	 * _avg must be null when _sum are null because _avg = _sum / divider
> 	 * Make sure that rounding and/or propagation of PELT values never
> 	 * break this.
> 	 */
> 	SCHED_WARN_ON(cfs_rq->avg.load_avg ||
> 		      cfs_rq->avg.util_avg ||
> 		      cfs_rq->avg.runnable_avg);
>
> PeterZ, does that ring any bell?
I also assume that the first BUG triggers this one.
>
>> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>> CPU: 3 PID: 455 Comm: kworker/3:2 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
>> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
>> Workqueue:  0x0 (events)
>> RIP: e030:__update_blocked_fair+0x49b/0x4b0
>> Code: 6b fd ff ff 49 8b 96 48 01 00 00 48 89 90 50 09 00 00 e9 ff fc ff ff 48 c7 c7 10 7a 5e 82 c6 05 f3 35 9e 01 01 e8 1f f3 b2 00 <0f> 0b 41 8b 86 38 01 00 00 e9 c6 fc ff ff 0f 1f 80 00 00 00 00 0f
>> RSP: e02b:ffffc900410d7ce0 EFLAGS: 00010082
>> RAX: 0000000000000000 RBX: 0000000000000018 RCX: ffff8881406d8a08
>> RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8881406d8a00
>> RBP: ffff8881406e9800 R08: 0000000000000048 R09: ffffc900410d7c78
>> R10: 0000000000000049 R11: 000000002d2d2d2d R12: ffff8881406e9f80
>> R13: ffff8881406e9e40 R14: ffff8881406e96c0 R15: 0000000000000000
>> FS:  0000000000000000(0000) GS:ffff8881406c0000(0000) knlGS:0000000000000000
>> CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
>> CR2: 0000782e51820000 CR3: 0000000002810000 CR4: 0000000000050660
>> Call Trace:
>>  update_blocked_averages+0xa8/0x180
>>  newidle_balance+0x175/0x380
>>  pick_next_task_fair+0x39/0x3e0
>>  pick_next_task+0x4c/0xbd0
>>  ? dequeue_task_fair+0xba/0x390
>>  __schedule+0x13a/0x570
>>  schedule+0x44/0xa0
>>  worker_thread+0xc0/0x320
>>  ? process_one_work+0x390/0x390
>>  kthread+0x10f/0x130
>>  ? set_kthread_struct+0x40/0x40
>>  ret_from_fork+0x22/0x30
>> ---[ end trace b068d3cd1b7f5f4c ]---
>> cpu 4 spinlock event irq 85
>> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
>> ACPI: \_SB_.PLTF.C004: Found 3 idle states
>> ACPI: FW issue: working around C-state latencies out of order
>> CPU4 is up
>> installing Xen timer for CPU 5
>> cpu 5 spinlock event irq 91
>> [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
>> ACPI: \_SB_.PLTF.C005: Found 3 idle states
>> ACPI: FW issue: working around C-state latencies out of order
>> CPU5 is up

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Peter Zijlstra]

On Thu, Nov 04, 2021 at 12:45:29AM +0100, Thomas Gleixner wrote:
> On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
> > ------------[ cut here ]------------
> > cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
> > installing Xen timer for CPU 4
> > WARNING: CPU: 3 PID: 455 at kernel/sched/fair.c:3339  __update_blocked_fair+0x49b/0x4b0
> 
> 	/*
> 	 * _avg must be null when _sum are null because _avg = _sum / divider
> 	 * Make sure that rounding and/or propagation of PELT values never
> 	 * break this.
> 	 */
> 	SCHED_WARN_ON(cfs_rq->avg.load_avg ||
> 		      cfs_rq->avg.util_avg ||
> 		      cfs_rq->avg.runnable_avg);
> 
> PeterZ, does that ring any bell?

Hurrr.. I thought we fixed all those. Vincent?

> > Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> > CPU: 3 PID: 455 Comm: kworker/3:2 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> > Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> > Workqueue:  0x0 (events)
> > RIP: e030:__update_blocked_fair+0x49b/0x4b0
> > Code: 6b fd ff ff 49 8b 96 48 01 00 00 48 89 90 50 09 00 00 e9 ff fc ff ff 48 c7 c7 10 7a 5e 82 c6 05 f3 35 9e 01 01 e8 1f f3 b2 00 <0f> 0b 41 8b 86 38 01 00 00 e9 c6 fc ff ff 0f 1f 80 00 00 00 00 0f
> > RSP: e02b:ffffc900410d7ce0 EFLAGS: 00010082
> > RAX: 0000000000000000 RBX: 0000000000000018 RCX: ffff8881406d8a08
> > RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8881406d8a00
> > RBP: ffff8881406e9800 R08: 0000000000000048 R09: ffffc900410d7c78
> > R10: 0000000000000049 R11: 000000002d2d2d2d R12: ffff8881406e9f80
> > R13: ffff8881406e9e40 R14: ffff8881406e96c0 R15: 0000000000000000
> > FS:  0000000000000000(0000) GS:ffff8881406c0000(0000) knlGS:0000000000000000
> > CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000782e51820000 CR3: 0000000002810000 CR4: 0000000000050660
> > Call Trace:
> >  update_blocked_averages+0xa8/0x180
> >  newidle_balance+0x175/0x380
> >  pick_next_task_fair+0x39/0x3e0
> >  pick_next_task+0x4c/0xbd0
> >  ? dequeue_task_fair+0xba/0x390
> >  __schedule+0x13a/0x570
> >  schedule+0x44/0xa0
> >  worker_thread+0xc0/0x320
> >  ? process_one_work+0x390/0x390
> >  kthread+0x10f/0x130
> >  ? set_kthread_struct+0x40/0x40
> >  ret_from_fork+0x22/0x30

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Vincent Guittot]

On Thu, 4 Nov 2021 at 00:45, Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
>
> CC+: EFIFB and scheduler folks
>
> > On 10/27/21 14:01, Josef Johansson wrote:
> > When I suspend I get errors from Xen, including stacktraces below
> > if anyone has any clue, if this might be related. I get one each time I
> > suspend
> > and the third time amdgpu gives up.
> >
> > rtc_cmos 00:01: registered as rtc0
> > rtc_cmos 00:01: setting system clock to 2021-10-27T15:04:35 UTC (1635347075)
> > rtc_cmos 00:01: no alarms, y3k, 114 bytes nvram
> > device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.
> > device-mapper: uevent: version 1.0.3
> > device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
> > efifb: probing for efifb
> > efifb: cannot reserve video memory at 0x60000000
> > ------------[ cut here ]------------
> > ioremap on RAM at 0x0000000060000000 - 0x00000000607e8fff
> > WARNING: CPU: 7 PID: 1 at arch/x86/mm/ioremap.c:210 __ioremap_caller+0x332/0x350
>
> That's this warning:
>
>         /*
>          * Don't allow anybody to remap normal RAM that we're using..
>          */
>         if (io_desc.flags & IORES_MAP_SYSTEM_RAM) {
>                 WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
>                           &phys_addr, &last_addr);
>                 return NULL;
>         }
>
>
> > Modules linked in:
> > CPU: 7 PID: 1 Comm: swapper/0 Not tainted 5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> > Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> > RIP: e030:__ioremap_caller+0x332/0x350
> > Code: e8 c3 ca ff ff 49 09 c6 e9 32 fe ff ff 48 8d 54 24 28 48 8d 74 24 18 48 c7 c7 35 f2 5d 82 c6 05 e8 7b a9 01 01 e8 48 39 be 00 <0f> 0b 45 31 e4 e9 ac fe ff ff e8 ff f5 c3 00 66 66 2e 0f 1f 84 00
> > RSP: e02b:ffffc9004007bb00 EFLAGS: 00010286
> > RAX: 0000000000000000 RBX: 00000000007e9000 RCX: ffffffff82915ca8
> > RDX: c0000000ffffdfff RSI: 0000000000000000 RDI: ffffffff82865ca0
> > RBP: 0000000060000000 R08: 0000000000000000 R09: ffffc9004007b948
> > R10: ffffc9004007b940 R11: ffffffff82945ce8 R12: 0000000000000001
> > R13: 00000000007e9000 R14: 00000000007e9000 R15: ffffffff81c8f772
> > FS:  0000000000000000(0000) GS:ffff8881407c0000(0000) knlGS:0000000000000000
> > CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000000000000000 CR3: 0000000002810000 CR4: 0000000000050660
> > Call Trace:
> >  efifb_probe.cold+0x2e6/0x688
>
> Why is this probing EFIFB at resume? Josef is that hibernate or suspend
> to RAM?
>
> >  platform_probe+0x3f/0x90
> >  call_driver_probe+0x24/0xc0
> >  really_probe+0x1e7/0x310
> >  __driver_probe_device+0xfe/0x180
> >  driver_probe_device+0x1e/0x90
> >  __device_attach_driver+0x72/0xe0
> >  ? driver_allows_async_probing+0x50/0x50
> >  ? driver_allows_async_probing+0x50/0x50
> >  bus_for_each_drv+0x8f/0xd0
> >  __device_attach+0xe9/0x1f0
> >  bus_probe_device+0x8e/0xa0
> >  device_add+0x3fb/0x630
> >  platform_device_add+0x102/0x230
> >  sysfb_init+0xea/0x141
> >  ? firmware_map_add_early+0xb8/0xb8
> >  do_one_initcall+0x57/0x200
> >  do_initcalls+0x109/0x166
> >  kernel_init_freeable+0x23c/0x2bd
> >  ? rest_init+0xc0/0xc0
> >  kernel_init+0x16/0x120
> >  ret_from_fork+0x22/0x30
> > ---[ end trace b068d3cd1b7f5f49 ]---
> > efifb: abort, cannot remap video memory 0x7e9000 @ 0x60000000
> > efi-framebuffer: probe of efi-framebuffer.0 failed with error -5
> > --
> > printk: Suspending console(s) (use no_console_suspend to debug)
> > [drm] free PSP TMR buffer
> > PM: suspend devices took 0.428 seconds
> > ACPI: EC: interrupt blocked
> > ACPI: PM: Preparing to enter system sleep state S3
> > ACPI: EC: event blocked
> > ACPI: EC: EC stopped
> > ACPI: PM: Saving platform NVS memory
> > Disabling non-boot CPUs ...
> > ------------[ cut here ]------------
> > WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522  switch_mm_irqs_off+0x3c5/0x400
>
>         if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {
>
> > Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> > CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> > Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> > RIP: e030:switch_mm_irqs_off+0x3c5/0x400
> > Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
> > RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
> > RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
> > RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
> > RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
> > R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
> > R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
> > FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
> > CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
> > CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
> > Call Trace:
> >  switch_mm+0x1c/0x30
> >  idle_task_exit+0x55/0x60
> >  play_dead_common+0xa/0x20
> >  xen_pv_play_dead+0xa/0x60
>
> So this is when bringing the non boot CPUs down and the switch_mm() code
> discovers inconsistency between CR3 and the expected value.
>
> Would probably be interesting to print the actual values, but XEN folks
> might have an idea.
>
> >  do_idle+0xd1/0xe0
> >  cpu_startup_entry+0x19/0x20
> >  asm_cpu_bringup_and_idle+0x5/0x1000
> > ---[ end trace b068d3cd1b7f5f4b ]---
> > smpboot: CPU 1 is now offline
> > smpboot: CPU 2 is now offline
> > smpboot: CPU 3 is now offline
> > smpboot: CPU 4 is now offline
> > smpboot: CPU 5 is now offline
> > smpboot: CPU 6 is now offline
> > smpboot: CPU 7 is now offline
> > ACPI: PM: Low-level resume complete
> > ACPI: EC: EC started
> > ACPI: PM: Restoring platform NVS memory
> > xen_acpi_processor: Uploading Xen processor PM info
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
> > xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU11
> > --
> > CPU2 is up
> > installing Xen timer for CPU 3
> > cpu 3 spinlock event irq 79
> > [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> > ACPI: \_SB_.PLTF.C003: Found 3 idle states
> > ACPI: FW issue: working around C-state latencies out of order
> > CPU3 is up
> > ------------[ cut here ]------------
> > cfs_rq->avg.load_avg || cfs_rq->avg.util_avg || cfs_rq->avg.runnable_avg
> > installing Xen timer for CPU 4
> > WARNING: CPU: 3 PID: 455 at kernel/sched/fair.c:3339  __update_blocked_fair+0x49b/0x4b0
>
>         /*
>          * _avg must be null when _sum are null because _avg = _sum / divider
>          * Make sure that rounding and/or propagation of PELT values never
>          * break this.
>          */
>         SCHED_WARN_ON(cfs_rq->avg.load_avg ||
>                       cfs_rq->avg.util_avg ||
>                       cfs_rq->avg.runnable_avg);
>
> PeterZ, does that ring any bell?

This warning raises when the PELT signal is not propagated correctly
in the cfs hierarchy which can impact the fairness. This is a bit
strange to get this warning during a resume. CPU 3 has just been put
online so no propagation already  happened and everything should be
synced

>
> > Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
> > CPU: 3 PID: 455 Comm: kworker/3:2 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
> > Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
> > Workqueue:  0x0 (events)
> > RIP: e030:__update_blocked_fair+0x49b/0x4b0
> > Code: 6b fd ff ff 49 8b 96 48 01 00 00 48 89 90 50 09 00 00 e9 ff fc ff ff 48 c7 c7 10 7a 5e 82 c6 05 f3 35 9e 01 01 e8 1f f3 b2 00 <0f> 0b 41 8b 86 38 01 00 00 e9 c6 fc ff ff 0f 1f 80 00 00 00 00 0f
> > RSP: e02b:ffffc900410d7ce0 EFLAGS: 00010082
> > RAX: 0000000000000000 RBX: 0000000000000018 RCX: ffff8881406d8a08
> > RDX: 00000000ffffffd8 RSI: 0000000000000027 RDI: ffff8881406d8a00
> > RBP: ffff8881406e9800 R08: 0000000000000048 R09: ffffc900410d7c78
> > R10: 0000000000000049 R11: 000000002d2d2d2d R12: ffff8881406e9f80
> > R13: ffff8881406e9e40 R14: ffff8881406e96c0 R15: 0000000000000000
> > FS:  0000000000000000(0000) GS:ffff8881406c0000(0000) knlGS:0000000000000000
> > CS:  10000e030 DS: 0000 ES: 0000 CR0: 0000000080050033
> > CR2: 0000782e51820000 CR3: 0000000002810000 CR4: 0000000000050660
> > Call Trace:
> >  update_blocked_averages+0xa8/0x180
> >  newidle_balance+0x175/0x380
> >  pick_next_task_fair+0x39/0x3e0
> >  pick_next_task+0x4c/0xbd0
> >  ? dequeue_task_fair+0xba/0x390
> >  __schedule+0x13a/0x570
> >  schedule+0x44/0xa0
> >  worker_thread+0xc0/0x320
> >  ? process_one_work+0x390/0x390
> >  kthread+0x10f/0x130
> >  ? set_kthread_struct+0x40/0x40
> >  ret_from_fork+0x22/0x30
> > ---[ end trace b068d3cd1b7f5f4c ]---
> > cpu 4 spinlock event irq 85
> > [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> > ACPI: \_SB_.PLTF.C004: Found 3 idle states
> > ACPI: FW issue: working around C-state latencies out of order
> > CPU4 is up
> > installing Xen timer for CPU 5
> > cpu 5 spinlock event irq 91
> > [Firmware Bug]: ACPI MWAIT C-state 0x0 not supported by HW (0x0)
> > ACPI: \_SB_.PLTF.C005: Found 3 idle states
> > ACPI: FW issue: working around C-state latencies out of order
> > CPU5 is up

Re: [PATCH v2] PCI/MSI: Move non-mask check back into low level accessors

[Thomas Gleixner]

On Thu, Nov 04 2021 at 00:27, Thomas Gleixner wrote:
>  
> -		if (!entry->msi_attrib.is_virtual) {
> +		if (!entry->msi_attrib.can_mask) {

Groan. I'm a moron. This obviously needs to be

		if (entry->msi_attrib.can_mask) {

>  			addr = pci_msix_desc_addr(entry);
>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>  		}

Re: [PATCH v2] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 11/9/21 15:53, Thomas Gleixner wrote:
> On Thu, Nov 04 2021 at 00:27, Thomas Gleixner wrote:
>>  
>> -		if (!entry->msi_attrib.is_virtual) {
>> +		if (!entry->msi_attrib.can_mask) {
> Groan. I'm a moron. This obviously needs to be
>
> 		if (entry->msi_attrib.can_mask) {
I will compile and check. Thanks for being thorough.
>>  			addr = pci_msix_desc_addr(entry);
>>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>  		}

Re: [PATCH v2] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 11/10/21 14:31, Josef Johansson wrote:
> On 11/9/21 15:53, Thomas Gleixner wrote:
>> On Thu, Nov 04 2021 at 00:27, Thomas Gleixner wrote:
>>>  
>>> -		if (!entry->msi_attrib.is_virtual) {
>>> +		if (!entry->msi_attrib.can_mask) {
>> Groan. I'm a moron. This obviously needs to be
>>
>> 		if (entry->msi_attrib.can_mask) {
> I will compile and check. Thanks for being thorough.
This worked as well.
>>>  			addr = pci_msix_desc_addr(entry);
>>>  			entry->msix_ctrl = readl(addr + PCI_MSIX_ENTRY_VECTOR_CTRL);
>>>  		}

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 11/4/21 00:45, Thomas Gleixner wrote:
> On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
>
> CC+: EFIFB and scheduler folks
>
>> On 10/27/21 14:01, Josef Johansson wrote:
>>
>> printk: Suspending console(s) (use no_console_suspend to debug)
>> [drm] free PSP TMR buffer
>> PM: suspend devices took 0.428 seconds
>> ACPI: EC: interrupt blocked
>> ACPI: PM: Preparing to enter system sleep state S3
>> ACPI: EC: event blocked
>> ACPI: EC: EC stopped
>> ACPI: PM: Saving platform NVS memory
>> Disabling non-boot CPUs ...
>> ------------[ cut here ]------------
>> WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522  switch_mm_irqs_off+0x3c5/0x400
> 	if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {
>
>> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>> CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
>> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
>> RIP: e030:switch_mm_irqs_off+0x3c5/0x400
>> Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
>> RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
>> RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
>> RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
>> RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
>> R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
>> R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
>> FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
>> CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
>> CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
>> Call Trace:
>>  switch_mm+0x1c/0x30
>>  idle_task_exit+0x55/0x60
>>  play_dead_common+0xa/0x20
>>  xen_pv_play_dead+0xa/0x60
> So this is when bringing the non boot CPUs down and the switch_mm() code
> discovers inconsistency between CR3 and the expected value.
>
> Would probably be interesting to print the actual values, but XEN folks
> might have an idea.
I can install some print-statements showing some more info here.
I guess I will be getting memory addresses, we already know that CR3 is
0000000002810000

If you have any hints on how to do an effective print statement for this
please do say so :)
I'll try though and see what I find out.
>>  do_idle+0xd1/0xe0
>>  cpu_startup_entry+0x19/0x20
>>  asm_cpu_bringup_and_idle+0x5/0x1000
>> ---[ end trace b068d3cd1b7f5f4b ]---
>> smpboot: CPU 1 is now offline
>> smpboot: CPU 2 is now offline
>> smpboot: CPU 3 is now offline
>> smpboot: CPU 4 is now offline
>> smpboot: CPU 5 is now offline
>> smpboot: CPU 6 is now offline
>> smpboot: CPU 7 is now offline
>> ACPI: PM: Low-level resume complete
>> ACPI: EC: EC started
>> ACPI: PM: Restoring platform NVS memory
>> xen_acpi_processor: Uploading Xen processor PM info
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
>>

Re: [PATCH] PCI/MSI: Move non-mask check back into low level accessors

[Josef Johansson]

On 11/10/21 21:30, Josef Johansson wrote:
> On 11/4/21 00:45, Thomas Gleixner wrote:
>> On Wed, Oct 27 2021 at 17:29, Josef Johansson wrote:
>>
>> CC+: EFIFB and scheduler folks
>>
>>> On 10/27/21 14:01, Josef Johansson wrote:
>>>
>>> printk: Suspending console(s) (use no_console_suspend to debug)
>>> [drm] free PSP TMR buffer
>>> PM: suspend devices took 0.428 seconds
>>> ACPI: EC: interrupt blocked
>>> ACPI: PM: Preparing to enter system sleep state S3
>>> ACPI: EC: event blocked
>>> ACPI: EC: EC stopped
>>> ACPI: PM: Saving platform NVS memory
>>> Disabling non-boot CPUs ...
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 1 PID: 0 at arch/x86/mm/tlb.c:522  switch_mm_irqs_off+0x3c5/0x400
>> 	if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {
>>
>>> Modules linked in: snd_seq_dummy snd_hrtimer snd_seq snd_seq_device snd_timer nf_tables nfnetlink vfat fat intel_rapl_msr think_lmi firmware_attributes_class wmi_bmof intel_rapl_common pcspkr uvcvideo videobuf2_vmalloc videobuf2_memops joydev videobuf2_v4l2 sp5100_tco k10temp videobuf2_common i2c_piix4 iwlwifi videodev mc cfg80211 thinkpad_acpi ipmi_devintf ucsi_acpi platform_profile typec_ucsi ledtrig_audio ipmi_msghandler r8169 rfkill typec snd wmi soundcore video i2c_scmi fuse xenfs ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_crypt trusted asn1_encoder hid_multitouch amdgpu crct10dif_pclmul crc32_pclmul crc32c_intel gpu_sched i2c_algo_bit drm_ttm_helper ghash_clmulni_intel ttm serio_raw drm_kms_helper cec sdhci_pci cqhci sdhci xhci_pci drm xhci_pci_renesas nvme xhci_hcd ehci_pci mmc_core ehci_hcd nvme_core xen_acpi_processor xen_privcmd xen_pciback xen_blkback xen_gntalloc xen_gntdev xen_evtchn uinput
>>> CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W        --------- ---  5.15.0-0.rc7.0.fc32.qubes.x86_64 #1
>>> Hardware name: LENOVO 20Y1S02400/20Y1S02400, BIOS R1BET65W(1.34 ) 06/17/2021
>>> RIP: e030:switch_mm_irqs_off+0x3c5/0x400
>>> Code: f0 41 80 65 01 fb ba 01 00 00 00 49 8d b5 60 23 00 00 4c 89 ef 49 c7 85 68 23 00 00 60 1d 08 81 e8 a0 f3 08 00 e9 15 fd ff ff <0f> 0b e8 34 fa ff ff e9 ad fc ff ff 0f 0b e9 31 fe ff ff 0f 0b e9
>>> RSP: e02b:ffffc900400f3eb0 EFLAGS: 00010006
>>> RAX: 00000001336c6000 RBX: ffff888140660000 RCX: 0000000000000040
>>> RDX: ffff8881003027c0 RSI: 0000000000000000 RDI: ffff8881b36c6000
>>> RBP: ffffffff829d91c0 R08: 0000000000000000 R09: 0000000000000000
>>> R10: 0000000000000008 R11: 0000000000000000 R12: ffff888104e88440
>>> R13: ffff8881003027c0 R14: 0000000000000000 R15: 0000000000000001
>>> FS:  0000000000000000(0000) GS:ffff888140640000(0000) knlGS:0000000000000000
>>> CS:  10000e030 DS: 002b ES: 002b CR0: 0000000080050033
>>> CR2: 000060b7d78bf198 CR3: 0000000002810000 CR4: 0000000000050660
>>> Call Trace:
>>>  switch_mm+0x1c/0x30
>>>  idle_task_exit+0x55/0x60
>>>  play_dead_common+0xa/0x20
>>>  xen_pv_play_dead+0xa/0x60
>> So this is when bringing the non boot CPUs down and the switch_mm() code
>> discovers inconsistency between CR3 and the expected value.
>>
>> Would probably be interesting to print the actual values, but XEN folks
>> might have an idea.
> I can install some print-statements showing some more info here.
> I guess I will be getting memory addresses, we already know that CR3 is
> 0000000002810000
>
> If you have any hints on how to do an effective print statement for this
> please do say so :)
> I'll try though and see what I find out.

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 59ba2968af1b..511792852e9e 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -520,6 +520,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
 	 */
 #ifdef CONFIG_DEBUG_VM
 	if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) {
+		printk("josef-debug: cr3: %lx, build_cr3: %lx, (%px, %x)",
+			__read_cr3(),
+			build_cr3(real_prev->pgd, prev_asid),
+			real_prev->pgd, prev_asid);
 		/*
 		 * If we were to BUG here, we'd be very likely to kill
 		 * the system so hard that we don't see the call trace.

this patch gave me the three values which where already known,
__read_cr3 = CR3 = 0000000002810000
build_cr3 = RAX = 00000001336c6000
real_prev->pgd = RDI = ffff8881b36c6000
prev_asid = RSI = 0

Not sure what conclusions I should draw though.

>>>  do_idle+0xd1/0xe0
>>>  cpu_startup_entry+0x19/0x20
>>>  asm_cpu_bringup_and_idle+0x5/0x1000
>>> ---[ end trace b068d3cd1b7f5f4b ]---
>>> smpboot: CPU 1 is now offline
>>> smpboot: CPU 2 is now offline
>>> smpboot: CPU 3 is now offline
>>> smpboot: CPU 4 is now offline
>>> smpboot: CPU 5 is now offline
>>> smpboot: CPU 6 is now offline
>>> smpboot: CPU 7 is now offline
>>> ACPI: PM: Low-level resume complete
>>> ACPI: EC: EC started
>>> ACPI: PM: Restoring platform NVS memory
>>> xen_acpi_processor: Uploading Xen processor PM info
>>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU1
>>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU3
>>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU5
>>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU7
>>> xen_acpi_processor: (_PXX): Hypervisor error (-19) for ACPI CPU9
>>>