Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[ydirson]

Hello,

Xen passthrough of a boot GPU those days (at least in the small QubesOS world)
is mostly tested/documented for Intel iGPUs (or I missed something).
I've been trying to do that with a Renoir GPU (for context, the goal is
to have a xen domU dedicated to the GUI [3]).  I won't go into all the details
of my successive attempts in this email, various (relative) progress reports are
available at [0] (there are other things to be investigated listed there, but
at least some of them can possibly wait).  And I have surely missed more than
a couple of key points.

Summary of the setup:
- GPU protected from dom0 driver using pci-stub (gets access to the GPU via efifb
  until hopefully the GUI domain seizes it)
- host is Xen 4.14, dom0 uses Linux 5.14 (Qubes' kernel-latest)
- guest is a Xen HVM with running in a stub domain, launched through libvirt/libxl
- hackish enablement of the IGD passthrough codepaths through
  - libxl PCI VID hack: https://github.com/ydirson/xen/commit/4c9d4cb5c3dc1282ba83f17d15072c197b60281c
  - qemu BDF hack: https://github.com/ydirson/qemu/commit/6a165467e25864f1ae17390a44a9c1425ba67aed

The first problem encountered, i.e. that the guest amdgpu driver was not able
to access the PCI expansion ROM, I have hacked around for now by letting the
driver load as firmware a copy of the ROMdriver [1] - this was a 5.14.15 kernel
with the QubesOS patches (all reachable from this commit).

Doing this seems to make the driver happy on this aspect, but several issues
now become visible, and after some digging I feel some insights from people
familiar with the code gets really necessary :)

The first problems are shown below as [T0], my interpretation being:
1. Xorg aborts (audit: type=1701) -- should find a way to get more details, but
   that is surely not the root cause of what follows
2. a PSP command fails -- I cannot find any AMD documentation on how PSP works,
   that could possibly help
3. the PSP fails to load some firmware as part of its own init -- here I'm quite
   uncomfortable, I thought of the PSP as being distinct from the cpu cores and
   gpu, but here it appears as a disting IP *within* the gpu.  I also failed to
   find any detailed description of the whole stuff and their interactions.
4. following this failure the driver finishes (while initialization was still
   ongoing)
5. then vcn_v2_0_sw_fini() triggers a bad memory access, which appeared to be
   while dereferencing adev->vcn.inst->fw_shared_cpu_addr.

After adding traces on the individual IPs init/fini [2] showed that the vcn
sw_init was indeed run, and likely initialized this pointer.  Any idea how
it became invalid ?  One track I briefly followed was that some of the IP
init appears to be asynchronous (the failure in PSP init occurs after later
IPs get initialized), but that pointer seems to be initialized early and
synchronously by VCN sw_init.


Then, to workaround the problem with PSP not being able to initialized, I used
fw_load_type=0 to use direct loading (and noted that fw_load_type=1, advertised
as loading firmware using SMU, just does not do anything in the code).

The result, using 5.15.4 at this time, resulted in trace [T1].  The error surfacing
now is "ring kiq_2.1.0 test failed" with a timeout.  I had to dig the kernel commit
messages to discover that KIQ is a Kernel Interface Queue, and there are various
other acronyms around this (eg. "eop", whose introduction seems older than the
landing of the driver in the kernel) which really make it hard to be efficient at
understanding the code.  Will gladly be enlightened :)

And this also ends with the VCN sw_fini going fireworks, and a quick look at the
assembler seems to hint that although the code changed a bit, it is still the
same statement crashing.

Also noticed that ip_block_mask=0xfff7 to disable the PSP on this ASIC will do slightly
different things, but end up with the same errors.


I will gladly take any suggestion, pointers to additional information, etc :)

Best regards,
-- 
Yann


[0] https://forum.qubes-os.org/t/amd-igpu-passthrough-attempt/6766/
[1] https://github.com/ydirson/linux/commit/4ca50829aa44b29e8428328e913a0546568bf1c0
[2] https://github.com/ydirson/linux/commit/87004f9542b9a80b4fb838697312778cf47e4146
[3] https://www.qubes-os.org/news/2020/03/18/gui-domain/#gpu-passthrough-the-perfect-world-desktop-solution

[T0] 

[2021-11-23 21:05:52] [    4.297684] amdgpu 0000:00:05.0: amdgpu: Fetched VBIOS from firmware file
[2021-11-23 21:05:52] [    4.297709] amdgpu: ATOM BIOS: 113-RENOIR-025
[2021-11-23 21:05:52] [    4.302046] [drm] VCN decode is enabled in VM mode
[2021-11-23 21:05:52] [    4.302066] [drm] VCN encode is enabled in VM mode
[2021-11-23 21:05:52] [    4.302078] [drm] JPEG decode is enabled in VM mode
[2021-11-23 21:05:52] [    4.302144] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[2021-11-23 21:05:52] [    4.302181] amdgpu 0000:00:05.0: amdgpu: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
[2021-11-23 21:05:52] [    4.302217] amdgpu 0000:00:05.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
[2021-11-23 21:05:52] [    4.302246] amdgpu 0000:00:05.0: amdgpu: AGP: 267419648M 0x000000F800000000 - 0x0000FFFFFFFFFFFF
[2021-11-23 21:05:52] [    4.302268] mtrr: base(0x430000000) is not aligned on a size(0x20000000) boundary
[2021-11-23 21:05:52] [    4.302289] Failed to add WC MTRR for [000000000998bb55-00000000eb9e681e]; performance may suffer.
[2021-11-23 21:05:52] [    4.302295] [drm] Detected VRAM RAM=512M, BAR=512M
[2021-11-23 21:05:52] [    4.302341] [drm] RAM width 128bits DDR4
[2021-11-23 21:05:52] [    4.302401] [drm] amdgpu: 512M of VRAM memory ready
[2021-11-23 21:05:52] [    4.302412] [drm] amdgpu: 691M of GTT memory ready.
[2021-11-23 21:05:52] [    4.302437] [drm] GART: num cpu pages 262144, num gpu pages 262144
[2021-11-23 21:05:52] [    4.302565] [drm] PCIE GART of 1024M enabled.
[2021-11-23 21:05:52] [    4.302575] [drm] PTB located at 0x000000F400900000
[2021-11-23 21:05:52] [    4.312921] amdgpu 0000:00:05.0: amdgpu: PSP runtime database doesn't exist
[2021-11-23 21:05:52] [    4.342353] [drm] Loading DMUB firmware via PSP: version=0x01010019
[2021-11-23 21:05:52] [    4.346679] [drm] Found VCN firmware Version ENC: 1.14 DEC: 5 VEP: 0 Revision: 20
[2021-11-23 21:05:52] [    4.346723] amdgpu 0000:00:05.0: amdgpu: Will use PSP to load VCN firmware
[2021-11-23 21:05:52] [    4.978736] [drm] reserve 0x400000 from 0xf41f800000 for PSP TMR
...
[2021-11-23 21:05:53] [    5.379127] kauditd_printk_skb: 166 callbacks suppressed
[2021-11-23 21:05:53] [    5.379130] audit: type=1130 audit(1637697953.112:140): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lightdm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2021-11-23 21:05:53] [    5.398103] audit: type=1701 audit(1637697953.131:141): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=873 comm="Xorg" exe="/usr/libexec/Xorg" sig=6 res=1
[2021-11-23 21:05:53] [    5.418058] audit: type=1334 audit(1637697953.150:142): prog-id=32 op=LOAD
[2021-11-23 21:05:53] [    5.418081] audit: type=1334 audit(1637697953.150:143): prog-id=33 op=LOAD
[2021-11-23 21:05:53] [    5.418096] audit: type=1334 audit(1637697953.150:144): prog-id=34 op=LOAD
[2021-11-23 21:05:53] [    5.418401] audit: type=1130 audit(1637697953.151:145): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@2-878-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2021-11-23 21:05:53] [    5.454438] audit: type=1130 audit(1637697953.184:146): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-input-sender-keyboard@event7 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2021-11-23 21:05:53] [    5.464666] audit: type=1130 audit(1637697953.197:147): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-input-sender-keyboard@event6 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2021-11-23 21:05:53] [    5.515827] audit: type=1131 audit(1637697953.248:148): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@2-878-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[2021-11-23 21:05:53] [    5.521839] audit: type=1131 audit(1637697953.254:149): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lightdm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[2021-11-23 21:05:55] [    7.675982] [drm] psp command (0xFFFFFFFF) failed and response status is (0xFFFFFFFF)
[2021-11-23 21:05:55] [    7.676007] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed!
[2021-11-23 21:05:55] [    7.676213] [drm:psp_hw_init [amdgpu]] *ERROR* PSP firmware loading failed
[2021-11-23 21:05:55] [    7.676371] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* hw_init of IP block <psp> failed -22
[2021-11-23 21:05:55] [    7.676530] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed
[2021-11-23 21:05:55] [    7.676563] amdgpu 0000:00:05.0: amdgpu: Fatal error during GPU init
[2021-11-23 21:05:55] [    7.676578] amdgpu 0000:00:05.0: amdgpu: amdgpu: finishing device.
[2021-11-23 21:05:55] [    7.679044] amdgpu: probe of 0000:00:05.0 failed with error -22
[2021-11-23 21:05:55] [    7.679102] BUG: unable to handle page fault for address: ffffb1f120cdf000
[2021-11-23 21:05:55] [    7.679117] #PF: supervisor write access in kernel mode
[2021-11-23 21:05:55] [    7.679129] #PF: error_code(0x0002) - not-present page
[2021-11-23 21:05:55] [    7.679140] PGD 1000067 P4D 1000067 PUD 11dc067 PMD 0 
[2021-11-23 21:05:55] [    7.679154] Oops: 0002 [#1] SMP NOPTI
[2021-11-23 21:05:55] [    7.679163] CPU: 0 PID: 276 Comm: systemd-udevd Not tainted 5.14.15-1.fc32.qubes.x86_64 #1
[2021-11-23 21:05:55] [    7.679180] Hardware name: Xen HVM domU, BIOS 4.14.3 11/14/2021
[2021-11-23 21:05:55] [    7.679194] RIP: 0010:vcn_v2_0_sw_fini+0x10/0x40 [amdgpu]
[2021-11-23 21:05:55] [    7.679367] Code: 66 f0 83 c2 81 c6 ea 05 00 00 31 c9 4c 89 cf e9 b6 4d ee ff 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 8b 87 38 17 01 00 48 89 fd <c7> 00 00 00 00 00 e8 d5 d5 f1 ff 48 89 ef e8 2d 20 ff ff 85 c0 74
[2021-11-23 21:05:55] [    7.679402] RSP: 0018:ffffb1f1002cfc30 EFLAGS: 00010206
[2021-11-23 21:05:55] [    7.679414] RAX: ffffb1f120cdf000 RBX: ffff8b4d9a675620 RCX: 0000000000000000
[2021-11-23 21:05:55] [    7.679429] RDX: 000000000000000e RSI: 0000000000000003 RDI: ffff8b4d9a660000
[2021-11-23 21:05:55] [    7.679444] RBP: ffff8b4d9a660000 R08: 000000000000000f R09: 000000008010000f
[2021-11-23 21:05:55] [    7.679459] R10: 0000000040000000 R11: 000000001b99d000 R12: ffff8b4d9a675590
[2021-11-23 21:05:55] [    7.679474] R13: ffff8b4d9a676400 R14: 000000000000000c R15: ffff8b4d813ef36c
[2021-11-23 21:05:55] [    7.679490] FS:  000073bc16d48380(0000) GS:ffff8b4dbcc00000(0000) knlGS:0000000000000000
[2021-11-23 21:05:55] [    7.679507] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2021-11-23 21:05:55] [    7.679520] CR2: ffffb1f120cdf000 CR3: 0000000004160000 CR4: 0000000000350ef0
[2021-11-23 21:05:55] [    7.679536] Call Trace:
[2021-11-23 21:05:55] [    7.679545]  amdgpu_device_ip_fini.isra.0+0xb6/0x1e0 [amdgpu]
[2021-11-23 21:05:55] [    7.679691]  amdgpu_device_fini_sw+0xe/0x100 [amdgpu]
[2021-11-23 21:05:55] [    7.679835]  amdgpu_driver_release_kms+0x12/0x30 [amdgpu]
[2021-11-23 21:05:55] [    7.679978]  devm_drm_dev_init_release+0x3d/0x60 [drm]
[2021-11-23 21:05:55] [    7.680008]  devres_release_all+0xb8/0x100
[2021-11-23 21:05:55] [    7.680019]  really_probe+0x100/0x310
[2021-11-23 21:05:55] [    7.680029]  __driver_probe_device+0xfe/0x180
[2021-11-23 21:05:55] [    7.680040]  driver_probe_device+0x1e/0x90
[2021-11-23 21:05:55] [    7.680050]  __driver_attach+0xc0/0x1c0
[2021-11-23 21:05:55] [    7.680059]  ? __device_attach_driver+0xe0/0xe0
[2021-11-23 21:05:55] [    7.680070]  ? __device_attach_driver+0xe0/0xe0
[2021-11-23 21:05:55] [    7.680081]  bus_for_each_dev+0x89/0xd0
[2021-11-23 21:05:55] [    7.680090]  bus_add_driver+0x12b/0x1e0
[2021-11-23 21:05:55] [    7.680099]  driver_register+0x8f/0xe0
[2021-11-23 21:05:55] [    7.680109]  ? 0xffffffffc0e7b000
[2021-11-23 21:05:55] [    7.680117]  do_one_initcall+0x57/0x200
[2021-11-23 21:05:55] [    7.680128]  do_init_module+0x5c/0x260
[2021-11-23 21:05:55] [    7.680137]  __do_sys_finit_module+0xae/0x110
[2021-11-23 21:05:55] [    7.680149]  do_syscall_64+0x3b/0x90
[2021-11-23 21:05:55] [    7.680158]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[2021-11-23 21:05:55] [    7.680170] RIP: 0033:0x73bc17ce9edd
[2021-11-23 21:05:55] [    7.680180] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6b 7f 0c 00 f7 d8 64 89 01 48
[2021-11-23 21:05:55] [    7.680215] RSP: 002b:00007fffa9b51688 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[2021-11-23 21:05:55] [    7.680231] RAX: ffffffffffffffda RBX: 0000602da93e3120 RCX: 000073bc17ce9edd
[2021-11-23 21:05:55] [    7.680246] RDX: 0000000000000000 RSI: 000073bc17e2732c RDI: 0000000000000014
[2021-11-23 21:05:55] [    7.680260] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000602da93e3bb0
[2021-11-23 21:05:55] [    7.680275] R10: 0000000000000014 R11: 0000000000000246 R12: 000073bc17e2732c
[2021-11-23 21:05:55] [    7.680290] R13: 0000602da9338960 R14: 0000000000000007 R15: 0000602da93e4000
[2021-11-23 21:05:55] [    7.680306] Modules linked in: joydev intel_rapl_msr amdgpu(+) intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ip6table_filter ip6table_mangle ip6table_raw ip6_tables iommu_v2 gpu_sched ipt_REJECT i2c_algo_bit nf_reject_ipv4 drm_ttm_helper ttm xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw drm_kms_helper ehci_pci xt_MASQUERADE iptable_nat nf_nat nf_conntrack ehci_hcd cec nf_defrag_ipv6 serio_raw nf_defrag_ipv4 i2c_piix4 ata_generic pata_acpi pcspkr xen_scsiback target_core_mod xen_netback uinput xen_privcmd xen_gntdev drm xen_gntalloc xen_blkback fuse xen_evtchn bpf_preload ip_tables overlay xen_blkfront
[2021-11-23 21:05:55] [    7.876218] CR2: ffffb1f120cdf000
[2021-11-23 21:05:55] [    7.876227] ---[ end trace 36c4552e098fcc4e ]---
[2021-11-23 21:05:55] [    7.876239] RIP: 0010:vcn_v2_0_sw_fini+0x10/0x40 [amdgpu]
[2021-11-23 21:05:55] [    7.876400] Code: 66 f0 83 c2 81 c6 ea 05 00 00 31 c9 4c 89 cf e9 b6 4d ee ff 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 8b 87 38 17 01 00 48 89 fd <c7> 00 00 00 00 00 e8 d5 d5 f1 ff 48 89 ef e8 2d 20 ff ff 85 c0 74
[2021-11-23 21:05:55] [    7.876439] RSP: 0018:ffffb1f1002cfc30 EFLAGS: 00010206
[2021-11-23 21:05:55] [    7.876451] RAX: ffffb1f120cdf000 RBX: ffff8b4d9a675620 RCX: 0000000000000000
[2021-11-23 21:05:55] [    7.876467] RDX: 000000000000000e RSI: 0000000000000003 RDI: ffff8b4d9a660000
[2021-11-23 21:05:55] [    7.876483] RBP: ffff8b4d9a660000 R08: 000000000000000f R09: 000000008010000f
[2021-11-23 21:05:55] [    7.876500] R10: 0000000040000000 R11: 000000001b99d000 R12: ffff8b4d9a675590
[2021-11-23 21:05:55] [    7.876515] R13: ffff8b4d9a676400 R14: 000000000000000c R15: ffff8b4d813ef36c
[2021-11-23 21:05:55] [    7.876533] FS:  000073bc16d48380(0000) GS:ffff8b4dbcc00000(0000) knlGS:0000000000000000
[2021-11-23 21:05:55] [    7.876551] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2021-11-23 21:05:55] [    7.876565] CR2: ffffb1f120cdf000 CR3: 0000000004160000 CR4: 0000000000350ef0
[2021-11-23 21:05:55] [    7.876582] Kernel panic - not syncing: Fatal exception
[2021-11-23 21:05:55] [    7.877654] Kernel Offset: 0x1000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)


[T1]

[2021-11-27 00:56:45] [    3.183789] [drm] Found VCN firmware Version ENC: 1.14 DEC: 5 VEP: 0 Revision: 20
[2021-11-27 00:56:45] [    3.207583] [drm] sw_init of IP block <jpeg_v2_0>...
[2021-11-27 00:56:45] [    3.210594] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!
[2021-11-27 00:56:45] [    3.227177] [drm] kiq ring mec 2 pipe 1 q 0
...
[2021-11-27 00:56:45] [    3.420287] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[2021-11-27 00:56:45] [    3.420485] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] *ERROR* KCQ enable failed
[2021-11-27 00:56:45] [    3.420681] [drm:amdgpu_device_ip_init [amdgpu]] *ERROR* hw_init of IP block <gfx_v9_0> failed -110
[2021-11-27 00:56:45] [    3.420878] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed
[2021-11-27 00:56:45] [    3.420893] amdgpu 0000:00:05.0: amdgpu: Fatal error during GPU init
[2021-11-27 00:56:45] [    3.420919] usb 1-1: New USB device found, idVendor=0627, idProduct=0001, bcdDevice= 0.00
[2021-11-27 00:56:45] [    3.420937] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=10
[2021-11-27 00:56:45] [    3.420954] usb 1-1: Product: QEMU USB Tablet
[2021-11-27 00:56:45] [    3.420964] usb 1-1: Manufacturer: QEMU
[2021-11-27 00:56:45] [    3.420973] usb 1-1: SerialNumber: 28754-0000:00:04.0-1
[2021-11-27 00:56:45] [    3.424285] amdgpu 0000:00:05.0: amdgpu: amdgpu: finishing device.
...
[2021-11-27 00:56:45] [    3.491635] amdgpu: probe of 0000:00:05.0 failed with error -110
[2021-11-27 00:56:45] [    3.492802] [drm] sw_fini of IP block <jpeg_v2_0>...
[2021-11-27 00:56:45] [    3.492827] [drm] sw_fini of IP block <vcn_v2_0>...
[2021-11-27 00:56:45] [    3.492843] BUG: unable to handle page fault for address: ffff9e9ea0d41000
[2021-11-27 00:56:45] [    3.492857] #PF: supervisor write access in kernel mode
[2021-11-27 00:56:45] [    3.492868] #PF: error_code(0x0002) - not-present page
[2021-11-27 00:56:45] [    3.492880] PGD 1000067 P4D 1000067 PUD 11db067 PMD 0 
[2021-11-27 00:56:45] [    3.492895] Oops: 0002 [#1] SMP NOPTI
[2021-11-27 00:56:45] [    3.492905] CPU: 0 PID: 280 Comm: systemd-udevd Not tainted 5.15.4-1.fc32.qubes.x86_64 #1
[2021-11-27 00:56:45] [    3.558408] Hardware name: Xen HVM domU, BIOS 4.14.3 11/25/2021
[2021-11-27 00:56:45] [    3.558422] RIP: 0010:vcn_v2_0_sw_fini+0x72/0x90 [amdgpu]
[2021-11-27 00:56:45] [    3.558598] Code: 89 ef e8 f1 1f ff ff 85 c0 75 08 48 89 ef e8 15 1e ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 18 48 83 c4 10 5b 5d c3 <c7> 03 00 00 00 00 8b 7c 24 04 e8 4f 48 ba ff eb bc e8 88 a2 58 ee
[2021-11-27 00:56:45] [    3.558637] RSP: 0018:ffff9e9e8054bc18 EFLAGS: 00010202
[2021-11-27 00:56:45] [    3.558649] RAX: 0000000000000001 RBX: ffff9e9ea0d41000 RCX: ffff8df77cc20a08
[2021-11-27 00:56:45] [    3.558665] RDX: 000030a703011a40 RSI: ffff9e9e8054bc1c RDI: ffffffffc0340f40
[2021-11-27 00:56:45] [    3.558682] RBP: ffff8df74e000000 R08: 0000000000000000 R09: ffff9e9e8054ba90
[2021-11-27 00:56:45] [    3.558698] R10: ffff9e9e8054ba88 R11: ffffffffaf945d68 R12: ffff8df74e015a70
[2021-11-27 00:56:45] [    3.558713] R13: ffff8df74e0168e0 R14: 000000000000000a R15: ffff8df741be637c
[2021-11-27 00:56:45] [    3.558731] FS:  00007f73d2d14380(0000) GS:ffff8df77cc00000(0000) knlGS:0000000000000000
[2021-11-27 00:56:45] [    3.558749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2021-11-27 00:56:45] [    3.558763] CR2: ffff9e9ea0d41000 CR3: 0000000006b82000 CR4: 0000000000350ef0
[2021-11-27 00:56:45] [    3.558781] Call Trace:
[2021-11-27 00:56:45] [    3.558788]  <TASK>
[2021-11-27 00:56:45] [    3.558795]  amdgpu_device_ip_fini.isra.0.cold+0x27/0x55 [amdgpu]
[2021-11-27 00:56:45] [    3.558992]  amdgpu_device_fini_sw+0x16/0x100 [amdgpu]
[2021-11-27 00:56:45] [    3.559144]  amdgpu_driver_release_kms+0x12/0x30 [amdgpu]
[2021-11-27 00:56:45] [    3.559290]  devm_drm_dev_init_release+0x3d/0x60 [drm]
[2021-11-27 00:56:45] [    3.559326]  devres_release_all+0xb8/0x100
[2021-11-27 00:56:45] [    3.559339]  really_probe+0x100/0x310
[2021-11-27 00:56:45] [    3.559350]  __driver_probe_device+0xfe/0x180
[2021-11-27 00:56:45] [    3.559362]  driver_probe_device+0x1e/0x90
[2021-11-27 00:56:45] [    3.559372]  __driver_attach+0xc0/0x1c0
[2021-11-27 00:56:45] [    3.559382]  ? __device_attach_driver+0xe0/0xe0
[2021-11-27 00:56:45] [    3.559393]  ? __device_attach_driver+0xe0/0xe0
[2021-11-27 00:56:45] [    3.559405]  bus_for_each_dev+0x89/0xd0
[2021-11-27 00:56:45] [    3.559414]  bus_add_driver+0x12b/0x1e0
[2021-11-27 00:56:45] [    3.559424]  driver_register+0x8f/0xe0
[2021-11-27 00:56:45] [    3.559433]  ? 0xffffffffc0df1000
[2021-11-27 00:56:45] [    3.559443]  do_one_initcall+0x57/0x200
[2021-11-27 00:56:45] [    3.559455]  do_init_module+0x5c/0x260
[2021-11-27 00:56:45] [    3.559465]  __do_sys_finit_module+0xae/0x110
[2021-11-27 00:56:45] [    3.559477]  do_syscall_64+0x3b/0x90
[2021-11-27 00:56:45] [    3.559488]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[2021-11-27 00:56:45] [    3.559502] RIP: 0033:0x7f73d3cb5edd
[2021-11-27 00:56:45] [    3.559512] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6b 7f 0c 00 f7 d8 64 89 01 48
[2021-11-27 00:56:45] [    3.559551] RSP: 002b:00007ffc83eee328 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[2021-11-27 00:56:45] [    3.559569] RAX: ffffffffffffffda RBX: 000060d4f098bf90 RCX: 00007f73d3cb5edd
[2021-11-27 00:56:45] [    3.559584] RDX: 0000000000000000 RSI: 000060d4f098c680 RDI: 0000000000000014
[2021-11-27 00:56:45] [    3.559600] RBP: 0000000000020000 R08: 0000000000000000 R09: 000060d4f098c6e0
[2021-11-27 00:56:45] [    3.559617] R10: 0000000000000014 R11: 0000000000000246 R12: 000060d4f098c680
[2021-11-27 00:56:45] [    3.559633] R13: 000060d4f08dc080 R14: 0000000000000000 R15: 000060d4f098cbf0
[2021-11-27 00:56:45] [    3.559650]  </TASK>
[2021-11-27 00:56:45] [    3.559656] Modules linked in: ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 amdgpu(+) intel_rapl_msr iommu_v2 gpu_sched i2c_algo_bit drm_ttm_helper intel_rapl_common ttm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper pcspkr cec serio_raw ehci_pci ehci_hcd i2c_piix4 ata_generic pata_acpi xen_scsiback target_core_mod xen_netback uinput xen_privcmd xen_gntdev xen_gntalloc fuse xen_blkback drm xen_evtchn bpf_preload ip_tables overlay xen_blkfront
[2021-11-27 00:56:45] [    3.559807] CR2: ffff9e9ea0d41000
[2021-11-27 00:56:45] [    3.559817] ---[ end trace 7cad684cd6eb2013 ]---
[2021-11-27 00:56:45] [    3.559829] RIP: 0010:vcn_v2_0_sw_fini+0x72/0x90 [amdgpu]
[2021-11-27 00:56:45] [    3.559994] Code: 89 ef e8 f1 1f ff ff 85 c0 75 08 48 89 ef e8 15 1e ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 18 48 83 c4 10 5b 5d c3 <c7> 03 00 00 00 00 8b 7c 24 04 e8 4f 48 ba ff eb bc e8 88 a2 58 ee
[2021-11-27 00:56:45] [    3.560033] RSP: 0018:ffff9e9e8054bc18 EFLAGS: 00010202
[2021-11-27 00:56:45] [    3.560045] RAX: 0000000000000001 RBX: ffff9e9ea0d41000 RCX: ffff8df77cc20a08
[2021-11-27 00:56:45] [    3.560062] RDX: 000030a703011a40 RSI: ffff9e9e8054bc1c RDI: ffffffffc0340f40
[2021-11-27 00:56:45] [    3.560078] RBP: ffff8df74e000000 R08: 0000000000000000 R09: ffff9e9e8054ba90
[2021-11-27 00:56:45] [    3.560094] R10: ffff9e9e8054ba88 R11: ffffffffaf945d68 R12: ffff8df74e015a70
[2021-11-27 00:56:45] [    3.560111] R13: ffff8df74e0168e0 R14: 000000000000000a R15: ffff8df741be637c
[2021-11-27 00:56:45] [    3.560130] FS:  00007f73d2d14380(0000) GS:ffff8df77cc00000(0000) knlGS:0000000000000000
[2021-11-27 00:56:45] [    3.560147] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2021-11-27 00:56:45] [    3.560159] CR2: ffff9e9ea0d41000 CR3: 0000000006b82000 CR4: 0000000000350ef0
[2021-11-27 00:56:45] [    3.560175] Kernel panic - not syncing: Fatal exception
[2021-11-27 00:56:45] [    3.561248] Kernel Offset: 0x2d000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Sat, Nov 27, 2021 at 11:28 AM <ydirson@free.fr> wrote:
>
> Hello,
>
> Xen passthrough of a boot GPU those days (at least in the small QubesOS world)
> is mostly tested/documented for Intel iGPUs (or I missed something).
> I've been trying to do that with a Renoir GPU (for context, the goal is
> to have a xen domU dedicated to the GUI [3]).  I won't go into all the details
> of my successive attempts in this email, various (relative) progress reports are
> available at [0] (there are other things to be investigated listed there, but
> at least some of them can possibly wait).  And I have surely missed more than
> a couple of key points.

We have not validated virtualization of our integrated GPUs.  I don't
know that it will work at all.  We had done a bit of testing but ran
into the same issues with the PSP, but never had a chance to debug
further because this feature is not productized.

>
> Summary of the setup:
> - GPU protected from dom0 driver using pci-stub (gets access to the GPU via efifb
>   until hopefully the GUI domain seizes it)
> - host is Xen 4.14, dom0 uses Linux 5.14 (Qubes' kernel-latest)
> - guest is a Xen HVM with running in a stub domain, launched through libvirt/libxl
> - hackish enablement of the IGD passthrough codepaths through
>   - libxl PCI VID hack: https://github.com/ydirson/xen/commit/4c9d4cb5c3dc1282ba83f17d15072c197b60281c
>   - qemu BDF hack: https://github.com/ydirson/qemu/commit/6a165467e25864f1ae17390a44a9c1425ba67aed
>
> The first problem encountered, i.e. that the guest amdgpu driver was not able
> to access the PCI expansion ROM, I have hacked around for now by letting the
> driver load as firmware a copy of the ROMdriver [1] - this was a 5.14.15 kernel
> with the QubesOS patches (all reachable from this commit).
>
> Doing this seems to make the driver happy on this aspect, but several issues
> now become visible, and after some digging I feel some insights from people
> familiar with the code gets really necessary :)
>
> The first problems are shown below as [T0], my interpretation being:
> 1. Xorg aborts (audit: type=1701) -- should find a way to get more details, but
>    that is surely not the root cause of what follows
> 2. a PSP command fails -- I cannot find any AMD documentation on how PSP works,
>    that could possibly help
> 3. the PSP fails to load some firmware as part of its own init -- here I'm quite
>    uncomfortable, I thought of the PSP as being distinct from the cpu cores and
>    gpu, but here it appears as a disting IP *within* the gpu.  I also failed to
>    find any detailed description of the whole stuff and their interactions.
> 4. following this failure the driver finishes (while initialization was still
>    ongoing)

You need a functional PSP to get the GPU driver up and running.

> 5. then vcn_v2_0_sw_fini() triggers a bad memory access, which appeared to be
>    while dereferencing adev->vcn.inst->fw_shared_cpu_addr.
>
> After adding traces on the individual IPs init/fini [2] showed that the vcn
> sw_init was indeed run, and likely initialized this pointer.  Any idea how
> it became invalid ?  One track I briefly followed was that some of the IP
> init appears to be asynchronous (the failure in PSP init occurs after later
> IPs get initialized), but that pointer seems to be initialized early and
> synchronously by VCN sw_init.
>
>
> Then, to workaround the problem with PSP not being able to initialized, I used
> fw_load_type=0 to use direct loading (and noted that fw_load_type=1, advertised
> as loading firmware using SMU, just does not do anything in the code).

That will not work on modern GPUs.  The PSP is required for firmware
loading.  Without firmware the various engines on the GPU (GFX,
compute, VCN) won't work.

>
> The result, using 5.15.4 at this time, resulted in trace [T1].  The error surfacing
> now is "ring kiq_2.1.0 test failed" with a timeout.  I had to dig the kernel commit
> messages to discover that KIQ is a Kernel Interface Queue, and there are various
> other acronyms around this (eg. "eop", whose introduction seems older than the
> landing of the driver in the kernel) which really make it hard to be efficient at
> understanding the code.  Will gladly be enlightened :)
>
> And this also ends with the VCN sw_fini going fireworks, and a quick look at the
> assembler seems to hint that although the code changed a bit, it is still the
> same statement crashing.
>
> Also noticed that ip_block_mask=0xfff7 to disable the PSP on this ASIC will do slightly
> different things, but end up with the same errors.
>
>
> I will gladly take any suggestion, pointers to additional information, etc :)

PSP is fundamental to the operation of the GPU.

Alex


>
> Best regards,
> --
> Yann
>
>
> [0] https://forum.qubes-os.org/t/amd-igpu-passthrough-attempt/6766/
> [1] https://github.com/ydirson/linux/commit/4ca50829aa44b29e8428328e913a0546568bf1c0
> [2] https://github.com/ydirson/linux/commit/87004f9542b9a80b4fb838697312778cf47e4146
> [3] https://www.qubes-os.org/news/2020/03/18/gui-domain/#gpu-passthrough-the-perfect-world-desktop-solution
>
> [T0]
>
> [2021-11-23 21:05:52] [    4.297684] amdgpu 0000:00:05.0: amdgpu: Fetched VBIOS from firmware file
> [2021-11-23 21:05:52] [    4.297709] amdgpu: ATOM BIOS: 113-RENOIR-025
> [2021-11-23 21:05:52] [    4.302046] [drm] VCN decode is enabled in VM mode
> [2021-11-23 21:05:52] [    4.302066] [drm] VCN encode is enabled in VM mode
> [2021-11-23 21:05:52] [    4.302078] [drm] JPEG decode is enabled in VM mode
> [2021-11-23 21:05:52] [    4.302144] [drm] vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
> [2021-11-23 21:05:52] [    4.302181] amdgpu 0000:00:05.0: amdgpu: VRAM: 512M 0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
> [2021-11-23 21:05:52] [    4.302217] amdgpu 0000:00:05.0: amdgpu: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
> [2021-11-23 21:05:52] [    4.302246] amdgpu 0000:00:05.0: amdgpu: AGP: 267419648M 0x000000F800000000 - 0x0000FFFFFFFFFFFF
> [2021-11-23 21:05:52] [    4.302268] mtrr: base(0x430000000) is not aligned on a size(0x20000000) boundary
> [2021-11-23 21:05:52] [    4.302289] Failed to add WC MTRR for [000000000998bb55-00000000eb9e681e]; performance may suffer.
> [2021-11-23 21:05:52] [    4.302295] [drm] Detected VRAM RAM=512M, BAR=512M
> [2021-11-23 21:05:52] [    4.302341] [drm] RAM width 128bits DDR4
> [2021-11-23 21:05:52] [    4.302401] [drm] amdgpu: 512M of VRAM memory ready
> [2021-11-23 21:05:52] [    4.302412] [drm] amdgpu: 691M of GTT memory ready.
> [2021-11-23 21:05:52] [    4.302437] [drm] GART: num cpu pages 262144, num gpu pages 262144
> [2021-11-23 21:05:52] [    4.302565] [drm] PCIE GART of 1024M enabled.
> [2021-11-23 21:05:52] [    4.302575] [drm] PTB located at 0x000000F400900000
> [2021-11-23 21:05:52] [    4.312921] amdgpu 0000:00:05.0: amdgpu: PSP runtime database doesn't exist
> [2021-11-23 21:05:52] [    4.342353] [drm] Loading DMUB firmware via PSP: version=0x01010019
> [2021-11-23 21:05:52] [    4.346679] [drm] Found VCN firmware Version ENC: 1.14 DEC: 5 VEP: 0 Revision: 20
> [2021-11-23 21:05:52] [    4.346723] amdgpu 0000:00:05.0: amdgpu: Will use PSP to load VCN firmware
> [2021-11-23 21:05:52] [    4.978736] [drm] reserve 0x400000 from 0xf41f800000 for PSP TMR
> ...
> [2021-11-23 21:05:53] [    5.379127] kauditd_printk_skb: 166 callbacks suppressed
> [2021-11-23 21:05:53] [    5.379130] audit: type=1130 audit(1637697953.112:140): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lightdm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> [2021-11-23 21:05:53] [    5.398103] audit: type=1701 audit(1637697953.131:141): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=873 comm="Xorg" exe="/usr/libexec/Xorg" sig=6 res=1
> [2021-11-23 21:05:53] [    5.418058] audit: type=1334 audit(1637697953.150:142): prog-id=32 op=LOAD
> [2021-11-23 21:05:53] [    5.418081] audit: type=1334 audit(1637697953.150:143): prog-id=33 op=LOAD
> [2021-11-23 21:05:53] [    5.418096] audit: type=1334 audit(1637697953.150:144): prog-id=34 op=LOAD
> [2021-11-23 21:05:53] [    5.418401] audit: type=1130 audit(1637697953.151:145): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@2-878-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> [2021-11-23 21:05:53] [    5.454438] audit: type=1130 audit(1637697953.184:146): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-input-sender-keyboard@event7 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> [2021-11-23 21:05:53] [    5.464666] audit: type=1130 audit(1637697953.197:147): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=qubes-input-sender-keyboard@event6 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> [2021-11-23 21:05:53] [    5.515827] audit: type=1131 audit(1637697953.248:148): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@2-878-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
> [2021-11-23 21:05:53] [    5.521839] audit: type=1131 audit(1637697953.254:149): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=lightdm comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
> [2021-11-23 21:05:55] [    7.675982] [drm] psp command (0xFFFFFFFF) failed and response status is (0xFFFFFFFF)
> [2021-11-23 21:05:55] [    7.676007] [drm:psp_hw_start [amdgpu]] *ERROR* PSP load tmr failed!
> [2021-11-23 21:05:55] [    7.676213] [drm:psp_hw_init [amdgpu]] *ERROR* PSP firmware loading failed
> [2021-11-23 21:05:55] [    7.676371] [drm:amdgpu_device_fw_loading [amdgpu]] *ERROR* hw_init of IP block <psp> failed -22
> [2021-11-23 21:05:55] [    7.676530] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed
> [2021-11-23 21:05:55] [    7.676563] amdgpu 0000:00:05.0: amdgpu: Fatal error during GPU init
> [2021-11-23 21:05:55] [    7.676578] amdgpu 0000:00:05.0: amdgpu: amdgpu: finishing device.
> [2021-11-23 21:05:55] [    7.679044] amdgpu: probe of 0000:00:05.0 failed with error -22
> [2021-11-23 21:05:55] [    7.679102] BUG: unable to handle page fault for address: ffffb1f120cdf000
> [2021-11-23 21:05:55] [    7.679117] #PF: supervisor write access in kernel mode
> [2021-11-23 21:05:55] [    7.679129] #PF: error_code(0x0002) - not-present page
> [2021-11-23 21:05:55] [    7.679140] PGD 1000067 P4D 1000067 PUD 11dc067 PMD 0
> [2021-11-23 21:05:55] [    7.679154] Oops: 0002 [#1] SMP NOPTI
> [2021-11-23 21:05:55] [    7.679163] CPU: 0 PID: 276 Comm: systemd-udevd Not tainted 5.14.15-1.fc32.qubes.x86_64 #1
> [2021-11-23 21:05:55] [    7.679180] Hardware name: Xen HVM domU, BIOS 4.14.3 11/14/2021
> [2021-11-23 21:05:55] [    7.679194] RIP: 0010:vcn_v2_0_sw_fini+0x10/0x40 [amdgpu]
> [2021-11-23 21:05:55] [    7.679367] Code: 66 f0 83 c2 81 c6 ea 05 00 00 31 c9 4c 89 cf e9 b6 4d ee ff 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 8b 87 38 17 01 00 48 89 fd <c7> 00 00 00 00 00 e8 d5 d5 f1 ff 48 89 ef e8 2d 20 ff ff 85 c0 74
> [2021-11-23 21:05:55] [    7.679402] RSP: 0018:ffffb1f1002cfc30 EFLAGS: 00010206
> [2021-11-23 21:05:55] [    7.679414] RAX: ffffb1f120cdf000 RBX: ffff8b4d9a675620 RCX: 0000000000000000
> [2021-11-23 21:05:55] [    7.679429] RDX: 000000000000000e RSI: 0000000000000003 RDI: ffff8b4d9a660000
> [2021-11-23 21:05:55] [    7.679444] RBP: ffff8b4d9a660000 R08: 000000000000000f R09: 000000008010000f
> [2021-11-23 21:05:55] [    7.679459] R10: 0000000040000000 R11: 000000001b99d000 R12: ffff8b4d9a675590
> [2021-11-23 21:05:55] [    7.679474] R13: ffff8b4d9a676400 R14: 000000000000000c R15: ffff8b4d813ef36c
> [2021-11-23 21:05:55] [    7.679490] FS:  000073bc16d48380(0000) GS:ffff8b4dbcc00000(0000) knlGS:0000000000000000
> [2021-11-23 21:05:55] [    7.679507] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [2021-11-23 21:05:55] [    7.679520] CR2: ffffb1f120cdf000 CR3: 0000000004160000 CR4: 0000000000350ef0
> [2021-11-23 21:05:55] [    7.679536] Call Trace:
> [2021-11-23 21:05:55] [    7.679545]  amdgpu_device_ip_fini.isra.0+0xb6/0x1e0 [amdgpu]
> [2021-11-23 21:05:55] [    7.679691]  amdgpu_device_fini_sw+0xe/0x100 [amdgpu]
> [2021-11-23 21:05:55] [    7.679835]  amdgpu_driver_release_kms+0x12/0x30 [amdgpu]
> [2021-11-23 21:05:55] [    7.679978]  devm_drm_dev_init_release+0x3d/0x60 [drm]
> [2021-11-23 21:05:55] [    7.680008]  devres_release_all+0xb8/0x100
> [2021-11-23 21:05:55] [    7.680019]  really_probe+0x100/0x310
> [2021-11-23 21:05:55] [    7.680029]  __driver_probe_device+0xfe/0x180
> [2021-11-23 21:05:55] [    7.680040]  driver_probe_device+0x1e/0x90
> [2021-11-23 21:05:55] [    7.680050]  __driver_attach+0xc0/0x1c0
> [2021-11-23 21:05:55] [    7.680059]  ? __device_attach_driver+0xe0/0xe0
> [2021-11-23 21:05:55] [    7.680070]  ? __device_attach_driver+0xe0/0xe0
> [2021-11-23 21:05:55] [    7.680081]  bus_for_each_dev+0x89/0xd0
> [2021-11-23 21:05:55] [    7.680090]  bus_add_driver+0x12b/0x1e0
> [2021-11-23 21:05:55] [    7.680099]  driver_register+0x8f/0xe0
> [2021-11-23 21:05:55] [    7.680109]  ? 0xffffffffc0e7b000
> [2021-11-23 21:05:55] [    7.680117]  do_one_initcall+0x57/0x200
> [2021-11-23 21:05:55] [    7.680128]  do_init_module+0x5c/0x260
> [2021-11-23 21:05:55] [    7.680137]  __do_sys_finit_module+0xae/0x110
> [2021-11-23 21:05:55] [    7.680149]  do_syscall_64+0x3b/0x90
> [2021-11-23 21:05:55] [    7.680158]  entry_SYSCALL_64_after_hwframe+0x44/0xae
> [2021-11-23 21:05:55] [    7.680170] RIP: 0033:0x73bc17ce9edd
> [2021-11-23 21:05:55] [    7.680180] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6b 7f 0c 00 f7 d8 64 89 01 48
> [2021-11-23 21:05:55] [    7.680215] RSP: 002b:00007fffa9b51688 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
> [2021-11-23 21:05:55] [    7.680231] RAX: ffffffffffffffda RBX: 0000602da93e3120 RCX: 000073bc17ce9edd
> [2021-11-23 21:05:55] [    7.680246] RDX: 0000000000000000 RSI: 000073bc17e2732c RDI: 0000000000000014
> [2021-11-23 21:05:55] [    7.680260] RBP: 0000000000020000 R08: 0000000000000000 R09: 0000602da93e3bb0
> [2021-11-23 21:05:55] [    7.680275] R10: 0000000000000014 R11: 0000000000000246 R12: 000073bc17e2732c
> [2021-11-23 21:05:55] [    7.680290] R13: 0000602da9338960 R14: 0000000000000007 R15: 0000602da93e4000
> [2021-11-23 21:05:55] [    7.680306] Modules linked in: joydev intel_rapl_msr amdgpu(+) intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ip6table_filter ip6table_mangle ip6table_raw ip6_tables iommu_v2 gpu_sched ipt_REJECT i2c_algo_bit nf_reject_ipv4 drm_ttm_helper ttm xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw drm_kms_helper ehci_pci xt_MASQUERADE iptable_nat nf_nat nf_conntrack ehci_hcd cec nf_defrag_ipv6 serio_raw nf_defrag_ipv4 i2c_piix4 ata_generic pata_acpi pcspkr xen_scsiback target_core_mod xen_netback uinput xen_privcmd xen_gntdev drm xen_gntalloc xen_blkback fuse xen_evtchn bpf_preload ip_tables overlay xen_blkfront
> [2021-11-23 21:05:55] [    7.876218] CR2: ffffb1f120cdf000
> [2021-11-23 21:05:55] [    7.876227] ---[ end trace 36c4552e098fcc4e ]---
> [2021-11-23 21:05:55] [    7.876239] RIP: 0010:vcn_v2_0_sw_fini+0x10/0x40 [amdgpu]
> [2021-11-23 21:05:55] [    7.876400] Code: 66 f0 83 c2 81 c6 ea 05 00 00 31 c9 4c 89 cf e9 b6 4d ee ff 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 8b 87 38 17 01 00 48 89 fd <c7> 00 00 00 00 00 e8 d5 d5 f1 ff 48 89 ef e8 2d 20 ff ff 85 c0 74
> [2021-11-23 21:05:55] [    7.876439] RSP: 0018:ffffb1f1002cfc30 EFLAGS: 00010206
> [2021-11-23 21:05:55] [    7.876451] RAX: ffffb1f120cdf000 RBX: ffff8b4d9a675620 RCX: 0000000000000000
> [2021-11-23 21:05:55] [    7.876467] RDX: 000000000000000e RSI: 0000000000000003 RDI: ffff8b4d9a660000
> [2021-11-23 21:05:55] [    7.876483] RBP: ffff8b4d9a660000 R08: 000000000000000f R09: 000000008010000f
> [2021-11-23 21:05:55] [    7.876500] R10: 0000000040000000 R11: 000000001b99d000 R12: ffff8b4d9a675590
> [2021-11-23 21:05:55] [    7.876515] R13: ffff8b4d9a676400 R14: 000000000000000c R15: ffff8b4d813ef36c
> [2021-11-23 21:05:55] [    7.876533] FS:  000073bc16d48380(0000) GS:ffff8b4dbcc00000(0000) knlGS:0000000000000000
> [2021-11-23 21:05:55] [    7.876551] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [2021-11-23 21:05:55] [    7.876565] CR2: ffffb1f120cdf000 CR3: 0000000004160000 CR4: 0000000000350ef0
> [2021-11-23 21:05:55] [    7.876582] Kernel panic - not syncing: Fatal exception
> [2021-11-23 21:05:55] [    7.877654] Kernel Offset: 0x1000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
>
>
> [T1]
>
> [2021-11-27 00:56:45] [    3.183789] [drm] Found VCN firmware Version ENC: 1.14 DEC: 5 VEP: 0 Revision: 20
> [2021-11-27 00:56:45] [    3.207583] [drm] sw_init of IP block <jpeg_v2_0>...
> [2021-11-27 00:56:45] [    3.210594] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!
> [2021-11-27 00:56:45] [    3.227177] [drm] kiq ring mec 2 pipe 1 q 0
> ...
> [2021-11-27 00:56:45] [    3.420287] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
> [2021-11-27 00:56:45] [    3.420485] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] *ERROR* KCQ enable failed
> [2021-11-27 00:56:45] [    3.420681] [drm:amdgpu_device_ip_init [amdgpu]] *ERROR* hw_init of IP block <gfx_v9_0> failed -110
> [2021-11-27 00:56:45] [    3.420878] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed
> [2021-11-27 00:56:45] [    3.420893] amdgpu 0000:00:05.0: amdgpu: Fatal error during GPU init
> [2021-11-27 00:56:45] [    3.420919] usb 1-1: New USB device found, idVendor=0627, idProduct=0001, bcdDevice= 0.00
> [2021-11-27 00:56:45] [    3.420937] usb 1-1: New USB device strings: Mfr=1, Product=3, SerialNumber=10
> [2021-11-27 00:56:45] [    3.420954] usb 1-1: Product: QEMU USB Tablet
> [2021-11-27 00:56:45] [    3.420964] usb 1-1: Manufacturer: QEMU
> [2021-11-27 00:56:45] [    3.420973] usb 1-1: SerialNumber: 28754-0000:00:04.0-1
> [2021-11-27 00:56:45] [    3.424285] amdgpu 0000:00:05.0: amdgpu: amdgpu: finishing device.
> ...
> [2021-11-27 00:56:45] [    3.491635] amdgpu: probe of 0000:00:05.0 failed with error -110
> [2021-11-27 00:56:45] [    3.492802] [drm] sw_fini of IP block <jpeg_v2_0>...
> [2021-11-27 00:56:45] [    3.492827] [drm] sw_fini of IP block <vcn_v2_0>...
> [2021-11-27 00:56:45] [    3.492843] BUG: unable to handle page fault for address: ffff9e9ea0d41000
> [2021-11-27 00:56:45] [    3.492857] #PF: supervisor write access in kernel mode
> [2021-11-27 00:56:45] [    3.492868] #PF: error_code(0x0002) - not-present page
> [2021-11-27 00:56:45] [    3.492880] PGD 1000067 P4D 1000067 PUD 11db067 PMD 0
> [2021-11-27 00:56:45] [    3.492895] Oops: 0002 [#1] SMP NOPTI
> [2021-11-27 00:56:45] [    3.492905] CPU: 0 PID: 280 Comm: systemd-udevd Not tainted 5.15.4-1.fc32.qubes.x86_64 #1
> [2021-11-27 00:56:45] [    3.558408] Hardware name: Xen HVM domU, BIOS 4.14.3 11/25/2021
> [2021-11-27 00:56:45] [    3.558422] RIP: 0010:vcn_v2_0_sw_fini+0x72/0x90 [amdgpu]
> [2021-11-27 00:56:45] [    3.558598] Code: 89 ef e8 f1 1f ff ff 85 c0 75 08 48 89 ef e8 15 1e ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 18 48 83 c4 10 5b 5d c3 <c7> 03 00 00 00 00 8b 7c 24 04 e8 4f 48 ba ff eb bc e8 88 a2 58 ee
> [2021-11-27 00:56:45] [    3.558637] RSP: 0018:ffff9e9e8054bc18 EFLAGS: 00010202
> [2021-11-27 00:56:45] [    3.558649] RAX: 0000000000000001 RBX: ffff9e9ea0d41000 RCX: ffff8df77cc20a08
> [2021-11-27 00:56:45] [    3.558665] RDX: 000030a703011a40 RSI: ffff9e9e8054bc1c RDI: ffffffffc0340f40
> [2021-11-27 00:56:45] [    3.558682] RBP: ffff8df74e000000 R08: 0000000000000000 R09: ffff9e9e8054ba90
> [2021-11-27 00:56:45] [    3.558698] R10: ffff9e9e8054ba88 R11: ffffffffaf945d68 R12: ffff8df74e015a70
> [2021-11-27 00:56:45] [    3.558713] R13: ffff8df74e0168e0 R14: 000000000000000a R15: ffff8df741be637c
> [2021-11-27 00:56:45] [    3.558731] FS:  00007f73d2d14380(0000) GS:ffff8df77cc00000(0000) knlGS:0000000000000000
> [2021-11-27 00:56:45] [    3.558749] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [2021-11-27 00:56:45] [    3.558763] CR2: ffff9e9ea0d41000 CR3: 0000000006b82000 CR4: 0000000000350ef0
> [2021-11-27 00:56:45] [    3.558781] Call Trace:
> [2021-11-27 00:56:45] [    3.558788]  <TASK>
> [2021-11-27 00:56:45] [    3.558795]  amdgpu_device_ip_fini.isra.0.cold+0x27/0x55 [amdgpu]
> [2021-11-27 00:56:45] [    3.558992]  amdgpu_device_fini_sw+0x16/0x100 [amdgpu]
> [2021-11-27 00:56:45] [    3.559144]  amdgpu_driver_release_kms+0x12/0x30 [amdgpu]
> [2021-11-27 00:56:45] [    3.559290]  devm_drm_dev_init_release+0x3d/0x60 [drm]
> [2021-11-27 00:56:45] [    3.559326]  devres_release_all+0xb8/0x100
> [2021-11-27 00:56:45] [    3.559339]  really_probe+0x100/0x310
> [2021-11-27 00:56:45] [    3.559350]  __driver_probe_device+0xfe/0x180
> [2021-11-27 00:56:45] [    3.559362]  driver_probe_device+0x1e/0x90
> [2021-11-27 00:56:45] [    3.559372]  __driver_attach+0xc0/0x1c0
> [2021-11-27 00:56:45] [    3.559382]  ? __device_attach_driver+0xe0/0xe0
> [2021-11-27 00:56:45] [    3.559393]  ? __device_attach_driver+0xe0/0xe0
> [2021-11-27 00:56:45] [    3.559405]  bus_for_each_dev+0x89/0xd0
> [2021-11-27 00:56:45] [    3.559414]  bus_add_driver+0x12b/0x1e0
> [2021-11-27 00:56:45] [    3.559424]  driver_register+0x8f/0xe0
> [2021-11-27 00:56:45] [    3.559433]  ? 0xffffffffc0df1000
> [2021-11-27 00:56:45] [    3.559443]  do_one_initcall+0x57/0x200
> [2021-11-27 00:56:45] [    3.559455]  do_init_module+0x5c/0x260
> [2021-11-27 00:56:45] [    3.559465]  __do_sys_finit_module+0xae/0x110
> [2021-11-27 00:56:45] [    3.559477]  do_syscall_64+0x3b/0x90
> [2021-11-27 00:56:45] [    3.559488]  entry_SYSCALL_64_after_hwframe+0x44/0xae
> [2021-11-27 00:56:45] [    3.559502] RIP: 0033:0x7f73d3cb5edd
> [2021-11-27 00:56:45] [    3.559512] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 6b 7f 0c 00 f7 d8 64 89 01 48
> [2021-11-27 00:56:45] [    3.559551] RSP: 002b:00007ffc83eee328 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
> [2021-11-27 00:56:45] [    3.559569] RAX: ffffffffffffffda RBX: 000060d4f098bf90 RCX: 00007f73d3cb5edd
> [2021-11-27 00:56:45] [    3.559584] RDX: 0000000000000000 RSI: 000060d4f098c680 RDI: 0000000000000014
> [2021-11-27 00:56:45] [    3.559600] RBP: 0000000000020000 R08: 0000000000000000 R09: 000060d4f098c6e0
> [2021-11-27 00:56:45] [    3.559617] R10: 0000000000000014 R11: 0000000000000246 R12: 000060d4f098c680
> [2021-11-27 00:56:45] [    3.559633] R13: 000060d4f08dc080 R14: 0000000000000000 R15: 000060d4f098cbf0
> [2021-11-27 00:56:45] [    3.559650]  </TASK>
> [2021-11-27 00:56:45] [    3.559656] Modules linked in: ip6table_filter ip6table_mangle ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 amdgpu(+) intel_rapl_msr iommu_v2 gpu_sched i2c_algo_bit drm_ttm_helper intel_rapl_common ttm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm_kms_helper pcspkr cec serio_raw ehci_pci ehci_hcd i2c_piix4 ata_generic pata_acpi xen_scsiback target_core_mod xen_netback uinput xen_privcmd xen_gntdev xen_gntalloc fuse xen_blkback drm xen_evtchn bpf_preload ip_tables overlay xen_blkfront
> [2021-11-27 00:56:45] [    3.559807] CR2: ffff9e9ea0d41000
> [2021-11-27 00:56:45] [    3.559817] ---[ end trace 7cad684cd6eb2013 ]---
> [2021-11-27 00:56:45] [    3.559829] RIP: 0010:vcn_v2_0_sw_fini+0x72/0x90 [amdgpu]
> [2021-11-27 00:56:45] [    3.559994] Code: 89 ef e8 f1 1f ff ff 85 c0 75 08 48 89 ef e8 15 1e ff ff 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 18 48 83 c4 10 5b 5d c3 <c7> 03 00 00 00 00 8b 7c 24 04 e8 4f 48 ba ff eb bc e8 88 a2 58 ee
> [2021-11-27 00:56:45] [    3.560033] RSP: 0018:ffff9e9e8054bc18 EFLAGS: 00010202
> [2021-11-27 00:56:45] [    3.560045] RAX: 0000000000000001 RBX: ffff9e9ea0d41000 RCX: ffff8df77cc20a08
> [2021-11-27 00:56:45] [    3.560062] RDX: 000030a703011a40 RSI: ffff9e9e8054bc1c RDI: ffffffffc0340f40
> [2021-11-27 00:56:45] [    3.560078] RBP: ffff8df74e000000 R08: 0000000000000000 R09: ffff9e9e8054ba90
> [2021-11-27 00:56:45] [    3.560094] R10: ffff9e9e8054ba88 R11: ffffffffaf945d68 R12: ffff8df74e015a70
> [2021-11-27 00:56:45] [    3.560111] R13: ffff8df74e0168e0 R14: 000000000000000a R15: ffff8df741be637c
> [2021-11-27 00:56:45] [    3.560130] FS:  00007f73d2d14380(0000) GS:ffff8df77cc00000(0000) knlGS:0000000000000000
> [2021-11-27 00:56:45] [    3.560147] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [2021-11-27 00:56:45] [    3.560159] CR2: ffff9e9ea0d41000 CR3: 0000000006b82000 CR4: 0000000000350ef0
> [2021-11-27 00:56:45] [    3.560175] Kernel panic - not syncing: Fatal exception
> [2021-11-27 00:56:45] [    3.561248] Kernel Offset: 0x2d000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Hi Alex,

> We have not validated virtualization of our integrated GPUs.  I don't
> know that it will work at all.  We had done a bit of testing but ran
> into the same issues with the PSP, but never had a chance to debug
> further because this feature is not productized.
...
> You need a functional PSP to get the GPU driver up and running.

Ah, thanks for the hint :)

I guess that if I want to have any chance to get the PSP working I'm
going to need more details on it.  A quick search some time ago mostly
brought reverse-engineering work, rather than official AMD doc.  Are
there some AMD resources I missed ?

Best regards,
-- 
Yann

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
>
> Hi Alex,
>
> > We have not validated virtualization of our integrated GPUs.  I don't
> > know that it will work at all.  We had done a bit of testing but ran
> > into the same issues with the PSP, but never had a chance to debug
> > further because this feature is not productized.
> ...
> > You need a functional PSP to get the GPU driver up and running.
>
> Ah, thanks for the hint :)
>
> I guess that if I want to have any chance to get the PSP working I'm
> going to need more details on it.  A quick search some time ago mostly
> brought reverse-engineering work, rather than official AMD doc.  Are
> there some AMD resources I missed ?

The driver code is pretty much it.  On APUs, the PSP is shared with
the CPU and the rest of the platform.  The GPU driver just interacts
with it for a few specific tasks:
1. Loading Trusted Applications (e.g., trusted firmware applications
that run on the PSP for specific functionality, e.g., HDCP and content
protection, etc.)
2. Validating and loading firmware for other engines on the SoC.  This
is required to use those engines.

I'm not too familiar with the PSP's path to memory from the GPU
perspective.  IIRC, most memory used by the PSP goes through carve out
"vram" on APUs so it should work, but I would double check if there
are any system memory allocations that used to interact with the PSP
and see if changing them to vram helps.  It does work with the IOMMU
enabled on bare metal, so it should work in passthrough as well in
theory.

Alex


>
> Best regards,
> --
> Yann

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Hi Alex,

> 
> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Hi Alex,
> >
> > > We have not validated virtualization of our integrated GPUs.  I
> > > don't
> > > know that it will work at all.  We had done a bit of testing but
> > > ran
> > > into the same issues with the PSP, but never had a chance to
> > > debug
> > > further because this feature is not productized.
> > ...
> > > You need a functional PSP to get the GPU driver up and running.
> >
> > Ah, thanks for the hint :)
> >
> > I guess that if I want to have any chance to get the PSP working
> > I'm
> > going to need more details on it.  A quick search some time ago
> > mostly
> > brought reverse-engineering work, rather than official AMD doc.
> >  Are
> > there some AMD resources I missed ?
> 
> The driver code is pretty much it.  On APUs, the PSP is shared with
> the CPU and the rest of the platform.  The GPU driver just interacts
> with it for a few specific tasks:
> 1. Loading Trusted Applications (e.g., trusted firmware applications
> that run on the PSP for specific functionality, e.g., HDCP and
> content
> protection, etc.)
> 2. Validating and loading firmware for other engines on the SoC.
>  This
> is required to use those engines.

After some digging, if I understand correctly, the PSP is the 3rd IP
getting its hw_init() called.  First comes soc15_common, then vega10_ih.

- soc15_common_init_hw does some writes through nbio_v7.0 functions,
  but does not query the hw to check before or after
- vega10_init_hw does some register reads as part of its work, but once
  it has written it does not check either

So PSP is the first one to check that "soc15" (I'm still not sure what
this one represents, really) is in fact alive and well.

Can't we check earlier that the chip is really listening to us ?

> 
> I'm not too familiar with the PSP's path to memory from the GPU
> perspective.  IIRC, most memory used by the PSP goes through carve
> out
> "vram" on APUs so it should work, but I would double check if there
> are any system memory allocations that used to interact with the PSP
> and see if changing them to vram helps.  It does work with the IOMMU
> enabled on bare metal, so it should work in passthrough as well in
> theory.
> 
> Alex
> 
> 
> >
> > Best regards,
> > --
> > Yann
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Wed, Dec 8, 2021 at 5:50 PM Yann Dirson <ydirson@free.fr> wrote:
>
> Hi Alex,
>
> >
> > On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > > Hi Alex,
> > >
> > > > We have not validated virtualization of our integrated GPUs.  I
> > > > don't
> > > > know that it will work at all.  We had done a bit of testing but
> > > > ran
> > > > into the same issues with the PSP, but never had a chance to
> > > > debug
> > > > further because this feature is not productized.
> > > ...
> > > > You need a functional PSP to get the GPU driver up and running.
> > >
> > > Ah, thanks for the hint :)
> > >
> > > I guess that if I want to have any chance to get the PSP working
> > > I'm
> > > going to need more details on it.  A quick search some time ago
> > > mostly
> > > brought reverse-engineering work, rather than official AMD doc.
> > >  Are
> > > there some AMD resources I missed ?
> >
> > The driver code is pretty much it.  On APUs, the PSP is shared with
> > the CPU and the rest of the platform.  The GPU driver just interacts
> > with it for a few specific tasks:
> > 1. Loading Trusted Applications (e.g., trusted firmware applications
> > that run on the PSP for specific functionality, e.g., HDCP and
> > content
> > protection, etc.)
> > 2. Validating and loading firmware for other engines on the SoC.
> >  This
> > is required to use those engines.
>
> After some digging, if I understand correctly, the PSP is the 3rd IP
> getting its hw_init() called.  First comes soc15_common, then vega10_ih.
>
> - soc15_common_init_hw does some writes through nbio_v7.0 functions,
>   but does not query the hw to check before or after
> - vega10_init_hw does some register reads as part of its work, but once
>   it has written it does not check either
>
> So PSP is the first one to check that "soc15" (I'm still not sure what
> this one represents, really) is in fact alive and well.
>
> Can't we check earlier that the chip is really listening to us ?

Each SoC is made up of hardware blocks that provide various different
functionality.  They are mostly independent and mostly initialized
independently.  I'm not sure what you would want to check.  In your
case, I don't think it's an issue of the chip not being functional
overall, but rather a problem specific to the failing block somehow
related to being in a virtualized environment.

Alex


>
> >
> > I'm not too familiar with the PSP's path to memory from the GPU
> > perspective.  IIRC, most memory used by the PSP goes through carve
> > out
> > "vram" on APUs so it should work, but I would double check if there
> > are any system memory allocations that used to interact with the PSP
> > and see if changing them to vram helps.  It does work with the IOMMU
> > enabled on bare metal, so it should work in passthrough as well in
> > theory.
> >
> > Alex
> >
> >
> > >
> > > Best regards,
> > > --
> > > Yann
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Hi Alex,
> >
> > > We have not validated virtualization of our integrated GPUs.  I
> > > don't
> > > know that it will work at all.  We had done a bit of testing but
> > > ran
> > > into the same issues with the PSP, but never had a chance to
> > > debug
> > > further because this feature is not productized.
> > ...
> > > You need a functional PSP to get the GPU driver up and running.
> >
> > Ah, thanks for the hint :)
> >
> > I guess that if I want to have any chance to get the PSP working
> > I'm
> > going to need more details on it.  A quick search some time ago
> > mostly
> > brought reverse-engineering work, rather than official AMD doc.
> >  Are
> > there some AMD resources I missed ?
> 
> The driver code is pretty much it.

Let's try to shed some more light on how things work, taking as excuse
psp_v12_0_ring_create().

First, register access through [RW]REG32_SOC15() is implemented in
terms of __[RW]REG32_SOC15_RLC__(), which is basically a [RW]REG32(),
except it has to be more complex in the SR-IOV case.
Has the RLC anything to do with SR-IOV ?

It accesses registers in the MMIO range of the MP0 IP, and the "MP0"
name correlates highly with MMIO accesses in PSP-handling code.
Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0 version
reported at v11.0.3 by discovery seems to contradict the use of v12.0
for RENOIR as set by soc15_set_ip_blocks(), or do I miss something ?

More generally (and mostly out of curiosity while we're at it), do we
have a way to match IPs listed at discovery time with the ones used
in the driver ?

---

As for the register names, maybe we could have a short explanation of
how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to be
a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not sure
of the "SMN" part -- that could refer to the "System Management Network",
described in [0] as an internal bus.  Are we accessing this register
through this SMN ?


>  On APUs, the PSP is shared with
> the CPU and the rest of the platform.  The GPU driver just interacts
> with it for a few specific tasks:
> 1. Loading Trusted Applications (e.g., trusted firmware applications
> that run on the PSP for specific functionality, e.g., HDCP and
> content
> protection, etc.)
> 2. Validating and loading firmware for other engines on the SoC.
>  This
> is required to use those engines.

Trying to understand in more details how we start the PSP up, I noticed
that psp_v12_0 has support for loading a sOS firmware, but never calls
init_sos_microcode() - and anyway there is no sos firmware for renoir
and green_sardine, which seem to be the only ASICs with this PSP version.
Is it something that's just not been completely wired up yet ?

That also rings a bell, that we have nothing about Secure OS in the doc
yet (not even the acronym in the glossary).


> I'm not too familiar with the PSP's path to memory from the GPU
> perspective.  IIRC, most memory used by the PSP goes through carve
> out
> "vram" on APUs so it should work, but I would double check if there
> are any system memory allocations that used to interact with the PSP
> and see if changing them to vram helps.  It does work with the IOMMU
> enabled on bare metal, so it should work in passthrough as well in
> theory.

I can see a single case in the PSP code where GTT is used instead of
vram: to create fw_pri_bo when SR-IOV is not used (and there has
to be a reason, since the SR-IOV code path does use vram).
Changing it to vram does not make a difference, but then the
only bo that seems to be used at that point is the one for the psp ring,
which is allocated in vram, so I'm not too much surprised.

Maybe I should double-check bo_create calls to hunt for more ?


[0] https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr> wrote:
>
> Alex wrote:
> > On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > > Hi Alex,
> > >
> > > > We have not validated virtualization of our integrated GPUs.  I
> > > > don't
> > > > know that it will work at all.  We had done a bit of testing but
> > > > ran
> > > > into the same issues with the PSP, but never had a chance to
> > > > debug
> > > > further because this feature is not productized.
> > > ...
> > > > You need a functional PSP to get the GPU driver up and running.
> > >
> > > Ah, thanks for the hint :)
> > >
> > > I guess that if I want to have any chance to get the PSP working
> > > I'm
> > > going to need more details on it.  A quick search some time ago
> > > mostly
> > > brought reverse-engineering work, rather than official AMD doc.
> > >  Are
> > > there some AMD resources I missed ?
> >
> > The driver code is pretty much it.
>
> Let's try to shed some more light on how things work, taking as excuse
> psp_v12_0_ring_create().
>
> First, register access through [RW]REG32_SOC15() is implemented in
> terms of __[RW]REG32_SOC15_RLC__(), which is basically a [RW]REG32(),
> except it has to be more complex in the SR-IOV case.
> Has the RLC anything to do with SR-IOV ?

When running the driver on a SR-IOV virtual function (VF), some
registers are not available directly via the VF's MMIO aperture so
they need to go through the RLC.  For bare metal or passthrough this
is not relevant.

>
> It accesses registers in the MMIO range of the MP0 IP, and the "MP0"
> name correlates highly with MMIO accesses in PSP-handling code.
> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0 version

Yes.

> reported at v11.0.3 by discovery seems to contradict the use of v12.0
> for RENOIR as set by soc15_set_ip_blocks(), or do I miss something ?

Typo in the ip discovery table on renoir.

>
> More generally (and mostly out of curiosity while we're at it), do we
> have a way to match IPs listed at discovery time with the ones used
> in the driver ?

In general, barring typos, the code is shared at the major version
level.  The actual code may or may not need changes to handle minor
revision changes in an IP.  The driver maps the IP versions from the
ip discovery table to the code contained in the driver.

>
> ---
>
> As for the register names, maybe we could have a short explanation of
> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to be
> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not sure
> of the "SMN" part -- that could refer to the "System Management Network",
> described in [0] as an internal bus.  Are we accessing this register
> through this SMN ?

These registers are just mailboxes for the PSP firmware.  All of the
C2PMSG registers functionality is defined by the PSP firmware.  They
are basically scratch registers used to communicate between the driver
and the PSP firmware.

>
>
> >  On APUs, the PSP is shared with
> > the CPU and the rest of the platform.  The GPU driver just interacts
> > with it for a few specific tasks:
> > 1. Loading Trusted Applications (e.g., trusted firmware applications
> > that run on the PSP for specific functionality, e.g., HDCP and
> > content
> > protection, etc.)
> > 2. Validating and loading firmware for other engines on the SoC.
> >  This
> > is required to use those engines.
>
> Trying to understand in more details how we start the PSP up, I noticed
> that psp_v12_0 has support for loading a sOS firmware, but never calls
> init_sos_microcode() - and anyway there is no sos firmware for renoir
> and green_sardine, which seem to be the only ASICs with this PSP version.
> Is it something that's just not been completely wired up yet ?

On APUs, the PSP is shared with the CPU so the PSP firmware is part of
the sbios image.  The driver doesn't load it.  We only load it on
dGPUs where the driver is responsible for the chip initialization.

>
> That also rings a bell, that we have nothing about Secure OS in the doc
> yet (not even the acronym in the glossary).
>
>
> > I'm not too familiar with the PSP's path to memory from the GPU
> > perspective.  IIRC, most memory used by the PSP goes through carve
> > out
> > "vram" on APUs so it should work, but I would double check if there
> > are any system memory allocations that used to interact with the PSP
> > and see if changing them to vram helps.  It does work with the IOMMU
> > enabled on bare metal, so it should work in passthrough as well in
> > theory.
>
> I can see a single case in the PSP code where GTT is used instead of
> vram: to create fw_pri_bo when SR-IOV is not used (and there has
> to be a reason, since the SR-IOV code path does use vram).
> Changing it to vram does not make a difference, but then the
> only bo that seems to be used at that point is the one for the psp ring,
> which is allocated in vram, so I'm not too much surprised.
>
> Maybe I should double-check bo_create calls to hunt for more ?

We looked into this a bit ourselves and ran into the same issues.
We'd probably need to debug this with the PSP team to make further
progress, but this was not productized so neither team had the
resources to delve further.

Alex

>
>
> [0] https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

Thinking about this more, I think the problem might be related to CPU
access to "VRAM".  APUs don't have dedicated VRAM, they use a reserved
carve out region at the top of system memory.  For CPU access to this
memory, we kmap the physical address of the carve out region of system
memory.  You'll need to make sure that region is accessible to the
guest.

Alex

On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Alex wrote:
> > > On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr> wrote:
> > > >
> > > > Hi Alex,
> > > >
> > > > > We have not validated virtualization of our integrated GPUs.  I
> > > > > don't
> > > > > know that it will work at all.  We had done a bit of testing but
> > > > > ran
> > > > > into the same issues with the PSP, but never had a chance to
> > > > > debug
> > > > > further because this feature is not productized.
> > > > ...
> > > > > You need a functional PSP to get the GPU driver up and running.
> > > >
> > > > Ah, thanks for the hint :)
> > > >
> > > > I guess that if I want to have any chance to get the PSP working
> > > > I'm
> > > > going to need more details on it.  A quick search some time ago
> > > > mostly
> > > > brought reverse-engineering work, rather than official AMD doc.
> > > >  Are
> > > > there some AMD resources I missed ?
> > >
> > > The driver code is pretty much it.
> >
> > Let's try to shed some more light on how things work, taking as excuse
> > psp_v12_0_ring_create().
> >
> > First, register access through [RW]REG32_SOC15() is implemented in
> > terms of __[RW]REG32_SOC15_RLC__(), which is basically a [RW]REG32(),
> > except it has to be more complex in the SR-IOV case.
> > Has the RLC anything to do with SR-IOV ?
>
> When running the driver on a SR-IOV virtual function (VF), some
> registers are not available directly via the VF's MMIO aperture so
> they need to go through the RLC.  For bare metal or passthrough this
> is not relevant.
>
> >
> > It accesses registers in the MMIO range of the MP0 IP, and the "MP0"
> > name correlates highly with MMIO accesses in PSP-handling code.
> > Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0 version
>
> Yes.
>
> > reported at v11.0.3 by discovery seems to contradict the use of v12.0
> > for RENOIR as set by soc15_set_ip_blocks(), or do I miss something ?
>
> Typo in the ip discovery table on renoir.
>
> >
> > More generally (and mostly out of curiosity while we're at it), do we
> > have a way to match IPs listed at discovery time with the ones used
> > in the driver ?
>
> In general, barring typos, the code is shared at the major version
> level.  The actual code may or may not need changes to handle minor
> revision changes in an IP.  The driver maps the IP versions from the
> ip discovery table to the code contained in the driver.
>
> >
> > ---
> >
> > As for the register names, maybe we could have a short explanation of
> > how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to be
> > a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not sure
> > of the "SMN" part -- that could refer to the "System Management Network",
> > described in [0] as an internal bus.  Are we accessing this register
> > through this SMN ?
>
> These registers are just mailboxes for the PSP firmware.  All of the
> C2PMSG registers functionality is defined by the PSP firmware.  They
> are basically scratch registers used to communicate between the driver
> and the PSP firmware.
>
> >
> >
> > >  On APUs, the PSP is shared with
> > > the CPU and the rest of the platform.  The GPU driver just interacts
> > > with it for a few specific tasks:
> > > 1. Loading Trusted Applications (e.g., trusted firmware applications
> > > that run on the PSP for specific functionality, e.g., HDCP and
> > > content
> > > protection, etc.)
> > > 2. Validating and loading firmware for other engines on the SoC.
> > >  This
> > > is required to use those engines.
> >
> > Trying to understand in more details how we start the PSP up, I noticed
> > that psp_v12_0 has support for loading a sOS firmware, but never calls
> > init_sos_microcode() - and anyway there is no sos firmware for renoir
> > and green_sardine, which seem to be the only ASICs with this PSP version.
> > Is it something that's just not been completely wired up yet ?
>
> On APUs, the PSP is shared with the CPU so the PSP firmware is part of
> the sbios image.  The driver doesn't load it.  We only load it on
> dGPUs where the driver is responsible for the chip initialization.
>
> >
> > That also rings a bell, that we have nothing about Secure OS in the doc
> > yet (not even the acronym in the glossary).
> >
> >
> > > I'm not too familiar with the PSP's path to memory from the GPU
> > > perspective.  IIRC, most memory used by the PSP goes through carve
> > > out
> > > "vram" on APUs so it should work, but I would double check if there
> > > are any system memory allocations that used to interact with the PSP
> > > and see if changing them to vram helps.  It does work with the IOMMU
> > > enabled on bare metal, so it should work in passthrough as well in
> > > theory.
> >
> > I can see a single case in the PSP code where GTT is used instead of
> > vram: to create fw_pri_bo when SR-IOV is not used (and there has
> > to be a reason, since the SR-IOV code path does use vram).
> > Changing it to vram does not make a difference, but then the
> > only bo that seems to be used at that point is the one for the psp ring,
> > which is allocated in vram, so I'm not too much surprised.
> >
> > Maybe I should double-check bo_create calls to hunt for more ?
>
> We looked into this a bit ourselves and ran into the same issues.
> We'd probably need to debug this with the PSP team to make further
> progress, but this was not productized so neither team had the
> resources to delve further.
>
> Alex
>
> >
> >
> > [0] https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> Thinking about this more, I think the problem might be related to CPU
> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> reserved
> carve out region at the top of system memory.  For CPU access to this
> memory, we kmap the physical address of the carve out region of
> system
> memory.  You'll need to make sure that region is accessible to the
> guest.

So basically, the non-virt flow is is: (video?) BIOS reserves memory, marks it
as reserved in e820, stores the physaddr somewhere, which the GPU driver gets.
Since I suppose this includes the framebuffer, this probably has to occur around
the moment the driver calls drm_aperture_remove_conflicting_pci_framebuffers()
(which happens before this hw init step), right ?

... which brings me to a point that's been puzzling me for some time, which is
that as the hw init fails, the efifb driver is still using the framebuffer.

Am I right in suspecting that efifb should get stripped of its ownership of the
fb aperture first, and that if I don't get a black screen on hw_init failure
that issue should be the first focus point ?

> 
> Alex
> 
> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher <alexdeucher@gmail.com>
> wrote:
> >
> > On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> > wrote:
> > >
> > > Alex wrote:
> > > > On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
> > > > wrote:
> > > > >
> > > > > Hi Alex,
> > > > >
> > > > > > We have not validated virtualization of our integrated
> > > > > > GPUs.  I
> > > > > > don't
> > > > > > know that it will work at all.  We had done a bit of
> > > > > > testing but
> > > > > > ran
> > > > > > into the same issues with the PSP, but never had a chance
> > > > > > to
> > > > > > debug
> > > > > > further because this feature is not productized.
> > > > > ...
> > > > > > You need a functional PSP to get the GPU driver up and
> > > > > > running.
> > > > >
> > > > > Ah, thanks for the hint :)
> > > > >
> > > > > I guess that if I want to have any chance to get the PSP
> > > > > working
> > > > > I'm
> > > > > going to need more details on it.  A quick search some time
> > > > > ago
> > > > > mostly
> > > > > brought reverse-engineering work, rather than official AMD
> > > > > doc.
> > > > >  Are
> > > > > there some AMD resources I missed ?
> > > >
> > > > The driver code is pretty much it.
> > >
> > > Let's try to shed some more light on how things work, taking as
> > > excuse
> > > psp_v12_0_ring_create().
> > >
> > > First, register access through [RW]REG32_SOC15() is implemented
> > > in
> > > terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > [RW]REG32(),
> > > except it has to be more complex in the SR-IOV case.
> > > Has the RLC anything to do with SR-IOV ?
> >
> > When running the driver on a SR-IOV virtual function (VF), some
> > registers are not available directly via the VF's MMIO aperture so
> > they need to go through the RLC.  For bare metal or passthrough
> > this
> > is not relevant.
> >
> > >
> > > It accesses registers in the MMIO range of the MP0 IP, and the
> > > "MP0"
> > > name correlates highly with MMIO accesses in PSP-handling code.
> > > Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> > > version
> >
> > Yes.
> >
> > > reported at v11.0.3 by discovery seems to contradict the use of
> > > v12.0
> > > for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > something ?
> >
> > Typo in the ip discovery table on renoir.
> >
> > >
> > > More generally (and mostly out of curiosity while we're at it),
> > > do we
> > > have a way to match IPs listed at discovery time with the ones
> > > used
> > > in the driver ?
> >
> > In general, barring typos, the code is shared at the major version
> > level.  The actual code may or may not need changes to handle minor
> > revision changes in an IP.  The driver maps the IP versions from
> > the
> > ip discovery table to the code contained in the driver.
> >
> > >
> > > ---
> > >
> > > As for the register names, maybe we could have a short
> > > explanation of
> > > how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to
> > > be
> > > a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
> > > sure
> > > of the "SMN" part -- that could refer to the "System Management
> > > Network",
> > > described in [0] as an internal bus.  Are we accessing this
> > > register
> > > through this SMN ?
> >
> > These registers are just mailboxes for the PSP firmware.  All of
> > the
> > C2PMSG registers functionality is defined by the PSP firmware.
> >  They
> > are basically scratch registers used to communicate between the
> > driver
> > and the PSP firmware.
> >
> > >
> > >
> > > >  On APUs, the PSP is shared with
> > > > the CPU and the rest of the platform.  The GPU driver just
> > > > interacts
> > > > with it for a few specific tasks:
> > > > 1. Loading Trusted Applications (e.g., trusted firmware
> > > > applications
> > > > that run on the PSP for specific functionality, e.g., HDCP and
> > > > content
> > > > protection, etc.)
> > > > 2. Validating and loading firmware for other engines on the
> > > > SoC.
> > > >  This
> > > > is required to use those engines.
> > >
> > > Trying to understand in more details how we start the PSP up, I
> > > noticed
> > > that psp_v12_0 has support for loading a sOS firmware, but never
> > > calls
> > > init_sos_microcode() - and anyway there is no sos firmware for
> > > renoir
> > > and green_sardine, which seem to be the only ASICs with this PSP
> > > version.
> > > Is it something that's just not been completely wired up yet ?
> >
> > On APUs, the PSP is shared with the CPU so the PSP firmware is part
> > of
> > the sbios image.  The driver doesn't load it.  We only load it on
> > dGPUs where the driver is responsible for the chip initialization.
> >
> > >
> > > That also rings a bell, that we have nothing about Secure OS in
> > > the doc
> > > yet (not even the acronym in the glossary).
> > >
> > >
> > > > I'm not too familiar with the PSP's path to memory from the GPU
> > > > perspective.  IIRC, most memory used by the PSP goes through
> > > > carve
> > > > out
> > > > "vram" on APUs so it should work, but I would double check if
> > > > there
> > > > are any system memory allocations that used to interact with
> > > > the PSP
> > > > and see if changing them to vram helps.  It does work with the
> > > > IOMMU
> > > > enabled on bare metal, so it should work in passthrough as well
> > > > in
> > > > theory.
> > >
> > > I can see a single case in the PSP code where GTT is used instead
> > > of
> > > vram: to create fw_pri_bo when SR-IOV is not used (and there has
> > > to be a reason, since the SR-IOV code path does use vram).
> > > Changing it to vram does not make a difference, but then the
> > > only bo that seems to be used at that point is the one for the
> > > psp ring,
> > > which is allocated in vram, so I'm not too much surprised.
> > >
> > > Maybe I should double-check bo_create calls to hunt for more ?
> >
> > We looked into this a bit ourselves and ran into the same issues.
> > We'd probably need to debug this with the PSP team to make further
> > progress, but this was not productized so neither team had the
> > resources to delve further.
> >
> > Alex
> >
> > >
> > >
> > > [0]
> > > https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Christian König]

Am 19.12.21 um 17:00 schrieb Yann Dirson:
> Alex wrote:
>> Thinking about this more, I think the problem might be related to CPU
>> access to "VRAM".  APUs don't have dedicated VRAM, they use a
>> reserved
>> carve out region at the top of system memory.  For CPU access to this
>> memory, we kmap the physical address of the carve out region of
>> system
>> memory.  You'll need to make sure that region is accessible to the
>> guest.
> So basically, the non-virt flow is is: (video?) BIOS reserves memory, marks it
> as reserved in e820, stores the physaddr somewhere, which the GPU driver gets.
> Since I suppose this includes the framebuffer, this probably has to occur around
> the moment the driver calls drm_aperture_remove_conflicting_pci_framebuffers()
> (which happens before this hw init step), right ?

Well, that partially correct. The efifb is using the PCIe resources to 
access the framebuffer and as far as I know we use that one to kick it out.

The stolen memory we get over e820/registers is separate to that.

> ... which brings me to a point that's been puzzling me for some time, which is
> that as the hw init fails, the efifb driver is still using the framebuffer.

No, it isn't. You are probably just still seeing the same screen.

The issue is most likely that while efi was kicked out nobody 
re-programmed the display hardware to show something different.

> Am I right in suspecting that efifb should get stripped of its ownership of the
> fb aperture first, and that if I don't get a black screen on hw_init failure
> that issue should be the first focus point ?

You assumption with the black screen is incorrect. Since the hardware 
works independent even if you kick out efi you still have the same 
screen content, you just can't update it anymore.

But putting efi asside what Alex pointed out pretty much breaks your 
neck trying to forward the device. You maybe could try to hack the 
driver to use the PCIe BAR for framebuffer access, but that might be 
quite a bit slower.

Regards,
Christian.

>
>> Alex
>>
>> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher <alexdeucher@gmail.com>
>> wrote:
>>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
>>> wrote:
>>>> Alex wrote:
>>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
>>>>> wrote:
>>>>>> Hi Alex,
>>>>>>
>>>>>>> We have not validated virtualization of our integrated
>>>>>>> GPUs.  I
>>>>>>> don't
>>>>>>> know that it will work at all.  We had done a bit of
>>>>>>> testing but
>>>>>>> ran
>>>>>>> into the same issues with the PSP, but never had a chance
>>>>>>> to
>>>>>>> debug
>>>>>>> further because this feature is not productized.
>>>>>> ...
>>>>>>> You need a functional PSP to get the GPU driver up and
>>>>>>> running.
>>>>>> Ah, thanks for the hint :)
>>>>>>
>>>>>> I guess that if I want to have any chance to get the PSP
>>>>>> working
>>>>>> I'm
>>>>>> going to need more details on it.  A quick search some time
>>>>>> ago
>>>>>> mostly
>>>>>> brought reverse-engineering work, rather than official AMD
>>>>>> doc.
>>>>>>   Are
>>>>>> there some AMD resources I missed ?
>>>>> The driver code is pretty much it.
>>>> Let's try to shed some more light on how things work, taking as
>>>> excuse
>>>> psp_v12_0_ring_create().
>>>>
>>>> First, register access through [RW]REG32_SOC15() is implemented
>>>> in
>>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
>>>> [RW]REG32(),
>>>> except it has to be more complex in the SR-IOV case.
>>>> Has the RLC anything to do with SR-IOV ?
>>> When running the driver on a SR-IOV virtual function (VF), some
>>> registers are not available directly via the VF's MMIO aperture so
>>> they need to go through the RLC.  For bare metal or passthrough
>>> this
>>> is not relevant.
>>>
>>>> It accesses registers in the MMIO range of the MP0 IP, and the
>>>> "MP0"
>>>> name correlates highly with MMIO accesses in PSP-handling code.
>>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
>>>> version
>>> Yes.
>>>
>>>> reported at v11.0.3 by discovery seems to contradict the use of
>>>> v12.0
>>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
>>>> something ?
>>> Typo in the ip discovery table on renoir.
>>>
>>>> More generally (and mostly out of curiosity while we're at it),
>>>> do we
>>>> have a way to match IPs listed at discovery time with the ones
>>>> used
>>>> in the driver ?
>>> In general, barring typos, the code is shared at the major version
>>> level.  The actual code may or may not need changes to handle minor
>>> revision changes in an IP.  The driver maps the IP versions from
>>> the
>>> ip discovery table to the code contained in the driver.
>>>
>>>> ---
>>>>
>>>> As for the register names, maybe we could have a short
>>>> explanation of
>>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to
>>>> be
>>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
>>>> sure
>>>> of the "SMN" part -- that could refer to the "System Management
>>>> Network",
>>>> described in [0] as an internal bus.  Are we accessing this
>>>> register
>>>> through this SMN ?
>>> These registers are just mailboxes for the PSP firmware.  All of
>>> the
>>> C2PMSG registers functionality is defined by the PSP firmware.
>>>   They
>>> are basically scratch registers used to communicate between the
>>> driver
>>> and the PSP firmware.
>>>
>>>>
>>>>>   On APUs, the PSP is shared with
>>>>> the CPU and the rest of the platform.  The GPU driver just
>>>>> interacts
>>>>> with it for a few specific tasks:
>>>>> 1. Loading Trusted Applications (e.g., trusted firmware
>>>>> applications
>>>>> that run on the PSP for specific functionality, e.g., HDCP and
>>>>> content
>>>>> protection, etc.)
>>>>> 2. Validating and loading firmware for other engines on the
>>>>> SoC.
>>>>>   This
>>>>> is required to use those engines.
>>>> Trying to understand in more details how we start the PSP up, I
>>>> noticed
>>>> that psp_v12_0 has support for loading a sOS firmware, but never
>>>> calls
>>>> init_sos_microcode() - and anyway there is no sos firmware for
>>>> renoir
>>>> and green_sardine, which seem to be the only ASICs with this PSP
>>>> version.
>>>> Is it something that's just not been completely wired up yet ?
>>> On APUs, the PSP is shared with the CPU so the PSP firmware is part
>>> of
>>> the sbios image.  The driver doesn't load it.  We only load it on
>>> dGPUs where the driver is responsible for the chip initialization.
>>>
>>>> That also rings a bell, that we have nothing about Secure OS in
>>>> the doc
>>>> yet (not even the acronym in the glossary).
>>>>
>>>>
>>>>> I'm not too familiar with the PSP's path to memory from the GPU
>>>>> perspective.  IIRC, most memory used by the PSP goes through
>>>>> carve
>>>>> out
>>>>> "vram" on APUs so it should work, but I would double check if
>>>>> there
>>>>> are any system memory allocations that used to interact with
>>>>> the PSP
>>>>> and see if changing them to vram helps.  It does work with the
>>>>> IOMMU
>>>>> enabled on bare metal, so it should work in passthrough as well
>>>>> in
>>>>> theory.
>>>> I can see a single case in the PSP code where GTT is used instead
>>>> of
>>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has
>>>> to be a reason, since the SR-IOV code path does use vram).
>>>> Changing it to vram does not make a difference, but then the
>>>> only bo that seems to be used at that point is the one for the
>>>> psp ring,
>>>> which is allocated in vram, so I'm not too much surprised.
>>>>
>>>> Maybe I should double-check bo_create calls to hunt for more ?
>>> We looked into this a bit ourselves and ran into the same issues.
>>> We'd probably need to debug this with the PSP team to make further
>>> progress, but this was not productized so neither team had the
>>> resources to delve further.
>>>
>>> Alex
>>>
>>>>
>>>> [0]
>>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Christian wrote:
> Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > Alex wrote:
> >> Thinking about this more, I think the problem might be related to
> >> CPU
> >> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> >> reserved
> >> carve out region at the top of system memory.  For CPU access to
> >> this
> >> memory, we kmap the physical address of the carve out region of
> >> system
> >> memory.  You'll need to make sure that region is accessible to the
> >> guest.
> > So basically, the non-virt flow is is: (video?) BIOS reserves
> > memory, marks it
> > as reserved in e820, stores the physaddr somewhere, which the GPU
> > driver gets.
> > Since I suppose this includes the framebuffer, this probably has to
> > occur around
> > the moment the driver calls
> > drm_aperture_remove_conflicting_pci_framebuffers()
> > (which happens before this hw init step), right ?
> 
> Well, that partially correct. The efifb is using the PCIe resources
> to
> access the framebuffer and as far as I know we use that one to kick
> it out.
> 
> The stolen memory we get over e820/registers is separate to that.
> 
> > ... which brings me to a point that's been puzzling me for some
> > time, which is
> > that as the hw init fails, the efifb driver is still using the
> > framebuffer.
> 
> No, it isn't. You are probably just still seeing the same screen.
> 
> The issue is most likely that while efi was kicked out nobody
> re-programmed the display hardware to show something different.
> 
> > Am I right in suspecting that efifb should get stripped of its
> > ownership of the
> > fb aperture first, and that if I don't get a black screen on
> > hw_init failure
> > that issue should be the first focus point ?
> 
> You assumption with the black screen is incorrect. Since the hardware
> works independent even if you kick out efi you still have the same
> screen content, you just can't update it anymore.

It's not only that the screen keeps its contents, it's that the dom0
happily continues updating it.

> But putting efi asside what Alex pointed out pretty much breaks your
> neck trying to forward the device. You maybe could try to hack the
> driver to use the PCIe BAR for framebuffer access, but that might be
> quite a bit slower.
> 
> Regards,
> Christian.
> 
> >
> >> Alex
> >>
> >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> >> <alexdeucher@gmail.com>
> >> wrote:
> >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> >>> wrote:
> >>>> Alex wrote:
> >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
> >>>>> wrote:
> >>>>>> Hi Alex,
> >>>>>>
> >>>>>>> We have not validated virtualization of our integrated
> >>>>>>> GPUs.  I
> >>>>>>> don't
> >>>>>>> know that it will work at all.  We had done a bit of
> >>>>>>> testing but
> >>>>>>> ran
> >>>>>>> into the same issues with the PSP, but never had a chance
> >>>>>>> to
> >>>>>>> debug
> >>>>>>> further because this feature is not productized.
> >>>>>> ...
> >>>>>>> You need a functional PSP to get the GPU driver up and
> >>>>>>> running.
> >>>>>> Ah, thanks for the hint :)
> >>>>>>
> >>>>>> I guess that if I want to have any chance to get the PSP
> >>>>>> working
> >>>>>> I'm
> >>>>>> going to need more details on it.  A quick search some time
> >>>>>> ago
> >>>>>> mostly
> >>>>>> brought reverse-engineering work, rather than official AMD
> >>>>>> doc.
> >>>>>>   Are
> >>>>>> there some AMD resources I missed ?
> >>>>> The driver code is pretty much it.
> >>>> Let's try to shed some more light on how things work, taking as
> >>>> excuse
> >>>> psp_v12_0_ring_create().
> >>>>
> >>>> First, register access through [RW]REG32_SOC15() is implemented
> >>>> in
> >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> >>>> [RW]REG32(),
> >>>> except it has to be more complex in the SR-IOV case.
> >>>> Has the RLC anything to do with SR-IOV ?
> >>> When running the driver on a SR-IOV virtual function (VF), some
> >>> registers are not available directly via the VF's MMIO aperture
> >>> so
> >>> they need to go through the RLC.  For bare metal or passthrough
> >>> this
> >>> is not relevant.
> >>>
> >>>> It accesses registers in the MMIO range of the MP0 IP, and the
> >>>> "MP0"
> >>>> name correlates highly with MMIO accesses in PSP-handling code.
> >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> >>>> version
> >>> Yes.
> >>>
> >>>> reported at v11.0.3 by discovery seems to contradict the use of
> >>>> v12.0
> >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> >>>> something ?
> >>> Typo in the ip discovery table on renoir.
> >>>
> >>>> More generally (and mostly out of curiosity while we're at it),
> >>>> do we
> >>>> have a way to match IPs listed at discovery time with the ones
> >>>> used
> >>>> in the driver ?
> >>> In general, barring typos, the code is shared at the major
> >>> version
> >>> level.  The actual code may or may not need changes to handle
> >>> minor
> >>> revision changes in an IP.  The driver maps the IP versions from
> >>> the
> >>> ip discovery table to the code contained in the driver.
> >>>
> >>>> ---
> >>>>
> >>>> As for the register names, maybe we could have a short
> >>>> explanation of
> >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems
> >>>> to
> >>>> be
> >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
> >>>> sure
> >>>> of the "SMN" part -- that could refer to the "System Management
> >>>> Network",
> >>>> described in [0] as an internal bus.  Are we accessing this
> >>>> register
> >>>> through this SMN ?
> >>> These registers are just mailboxes for the PSP firmware.  All of
> >>> the
> >>> C2PMSG registers functionality is defined by the PSP firmware.
> >>>   They
> >>> are basically scratch registers used to communicate between the
> >>> driver
> >>> and the PSP firmware.
> >>>
> >>>>
> >>>>>   On APUs, the PSP is shared with
> >>>>> the CPU and the rest of the platform.  The GPU driver just
> >>>>> interacts
> >>>>> with it for a few specific tasks:
> >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> >>>>> applications
> >>>>> that run on the PSP for specific functionality, e.g., HDCP and
> >>>>> content
> >>>>> protection, etc.)
> >>>>> 2. Validating and loading firmware for other engines on the
> >>>>> SoC.
> >>>>>   This
> >>>>> is required to use those engines.
> >>>> Trying to understand in more details how we start the PSP up, I
> >>>> noticed
> >>>> that psp_v12_0 has support for loading a sOS firmware, but never
> >>>> calls
> >>>> init_sos_microcode() - and anyway there is no sos firmware for
> >>>> renoir
> >>>> and green_sardine, which seem to be the only ASICs with this PSP
> >>>> version.
> >>>> Is it something that's just not been completely wired up yet ?
> >>> On APUs, the PSP is shared with the CPU so the PSP firmware is
> >>> part
> >>> of
> >>> the sbios image.  The driver doesn't load it.  We only load it on
> >>> dGPUs where the driver is responsible for the chip
> >>> initialization.
> >>>
> >>>> That also rings a bell, that we have nothing about Secure OS in
> >>>> the doc
> >>>> yet (not even the acronym in the glossary).
> >>>>
> >>>>
> >>>>> I'm not too familiar with the PSP's path to memory from the GPU
> >>>>> perspective.  IIRC, most memory used by the PSP goes through
> >>>>> carve
> >>>>> out
> >>>>> "vram" on APUs so it should work, but I would double check if
> >>>>> there
> >>>>> are any system memory allocations that used to interact with
> >>>>> the PSP
> >>>>> and see if changing them to vram helps.  It does work with the
> >>>>> IOMMU
> >>>>> enabled on bare metal, so it should work in passthrough as well
> >>>>> in
> >>>>> theory.
> >>>> I can see a single case in the PSP code where GTT is used
> >>>> instead
> >>>> of
> >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has
> >>>> to be a reason, since the SR-IOV code path does use vram).
> >>>> Changing it to vram does not make a difference, but then the
> >>>> only bo that seems to be used at that point is the one for the
> >>>> psp ring,
> >>>> which is allocated in vram, so I'm not too much surprised.
> >>>>
> >>>> Maybe I should double-check bo_create calls to hunt for more ?
> >>> We looked into this a bit ourselves and ran into the same issues.
> >>> We'd probably need to debug this with the PSP team to make
> >>> further
> >>> progress, but this was not productized so neither team had the
> >>> resources to delve further.
> >>>
> >>> Alex
> >>>
> >>>>
> >>>> [0]
> >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> 
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@free.fr> wrote:
>
> Christian wrote:
> > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > Alex wrote:
> > >> Thinking about this more, I think the problem might be related to
> > >> CPU
> > >> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> > >> reserved
> > >> carve out region at the top of system memory.  For CPU access to
> > >> this
> > >> memory, we kmap the physical address of the carve out region of
> > >> system
> > >> memory.  You'll need to make sure that region is accessible to the
> > >> guest.
> > > So basically, the non-virt flow is is: (video?) BIOS reserves
> > > memory, marks it
> > > as reserved in e820, stores the physaddr somewhere, which the GPU
> > > driver gets.
> > > Since I suppose this includes the framebuffer, this probably has to
> > > occur around
> > > the moment the driver calls
> > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > (which happens before this hw init step), right ?
> >
> > Well, that partially correct. The efifb is using the PCIe resources
> > to
> > access the framebuffer and as far as I know we use that one to kick
> > it out.
> >
> > The stolen memory we get over e820/registers is separate to that.
> >
> > > ... which brings me to a point that's been puzzling me for some
> > > time, which is
> > > that as the hw init fails, the efifb driver is still using the
> > > framebuffer.
> >
> > No, it isn't. You are probably just still seeing the same screen.
> >
> > The issue is most likely that while efi was kicked out nobody
> > re-programmed the display hardware to show something different.
> >
> > > Am I right in suspecting that efifb should get stripped of its
> > > ownership of the
> > > fb aperture first, and that if I don't get a black screen on
> > > hw_init failure
> > > that issue should be the first focus point ?
> >
> > You assumption with the black screen is incorrect. Since the hardware
> > works independent even if you kick out efi you still have the same
> > screen content, you just can't update it anymore.
>
> It's not only that the screen keeps its contents, it's that the dom0
> happily continues updating it.

If the hypevisor is using efifb, then yes that could be a problem as
the hypervisor could be writing to the efifb resources which ends up
writing to the same physical memory.  That applies to any GPU on a
UEFI system.  You'll need to make sure efifb is not in use in the
hypervisor.

Alex


>
> > But putting efi asside what Alex pointed out pretty much breaks your
> > neck trying to forward the device. You maybe could try to hack the
> > driver to use the PCIe BAR for framebuffer access, but that might be
> > quite a bit slower.
> >
> > Regards,
> > Christian.
> >
> > >
> > >> Alex
> > >>
> > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > >> <alexdeucher@gmail.com>
> > >> wrote:
> > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> > >>> wrote:
> > >>>> Alex wrote:
> > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
> > >>>>> wrote:
> > >>>>>> Hi Alex,
> > >>>>>>
> > >>>>>>> We have not validated virtualization of our integrated
> > >>>>>>> GPUs.  I
> > >>>>>>> don't
> > >>>>>>> know that it will work at all.  We had done a bit of
> > >>>>>>> testing but
> > >>>>>>> ran
> > >>>>>>> into the same issues with the PSP, but never had a chance
> > >>>>>>> to
> > >>>>>>> debug
> > >>>>>>> further because this feature is not productized.
> > >>>>>> ...
> > >>>>>>> You need a functional PSP to get the GPU driver up and
> > >>>>>>> running.
> > >>>>>> Ah, thanks for the hint :)
> > >>>>>>
> > >>>>>> I guess that if I want to have any chance to get the PSP
> > >>>>>> working
> > >>>>>> I'm
> > >>>>>> going to need more details on it.  A quick search some time
> > >>>>>> ago
> > >>>>>> mostly
> > >>>>>> brought reverse-engineering work, rather than official AMD
> > >>>>>> doc.
> > >>>>>>   Are
> > >>>>>> there some AMD resources I missed ?
> > >>>>> The driver code is pretty much it.
> > >>>> Let's try to shed some more light on how things work, taking as
> > >>>> excuse
> > >>>> psp_v12_0_ring_create().
> > >>>>
> > >>>> First, register access through [RW]REG32_SOC15() is implemented
> > >>>> in
> > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > >>>> [RW]REG32(),
> > >>>> except it has to be more complex in the SR-IOV case.
> > >>>> Has the RLC anything to do with SR-IOV ?
> > >>> When running the driver on a SR-IOV virtual function (VF), some
> > >>> registers are not available directly via the VF's MMIO aperture
> > >>> so
> > >>> they need to go through the RLC.  For bare metal or passthrough
> > >>> this
> > >>> is not relevant.
> > >>>
> > >>>> It accesses registers in the MMIO range of the MP0 IP, and the
> > >>>> "MP0"
> > >>>> name correlates highly with MMIO accesses in PSP-handling code.
> > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> > >>>> version
> > >>> Yes.
> > >>>
> > >>>> reported at v11.0.3 by discovery seems to contradict the use of
> > >>>> v12.0
> > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > >>>> something ?
> > >>> Typo in the ip discovery table on renoir.
> > >>>
> > >>>> More generally (and mostly out of curiosity while we're at it),
> > >>>> do we
> > >>>> have a way to match IPs listed at discovery time with the ones
> > >>>> used
> > >>>> in the driver ?
> > >>> In general, barring typos, the code is shared at the major
> > >>> version
> > >>> level.  The actual code may or may not need changes to handle
> > >>> minor
> > >>> revision changes in an IP.  The driver maps the IP versions from
> > >>> the
> > >>> ip discovery table to the code contained in the driver.
> > >>>
> > >>>> ---
> > >>>>
> > >>>> As for the register names, maybe we could have a short
> > >>>> explanation of
> > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems
> > >>>> to
> > >>>> be
> > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
> > >>>> sure
> > >>>> of the "SMN" part -- that could refer to the "System Management
> > >>>> Network",
> > >>>> described in [0] as an internal bus.  Are we accessing this
> > >>>> register
> > >>>> through this SMN ?
> > >>> These registers are just mailboxes for the PSP firmware.  All of
> > >>> the
> > >>> C2PMSG registers functionality is defined by the PSP firmware.
> > >>>   They
> > >>> are basically scratch registers used to communicate between the
> > >>> driver
> > >>> and the PSP firmware.
> > >>>
> > >>>>
> > >>>>>   On APUs, the PSP is shared with
> > >>>>> the CPU and the rest of the platform.  The GPU driver just
> > >>>>> interacts
> > >>>>> with it for a few specific tasks:
> > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > >>>>> applications
> > >>>>> that run on the PSP for specific functionality, e.g., HDCP and
> > >>>>> content
> > >>>>> protection, etc.)
> > >>>>> 2. Validating and loading firmware for other engines on the
> > >>>>> SoC.
> > >>>>>   This
> > >>>>> is required to use those engines.
> > >>>> Trying to understand in more details how we start the PSP up, I
> > >>>> noticed
> > >>>> that psp_v12_0 has support for loading a sOS firmware, but never
> > >>>> calls
> > >>>> init_sos_microcode() - and anyway there is no sos firmware for
> > >>>> renoir
> > >>>> and green_sardine, which seem to be the only ASICs with this PSP
> > >>>> version.
> > >>>> Is it something that's just not been completely wired up yet ?
> > >>> On APUs, the PSP is shared with the CPU so the PSP firmware is
> > >>> part
> > >>> of
> > >>> the sbios image.  The driver doesn't load it.  We only load it on
> > >>> dGPUs where the driver is responsible for the chip
> > >>> initialization.
> > >>>
> > >>>> That also rings a bell, that we have nothing about Secure OS in
> > >>>> the doc
> > >>>> yet (not even the acronym in the glossary).
> > >>>>
> > >>>>
> > >>>>> I'm not too familiar with the PSP's path to memory from the GPU
> > >>>>> perspective.  IIRC, most memory used by the PSP goes through
> > >>>>> carve
> > >>>>> out
> > >>>>> "vram" on APUs so it should work, but I would double check if
> > >>>>> there
> > >>>>> are any system memory allocations that used to interact with
> > >>>>> the PSP
> > >>>>> and see if changing them to vram helps.  It does work with the
> > >>>>> IOMMU
> > >>>>> enabled on bare metal, so it should work in passthrough as well
> > >>>>> in
> > >>>>> theory.
> > >>>> I can see a single case in the PSP code where GTT is used
> > >>>> instead
> > >>>> of
> > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has
> > >>>> to be a reason, since the SR-IOV code path does use vram).
> > >>>> Changing it to vram does not make a difference, but then the
> > >>>> only bo that seems to be used at that point is the one for the
> > >>>> psp ring,
> > >>>> which is allocated in vram, so I'm not too much surprised.
> > >>>>
> > >>>> Maybe I should double-check bo_create calls to hunt for more ?
> > >>> We looked into this a bit ourselves and ran into the same issues.
> > >>> We'd probably need to debug this with the PSP team to make
> > >>> further
> > >>> progress, but this was not productized so neither team had the
> > >>> resources to delve further.
> > >>>
> > >>> Alex
> > >>>
> > >>>>
> > >>>> [0]
> > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> 
> On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Christian wrote:
> > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > Alex wrote:
> > > >> Thinking about this more, I think the problem might be related
> > > >> to
> > > >> CPU
> > > >> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> > > >> reserved
> > > >> carve out region at the top of system memory.  For CPU access
> > > >> to
> > > >> this
> > > >> memory, we kmap the physical address of the carve out region
> > > >> of
> > > >> system
> > > >> memory.  You'll need to make sure that region is accessible to
> > > >> the
> > > >> guest.
> > > > So basically, the non-virt flow is is: (video?) BIOS reserves
> > > > memory, marks it
> > > > as reserved in e820, stores the physaddr somewhere, which the
> > > > GPU
> > > > driver gets.
> > > > Since I suppose this includes the framebuffer, this probably
> > > > has to
> > > > occur around
> > > > the moment the driver calls
> > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > (which happens before this hw init step), right ?
> > >
> > > Well, that partially correct. The efifb is using the PCIe
> > > resources
> > > to
> > > access the framebuffer and as far as I know we use that one to
> > > kick
> > > it out.
> > >
> > > The stolen memory we get over e820/registers is separate to that.

How is the stolen memory communicated to the driver ?  That host physical
memory probably has to be mapped at the same guest physical address for
the magic to work, right ?

> > >
> > > > ... which brings me to a point that's been puzzling me for some
> > > > time, which is
> > > > that as the hw init fails, the efifb driver is still using the
> > > > framebuffer.
> > >
> > > No, it isn't. You are probably just still seeing the same screen.
> > >
> > > The issue is most likely that while efi was kicked out nobody
> > > re-programmed the display hardware to show something different.
> > >
> > > > Am I right in suspecting that efifb should get stripped of its
> > > > ownership of the
> > > > fb aperture first, and that if I don't get a black screen on
> > > > hw_init failure
> > > > that issue should be the first focus point ?
> > >
> > > You assumption with the black screen is incorrect. Since the
> > > hardware
> > > works independent even if you kick out efi you still have the
> > > same
> > > screen content, you just can't update it anymore.
> >
> > It's not only that the screen keeps its contents, it's that the
> > dom0
> > happily continues updating it.
> 
> If the hypevisor is using efifb, then yes that could be a problem as
> the hypervisor could be writing to the efifb resources which ends up
> writing to the same physical memory.  That applies to any GPU on a
> UEFI system.  You'll need to make sure efifb is not in use in the
> hypervisor.

That remark evokes several things to me.  First one is that every time
I've tried booting with efifb disabled in dom0, there was no visible
improvements in the guest driver - i.i. I really have to dig how vram mapping
is performed and check things are as expected anyway.

The other is that, when dom0 cannot use efifb, entering a luks key is
suddenly less user-friendly.  But in theory I'd think we could overcome
this by letting dom0 use efifb until ready to start the guest, a simple
driver unbind at the right moment should be expected to work, right ?
Going further and allowing the guest to use efifb on its own could
possibly be more tricky (starting with a different state?) but does
not seem to sound completely outlandish either - or does it ?

> 
> Alex
> 
> 
> >
> > > But putting efi asside what Alex pointed out pretty much breaks
> > > your
> > > neck trying to forward the device. You maybe could try to hack
> > > the
> > > driver to use the PCIe BAR for framebuffer access, but that might
> > > be
> > > quite a bit slower.
> > >
> > > Regards,
> > > Christian.
> > >
> > > >
> > > >> Alex
> > > >>
> > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > >> <alexdeucher@gmail.com>
> > > >> wrote:
> > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> > > >>> wrote:
> > > >>>> Alex wrote:
> > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > >>>>> <ydirson@free.fr>
> > > >>>>> wrote:
> > > >>>>>> Hi Alex,
> > > >>>>>>
> > > >>>>>>> We have not validated virtualization of our integrated
> > > >>>>>>> GPUs.  I
> > > >>>>>>> don't
> > > >>>>>>> know that it will work at all.  We had done a bit of
> > > >>>>>>> testing but
> > > >>>>>>> ran
> > > >>>>>>> into the same issues with the PSP, but never had a chance
> > > >>>>>>> to
> > > >>>>>>> debug
> > > >>>>>>> further because this feature is not productized.
> > > >>>>>> ...
> > > >>>>>>> You need a functional PSP to get the GPU driver up and
> > > >>>>>>> running.
> > > >>>>>> Ah, thanks for the hint :)
> > > >>>>>>
> > > >>>>>> I guess that if I want to have any chance to get the PSP
> > > >>>>>> working
> > > >>>>>> I'm
> > > >>>>>> going to need more details on it.  A quick search some
> > > >>>>>> time
> > > >>>>>> ago
> > > >>>>>> mostly
> > > >>>>>> brought reverse-engineering work, rather than official AMD
> > > >>>>>> doc.
> > > >>>>>>   Are
> > > >>>>>> there some AMD resources I missed ?
> > > >>>>> The driver code is pretty much it.
> > > >>>> Let's try to shed some more light on how things work, taking
> > > >>>> as
> > > >>>> excuse
> > > >>>> psp_v12_0_ring_create().
> > > >>>>
> > > >>>> First, register access through [RW]REG32_SOC15() is
> > > >>>> implemented
> > > >>>> in
> > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > >>>> [RW]REG32(),
> > > >>>> except it has to be more complex in the SR-IOV case.
> > > >>>> Has the RLC anything to do with SR-IOV ?
> > > >>> When running the driver on a SR-IOV virtual function (VF),
> > > >>> some
> > > >>> registers are not available directly via the VF's MMIO
> > > >>> aperture
> > > >>> so
> > > >>> they need to go through the RLC.  For bare metal or
> > > >>> passthrough
> > > >>> this
> > > >>> is not relevant.
> > > >>>
> > > >>>> It accesses registers in the MMIO range of the MP0 IP, and
> > > >>>> the
> > > >>>> "MP0"
> > > >>>> name correlates highly with MMIO accesses in PSP-handling
> > > >>>> code.
> > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> > > >>>> version
> > > >>> Yes.
> > > >>>
> > > >>>> reported at v11.0.3 by discovery seems to contradict the use
> > > >>>> of
> > > >>>> v12.0
> > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > >>>> something ?
> > > >>> Typo in the ip discovery table on renoir.
> > > >>>
> > > >>>> More generally (and mostly out of curiosity while we're at
> > > >>>> it),
> > > >>>> do we
> > > >>>> have a way to match IPs listed at discovery time with the
> > > >>>> ones
> > > >>>> used
> > > >>>> in the driver ?
> > > >>> In general, barring typos, the code is shared at the major
> > > >>> version
> > > >>> level.  The actual code may or may not need changes to handle
> > > >>> minor
> > > >>> revision changes in an IP.  The driver maps the IP versions
> > > >>> from
> > > >>> the
> > > >>> ip discovery table to the code contained in the driver.
> > > >>>
> > > >>>> ---
> > > >>>>
> > > >>>> As for the register names, maybe we could have a short
> > > >>>> explanation of
> > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > >>>> seems
> > > >>>> to
> > > >>>> be
> > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm
> > > >>>> not
> > > >>>> sure
> > > >>>> of the "SMN" part -- that could refer to the "System
> > > >>>> Management
> > > >>>> Network",
> > > >>>> described in [0] as an internal bus.  Are we accessing this
> > > >>>> register
> > > >>>> through this SMN ?
> > > >>> These registers are just mailboxes for the PSP firmware.  All
> > > >>> of
> > > >>> the
> > > >>> C2PMSG registers functionality is defined by the PSP
> > > >>> firmware.
> > > >>>   They
> > > >>> are basically scratch registers used to communicate between
> > > >>> the
> > > >>> driver
> > > >>> and the PSP firmware.
> > > >>>
> > > >>>>
> > > >>>>>   On APUs, the PSP is shared with
> > > >>>>> the CPU and the rest of the platform.  The GPU driver just
> > > >>>>> interacts
> > > >>>>> with it for a few specific tasks:
> > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > >>>>> applications
> > > >>>>> that run on the PSP for specific functionality, e.g., HDCP
> > > >>>>> and
> > > >>>>> content
> > > >>>>> protection, etc.)
> > > >>>>> 2. Validating and loading firmware for other engines on the
> > > >>>>> SoC.
> > > >>>>>   This
> > > >>>>> is required to use those engines.
> > > >>>> Trying to understand in more details how we start the PSP
> > > >>>> up, I
> > > >>>> noticed
> > > >>>> that psp_v12_0 has support for loading a sOS firmware, but
> > > >>>> never
> > > >>>> calls
> > > >>>> init_sos_microcode() - and anyway there is no sos firmware
> > > >>>> for
> > > >>>> renoir
> > > >>>> and green_sardine, which seem to be the only ASICs with this
> > > >>>> PSP
> > > >>>> version.
> > > >>>> Is it something that's just not been completely wired up yet
> > > >>>> ?
> > > >>> On APUs, the PSP is shared with the CPU so the PSP firmware
> > > >>> is
> > > >>> part
> > > >>> of
> > > >>> the sbios image.  The driver doesn't load it.  We only load
> > > >>> it on
> > > >>> dGPUs where the driver is responsible for the chip
> > > >>> initialization.
> > > >>>
> > > >>>> That also rings a bell, that we have nothing about Secure OS
> > > >>>> in
> > > >>>> the doc
> > > >>>> yet (not even the acronym in the glossary).
> > > >>>>
> > > >>>>
> > > >>>>> I'm not too familiar with the PSP's path to memory from the
> > > >>>>> GPU
> > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > >>>>> through
> > > >>>>> carve
> > > >>>>> out
> > > >>>>> "vram" on APUs so it should work, but I would double check
> > > >>>>> if
> > > >>>>> there
> > > >>>>> are any system memory allocations that used to interact
> > > >>>>> with
> > > >>>>> the PSP
> > > >>>>> and see if changing them to vram helps.  It does work with
> > > >>>>> the
> > > >>>>> IOMMU
> > > >>>>> enabled on bare metal, so it should work in passthrough as
> > > >>>>> well
> > > >>>>> in
> > > >>>>> theory.
> > > >>>> I can see a single case in the PSP code where GTT is used
> > > >>>> instead
> > > >>>> of
> > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there
> > > >>>> has
> > > >>>> to be a reason, since the SR-IOV code path does use vram).
> > > >>>> Changing it to vram does not make a difference, but then the
> > > >>>> only bo that seems to be used at that point is the one for
> > > >>>> the
> > > >>>> psp ring,
> > > >>>> which is allocated in vram, so I'm not too much surprised.
> > > >>>>
> > > >>>> Maybe I should double-check bo_create calls to hunt for more
> > > >>>> ?
> > > >>> We looked into this a bit ourselves and ran into the same
> > > >>> issues.
> > > >>> We'd probably need to debug this with the PSP team to make
> > > >>> further
> > > >>> progress, but this was not productized so neither team had
> > > >>> the
> > > >>> resources to delve further.
> > > >>>
> > > >>> Alex
> > > >>>
> > > >>>>
> > > >>>> [0]
> > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > >
> > >
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr> wrote:
>
>
> Alex wrote:
> >
> > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > > Christian wrote:
> > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > Alex wrote:
> > > > >> Thinking about this more, I think the problem might be related
> > > > >> to
> > > > >> CPU
> > > > >> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> > > > >> reserved
> > > > >> carve out region at the top of system memory.  For CPU access
> > > > >> to
> > > > >> this
> > > > >> memory, we kmap the physical address of the carve out region
> > > > >> of
> > > > >> system
> > > > >> memory.  You'll need to make sure that region is accessible to
> > > > >> the
> > > > >> guest.
> > > > > So basically, the non-virt flow is is: (video?) BIOS reserves
> > > > > memory, marks it
> > > > > as reserved in e820, stores the physaddr somewhere, which the
> > > > > GPU
> > > > > driver gets.
> > > > > Since I suppose this includes the framebuffer, this probably
> > > > > has to
> > > > > occur around
> > > > > the moment the driver calls
> > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > (which happens before this hw init step), right ?
> > > >
> > > > Well, that partially correct. The efifb is using the PCIe
> > > > resources
> > > > to
> > > > access the framebuffer and as far as I know we use that one to
> > > > kick
> > > > it out.
> > > >
> > > > The stolen memory we get over e820/registers is separate to that.
>
> How is the stolen memory communicated to the driver ?  That host physical
> memory probably has to be mapped at the same guest physical address for
> the magic to work, right ?

Correct.  The driver reads the physical location of that memory from
hardware registers.  Removing this chunk of code from gmc_v9_0.c will
force the driver to use the BAR, but I'm not sure if there are any
other places in the driver that make assumptions about using the
physical host address or not on APUs off hand.

        if ((adev->flags & AMD_IS_APU) ||
            (adev->gmc.xgmi.supported &&
             adev->gmc.xgmi.connected_to_cpu)) {
                adev->gmc.aper_base =
                        adev->gfxhub.funcs->get_mc_fb_offset(adev) +
                        adev->gmc.xgmi.physical_node_id *
                        adev->gmc.xgmi.node_segment_size;
                adev->gmc.aper_size = adev->gmc.real_vram_size;
        }



>
> > > >
> > > > > ... which brings me to a point that's been puzzling me for some
> > > > > time, which is
> > > > > that as the hw init fails, the efifb driver is still using the
> > > > > framebuffer.
> > > >
> > > > No, it isn't. You are probably just still seeing the same screen.
> > > >
> > > > The issue is most likely that while efi was kicked out nobody
> > > > re-programmed the display hardware to show something different.
> > > >
> > > > > Am I right in suspecting that efifb should get stripped of its
> > > > > ownership of the
> > > > > fb aperture first, and that if I don't get a black screen on
> > > > > hw_init failure
> > > > > that issue should be the first focus point ?
> > > >
> > > > You assumption with the black screen is incorrect. Since the
> > > > hardware
> > > > works independent even if you kick out efi you still have the
> > > > same
> > > > screen content, you just can't update it anymore.
> > >
> > > It's not only that the screen keeps its contents, it's that the
> > > dom0
> > > happily continues updating it.
> >
> > If the hypevisor is using efifb, then yes that could be a problem as
> > the hypervisor could be writing to the efifb resources which ends up
> > writing to the same physical memory.  That applies to any GPU on a
> > UEFI system.  You'll need to make sure efifb is not in use in the
> > hypervisor.
>
> That remark evokes several things to me.  First one is that every time
> I've tried booting with efifb disabled in dom0, there was no visible
> improvements in the guest driver - i.i. I really have to dig how vram mapping
> is performed and check things are as expected anyway.

Ultimately you end up at the same physical memory.  efifb uses the PCI
BAR which points to the same physical memory that the driver directly
maps.

>
> The other is that, when dom0 cannot use efifb, entering a luks key is
> suddenly less user-friendly.  But in theory I'd think we could overcome
> this by letting dom0 use efifb until ready to start the guest, a simple
> driver unbind at the right moment should be expected to work, right ?
> Going further and allowing the guest to use efifb on its own could
> possibly be more tricky (starting with a different state?) but does
> not seem to sound completely outlandish either - or does it ?
>

efifb just takes whatever hardware state the GOP driver in the pre-OS
environment left the GPU in.  Once you have a driver loaded in the OS,
that state is gone so I I don't see much value in using efifb once you
have a real driver in the mix.  If you want a console on the host,
it's probably better to use 2 GPU or just load the real driver as
needed in both the host and guest.

> >
> > Alex
> >
> >
> > >
> > > > But putting efi asside what Alex pointed out pretty much breaks
> > > > your
> > > > neck trying to forward the device. You maybe could try to hack
> > > > the
> > > > driver to use the PCIe BAR for framebuffer access, but that might
> > > > be
> > > > quite a bit slower.
> > > >
> > > > Regards,
> > > > Christian.
> > > >
> > > > >
> > > > >> Alex
> > > > >>
> > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > >> <alexdeucher@gmail.com>
> > > > >> wrote:
> > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> > > > >>> wrote:
> > > > >>>> Alex wrote:
> > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > >>>>> <ydirson@free.fr>
> > > > >>>>> wrote:
> > > > >>>>>> Hi Alex,
> > > > >>>>>>
> > > > >>>>>>> We have not validated virtualization of our integrated
> > > > >>>>>>> GPUs.  I
> > > > >>>>>>> don't
> > > > >>>>>>> know that it will work at all.  We had done a bit of
> > > > >>>>>>> testing but
> > > > >>>>>>> ran
> > > > >>>>>>> into the same issues with the PSP, but never had a chance
> > > > >>>>>>> to
> > > > >>>>>>> debug
> > > > >>>>>>> further because this feature is not productized.
> > > > >>>>>> ...
> > > > >>>>>>> You need a functional PSP to get the GPU driver up and
> > > > >>>>>>> running.
> > > > >>>>>> Ah, thanks for the hint :)
> > > > >>>>>>
> > > > >>>>>> I guess that if I want to have any chance to get the PSP
> > > > >>>>>> working
> > > > >>>>>> I'm
> > > > >>>>>> going to need more details on it.  A quick search some
> > > > >>>>>> time
> > > > >>>>>> ago
> > > > >>>>>> mostly
> > > > >>>>>> brought reverse-engineering work, rather than official AMD
> > > > >>>>>> doc.
> > > > >>>>>>   Are
> > > > >>>>>> there some AMD resources I missed ?
> > > > >>>>> The driver code is pretty much it.
> > > > >>>> Let's try to shed some more light on how things work, taking
> > > > >>>> as
> > > > >>>> excuse
> > > > >>>> psp_v12_0_ring_create().
> > > > >>>>
> > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > >>>> implemented
> > > > >>>> in
> > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > > >>>> [RW]REG32(),
> > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > >>> When running the driver on a SR-IOV virtual function (VF),
> > > > >>> some
> > > > >>> registers are not available directly via the VF's MMIO
> > > > >>> aperture
> > > > >>> so
> > > > >>> they need to go through the RLC.  For bare metal or
> > > > >>> passthrough
> > > > >>> this
> > > > >>> is not relevant.
> > > > >>>
> > > > >>>> It accesses registers in the MMIO range of the MP0 IP, and
> > > > >>>> the
> > > > >>>> "MP0"
> > > > >>>> name correlates highly with MMIO accesses in PSP-handling
> > > > >>>> code.
> > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> > > > >>>> version
> > > > >>> Yes.
> > > > >>>
> > > > >>>> reported at v11.0.3 by discovery seems to contradict the use
> > > > >>>> of
> > > > >>>> v12.0
> > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > > >>>> something ?
> > > > >>> Typo in the ip discovery table on renoir.
> > > > >>>
> > > > >>>> More generally (and mostly out of curiosity while we're at
> > > > >>>> it),
> > > > >>>> do we
> > > > >>>> have a way to match IPs listed at discovery time with the
> > > > >>>> ones
> > > > >>>> used
> > > > >>>> in the driver ?
> > > > >>> In general, barring typos, the code is shared at the major
> > > > >>> version
> > > > >>> level.  The actual code may or may not need changes to handle
> > > > >>> minor
> > > > >>> revision changes in an IP.  The driver maps the IP versions
> > > > >>> from
> > > > >>> the
> > > > >>> ip discovery table to the code contained in the driver.
> > > > >>>
> > > > >>>> ---
> > > > >>>>
> > > > >>>> As for the register names, maybe we could have a short
> > > > >>>> explanation of
> > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > > >>>> seems
> > > > >>>> to
> > > > >>>> be
> > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm
> > > > >>>> not
> > > > >>>> sure
> > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > >>>> Management
> > > > >>>> Network",
> > > > >>>> described in [0] as an internal bus.  Are we accessing this
> > > > >>>> register
> > > > >>>> through this SMN ?
> > > > >>> These registers are just mailboxes for the PSP firmware.  All
> > > > >>> of
> > > > >>> the
> > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > >>> firmware.
> > > > >>>   They
> > > > >>> are basically scratch registers used to communicate between
> > > > >>> the
> > > > >>> driver
> > > > >>> and the PSP firmware.
> > > > >>>
> > > > >>>>
> > > > >>>>>   On APUs, the PSP is shared with
> > > > >>>>> the CPU and the rest of the platform.  The GPU driver just
> > > > >>>>> interacts
> > > > >>>>> with it for a few specific tasks:
> > > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > > >>>>> applications
> > > > >>>>> that run on the PSP for specific functionality, e.g., HDCP
> > > > >>>>> and
> > > > >>>>> content
> > > > >>>>> protection, etc.)
> > > > >>>>> 2. Validating and loading firmware for other engines on the
> > > > >>>>> SoC.
> > > > >>>>>   This
> > > > >>>>> is required to use those engines.
> > > > >>>> Trying to understand in more details how we start the PSP
> > > > >>>> up, I
> > > > >>>> noticed
> > > > >>>> that psp_v12_0 has support for loading a sOS firmware, but
> > > > >>>> never
> > > > >>>> calls
> > > > >>>> init_sos_microcode() - and anyway there is no sos firmware
> > > > >>>> for
> > > > >>>> renoir
> > > > >>>> and green_sardine, which seem to be the only ASICs with this
> > > > >>>> PSP
> > > > >>>> version.
> > > > >>>> Is it something that's just not been completely wired up yet
> > > > >>>> ?
> > > > >>> On APUs, the PSP is shared with the CPU so the PSP firmware
> > > > >>> is
> > > > >>> part
> > > > >>> of
> > > > >>> the sbios image.  The driver doesn't load it.  We only load
> > > > >>> it on
> > > > >>> dGPUs where the driver is responsible for the chip
> > > > >>> initialization.
> > > > >>>
> > > > >>>> That also rings a bell, that we have nothing about Secure OS
> > > > >>>> in
> > > > >>>> the doc
> > > > >>>> yet (not even the acronym in the glossary).
> > > > >>>>
> > > > >>>>
> > > > >>>>> I'm not too familiar with the PSP's path to memory from the
> > > > >>>>> GPU
> > > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > > >>>>> through
> > > > >>>>> carve
> > > > >>>>> out
> > > > >>>>> "vram" on APUs so it should work, but I would double check
> > > > >>>>> if
> > > > >>>>> there
> > > > >>>>> are any system memory allocations that used to interact
> > > > >>>>> with
> > > > >>>>> the PSP
> > > > >>>>> and see if changing them to vram helps.  It does work with
> > > > >>>>> the
> > > > >>>>> IOMMU
> > > > >>>>> enabled on bare metal, so it should work in passthrough as
> > > > >>>>> well
> > > > >>>>> in
> > > > >>>>> theory.
> > > > >>>> I can see a single case in the PSP code where GTT is used
> > > > >>>> instead
> > > > >>>> of
> > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there
> > > > >>>> has
> > > > >>>> to be a reason, since the SR-IOV code path does use vram).
> > > > >>>> Changing it to vram does not make a difference, but then the
> > > > >>>> only bo that seems to be used at that point is the one for
> > > > >>>> the
> > > > >>>> psp ring,
> > > > >>>> which is allocated in vram, so I'm not too much surprised.
> > > > >>>>
> > > > >>>> Maybe I should double-check bo_create calls to hunt for more
> > > > >>>> ?
> > > > >>> We looked into this a bit ourselves and ran into the same
> > > > >>> issues.
> > > > >>> We'd probably need to debug this with the PSP team to make
> > > > >>> further
> > > > >>> progress, but this was not productized so neither team had
> > > > >>> the
> > > > >>> resources to delve further.
> > > > >>>
> > > > >>> Alex
> > > > >>>
> > > > >>>>
> > > > >>>> [0]
> > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > >
> > > >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

----- Mail original -----
> De: "Alex Deucher" <alexdeucher@gmail.com>
> À: "Yann Dirson" <ydirson@free.fr>
> Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>, "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> Envoyé: Mardi 21 Décembre 2021 23:31:01
> Objet: Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm
> 
> On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr> wrote:
> >
> >
> > Alex wrote:
> > >
> > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@free.fr>
> > > wrote:
> > > >
> > > > Christian wrote:
> > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > Alex wrote:
> > > > > >> Thinking about this more, I think the problem might be
> > > > > >> related
> > > > > >> to
> > > > > >> CPU
> > > > > >> access to "VRAM".  APUs don't have dedicated VRAM, they
> > > > > >> use a
> > > > > >> reserved
> > > > > >> carve out region at the top of system memory.  For CPU
> > > > > >> access
> > > > > >> to
> > > > > >> this
> > > > > >> memory, we kmap the physical address of the carve out
> > > > > >> region
> > > > > >> of
> > > > > >> system
> > > > > >> memory.  You'll need to make sure that region is
> > > > > >> accessible to
> > > > > >> the
> > > > > >> guest.
> > > > > > So basically, the non-virt flow is is: (video?) BIOS
> > > > > > reserves
> > > > > > memory, marks it
> > > > > > as reserved in e820, stores the physaddr somewhere, which
> > > > > > the
> > > > > > GPU
> > > > > > driver gets.
> > > > > > Since I suppose this includes the framebuffer, this
> > > > > > probably
> > > > > > has to
> > > > > > occur around
> > > > > > the moment the driver calls
> > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > (which happens before this hw init step), right ?
> > > > >
> > > > > Well, that partially correct. The efifb is using the PCIe
> > > > > resources
> > > > > to
> > > > > access the framebuffer and as far as I know we use that one
> > > > > to
> > > > > kick
> > > > > it out.
> > > > >
> > > > > The stolen memory we get over e820/registers is separate to
> > > > > that.
> >
> > How is the stolen memory communicated to the driver ?  That host
> > physical
> > memory probably has to be mapped at the same guest physical address
> > for
> > the magic to work, right ?
> 
> Correct.  The driver reads the physical location of that memory from
> hardware registers.  Removing this chunk of code from gmc_v9_0.c will
> force the driver to use the BAR, but I'm not sure if there are any
> other places in the driver that make assumptions about using the
> physical host address or not on APUs off hand.
> 
>         if ((adev->flags & AMD_IS_APU) ||
>             (adev->gmc.xgmi.supported &&
>              adev->gmc.xgmi.connected_to_cpu)) {
>                 adev->gmc.aper_base =
>                         adev->gfxhub.funcs->get_mc_fb_offset(adev) +
>                         adev->gmc.xgmi.physical_node_id *
>                         adev->gmc.xgmi.node_segment_size;
>                 adev->gmc.aper_size = adev->gmc.real_vram_size;
>         }
> 
> 
> 
> >
> > > > >
> > > > > > ... which brings me to a point that's been puzzling me for
> > > > > > some
> > > > > > time, which is
> > > > > > that as the hw init fails, the efifb driver is still using
> > > > > > the
> > > > > > framebuffer.
> > > > >
> > > > > No, it isn't. You are probably just still seeing the same
> > > > > screen.
> > > > >
> > > > > The issue is most likely that while efi was kicked out nobody
> > > > > re-programmed the display hardware to show something
> > > > > different.
> > > > >
> > > > > > Am I right in suspecting that efifb should get stripped of
> > > > > > its
> > > > > > ownership of the
> > > > > > fb aperture first, and that if I don't get a black screen
> > > > > > on
> > > > > > hw_init failure
> > > > > > that issue should be the first focus point ?
> > > > >
> > > > > You assumption with the black screen is incorrect. Since the
> > > > > hardware
> > > > > works independent even if you kick out efi you still have the
> > > > > same
> > > > > screen content, you just can't update it anymore.
> > > >
> > > > It's not only that the screen keeps its contents, it's that the
> > > > dom0
> > > > happily continues updating it.
> > >
> > > If the hypevisor is using efifb, then yes that could be a problem
> > > as
> > > the hypervisor could be writing to the efifb resources which ends
> > > up
> > > writing to the same physical memory.  That applies to any GPU on
> > > a
> > > UEFI system.  You'll need to make sure efifb is not in use in the
> > > hypervisor.
> >
> > That remark evokes several things to me.  First one is that every
> > time
> > I've tried booting with efifb disabled in dom0, there was no
> > visible
> > improvements in the guest driver - i.i. I really have to dig how
> > vram mapping
> > is performed and check things are as expected anyway.
> 
> Ultimately you end up at the same physical memory.  efifb uses the
> PCI
> BAR which points to the same physical memory that the driver directly
> maps.
> 
> >
> > The other is that, when dom0 cannot use efifb, entering a luks key
> > is
> > suddenly less user-friendly.  But in theory I'd think we could
> > overcome
> > this by letting dom0 use efifb until ready to start the guest, a
> > simple
> > driver unbind at the right moment should be expected to work, right
> > ?
> > Going further and allowing the guest to use efifb on its own could
> > possibly be more tricky (starting with a different state?) but does
> > not seem to sound completely outlandish either - or does it ?
> >
> 
> efifb just takes whatever hardware state the GOP driver in the pre-OS
> environment left the GPU in.  Once you have a driver loaded in the
> OS,
> that state is gone so I I don't see much value in using efifb once
> you
> have a real driver in the mix.  If you want a console on the host,
> it's probably better to use 2 GPU or just load the real driver as
> needed in both the host and guest.

My point is essentially making sure we always have a usable display.
Dom0 will essentially use it for entering luks key (but having boot logs
visible may also be useful for some troubleshooting), and I was figuring
out that once dom0 has relinquished it to start a guest with passthrough,
the guest might as well make use of it before it can start amdgpu.  
Could be useful as a first step of seeing the GPU in the guest, in
case there are more issues to be solved to be able to start amdgpu.

> 
> > >
> > > Alex
> > >
> > >
> > > >
> > > > > But putting efi asside what Alex pointed out pretty much
> > > > > breaks
> > > > > your
> > > > > neck trying to forward the device. You maybe could try to
> > > > > hack
> > > > > the
> > > > > driver to use the PCIe BAR for framebuffer access, but that
> > > > > might
> > > > > be
> > > > > quite a bit slower.
> > > > >
> > > > > Regards,
> > > > > Christian.
> > > > >
> > > > > >
> > > > > >> Alex
> > > > > >>
> > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > >> <alexdeucher@gmail.com>
> > > > > >> wrote:
> > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > >>> <ydirson@free.fr>
> > > > > >>> wrote:
> > > > > >>>> Alex wrote:
> > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > >>>>> <ydirson@free.fr>
> > > > > >>>>> wrote:
> > > > > >>>>>> Hi Alex,
> > > > > >>>>>>
> > > > > >>>>>>> We have not validated virtualization of our
> > > > > >>>>>>> integrated
> > > > > >>>>>>> GPUs.  I
> > > > > >>>>>>> don't
> > > > > >>>>>>> know that it will work at all.  We had done a bit of
> > > > > >>>>>>> testing but
> > > > > >>>>>>> ran
> > > > > >>>>>>> into the same issues with the PSP, but never had a
> > > > > >>>>>>> chance
> > > > > >>>>>>> to
> > > > > >>>>>>> debug
> > > > > >>>>>>> further because this feature is not productized.
> > > > > >>>>>> ...
> > > > > >>>>>>> You need a functional PSP to get the GPU driver up
> > > > > >>>>>>> and
> > > > > >>>>>>> running.
> > > > > >>>>>> Ah, thanks for the hint :)
> > > > > >>>>>>
> > > > > >>>>>> I guess that if I want to have any chance to get the
> > > > > >>>>>> PSP
> > > > > >>>>>> working
> > > > > >>>>>> I'm
> > > > > >>>>>> going to need more details on it.  A quick search some
> > > > > >>>>>> time
> > > > > >>>>>> ago
> > > > > >>>>>> mostly
> > > > > >>>>>> brought reverse-engineering work, rather than official
> > > > > >>>>>> AMD
> > > > > >>>>>> doc.
> > > > > >>>>>>   Are
> > > > > >>>>>> there some AMD resources I missed ?
> > > > > >>>>> The driver code is pretty much it.
> > > > > >>>> Let's try to shed some more light on how things work,
> > > > > >>>> taking
> > > > > >>>> as
> > > > > >>>> excuse
> > > > > >>>> psp_v12_0_ring_create().
> > > > > >>>>
> > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > >>>> implemented
> > > > > >>>> in
> > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > > > >>>> [RW]REG32(),
> > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > >>> (VF),
> > > > > >>> some
> > > > > >>> registers are not available directly via the VF's MMIO
> > > > > >>> aperture
> > > > > >>> so
> > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > >>> passthrough
> > > > > >>> this
> > > > > >>> is not relevant.
> > > > > >>>
> > > > > >>>> It accesses registers in the MMIO range of the MP0 IP,
> > > > > >>>> and
> > > > > >>>> the
> > > > > >>>> "MP0"
> > > > > >>>> name correlates highly with MMIO accesses in
> > > > > >>>> PSP-handling
> > > > > >>>> code.
> > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The
> > > > > >>>> MP0
> > > > > >>>> version
> > > > > >>> Yes.
> > > > > >>>
> > > > > >>>> reported at v11.0.3 by discovery seems to contradict the
> > > > > >>>> use
> > > > > >>>> of
> > > > > >>>> v12.0
> > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > > > >>>> something ?
> > > > > >>> Typo in the ip discovery table on renoir.
> > > > > >>>
> > > > > >>>> More generally (and mostly out of curiosity while we're
> > > > > >>>> at
> > > > > >>>> it),
> > > > > >>>> do we
> > > > > >>>> have a way to match IPs listed at discovery time with
> > > > > >>>> the
> > > > > >>>> ones
> > > > > >>>> used
> > > > > >>>> in the driver ?
> > > > > >>> In general, barring typos, the code is shared at the
> > > > > >>> major
> > > > > >>> version
> > > > > >>> level.  The actual code may or may not need changes to
> > > > > >>> handle
> > > > > >>> minor
> > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > >>> versions
> > > > > >>> from
> > > > > >>> the
> > > > > >>> ip discovery table to the code contained in the driver.
> > > > > >>>
> > > > > >>>> ---
> > > > > >>>>
> > > > > >>>> As for the register names, maybe we could have a short
> > > > > >>>> explanation of
> > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > > > >>>> seems
> > > > > >>>> to
> > > > > >>>> be
> > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but
> > > > > >>>> I'm
> > > > > >>>> not
> > > > > >>>> sure
> > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > >>>> Management
> > > > > >>>> Network",
> > > > > >>>> described in [0] as an internal bus.  Are we accessing
> > > > > >>>> this
> > > > > >>>> register
> > > > > >>>> through this SMN ?
> > > > > >>> These registers are just mailboxes for the PSP firmware.
> > > > > >>>  All
> > > > > >>> of
> > > > > >>> the
> > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > >>> firmware.
> > > > > >>>   They
> > > > > >>> are basically scratch registers used to communicate
> > > > > >>> between
> > > > > >>> the
> > > > > >>> driver
> > > > > >>> and the PSP firmware.
> > > > > >>>
> > > > > >>>>
> > > > > >>>>>   On APUs, the PSP is shared with
> > > > > >>>>> the CPU and the rest of the platform.  The GPU driver
> > > > > >>>>> just
> > > > > >>>>> interacts
> > > > > >>>>> with it for a few specific tasks:
> > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > > > >>>>> applications
> > > > > >>>>> that run on the PSP for specific functionality, e.g.,
> > > > > >>>>> HDCP
> > > > > >>>>> and
> > > > > >>>>> content
> > > > > >>>>> protection, etc.)
> > > > > >>>>> 2. Validating and loading firmware for other engines on
> > > > > >>>>> the
> > > > > >>>>> SoC.
> > > > > >>>>>   This
> > > > > >>>>> is required to use those engines.
> > > > > >>>> Trying to understand in more details how we start the
> > > > > >>>> PSP
> > > > > >>>> up, I
> > > > > >>>> noticed
> > > > > >>>> that psp_v12_0 has support for loading a sOS firmware,
> > > > > >>>> but
> > > > > >>>> never
> > > > > >>>> calls
> > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > >>>> firmware
> > > > > >>>> for
> > > > > >>>> renoir
> > > > > >>>> and green_sardine, which seem to be the only ASICs with
> > > > > >>>> this
> > > > > >>>> PSP
> > > > > >>>> version.
> > > > > >>>> Is it something that's just not been completely wired up
> > > > > >>>> yet
> > > > > >>>> ?
> > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > >>> firmware
> > > > > >>> is
> > > > > >>> part
> > > > > >>> of
> > > > > >>> the sbios image.  The driver doesn't load it.  We only
> > > > > >>> load
> > > > > >>> it on
> > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > >>> initialization.
> > > > > >>>
> > > > > >>>> That also rings a bell, that we have nothing about
> > > > > >>>> Secure OS
> > > > > >>>> in
> > > > > >>>> the doc
> > > > > >>>> yet (not even the acronym in the glossary).
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> I'm not too familiar with the PSP's path to memory from
> > > > > >>>>> the
> > > > > >>>>> GPU
> > > > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > > > >>>>> through
> > > > > >>>>> carve
> > > > > >>>>> out
> > > > > >>>>> "vram" on APUs so it should work, but I would double
> > > > > >>>>> check
> > > > > >>>>> if
> > > > > >>>>> there
> > > > > >>>>> are any system memory allocations that used to interact
> > > > > >>>>> with
> > > > > >>>>> the PSP
> > > > > >>>>> and see if changing them to vram helps.  It does work
> > > > > >>>>> with
> > > > > >>>>> the
> > > > > >>>>> IOMMU
> > > > > >>>>> enabled on bare metal, so it should work in passthrough
> > > > > >>>>> as
> > > > > >>>>> well
> > > > > >>>>> in
> > > > > >>>>> theory.
> > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > >>>> used
> > > > > >>>> instead
> > > > > >>>> of
> > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and
> > > > > >>>> there
> > > > > >>>> has
> > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > >>>> vram).
> > > > > >>>> Changing it to vram does not make a difference, but then
> > > > > >>>> the
> > > > > >>>> only bo that seems to be used at that point is the one
> > > > > >>>> for
> > > > > >>>> the
> > > > > >>>> psp ring,
> > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > >>>> surprised.
> > > > > >>>>
> > > > > >>>> Maybe I should double-check bo_create calls to hunt for
> > > > > >>>> more
> > > > > >>>> ?
> > > > > >>> We looked into this a bit ourselves and ran into the same
> > > > > >>> issues.
> > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > >>> make
> > > > > >>> further
> > > > > >>> progress, but this was not productized so neither team
> > > > > >>> had
> > > > > >>> the
> > > > > >>> resources to delve further.
> > > > > >>>
> > > > > >>> Alex
> > > > > >>>
> > > > > >>>>
> > > > > >>>> [0]
> > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > >
> > > > >
> > >
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Tue, Dec 21, 2021 at 6:09 PM Yann Dirson <ydirson@free.fr> wrote:
>
>
>
> ----- Mail original -----
> > De: "Alex Deucher" <alexdeucher@gmail.com>
> > À: "Yann Dirson" <ydirson@free.fr>
> > Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>, "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> > Envoyé: Mardi 21 Décembre 2021 23:31:01
> > Objet: Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm
> >
> > On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > >
> > > Alex wrote:
> > > >
> > > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@free.fr>
> > > > wrote:
> > > > >
> > > > > Christian wrote:
> > > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > > Alex wrote:
> > > > > > >> Thinking about this more, I think the problem might be
> > > > > > >> related
> > > > > > >> to
> > > > > > >> CPU
> > > > > > >> access to "VRAM".  APUs don't have dedicated VRAM, they
> > > > > > >> use a
> > > > > > >> reserved
> > > > > > >> carve out region at the top of system memory.  For CPU
> > > > > > >> access
> > > > > > >> to
> > > > > > >> this
> > > > > > >> memory, we kmap the physical address of the carve out
> > > > > > >> region
> > > > > > >> of
> > > > > > >> system
> > > > > > >> memory.  You'll need to make sure that region is
> > > > > > >> accessible to
> > > > > > >> the
> > > > > > >> guest.
> > > > > > > So basically, the non-virt flow is is: (video?) BIOS
> > > > > > > reserves
> > > > > > > memory, marks it
> > > > > > > as reserved in e820, stores the physaddr somewhere, which
> > > > > > > the
> > > > > > > GPU
> > > > > > > driver gets.
> > > > > > > Since I suppose this includes the framebuffer, this
> > > > > > > probably
> > > > > > > has to
> > > > > > > occur around
> > > > > > > the moment the driver calls
> > > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > > (which happens before this hw init step), right ?
> > > > > >
> > > > > > Well, that partially correct. The efifb is using the PCIe
> > > > > > resources
> > > > > > to
> > > > > > access the framebuffer and as far as I know we use that one
> > > > > > to
> > > > > > kick
> > > > > > it out.
> > > > > >
> > > > > > The stolen memory we get over e820/registers is separate to
> > > > > > that.
> > >
> > > How is the stolen memory communicated to the driver ?  That host
> > > physical
> > > memory probably has to be mapped at the same guest physical address
> > > for
> > > the magic to work, right ?
> >
> > Correct.  The driver reads the physical location of that memory from
> > hardware registers.  Removing this chunk of code from gmc_v9_0.c will
> > force the driver to use the BAR, but I'm not sure if there are any
> > other places in the driver that make assumptions about using the
> > physical host address or not on APUs off hand.
> >
> >         if ((adev->flags & AMD_IS_APU) ||
> >             (adev->gmc.xgmi.supported &&
> >              adev->gmc.xgmi.connected_to_cpu)) {
> >                 adev->gmc.aper_base =
> >                         adev->gfxhub.funcs->get_mc_fb_offset(adev) +
> >                         adev->gmc.xgmi.physical_node_id *
> >                         adev->gmc.xgmi.node_segment_size;
> >                 adev->gmc.aper_size = adev->gmc.real_vram_size;
> >         }
> >
> >
> >
> > >
> > > > > >
> > > > > > > ... which brings me to a point that's been puzzling me for
> > > > > > > some
> > > > > > > time, which is
> > > > > > > that as the hw init fails, the efifb driver is still using
> > > > > > > the
> > > > > > > framebuffer.
> > > > > >
> > > > > > No, it isn't. You are probably just still seeing the same
> > > > > > screen.
> > > > > >
> > > > > > The issue is most likely that while efi was kicked out nobody
> > > > > > re-programmed the display hardware to show something
> > > > > > different.
> > > > > >
> > > > > > > Am I right in suspecting that efifb should get stripped of
> > > > > > > its
> > > > > > > ownership of the
> > > > > > > fb aperture first, and that if I don't get a black screen
> > > > > > > on
> > > > > > > hw_init failure
> > > > > > > that issue should be the first focus point ?
> > > > > >
> > > > > > You assumption with the black screen is incorrect. Since the
> > > > > > hardware
> > > > > > works independent even if you kick out efi you still have the
> > > > > > same
> > > > > > screen content, you just can't update it anymore.
> > > > >
> > > > > It's not only that the screen keeps its contents, it's that the
> > > > > dom0
> > > > > happily continues updating it.
> > > >
> > > > If the hypevisor is using efifb, then yes that could be a problem
> > > > as
> > > > the hypervisor could be writing to the efifb resources which ends
> > > > up
> > > > writing to the same physical memory.  That applies to any GPU on
> > > > a
> > > > UEFI system.  You'll need to make sure efifb is not in use in the
> > > > hypervisor.
> > >
> > > That remark evokes several things to me.  First one is that every
> > > time
> > > I've tried booting with efifb disabled in dom0, there was no
> > > visible
> > > improvements in the guest driver - i.i. I really have to dig how
> > > vram mapping
> > > is performed and check things are as expected anyway.
> >
> > Ultimately you end up at the same physical memory.  efifb uses the
> > PCI
> > BAR which points to the same physical memory that the driver directly
> > maps.
> >
> > >
> > > The other is that, when dom0 cannot use efifb, entering a luks key
> > > is
> > > suddenly less user-friendly.  But in theory I'd think we could
> > > overcome
> > > this by letting dom0 use efifb until ready to start the guest, a
> > > simple
> > > driver unbind at the right moment should be expected to work, right
> > > ?
> > > Going further and allowing the guest to use efifb on its own could
> > > possibly be more tricky (starting with a different state?) but does
> > > not seem to sound completely outlandish either - or does it ?
> > >
> >
> > efifb just takes whatever hardware state the GOP driver in the pre-OS
> > environment left the GPU in.  Once you have a driver loaded in the
> > OS,
> > that state is gone so I I don't see much value in using efifb once
> > you
> > have a real driver in the mix.  If you want a console on the host,
> > it's probably better to use 2 GPU or just load the real driver as
> > needed in both the host and guest.
>
> My point is essentially making sure we always have a usable display.
> Dom0 will essentially use it for entering luks key (but having boot logs
> visible may also be useful for some troubleshooting), and I was figuring
> out that once dom0 has relinquished it to start a guest with passthrough,
> the guest might as well make use of it before it can start amdgpu.
> Could be useful as a first step of seeing the GPU in the guest, in
> case there are more issues to be solved to be able to start amdgpu.


Something has to program the GPU to get the displays up.  You need a
driver for that.  Once the OS has loaded the UEFI GOP driver is gone.

Alex

>
> >
> > > >
> > > > Alex
> > > >
> > > >
> > > > >
> > > > > > But putting efi asside what Alex pointed out pretty much
> > > > > > breaks
> > > > > > your
> > > > > > neck trying to forward the device. You maybe could try to
> > > > > > hack
> > > > > > the
> > > > > > driver to use the PCIe BAR for framebuffer access, but that
> > > > > > might
> > > > > > be
> > > > > > quite a bit slower.
> > > > > >
> > > > > > Regards,
> > > > > > Christian.
> > > > > >
> > > > > > >
> > > > > > >> Alex
> > > > > > >>
> > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > >> <alexdeucher@gmail.com>
> > > > > > >> wrote:
> > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > >>> <ydirson@free.fr>
> > > > > > >>> wrote:
> > > > > > >>>> Alex wrote:
> > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > >>>>> <ydirson@free.fr>
> > > > > > >>>>> wrote:
> > > > > > >>>>>> Hi Alex,
> > > > > > >>>>>>
> > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > >>>>>>> integrated
> > > > > > >>>>>>> GPUs.  I
> > > > > > >>>>>>> don't
> > > > > > >>>>>>> know that it will work at all.  We had done a bit of
> > > > > > >>>>>>> testing but
> > > > > > >>>>>>> ran
> > > > > > >>>>>>> into the same issues with the PSP, but never had a
> > > > > > >>>>>>> chance
> > > > > > >>>>>>> to
> > > > > > >>>>>>> debug
> > > > > > >>>>>>> further because this feature is not productized.
> > > > > > >>>>>> ...
> > > > > > >>>>>>> You need a functional PSP to get the GPU driver up
> > > > > > >>>>>>> and
> > > > > > >>>>>>> running.
> > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > >>>>>>
> > > > > > >>>>>> I guess that if I want to have any chance to get the
> > > > > > >>>>>> PSP
> > > > > > >>>>>> working
> > > > > > >>>>>> I'm
> > > > > > >>>>>> going to need more details on it.  A quick search some
> > > > > > >>>>>> time
> > > > > > >>>>>> ago
> > > > > > >>>>>> mostly
> > > > > > >>>>>> brought reverse-engineering work, rather than official
> > > > > > >>>>>> AMD
> > > > > > >>>>>> doc.
> > > > > > >>>>>>   Are
> > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > >>>>> The driver code is pretty much it.
> > > > > > >>>> Let's try to shed some more light on how things work,
> > > > > > >>>> taking
> > > > > > >>>> as
> > > > > > >>>> excuse
> > > > > > >>>> psp_v12_0_ring_create().
> > > > > > >>>>
> > > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > > >>>> implemented
> > > > > > >>>> in
> > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > > > > >>>> [RW]REG32(),
> > > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > > >>> (VF),
> > > > > > >>> some
> > > > > > >>> registers are not available directly via the VF's MMIO
> > > > > > >>> aperture
> > > > > > >>> so
> > > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > > >>> passthrough
> > > > > > >>> this
> > > > > > >>> is not relevant.
> > > > > > >>>
> > > > > > >>>> It accesses registers in the MMIO range of the MP0 IP,
> > > > > > >>>> and
> > > > > > >>>> the
> > > > > > >>>> "MP0"
> > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > >>>> PSP-handling
> > > > > > >>>> code.
> > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The
> > > > > > >>>> MP0
> > > > > > >>>> version
> > > > > > >>> Yes.
> > > > > > >>>
> > > > > > >>>> reported at v11.0.3 by discovery seems to contradict the
> > > > > > >>>> use
> > > > > > >>>> of
> > > > > > >>>> v12.0
> > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > > > > >>>> something ?
> > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > >>>
> > > > > > >>>> More generally (and mostly out of curiosity while we're
> > > > > > >>>> at
> > > > > > >>>> it),
> > > > > > >>>> do we
> > > > > > >>>> have a way to match IPs listed at discovery time with
> > > > > > >>>> the
> > > > > > >>>> ones
> > > > > > >>>> used
> > > > > > >>>> in the driver ?
> > > > > > >>> In general, barring typos, the code is shared at the
> > > > > > >>> major
> > > > > > >>> version
> > > > > > >>> level.  The actual code may or may not need changes to
> > > > > > >>> handle
> > > > > > >>> minor
> > > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > > >>> versions
> > > > > > >>> from
> > > > > > >>> the
> > > > > > >>> ip discovery table to the code contained in the driver.
> > > > > > >>>
> > > > > > >>>> ---
> > > > > > >>>>
> > > > > > >>>> As for the register names, maybe we could have a short
> > > > > > >>>> explanation of
> > > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > > > > >>>> seems
> > > > > > >>>> to
> > > > > > >>>> be
> > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but
> > > > > > >>>> I'm
> > > > > > >>>> not
> > > > > > >>>> sure
> > > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > > >>>> Management
> > > > > > >>>> Network",
> > > > > > >>>> described in [0] as an internal bus.  Are we accessing
> > > > > > >>>> this
> > > > > > >>>> register
> > > > > > >>>> through this SMN ?
> > > > > > >>> These registers are just mailboxes for the PSP firmware.
> > > > > > >>>  All
> > > > > > >>> of
> > > > > > >>> the
> > > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > > >>> firmware.
> > > > > > >>>   They
> > > > > > >>> are basically scratch registers used to communicate
> > > > > > >>> between
> > > > > > >>> the
> > > > > > >>> driver
> > > > > > >>> and the PSP firmware.
> > > > > > >>>
> > > > > > >>>>
> > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > >>>>> the CPU and the rest of the platform.  The GPU driver
> > > > > > >>>>> just
> > > > > > >>>>> interacts
> > > > > > >>>>> with it for a few specific tasks:
> > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > > > > >>>>> applications
> > > > > > >>>>> that run on the PSP for specific functionality, e.g.,
> > > > > > >>>>> HDCP
> > > > > > >>>>> and
> > > > > > >>>>> content
> > > > > > >>>>> protection, etc.)
> > > > > > >>>>> 2. Validating and loading firmware for other engines on
> > > > > > >>>>> the
> > > > > > >>>>> SoC.
> > > > > > >>>>>   This
> > > > > > >>>>> is required to use those engines.
> > > > > > >>>> Trying to understand in more details how we start the
> > > > > > >>>> PSP
> > > > > > >>>> up, I
> > > > > > >>>> noticed
> > > > > > >>>> that psp_v12_0 has support for loading a sOS firmware,
> > > > > > >>>> but
> > > > > > >>>> never
> > > > > > >>>> calls
> > > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > > >>>> firmware
> > > > > > >>>> for
> > > > > > >>>> renoir
> > > > > > >>>> and green_sardine, which seem to be the only ASICs with
> > > > > > >>>> this
> > > > > > >>>> PSP
> > > > > > >>>> version.
> > > > > > >>>> Is it something that's just not been completely wired up
> > > > > > >>>> yet
> > > > > > >>>> ?
> > > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > > >>> firmware
> > > > > > >>> is
> > > > > > >>> part
> > > > > > >>> of
> > > > > > >>> the sbios image.  The driver doesn't load it.  We only
> > > > > > >>> load
> > > > > > >>> it on
> > > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > > >>> initialization.
> > > > > > >>>
> > > > > > >>>> That also rings a bell, that we have nothing about
> > > > > > >>>> Secure OS
> > > > > > >>>> in
> > > > > > >>>> the doc
> > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>> I'm not too familiar with the PSP's path to memory from
> > > > > > >>>>> the
> > > > > > >>>>> GPU
> > > > > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > > > > >>>>> through
> > > > > > >>>>> carve
> > > > > > >>>>> out
> > > > > > >>>>> "vram" on APUs so it should work, but I would double
> > > > > > >>>>> check
> > > > > > >>>>> if
> > > > > > >>>>> there
> > > > > > >>>>> are any system memory allocations that used to interact
> > > > > > >>>>> with
> > > > > > >>>>> the PSP
> > > > > > >>>>> and see if changing them to vram helps.  It does work
> > > > > > >>>>> with
> > > > > > >>>>> the
> > > > > > >>>>> IOMMU
> > > > > > >>>>> enabled on bare metal, so it should work in passthrough
> > > > > > >>>>> as
> > > > > > >>>>> well
> > > > > > >>>>> in
> > > > > > >>>>> theory.
> > > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > > >>>> used
> > > > > > >>>> instead
> > > > > > >>>> of
> > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and
> > > > > > >>>> there
> > > > > > >>>> has
> > > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > > >>>> vram).
> > > > > > >>>> Changing it to vram does not make a difference, but then
> > > > > > >>>> the
> > > > > > >>>> only bo that seems to be used at that point is the one
> > > > > > >>>> for
> > > > > > >>>> the
> > > > > > >>>> psp ring,
> > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > >>>> surprised.
> > > > > > >>>>
> > > > > > >>>> Maybe I should double-check bo_create calls to hunt for
> > > > > > >>>> more
> > > > > > >>>> ?
> > > > > > >>> We looked into this a bit ourselves and ran into the same
> > > > > > >>> issues.
> > > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > > >>> make
> > > > > > >>> further
> > > > > > >>> progress, but this was not productized so neither team
> > > > > > >>> had
> > > > > > >>> the
> > > > > > >>> resources to delve further.
> > > > > > >>>
> > > > > > >>> Alex
> > > > > > >>>
> > > > > > >>>>
> > > > > > >>>> [0]
> > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > >
> > > > > >
> > > >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> On Tue, Dec 21, 2021 at 6:09 PM Yann Dirson <ydirson@free.fr> wrote:
> >
> >
> >
> > ----- Mail original -----
> > > De: "Alex Deucher" <alexdeucher@gmail.com>
> > > À: "Yann Dirson" <ydirson@free.fr>
> > > Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>,
> > > "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> > > Envoyé: Mardi 21 Décembre 2021 23:31:01
> > > Objet: Re: Various problems trying to vga-passthrough a Renoir
> > > iGPU to a xen/qubes-os hvm
> > >
> > > On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr>
> > > wrote:
> > > >
> > > >
> > > > Alex wrote:
> > > > >
> > > > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson
> > > > > <ydirson@free.fr>
> > > > > wrote:
> > > > > >
> > > > > > Christian wrote:
> > > > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > > > Alex wrote:
> > > > > > > >> Thinking about this more, I think the problem might be
> > > > > > > >> related
> > > > > > > >> to
> > > > > > > >> CPU
> > > > > > > >> access to "VRAM".  APUs don't have dedicated VRAM,
> > > > > > > >> they
> > > > > > > >> use a
> > > > > > > >> reserved
> > > > > > > >> carve out region at the top of system memory.  For CPU
> > > > > > > >> access
> > > > > > > >> to
> > > > > > > >> this
> > > > > > > >> memory, we kmap the physical address of the carve out
> > > > > > > >> region
> > > > > > > >> of
> > > > > > > >> system
> > > > > > > >> memory.  You'll need to make sure that region is
> > > > > > > >> accessible to
> > > > > > > >> the
> > > > > > > >> guest.
> > > > > > > > So basically, the non-virt flow is is: (video?) BIOS
> > > > > > > > reserves
> > > > > > > > memory, marks it
> > > > > > > > as reserved in e820, stores the physaddr somewhere,
> > > > > > > > which
> > > > > > > > the
> > > > > > > > GPU
> > > > > > > > driver gets.
> > > > > > > > Since I suppose this includes the framebuffer, this
> > > > > > > > probably
> > > > > > > > has to
> > > > > > > > occur around
> > > > > > > > the moment the driver calls
> > > > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > > > (which happens before this hw init step), right ?
> > > > > > >
> > > > > > > Well, that partially correct. The efifb is using the PCIe
> > > > > > > resources
> > > > > > > to
> > > > > > > access the framebuffer and as far as I know we use that
> > > > > > > one
> > > > > > > to
> > > > > > > kick
> > > > > > > it out.
> > > > > > >
> > > > > > > The stolen memory we get over e820/registers is separate
> > > > > > > to
> > > > > > > that.
> > > >
> > > > How is the stolen memory communicated to the driver ?  That
> > > > host
> > > > physical
> > > > memory probably has to be mapped at the same guest physical
> > > > address
> > > > for
> > > > the magic to work, right ?
> > >
> > > Correct.  The driver reads the physical location of that memory
> > > from
> > > hardware registers.  Removing this chunk of code from gmc_v9_0.c
> > > will
> > > force the driver to use the BAR, but I'm not sure if there are
> > > any
> > > other places in the driver that make assumptions about using the
> > > physical host address or not on APUs off hand.
> > >
> > >         if ((adev->flags & AMD_IS_APU) ||
> > >             (adev->gmc.xgmi.supported &&
> > >              adev->gmc.xgmi.connected_to_cpu)) {
> > >                 adev->gmc.aper_base =
> > >                         adev->gfxhub.funcs->get_mc_fb_offset(adev)
> > >                         +
> > >                         adev->gmc.xgmi.physical_node_id *
> > >                         adev->gmc.xgmi.node_segment_size;
> > >                 adev->gmc.aper_size = adev->gmc.real_vram_size;
> > >         }
> > >
> > >
> > >
> > > >
> > > > > > >
> > > > > > > > ... which brings me to a point that's been puzzling me
> > > > > > > > for
> > > > > > > > some
> > > > > > > > time, which is
> > > > > > > > that as the hw init fails, the efifb driver is still
> > > > > > > > using
> > > > > > > > the
> > > > > > > > framebuffer.
> > > > > > >
> > > > > > > No, it isn't. You are probably just still seeing the same
> > > > > > > screen.
> > > > > > >
> > > > > > > The issue is most likely that while efi was kicked out
> > > > > > > nobody
> > > > > > > re-programmed the display hardware to show something
> > > > > > > different.
> > > > > > >
> > > > > > > > Am I right in suspecting that efifb should get stripped
> > > > > > > > of
> > > > > > > > its
> > > > > > > > ownership of the
> > > > > > > > fb aperture first, and that if I don't get a black
> > > > > > > > screen
> > > > > > > > on
> > > > > > > > hw_init failure
> > > > > > > > that issue should be the first focus point ?
> > > > > > >
> > > > > > > You assumption with the black screen is incorrect. Since
> > > > > > > the
> > > > > > > hardware
> > > > > > > works independent even if you kick out efi you still have
> > > > > > > the
> > > > > > > same
> > > > > > > screen content, you just can't update it anymore.
> > > > > >
> > > > > > It's not only that the screen keeps its contents, it's that
> > > > > > the
> > > > > > dom0
> > > > > > happily continues updating it.
> > > > >
> > > > > If the hypevisor is using efifb, then yes that could be a
> > > > > problem
> > > > > as
> > > > > the hypervisor could be writing to the efifb resources which
> > > > > ends
> > > > > up
> > > > > writing to the same physical memory.  That applies to any GPU
> > > > > on
> > > > > a
> > > > > UEFI system.  You'll need to make sure efifb is not in use in
> > > > > the
> > > > > hypervisor.
> > > >
> > > > That remark evokes several things to me.  First one is that
> > > > every
> > > > time
> > > > I've tried booting with efifb disabled in dom0, there was no
> > > > visible
> > > > improvements in the guest driver - i.i. I really have to dig
> > > > how
> > > > vram mapping
> > > > is performed and check things are as expected anyway.
> > >
> > > Ultimately you end up at the same physical memory.  efifb uses
> > > the
> > > PCI
> > > BAR which points to the same physical memory that the driver
> > > directly
> > > maps.
> > >
> > > >
> > > > The other is that, when dom0 cannot use efifb, entering a luks
> > > > key
> > > > is
> > > > suddenly less user-friendly.  But in theory I'd think we could
> > > > overcome
> > > > this by letting dom0 use efifb until ready to start the guest,
> > > > a
> > > > simple
> > > > driver unbind at the right moment should be expected to work,
> > > > right
> > > > ?
> > > > Going further and allowing the guest to use efifb on its own
> > > > could
> > > > possibly be more tricky (starting with a different state?) but
> > > > does
> > > > not seem to sound completely outlandish either - or does it ?
> > > >
> > >
> > > efifb just takes whatever hardware state the GOP driver in the
> > > pre-OS
> > > environment left the GPU in.  Once you have a driver loaded in
> > > the
> > > OS,
> > > that state is gone so I I don't see much value in using efifb
> > > once
> > > you
> > > have a real driver in the mix.  If you want a console on the
> > > host,
> > > it's probably better to use 2 GPU or just load the real driver as
> > > needed in both the host and guest.
> >
> > My point is essentially making sure we always have a usable
> > display.
> > Dom0 will essentially use it for entering luks key (but having boot
> > logs
> > visible may also be useful for some troubleshooting), and I was
> > figuring
> > out that once dom0 has relinquished it to start a guest with
> > passthrough,
> > the guest might as well make use of it before it can start amdgpu.
> > Could be useful as a first step of seeing the GPU in the guest, in
> > case there are more issues to be solved to be able to start amdgpu.
> 
> 
> Something has to program the GPU to get the displays up.  You need a
> driver for that.  Once the OS has loaded the UEFI GOP driver is gone.

My understanding is that efifb is only making use of the framebuffer set up
by the GOP.  If that's correct I don't see why the same framebuffer could
not be used by a guest once the host releases it (provided the guest boots
Tiano Core and we can tell the latter to pass the GOP framebuffer through).
Where do I get it wrong ?

> > > > >
> > > > > >
> > > > > > > But putting efi asside what Alex pointed out pretty much
> > > > > > > breaks
> > > > > > > your
> > > > > > > neck trying to forward the device. You maybe could try to
> > > > > > > hack
> > > > > > > the
> > > > > > > driver to use the PCIe BAR for framebuffer access, but
> > > > > > > that
> > > > > > > might
> > > > > > > be
> > > > > > > quite a bit slower.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Christian.
> > > > > > >
> > > > > > > >
> > > > > > > >> Alex
> > > > > > > >>
> > > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > > >> <alexdeucher@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > > >>> <ydirson@free.fr>
> > > > > > > >>> wrote:
> > > > > > > >>>> Alex wrote:
> > > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > > >>>>> <ydirson@free.fr>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>> Hi Alex,
> > > > > > > >>>>>>
> > > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > > >>>>>>> integrated
> > > > > > > >>>>>>> GPUs.  I
> > > > > > > >>>>>>> don't
> > > > > > > >>>>>>> know that it will work at all.  We had done a bit
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>> testing but
> > > > > > > >>>>>>> ran
> > > > > > > >>>>>>> into the same issues with the PSP, but never had
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>> chance
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>> debug
> > > > > > > >>>>>>> further because this feature is not productized.
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>> You need a functional PSP to get the GPU driver
> > > > > > > >>>>>>> up
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>> running.
> > > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > > >>>>>>
> > > > > > > >>>>>> I guess that if I want to have any chance to get
> > > > > > > >>>>>> the
> > > > > > > >>>>>> PSP
> > > > > > > >>>>>> working
> > > > > > > >>>>>> I'm
> > > > > > > >>>>>> going to need more details on it.  A quick search
> > > > > > > >>>>>> some
> > > > > > > >>>>>> time
> > > > > > > >>>>>> ago
> > > > > > > >>>>>> mostly
> > > > > > > >>>>>> brought reverse-engineering work, rather than
> > > > > > > >>>>>> official
> > > > > > > >>>>>> AMD
> > > > > > > >>>>>> doc.
> > > > > > > >>>>>>   Are
> > > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > > >>>>> The driver code is pretty much it.
> > > > > > > >>>> Let's try to shed some more light on how things
> > > > > > > >>>> work,
> > > > > > > >>>> taking
> > > > > > > >>>> as
> > > > > > > >>>> excuse
> > > > > > > >>>> psp_v12_0_ring_create().
> > > > > > > >>>>
> > > > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > > > >>>> implemented
> > > > > > > >>>> in
> > > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is
> > > > > > > >>>> basically a
> > > > > > > >>>> [RW]REG32(),
> > > > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > > > >>> (VF),
> > > > > > > >>> some
> > > > > > > >>> registers are not available directly via the VF's
> > > > > > > >>> MMIO
> > > > > > > >>> aperture
> > > > > > > >>> so
> > > > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > > > >>> passthrough
> > > > > > > >>> this
> > > > > > > >>> is not relevant.
> > > > > > > >>>
> > > > > > > >>>> It accesses registers in the MMIO range of the MP0
> > > > > > > >>>> IP,
> > > > > > > >>>> and
> > > > > > > >>>> the
> > > > > > > >>>> "MP0"
> > > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > > >>>> PSP-handling
> > > > > > > >>>> code.
> > > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?
> > > > > > > >>>>  The
> > > > > > > >>>> MP0
> > > > > > > >>>> version
> > > > > > > >>> Yes.
> > > > > > > >>>
> > > > > > > >>>> reported at v11.0.3 by discovery seems to contradict
> > > > > > > >>>> the
> > > > > > > >>>> use
> > > > > > > >>>> of
> > > > > > > >>>> v12.0
> > > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I
> > > > > > > >>>> miss
> > > > > > > >>>> something ?
> > > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > > >>>
> > > > > > > >>>> More generally (and mostly out of curiosity while
> > > > > > > >>>> we're
> > > > > > > >>>> at
> > > > > > > >>>> it),
> > > > > > > >>>> do we
> > > > > > > >>>> have a way to match IPs listed at discovery time
> > > > > > > >>>> with
> > > > > > > >>>> the
> > > > > > > >>>> ones
> > > > > > > >>>> used
> > > > > > > >>>> in the driver ?
> > > > > > > >>> In general, barring typos, the code is shared at the
> > > > > > > >>> major
> > > > > > > >>> version
> > > > > > > >>> level.  The actual code may or may not need changes
> > > > > > > >>> to
> > > > > > > >>> handle
> > > > > > > >>> minor
> > > > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > > > >>> versions
> > > > > > > >>> from
> > > > > > > >>> the
> > > > > > > >>> ip discovery table to the code contained in the
> > > > > > > >>> driver.
> > > > > > > >>>
> > > > > > > >>>> ---
> > > > > > > >>>>
> > > > > > > >>>> As for the register names, maybe we could have a
> > > > > > > >>>> short
> > > > > > > >>>> explanation of
> > > > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69:
> > > > > > > >>>> that
> > > > > > > >>>> seems
> > > > > > > >>>> to
> > > > > > > >>>> be
> > > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP,
> > > > > > > >>>> but
> > > > > > > >>>> I'm
> > > > > > > >>>> not
> > > > > > > >>>> sure
> > > > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > > > >>>> Management
> > > > > > > >>>> Network",
> > > > > > > >>>> described in [0] as an internal bus.  Are we
> > > > > > > >>>> accessing
> > > > > > > >>>> this
> > > > > > > >>>> register
> > > > > > > >>>> through this SMN ?
> > > > > > > >>> These registers are just mailboxes for the PSP
> > > > > > > >>> firmware.
> > > > > > > >>>  All
> > > > > > > >>> of
> > > > > > > >>> the
> > > > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > > > >>> firmware.
> > > > > > > >>>   They
> > > > > > > >>> are basically scratch registers used to communicate
> > > > > > > >>> between
> > > > > > > >>> the
> > > > > > > >>> driver
> > > > > > > >>> and the PSP firmware.
> > > > > > > >>>
> > > > > > > >>>>
> > > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > > >>>>> the CPU and the rest of the platform.  The GPU
> > > > > > > >>>>> driver
> > > > > > > >>>>> just
> > > > > > > >>>>> interacts
> > > > > > > >>>>> with it for a few specific tasks:
> > > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted
> > > > > > > >>>>> firmware
> > > > > > > >>>>> applications
> > > > > > > >>>>> that run on the PSP for specific functionality,
> > > > > > > >>>>> e.g.,
> > > > > > > >>>>> HDCP
> > > > > > > >>>>> and
> > > > > > > >>>>> content
> > > > > > > >>>>> protection, etc.)
> > > > > > > >>>>> 2. Validating and loading firmware for other
> > > > > > > >>>>> engines on
> > > > > > > >>>>> the
> > > > > > > >>>>> SoC.
> > > > > > > >>>>>   This
> > > > > > > >>>>> is required to use those engines.
> > > > > > > >>>> Trying to understand in more details how we start
> > > > > > > >>>> the
> > > > > > > >>>> PSP
> > > > > > > >>>> up, I
> > > > > > > >>>> noticed
> > > > > > > >>>> that psp_v12_0 has support for loading a sOS
> > > > > > > >>>> firmware,
> > > > > > > >>>> but
> > > > > > > >>>> never
> > > > > > > >>>> calls
> > > > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > > > >>>> firmware
> > > > > > > >>>> for
> > > > > > > >>>> renoir
> > > > > > > >>>> and green_sardine, which seem to be the only ASICs
> > > > > > > >>>> with
> > > > > > > >>>> this
> > > > > > > >>>> PSP
> > > > > > > >>>> version.
> > > > > > > >>>> Is it something that's just not been completely
> > > > > > > >>>> wired up
> > > > > > > >>>> yet
> > > > > > > >>>> ?
> > > > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > > > >>> firmware
> > > > > > > >>> is
> > > > > > > >>> part
> > > > > > > >>> of
> > > > > > > >>> the sbios image.  The driver doesn't load it.  We
> > > > > > > >>> only
> > > > > > > >>> load
> > > > > > > >>> it on
> > > > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > > > >>> initialization.
> > > > > > > >>>
> > > > > > > >>>> That also rings a bell, that we have nothing about
> > > > > > > >>>> Secure OS
> > > > > > > >>>> in
> > > > > > > >>>> the doc
> > > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>> I'm not too familiar with the PSP's path to memory
> > > > > > > >>>>> from
> > > > > > > >>>>> the
> > > > > > > >>>>> GPU
> > > > > > > >>>>> perspective.  IIRC, most memory used by the PSP
> > > > > > > >>>>> goes
> > > > > > > >>>>> through
> > > > > > > >>>>> carve
> > > > > > > >>>>> out
> > > > > > > >>>>> "vram" on APUs so it should work, but I would
> > > > > > > >>>>> double
> > > > > > > >>>>> check
> > > > > > > >>>>> if
> > > > > > > >>>>> there
> > > > > > > >>>>> are any system memory allocations that used to
> > > > > > > >>>>> interact
> > > > > > > >>>>> with
> > > > > > > >>>>> the PSP
> > > > > > > >>>>> and see if changing them to vram helps.  It does
> > > > > > > >>>>> work
> > > > > > > >>>>> with
> > > > > > > >>>>> the
> > > > > > > >>>>> IOMMU
> > > > > > > >>>>> enabled on bare metal, so it should work in
> > > > > > > >>>>> passthrough
> > > > > > > >>>>> as
> > > > > > > >>>>> well
> > > > > > > >>>>> in
> > > > > > > >>>>> theory.
> > > > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > > > >>>> used
> > > > > > > >>>> instead
> > > > > > > >>>> of
> > > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used
> > > > > > > >>>> (and
> > > > > > > >>>> there
> > > > > > > >>>> has
> > > > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > > > >>>> vram).
> > > > > > > >>>> Changing it to vram does not make a difference, but
> > > > > > > >>>> then
> > > > > > > >>>> the
> > > > > > > >>>> only bo that seems to be used at that point is the
> > > > > > > >>>> one
> > > > > > > >>>> for
> > > > > > > >>>> the
> > > > > > > >>>> psp ring,
> > > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > > >>>> surprised.
> > > > > > > >>>>
> > > > > > > >>>> Maybe I should double-check bo_create calls to hunt
> > > > > > > >>>> for
> > > > > > > >>>> more
> > > > > > > >>>> ?
> > > > > > > >>> We looked into this a bit ourselves and ran into the
> > > > > > > >>> same
> > > > > > > >>> issues.
> > > > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > > > >>> make
> > > > > > > >>> further
> > > > > > > >>> progress, but this was not productized so neither
> > > > > > > >>> team
> > > > > > > >>> had
> > > > > > > >>> the
> > > > > > > >>> resources to delve further.
> > > > > > > >>>
> > > > > > > >>> Alex
> > > > > > > >>>
> > > > > > > >>>>
> > > > > > > >>>> [0]
> > > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > > >
> > > > > > >
> > > > >
> > >
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Wed, Dec 29, 2021 at 11:59 AM Yann Dirson <ydirson@free.fr> wrote:
>
> Alex wrote:
> > On Tue, Dec 21, 2021 at 6:09 PM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > >
> > >
> > > ----- Mail original -----
> > > > De: "Alex Deucher" <alexdeucher@gmail.com>
> > > > À: "Yann Dirson" <ydirson@free.fr>
> > > > Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>,
> > > > "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> > > > Envoyé: Mardi 21 Décembre 2021 23:31:01
> > > > Objet: Re: Various problems trying to vga-passthrough a Renoir
> > > > iGPU to a xen/qubes-os hvm
> > > >
> > > > On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr>
> > > > wrote:
> > > > >
> > > > >
> > > > > Alex wrote:
> > > > > >
> > > > > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson
> > > > > > <ydirson@free.fr>
> > > > > > wrote:
> > > > > > >
> > > > > > > Christian wrote:
> > > > > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > > > > Alex wrote:
> > > > > > > > >> Thinking about this more, I think the problem might be
> > > > > > > > >> related
> > > > > > > > >> to
> > > > > > > > >> CPU
> > > > > > > > >> access to "VRAM".  APUs don't have dedicated VRAM,
> > > > > > > > >> they
> > > > > > > > >> use a
> > > > > > > > >> reserved
> > > > > > > > >> carve out region at the top of system memory.  For CPU
> > > > > > > > >> access
> > > > > > > > >> to
> > > > > > > > >> this
> > > > > > > > >> memory, we kmap the physical address of the carve out
> > > > > > > > >> region
> > > > > > > > >> of
> > > > > > > > >> system
> > > > > > > > >> memory.  You'll need to make sure that region is
> > > > > > > > >> accessible to
> > > > > > > > >> the
> > > > > > > > >> guest.
> > > > > > > > > So basically, the non-virt flow is is: (video?) BIOS
> > > > > > > > > reserves
> > > > > > > > > memory, marks it
> > > > > > > > > as reserved in e820, stores the physaddr somewhere,
> > > > > > > > > which
> > > > > > > > > the
> > > > > > > > > GPU
> > > > > > > > > driver gets.
> > > > > > > > > Since I suppose this includes the framebuffer, this
> > > > > > > > > probably
> > > > > > > > > has to
> > > > > > > > > occur around
> > > > > > > > > the moment the driver calls
> > > > > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > > > > (which happens before this hw init step), right ?
> > > > > > > >
> > > > > > > > Well, that partially correct. The efifb is using the PCIe
> > > > > > > > resources
> > > > > > > > to
> > > > > > > > access the framebuffer and as far as I know we use that
> > > > > > > > one
> > > > > > > > to
> > > > > > > > kick
> > > > > > > > it out.
> > > > > > > >
> > > > > > > > The stolen memory we get over e820/registers is separate
> > > > > > > > to
> > > > > > > > that.
> > > > >
> > > > > How is the stolen memory communicated to the driver ?  That
> > > > > host
> > > > > physical
> > > > > memory probably has to be mapped at the same guest physical
> > > > > address
> > > > > for
> > > > > the magic to work, right ?
> > > >
> > > > Correct.  The driver reads the physical location of that memory
> > > > from
> > > > hardware registers.  Removing this chunk of code from gmc_v9_0.c
> > > > will
> > > > force the driver to use the BAR, but I'm not sure if there are
> > > > any
> > > > other places in the driver that make assumptions about using the
> > > > physical host address or not on APUs off hand.
> > > >
> > > >         if ((adev->flags & AMD_IS_APU) ||
> > > >             (adev->gmc.xgmi.supported &&
> > > >              adev->gmc.xgmi.connected_to_cpu)) {
> > > >                 adev->gmc.aper_base =
> > > >                         adev->gfxhub.funcs->get_mc_fb_offset(adev)
> > > >                         +
> > > >                         adev->gmc.xgmi.physical_node_id *
> > > >                         adev->gmc.xgmi.node_segment_size;
> > > >                 adev->gmc.aper_size = adev->gmc.real_vram_size;
> > > >         }
> > > >
> > > >
> > > >
> > > > >
> > > > > > > >
> > > > > > > > > ... which brings me to a point that's been puzzling me
> > > > > > > > > for
> > > > > > > > > some
> > > > > > > > > time, which is
> > > > > > > > > that as the hw init fails, the efifb driver is still
> > > > > > > > > using
> > > > > > > > > the
> > > > > > > > > framebuffer.
> > > > > > > >
> > > > > > > > No, it isn't. You are probably just still seeing the same
> > > > > > > > screen.
> > > > > > > >
> > > > > > > > The issue is most likely that while efi was kicked out
> > > > > > > > nobody
> > > > > > > > re-programmed the display hardware to show something
> > > > > > > > different.
> > > > > > > >
> > > > > > > > > Am I right in suspecting that efifb should get stripped
> > > > > > > > > of
> > > > > > > > > its
> > > > > > > > > ownership of the
> > > > > > > > > fb aperture first, and that if I don't get a black
> > > > > > > > > screen
> > > > > > > > > on
> > > > > > > > > hw_init failure
> > > > > > > > > that issue should be the first focus point ?
> > > > > > > >
> > > > > > > > You assumption with the black screen is incorrect. Since
> > > > > > > > the
> > > > > > > > hardware
> > > > > > > > works independent even if you kick out efi you still have
> > > > > > > > the
> > > > > > > > same
> > > > > > > > screen content, you just can't update it anymore.
> > > > > > >
> > > > > > > It's not only that the screen keeps its contents, it's that
> > > > > > > the
> > > > > > > dom0
> > > > > > > happily continues updating it.
> > > > > >
> > > > > > If the hypevisor is using efifb, then yes that could be a
> > > > > > problem
> > > > > > as
> > > > > > the hypervisor could be writing to the efifb resources which
> > > > > > ends
> > > > > > up
> > > > > > writing to the same physical memory.  That applies to any GPU
> > > > > > on
> > > > > > a
> > > > > > UEFI system.  You'll need to make sure efifb is not in use in
> > > > > > the
> > > > > > hypervisor.
> > > > >
> > > > > That remark evokes several things to me.  First one is that
> > > > > every
> > > > > time
> > > > > I've tried booting with efifb disabled in dom0, there was no
> > > > > visible
> > > > > improvements in the guest driver - i.i. I really have to dig
> > > > > how
> > > > > vram mapping
> > > > > is performed and check things are as expected anyway.
> > > >
> > > > Ultimately you end up at the same physical memory.  efifb uses
> > > > the
> > > > PCI
> > > > BAR which points to the same physical memory that the driver
> > > > directly
> > > > maps.
> > > >
> > > > >
> > > > > The other is that, when dom0 cannot use efifb, entering a luks
> > > > > key
> > > > > is
> > > > > suddenly less user-friendly.  But in theory I'd think we could
> > > > > overcome
> > > > > this by letting dom0 use efifb until ready to start the guest,
> > > > > a
> > > > > simple
> > > > > driver unbind at the right moment should be expected to work,
> > > > > right
> > > > > ?
> > > > > Going further and allowing the guest to use efifb on its own
> > > > > could
> > > > > possibly be more tricky (starting with a different state?) but
> > > > > does
> > > > > not seem to sound completely outlandish either - or does it ?
> > > > >
> > > >
> > > > efifb just takes whatever hardware state the GOP driver in the
> > > > pre-OS
> > > > environment left the GPU in.  Once you have a driver loaded in
> > > > the
> > > > OS,
> > > > that state is gone so I I don't see much value in using efifb
> > > > once
> > > > you
> > > > have a real driver in the mix.  If you want a console on the
> > > > host,
> > > > it's probably better to use 2 GPU or just load the real driver as
> > > > needed in both the host and guest.
> > >
> > > My point is essentially making sure we always have a usable
> > > display.
> > > Dom0 will essentially use it for entering luks key (but having boot
> > > logs
> > > visible may also be useful for some troubleshooting), and I was
> > > figuring
> > > out that once dom0 has relinquished it to start a guest with
> > > passthrough,
> > > the guest might as well make use of it before it can start amdgpu.
> > > Could be useful as a first step of seeing the GPU in the guest, in
> > > case there are more issues to be solved to be able to start amdgpu.
> >
> >
> > Something has to program the GPU to get the displays up.  You need a
> > driver for that.  Once the OS has loaded the UEFI GOP driver is gone.
>
> My understanding is that efifb is only making use of the framebuffer set up
> by the GOP.  If that's correct I don't see why the same framebuffer could
> not be used by a guest once the host releases it (provided the guest boots
> Tiano Core and we can tell the latter to pass the GOP framebuffer through).
> Where do I get it wrong ?

Sure, you could re-use the GPU memory that backs the framebuffer, but
something needs to reprogram the display timing so that you get a
signal to your monitor.  That is what the driver does (GOP or OS
driver).

Alex

>
> > > > > >
> > > > > > >
> > > > > > > > But putting efi asside what Alex pointed out pretty much
> > > > > > > > breaks
> > > > > > > > your
> > > > > > > > neck trying to forward the device. You maybe could try to
> > > > > > > > hack
> > > > > > > > the
> > > > > > > > driver to use the PCIe BAR for framebuffer access, but
> > > > > > > > that
> > > > > > > > might
> > > > > > > > be
> > > > > > > > quite a bit slower.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Christian.
> > > > > > > >
> > > > > > > > >
> > > > > > > > >> Alex
> > > > > > > > >>
> > > > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > > > >> <alexdeucher@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > > > >>> <ydirson@free.fr>
> > > > > > > > >>> wrote:
> > > > > > > > >>>> Alex wrote:
> > > > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > > > >>>>> <ydirson@free.fr>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>> Hi Alex,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > > > >>>>>>> integrated
> > > > > > > > >>>>>>> GPUs.  I
> > > > > > > > >>>>>>> don't
> > > > > > > > >>>>>>> know that it will work at all.  We had done a bit
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>> testing but
> > > > > > > > >>>>>>> ran
> > > > > > > > >>>>>>> into the same issues with the PSP, but never had
> > > > > > > > >>>>>>> a
> > > > > > > > >>>>>>> chance
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>> debug
> > > > > > > > >>>>>>> further because this feature is not productized.
> > > > > > > > >>>>>> ...
> > > > > > > > >>>>>>> You need a functional PSP to get the GPU driver
> > > > > > > > >>>>>>> up
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>> running.
> > > > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I guess that if I want to have any chance to get
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>> PSP
> > > > > > > > >>>>>> working
> > > > > > > > >>>>>> I'm
> > > > > > > > >>>>>> going to need more details on it.  A quick search
> > > > > > > > >>>>>> some
> > > > > > > > >>>>>> time
> > > > > > > > >>>>>> ago
> > > > > > > > >>>>>> mostly
> > > > > > > > >>>>>> brought reverse-engineering work, rather than
> > > > > > > > >>>>>> official
> > > > > > > > >>>>>> AMD
> > > > > > > > >>>>>> doc.
> > > > > > > > >>>>>>   Are
> > > > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > > > >>>>> The driver code is pretty much it.
> > > > > > > > >>>> Let's try to shed some more light on how things
> > > > > > > > >>>> work,
> > > > > > > > >>>> taking
> > > > > > > > >>>> as
> > > > > > > > >>>> excuse
> > > > > > > > >>>> psp_v12_0_ring_create().
> > > > > > > > >>>>
> > > > > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > > > > >>>> implemented
> > > > > > > > >>>> in
> > > > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is
> > > > > > > > >>>> basically a
> > > > > > > > >>>> [RW]REG32(),
> > > > > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > > > > >>> (VF),
> > > > > > > > >>> some
> > > > > > > > >>> registers are not available directly via the VF's
> > > > > > > > >>> MMIO
> > > > > > > > >>> aperture
> > > > > > > > >>> so
> > > > > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > > > > >>> passthrough
> > > > > > > > >>> this
> > > > > > > > >>> is not relevant.
> > > > > > > > >>>
> > > > > > > > >>>> It accesses registers in the MMIO range of the MP0
> > > > > > > > >>>> IP,
> > > > > > > > >>>> and
> > > > > > > > >>>> the
> > > > > > > > >>>> "MP0"
> > > > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > > > >>>> PSP-handling
> > > > > > > > >>>> code.
> > > > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?
> > > > > > > > >>>>  The
> > > > > > > > >>>> MP0
> > > > > > > > >>>> version
> > > > > > > > >>> Yes.
> > > > > > > > >>>
> > > > > > > > >>>> reported at v11.0.3 by discovery seems to contradict
> > > > > > > > >>>> the
> > > > > > > > >>>> use
> > > > > > > > >>>> of
> > > > > > > > >>>> v12.0
> > > > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I
> > > > > > > > >>>> miss
> > > > > > > > >>>> something ?
> > > > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > > > >>>
> > > > > > > > >>>> More generally (and mostly out of curiosity while
> > > > > > > > >>>> we're
> > > > > > > > >>>> at
> > > > > > > > >>>> it),
> > > > > > > > >>>> do we
> > > > > > > > >>>> have a way to match IPs listed at discovery time
> > > > > > > > >>>> with
> > > > > > > > >>>> the
> > > > > > > > >>>> ones
> > > > > > > > >>>> used
> > > > > > > > >>>> in the driver ?
> > > > > > > > >>> In general, barring typos, the code is shared at the
> > > > > > > > >>> major
> > > > > > > > >>> version
> > > > > > > > >>> level.  The actual code may or may not need changes
> > > > > > > > >>> to
> > > > > > > > >>> handle
> > > > > > > > >>> minor
> > > > > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > > > > >>> versions
> > > > > > > > >>> from
> > > > > > > > >>> the
> > > > > > > > >>> ip discovery table to the code contained in the
> > > > > > > > >>> driver.
> > > > > > > > >>>
> > > > > > > > >>>> ---
> > > > > > > > >>>>
> > > > > > > > >>>> As for the register names, maybe we could have a
> > > > > > > > >>>> short
> > > > > > > > >>>> explanation of
> > > > > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69:
> > > > > > > > >>>> that
> > > > > > > > >>>> seems
> > > > > > > > >>>> to
> > > > > > > > >>>> be
> > > > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP,
> > > > > > > > >>>> but
> > > > > > > > >>>> I'm
> > > > > > > > >>>> not
> > > > > > > > >>>> sure
> > > > > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > > > > >>>> Management
> > > > > > > > >>>> Network",
> > > > > > > > >>>> described in [0] as an internal bus.  Are we
> > > > > > > > >>>> accessing
> > > > > > > > >>>> this
> > > > > > > > >>>> register
> > > > > > > > >>>> through this SMN ?
> > > > > > > > >>> These registers are just mailboxes for the PSP
> > > > > > > > >>> firmware.
> > > > > > > > >>>  All
> > > > > > > > >>> of
> > > > > > > > >>> the
> > > > > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > > > > >>> firmware.
> > > > > > > > >>>   They
> > > > > > > > >>> are basically scratch registers used to communicate
> > > > > > > > >>> between
> > > > > > > > >>> the
> > > > > > > > >>> driver
> > > > > > > > >>> and the PSP firmware.
> > > > > > > > >>>
> > > > > > > > >>>>
> > > > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > > > >>>>> the CPU and the rest of the platform.  The GPU
> > > > > > > > >>>>> driver
> > > > > > > > >>>>> just
> > > > > > > > >>>>> interacts
> > > > > > > > >>>>> with it for a few specific tasks:
> > > > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted
> > > > > > > > >>>>> firmware
> > > > > > > > >>>>> applications
> > > > > > > > >>>>> that run on the PSP for specific functionality,
> > > > > > > > >>>>> e.g.,
> > > > > > > > >>>>> HDCP
> > > > > > > > >>>>> and
> > > > > > > > >>>>> content
> > > > > > > > >>>>> protection, etc.)
> > > > > > > > >>>>> 2. Validating and loading firmware for other
> > > > > > > > >>>>> engines on
> > > > > > > > >>>>> the
> > > > > > > > >>>>> SoC.
> > > > > > > > >>>>>   This
> > > > > > > > >>>>> is required to use those engines.
> > > > > > > > >>>> Trying to understand in more details how we start
> > > > > > > > >>>> the
> > > > > > > > >>>> PSP
> > > > > > > > >>>> up, I
> > > > > > > > >>>> noticed
> > > > > > > > >>>> that psp_v12_0 has support for loading a sOS
> > > > > > > > >>>> firmware,
> > > > > > > > >>>> but
> > > > > > > > >>>> never
> > > > > > > > >>>> calls
> > > > > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > > > > >>>> firmware
> > > > > > > > >>>> for
> > > > > > > > >>>> renoir
> > > > > > > > >>>> and green_sardine, which seem to be the only ASICs
> > > > > > > > >>>> with
> > > > > > > > >>>> this
> > > > > > > > >>>> PSP
> > > > > > > > >>>> version.
> > > > > > > > >>>> Is it something that's just not been completely
> > > > > > > > >>>> wired up
> > > > > > > > >>>> yet
> > > > > > > > >>>> ?
> > > > > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > > > > >>> firmware
> > > > > > > > >>> is
> > > > > > > > >>> part
> > > > > > > > >>> of
> > > > > > > > >>> the sbios image.  The driver doesn't load it.  We
> > > > > > > > >>> only
> > > > > > > > >>> load
> > > > > > > > >>> it on
> > > > > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > > > > >>> initialization.
> > > > > > > > >>>
> > > > > > > > >>>> That also rings a bell, that we have nothing about
> > > > > > > > >>>> Secure OS
> > > > > > > > >>>> in
> > > > > > > > >>>> the doc
> > > > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>> I'm not too familiar with the PSP's path to memory
> > > > > > > > >>>>> from
> > > > > > > > >>>>> the
> > > > > > > > >>>>> GPU
> > > > > > > > >>>>> perspective.  IIRC, most memory used by the PSP
> > > > > > > > >>>>> goes
> > > > > > > > >>>>> through
> > > > > > > > >>>>> carve
> > > > > > > > >>>>> out
> > > > > > > > >>>>> "vram" on APUs so it should work, but I would
> > > > > > > > >>>>> double
> > > > > > > > >>>>> check
> > > > > > > > >>>>> if
> > > > > > > > >>>>> there
> > > > > > > > >>>>> are any system memory allocations that used to
> > > > > > > > >>>>> interact
> > > > > > > > >>>>> with
> > > > > > > > >>>>> the PSP
> > > > > > > > >>>>> and see if changing them to vram helps.  It does
> > > > > > > > >>>>> work
> > > > > > > > >>>>> with
> > > > > > > > >>>>> the
> > > > > > > > >>>>> IOMMU
> > > > > > > > >>>>> enabled on bare metal, so it should work in
> > > > > > > > >>>>> passthrough
> > > > > > > > >>>>> as
> > > > > > > > >>>>> well
> > > > > > > > >>>>> in
> > > > > > > > >>>>> theory.
> > > > > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > > > > >>>> used
> > > > > > > > >>>> instead
> > > > > > > > >>>> of
> > > > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used
> > > > > > > > >>>> (and
> > > > > > > > >>>> there
> > > > > > > > >>>> has
> > > > > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > > > > >>>> vram).
> > > > > > > > >>>> Changing it to vram does not make a difference, but
> > > > > > > > >>>> then
> > > > > > > > >>>> the
> > > > > > > > >>>> only bo that seems to be used at that point is the
> > > > > > > > >>>> one
> > > > > > > > >>>> for
> > > > > > > > >>>> the
> > > > > > > > >>>> psp ring,
> > > > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > > > >>>> surprised.
> > > > > > > > >>>>
> > > > > > > > >>>> Maybe I should double-check bo_create calls to hunt
> > > > > > > > >>>> for
> > > > > > > > >>>> more
> > > > > > > > >>>> ?
> > > > > > > > >>> We looked into this a bit ourselves and ran into the
> > > > > > > > >>> same
> > > > > > > > >>> issues.
> > > > > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > > > > >>> make
> > > > > > > > >>> further
> > > > > > > > >>> progress, but this was not productized so neither
> > > > > > > > >>> team
> > > > > > > > >>> had
> > > > > > > > >>> the
> > > > > > > > >>> resources to delve further.
> > > > > > > > >>>
> > > > > > > > >>> Alex
> > > > > > > > >>>
> > > > > > > > >>>>
> > > > > > > > >>>> [0]
> > > > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> On Wed, Dec 29, 2021 at 11:59 AM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Alex wrote:
> > > On Tue, Dec 21, 2021 at 6:09 PM Yann Dirson <ydirson@free.fr>
> > > wrote:
> > > >
> > > >
> > > >
> > > > ----- Mail original -----
> > > > > De: "Alex Deucher" <alexdeucher@gmail.com>
> > > > > À: "Yann Dirson" <ydirson@free.fr>
> > > > > Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>,
> > > > > "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> > > > > Envoyé: Mardi 21 Décembre 2021 23:31:01
> > > > > Objet: Re: Various problems trying to vga-passthrough a
> > > > > Renoir
> > > > > iGPU to a xen/qubes-os hvm
> > > > >
> > > > > On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr>
> > > > > wrote:
> > > > > >
> > > > > >
> > > > > > Alex wrote:
> > > > > > >
> > > > > > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson
> > > > > > > <ydirson@free.fr>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Christian wrote:
> > > > > > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > > > > > Alex wrote:
> > > > > > > > > >> Thinking about this more, I think the problem
> > > > > > > > > >> might be
> > > > > > > > > >> related
> > > > > > > > > >> to
> > > > > > > > > >> CPU
> > > > > > > > > >> access to "VRAM".  APUs don't have dedicated VRAM,
> > > > > > > > > >> they
> > > > > > > > > >> use a
> > > > > > > > > >> reserved
> > > > > > > > > >> carve out region at the top of system memory.  For
> > > > > > > > > >> CPU
> > > > > > > > > >> access
> > > > > > > > > >> to
> > > > > > > > > >> this
> > > > > > > > > >> memory, we kmap the physical address of the carve
> > > > > > > > > >> out
> > > > > > > > > >> region
> > > > > > > > > >> of
> > > > > > > > > >> system
> > > > > > > > > >> memory.  You'll need to make sure that region is
> > > > > > > > > >> accessible to
> > > > > > > > > >> the
> > > > > > > > > >> guest.
> > > > > > > > > > So basically, the non-virt flow is is: (video?)
> > > > > > > > > > BIOS
> > > > > > > > > > reserves
> > > > > > > > > > memory, marks it
> > > > > > > > > > as reserved in e820, stores the physaddr somewhere,
> > > > > > > > > > which
> > > > > > > > > > the
> > > > > > > > > > GPU
> > > > > > > > > > driver gets.
> > > > > > > > > > Since I suppose this includes the framebuffer, this
> > > > > > > > > > probably
> > > > > > > > > > has to
> > > > > > > > > > occur around
> > > > > > > > > > the moment the driver calls
> > > > > > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > > > > > (which happens before this hw init step), right ?
> > > > > > > > >
> > > > > > > > > Well, that partially correct. The efifb is using the
> > > > > > > > > PCIe
> > > > > > > > > resources
> > > > > > > > > to
> > > > > > > > > access the framebuffer and as far as I know we use
> > > > > > > > > that
> > > > > > > > > one
> > > > > > > > > to
> > > > > > > > > kick
> > > > > > > > > it out.
> > > > > > > > >
> > > > > > > > > The stolen memory we get over e820/registers is
> > > > > > > > > separate
> > > > > > > > > to
> > > > > > > > > that.
> > > > > >
> > > > > > How is the stolen memory communicated to the driver ?  That
> > > > > > host
> > > > > > physical
> > > > > > memory probably has to be mapped at the same guest physical
> > > > > > address
> > > > > > for
> > > > > > the magic to work, right ?
> > > > >
> > > > > Correct.  The driver reads the physical location of that
> > > > > memory
> > > > > from
> > > > > hardware registers.  Removing this chunk of code from
> > > > > gmc_v9_0.c
> > > > > will
> > > > > force the driver to use the BAR, but I'm not sure if there
> > > > > are
> > > > > any
> > > > > other places in the driver that make assumptions about using
> > > > > the
> > > > > physical host address or not on APUs off hand.
> > > > >
> > > > >         if ((adev->flags & AMD_IS_APU) ||
> > > > >             (adev->gmc.xgmi.supported &&
> > > > >              adev->gmc.xgmi.connected_to_cpu)) {
> > > > >                 adev->gmc.aper_base =
> > > > >                         adev->gfxhub.funcs->get_mc_fb_offset(adev)
> > > > >                         +
> > > > >                         adev->gmc.xgmi.physical_node_id *
> > > > >                         adev->gmc.xgmi.node_segment_size;
> > > > >                 adev->gmc.aper_size =
> > > > >                 adev->gmc.real_vram_size;
> > > > >         }
> > > > >
> > > > >
> > > > >
> > > > > >
> > > > > > > > >
> > > > > > > > > > ... which brings me to a point that's been puzzling
> > > > > > > > > > me
> > > > > > > > > > for
> > > > > > > > > > some
> > > > > > > > > > time, which is
> > > > > > > > > > that as the hw init fails, the efifb driver is
> > > > > > > > > > still
> > > > > > > > > > using
> > > > > > > > > > the
> > > > > > > > > > framebuffer.
> > > > > > > > >
> > > > > > > > > No, it isn't. You are probably just still seeing the
> > > > > > > > > same
> > > > > > > > > screen.
> > > > > > > > >
> > > > > > > > > The issue is most likely that while efi was kicked
> > > > > > > > > out
> > > > > > > > > nobody
> > > > > > > > > re-programmed the display hardware to show something
> > > > > > > > > different.
> > > > > > > > >
> > > > > > > > > > Am I right in suspecting that efifb should get
> > > > > > > > > > stripped
> > > > > > > > > > of
> > > > > > > > > > its
> > > > > > > > > > ownership of the
> > > > > > > > > > fb aperture first, and that if I don't get a black
> > > > > > > > > > screen
> > > > > > > > > > on
> > > > > > > > > > hw_init failure
> > > > > > > > > > that issue should be the first focus point ?
> > > > > > > > >
> > > > > > > > > You assumption with the black screen is incorrect.
> > > > > > > > > Since
> > > > > > > > > the
> > > > > > > > > hardware
> > > > > > > > > works independent even if you kick out efi you still
> > > > > > > > > have
> > > > > > > > > the
> > > > > > > > > same
> > > > > > > > > screen content, you just can't update it anymore.
> > > > > > > >
> > > > > > > > It's not only that the screen keeps its contents, it's
> > > > > > > > that
> > > > > > > > the
> > > > > > > > dom0
> > > > > > > > happily continues updating it.
> > > > > > >
> > > > > > > If the hypevisor is using efifb, then yes that could be a
> > > > > > > problem
> > > > > > > as
> > > > > > > the hypervisor could be writing to the efifb resources
> > > > > > > which
> > > > > > > ends
> > > > > > > up
> > > > > > > writing to the same physical memory.  That applies to any
> > > > > > > GPU
> > > > > > > on
> > > > > > > a
> > > > > > > UEFI system.  You'll need to make sure efifb is not in
> > > > > > > use in
> > > > > > > the
> > > > > > > hypervisor.
> > > > > >
> > > > > > That remark evokes several things to me.  First one is that
> > > > > > every
> > > > > > time
> > > > > > I've tried booting with efifb disabled in dom0, there was
> > > > > > no
> > > > > > visible
> > > > > > improvements in the guest driver - i.i. I really have to
> > > > > > dig
> > > > > > how
> > > > > > vram mapping
> > > > > > is performed and check things are as expected anyway.
> > > > >
> > > > > Ultimately you end up at the same physical memory.  efifb
> > > > > uses
> > > > > the
> > > > > PCI
> > > > > BAR which points to the same physical memory that the driver
> > > > > directly
> > > > > maps.
> > > > >
> > > > > >
> > > > > > The other is that, when dom0 cannot use efifb, entering a
> > > > > > luks
> > > > > > key
> > > > > > is
> > > > > > suddenly less user-friendly.  But in theory I'd think we
> > > > > > could
> > > > > > overcome
> > > > > > this by letting dom0 use efifb until ready to start the
> > > > > > guest,
> > > > > > a
> > > > > > simple
> > > > > > driver unbind at the right moment should be expected to
> > > > > > work,
> > > > > > right
> > > > > > ?
> > > > > > Going further and allowing the guest to use efifb on its
> > > > > > own
> > > > > > could
> > > > > > possibly be more tricky (starting with a different state?)
> > > > > > but
> > > > > > does
> > > > > > not seem to sound completely outlandish either - or does it
> > > > > > ?
> > > > > >
> > > > >
> > > > > efifb just takes whatever hardware state the GOP driver in
> > > > > the
> > > > > pre-OS
> > > > > environment left the GPU in.  Once you have a driver loaded
> > > > > in
> > > > > the
> > > > > OS,
> > > > > that state is gone so I I don't see much value in using efifb
> > > > > once
> > > > > you
> > > > > have a real driver in the mix.  If you want a console on the
> > > > > host,
> > > > > it's probably better to use 2 GPU or just load the real
> > > > > driver as
> > > > > needed in both the host and guest.
> > > >
> > > > My point is essentially making sure we always have a usable
> > > > display.
> > > > Dom0 will essentially use it for entering luks key (but having
> > > > boot
> > > > logs
> > > > visible may also be useful for some troubleshooting), and I was
> > > > figuring
> > > > out that once dom0 has relinquished it to start a guest with
> > > > passthrough,
> > > > the guest might as well make use of it before it can start
> > > > amdgpu.
> > > > Could be useful as a first step of seeing the GPU in the guest,
> > > > in
> > > > case there are more issues to be solved to be able to start
> > > > amdgpu.
> > >
> > >
> > > Something has to program the GPU to get the displays up.  You
> > > need a
> > > driver for that.  Once the OS has loaded the UEFI GOP driver is
> > > gone.
> >
> > My understanding is that efifb is only making use of the
> > framebuffer set up
> > by the GOP.  If that's correct I don't see why the same framebuffer
> > could
> > not be used by a guest once the host releases it (provided the
> > guest boots
> > Tiano Core and we can tell the latter to pass the GOP framebuffer
> > through).
> > Where do I get it wrong ?
> 
> Sure, you could re-use the GPU memory that backs the framebuffer, but
> something needs to reprogram the display timing so that you get a
> signal to your monitor.  That is what the driver does (GOP or OS
> driver).

Yes, GOP programs timings early, so host and early guest can use efifb, then
at some time the guest OS driver takes over the passed-through device and
at that point we forget about efifb.  That seems to make sense ?

> 
> Alex
> 
> >
> > > > > > >
> > > > > > > >
> > > > > > > > > But putting efi asside what Alex pointed out pretty
> > > > > > > > > much
> > > > > > > > > breaks
> > > > > > > > > your
> > > > > > > > > neck trying to forward the device. You maybe could
> > > > > > > > > try to
> > > > > > > > > hack
> > > > > > > > > the
> > > > > > > > > driver to use the PCIe BAR for framebuffer access,
> > > > > > > > > but
> > > > > > > > > that
> > > > > > > > > might
> > > > > > > > > be
> > > > > > > > > quite a bit slower.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Christian.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >> Alex
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > > > > >> <alexdeucher@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > > > > >>> <ydirson@free.fr>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>> Alex wrote:
> > > > > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > > > > >>>>> <ydirson@free.fr>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>> Hi Alex,
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > > > > >>>>>>> integrated
> > > > > > > > > >>>>>>> GPUs.  I
> > > > > > > > > >>>>>>> don't
> > > > > > > > > >>>>>>> know that it will work at all.  We had done a
> > > > > > > > > >>>>>>> bit
> > > > > > > > > >>>>>>> of
> > > > > > > > > >>>>>>> testing but
> > > > > > > > > >>>>>>> ran
> > > > > > > > > >>>>>>> into the same issues with the PSP, but never
> > > > > > > > > >>>>>>> had
> > > > > > > > > >>>>>>> a
> > > > > > > > > >>>>>>> chance
> > > > > > > > > >>>>>>> to
> > > > > > > > > >>>>>>> debug
> > > > > > > > > >>>>>>> further because this feature is not
> > > > > > > > > >>>>>>> productized.
> > > > > > > > > >>>>>> ...
> > > > > > > > > >>>>>>> You need a functional PSP to get the GPU
> > > > > > > > > >>>>>>> driver
> > > > > > > > > >>>>>>> up
> > > > > > > > > >>>>>>> and
> > > > > > > > > >>>>>>> running.
> > > > > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> I guess that if I want to have any chance to
> > > > > > > > > >>>>>> get
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>> PSP
> > > > > > > > > >>>>>> working
> > > > > > > > > >>>>>> I'm
> > > > > > > > > >>>>>> going to need more details on it.  A quick
> > > > > > > > > >>>>>> search
> > > > > > > > > >>>>>> some
> > > > > > > > > >>>>>> time
> > > > > > > > > >>>>>> ago
> > > > > > > > > >>>>>> mostly
> > > > > > > > > >>>>>> brought reverse-engineering work, rather than
> > > > > > > > > >>>>>> official
> > > > > > > > > >>>>>> AMD
> > > > > > > > > >>>>>> doc.
> > > > > > > > > >>>>>>   Are
> > > > > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > > > > >>>>> The driver code is pretty much it.
> > > > > > > > > >>>> Let's try to shed some more light on how things
> > > > > > > > > >>>> work,
> > > > > > > > > >>>> taking
> > > > > > > > > >>>> as
> > > > > > > > > >>>> excuse
> > > > > > > > > >>>> psp_v12_0_ring_create().
> > > > > > > > > >>>>
> > > > > > > > > >>>> First, register access through [RW]REG32_SOC15()
> > > > > > > > > >>>> is
> > > > > > > > > >>>> implemented
> > > > > > > > > >>>> in
> > > > > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is
> > > > > > > > > >>>> basically a
> > > > > > > > > >>>> [RW]REG32(),
> > > > > > > > > >>>> except it has to be more complex in the SR-IOV
> > > > > > > > > >>>> case.
> > > > > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > > > > >>> When running the driver on a SR-IOV virtual
> > > > > > > > > >>> function
> > > > > > > > > >>> (VF),
> > > > > > > > > >>> some
> > > > > > > > > >>> registers are not available directly via the VF's
> > > > > > > > > >>> MMIO
> > > > > > > > > >>> aperture
> > > > > > > > > >>> so
> > > > > > > > > >>> they need to go through the RLC.  For bare metal
> > > > > > > > > >>> or
> > > > > > > > > >>> passthrough
> > > > > > > > > >>> this
> > > > > > > > > >>> is not relevant.
> > > > > > > > > >>>
> > > > > > > > > >>>> It accesses registers in the MMIO range of the
> > > > > > > > > >>>> MP0
> > > > > > > > > >>>> IP,
> > > > > > > > > >>>> and
> > > > > > > > > >>>> the
> > > > > > > > > >>>> "MP0"
> > > > > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > > > > >>>> PSP-handling
> > > > > > > > > >>>> code.
> > > > > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for
> > > > > > > > > >>>> SMU) ?
> > > > > > > > > >>>>  The
> > > > > > > > > >>>> MP0
> > > > > > > > > >>>> version
> > > > > > > > > >>> Yes.
> > > > > > > > > >>>
> > > > > > > > > >>>> reported at v11.0.3 by discovery seems to
> > > > > > > > > >>>> contradict
> > > > > > > > > >>>> the
> > > > > > > > > >>>> use
> > > > > > > > > >>>> of
> > > > > > > > > >>>> v12.0
> > > > > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or
> > > > > > > > > >>>> do I
> > > > > > > > > >>>> miss
> > > > > > > > > >>>> something ?
> > > > > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > > > > >>>
> > > > > > > > > >>>> More generally (and mostly out of curiosity
> > > > > > > > > >>>> while
> > > > > > > > > >>>> we're
> > > > > > > > > >>>> at
> > > > > > > > > >>>> it),
> > > > > > > > > >>>> do we
> > > > > > > > > >>>> have a way to match IPs listed at discovery time
> > > > > > > > > >>>> with
> > > > > > > > > >>>> the
> > > > > > > > > >>>> ones
> > > > > > > > > >>>> used
> > > > > > > > > >>>> in the driver ?
> > > > > > > > > >>> In general, barring typos, the code is shared at
> > > > > > > > > >>> the
> > > > > > > > > >>> major
> > > > > > > > > >>> version
> > > > > > > > > >>> level.  The actual code may or may not need
> > > > > > > > > >>> changes
> > > > > > > > > >>> to
> > > > > > > > > >>> handle
> > > > > > > > > >>> minor
> > > > > > > > > >>> revision changes in an IP.  The driver maps the
> > > > > > > > > >>> IP
> > > > > > > > > >>> versions
> > > > > > > > > >>> from
> > > > > > > > > >>> the
> > > > > > > > > >>> ip discovery table to the code contained in the
> > > > > > > > > >>> driver.
> > > > > > > > > >>>
> > > > > > > > > >>>> ---
> > > > > > > > > >>>>
> > > > > > > > > >>>> As for the register names, maybe we could have a
> > > > > > > > > >>>> short
> > > > > > > > > >>>> explanation of
> > > > > > > > > >>>> how they are structured ?  Eg.
> > > > > > > > > >>>> mmMP0_SMN_C2PMSG_69:
> > > > > > > > > >>>> that
> > > > > > > > > >>>> seems
> > > > > > > > > >>>> to
> > > > > > > > > >>>> be
> > > > > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0"
> > > > > > > > > >>>> IP,
> > > > > > > > > >>>> but
> > > > > > > > > >>>> I'm
> > > > > > > > > >>>> not
> > > > > > > > > >>>> sure
> > > > > > > > > >>>> of the "SMN" part -- that could refer to the
> > > > > > > > > >>>> "System
> > > > > > > > > >>>> Management
> > > > > > > > > >>>> Network",
> > > > > > > > > >>>> described in [0] as an internal bus.  Are we
> > > > > > > > > >>>> accessing
> > > > > > > > > >>>> this
> > > > > > > > > >>>> register
> > > > > > > > > >>>> through this SMN ?
> > > > > > > > > >>> These registers are just mailboxes for the PSP
> > > > > > > > > >>> firmware.
> > > > > > > > > >>>  All
> > > > > > > > > >>> of
> > > > > > > > > >>> the
> > > > > > > > > >>> C2PMSG registers functionality is defined by the
> > > > > > > > > >>> PSP
> > > > > > > > > >>> firmware.
> > > > > > > > > >>>   They
> > > > > > > > > >>> are basically scratch registers used to
> > > > > > > > > >>> communicate
> > > > > > > > > >>> between
> > > > > > > > > >>> the
> > > > > > > > > >>> driver
> > > > > > > > > >>> and the PSP firmware.
> > > > > > > > > >>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > > > > >>>>> the CPU and the rest of the platform.  The GPU
> > > > > > > > > >>>>> driver
> > > > > > > > > >>>>> just
> > > > > > > > > >>>>> interacts
> > > > > > > > > >>>>> with it for a few specific tasks:
> > > > > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted
> > > > > > > > > >>>>> firmware
> > > > > > > > > >>>>> applications
> > > > > > > > > >>>>> that run on the PSP for specific functionality,
> > > > > > > > > >>>>> e.g.,
> > > > > > > > > >>>>> HDCP
> > > > > > > > > >>>>> and
> > > > > > > > > >>>>> content
> > > > > > > > > >>>>> protection, etc.)
> > > > > > > > > >>>>> 2. Validating and loading firmware for other
> > > > > > > > > >>>>> engines on
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> SoC.
> > > > > > > > > >>>>>   This
> > > > > > > > > >>>>> is required to use those engines.
> > > > > > > > > >>>> Trying to understand in more details how we
> > > > > > > > > >>>> start
> > > > > > > > > >>>> the
> > > > > > > > > >>>> PSP
> > > > > > > > > >>>> up, I
> > > > > > > > > >>>> noticed
> > > > > > > > > >>>> that psp_v12_0 has support for loading a sOS
> > > > > > > > > >>>> firmware,
> > > > > > > > > >>>> but
> > > > > > > > > >>>> never
> > > > > > > > > >>>> calls
> > > > > > > > > >>>> init_sos_microcode() - and anyway there is no
> > > > > > > > > >>>> sos
> > > > > > > > > >>>> firmware
> > > > > > > > > >>>> for
> > > > > > > > > >>>> renoir
> > > > > > > > > >>>> and green_sardine, which seem to be the only
> > > > > > > > > >>>> ASICs
> > > > > > > > > >>>> with
> > > > > > > > > >>>> this
> > > > > > > > > >>>> PSP
> > > > > > > > > >>>> version.
> > > > > > > > > >>>> Is it something that's just not been completely
> > > > > > > > > >>>> wired up
> > > > > > > > > >>>> yet
> > > > > > > > > >>>> ?
> > > > > > > > > >>> On APUs, the PSP is shared with the CPU so the
> > > > > > > > > >>> PSP
> > > > > > > > > >>> firmware
> > > > > > > > > >>> is
> > > > > > > > > >>> part
> > > > > > > > > >>> of
> > > > > > > > > >>> the sbios image.  The driver doesn't load it.  We
> > > > > > > > > >>> only
> > > > > > > > > >>> load
> > > > > > > > > >>> it on
> > > > > > > > > >>> dGPUs where the driver is responsible for the
> > > > > > > > > >>> chip
> > > > > > > > > >>> initialization.
> > > > > > > > > >>>
> > > > > > > > > >>>> That also rings a bell, that we have nothing
> > > > > > > > > >>>> about
> > > > > > > > > >>>> Secure OS
> > > > > > > > > >>>> in
> > > > > > > > > >>>> the doc
> > > > > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>> I'm not too familiar with the PSP's path to
> > > > > > > > > >>>>> memory
> > > > > > > > > >>>>> from
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> GPU
> > > > > > > > > >>>>> perspective.  IIRC, most memory used by the PSP
> > > > > > > > > >>>>> goes
> > > > > > > > > >>>>> through
> > > > > > > > > >>>>> carve
> > > > > > > > > >>>>> out
> > > > > > > > > >>>>> "vram" on APUs so it should work, but I would
> > > > > > > > > >>>>> double
> > > > > > > > > >>>>> check
> > > > > > > > > >>>>> if
> > > > > > > > > >>>>> there
> > > > > > > > > >>>>> are any system memory allocations that used to
> > > > > > > > > >>>>> interact
> > > > > > > > > >>>>> with
> > > > > > > > > >>>>> the PSP
> > > > > > > > > >>>>> and see if changing them to vram helps.  It
> > > > > > > > > >>>>> does
> > > > > > > > > >>>>> work
> > > > > > > > > >>>>> with
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> IOMMU
> > > > > > > > > >>>>> enabled on bare metal, so it should work in
> > > > > > > > > >>>>> passthrough
> > > > > > > > > >>>>> as
> > > > > > > > > >>>>> well
> > > > > > > > > >>>>> in
> > > > > > > > > >>>>> theory.
> > > > > > > > > >>>> I can see a single case in the PSP code where
> > > > > > > > > >>>> GTT is
> > > > > > > > > >>>> used
> > > > > > > > > >>>> instead
> > > > > > > > > >>>> of
> > > > > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not
> > > > > > > > > >>>> used
> > > > > > > > > >>>> (and
> > > > > > > > > >>>> there
> > > > > > > > > >>>> has
> > > > > > > > > >>>> to be a reason, since the SR-IOV code path does
> > > > > > > > > >>>> use
> > > > > > > > > >>>> vram).
> > > > > > > > > >>>> Changing it to vram does not make a difference,
> > > > > > > > > >>>> but
> > > > > > > > > >>>> then
> > > > > > > > > >>>> the
> > > > > > > > > >>>> only bo that seems to be used at that point is
> > > > > > > > > >>>> the
> > > > > > > > > >>>> one
> > > > > > > > > >>>> for
> > > > > > > > > >>>> the
> > > > > > > > > >>>> psp ring,
> > > > > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > > > > >>>> surprised.
> > > > > > > > > >>>>
> > > > > > > > > >>>> Maybe I should double-check bo_create calls to
> > > > > > > > > >>>> hunt
> > > > > > > > > >>>> for
> > > > > > > > > >>>> more
> > > > > > > > > >>>> ?
> > > > > > > > > >>> We looked into this a bit ourselves and ran into
> > > > > > > > > >>> the
> > > > > > > > > >>> same
> > > > > > > > > >>> issues.
> > > > > > > > > >>> We'd probably need to debug this with the PSP
> > > > > > > > > >>> team to
> > > > > > > > > >>> make
> > > > > > > > > >>> further
> > > > > > > > > >>> progress, but this was not productized so neither
> > > > > > > > > >>> team
> > > > > > > > > >>> had
> > > > > > > > > >>> the
> > > > > > > > > >>> resources to delve further.
> > > > > > > > > >>>
> > > > > > > > > >>> Alex
> > > > > > > > > >>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> [0]
> > > > > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Wed, Dec 29, 2021 at 12:34 PM Yann Dirson <ydirson@free.fr> wrote:
>
> Alex wrote:
> > On Wed, Dec 29, 2021 at 11:59 AM Yann Dirson <ydirson@free.fr> wrote:
> > >
> > > Alex wrote:
> > > > On Tue, Dec 21, 2021 at 6:09 PM Yann Dirson <ydirson@free.fr>
> > > > wrote:
> > > > >
> > > > >
> > > > >
> > > > > ----- Mail original -----
> > > > > > De: "Alex Deucher" <alexdeucher@gmail.com>
> > > > > > À: "Yann Dirson" <ydirson@free.fr>
> > > > > > Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>,
> > > > > > "amd-gfx list" <amd-gfx@lists.freedesktop.org>
> > > > > > Envoyé: Mardi 21 Décembre 2021 23:31:01
> > > > > > Objet: Re: Various problems trying to vga-passthrough a
> > > > > > Renoir
> > > > > > iGPU to a xen/qubes-os hvm
> > > > > >
> > > > > > On Tue, Dec 21, 2021 at 5:12 PM Yann Dirson <ydirson@free.fr>
> > > > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > > Alex wrote:
> > > > > > > >
> > > > > > > > On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson
> > > > > > > > <ydirson@free.fr>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Christian wrote:
> > > > > > > > > > Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > > > > > > > > > > Alex wrote:
> > > > > > > > > > >> Thinking about this more, I think the problem
> > > > > > > > > > >> might be
> > > > > > > > > > >> related
> > > > > > > > > > >> to
> > > > > > > > > > >> CPU
> > > > > > > > > > >> access to "VRAM".  APUs don't have dedicated VRAM,
> > > > > > > > > > >> they
> > > > > > > > > > >> use a
> > > > > > > > > > >> reserved
> > > > > > > > > > >> carve out region at the top of system memory.  For
> > > > > > > > > > >> CPU
> > > > > > > > > > >> access
> > > > > > > > > > >> to
> > > > > > > > > > >> this
> > > > > > > > > > >> memory, we kmap the physical address of the carve
> > > > > > > > > > >> out
> > > > > > > > > > >> region
> > > > > > > > > > >> of
> > > > > > > > > > >> system
> > > > > > > > > > >> memory.  You'll need to make sure that region is
> > > > > > > > > > >> accessible to
> > > > > > > > > > >> the
> > > > > > > > > > >> guest.
> > > > > > > > > > > So basically, the non-virt flow is is: (video?)
> > > > > > > > > > > BIOS
> > > > > > > > > > > reserves
> > > > > > > > > > > memory, marks it
> > > > > > > > > > > as reserved in e820, stores the physaddr somewhere,
> > > > > > > > > > > which
> > > > > > > > > > > the
> > > > > > > > > > > GPU
> > > > > > > > > > > driver gets.
> > > > > > > > > > > Since I suppose this includes the framebuffer, this
> > > > > > > > > > > probably
> > > > > > > > > > > has to
> > > > > > > > > > > occur around
> > > > > > > > > > > the moment the driver calls
> > > > > > > > > > > drm_aperture_remove_conflicting_pci_framebuffers()
> > > > > > > > > > > (which happens before this hw init step), right ?
> > > > > > > > > >
> > > > > > > > > > Well, that partially correct. The efifb is using the
> > > > > > > > > > PCIe
> > > > > > > > > > resources
> > > > > > > > > > to
> > > > > > > > > > access the framebuffer and as far as I know we use
> > > > > > > > > > that
> > > > > > > > > > one
> > > > > > > > > > to
> > > > > > > > > > kick
> > > > > > > > > > it out.
> > > > > > > > > >
> > > > > > > > > > The stolen memory we get over e820/registers is
> > > > > > > > > > separate
> > > > > > > > > > to
> > > > > > > > > > that.
> > > > > > >
> > > > > > > How is the stolen memory communicated to the driver ?  That
> > > > > > > host
> > > > > > > physical
> > > > > > > memory probably has to be mapped at the same guest physical
> > > > > > > address
> > > > > > > for
> > > > > > > the magic to work, right ?
> > > > > >
> > > > > > Correct.  The driver reads the physical location of that
> > > > > > memory
> > > > > > from
> > > > > > hardware registers.  Removing this chunk of code from
> > > > > > gmc_v9_0.c
> > > > > > will
> > > > > > force the driver to use the BAR, but I'm not sure if there
> > > > > > are
> > > > > > any
> > > > > > other places in the driver that make assumptions about using
> > > > > > the
> > > > > > physical host address or not on APUs off hand.
> > > > > >
> > > > > >         if ((adev->flags & AMD_IS_APU) ||
> > > > > >             (adev->gmc.xgmi.supported &&
> > > > > >              adev->gmc.xgmi.connected_to_cpu)) {
> > > > > >                 adev->gmc.aper_base =
> > > > > >                         adev->gfxhub.funcs->get_mc_fb_offset(adev)
> > > > > >                         +
> > > > > >                         adev->gmc.xgmi.physical_node_id *
> > > > > >                         adev->gmc.xgmi.node_segment_size;
> > > > > >                 adev->gmc.aper_size =
> > > > > >                 adev->gmc.real_vram_size;
> > > > > >         }
> > > > > >
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > > > >
> > > > > > > > > > > ... which brings me to a point that's been puzzling
> > > > > > > > > > > me
> > > > > > > > > > > for
> > > > > > > > > > > some
> > > > > > > > > > > time, which is
> > > > > > > > > > > that as the hw init fails, the efifb driver is
> > > > > > > > > > > still
> > > > > > > > > > > using
> > > > > > > > > > > the
> > > > > > > > > > > framebuffer.
> > > > > > > > > >
> > > > > > > > > > No, it isn't. You are probably just still seeing the
> > > > > > > > > > same
> > > > > > > > > > screen.
> > > > > > > > > >
> > > > > > > > > > The issue is most likely that while efi was kicked
> > > > > > > > > > out
> > > > > > > > > > nobody
> > > > > > > > > > re-programmed the display hardware to show something
> > > > > > > > > > different.
> > > > > > > > > >
> > > > > > > > > > > Am I right in suspecting that efifb should get
> > > > > > > > > > > stripped
> > > > > > > > > > > of
> > > > > > > > > > > its
> > > > > > > > > > > ownership of the
> > > > > > > > > > > fb aperture first, and that if I don't get a black
> > > > > > > > > > > screen
> > > > > > > > > > > on
> > > > > > > > > > > hw_init failure
> > > > > > > > > > > that issue should be the first focus point ?
> > > > > > > > > >
> > > > > > > > > > You assumption with the black screen is incorrect.
> > > > > > > > > > Since
> > > > > > > > > > the
> > > > > > > > > > hardware
> > > > > > > > > > works independent even if you kick out efi you still
> > > > > > > > > > have
> > > > > > > > > > the
> > > > > > > > > > same
> > > > > > > > > > screen content, you just can't update it anymore.
> > > > > > > > >
> > > > > > > > > It's not only that the screen keeps its contents, it's
> > > > > > > > > that
> > > > > > > > > the
> > > > > > > > > dom0
> > > > > > > > > happily continues updating it.
> > > > > > > >
> > > > > > > > If the hypevisor is using efifb, then yes that could be a
> > > > > > > > problem
> > > > > > > > as
> > > > > > > > the hypervisor could be writing to the efifb resources
> > > > > > > > which
> > > > > > > > ends
> > > > > > > > up
> > > > > > > > writing to the same physical memory.  That applies to any
> > > > > > > > GPU
> > > > > > > > on
> > > > > > > > a
> > > > > > > > UEFI system.  You'll need to make sure efifb is not in
> > > > > > > > use in
> > > > > > > > the
> > > > > > > > hypervisor.
> > > > > > >
> > > > > > > That remark evokes several things to me.  First one is that
> > > > > > > every
> > > > > > > time
> > > > > > > I've tried booting with efifb disabled in dom0, there was
> > > > > > > no
> > > > > > > visible
> > > > > > > improvements in the guest driver - i.i. I really have to
> > > > > > > dig
> > > > > > > how
> > > > > > > vram mapping
> > > > > > > is performed and check things are as expected anyway.
> > > > > >
> > > > > > Ultimately you end up at the same physical memory.  efifb
> > > > > > uses
> > > > > > the
> > > > > > PCI
> > > > > > BAR which points to the same physical memory that the driver
> > > > > > directly
> > > > > > maps.
> > > > > >
> > > > > > >
> > > > > > > The other is that, when dom0 cannot use efifb, entering a
> > > > > > > luks
> > > > > > > key
> > > > > > > is
> > > > > > > suddenly less user-friendly.  But in theory I'd think we
> > > > > > > could
> > > > > > > overcome
> > > > > > > this by letting dom0 use efifb until ready to start the
> > > > > > > guest,
> > > > > > > a
> > > > > > > simple
> > > > > > > driver unbind at the right moment should be expected to
> > > > > > > work,
> > > > > > > right
> > > > > > > ?
> > > > > > > Going further and allowing the guest to use efifb on its
> > > > > > > own
> > > > > > > could
> > > > > > > possibly be more tricky (starting with a different state?)
> > > > > > > but
> > > > > > > does
> > > > > > > not seem to sound completely outlandish either - or does it
> > > > > > > ?
> > > > > > >
> > > > > >
> > > > > > efifb just takes whatever hardware state the GOP driver in
> > > > > > the
> > > > > > pre-OS
> > > > > > environment left the GPU in.  Once you have a driver loaded
> > > > > > in
> > > > > > the
> > > > > > OS,
> > > > > > that state is gone so I I don't see much value in using efifb
> > > > > > once
> > > > > > you
> > > > > > have a real driver in the mix.  If you want a console on the
> > > > > > host,
> > > > > > it's probably better to use 2 GPU or just load the real
> > > > > > driver as
> > > > > > needed in both the host and guest.
> > > > >
> > > > > My point is essentially making sure we always have a usable
> > > > > display.
> > > > > Dom0 will essentially use it for entering luks key (but having
> > > > > boot
> > > > > logs
> > > > > visible may also be useful for some troubleshooting), and I was
> > > > > figuring
> > > > > out that once dom0 has relinquished it to start a guest with
> > > > > passthrough,
> > > > > the guest might as well make use of it before it can start
> > > > > amdgpu.
> > > > > Could be useful as a first step of seeing the GPU in the guest,
> > > > > in
> > > > > case there are more issues to be solved to be able to start
> > > > > amdgpu.
> > > >
> > > >
> > > > Something has to program the GPU to get the displays up.  You
> > > > need a
> > > > driver for that.  Once the OS has loaded the UEFI GOP driver is
> > > > gone.
> > >
> > > My understanding is that efifb is only making use of the
> > > framebuffer set up
> > > by the GOP.  If that's correct I don't see why the same framebuffer
> > > could
> > > not be used by a guest once the host releases it (provided the
> > > guest boots
> > > Tiano Core and we can tell the latter to pass the GOP framebuffer
> > > through).
> > > Where do I get it wrong ?
> >
> > Sure, you could re-use the GPU memory that backs the framebuffer, but
> > something needs to reprogram the display timing so that you get a
> > signal to your monitor.  That is what the driver does (GOP or OS
> > driver).
>
> Yes, GOP programs timings early, so host and early guest can use efifb, then
> at some time the guest OS driver takes over the passed-through device and
> at that point we forget about efifb.  That seems to make sense ?

Yes.  on boot up, the GOP driver loads and initializes the hardware,
enumerates the attached displays, and lights them up.  Once the OS
loads, the pre-OS environment is gone, so the GOP driver is gone.  The
OS just inherits the memory and timings that the GOP driver set up.
Then the OS loads the OS driver which enumerates the displays and
lights them up, etc.  When you unload the OS driver and you want to
use the hardware again, something has to re-initialize it.

Alex


>
> >
> > Alex
> >
> > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > > But putting efi asside what Alex pointed out pretty
> > > > > > > > > > much
> > > > > > > > > > breaks
> > > > > > > > > > your
> > > > > > > > > > neck trying to forward the device. You maybe could
> > > > > > > > > > try to
> > > > > > > > > > hack
> > > > > > > > > > the
> > > > > > > > > > driver to use the PCIe BAR for framebuffer access,
> > > > > > > > > > but
> > > > > > > > > > that
> > > > > > > > > > might
> > > > > > > > > > be
> > > > > > > > > > quite a bit slower.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Christian.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >> Alex
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > > > > > >> <alexdeucher@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > > > > > >>> <ydirson@free.fr>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>> Alex wrote:
> > > > > > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > > > > > >>>>> <ydirson@free.fr>
> > > > > > > > > > >>>>> wrote:
> > > > > > > > > > >>>>>> Hi Alex,
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > > > > > >>>>>>> integrated
> > > > > > > > > > >>>>>>> GPUs.  I
> > > > > > > > > > >>>>>>> don't
> > > > > > > > > > >>>>>>> know that it will work at all.  We had done a
> > > > > > > > > > >>>>>>> bit
> > > > > > > > > > >>>>>>> of
> > > > > > > > > > >>>>>>> testing but
> > > > > > > > > > >>>>>>> ran
> > > > > > > > > > >>>>>>> into the same issues with the PSP, but never
> > > > > > > > > > >>>>>>> had
> > > > > > > > > > >>>>>>> a
> > > > > > > > > > >>>>>>> chance
> > > > > > > > > > >>>>>>> to
> > > > > > > > > > >>>>>>> debug
> > > > > > > > > > >>>>>>> further because this feature is not
> > > > > > > > > > >>>>>>> productized.
> > > > > > > > > > >>>>>> ...
> > > > > > > > > > >>>>>>> You need a functional PSP to get the GPU
> > > > > > > > > > >>>>>>> driver
> > > > > > > > > > >>>>>>> up
> > > > > > > > > > >>>>>>> and
> > > > > > > > > > >>>>>>> running.
> > > > > > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> I guess that if I want to have any chance to
> > > > > > > > > > >>>>>> get
> > > > > > > > > > >>>>>> the
> > > > > > > > > > >>>>>> PSP
> > > > > > > > > > >>>>>> working
> > > > > > > > > > >>>>>> I'm
> > > > > > > > > > >>>>>> going to need more details on it.  A quick
> > > > > > > > > > >>>>>> search
> > > > > > > > > > >>>>>> some
> > > > > > > > > > >>>>>> time
> > > > > > > > > > >>>>>> ago
> > > > > > > > > > >>>>>> mostly
> > > > > > > > > > >>>>>> brought reverse-engineering work, rather than
> > > > > > > > > > >>>>>> official
> > > > > > > > > > >>>>>> AMD
> > > > > > > > > > >>>>>> doc.
> > > > > > > > > > >>>>>>   Are
> > > > > > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > > > > > >>>>> The driver code is pretty much it.
> > > > > > > > > > >>>> Let's try to shed some more light on how things
> > > > > > > > > > >>>> work,
> > > > > > > > > > >>>> taking
> > > > > > > > > > >>>> as
> > > > > > > > > > >>>> excuse
> > > > > > > > > > >>>> psp_v12_0_ring_create().
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> First, register access through [RW]REG32_SOC15()
> > > > > > > > > > >>>> is
> > > > > > > > > > >>>> implemented
> > > > > > > > > > >>>> in
> > > > > > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is
> > > > > > > > > > >>>> basically a
> > > > > > > > > > >>>> [RW]REG32(),
> > > > > > > > > > >>>> except it has to be more complex in the SR-IOV
> > > > > > > > > > >>>> case.
> > > > > > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > > > > > >>> When running the driver on a SR-IOV virtual
> > > > > > > > > > >>> function
> > > > > > > > > > >>> (VF),
> > > > > > > > > > >>> some
> > > > > > > > > > >>> registers are not available directly via the VF's
> > > > > > > > > > >>> MMIO
> > > > > > > > > > >>> aperture
> > > > > > > > > > >>> so
> > > > > > > > > > >>> they need to go through the RLC.  For bare metal
> > > > > > > > > > >>> or
> > > > > > > > > > >>> passthrough
> > > > > > > > > > >>> this
> > > > > > > > > > >>> is not relevant.
> > > > > > > > > > >>>
> > > > > > > > > > >>>> It accesses registers in the MMIO range of the
> > > > > > > > > > >>>> MP0
> > > > > > > > > > >>>> IP,
> > > > > > > > > > >>>> and
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> "MP0"
> > > > > > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > > > > > >>>> PSP-handling
> > > > > > > > > > >>>> code.
> > > > > > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for
> > > > > > > > > > >>>> SMU) ?
> > > > > > > > > > >>>>  The
> > > > > > > > > > >>>> MP0
> > > > > > > > > > >>>> version
> > > > > > > > > > >>> Yes.
> > > > > > > > > > >>>
> > > > > > > > > > >>>> reported at v11.0.3 by discovery seems to
> > > > > > > > > > >>>> contradict
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> use
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>> v12.0
> > > > > > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or
> > > > > > > > > > >>>> do I
> > > > > > > > > > >>>> miss
> > > > > > > > > > >>>> something ?
> > > > > > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > > > > > >>>
> > > > > > > > > > >>>> More generally (and mostly out of curiosity
> > > > > > > > > > >>>> while
> > > > > > > > > > >>>> we're
> > > > > > > > > > >>>> at
> > > > > > > > > > >>>> it),
> > > > > > > > > > >>>> do we
> > > > > > > > > > >>>> have a way to match IPs listed at discovery time
> > > > > > > > > > >>>> with
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> ones
> > > > > > > > > > >>>> used
> > > > > > > > > > >>>> in the driver ?
> > > > > > > > > > >>> In general, barring typos, the code is shared at
> > > > > > > > > > >>> the
> > > > > > > > > > >>> major
> > > > > > > > > > >>> version
> > > > > > > > > > >>> level.  The actual code may or may not need
> > > > > > > > > > >>> changes
> > > > > > > > > > >>> to
> > > > > > > > > > >>> handle
> > > > > > > > > > >>> minor
> > > > > > > > > > >>> revision changes in an IP.  The driver maps the
> > > > > > > > > > >>> IP
> > > > > > > > > > >>> versions
> > > > > > > > > > >>> from
> > > > > > > > > > >>> the
> > > > > > > > > > >>> ip discovery table to the code contained in the
> > > > > > > > > > >>> driver.
> > > > > > > > > > >>>
> > > > > > > > > > >>>> ---
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> As for the register names, maybe we could have a
> > > > > > > > > > >>>> short
> > > > > > > > > > >>>> explanation of
> > > > > > > > > > >>>> how they are structured ?  Eg.
> > > > > > > > > > >>>> mmMP0_SMN_C2PMSG_69:
> > > > > > > > > > >>>> that
> > > > > > > > > > >>>> seems
> > > > > > > > > > >>>> to
> > > > > > > > > > >>>> be
> > > > > > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0"
> > > > > > > > > > >>>> IP,
> > > > > > > > > > >>>> but
> > > > > > > > > > >>>> I'm
> > > > > > > > > > >>>> not
> > > > > > > > > > >>>> sure
> > > > > > > > > > >>>> of the "SMN" part -- that could refer to the
> > > > > > > > > > >>>> "System
> > > > > > > > > > >>>> Management
> > > > > > > > > > >>>> Network",
> > > > > > > > > > >>>> described in [0] as an internal bus.  Are we
> > > > > > > > > > >>>> accessing
> > > > > > > > > > >>>> this
> > > > > > > > > > >>>> register
> > > > > > > > > > >>>> through this SMN ?
> > > > > > > > > > >>> These registers are just mailboxes for the PSP
> > > > > > > > > > >>> firmware.
> > > > > > > > > > >>>  All
> > > > > > > > > > >>> of
> > > > > > > > > > >>> the
> > > > > > > > > > >>> C2PMSG registers functionality is defined by the
> > > > > > > > > > >>> PSP
> > > > > > > > > > >>> firmware.
> > > > > > > > > > >>>   They
> > > > > > > > > > >>> are basically scratch registers used to
> > > > > > > > > > >>> communicate
> > > > > > > > > > >>> between
> > > > > > > > > > >>> the
> > > > > > > > > > >>> driver
> > > > > > > > > > >>> and the PSP firmware.
> > > > > > > > > > >>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > > > > > >>>>> the CPU and the rest of the platform.  The GPU
> > > > > > > > > > >>>>> driver
> > > > > > > > > > >>>>> just
> > > > > > > > > > >>>>> interacts
> > > > > > > > > > >>>>> with it for a few specific tasks:
> > > > > > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted
> > > > > > > > > > >>>>> firmware
> > > > > > > > > > >>>>> applications
> > > > > > > > > > >>>>> that run on the PSP for specific functionality,
> > > > > > > > > > >>>>> e.g.,
> > > > > > > > > > >>>>> HDCP
> > > > > > > > > > >>>>> and
> > > > > > > > > > >>>>> content
> > > > > > > > > > >>>>> protection, etc.)
> > > > > > > > > > >>>>> 2. Validating and loading firmware for other
> > > > > > > > > > >>>>> engines on
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> SoC.
> > > > > > > > > > >>>>>   This
> > > > > > > > > > >>>>> is required to use those engines.
> > > > > > > > > > >>>> Trying to understand in more details how we
> > > > > > > > > > >>>> start
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> PSP
> > > > > > > > > > >>>> up, I
> > > > > > > > > > >>>> noticed
> > > > > > > > > > >>>> that psp_v12_0 has support for loading a sOS
> > > > > > > > > > >>>> firmware,
> > > > > > > > > > >>>> but
> > > > > > > > > > >>>> never
> > > > > > > > > > >>>> calls
> > > > > > > > > > >>>> init_sos_microcode() - and anyway there is no
> > > > > > > > > > >>>> sos
> > > > > > > > > > >>>> firmware
> > > > > > > > > > >>>> for
> > > > > > > > > > >>>> renoir
> > > > > > > > > > >>>> and green_sardine, which seem to be the only
> > > > > > > > > > >>>> ASICs
> > > > > > > > > > >>>> with
> > > > > > > > > > >>>> this
> > > > > > > > > > >>>> PSP
> > > > > > > > > > >>>> version.
> > > > > > > > > > >>>> Is it something that's just not been completely
> > > > > > > > > > >>>> wired up
> > > > > > > > > > >>>> yet
> > > > > > > > > > >>>> ?
> > > > > > > > > > >>> On APUs, the PSP is shared with the CPU so the
> > > > > > > > > > >>> PSP
> > > > > > > > > > >>> firmware
> > > > > > > > > > >>> is
> > > > > > > > > > >>> part
> > > > > > > > > > >>> of
> > > > > > > > > > >>> the sbios image.  The driver doesn't load it.  We
> > > > > > > > > > >>> only
> > > > > > > > > > >>> load
> > > > > > > > > > >>> it on
> > > > > > > > > > >>> dGPUs where the driver is responsible for the
> > > > > > > > > > >>> chip
> > > > > > > > > > >>> initialization.
> > > > > > > > > > >>>
> > > > > > > > > > >>>> That also rings a bell, that we have nothing
> > > > > > > > > > >>>> about
> > > > > > > > > > >>>> Secure OS
> > > > > > > > > > >>>> in
> > > > > > > > > > >>>> the doc
> > > > > > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> I'm not too familiar with the PSP's path to
> > > > > > > > > > >>>>> memory
> > > > > > > > > > >>>>> from
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> GPU
> > > > > > > > > > >>>>> perspective.  IIRC, most memory used by the PSP
> > > > > > > > > > >>>>> goes
> > > > > > > > > > >>>>> through
> > > > > > > > > > >>>>> carve
> > > > > > > > > > >>>>> out
> > > > > > > > > > >>>>> "vram" on APUs so it should work, but I would
> > > > > > > > > > >>>>> double
> > > > > > > > > > >>>>> check
> > > > > > > > > > >>>>> if
> > > > > > > > > > >>>>> there
> > > > > > > > > > >>>>> are any system memory allocations that used to
> > > > > > > > > > >>>>> interact
> > > > > > > > > > >>>>> with
> > > > > > > > > > >>>>> the PSP
> > > > > > > > > > >>>>> and see if changing them to vram helps.  It
> > > > > > > > > > >>>>> does
> > > > > > > > > > >>>>> work
> > > > > > > > > > >>>>> with
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> IOMMU
> > > > > > > > > > >>>>> enabled on bare metal, so it should work in
> > > > > > > > > > >>>>> passthrough
> > > > > > > > > > >>>>> as
> > > > > > > > > > >>>>> well
> > > > > > > > > > >>>>> in
> > > > > > > > > > >>>>> theory.
> > > > > > > > > > >>>> I can see a single case in the PSP code where
> > > > > > > > > > >>>> GTT is
> > > > > > > > > > >>>> used
> > > > > > > > > > >>>> instead
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not
> > > > > > > > > > >>>> used
> > > > > > > > > > >>>> (and
> > > > > > > > > > >>>> there
> > > > > > > > > > >>>> has
> > > > > > > > > > >>>> to be a reason, since the SR-IOV code path does
> > > > > > > > > > >>>> use
> > > > > > > > > > >>>> vram).
> > > > > > > > > > >>>> Changing it to vram does not make a difference,
> > > > > > > > > > >>>> but
> > > > > > > > > > >>>> then
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> only bo that seems to be used at that point is
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> one
> > > > > > > > > > >>>> for
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>> psp ring,
> > > > > > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > > > > > >>>> surprised.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Maybe I should double-check bo_create calls to
> > > > > > > > > > >>>> hunt
> > > > > > > > > > >>>> for
> > > > > > > > > > >>>> more
> > > > > > > > > > >>>> ?
> > > > > > > > > > >>> We looked into this a bit ourselves and ran into
> > > > > > > > > > >>> the
> > > > > > > > > > >>> same
> > > > > > > > > > >>> issues.
> > > > > > > > > > >>> We'd probably need to debug this with the PSP
> > > > > > > > > > >>> team to
> > > > > > > > > > >>> make
> > > > > > > > > > >>> further
> > > > > > > > > > >>> progress, but this was not productized so neither
> > > > > > > > > > >>> team
> > > > > > > > > > >>> had
> > > > > > > > > > >>> the
> > > > > > > > > > >>> resources to delve further.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Alex
> > > > > > > > > > >>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [0]
> > > > > > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> > How is the stolen memory communicated to the driver ?  That host
> > physical
> > memory probably has to be mapped at the same guest physical address
> > for
> > the magic to work, right ?
> 
> Correct.  The driver reads the physical location of that memory from
> hardware registers.  Removing this chunk of code from gmc_v9_0.c will
> force the driver to use the BAR,

That would only be a workaround for a missing mapping of stolen
memory to the guest, right ?


> but I'm not sure if there are any
> other places in the driver that make assumptions about using the
> physical host address or not on APUs off hand.

gmc_v9_0_vram_gtt_location() updates vm_manager.vram_base_offset from
the same value.  I'm not sure I understand why in this case there is
no reason to use the BAR while there are some in gmc_v9_0_mc_init().

vram_base_offset then gets used in several places:

* amdgpu_gmc_init_pdb0, that seems likely enough to be problematic,
  right ?  
  As a sidenote the XGMI offset added earlier gets substracted
  here to deduce vram base addr
  (a couple of new acronyms there: PDB, PDE -- page directory base/entry?)

* amdgpu_ttm_map_buffer, amdgpu_vm_bo_update_mapping: those seem to be
  as problematic

* amdgpu_gmc_vram_mc2pa: until I got there I had assumed MC could stand for 
  "memory controller", but then "MC address of buffer" makes me doubt


> 
>         if ((adev->flags & AMD_IS_APU) ||
>             (adev->gmc.xgmi.supported &&
>              adev->gmc.xgmi.connected_to_cpu)) {
>                 adev->gmc.aper_base =
>                         adev->gfxhub.funcs->get_mc_fb_offset(adev) +
>                         adev->gmc.xgmi.physical_node_id *
>                         adev->gmc.xgmi.node_segment_size;
>                 adev->gmc.aper_size = adev->gmc.real_vram_size;
>         }


Now for the test... it does indeed seem to go much further, I even
loose the dom0's efifb to that black screen hopefully showing the
driver started to setup the hardware.  Will probably still have to
hunt down whether it still tries to use efifb afterwards (can't see
why it would not, TBH, given the previous behaviour where it kept
using it after the guest failed to start).

The log shows many details about TMR loading

Then as expected:

[2022-01-06 15:16:09] <6>[    5.844589] amdgpu 0000:00:05.0: amdgpu: RAP: optional rap ta ucode is not available
[2022-01-06 15:16:09] <6>[    5.844619] amdgpu 0000:00:05.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
[2022-01-06 15:16:09] <7>[    5.844639] [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP block <smu>...
[2022-01-06 15:16:09] <6>[    5.845515] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!


not sure about that unhandled interrupt (and a bit worried about messed-up logs):

[2022-01-06 15:16:09] <7>[    6.010681] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_hel[2022-01-06 15:16:10] per [amdgpu]] ring test on sdma0 succeeded
[2022-01-06 15:16:10] <7>[    6.010831] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 0, wptr 32
[2022-01-06 15:16:10] <7>[    6.011002] [drm:amdgpu_irq_dispatch [amdgpu]] Unhandled interrupt src_id: 243


then comes a first error:

[2022-01-06 15:16:10] <6>[    6.011785] [drm] Display Core initialized with v3.2.149!
[2022-01-06 15:16:10] <6>[    6.012714] [drm] DMUB hardware initialized: version=0x0101001C
[2022-01-06 15:16:10] <3>[    6.228263] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
[2022-01-06 15:16:10] <7>[    6.229125] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: freesync_module init done 0000000076c7b459.
[2022-01-06 15:16:10] <7>[    6.229677] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: hdcp_workqueue init done 0000000087e28b47.
[2022-01-06 15:16:10] <7>[    6.229979] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu_dm_connector_init()

... which we can see again several times later though the driver seems sufficient to finish init:

[2022-01-06 15:16:10] <6>[    6.615615] [drm] late_init of IP block <smu>...
[2022-01-06 15:16:10] <6>[    6.615772] [drm] late_init of IP block <gfx_v9_0>...
[2022-01-06 15:16:10] <6>[    6.615801] [drm] late_init of IP block <sdma_v4_0>...
[2022-01-06 15:16:10] <6>[    6.615827] [drm] late_init of IP block <dm>...
[2022-01-06 15:16:10] <3>[    6.801790] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
[2022-01-06 15:16:10] <7>[    6.806079] [drm:drm_minor_register [drm]] 
[2022-01-06 15:16:10] <7>[    6.806195] [drm:drm_minor_register [drm]] new minor registered 128
[2022-01-06 15:16:10] <7>[    6.806223] [drm:drm_minor_register [drm]] 
[2022-01-06 15:16:10] <7>[    6.806289] [drm:drm_minor_register [drm]] new minor registered 0
[2022-01-06 15:16:10] <7>[    6.806355] [drm:drm_sysfs_connector_add [drm]] adding "eDP-1" to sysfs
[2022-01-06 15:16:10] <7>[    6.806424] [drm:drm_dp_aux_register_devnode [drm_kms_helper]] drm_dp_aux_dev: aux [AMDGPU DM aux hw bus 0] registered as minor 0
[2022-01-06 15:16:10] <7>[    6.806498] [drm:drm_sysfs_hotplug_event [drm]] generating hotplug event
[2022-01-06 15:16:10] <6>[    6.806533] [drm] Initialized amdgpu 3.42.0 20150101 for 0000:00:05.0 on minor 0


At one point though a new problem shows: it seem to have issues driving the CRTC in the end:

[2022-01-06 15:16:25] <7>[   11.140807] amdgpu 0000:00:05.0: [drm:drm_vblank_enable [drm]] enabling vblank on crtc 0, ret: 0
[2022-01-06 15:16:25] <3>[   11.329306] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
[2022-01-06 15:16:25] <3>[   11.524327] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
[2022-01-06 15:16:25] <4>[   11.641814] [drm] Fence fallback timer expired on ring comp_1.3.0
[2022-01-06 15:16:25] <7>[   11.641877] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.0 succeeded
[2022-01-06 15:16:25] <4>[   12.145804] [drm] Fence fallback timer expired on ring comp_1.0.1
[2022-01-06 15:16:25] <7>[   12.145862] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.0.1 succeeded
[2022-01-06 15:16:25] <4>[   12.649771] [drm] Fence fallback timer expired on ring comp_1.1.1
[2022-01-06 15:16:25] <7>[   12.649789] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.1.1 succeeded
[2022-01-06 15:16:25] <4>[   13.153815] [drm] Fence fallback timer expired on ring comp_1.2.1
[2022-01-06 15:16:25] <7>[   13.153836] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.2.1 succeeded
[2022-01-06 15:16:25] <4>[   13.657756] [drm] Fence fallback timer expired on ring comp_1.3.1
[2022-01-06 15:16:25] <7>[   13.657767] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.1 succeeded
[2022-01-06 15:16:25] <7>[   13.657899] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Setting write pointer
[2022-01-06 15:16:25] <7>[   13.658008] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Using doorbell -- wptr_offs == 0x00000198 lower_32_bits(ring->wptr) << 2 == 0x00000100 upper_32_bits(ring->wptr) << 2 == 0x00000000
[2022-01-06 15:16:25] <7>[   13.658114] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] calling WDOORBELL64(0x000001e0, 0x0000000000000100)
[2022-01-06 15:16:25] <4>[   14.161792] [drm] Fence fallback timer expired on ring sdma0
[2022-01-06 15:16:25] <7>[   14.161811] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on sdma0 succeeded
[2022-01-06 15:16:25] <3>[   21.609821] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out


No visible change if I boot with efifb:off (aside from entering LUKS
passphrase in the dark, that is).


Tried patching gmc_v9_0_vram_gtt_location() to use the BAR too [2], but
that turns out to work even less:

[2022-01-06 16:27:48] <6>[    6.230166] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!
[2022-01-06 16:27:48] <7>[    6.230168] [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP block <gfx_v9_0>...
[2022-01-06 16:27:48] <6>[    6.231948] [drm] kiq ring mec 2 pipe 1 q 0
[2022-01-06 16:27:48] <7>[    6.231861] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 448, wptr 512
[2022-01-06 16:27:48] <7>[    6.231962] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq alloc'd 64
[2022-01-06 16:27:48] <7>[    6.232172] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size init: 256
[2022-01-06 16:27:48] <7>[    6.232344] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after set_res: 248
[2022-01-06 16:27:48] <7>[    6.232530] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after map_q: 192
[2022-01-06 16:27:48] <7>[    6.232725] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 512, wptr 544
[2022-01-06 16:27:48] <3>[    6.429974] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
[2022-01-06 16:27:48] <7>[    6.430167] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after test: 0
[2022-01-06 16:27:48] <3>[    6.430353] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] *ERROR* KCQ enable failed
[2022-01-06 16:27:48] <3>[    6.430532] [drm:amdgpu_device_init.cold [amdgpu]] *ERROR* hw_init of IP block <gfx_v9_0> failed -110
[2022-01-06 16:27:48] <3>[    6.430720] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed




As a sidenote, my warning on ring_alloc() being called twice without
commiting or undoing [1] gets triggered.  Given the call chain it looks
like this would happen in the previous usage of that ring, would have to
dig deeper to understand that.  Unless I'm missing something and this would
be legal ?

[2022-01-06 15:52:17] <4>[    5.929158] ------------[ cut here ]------------
[2022-01-06 15:52:17] <4>[    5.929170] WARNING: CPU: 1 PID: 458 at drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:74 amdgpu_ring_alloc+0x62/0x70 [amdgpu]
[2022-01-06 15:52:17] <4>[    5.929323] Modules linked in: ip6table_filter ip6table_mangle joydev ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel amdgpu(+) iommu_v2 gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ehci_pci cec pcspkr ehci_hcd i2c_piix4 serio_raw ata_generic pata_acpi xen_scsiback target_core_mod xen_netback xen_privcmd xen_gntdev xen_gntalloc xen_blkback fuse drm xen_evtchn bpf_preload ip_tables overlay xen_blkfront
[2022-01-06 15:52:17] <4>[    5.929458] CPU: 1 PID: 458 Comm: sdma0 Not tainted 5.15.4-1.fc32.qubes.x86_64+ #8
[2022-01-06 15:52:17] <4>[    5.929474] Hardware name: Xen HVM domU, BIOS 4.14.3 01/03/2022
[2022-01-06 15:52:17] <4>[    5.929487] RIP: 0010:amdgpu_ring_alloc+0x62/0x70 [amdgpu]
[2022-01-06 15:52:17] <4>[    5.929628] Code: 87 28 02 00 00 48 8b 82 b8 00 00 00 48 85 c0 74 05 e8 b2 ae 90 ee 44 89 e0 41 5c c3 0f 0b 41 bc f4 ff ff ff 44 89 e0 41 5c c3 <0f> 0b 48 8b 57 08 eb bc 66 0f 1f 44 00 00 0f 1f 44 00 00 85 f6 0f
[2022-01-06 15:52:17] <4>[    5.929667] RSP: 0018:ffffb129005f3dd8 EFLAGS: 00010206
[2022-01-06 15:52:17] <4>[    5.929678] RAX: 0000000000000060 RBX: ffff96209112d230 RCX: 0000000000000050
[2022-01-06 15:52:17] <4>[    5.929693] RDX: ffffffffc0ac6c60 RSI: 000000000000006d RDI: ffff96208c5eb8f8
[2022-01-06 15:52:17] <4>[    5.929707] RBP: ffff96209112d000 R08: ffffb129005f3e50 R09: ffff96208c5eba98
[2022-01-06 15:52:17] <4>[    5.929722] R10: 0000000000000000 R11: 0000000000000001 R12: ffff962090a0c780
[2022-01-06 15:52:17] <4>[    5.929736] R13: 0000000000000001 R14: ffff96208c5eb8f8 R15: ffff96208c5eb970
[2022-01-06 15:52:17] <4>[    5.929752] FS:  0000000000000000(0000) GS:ffff9620bcd00000(0000) knlGS:0000000000000000
[2022-01-06 15:52:17] <4>[    5.929768] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2022-01-06 15:52:17] <4>[    5.929781] CR2: 00007c1130d0f860 CR3: 00000000040c4000 CR4: 0000000000350ee0
[2022-01-06 15:52:17] <4>[    5.929797] Call Trace:
[2022-01-06 15:52:17] <4>[    5.929805]  <TASK>
[2022-01-06 15:52:17] <4>[    5.929812]  amdgpu_ib_schedule+0xa9/0x540 [amdgpu]
[2022-01-06 15:52:17] <4>[    5.929956]  ? _raw_spin_unlock_irqrestore+0xa/0x20
[2022-01-06 15:52:17] <4>[    5.929969]  amdgpu_job_run+0xce/0x1f0 [amdgpu]
[2022-01-06 15:52:17] <4>[    5.930131]  drm_sched_main+0x300/0x500 [gpu_sched]
[2022-01-06 15:52:17] <4>[    5.930146]  ? finish_wait+0x80/0x80
[2022-01-06 15:52:17] <4>[    5.930156]  ? drm_sched_rq_select_entity+0xa0/0xa0 [gpu_sched]
[2022-01-06 15:52:17] <4>[    5.930171]  kthread+0x127/0x150
[2022-01-06 15:52:17] <4>[    5.930181]  ? set_kthread_struct+0x40/0x40
[2022-01-06 15:52:17] <4>[    5.930192]  ret_from_fork+0x22/0x30
[2022-01-06 15:52:17] <4>[    5.930203]  </TASK>
[2022-01-06 15:52:17] <4>[    5.930208] ---[ end trace cf0edb400b0116c7 ]---


[1] https://github.com/ydirson/linux/commit/4a010943e74d6bf621bd9e72a7620a65af23ecc9
[2] https://github.com/ydirson/linux/commit/e90230e008ce204d822f07e36b3c3e196d561c28

> 
> 
> 
> >
> > > > >
> > > > > > ... which brings me to a point that's been puzzling me for
> > > > > > some
> > > > > > time, which is
> > > > > > that as the hw init fails, the efifb driver is still using
> > > > > > the
> > > > > > framebuffer.
> > > > >
> > > > > No, it isn't. You are probably just still seeing the same
> > > > > screen.
> > > > >
> > > > > The issue is most likely that while efi was kicked out nobody
> > > > > re-programmed the display hardware to show something
> > > > > different.
> > > > >
> > > > > > Am I right in suspecting that efifb should get stripped of
> > > > > > its
> > > > > > ownership of the
> > > > > > fb aperture first, and that if I don't get a black screen
> > > > > > on
> > > > > > hw_init failure
> > > > > > that issue should be the first focus point ?
> > > > >
> > > > > You assumption with the black screen is incorrect. Since the
> > > > > hardware
> > > > > works independent even if you kick out efi you still have the
> > > > > same
> > > > > screen content, you just can't update it anymore.
> > > >
> > > > It's not only that the screen keeps its contents, it's that the
> > > > dom0
> > > > happily continues updating it.
> > >
> > > If the hypevisor is using efifb, then yes that could be a problem
> > > as
> > > the hypervisor could be writing to the efifb resources which ends
> > > up
> > > writing to the same physical memory.  That applies to any GPU on
> > > a
> > > UEFI system.  You'll need to make sure efifb is not in use in the
> > > hypervisor.

> >
> > That remark evokes several things to me.  First one is that every
> > time
> > I've tried booting with efifb disabled in dom0, there was no
> > visible
> > improvements in the guest driver - i.i. I really have to dig how
> > vram mapping
> > is performed and check things are as expected anyway.
> 
> Ultimately you end up at the same physical memory.  efifb uses the
> PCI
> BAR which points to the same physical memory that the driver directly
> maps.
> 
> >
> > The other is that, when dom0 cannot use efifb, entering a luks key
> > is
> > suddenly less user-friendly.  But in theory I'd think we could
> > overcome
> > this by letting dom0 use efifb until ready to start the guest, a
> > simple
> > driver unbind at the right moment should be expected to work, right
> > ?
> > Going further and allowing the guest to use efifb on its own could
> > possibly be more tricky (starting with a different state?) but does
> > not seem to sound completely outlandish either - or does it ?
> >
> 
> efifb just takes whatever hardware state the GOP driver in the pre-OS
> environment left the GPU in.  Once you have a driver loaded in the
> OS,
> that state is gone so I I don't see much value in using efifb once
> you
> have a real driver in the mix.  If you want a console on the host,
> it's probably better to use 2 GPU or just load the real driver as
> needed in both the host and guest.
> 
> > >
> > > Alex
> > >
> > >
> > > >
> > > > > But putting efi asside what Alex pointed out pretty much
> > > > > breaks
> > > > > your
> > > > > neck trying to forward the device. You maybe could try to
> > > > > hack
> > > > > the
> > > > > driver to use the PCIe BAR for framebuffer access, but that
> > > > > might
> > > > > be
> > > > > quite a bit slower.
> > > > >
> > > > > Regards,
> > > > > Christian.
> > > > >
> > > > > >
> > > > > >> Alex
> > > > > >>
> > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > >> <alexdeucher@gmail.com>
> > > > > >> wrote:
> > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > >>> <ydirson@free.fr>
> > > > > >>> wrote:
> > > > > >>>> Alex wrote:
> > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > >>>>> <ydirson@free.fr>
> > > > > >>>>> wrote:
> > > > > >>>>>> Hi Alex,
> > > > > >>>>>>
> > > > > >>>>>>> We have not validated virtualization of our
> > > > > >>>>>>> integrated
> > > > > >>>>>>> GPUs.  I
> > > > > >>>>>>> don't
> > > > > >>>>>>> know that it will work at all.  We had done a bit of
> > > > > >>>>>>> testing but
> > > > > >>>>>>> ran
> > > > > >>>>>>> into the same issues with the PSP, but never had a
> > > > > >>>>>>> chance
> > > > > >>>>>>> to
> > > > > >>>>>>> debug
> > > > > >>>>>>> further because this feature is not productized.
> > > > > >>>>>> ...
> > > > > >>>>>>> You need a functional PSP to get the GPU driver up
> > > > > >>>>>>> and
> > > > > >>>>>>> running.
> > > > > >>>>>> Ah, thanks for the hint :)
> > > > > >>>>>>
> > > > > >>>>>> I guess that if I want to have any chance to get the
> > > > > >>>>>> PSP
> > > > > >>>>>> working
> > > > > >>>>>> I'm
> > > > > >>>>>> going to need more details on it.  A quick search some
> > > > > >>>>>> time
> > > > > >>>>>> ago
> > > > > >>>>>> mostly
> > > > > >>>>>> brought reverse-engineering work, rather than official
> > > > > >>>>>> AMD
> > > > > >>>>>> doc.
> > > > > >>>>>>   Are
> > > > > >>>>>> there some AMD resources I missed ?
> > > > > >>>>> The driver code is pretty much it.
> > > > > >>>> Let's try to shed some more light on how things work,
> > > > > >>>> taking
> > > > > >>>> as
> > > > > >>>> excuse
> > > > > >>>> psp_v12_0_ring_create().
> > > > > >>>>
> > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > >>>> implemented
> > > > > >>>> in
> > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > > > >>>> [RW]REG32(),
> > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > >>> (VF),
> > > > > >>> some
> > > > > >>> registers are not available directly via the VF's MMIO
> > > > > >>> aperture
> > > > > >>> so
> > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > >>> passthrough
> > > > > >>> this
> > > > > >>> is not relevant.
> > > > > >>>
> > > > > >>>> It accesses registers in the MMIO range of the MP0 IP,
> > > > > >>>> and
> > > > > >>>> the
> > > > > >>>> "MP0"
> > > > > >>>> name correlates highly with MMIO accesses in
> > > > > >>>> PSP-handling
> > > > > >>>> code.
> > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The
> > > > > >>>> MP0
> > > > > >>>> version
> > > > > >>> Yes.
> > > > > >>>
> > > > > >>>> reported at v11.0.3 by discovery seems to contradict the
> > > > > >>>> use
> > > > > >>>> of
> > > > > >>>> v12.0
> > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > > > >>>> something ?
> > > > > >>> Typo in the ip discovery table on renoir.
> > > > > >>>
> > > > > >>>> More generally (and mostly out of curiosity while we're
> > > > > >>>> at
> > > > > >>>> it),
> > > > > >>>> do we
> > > > > >>>> have a way to match IPs listed at discovery time with
> > > > > >>>> the
> > > > > >>>> ones
> > > > > >>>> used
> > > > > >>>> in the driver ?
> > > > > >>> In general, barring typos, the code is shared at the
> > > > > >>> major
> > > > > >>> version
> > > > > >>> level.  The actual code may or may not need changes to
> > > > > >>> handle
> > > > > >>> minor
> > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > >>> versions
> > > > > >>> from
> > > > > >>> the
> > > > > >>> ip discovery table to the code contained in the driver.
> > > > > >>>
> > > > > >>>> ---
> > > > > >>>>
> > > > > >>>> As for the register names, maybe we could have a short
> > > > > >>>> explanation of
> > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > > > >>>> seems
> > > > > >>>> to
> > > > > >>>> be
> > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but
> > > > > >>>> I'm
> > > > > >>>> not
> > > > > >>>> sure
> > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > >>>> Management
> > > > > >>>> Network",
> > > > > >>>> described in [0] as an internal bus.  Are we accessing
> > > > > >>>> this
> > > > > >>>> register
> > > > > >>>> through this SMN ?
> > > > > >>> These registers are just mailboxes for the PSP firmware.
> > > > > >>>  All
> > > > > >>> of
> > > > > >>> the
> > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > >>> firmware.
> > > > > >>>   They
> > > > > >>> are basically scratch registers used to communicate
> > > > > >>> between
> > > > > >>> the
> > > > > >>> driver
> > > > > >>> and the PSP firmware.
> > > > > >>>
> > > > > >>>>
> > > > > >>>>>   On APUs, the PSP is shared with
> > > > > >>>>> the CPU and the rest of the platform.  The GPU driver
> > > > > >>>>> just
> > > > > >>>>> interacts
> > > > > >>>>> with it for a few specific tasks:
> > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > > > >>>>> applications
> > > > > >>>>> that run on the PSP for specific functionality, e.g.,
> > > > > >>>>> HDCP
> > > > > >>>>> and
> > > > > >>>>> content
> > > > > >>>>> protection, etc.)
> > > > > >>>>> 2. Validating and loading firmware for other engines on
> > > > > >>>>> the
> > > > > >>>>> SoC.
> > > > > >>>>>   This
> > > > > >>>>> is required to use those engines.
> > > > > >>>> Trying to understand in more details how we start the
> > > > > >>>> PSP
> > > > > >>>> up, I
> > > > > >>>> noticed
> > > > > >>>> that psp_v12_0 has support for loading a sOS firmware,
> > > > > >>>> but
> > > > > >>>> never
> > > > > >>>> calls
> > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > >>>> firmware
> > > > > >>>> for
> > > > > >>>> renoir
> > > > > >>>> and green_sardine, which seem to be the only ASICs with
> > > > > >>>> this
> > > > > >>>> PSP
> > > > > >>>> version.
> > > > > >>>> Is it something that's just not been completely wired up
> > > > > >>>> yet
> > > > > >>>> ?
> > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > >>> firmware
> > > > > >>> is
> > > > > >>> part
> > > > > >>> of
> > > > > >>> the sbios image.  The driver doesn't load it.  We only
> > > > > >>> load
> > > > > >>> it on
> > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > >>> initialization.
> > > > > >>>
> > > > > >>>> That also rings a bell, that we have nothing about
> > > > > >>>> Secure OS
> > > > > >>>> in
> > > > > >>>> the doc
> > > > > >>>> yet (not even the acronym in the glossary).
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>> I'm not too familiar with the PSP's path to memory from
> > > > > >>>>> the
> > > > > >>>>> GPU
> > > > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > > > >>>>> through
> > > > > >>>>> carve
> > > > > >>>>> out
> > > > > >>>>> "vram" on APUs so it should work, but I would double
> > > > > >>>>> check
> > > > > >>>>> if
> > > > > >>>>> there
> > > > > >>>>> are any system memory allocations that used to interact
> > > > > >>>>> with
> > > > > >>>>> the PSP
> > > > > >>>>> and see if changing them to vram helps.  It does work
> > > > > >>>>> with
> > > > > >>>>> the
> > > > > >>>>> IOMMU
> > > > > >>>>> enabled on bare metal, so it should work in passthrough
> > > > > >>>>> as
> > > > > >>>>> well
> > > > > >>>>> in
> > > > > >>>>> theory.
> > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > >>>> used
> > > > > >>>> instead
> > > > > >>>> of
> > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and
> > > > > >>>> there
> > > > > >>>> has
> > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > >>>> vram).
> > > > > >>>> Changing it to vram does not make a difference, but then
> > > > > >>>> the
> > > > > >>>> only bo that seems to be used at that point is the one
> > > > > >>>> for
> > > > > >>>> the
> > > > > >>>> psp ring,
> > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > >>>> surprised.
> > > > > >>>>
> > > > > >>>> Maybe I should double-check bo_create calls to hunt for
> > > > > >>>> more
> > > > > >>>> ?
> > > > > >>> We looked into this a bit ourselves and ran into the same
> > > > > >>> issues.
> > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > >>> make
> > > > > >>> further
> > > > > >>> progress, but this was not productized so neither team
> > > > > >>> had
> > > > > >>> the
> > > > > >>> resources to delve further.
> > > > > >>>
> > > > > >>> Alex
> > > > > >>>
> > > > > >>>>
> > > > > >>>> [0]
> > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > >
> > > > >
> > >
> 

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Alex Deucher]

On Thu, Jan 6, 2022 at 10:38 AM Yann Dirson <ydirson@free.fr> wrote:
>
> Alex wrote:
> > > How is the stolen memory communicated to the driver ?  That host
> > > physical
> > > memory probably has to be mapped at the same guest physical address
> > > for
> > > the magic to work, right ?
> >
> > Correct.  The driver reads the physical location of that memory from
> > hardware registers.  Removing this chunk of code from gmc_v9_0.c will
> > force the driver to use the BAR,
>
> That would only be a workaround for a missing mapping of stolen
> memory to the guest, right ?


Correct. That will use the PCI BAR rather than the underlying physical
memory for CPU access to the carve out region.

>
>
> > but I'm not sure if there are any
> > other places in the driver that make assumptions about using the
> > physical host address or not on APUs off hand.
>
> gmc_v9_0_vram_gtt_location() updates vm_manager.vram_base_offset from
> the same value.  I'm not sure I understand why in this case there is
> no reason to use the BAR while there are some in gmc_v9_0_mc_init().
>
> vram_base_offset then gets used in several places:
>
> * amdgpu_gmc_init_pdb0, that seems likely enough to be problematic,
>   right ?
>   As a sidenote the XGMI offset added earlier gets substracted
>   here to deduce vram base addr
>   (a couple of new acronyms there: PDB, PDE -- page directory base/entry?)
>
> * amdgpu_ttm_map_buffer, amdgpu_vm_bo_update_mapping: those seem to be
>   as problematic
>
> * amdgpu_gmc_vram_mc2pa: until I got there I had assumed MC could stand for
>   "memory controller", but then "MC address of buffer" makes me doubt
>
>

MC = memory controller (as in graphics memory controller).

These are GPU addresses not CPU addresses so they should be fine.

> >
> >         if ((adev->flags & AMD_IS_APU) ||
> >             (adev->gmc.xgmi.supported &&
> >              adev->gmc.xgmi.connected_to_cpu)) {
> >                 adev->gmc.aper_base =
> >                         adev->gfxhub.funcs->get_mc_fb_offset(adev) +
> >                         adev->gmc.xgmi.physical_node_id *
> >                         adev->gmc.xgmi.node_segment_size;
> >                 adev->gmc.aper_size = adev->gmc.real_vram_size;
> >         }
>
>
> Now for the test... it does indeed seem to go much further, I even
> loose the dom0's efifb to that black screen hopefully showing the
> driver started to setup the hardware.  Will probably still have to
> hunt down whether it still tries to use efifb afterwards (can't see
> why it would not, TBH, given the previous behaviour where it kept
> using it after the guest failed to start).
>
> The log shows many details about TMR loading
>
> Then as expected:
>
> [2022-01-06 15:16:09] <6>[    5.844589] amdgpu 0000:00:05.0: amdgpu: RAP: optional rap ta ucode is not available
> [2022-01-06 15:16:09] <6>[    5.844619] amdgpu 0000:00:05.0: amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
> [2022-01-06 15:16:09] <7>[    5.844639] [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP block <smu>...
> [2022-01-06 15:16:09] <6>[    5.845515] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!
>
>
> not sure about that unhandled interrupt (and a bit worried about messed-up logs):
>
> [2022-01-06 15:16:09] <7>[    6.010681] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_hel[2022-01-06 15:16:10] per [amdgpu]] ring test on sdma0 succeeded
> [2022-01-06 15:16:10] <7>[    6.010831] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 0, wptr 32
> [2022-01-06 15:16:10] <7>[    6.011002] [drm:amdgpu_irq_dispatch [amdgpu]] Unhandled interrupt src_id: 243
>
>
> then comes a first error:
>
> [2022-01-06 15:16:10] <6>[    6.011785] [drm] Display Core initialized with v3.2.149!
> [2022-01-06 15:16:10] <6>[    6.012714] [drm] DMUB hardware initialized: version=0x0101001C
> [2022-01-06 15:16:10] <3>[    6.228263] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> [2022-01-06 15:16:10] <7>[    6.229125] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: freesync_module init done 0000000076c7b459.
> [2022-01-06 15:16:10] <7>[    6.229677] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: hdcp_workqueue init done 0000000087e28b47.
> [2022-01-06 15:16:10] <7>[    6.229979] [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu_dm_connector_init()
>
> ... which we can see again several times later though the driver seems sufficient to finish init:
>
> [2022-01-06 15:16:10] <6>[    6.615615] [drm] late_init of IP block <smu>...
> [2022-01-06 15:16:10] <6>[    6.615772] [drm] late_init of IP block <gfx_v9_0>...
> [2022-01-06 15:16:10] <6>[    6.615801] [drm] late_init of IP block <sdma_v4_0>...
> [2022-01-06 15:16:10] <6>[    6.615827] [drm] late_init of IP block <dm>...
> [2022-01-06 15:16:10] <3>[    6.801790] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> [2022-01-06 15:16:10] <7>[    6.806079] [drm:drm_minor_register [drm]]
> [2022-01-06 15:16:10] <7>[    6.806195] [drm:drm_minor_register [drm]] new minor registered 128
> [2022-01-06 15:16:10] <7>[    6.806223] [drm:drm_minor_register [drm]]
> [2022-01-06 15:16:10] <7>[    6.806289] [drm:drm_minor_register [drm]] new minor registered 0
> [2022-01-06 15:16:10] <7>[    6.806355] [drm:drm_sysfs_connector_add [drm]] adding "eDP-1" to sysfs
> [2022-01-06 15:16:10] <7>[    6.806424] [drm:drm_dp_aux_register_devnode [drm_kms_helper]] drm_dp_aux_dev: aux [AMDGPU DM aux hw bus 0] registered as minor 0
> [2022-01-06 15:16:10] <7>[    6.806498] [drm:drm_sysfs_hotplug_event [drm]] generating hotplug event
> [2022-01-06 15:16:10] <6>[    6.806533] [drm] Initialized amdgpu 3.42.0 20150101 for 0000:00:05.0 on minor 0
>
>

Looks like it initialized fine.  I guess the DMCUB firmware issues are
not fatal.  Probably need input from one of the display guys on that.

> At one point though a new problem shows: it seem to have issues driving the CRTC in the end:
>
> [2022-01-06 15:16:25] <7>[   11.140807] amdgpu 0000:00:05.0: [drm:drm_vblank_enable [drm]] enabling vblank on crtc 0, ret: 0
> [2022-01-06 15:16:25] <3>[   11.329306] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> [2022-01-06 15:16:25] <3>[   11.524327] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> [2022-01-06 15:16:25] <4>[   11.641814] [drm] Fence fallback timer expired on ring comp_1.3.0
> [2022-01-06 15:16:25] <7>[   11.641877] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.0 succeeded
> [2022-01-06 15:16:25] <4>[   12.145804] [drm] Fence fallback timer expired on ring comp_1.0.1
> [2022-01-06 15:16:25] <7>[   12.145862] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.0.1 succeeded
> [2022-01-06 15:16:25] <4>[   12.649771] [drm] Fence fallback timer expired on ring comp_1.1.1
> [2022-01-06 15:16:25] <7>[   12.649789] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.1.1 succeeded
> [2022-01-06 15:16:25] <4>[   13.153815] [drm] Fence fallback timer expired on ring comp_1.2.1
> [2022-01-06 15:16:25] <7>[   13.153836] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.2.1 succeeded
> [2022-01-06 15:16:25] <4>[   13.657756] [drm] Fence fallback timer expired on ring comp_1.3.1
> [2022-01-06 15:16:25] <7>[   13.657767] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.1 succeeded
> [2022-01-06 15:16:25] <7>[   13.657899] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Setting write pointer
> [2022-01-06 15:16:25] <7>[   13.658008] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Using doorbell -- wptr_offs == 0x00000198 lower_32_bits(ring->wptr) << 2 == 0x00000100 upper_32_bits(ring->wptr) << 2 == 0x00000000
> [2022-01-06 15:16:25] <7>[   13.658114] [drm:sdma_v4_0_ring_set_wptr [amdgpu]] calling WDOORBELL64(0x000001e0, 0x0000000000000100)
> [2022-01-06 15:16:25] <4>[   14.161792] [drm] Fence fallback timer expired on ring sdma0
> [2022-01-06 15:16:25] <7>[   14.161811] amdgpu 0000:00:05.0: [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on sdma0 succeeded
> [2022-01-06 15:16:25] <3>[   21.609821] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:67:crtc-0] flip_done timed out
>
>
> No visible change if I boot with efifb:off (aside from entering LUKS
> passphrase in the dark, that is).
>
>
> Tried patching gmc_v9_0_vram_gtt_location() to use the BAR too [2], but
> that turns out to work even less:


That won't work.  These are GPU addresses not CPU addresses.

>
> [2022-01-06 16:27:48] <6>[    6.230166] amdgpu 0000:00:05.0: amdgpu: SMU is initialized successfully!
> [2022-01-06 16:27:48] <7>[    6.230168] [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP block <gfx_v9_0>...
> [2022-01-06 16:27:48] <6>[    6.231948] [drm] kiq ring mec 2 pipe 1 q 0
> [2022-01-06 16:27:48] <7>[    6.231861] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 448, wptr 512
> [2022-01-06 16:27:48] <7>[    6.231962] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq alloc'd 64
> [2022-01-06 16:27:48] <7>[    6.232172] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size init: 256
> [2022-01-06 16:27:48] <7>[    6.232344] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after set_res: 248
> [2022-01-06 16:27:48] <7>[    6.232530] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after map_q: 192
> [2022-01-06 16:27:48] <7>[    6.232725] [drm:amdgpu_ih_process [amdgpu]] amdgpu_ih_process: rptr 512, wptr 544
> [2022-01-06 16:27:48] <3>[    6.429974] amdgpu 0000:00:05.0: [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test failed (-110)
> [2022-01-06 16:27:48] <7>[    6.430167] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after test: 0
> [2022-01-06 16:27:48] <3>[    6.430353] [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] *ERROR* KCQ enable failed
> [2022-01-06 16:27:48] <3>[    6.430532] [drm:amdgpu_device_init.cold [amdgpu]] *ERROR* hw_init of IP block <gfx_v9_0> failed -110
> [2022-01-06 16:27:48] <3>[    6.430720] amdgpu 0000:00:05.0: amdgpu: amdgpu_device_ip_init failed
>
>
>
>
> As a sidenote, my warning on ring_alloc() being called twice without
> commiting or undoing [1] gets triggered.  Given the call chain it looks
> like this would happen in the previous usage of that ring, would have to
> dig deeper to understand that.  Unless I'm missing something and this would
> be legal ?

I don't remember off hand.

Alex

>
> [2022-01-06 15:52:17] <4>[    5.929158] ------------[ cut here ]------------
> [2022-01-06 15:52:17] <4>[    5.929170] WARNING: CPU: 1 PID: 458 at drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:74 amdgpu_ring_alloc+0x62/0x70 [amdgpu]
> [2022-01-06 15:52:17] <4>[    5.929323] Modules linked in: ip6table_filter ip6table_mangle joydev ip6table_raw ip6_tables ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack iptable_filter iptable_mangle iptable_raw xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel amdgpu(+) iommu_v2 gpu_sched i2c_algo_bit drm_ttm_helper ttm drm_kms_helper ehci_pci cec pcspkr ehci_hcd i2c_piix4 serio_raw ata_generic pata_acpi xen_scsiback target_core_mod xen_netback xen_privcmd xen_gntdev xen_gntalloc xen_blkback fuse drm xen_evtchn bpf_preload ip_tables overlay xen_blkfront
> [2022-01-06 15:52:17] <4>[    5.929458] CPU: 1 PID: 458 Comm: sdma0 Not tainted 5.15.4-1.fc32.qubes.x86_64+ #8
> [2022-01-06 15:52:17] <4>[    5.929474] Hardware name: Xen HVM domU, BIOS 4.14.3 01/03/2022
> [2022-01-06 15:52:17] <4>[    5.929487] RIP: 0010:amdgpu_ring_alloc+0x62/0x70 [amdgpu]
> [2022-01-06 15:52:17] <4>[    5.929628] Code: 87 28 02 00 00 48 8b 82 b8 00 00 00 48 85 c0 74 05 e8 b2 ae 90 ee 44 89 e0 41 5c c3 0f 0b 41 bc f4 ff ff ff 44 89 e0 41 5c c3 <0f> 0b 48 8b 57 08 eb bc 66 0f 1f 44 00 00 0f 1f 44 00 00 85 f6 0f
> [2022-01-06 15:52:17] <4>[    5.929667] RSP: 0018:ffffb129005f3dd8 EFLAGS: 00010206
> [2022-01-06 15:52:17] <4>[    5.929678] RAX: 0000000000000060 RBX: ffff96209112d230 RCX: 0000000000000050
> [2022-01-06 15:52:17] <4>[    5.929693] RDX: ffffffffc0ac6c60 RSI: 000000000000006d RDI: ffff96208c5eb8f8
> [2022-01-06 15:52:17] <4>[    5.929707] RBP: ffff96209112d000 R08: ffffb129005f3e50 R09: ffff96208c5eba98
> [2022-01-06 15:52:17] <4>[    5.929722] R10: 0000000000000000 R11: 0000000000000001 R12: ffff962090a0c780
> [2022-01-06 15:52:17] <4>[    5.929736] R13: 0000000000000001 R14: ffff96208c5eb8f8 R15: ffff96208c5eb970
> [2022-01-06 15:52:17] <4>[    5.929752] FS:  0000000000000000(0000) GS:ffff9620bcd00000(0000) knlGS:0000000000000000
> [2022-01-06 15:52:17] <4>[    5.929768] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [2022-01-06 15:52:17] <4>[    5.929781] CR2: 00007c1130d0f860 CR3: 00000000040c4000 CR4: 0000000000350ee0
> [2022-01-06 15:52:17] <4>[    5.929797] Call Trace:
> [2022-01-06 15:52:17] <4>[    5.929805]  <TASK>
> [2022-01-06 15:52:17] <4>[    5.929812]  amdgpu_ib_schedule+0xa9/0x540 [amdgpu]
> [2022-01-06 15:52:17] <4>[    5.929956]  ? _raw_spin_unlock_irqrestore+0xa/0x20
> [2022-01-06 15:52:17] <4>[    5.929969]  amdgpu_job_run+0xce/0x1f0 [amdgpu]
> [2022-01-06 15:52:17] <4>[    5.930131]  drm_sched_main+0x300/0x500 [gpu_sched]
> [2022-01-06 15:52:17] <4>[    5.930146]  ? finish_wait+0x80/0x80
> [2022-01-06 15:52:17] <4>[    5.930156]  ? drm_sched_rq_select_entity+0xa0/0xa0 [gpu_sched]
> [2022-01-06 15:52:17] <4>[    5.930171]  kthread+0x127/0x150
> [2022-01-06 15:52:17] <4>[    5.930181]  ? set_kthread_struct+0x40/0x40
> [2022-01-06 15:52:17] <4>[    5.930192]  ret_from_fork+0x22/0x30
> [2022-01-06 15:52:17] <4>[    5.930203]  </TASK>
> [2022-01-06 15:52:17] <4>[    5.930208] ---[ end trace cf0edb400b0116c7 ]---
>
>
> [1] https://github.com/ydirson/linux/commit/4a010943e74d6bf621bd9e72a7620a65af23ecc9
> [2] https://github.com/ydirson/linux/commit/e90230e008ce204d822f07e36b3c3e196d561c28
>
> >
> >
> >
> > >
> > > > > >
> > > > > > > ... which brings me to a point that's been puzzling me for
> > > > > > > some
> > > > > > > time, which is
> > > > > > > that as the hw init fails, the efifb driver is still using
> > > > > > > the
> > > > > > > framebuffer.
> > > > > >
> > > > > > No, it isn't. You are probably just still seeing the same
> > > > > > screen.
> > > > > >
> > > > > > The issue is most likely that while efi was kicked out nobody
> > > > > > re-programmed the display hardware to show something
> > > > > > different.
> > > > > >
> > > > > > > Am I right in suspecting that efifb should get stripped of
> > > > > > > its
> > > > > > > ownership of the
> > > > > > > fb aperture first, and that if I don't get a black screen
> > > > > > > on
> > > > > > > hw_init failure
> > > > > > > that issue should be the first focus point ?
> > > > > >
> > > > > > You assumption with the black screen is incorrect. Since the
> > > > > > hardware
> > > > > > works independent even if you kick out efi you still have the
> > > > > > same
> > > > > > screen content, you just can't update it anymore.
> > > > >
> > > > > It's not only that the screen keeps its contents, it's that the
> > > > > dom0
> > > > > happily continues updating it.
> > > >
> > > > If the hypevisor is using efifb, then yes that could be a problem
> > > > as
> > > > the hypervisor could be writing to the efifb resources which ends
> > > > up
> > > > writing to the same physical memory.  That applies to any GPU on
> > > > a
> > > > UEFI system.  You'll need to make sure efifb is not in use in the
> > > > hypervisor.
>
> > >
> > > That remark evokes several things to me.  First one is that every
> > > time
> > > I've tried booting with efifb disabled in dom0, there was no
> > > visible
> > > improvements in the guest driver - i.i. I really have to dig how
> > > vram mapping
> > > is performed and check things are as expected anyway.
> >
> > Ultimately you end up at the same physical memory.  efifb uses the
> > PCI
> > BAR which points to the same physical memory that the driver directly
> > maps.
> >
> > >
> > > The other is that, when dom0 cannot use efifb, entering a luks key
> > > is
> > > suddenly less user-friendly.  But in theory I'd think we could
> > > overcome
> > > this by letting dom0 use efifb until ready to start the guest, a
> > > simple
> > > driver unbind at the right moment should be expected to work, right
> > > ?
> > > Going further and allowing the guest to use efifb on its own could
> > > possibly be more tricky (starting with a different state?) but does
> > > not seem to sound completely outlandish either - or does it ?
> > >
> >
> > efifb just takes whatever hardware state the GOP driver in the pre-OS
> > environment left the GPU in.  Once you have a driver loaded in the
> > OS,
> > that state is gone so I I don't see much value in using efifb once
> > you
> > have a real driver in the mix.  If you want a console on the host,
> > it's probably better to use 2 GPU or just load the real driver as
> > needed in both the host and guest.
> >
> > > >
> > > > Alex
> > > >
> > > >
> > > > >
> > > > > > But putting efi asside what Alex pointed out pretty much
> > > > > > breaks
> > > > > > your
> > > > > > neck trying to forward the device. You maybe could try to
> > > > > > hack
> > > > > > the
> > > > > > driver to use the PCIe BAR for framebuffer access, but that
> > > > > > might
> > > > > > be
> > > > > > quite a bit slower.
> > > > > >
> > > > > > Regards,
> > > > > > Christian.
> > > > > >
> > > > > > >
> > > > > > >> Alex
> > > > > > >>
> > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > >> <alexdeucher@gmail.com>
> > > > > > >> wrote:
> > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > >>> <ydirson@free.fr>
> > > > > > >>> wrote:
> > > > > > >>>> Alex wrote:
> > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > >>>>> <ydirson@free.fr>
> > > > > > >>>>> wrote:
> > > > > > >>>>>> Hi Alex,
> > > > > > >>>>>>
> > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > >>>>>>> integrated
> > > > > > >>>>>>> GPUs.  I
> > > > > > >>>>>>> don't
> > > > > > >>>>>>> know that it will work at all.  We had done a bit of
> > > > > > >>>>>>> testing but
> > > > > > >>>>>>> ran
> > > > > > >>>>>>> into the same issues with the PSP, but never had a
> > > > > > >>>>>>> chance
> > > > > > >>>>>>> to
> > > > > > >>>>>>> debug
> > > > > > >>>>>>> further because this feature is not productized.
> > > > > > >>>>>> ...
> > > > > > >>>>>>> You need a functional PSP to get the GPU driver up
> > > > > > >>>>>>> and
> > > > > > >>>>>>> running.
> > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > >>>>>>
> > > > > > >>>>>> I guess that if I want to have any chance to get the
> > > > > > >>>>>> PSP
> > > > > > >>>>>> working
> > > > > > >>>>>> I'm
> > > > > > >>>>>> going to need more details on it.  A quick search some
> > > > > > >>>>>> time
> > > > > > >>>>>> ago
> > > > > > >>>>>> mostly
> > > > > > >>>>>> brought reverse-engineering work, rather than official
> > > > > > >>>>>> AMD
> > > > > > >>>>>> doc.
> > > > > > >>>>>>   Are
> > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > >>>>> The driver code is pretty much it.
> > > > > > >>>> Let's try to shed some more light on how things work,
> > > > > > >>>> taking
> > > > > > >>>> as
> > > > > > >>>> excuse
> > > > > > >>>> psp_v12_0_ring_create().
> > > > > > >>>>
> > > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > > >>>> implemented
> > > > > > >>>> in
> > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > > > > >>>> [RW]REG32(),
> > > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > > >>> (VF),
> > > > > > >>> some
> > > > > > >>> registers are not available directly via the VF's MMIO
> > > > > > >>> aperture
> > > > > > >>> so
> > > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > > >>> passthrough
> > > > > > >>> this
> > > > > > >>> is not relevant.
> > > > > > >>>
> > > > > > >>>> It accesses registers in the MMIO range of the MP0 IP,
> > > > > > >>>> and
> > > > > > >>>> the
> > > > > > >>>> "MP0"
> > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > >>>> PSP-handling
> > > > > > >>>> code.
> > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The
> > > > > > >>>> MP0
> > > > > > >>>> version
> > > > > > >>> Yes.
> > > > > > >>>
> > > > > > >>>> reported at v11.0.3 by discovery seems to contradict the
> > > > > > >>>> use
> > > > > > >>>> of
> > > > > > >>>> v12.0
> > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > > > > >>>> something ?
> > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > >>>
> > > > > > >>>> More generally (and mostly out of curiosity while we're
> > > > > > >>>> at
> > > > > > >>>> it),
> > > > > > >>>> do we
> > > > > > >>>> have a way to match IPs listed at discovery time with
> > > > > > >>>> the
> > > > > > >>>> ones
> > > > > > >>>> used
> > > > > > >>>> in the driver ?
> > > > > > >>> In general, barring typos, the code is shared at the
> > > > > > >>> major
> > > > > > >>> version
> > > > > > >>> level.  The actual code may or may not need changes to
> > > > > > >>> handle
> > > > > > >>> minor
> > > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > > >>> versions
> > > > > > >>> from
> > > > > > >>> the
> > > > > > >>> ip discovery table to the code contained in the driver.
> > > > > > >>>
> > > > > > >>>> ---
> > > > > > >>>>
> > > > > > >>>> As for the register names, maybe we could have a short
> > > > > > >>>> explanation of
> > > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that
> > > > > > >>>> seems
> > > > > > >>>> to
> > > > > > >>>> be
> > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but
> > > > > > >>>> I'm
> > > > > > >>>> not
> > > > > > >>>> sure
> > > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > > >>>> Management
> > > > > > >>>> Network",
> > > > > > >>>> described in [0] as an internal bus.  Are we accessing
> > > > > > >>>> this
> > > > > > >>>> register
> > > > > > >>>> through this SMN ?
> > > > > > >>> These registers are just mailboxes for the PSP firmware.
> > > > > > >>>  All
> > > > > > >>> of
> > > > > > >>> the
> > > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > > >>> firmware.
> > > > > > >>>   They
> > > > > > >>> are basically scratch registers used to communicate
> > > > > > >>> between
> > > > > > >>> the
> > > > > > >>> driver
> > > > > > >>> and the PSP firmware.
> > > > > > >>>
> > > > > > >>>>
> > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > >>>>> the CPU and the rest of the platform.  The GPU driver
> > > > > > >>>>> just
> > > > > > >>>>> interacts
> > > > > > >>>>> with it for a few specific tasks:
> > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> > > > > > >>>>> applications
> > > > > > >>>>> that run on the PSP for specific functionality, e.g.,
> > > > > > >>>>> HDCP
> > > > > > >>>>> and
> > > > > > >>>>> content
> > > > > > >>>>> protection, etc.)
> > > > > > >>>>> 2. Validating and loading firmware for other engines on
> > > > > > >>>>> the
> > > > > > >>>>> SoC.
> > > > > > >>>>>   This
> > > > > > >>>>> is required to use those engines.
> > > > > > >>>> Trying to understand in more details how we start the
> > > > > > >>>> PSP
> > > > > > >>>> up, I
> > > > > > >>>> noticed
> > > > > > >>>> that psp_v12_0 has support for loading a sOS firmware,
> > > > > > >>>> but
> > > > > > >>>> never
> > > > > > >>>> calls
> > > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > > >>>> firmware
> > > > > > >>>> for
> > > > > > >>>> renoir
> > > > > > >>>> and green_sardine, which seem to be the only ASICs with
> > > > > > >>>> this
> > > > > > >>>> PSP
> > > > > > >>>> version.
> > > > > > >>>> Is it something that's just not been completely wired up
> > > > > > >>>> yet
> > > > > > >>>> ?
> > > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > > >>> firmware
> > > > > > >>> is
> > > > > > >>> part
> > > > > > >>> of
> > > > > > >>> the sbios image.  The driver doesn't load it.  We only
> > > > > > >>> load
> > > > > > >>> it on
> > > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > > >>> initialization.
> > > > > > >>>
> > > > > > >>>> That also rings a bell, that we have nothing about
> > > > > > >>>> Secure OS
> > > > > > >>>> in
> > > > > > >>>> the doc
> > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>> I'm not too familiar with the PSP's path to memory from
> > > > > > >>>>> the
> > > > > > >>>>> GPU
> > > > > > >>>>> perspective.  IIRC, most memory used by the PSP goes
> > > > > > >>>>> through
> > > > > > >>>>> carve
> > > > > > >>>>> out
> > > > > > >>>>> "vram" on APUs so it should work, but I would double
> > > > > > >>>>> check
> > > > > > >>>>> if
> > > > > > >>>>> there
> > > > > > >>>>> are any system memory allocations that used to interact
> > > > > > >>>>> with
> > > > > > >>>>> the PSP
> > > > > > >>>>> and see if changing them to vram helps.  It does work
> > > > > > >>>>> with
> > > > > > >>>>> the
> > > > > > >>>>> IOMMU
> > > > > > >>>>> enabled on bare metal, so it should work in passthrough
> > > > > > >>>>> as
> > > > > > >>>>> well
> > > > > > >>>>> in
> > > > > > >>>>> theory.
> > > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > > >>>> used
> > > > > > >>>> instead
> > > > > > >>>> of
> > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and
> > > > > > >>>> there
> > > > > > >>>> has
> > > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > > >>>> vram).
> > > > > > >>>> Changing it to vram does not make a difference, but then
> > > > > > >>>> the
> > > > > > >>>> only bo that seems to be used at that point is the one
> > > > > > >>>> for
> > > > > > >>>> the
> > > > > > >>>> psp ring,
> > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > >>>> surprised.
> > > > > > >>>>
> > > > > > >>>> Maybe I should double-check bo_create calls to hunt for
> > > > > > >>>> more
> > > > > > >>>> ?
> > > > > > >>> We looked into this a bit ourselves and ran into the same
> > > > > > >>> issues.
> > > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > > >>> make
> > > > > > >>> further
> > > > > > >>> progress, but this was not productized so neither team
> > > > > > >>> had
> > > > > > >>> the
> > > > > > >>> resources to delve further.
> > > > > > >>>
> > > > > > >>> Alex
> > > > > > >>>
> > > > > > >>>>
> > > > > > >>>> [0]
> > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > >
> > > > > >
> > > >
> >

Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson]

Alex wrote:
> On Thu, Jan 6, 2022 at 10:38 AM Yann Dirson <ydirson@free.fr> wrote:
> >
> > Alex wrote:
> > > > How is the stolen memory communicated to the driver ?  That
> > > > host
> > > > physical
> > > > memory probably has to be mapped at the same guest physical
> > > > address
> > > > for
> > > > the magic to work, right ?
> > >
> > > Correct.  The driver reads the physical location of that memory
> > > from
> > > hardware registers.  Removing this chunk of code from gmc_v9_0.c
> > > will
> > > force the driver to use the BAR,
> >
> > That would only be a workaround for a missing mapping of stolen
> > memory to the guest, right ?
> 
> 
> Correct. That will use the PCI BAR rather than the underlying
> physical
> memory for CPU access to the carve out region.
> 
> >
> >
> > > but I'm not sure if there are any
> > > other places in the driver that make assumptions about using the
> > > physical host address or not on APUs off hand.
> >
> > gmc_v9_0_vram_gtt_location() updates vm_manager.vram_base_offset
> > from
> > the same value.  I'm not sure I understand why in this case there
> > is
> > no reason to use the BAR while there are some in
> > gmc_v9_0_mc_init().
> >
> > vram_base_offset then gets used in several places:
> >
> > * amdgpu_gmc_init_pdb0, that seems likely enough to be problematic,
> >   right ?
> >   As a sidenote the XGMI offset added earlier gets substracted
> >   here to deduce vram base addr
> >   (a couple of new acronyms there: PDB, PDE -- page directory
> >   base/entry?)
> >
> > * amdgpu_ttm_map_buffer, amdgpu_vm_bo_update_mapping: those seem to
> > be
> >   as problematic
> >
> > * amdgpu_gmc_vram_mc2pa: until I got there I had assumed MC could
> > stand for
> >   "memory controller", but then "MC address of buffer" makes me
> >   doubt
> >
> >
> 
> MC = memory controller (as in graphics memory controller).
> 
> These are GPU addresses not CPU addresses so they should be fine.
> 
> > >
> > >         if ((adev->flags & AMD_IS_APU) ||
> > >             (adev->gmc.xgmi.supported &&
> > >              adev->gmc.xgmi.connected_to_cpu)) {
> > >                 adev->gmc.aper_base =
> > >                         adev->gfxhub.funcs->get_mc_fb_offset(adev)
> > >                         +
> > >                         adev->gmc.xgmi.physical_node_id *
> > >                         adev->gmc.xgmi.node_segment_size;
> > >                 adev->gmc.aper_size = adev->gmc.real_vram_size;
> > >         }
> >
> >
> > Now for the test... it does indeed seem to go much further, I even
> > loose the dom0's efifb to that black screen hopefully showing the
> > driver started to setup the hardware.  Will probably still have to
> > hunt down whether it still tries to use efifb afterwards (can't see
> > why it would not, TBH, given the previous behaviour where it kept
> > using it after the guest failed to start).
> >
> > The log shows many details about TMR loading
> >
> > Then as expected:
> >
> > [2022-01-06 15:16:09] <6>[    5.844589] amdgpu 0000:00:05.0:
> > amdgpu: RAP: optional rap ta ucode is not available
> > [2022-01-06 15:16:09] <6>[    5.844619] amdgpu 0000:00:05.0:
> > amdgpu: SECUREDISPLAY: securedisplay ta ucode is not available
> > [2022-01-06 15:16:09] <7>[    5.844639]
> > [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP
> > block <smu>...
> > [2022-01-06 15:16:09] <6>[    5.845515] amdgpu 0000:00:05.0:
> > amdgpu: SMU is initialized successfully!
> >
> >
> > not sure about that unhandled interrupt (and a bit worried about
> > messed-up logs):
> >
> > [2022-01-06 15:16:09] <7>[    6.010681] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ring_test_hel[2022-01-06 15:16:10] per [amdgpu]] ring
> > test on sdma0 succeeded
> > [2022-01-06 15:16:10] <7>[    6.010831] [drm:amdgpu_ih_process
> > [amdgpu]] amdgpu_ih_process: rptr 0, wptr 32
> > [2022-01-06 15:16:10] <7>[    6.011002] [drm:amdgpu_irq_dispatch
> > [amdgpu]] Unhandled interrupt src_id: 243
> >
> >
> > then comes a first error:
> >
> > [2022-01-06 15:16:10] <6>[    6.011785] [drm] Display Core
> > initialized with v3.2.149!
> > [2022-01-06 15:16:10] <6>[    6.012714] [drm] DMUB hardware
> > initialized: version=0x0101001C
> > [2022-01-06 15:16:10] <3>[    6.228263] [drm:dc_dmub_srv_wait_idle
> > [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> > [2022-01-06 15:16:10] <7>[    6.229125]
> > [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: freesync_module
> > init done 0000000076c7b459.
> > [2022-01-06 15:16:10] <7>[    6.229677]
> > [drm:amdgpu_dm_init.isra.0.cold [amdgpu]] amdgpu: hdcp_workqueue
> > init done 0000000087e28b47.
> > [2022-01-06 15:16:10] <7>[    6.229979]
> > [drm:amdgpu_dm_init.isra.0.cold [amdgpu]]
> > amdgpu_dm_connector_init()
> >
> > ... which we can see again several times later though the driver
> > seems sufficient to finish init:
> >
> > [2022-01-06 15:16:10] <6>[    6.615615] [drm] late_init of IP block
> > <smu>...
> > [2022-01-06 15:16:10] <6>[    6.615772] [drm] late_init of IP block
> > <gfx_v9_0>...
> > [2022-01-06 15:16:10] <6>[    6.615801] [drm] late_init of IP block
> > <sdma_v4_0>...
> > [2022-01-06 15:16:10] <6>[    6.615827] [drm] late_init of IP block
> > <dm>...
> > [2022-01-06 15:16:10] <3>[    6.801790] [drm:dc_dmub_srv_wait_idle
> > [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> > [2022-01-06 15:16:10] <7>[    6.806079] [drm:drm_minor_register
> > [drm]]
> > [2022-01-06 15:16:10] <7>[    6.806195] [drm:drm_minor_register
> > [drm]] new minor registered 128
> > [2022-01-06 15:16:10] <7>[    6.806223] [drm:drm_minor_register
> > [drm]]
> > [2022-01-06 15:16:10] <7>[    6.806289] [drm:drm_minor_register
> > [drm]] new minor registered 0
> > [2022-01-06 15:16:10] <7>[    6.806355]
> > [drm:drm_sysfs_connector_add [drm]] adding "eDP-1" to sysfs
> > [2022-01-06 15:16:10] <7>[    6.806424]
> > [drm:drm_dp_aux_register_devnode [drm_kms_helper]] drm_dp_aux_dev:
> > aux [AMDGPU DM aux hw bus 0] registered as minor 0
> > [2022-01-06 15:16:10] <7>[    6.806498]
> > [drm:drm_sysfs_hotplug_event [drm]] generating hotplug event
> > [2022-01-06 15:16:10] <6>[    6.806533] [drm] Initialized amdgpu
> > 3.42.0 20150101 for 0000:00:05.0 on minor 0
> >
> >
> 
> Looks like it initialized fine.  I guess the DMCUB firmware issues
> are
> not fatal.  Probably need input from one of the display guys on that.

Not sure what's the best way of getting display guys on board here,
splitting the thread under a new name could do, but then the thread
link is partially lost :)


I enabled a couple more logs, but that did not reveal much yet.

The "Error waiting for DMUB idle" gets more detailed, with this
systematically-identical dump:

[2022-01-09 11:27:58] <3>[   12.755512] [drm:dc_dmub_srv_wait_idle [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
[2022-01-09 11:27:58] <7>[   12.755696] [drm:dc_dmub_srv_log_diagnostic_data [amdgpu]] DMCUB STATE
[2022-01-09 11:27:58] <7>[   12.755696]     dmcub_version      : 01011c00
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [0]       : 00000003
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [1]       : 01011c00
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [2]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [3]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [4]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [5]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [6]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [7]       : deaddead
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [8]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch  [9]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [10]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [11]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [12]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [13]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [14]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     scratch [15]       : 00000000
[2022-01-09 11:27:58] <7>[   12.755696]     pc                 : 00000000





> 
> > At one point though a new problem shows: it seem to have issues
> > driving the CRTC in the end:
> >
> > [2022-01-06 15:16:25] <7>[   11.140807] amdgpu 0000:00:05.0:
> > [drm:drm_vblank_enable [drm]] enabling vblank on crtc 0, ret: 0
> > [2022-01-06 15:16:25] <3>[   11.329306] [drm:dc_dmub_srv_wait_idle
> > [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> > [2022-01-06 15:16:25] <3>[   11.524327] [drm:dc_dmub_srv_wait_idle
> > [amdgpu]] *ERROR* Error waiting for DMUB idle: status=3
> > [2022-01-06 15:16:25] <4>[   11.641814] [drm] Fence fallback timer
> > expired on ring comp_1.3.0
> > [2022-01-06 15:16:25] <7>[   11.641877] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.0
> > succeeded
> > [2022-01-06 15:16:25] <4>[   12.145804] [drm] Fence fallback timer
> > expired on ring comp_1.0.1
> > [2022-01-06 15:16:25] <7>[   12.145862] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.0.1
> > succeeded
> > [2022-01-06 15:16:25] <4>[   12.649771] [drm] Fence fallback timer
> > expired on ring comp_1.1.1
> > [2022-01-06 15:16:25] <7>[   12.649789] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.1.1
> > succeeded
> > [2022-01-06 15:16:25] <4>[   13.153815] [drm] Fence fallback timer
> > expired on ring comp_1.2.1
> > [2022-01-06 15:16:25] <7>[   13.153836] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.2.1
> > succeeded
> > [2022-01-06 15:16:25] <4>[   13.657756] [drm] Fence fallback timer
> > expired on ring comp_1.3.1
> > [2022-01-06 15:16:25] <7>[   13.657767] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on comp_1.3.1
> > succeeded
> > [2022-01-06 15:16:25] <7>[   13.657899]
> > [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Setting write pointer
> > [2022-01-06 15:16:25] <7>[   13.658008]
> > [drm:sdma_v4_0_ring_set_wptr [amdgpu]] Using doorbell -- wptr_offs
> > == 0x00000198 lower_32_bits(ring->wptr) << 2 == 0x00000100
> > upper_32_bits(ring->wptr) << 2 == 0x00000000
> > [2022-01-06 15:16:25] <7>[   13.658114]
> > [drm:sdma_v4_0_ring_set_wptr [amdgpu]] calling
> > WDOORBELL64(0x000001e0, 0x0000000000000100)
> > [2022-01-06 15:16:25] <4>[   14.161792] [drm] Fence fallback timer
> > expired on ring sdma0
> > [2022-01-06 15:16:25] <7>[   14.161811] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ib_ring_tests [amdgpu]] ib test on sdma0 succeeded
> > [2022-01-06 15:16:25] <3>[   21.609821]
> > [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]]
> > *ERROR* [CRTC:67:crtc-0] flip_done timed out
> >
> >
> > No visible change if I boot with efifb:off (aside from entering
> > LUKS
> > passphrase in the dark, that is).
> >
> >
> > Tried patching gmc_v9_0_vram_gtt_location() to use the BAR too [2],
> > but
> > that turns out to work even less:
> 
> 
> That won't work.  These are GPU addresses not CPU addresses.
> 
> >
> > [2022-01-06 16:27:48] <6>[    6.230166] amdgpu 0000:00:05.0:
> > amdgpu: SMU is initialized successfully!
> > [2022-01-06 16:27:48] <7>[    6.230168]
> > [drm:amdgpu_device_init.cold [amdgpu]] hw_init (phase2) of IP
> > block <gfx_v9_0>...
> > [2022-01-06 16:27:48] <6>[    6.231948] [drm] kiq ring mec 2 pipe 1
> > q 0
> > [2022-01-06 16:27:48] <7>[    6.231861] [drm:amdgpu_ih_process
> > [amdgpu]] amdgpu_ih_process: rptr 448, wptr 512
> > [2022-01-06 16:27:48] <7>[    6.231962]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq alloc'd 64
> > [2022-01-06 16:27:48] <7>[    6.232172]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size init: 256
> > [2022-01-06 16:27:48] <7>[    6.232344]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after set_res:
> > 248
> > [2022-01-06 16:27:48] <7>[    6.232530]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after map_q:
> > 192
> > [2022-01-06 16:27:48] <7>[    6.232725] [drm:amdgpu_ih_process
> > [amdgpu]] amdgpu_ih_process: rptr 512, wptr 544
> > [2022-01-06 16:27:48] <3>[    6.429974] amdgpu 0000:00:05.0:
> > [drm:amdgpu_ring_test_helper [amdgpu]] *ERROR* ring kiq_2.1.0 test
> > failed (-110)
> > [2022-01-06 16:27:48] <7>[    6.430167]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] kiq size after test: 0
> > [2022-01-06 16:27:48] <3>[    6.430353]
> > [drm:amdgpu_gfx_enable_kcq.cold [amdgpu]] *ERROR* KCQ enable
> > failed
> > [2022-01-06 16:27:48] <3>[    6.430532]
> > [drm:amdgpu_device_init.cold [amdgpu]] *ERROR* hw_init of IP block
> > <gfx_v9_0> failed -110
> > [2022-01-06 16:27:48] <3>[    6.430720] amdgpu 0000:00:05.0:
> > amdgpu: amdgpu_device_ip_init failed
> >
> >
> >
> >
> > As a sidenote, my warning on ring_alloc() being called twice
> > without
> > commiting or undoing [1] gets triggered.  Given the call chain it
> > looks
> > like this would happen in the previous usage of that ring, would
> > have to
> > dig deeper to understand that.  Unless I'm missing something and
> > this would
> > be legal ?
> 
> I don't remember off hand.
> 
> Alex
> 
> >
> > [2022-01-06 15:52:17] <4>[    5.929158] ------------[ cut here
> > ]------------
> > [2022-01-06 15:52:17] <4>[    5.929170] WARNING: CPU: 1 PID: 458 at
> > drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c:74
> > amdgpu_ring_alloc+0x62/0x70 [amdgpu]
> > [2022-01-06 15:52:17] <4>[    5.929323] Modules linked in:
> > ip6table_filter ip6table_mangle joydev ip6table_raw ip6_tables
> > ipt_REJECT nf_reject_ipv4 xt_state xt_conntrack iptable_filter
> > iptable_mangle iptable_raw xt_MASQUERADE iptable_nat nf_nat
> > nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 intel_rapl_msr
> > intel_rapl_common crct10dif_pclmul crc32_pclmul crc32c_intel
> > ghash_clmulni_intel amdgpu(+) iommu_v2 gpu_sched i2c_algo_bit
> > drm_ttm_helper ttm drm_kms_helper ehci_pci cec pcspkr ehci_hcd
> > i2c_piix4 serio_raw ata_generic pata_acpi xen_scsiback
> > target_core_mod xen_netback xen_privcmd xen_gntdev xen_gntalloc
> > xen_blkback fuse drm xen_evtchn bpf_preload ip_tables overlay
> > xen_blkfront
> > [2022-01-06 15:52:17] <4>[    5.929458] CPU: 1 PID: 458 Comm: sdma0
> > Not tainted 5.15.4-1.fc32.qubes.x86_64+ #8
> > [2022-01-06 15:52:17] <4>[    5.929474] Hardware name: Xen HVM
> > domU, BIOS 4.14.3 01/03/2022
> > [2022-01-06 15:52:17] <4>[    5.929487] RIP:
> > 0010:amdgpu_ring_alloc+0x62/0x70 [amdgpu]
> > [2022-01-06 15:52:17] <4>[    5.929628] Code: 87 28 02 00 00 48 8b
> > 82 b8 00 00 00 48 85 c0 74 05 e8 b2 ae 90 ee 44 89 e0 41 5c c3 0f
> > 0b 41 bc f4 ff ff ff 44 89 e0 41 5c c3 <0f> 0b 48 8b 57 08 eb bc
> > 66 0f 1f 44 00 00 0f 1f 44 00 00 85 f6 0f
> > [2022-01-06 15:52:17] <4>[    5.929667] RSP: 0018:ffffb129005f3dd8
> > EFLAGS: 00010206
> > [2022-01-06 15:52:17] <4>[    5.929678] RAX: 0000000000000060 RBX:
> > ffff96209112d230 RCX: 0000000000000050
> > [2022-01-06 15:52:17] <4>[    5.929693] RDX: ffffffffc0ac6c60 RSI:
> > 000000000000006d RDI: ffff96208c5eb8f8
> > [2022-01-06 15:52:17] <4>[    5.929707] RBP: ffff96209112d000 R08:
> > ffffb129005f3e50 R09: ffff96208c5eba98
> > [2022-01-06 15:52:17] <4>[    5.929722] R10: 0000000000000000 R11:
> > 0000000000000001 R12: ffff962090a0c780
> > [2022-01-06 15:52:17] <4>[    5.929736] R13: 0000000000000001 R14:
> > ffff96208c5eb8f8 R15: ffff96208c5eb970
> > [2022-01-06 15:52:17] <4>[    5.929752] FS:  0000000000000000(0000)
> > GS:ffff9620bcd00000(0000) knlGS:0000000000000000
> > [2022-01-06 15:52:17] <4>[    5.929768] CS:  0010 DS: 0000 ES: 0000
> > CR0: 0000000080050033
> > [2022-01-06 15:52:17] <4>[    5.929781] CR2: 00007c1130d0f860 CR3:
> > 00000000040c4000 CR4: 0000000000350ee0
> > [2022-01-06 15:52:17] <4>[    5.929797] Call Trace:
> > [2022-01-06 15:52:17] <4>[    5.929805]  <TASK>
> > [2022-01-06 15:52:17] <4>[    5.929812]
> >  amdgpu_ib_schedule+0xa9/0x540 [amdgpu]
> > [2022-01-06 15:52:17] <4>[    5.929956]  ?
> > _raw_spin_unlock_irqrestore+0xa/0x20
> > [2022-01-06 15:52:17] <4>[    5.929969]  amdgpu_job_run+0xce/0x1f0
> > [amdgpu]
> > [2022-01-06 15:52:17] <4>[    5.930131]  drm_sched_main+0x300/0x500
> > [gpu_sched]
> > [2022-01-06 15:52:17] <4>[    5.930146]  ? finish_wait+0x80/0x80
> > [2022-01-06 15:52:17] <4>[    5.930156]  ?
> > drm_sched_rq_select_entity+0xa0/0xa0 [gpu_sched]
> > [2022-01-06 15:52:17] <4>[    5.930171]  kthread+0x127/0x150
> > [2022-01-06 15:52:17] <4>[    5.930181]  ?
> > set_kthread_struct+0x40/0x40
> > [2022-01-06 15:52:17] <4>[    5.930192]  ret_from_fork+0x22/0x30
> > [2022-01-06 15:52:17] <4>[    5.930203]  </TASK>
> > [2022-01-06 15:52:17] <4>[    5.930208] ---[ end trace
> > cf0edb400b0116c7 ]---
> >
> >
> > [1]
> > https://github.com/ydirson/linux/commit/4a010943e74d6bf621bd9e72a7620a65af23ecc9
> > [2]
> > https://github.com/ydirson/linux/commit/e90230e008ce204d822f07e36b3c3e196d561c28
> >
> > >
> > >
> > >
> > > >
> > > > > > >
> > > > > > > > ... which brings me to a point that's been puzzling me
> > > > > > > > for
> > > > > > > > some
> > > > > > > > time, which is
> > > > > > > > that as the hw init fails, the efifb driver is still
> > > > > > > > using
> > > > > > > > the
> > > > > > > > framebuffer.
> > > > > > >
> > > > > > > No, it isn't. You are probably just still seeing the same
> > > > > > > screen.
> > > > > > >
> > > > > > > The issue is most likely that while efi was kicked out
> > > > > > > nobody
> > > > > > > re-programmed the display hardware to show something
> > > > > > > different.
> > > > > > >
> > > > > > > > Am I right in suspecting that efifb should get stripped
> > > > > > > > of
> > > > > > > > its
> > > > > > > > ownership of the
> > > > > > > > fb aperture first, and that if I don't get a black
> > > > > > > > screen
> > > > > > > > on
> > > > > > > > hw_init failure
> > > > > > > > that issue should be the first focus point ?
> > > > > > >
> > > > > > > You assumption with the black screen is incorrect. Since
> > > > > > > the
> > > > > > > hardware
> > > > > > > works independent even if you kick out efi you still have
> > > > > > > the
> > > > > > > same
> > > > > > > screen content, you just can't update it anymore.
> > > > > >
> > > > > > It's not only that the screen keeps its contents, it's that
> > > > > > the
> > > > > > dom0
> > > > > > happily continues updating it.
> > > > >
> > > > > If the hypevisor is using efifb, then yes that could be a
> > > > > problem
> > > > > as
> > > > > the hypervisor could be writing to the efifb resources which
> > > > > ends
> > > > > up
> > > > > writing to the same physical memory.  That applies to any GPU
> > > > > on
> > > > > a
> > > > > UEFI system.  You'll need to make sure efifb is not in use in
> > > > > the
> > > > > hypervisor.
> >
> > > >
> > > > That remark evokes several things to me.  First one is that
> > > > every
> > > > time
> > > > I've tried booting with efifb disabled in dom0, there was no
> > > > visible
> > > > improvements in the guest driver - i.i. I really have to dig
> > > > how
> > > > vram mapping
> > > > is performed and check things are as expected anyway.
> > >
> > > Ultimately you end up at the same physical memory.  efifb uses
> > > the
> > > PCI
> > > BAR which points to the same physical memory that the driver
> > > directly
> > > maps.
> > >
> > > >
> > > > The other is that, when dom0 cannot use efifb, entering a luks
> > > > key
> > > > is
> > > > suddenly less user-friendly.  But in theory I'd think we could
> > > > overcome
> > > > this by letting dom0 use efifb until ready to start the guest,
> > > > a
> > > > simple
> > > > driver unbind at the right moment should be expected to work,
> > > > right
> > > > ?
> > > > Going further and allowing the guest to use efifb on its own
> > > > could
> > > > possibly be more tricky (starting with a different state?) but
> > > > does
> > > > not seem to sound completely outlandish either - or does it ?
> > > >
> > >
> > > efifb just takes whatever hardware state the GOP driver in the
> > > pre-OS
> > > environment left the GPU in.  Once you have a driver loaded in
> > > the
> > > OS,
> > > that state is gone so I I don't see much value in using efifb
> > > once
> > > you
> > > have a real driver in the mix.  If you want a console on the
> > > host,
> > > it's probably better to use 2 GPU or just load the real driver as
> > > needed in both the host and guest.
> > >
> > > > >
> > > > > Alex
> > > > >
> > > > >
> > > > > >
> > > > > > > But putting efi asside what Alex pointed out pretty much
> > > > > > > breaks
> > > > > > > your
> > > > > > > neck trying to forward the device. You maybe could try to
> > > > > > > hack
> > > > > > > the
> > > > > > > driver to use the PCIe BAR for framebuffer access, but
> > > > > > > that
> > > > > > > might
> > > > > > > be
> > > > > > > quite a bit slower.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Christian.
> > > > > > >
> > > > > > > >
> > > > > > > >> Alex
> > > > > > > >>
> > > > > > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> > > > > > > >> <alexdeucher@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson
> > > > > > > >>> <ydirson@free.fr>
> > > > > > > >>> wrote:
> > > > > > > >>>> Alex wrote:
> > > > > > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson
> > > > > > > >>>>> <ydirson@free.fr>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>> Hi Alex,
> > > > > > > >>>>>>
> > > > > > > >>>>>>> We have not validated virtualization of our
> > > > > > > >>>>>>> integrated
> > > > > > > >>>>>>> GPUs.  I
> > > > > > > >>>>>>> don't
> > > > > > > >>>>>>> know that it will work at all.  We had done a bit
> > > > > > > >>>>>>> of
> > > > > > > >>>>>>> testing but
> > > > > > > >>>>>>> ran
> > > > > > > >>>>>>> into the same issues with the PSP, but never had
> > > > > > > >>>>>>> a
> > > > > > > >>>>>>> chance
> > > > > > > >>>>>>> to
> > > > > > > >>>>>>> debug
> > > > > > > >>>>>>> further because this feature is not productized.
> > > > > > > >>>>>> ...
> > > > > > > >>>>>>> You need a functional PSP to get the GPU driver
> > > > > > > >>>>>>> up
> > > > > > > >>>>>>> and
> > > > > > > >>>>>>> running.
> > > > > > > >>>>>> Ah, thanks for the hint :)
> > > > > > > >>>>>>
> > > > > > > >>>>>> I guess that if I want to have any chance to get
> > > > > > > >>>>>> the
> > > > > > > >>>>>> PSP
> > > > > > > >>>>>> working
> > > > > > > >>>>>> I'm
> > > > > > > >>>>>> going to need more details on it.  A quick search
> > > > > > > >>>>>> some
> > > > > > > >>>>>> time
> > > > > > > >>>>>> ago
> > > > > > > >>>>>> mostly
> > > > > > > >>>>>> brought reverse-engineering work, rather than
> > > > > > > >>>>>> official
> > > > > > > >>>>>> AMD
> > > > > > > >>>>>> doc.
> > > > > > > >>>>>>   Are
> > > > > > > >>>>>> there some AMD resources I missed ?
> > > > > > > >>>>> The driver code is pretty much it.
> > > > > > > >>>> Let's try to shed some more light on how things
> > > > > > > >>>> work,
> > > > > > > >>>> taking
> > > > > > > >>>> as
> > > > > > > >>>> excuse
> > > > > > > >>>> psp_v12_0_ring_create().
> > > > > > > >>>>
> > > > > > > >>>> First, register access through [RW]REG32_SOC15() is
> > > > > > > >>>> implemented
> > > > > > > >>>> in
> > > > > > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is
> > > > > > > >>>> basically a
> > > > > > > >>>> [RW]REG32(),
> > > > > > > >>>> except it has to be more complex in the SR-IOV case.
> > > > > > > >>>> Has the RLC anything to do with SR-IOV ?
> > > > > > > >>> When running the driver on a SR-IOV virtual function
> > > > > > > >>> (VF),
> > > > > > > >>> some
> > > > > > > >>> registers are not available directly via the VF's
> > > > > > > >>> MMIO
> > > > > > > >>> aperture
> > > > > > > >>> so
> > > > > > > >>> they need to go through the RLC.  For bare metal or
> > > > > > > >>> passthrough
> > > > > > > >>> this
> > > > > > > >>> is not relevant.
> > > > > > > >>>
> > > > > > > >>>> It accesses registers in the MMIO range of the MP0
> > > > > > > >>>> IP,
> > > > > > > >>>> and
> > > > > > > >>>> the
> > > > > > > >>>> "MP0"
> > > > > > > >>>> name correlates highly with MMIO accesses in
> > > > > > > >>>> PSP-handling
> > > > > > > >>>> code.
> > > > > > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?
> > > > > > > >>>>  The
> > > > > > > >>>> MP0
> > > > > > > >>>> version
> > > > > > > >>> Yes.
> > > > > > > >>>
> > > > > > > >>>> reported at v11.0.3 by discovery seems to contradict
> > > > > > > >>>> the
> > > > > > > >>>> use
> > > > > > > >>>> of
> > > > > > > >>>> v12.0
> > > > > > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I
> > > > > > > >>>> miss
> > > > > > > >>>> something ?
> > > > > > > >>> Typo in the ip discovery table on renoir.
> > > > > > > >>>
> > > > > > > >>>> More generally (and mostly out of curiosity while
> > > > > > > >>>> we're
> > > > > > > >>>> at
> > > > > > > >>>> it),
> > > > > > > >>>> do we
> > > > > > > >>>> have a way to match IPs listed at discovery time
> > > > > > > >>>> with
> > > > > > > >>>> the
> > > > > > > >>>> ones
> > > > > > > >>>> used
> > > > > > > >>>> in the driver ?
> > > > > > > >>> In general, barring typos, the code is shared at the
> > > > > > > >>> major
> > > > > > > >>> version
> > > > > > > >>> level.  The actual code may or may not need changes
> > > > > > > >>> to
> > > > > > > >>> handle
> > > > > > > >>> minor
> > > > > > > >>> revision changes in an IP.  The driver maps the IP
> > > > > > > >>> versions
> > > > > > > >>> from
> > > > > > > >>> the
> > > > > > > >>> ip discovery table to the code contained in the
> > > > > > > >>> driver.
> > > > > > > >>>
> > > > > > > >>>> ---
> > > > > > > >>>>
> > > > > > > >>>> As for the register names, maybe we could have a
> > > > > > > >>>> short
> > > > > > > >>>> explanation of
> > > > > > > >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69:
> > > > > > > >>>> that
> > > > > > > >>>> seems
> > > > > > > >>>> to
> > > > > > > >>>> be
> > > > > > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP,
> > > > > > > >>>> but
> > > > > > > >>>> I'm
> > > > > > > >>>> not
> > > > > > > >>>> sure
> > > > > > > >>>> of the "SMN" part -- that could refer to the "System
> > > > > > > >>>> Management
> > > > > > > >>>> Network",
> > > > > > > >>>> described in [0] as an internal bus.  Are we
> > > > > > > >>>> accessing
> > > > > > > >>>> this
> > > > > > > >>>> register
> > > > > > > >>>> through this SMN ?
> > > > > > > >>> These registers are just mailboxes for the PSP
> > > > > > > >>> firmware.
> > > > > > > >>>  All
> > > > > > > >>> of
> > > > > > > >>> the
> > > > > > > >>> C2PMSG registers functionality is defined by the PSP
> > > > > > > >>> firmware.
> > > > > > > >>>   They
> > > > > > > >>> are basically scratch registers used to communicate
> > > > > > > >>> between
> > > > > > > >>> the
> > > > > > > >>> driver
> > > > > > > >>> and the PSP firmware.
> > > > > > > >>>
> > > > > > > >>>>
> > > > > > > >>>>>   On APUs, the PSP is shared with
> > > > > > > >>>>> the CPU and the rest of the platform.  The GPU
> > > > > > > >>>>> driver
> > > > > > > >>>>> just
> > > > > > > >>>>> interacts
> > > > > > > >>>>> with it for a few specific tasks:
> > > > > > > >>>>> 1. Loading Trusted Applications (e.g., trusted
> > > > > > > >>>>> firmware
> > > > > > > >>>>> applications
> > > > > > > >>>>> that run on the PSP for specific functionality,
> > > > > > > >>>>> e.g.,
> > > > > > > >>>>> HDCP
> > > > > > > >>>>> and
> > > > > > > >>>>> content
> > > > > > > >>>>> protection, etc.)
> > > > > > > >>>>> 2. Validating and loading firmware for other
> > > > > > > >>>>> engines on
> > > > > > > >>>>> the
> > > > > > > >>>>> SoC.
> > > > > > > >>>>>   This
> > > > > > > >>>>> is required to use those engines.
> > > > > > > >>>> Trying to understand in more details how we start
> > > > > > > >>>> the
> > > > > > > >>>> PSP
> > > > > > > >>>> up, I
> > > > > > > >>>> noticed
> > > > > > > >>>> that psp_v12_0 has support for loading a sOS
> > > > > > > >>>> firmware,
> > > > > > > >>>> but
> > > > > > > >>>> never
> > > > > > > >>>> calls
> > > > > > > >>>> init_sos_microcode() - and anyway there is no sos
> > > > > > > >>>> firmware
> > > > > > > >>>> for
> > > > > > > >>>> renoir
> > > > > > > >>>> and green_sardine, which seem to be the only ASICs
> > > > > > > >>>> with
> > > > > > > >>>> this
> > > > > > > >>>> PSP
> > > > > > > >>>> version.
> > > > > > > >>>> Is it something that's just not been completely
> > > > > > > >>>> wired up
> > > > > > > >>>> yet
> > > > > > > >>>> ?
> > > > > > > >>> On APUs, the PSP is shared with the CPU so the PSP
> > > > > > > >>> firmware
> > > > > > > >>> is
> > > > > > > >>> part
> > > > > > > >>> of
> > > > > > > >>> the sbios image.  The driver doesn't load it.  We
> > > > > > > >>> only
> > > > > > > >>> load
> > > > > > > >>> it on
> > > > > > > >>> dGPUs where the driver is responsible for the chip
> > > > > > > >>> initialization.
> > > > > > > >>>
> > > > > > > >>>> That also rings a bell, that we have nothing about
> > > > > > > >>>> Secure OS
> > > > > > > >>>> in
> > > > > > > >>>> the doc
> > > > > > > >>>> yet (not even the acronym in the glossary).
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>> I'm not too familiar with the PSP's path to memory
> > > > > > > >>>>> from
> > > > > > > >>>>> the
> > > > > > > >>>>> GPU
> > > > > > > >>>>> perspective.  IIRC, most memory used by the PSP
> > > > > > > >>>>> goes
> > > > > > > >>>>> through
> > > > > > > >>>>> carve
> > > > > > > >>>>> out
> > > > > > > >>>>> "vram" on APUs so it should work, but I would
> > > > > > > >>>>> double
> > > > > > > >>>>> check
> > > > > > > >>>>> if
> > > > > > > >>>>> there
> > > > > > > >>>>> are any system memory allocations that used to
> > > > > > > >>>>> interact
> > > > > > > >>>>> with
> > > > > > > >>>>> the PSP
> > > > > > > >>>>> and see if changing them to vram helps.  It does
> > > > > > > >>>>> work
> > > > > > > >>>>> with
> > > > > > > >>>>> the
> > > > > > > >>>>> IOMMU
> > > > > > > >>>>> enabled on bare metal, so it should work in
> > > > > > > >>>>> passthrough
> > > > > > > >>>>> as
> > > > > > > >>>>> well
> > > > > > > >>>>> in
> > > > > > > >>>>> theory.
> > > > > > > >>>> I can see a single case in the PSP code where GTT is
> > > > > > > >>>> used
> > > > > > > >>>> instead
> > > > > > > >>>> of
> > > > > > > >>>> vram: to create fw_pri_bo when SR-IOV is not used
> > > > > > > >>>> (and
> > > > > > > >>>> there
> > > > > > > >>>> has
> > > > > > > >>>> to be a reason, since the SR-IOV code path does use
> > > > > > > >>>> vram).
> > > > > > > >>>> Changing it to vram does not make a difference, but
> > > > > > > >>>> then
> > > > > > > >>>> the
> > > > > > > >>>> only bo that seems to be used at that point is the
> > > > > > > >>>> one
> > > > > > > >>>> for
> > > > > > > >>>> the
> > > > > > > >>>> psp ring,
> > > > > > > >>>> which is allocated in vram, so I'm not too much
> > > > > > > >>>> surprised.
> > > > > > > >>>>
> > > > > > > >>>> Maybe I should double-check bo_create calls to hunt
> > > > > > > >>>> for
> > > > > > > >>>> more
> > > > > > > >>>> ?
> > > > > > > >>> We looked into this a bit ourselves and ran into the
> > > > > > > >>> same
> > > > > > > >>> issues.
> > > > > > > >>> We'd probably need to debug this with the PSP team to
> > > > > > > >>> make
> > > > > > > >>> further
> > > > > > > >>> progress, but this was not productized so neither
> > > > > > > >>> team
> > > > > > > >>> had
> > > > > > > >>> the
> > > > > > > >>> resources to delve further.
> > > > > > > >>>
> > > > > > > >>> Alex
> > > > > > > >>>
> > > > > > > >>>>
> > > > > > > >>>> [0]
> > > > > > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> > > > > > >
> > > > > > >
> > > > >
> > >
>