On Fri, Jan 19, 2024 at 6:06 AM Roger Pau Monné <roger.pau@citrix.com>
wrote:

> On Fri, Jan 19, 2024 at 02:44:35AM -0500, Patrick Plenefisch wrote:
> > On Thu, Jan 18, 2024 at 7:41 AM Roger Pau Monné <roger.pau@citrix.com>
> > wrote:
> >
> > >
> > > From that environment (PVH dom0) can you see if you can dump the
> > > contents of the VFCT table?  I don't have a system with that table, so
> > > not sure if this will work (because iasl is unlikely to know how to
> > > decode it):
> > >
> > > # acpidump -n VFCT -o table.dump
> > > # acpixtract -a table.dump
> > > # iasl -d vfct.dat
> > > # cat vfct.dsl
> > >
> > > Would be good if you can compare the output from what you get on a PV
> > > dom0 or when running native Linux.
> > >
> > > I'm also adding some AMD folks, as IIRC they did some fixes to Linux
> > > in order to get some AMD graphics cards running on a PVH dom0, maybe
> > > they can provide some additional input.
> > >
> > >
> > Well, this is pretty weird. I'll go into more details because it may be
> > relevant.
>
> Wow, you have certainly gone out of the beaten path here.
>

Yeah, I wasn't expecting this many issues for being an early adopter of
Threaderipper 7000!


>
> > I had been working with ASRock support ever since I got my brand
> > new motherboard because I couldn't see the BIOS/UEFI screens. I could
> boot
> > up, and once native linux took control amdgpu got the screens/gpu working
> > fine. I finally managed to be able to see the UEFI/BIOS setup screens by
> > patching my VBIOS: I extracted the VBIOS image of a cheap R5 430 OEM, ran
> > GOPupd to update the VBIOS UEFI component (GOP) from version 1.60 to
> 1.67.
> > That allowed the UEFI to actually initialize and use a screen. However, I
> > later realized that only 1 monitor was lighting up in the bios: my
> monitor
> > plugged into the Radeon RX 480 that was still on VBIOS GOP 1.60. It
> appears
> > the GOP was initializing the RX 480 too, despite not being flashed with
> the
> > latest itself. I am working on an email to asrock support about that.
> Once
> > I get into linux (native or PV), both monitors light up as expected.
> Also,
> > If I boot linux PVH from grub, they also work.
>
> OK, that's good, so that would be UEFI -> grub -> Xen -> Linux PVH?
>

Correct. Inserting grub into the chain "fixes" the acpi tables and things
work correctly.


>
> > Those usage scenarios have
> > acpidump output as identical. Booting linux PVH directly from UEFI (no
> > grub), the monitors go to sleep on the RX 480, and amdgpu errors out
> about
> > VFCT, but the R5 430 OEM does still have output. Interestingly, there is
> a
> > different screen mode booting UEFI+PVH, the characters are basically
> > squares, with more vertical lines than "default", maybe close to native
> > screen resolution, but horizontally it's still "default". Booting from
> grub
> > gives everything in the "default" resolution.
>
> Hm, maybe we are not passing the correct video information to Linux
> dom0 when booted from UEFI.  I'm afraid I don't have such setup
> myself, so it's going to be hard to test.
>
> To clarify, the output from Xen is fine, is the video output from
> Linux dom0 in PVH mode that's corrupt?
>

Xen output in EFI+PV/GRUB+PV/GRUB+PVH: Rectangular letters (expected/good)
Linux output in EFI+PV/GRUB+PV/GRUB+PVH: Rectangular letters -> kms fb ->
X11 (using this mode as I type this email right now)
Xen output in EFI+PVH: Squashed square letters (unexpected)
Linux output in EFI+PVH: No output on RX 480, but R5 430 shows a (static?)
login console on hvc0 that I can't interact with.


>
> >
> > So what is in the VFCT Table? VFCT contains the non-GOP VIOS image of my
> > Radeon RX 480 GPU. You can compare it to the VBIOS hosted at
> > https://www.techpowerup.com/vgabios/185789/msi-rx480-4096-160720
> (Compare
> > the end at E667 in the VFCT table to E5ff in that vbios link). I find
> this
> > extra suspicious due to the above.
> >
> > Now for the extra horrible things:
> >
> > UEFI-booted PVH Linux doesn't support keyboard getty input, and at least
> > some of the USB devices are not in lsusb. It also decided to vanish one
> of
> > my HDD's. The `reboot` command hangs. The Power button doesn't do
> anything.
>
> Yes, it does seem Lunux has issues reserving some BARs:
>
> [    6.520615] ahci 0000:07:00.1: version 3.0
> [    6.520701] ahci 0000:07:00.1: BAR 5: can't reserve [mem
> 0xf0e00000-0xf0e007ff]
> ...
> [   17.130099] ccp 0000:06:00.5: enabling device (0000 -> 0002)
> [   17.137025] ccp 0000:06:00.5: BAR 2: can't reserve [mem
> 0xf0b00000-0xf0bfffff]
> [   17.145210] ccp 0000:06:00.5: pcim_iomap_regions failed (-16)
> [   17.151868] ccp 0000:06:00.5: initialization failed
> [   17.157615] ccp: probe of 0000:06:00.5 failed with error -16
> ...
> [   17.993532] snd_hda_intel 0000:01:00.1: enabling device (0000 -> 0002)
> [   18.001207] snd_hda_intel 0000:01:00.1: Force to non-snoop mode
> [   18.007997] snd_hda_intel 0000:01:00.1: BAR 0: can't reserve [mem
> 0xf0f60000-0xf0f63fff 64bit]
> [   18.018053] snd_hda_intel 0000:06:00.7: enabling device (0000 -> 0002)
> [   18.025723] snd_hda_intel 0000:06:00.7: BAR 0: can't reserve [mem
> 0xf0d00000-0xf0d07fff]
> [   18.033679] snd_hda_intel 0000:41:00.1: enabling device (0000 -> 0002)
> [   18.043165] snd_hda_intel 0000:41:00.1: Force to non-snoop mode
>
> I bet this because Xen balloon driver has started picking up those
> regions in order to map foreign pages.
>
> > There are several stack traces in dmesg. But Alt-SysRq-B does reboot!
> > Luckily I could ssh in and capture the ACPI tables. They are much
> smaller,
> > and VFCT is empty.  Booting back to one of the working scenarios (direct
> > linux, Grub PV, Grub PVH, UEFI PV), all of this is normal.
> >
> > I've attached:
> >
> > xenboot.log which is the serial log of xen+linux booting in UEFI PVH
> > (kernel is debian's config, but patched to start at 2MiB)
> > dmesg.txt which is the linux dmesg that contains some nice stack traces
> > (kernel is debian's config, but patched to start at 2MiB)
> > efipvh-tables.dump is ALL acpi tables from UEFI+PVH mode (acpidump -o
> > efipvh-tables.dump)
> > working-tables.dump is ALL acpi tables from the other modes (acpidump -o
> > working-tables.dump)
> > efipvh-vfct.dump is attached in spirit, as it is 0 bytes long (acpidump
> -n
> > VFCT -o efipvh-vfct.dump)
> >
> > I ran iasl, but it just said **** Unknown ACPI table signature [VFCT] and
> > spat out the raw data table, nothing interesting
> >
> > Something I can try, but have been nervous to try due to GOPupd warnings
> is
> > to also flash the 1.67 GOP to the VBIOS on the RX 480. The R5 430 OEM had
> > no such warnings.
> >
> > Patrick
>
> > [    0.000000] Linux version 6.1.69 (root@pollux) (gcc (Debian
> 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #10 SMP
> PREEMPT_DYNAMIC Wed Jan 17 23:44:31 EST 2024
> > [    0.000000] Command line:
> root=UUID=922b10f2-a826-47fb-ab38-836f9b397ff7 ro rootflags=subvol=@rootfs
> earlyprintk=xen console=hvc0
> > [    0.000000] [Firmware Bug]: TSC doesn't count with P0 frequency!
> > [    0.000000] BIOS-provided physical RAM map:
> > [    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000003ffffff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x0000000004000000-0x0000000004045fff]
> ACPI NVS
> > [    0.000000] BIOS-e820: [mem 0x0000000004046000-0x0000000009afefff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x0000000009aff000-0x0000000009ffffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x000000000a000000-0x000000000affffff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x000000000b000000-0x000000000b020fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x000000000b021000-0x00000000a04e8fff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x00000000a04e9000-0x00000000a64e8fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000a64e9000-0x00000000a747efff]
> ACPI data
> > [    0.000000] BIOS-e820: [mem 0x00000000a747f000-0x00000000a947efff]
> ACPI NVS
> > [    0.000000] BIOS-e820: [mem 0x00000000a947f000-0x00000000addfefff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000addff000-0x00000000afffab9b]
> usable
> > [    0.000000] BIOS-e820: [mem 0x00000000afffab9c-0x00000000afffaf17]
> ACPI data
> > [    0.000000] BIOS-e820: [mem 0x00000000afffb000-0x00000000bfffffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000df200000-0x00000000df2fffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fea00000-0x00000000feafffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fec10000-0x00000000fec10fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fec30000-0x00000000fec30fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fed00000-0x00000000fed00fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000fed80000-0x00000000fed8ffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff]
> reserved
> > [    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000025dee2fff]
> usable
> > [    0.000000] BIOS-e820: [mem 0x000000025dee3000-0x000000403dbbffff]
> unusable
> > [    0.000000] BIOS-e820: [mem 0x000000403dbc0000-0x000000403fffffff]
> reserved
> > [    0.000000] printk: bootconsole [xenboot0] enabled
> > [    0.000000] NX (Execute Disable) protection: active
> > [    0.000000] efi: EFI v2.90 by American Megatrends
> > [    0.000000] efi: ACPI=0xa9463000 ACPI 2.0=0xa9463014
> SMBIOS=0xad9f4000 SMBIOS 3.0=0xad9f3000 MEMATTR=0x99e26698 ESRT=0xa7466018
> > [    0.000000] secureboot: Secure boot disabled
> > [    0.000000] SMBIOS 3.6.0 present.
> > [    0.000000] DMI: To Be Filled By O.E.M. TRX50 WS/TRX50 WS, BIOS 7.08
> 01/16/2024
> > [    0.000000] Hypervisor detected: Xen HVM
> > [    0.000000] Xen version 4.17.
> > [    0.000003] HVMOP_pagetable_dying not supported
> > [    0.043916] tsc: Fast TSC calibration failed
> > [    0.048843] tsc: Detected 4199.960 MHz processor
> > [    0.054365] e820: update [mem 0x00000000-0x00000fff] usable ==>
> reserved
> > [    0.054368] e820: remove [mem 0x000a0000-0x000fffff] usable
> > [    0.054375] last_pfn = 0x25dee3 max_arch_pfn = 0x400000000
> > [    0.062033] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC-
> WT
> > [    0.070417] CPU MTRRs all blank - virtualized system.
> > [    0.076209] last_pfn = 0xafffa max_arch_pfn = 0x400000000
> > [    0.085594] Using GB pages for direct mapping
> > [    0.091184] RAMDISK: [mem 0x04046000-0x068cdfff]
> > [    0.096533] ACPI: Early table checksum verification disabled
> > [    0.103248] ACPI: RSDP 0x00000000AFFFAB9C 000024 (v02 AMD   )
> > [    0.109716] ACPI: XSDT 0x00000000AFFFABC0 0000C4 (v01 AMD    A M I
> 00000001 AMI  01000013)
> > [    0.119663] ACPI: APIC 0x00000000AFFFAC84 00037C (v04 AMD    A M I
> 00000001 AMI  00010013)
> > [    0.129538] ACPI: FACP 0x00000000A747C000 000114 (v06 AMD    A M I
> 00000001 AMI  00010013)
> > [    0.139250] ACPI: DSDT 0x00000000A746E000 00D2F6 (v02 AMD    A M I
> 00000001 INTL 20230331)
> > [    0.148717] ACPI: FACS 0x00000000A945A000 000040
> > [    0.153805] ACPI: SSDT 0x00000000A747E000 0009CC (v02 AMD    BOULDERG
> 00000002 MSFT 04000000)
> > [    0.163309] ACPI: SSDT 0x00000000A747D000 000067 (v02 AMD    CPMDSM
>  00000001 INTL 20230331)
> > [    0.172813] ACPI: MCFG 0x00000000A746C000 00003C (v01 AMD    A M I
> 00000001 MSFT 00010013)
> > [    0.182351] ACPI: SSDT 0x00000000A746B000 0005C5 (v02 AMD    CPUSSDT
> 00000001 AMI  00000001)
> > [    0.191806] ACPI: SSDT 0x00000000A7468000 001B53 (v02 AMD    CPMRAS
>  00000001 INTL 20230331)
> > [    0.201308] ACPI: FPDT 0x00000000A7465000 000044 (v01 AMD    A M I
> 01072009 AMI  01000013)
> > [    0.210819] ACPI: SSDT 0x00000000A7462000 002448 (v02 AMD    GPP_PME_
> 00000001 INTL 20230331)
> > [    0.220343] ACPI: SSDT 0x00000000A745F000 002448 (v02 AMD    GPP_PME_
> 00000001 INTL 20230331)
> > [    0.229884] ACPI: SSDT 0x00000000A745C000 002448 (v02 AMD    GPP_PME_
> 00000001 INTL 20230331)
> > [    0.239351] ACPI: SSDT 0x00000000A7459000 002448 (v02 AMD    GPP_PME_
> 00000001 INTL 20230331)
> > [    0.248780] ACPI: SSDT 0x00000000A743E000 00A40E (v02 AMD    AMD CPU
> 00000001 AMD  00000001)
> > [    0.258288] ACPI: SSDT 0x00000000A684C000 0006D4 (v02 AMD    CPMWLRC
> 00000001 INTL 20230331)
> > [    0.267815] ACPI: SSDT 0x00000000A6842000 00982F (v02 AMD    CPMCMN
>  00000001 INTL 20230331)
> > [    0.277339] ACPI: SSDT 0x00000000A683C000 002387 (v02 AMD    AOD
> 00000001 INTL 20230331)
> > [    0.286840] ACPI: SSDT 0x00000000A683B000 000500 (v02 AMD    MEMTOOL0
> 00000002 INTL 20230331)
> > [    0.296283] ACPI: SSDT 0x00000000A683A000 00096A (v02 AMD    CPMMSOSC
> 00000001 INTL 20230331)
> > [    0.305841] ACPI: SSDT 0x00000000A6839000 000B72 (v02 AMD    CPMACPV6
> 00000001 INTL 20230331)
> > [    0.315288] ACPI: SSDT 0x00000000A6838000 00044E (v02 AMD    AmdTable
> 00000001 INTL 20230331)
> > [    0.324802] ACPI: Reserving APIC table memory at [mem
> 0xafffac84-0xafffafff]
> > [    0.332592] ACPI: Reserving FACP table memory at [mem
> 0xa747c000-0xa747c113]
> > [    0.340419] ACPI: Reserving DSDT table memory at [mem
> 0xa746e000-0xa747b2f5]
> > [    0.348189] ACPI: Reserving FACS table memory at [mem
> 0xa945a000-0xa945a03f]
> > [    0.356058] ACPI: Reserving SSDT table memory at [mem
> 0xa747e000-0xa747e9cb]
> > [    0.363906] ACPI: Reserving SSDT table memory at [mem
> 0xa747d000-0xa747d066]
> > [    0.371757] ACPI: Reserving MCFG table memory at [mem
> 0xa746c000-0xa746c03b]
> > [    0.379576] ACPI: Reserving SSDT table memory at [mem
> 0xa746b000-0xa746b5c4]
> > [    0.387437] ACPI: Reserving SSDT table memory at [mem
> 0xa7468000-0xa7469b52]
> > [    0.395337] ACPI: Reserving FPDT table memory at [mem
> 0xa7465000-0xa7465043]
> > [    0.403153] ACPI: Reserving SSDT table memory at [mem
> 0xa7462000-0xa7464447]
> > [    0.410989] ACPI: Reserving SSDT table memory at [mem
> 0xa745f000-0xa7461447]
> > [    0.418802] ACPI: Reserving SSDT table memory at [mem
> 0xa745c000-0xa745e447]
> > [    0.426642] ACPI: Reserving SSDT table memory at [mem
> 0xa7459000-0xa745b447]
> > [    0.434529] ACPI: Reserving SSDT table memory at [mem
> 0xa743e000-0xa744840d]
> > [    0.442393] ACPI: Reserving SSDT table memory at [mem
> 0xa684c000-0xa684c6d3]
> > [    0.450203] ACPI: Reserving SSDT table memory at [mem
> 0xa6842000-0xa684b82e]
> > [    0.458038] ACPI: Reserving SSDT table memory at [mem
> 0xa683c000-0xa683e386]
> > [    0.465853] ACPI: Reserving SSDT table memory at [mem
> 0xa683b000-0xa683b4ff]
> > [    0.473805] ACPI: Reserving SSDT table memory at [mem
> 0xa683a000-0xa683a969]
> > [    0.481626] ACPI: Reserving SSDT table memory at [mem
> 0xa6839000-0xa6839b71]
> > [    0.489475] ACPI: Reserving SSDT table memory at [mem
> 0xa6838000-0xa683844d]
>
> Weird, there's no VFCT listed here.
>
> Thanks, Roger.
>