Issues with IGD legacy passthrough with a HP Probook 430 G4 and other hardware

Issues with IGD legacy passthrough with a HP Probook 430 G4 and other hardware

[Tony Cheneau]

Hello folks,



A colleague and I are trying to get legacy passthrough to work with a HP 

Probook

430 G4 in BIOS mode. This is part of a "bigger" pet project where we are 

trying

to use legacy passthrough on all our laptop that have a working IOMMU and

compatible hardware. Ultimately, we trying to run a Windows 10 guest within 


that

VM.



We've experienced a success using a same generation IGD (on Lenovo Thinkpad

X260) where we got legacy passthrough to work and that encouraged us to 

push

things further. We're now asking for help because we can't seem to find how 


to

get it to work on an HP Probook G4. Qemu is stuck when loading the VBIOS.



Here is the command line for starting the VM:

 $ qemu-system-x86_64 -D /var/log/qemu.log \

  -serial file:/root/serial.logs -monitor stdio \

  -bios /root/seabios-debug.bin -smbios type=0 \

  -m 6G -enable-kvm -machine pc,accel=kvm \

  -rtc driftfix=slew,base=localtime \

  -cpu 

host,hv_relaxed,hv_reset,hv_vpindex,hv_runtime,hv_spinlocks=0x1fff,hv_vapic
,hv_time,kvm=off 

\

  -smp sockets=1,cores=2,threads=2 \

  -drive format=qcow2,file=/root/windows.qcow2,index=0,if=virtio \

  -vga none -display none -device 

vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x02,romfile=/root
/image.rom,x-igd-gms=1 

\

  -device 

virtio-input-host-pci,evdev=/dev/input/by-path/platform-i8042-serio-0-event
-kbd 

\

  -device 

virtio-input-host-pci,evdev=/dev/input/by-path/platform-i8042-serio-3-event
-mouse 

\

  -net nic,model=virtio,addr=0x14 -net user 



The image.rom VBIOS image has been extracted from a firmware update for HP

laptop (using UEFI tool) and patched with rom-fixer. The firmware update is

confirmed to be compatible with this laptop (it applied without bricking 

the

laptop). We also tried using VBIOS from other hardware, but qemu then 

rejects it

("Turning on vga text mode console" error message).



We compiled a Seabios with a debug serial console and obtained the 

following output:

	Changing serial settings was 0/0 now 3/0

	SeaBIOS (version rel-1.11.0-25-g5adc8bd)

	BUILD: gcc: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609 binutils: (GNU 


Binutils for Ubuntu) 2.26.1

	No Xen hypervisor found.

	enabling shadow ram

	RamSize: 0xc0000000 [cmos]

	malloc preinit

	Relocating init from 0x000d8160 to 0xbffada00 (size 75104)

	malloc init

	Found QEMU fw_cfg

	QEMU fw_cfg DMA interface supported

	Add romfile: etc/show-boot-menu (size=2)

	Add romfile: etc/irq0-override (size=1)

	Add romfile: etc/max-cpus (size=2)

	Add romfile: etc/numa-cpu-map (size=32)

	Add romfile: etc/numa-nodes (size=0)

	Add romfile: bootorder (size=0)

	Add romfile: etc/acpi/rsdp (size=36)

	Add romfile: etc/acpi/tables (size=131072)

	Add romfile: etc/boot-fail-wait (size=4)

	Add romfile: etc/e820 (size=60)

	Add romfile: etc/igd-bdsm-size (size=8)

	Add romfile: etc/igd-opregion (size=8192)

	Add romfile: etc/msr_feature_control (size=8)

	Add romfile: etc/smbios/smbios-anchor (size=31)

	Add romfile: etc/smbios/smbios-tables (size=387)

	Add romfile: etc/system-states (size=6)

	Add romfile: etc/table-loader (size=4096)

	Add romfile: etc/tpm/log (size=0)

	Add romfile: genroms/kvmvapic.bin (size=9216)

	RamBlock: addr 0x0000000000000000 len 0x00000000c0000000 [e820]

	RamBlock: addr 0x0000000100000000 len 0x00000000c0000000 [e820]

	Moving pm_base to 0x600

	init ivt

	init bda

	init bios32

	init PMM

	init PNPBIOS table

	init keyboard

	init mouse

	init pic

	math cp init

	pci setup

	=== PCI bus & bridge init ===

	PCI: pci_bios_init_bus_rec bus = 0x0

	=== PCI device probing ===

	PCI probe

	Found 14 PCI devices (max PCI bus is 00)

	=== PCI new allocation pass #1 ===

	PCI: check devices

	=== PCI new allocation pass #2 ===

	PCI: IO: c000 - c0ef

	PCI: 32: 00000000c0000000 - 00000000fec00000

	PCI: map device bdf=00:02.0  bar 4, addr 0000c000, size 00000040 [io]

	PCI: map device bdf=00:08.0  bar 0, addr 0000c040, size 00000040 [io]

	PCI: map device bdf=00:01.2  bar 4, addr 0000c080, size 00000020 [io]

	PCI: map device bdf=00:05.0  bar 0, addr 0000c0a0, size 00000020 [io]

	PCI: map device bdf=00:14.0  bar 0, addr 0000c0c0, size 00000020 [io]

	PCI: map device bdf=00:01.1  bar 4, addr 0000c0e0, size 00000010 [io]

	PCI: map device bdf=00:02.0  bar 0, addr fd000000, size 01000000 [mem]

	PCI: map device bdf=00:14.0  bar 6, addr fe000000, size 00040000 [mem]

	PCI: map device bdf=00:02.0  bar 6, addr fe040000, size 00010000 [mem]

	PCI: map device bdf=00:06.0  bar 0, addr fe050000, size 00004000 [mem]

	PCI: map device bdf=00:03.0  bar 1, addr fe054000, size 00001000 [mem]

	PCI: map device bdf=00:04.0  bar 1, addr fe055000, size 00001000 [mem]

	PCI: map device bdf=00:05.0  bar 1, addr fe056000, size 00001000 [mem]

	PCI: map device bdf=00:07.0  bar 0, addr fe057000, size 00001000 [mem]

	PCI: map device bdf=00:08.0  bar 1, addr fe058000, size 00001000 [mem]

	PCI: map device bdf=00:14.0  bar 1, addr fe059000, size 00001000 [mem]

	PCI: map device bdf=00:02.0  bar 2, addr e0000000, size 10000000 [prefme
m]

	PCI: map device bdf=00:03.0  bar 4, addr f0000000, size 00004000 [prefme
m]

	PCI: map device bdf=00:04.0  bar 4, addr f0004000, size 00004000 [prefme
m]

	PCI: map device bdf=00:05.0  bar 4, addr f0008000, size 00004000 [prefme
m]

	PCI: map device bdf=00:08.0  bar 4, addr f000c000, size 00004000 [prefme
m]

	PCI: map device bdf=00:14.0  bar 4, addr f0010000, size 00004000 [prefme
m]

	PCI: init bdf=00:00.0 id=8086:1237

	PCI: init bdf=00:01.0 id=8086:7000

	PIIX3/PIIX4 init: elcr=00 0c

	PCI: init bdf=00:01.1 id=8086:7010

	PCI: init bdf=00:01.2 id=8086:7020

	PCI: init bdf=00:01.3 id=8086:7113

	Using pmtimer, ioport 0x608

	PCI: init bdf=00:02.0 id=8086:5916

	Intel IGD OpRegion enabled at 0xbfffe000, size 8KB, dev 00:02.0

	Intel IGD BDSM enabled at 0xbf700000, size 8MB, dev 00:02.0

	PCI: init bdf=00:03.0 id=1af4:1052

	PCI: init bdf=00:04.0 id=1af4:1052

	PCI: init bdf=00:05.0 id=1af4:1012

	PCI: init bdf=00:06.0 id=8086:293e

	PCI: init bdf=00:07.0 id=1b36:0007

	PCI: init bdf=00:08.0 id=1af4:1001

	PCI: init bdf=00:14.0 id=1af4:1000

	PCI: init bdf=00:1f.0 id=8086:9d58

	PCI: Using 00:02.0 for primary VGA

	init smm

	init mtrr

	handle_smp: apic_id=0x1

	handle_smp: apic_id=0x3

	handle_smp: apic_id=0x2

	Found 4 cpu(s) max supported 4 cpu(s)

	init PIR table

	Copying PIR from 0xbffbfcc0 to 0x000f4ea0

	init MPTable

	Copying MPTABLE from 0x00006e60/bffa4470 to 0x000f4d80

	Copying SMBIOS entry point from 0x00006e60 to 0x000f4bd0

	load ACPI tables

	init timer

	WARNING - Timeout at wait_reg8:81!

	Scan for VGA option rom

	Running option rom at c000:0003





Attaching gdb on qemu only confirmed that we were looping "somewhere" in 

the

guest VM BIOS (judging from the address space between 0x6702 and 0x670C).



As a side note, we never managed to get the UPT mode to output on the 

displays

(even on the Thinkpad X260, where legacy PT works). Perhaps because we 

tried on

a Q35/UEFI platform back then. On the guest, we got rid of the "error 43" 

in

Windows after updating to the latest intel driver. Also, it always 

indicates 

the presence of a second monitor in the configuration dialogs (even without 


the

x-igd-opregion=on option which make me doubt the actual screen is even 

detected).

On the Lenovo X260, we were luckier with the Q35/Seabios combo: we managed

once to get the external display to work and even then the internal display 


had

brightness issues and was unreadable.



We identified some differences between our test hardware:

- HP Probook's processor is a i3-7100U (Kabylake), graphic chip is a Intel 


HD Graphics 620 (8086:5916)

- Lenovo X260's processor is a i3-6100U (Skylake), graphic chip is a Intel 


HD Graphics 520 GT2 (8086:1916)



On the Linux host, we are using:

- a mix of linux Kernel 4.15.15 (for some early tests) and 4.16 (for all 

the debug output in the email)

- Qemu 2.12 or git master (HEAD of master on 4/25/2018)



We believe that Qemu could currently be lacking support for Gen9 KBL_GT2

graphics chips and that some memory regions need to be "reserved/mapped" or 


some

"stolen memory" should be declared (we are clearly not expert here, we just

looked at how the previous commits on pci_quirks.c handled newer generation 


IGD

hardware in Qemu). We haven't ruled out yet that an interrupt could be 

triggered

and we would be enterring an interrupt loop. We are a little at odds on how 


to

direct our debug next:

- do you know ways to check for memory write error in Qemu (in case the 

device

  is trying to communicate with a memory region that have not been mapped)?

- how do we check for interrupt loop?



We are also more than open to other suggestions. :-)



Best regards,

Tony Cheneau

Re: [vfio-users] Issues with IGD legacy passthrough with a HP Probook 430 G4 and other hardware

[Alex Williamson]

On Thu, 26 Apr 2018 14:46:08 +0200
Tony Cheneau <tony.cheneau-jNfjcPZKvDhg9hUCZPvPmw@public.gmane.org> wrote:

> Hello folks,
> 
> A colleague and I are trying to get legacy passthrough to work with a HP 
> Probook
> 430 G4 in BIOS mode. This is part of a "bigger" pet project where we are 
> trying
> to use legacy passthrough on all our laptop that have a working IOMMU and
> compatible hardware. Ultimately, we trying to run a Windows 10 guest within 
> that
> VM.
> 
> We've experienced a success using a same generation IGD (on Lenovo Thinkpad
> X260) where we got legacy passthrough to work and that encouraged us to 
> push
> things further. We're now asking for help because we can't seem to find how 
> to
> get it to work on an HP Probook G4. Qemu is stuck when loading the VBIOS.
> 
> Here is the command line for starting the VM:
>  $ qemu-system-x86_64 -D /var/log/qemu.log \
>   -serial file:/root/serial.logs -monitor stdio \
>   -bios /root/seabios-debug.bin -smbios type=0 \
>   -m 6G -enable-kvm -machine pc,accel=kvm \
>   -rtc driftfix=slew,base=localtime \
>   -cpu 
> host,hv_relaxed,hv_reset,hv_vpindex,hv_runtime,hv_spinlocks=0x1fff,hv_vapic,hv_time,kvm=off 
> \
>   -smp sockets=1,cores=2,threads=2 \
>   -drive format=qcow2,file=/root/windows.qcow2,index=0,if=virtio \
>   -vga none -display none -device 
> vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x02,romfile=/root/image.rom,x-igd-gms=1 
> \
>   -device 
> virtio-input-host-pci,evdev=/dev/input/by-path/platform-i8042-serio-0-event-kbd 
> \
>   -device 
> virtio-input-host-pci,evdev=/dev/input/by-path/platform-i8042-serio-3-event-mouse 
> \
>   -net nic,model=virtio,addr=0x14 -net user 
> 
> The image.rom VBIOS image has been extracted from a firmware update for HP
> laptop (using UEFI tool) and patched with rom-fixer. The firmware update is
> confirmed to be compatible with this laptop (it applied without bricking 
> the
> laptop). We also tried using VBIOS from other hardware, but qemu then 
> rejects it
> ("Turning on vga text mode console" error message).

The VBIOS seems pretty suspect here, are you scraping it out of the
firmware update image because the laptop is running in UEFI mode and
doesn't expose a PCI ROM?  A strategy that I've used for such a case is
to enable legacy boot in the BIOS, boot a live CD and use that to dump
the ROM.  Stash it somewhere, revert the BIOS changes, use rom-fixer to
cleanup the new image and see if it behaves more reasonably.  (BTW,
kvm=off should only ever be necessary for GeForce assignment)
 
[snip]
> As a side note, we never managed to get the UPT mode to output on the 
> displays
> (even on the Thinkpad X260, where legacy PT works). Perhaps because we 
> tried on
> a Q35/UEFI platform back then. On the guest, we got rid of the "error 43" 
> in
> Windows after updating to the latest intel driver. Also, it always 
> indicates 
> the presence of a second monitor in the configuration dialogs (even without 
> the
> x-igd-opregion=on option which make me doubt the actual screen is even 
> detected).

Display output in UPT mode requires opregion support, but OVMF doesn't
include any support for the fw_cfg used to handle that, so it would be
expected not to get output in this configuration.

> On the Lenovo X260, we were luckier with the Q35/Seabios combo: we managed
> once to get the external display to work and even then the internal display 
> had
> brightness issues and was unreadable.

It might depend on the system how brightness is wired, it's not
entirely clear that a VM owning the GPU also necessarily owns the
hardware necessary for brightness.

> We identified some differences between our test hardware:
> - HP Probook's processor is a i3-7100U (Kabylake), graphic chip is a Intel 
> HD Graphics 620 (8086:5916)
> - Lenovo X260's processor is a i3-6100U (Skylake), graphic chip is a Intel 
> HD Graphics 520 GT2 (8086:1916)
> 
> On the Linux host, we are using:
> - a mix of linux Kernel 4.15.15 (for some early tests) and 4.16 (for all 
> the debug output in the email)
> - Qemu 2.12 or git master (HEAD of master on 4/25/2018)
> 
> We believe that Qemu could currently be lacking support for Gen9 KBL_GT2
> graphics chips and that some memory regions need to be "reserved/mapped" or 
> some
> "stolen memory" should be declared (we are clearly not expert here, we just
> looked at how the previous commits on pci_quirks.c handled newer generation 
> IGD
> hardware in Qemu).

Entirely possible.  IIRC we initially had a lot or reports of corrupted
displays with the i3-6100U but I think we fixed that by recognizing a
different page table format when programming the GTT.  Intel seems to
change the GPU hardware on a whim, but I haven't specifically heard of
issues with Kabylake.

> We haven't ruled out yet that an interrupt could be 
> triggered
> and we would be enterring an interrupt loop. We are a little at odds on how 
> to
> direct our debug next:
> - do you know ways to check for memory write error in Qemu (in case the 
> device
>   is trying to communicate with a memory region that have not been mapped)?

Uncomment memory.c://#define DEBUG_UNASSIGNED

You'll also want to enable vfio* tracing and configure the vfio-device
with x-no-mmap=on to see all the interactions with the device.

> - how do we check for interrupt loop?

I don't think the VBIOS is doing anything with device interrupts but
there are a series of vfio-pci x-no-kvm-foo options that disable
interrupt bypass through KVM so you can see them in the traces.  Good
luck!

Alex