[Bug 1897481] Re: qemu crashes with VGA pass-through, e-GPU, nvidia 1060

[Sergiy K <1897481@bugs.launchpad.net>]

I recorded both lspci -vvvv and lspci -xxxx for the following
connections:

  - hotplug: when GPU is connected after the host was loaded
  - fresh: when GPU is connected before the host was started

The main difference is the following:

1c1
< # hotplug
---
> # fresh
6c6
< 	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
---
> 	Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
8c8
< 	Interrupt: pin A routed to IRQ 18
---
> 	Interrupt: pin A routed to IRQ 255
10,13c10,14
< 	Region 1: Memory at <unassigned> (64-bit, prefetchable) [disabled]
< 	Region 3: Memory at <unassigned> (64-bit, prefetchable) [disabled]
< 	Region 5: I/O ports at 4000 [size=128]
< 	Expansion ROM at f1400000 [virtual] [disabled] [size=512K]
---
> 	Region 0: Memory at f0000000 (32-bit, non-prefetchable) [size=16M]
> 	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=256M]
> 	Region 3: Memory at d0000000 (64-bit, prefetchable) [size=32M]
> 	Region 5: I/O ports at 4000 [disabled] [size=128]
> 	Expansion ROM at f1080000 [disabled] [size=512K]
30c31
< 		LnkSta:	Speed 5GT/s (downgraded), Width x1 (downgraded)
---
> 		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (downgraded)
35a37
> 			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
79c81
< 	Interrupt: pin B routed to IRQ 19
---
> 	Interrupt: pin B routed to IRQ 255
81c83
< 	Region 0: Memory at f1480000 (32-bit, non-prefetchable) [size=16K]
---
> 	Region 0: Memory at f1000000 (32-bit, non-prefetchable) [size=16K]
98c100
< 		LnkSta:	Speed 5GT/s (downgraded), Width x1 (downgraded)
---
> 		LnkSta:	Speed 2.5GT/s (downgraded), Width x1 (downgraded)
124,125c126,127

I can tell, that hotplug connects as 5GT/s and fresh - 2.5GT/s. And
there is an obvious difference between Regions.

The difference between lspci -xxxx but I don't know how to interpret the
result:

124,125c126,127
< 00: de 10 03 1c 01 00 10 00 a1 00 00 03 00 00 80 00
< 10: 00 00 00 00 0c 00 00 00 00 00 00 00 0c 00 00 00
---
> 00: de 10 03 1c 02 00 10 00 a1 00 00 03 10 00 80 00
> 10: 00 00 00 f0 0c 00 00 c0 00 00 00 00 0c 00 00 d0
127c129
< 30: 00 00 00 00 60 00 00 00 00 00 00 00 00 01 00 00
---
> 30: 00 00 f8 ff 60 00 00 00 00 00 00 00 ff 01 00 00
132c134
< 80: 10 29 09 00 03 3d 45 00 43 01 12 10 00 00 00 00
---
> 80: 10 29 09 00 03 3d 45 00 43 01 11 10 00 00 00 00
198c200
< 4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78
---
> 4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff
221c223
< 610: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
---
> 610: 01 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00
257c259
< 850: 00 00 00 00 78 00 00 00 ff 3f 00 00 00 00 00 00
---
> 850: 00 00 00 00 af 04 00 00 ff 3f 00 00 00 00 00 00
382,383c384,385
< 00: de 10 f1 10 02 00 10 00 a1 00 03 04 00 00 80 00
< 10: 00 00 48 f1 00 00 00 00 00 00 00 00 00 00 00 00
---
> 00: de 10 f1 10 02 00 10 00 a1 00 03 04 10 00 80 00
> 10: 00 00 00 f1 00 00 00 00 00 00 00 00 00 00 00 00
385c387
< 30: 00 00 00 00 60 00 00 00 00 00 00 00 00 02 00 00
---
> 30: 00 00 00 00 60 00 00 00 00 00 00 00 ff 02 00 00
390c392
< 80: 10 29 09 00 03 3d 45 00 43 01 12 10 00 00 00 00
---
> 80: 10 29 09 00 03 3d 45 00 43 01 11 10 00 00 00 00
456,457c458,459
< 4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 78
< 4b0: 78 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
---
> 4a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff
> 4b0: af 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00


** Attachment added: "lspci-hotplug"
   https://bugs.launchpad.net/qemu/+bug/1897481/+attachment/5417476/+files/lspci-hotplug

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1897481

Title:
  qemu crashes with VGA pass-through, e-GPU, nvidia 1060

Status in QEMU:
  New

Bug description:
  I try to pass-through nvidia 1060 6gb card, which is connected via
  ExpressCard (EXP-GDC converter).

  I can successfully run my virtual machine without pass-through, but
  when I try to add the devices, qemu crashes.

  The coredump contains:

  Stack trace of thread 3289311:
  #0  0x0000000000614c49 memory_region_update_container_subregions (qemu-system-x86_64 + 0x214c49)
  #1  0x00000000005c0e8c vfio_probe_nvidia_bar0_quirk (qemu-system-x86_64 + 0x1c0e8c)
  #2  0x00000000005bcec0 vfio_realize (qemu-system-x86_64 + 0x1bcec0)
  #3  0x000000000079b423 pci_qdev_realize (qemu-system-x86_64 + 0x39b423)
  #4  0x00000000006facda device_set_realized (qemu-system-x86_64 + 0x2facda)
  #5  0x0000000000887e57 property_set_bool (qemu-system-x86_64 + 0x487e57)
  #6  0x000000000088ac48 object_property_set (qemu-system-x86_64 + 0x48ac48)
  #7  0x000000000088d1d2 object_property_set_qobject (qemu-system-x86_64 + 0x48d1d2)
  #8  0x000000000088b1f7 object_property_set_bool (qemu-system-x86_64 + 0x48b1f7)
  #9  0x0000000000693785 qdev_device_add (qemu-system-x86_64 + 0x293785)
  #10 0x000000000061aad0 device_init_func (qemu-system-x86_64 + 0x21aad0)
  #11 0x000000000098c87b qemu_opts_foreach (qemu-system-x86_64 + 0x58c87b)
  #12 0x00000000006211cb qemu_init (qemu-system-x86_64 + 0x2211cb)
  #13 0x00000000005002aa main (qemu-system-x86_64 + 0x1002aa)
  #14 0x00007fce8af21152 __libc_start_main (libc.so.6 + 0x28152)
  #15 0x000000000050087e _start (qemu-system-x86_64 + 0x10087e)

  The whole running command is pretty long, since I use libvirt to
  manage my machines:

  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/bin \
  HOME=/var/lib/libvirt/qemu/domain-2-Win10 \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-Win10/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-Win10/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-Win10/.config \
  QEMU_AUDIO_DRV=spice \
  /usr/bin/qemu-system-x86_64 \
  -name guest=Win10,debug-threads=on \
  -S \
  -blockdev '{"driver":"file","filename":"/usr/share/edk2-ovmf/x64/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
  -blockdev '{"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/Win10_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
  -machine pc-q35-5.1,accel=kvm,usb=off,vmport=off,dump-guest-core=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
  -cpu host,migratable=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff \
  -m 8192 \
  -overcommit mem-lock=off \
  -smp 2,sockets=2,cores=1,threads=1 \
  -uuid 7043c77b-4903-4527-8089-9679d9a17fee \
  -no-user-config \
  -nodefaults \
  -chardev stdio,mux=on,id=charmonitor \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -rtc base=localtime,driftfix=slew \
  -global kvm-pit.lost_tick_policy=delay \
  -no-hpet \
  -no-shutdown \
  -global ICH9-LPC.disable_s3=1 \
  -global ICH9-LPC.disable_s4=1 \
  -boot strict=on \
  -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 \
  -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 \
  -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 \
  -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 \
  -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 \
  -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5 \
  -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 \
  -device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
  -blockdev '{"driver":"file","filename":"/home/sergiy/VirtualBox VMs/win4games.img","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"raw","file":"libvirt-2-storage"}' \
  -device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=1 \
  -blockdev '{"driver":"file","filename":"/home/sergiy/Downloads/Win10_2004_Ukrainian_x64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
  -device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1 \
  -chardev pty,id=charserial0 \
  -device isa-serial,chardev=charserial0,id=serial0 \
  -chardev spicevmc,id=charchannel0,name=vdagent \
  -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 \
  -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on \
  -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1 \
  -chardev spicevmc,id=charredir0,name=usbredir \
  -device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=1 \
  -chardev spicevmc,id=charredir1,name=usbredir \
  -device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=2 \
  -device vfio-pci,host=0000:04:00.0,id=hostdev0,bus=pci.4,multifunction=on,addr=0x0 \
  -device vfio-pci,host=0000:04:00.1,id=hostdev1,bus=pci.4,addr=0x0.0x1 \
  -device virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 \
  -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
  -msg timestamp=on

  I've forced vfio_pci module for the VGA, and ensured that lspci shows

    Kernel driver in use: vfio_pci

  My laptop is Thinkpad x230, that runs on Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz. 
  I run 5.8.6-1-MANJARO kernel and run QEMU emulator version 5.1.0.

  Thank you for your attention. I'd love to provide more information,
  but I don't know what else matters.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1897481/+subscriptions