* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
[not found] <alpine.DEB.2.21.1906241135080.2468@sstabellini-ThinkPad-T480s>
@ 2019-06-24 18:47 ` Stefano Stabellini
2019-06-25 7:44 ` Roger Pau Monné
2019-06-26 6:17 ` Juergen Gross
[not found] ` <f1b992ab-9e1d-0e37-ebb4-37fc609cfb5d@suse.com>
1 sibling, 2 replies; 11+ messages in thread
From: Stefano Stabellini @ 2019-06-24 18:47 UTC (permalink / raw)
To: Stefano Stabellini
Cc: jgross, wl, andrew.cooper3, jbeulich, xen-devel, boris.ostrovsky,
roger.pau, chao.gao
+ xen-devel
On Mon, 24 Jun 2019, Stefano Stabellini wrote:
> Hi all,
>
> I might have found a bug with PCI passthrough to a Linux HVM guest on
> x86 with Xen 4.12. It is not easy for me to get access, and especially
> change components, on this particular system, and I don't have access to
> other x86 boxes at the moment, so apologies for the partial information
> report. The setup is as follow:
>
> - two PCI devices have been assigned to a HVM guest, everything is fine
> - reboot the guest from inside, i.e. `reboot' in Linux
> - after the reboot completes, only one device is assigned
>
> Before the reboot, I see all the appropriate xenstore entries for both
> devices. Everything is fine. After the reboot, I can only see the
> xenstore entries of one device. It is as if the other device
> "disappeared" without throwing any errors.
>
> Have you seen this before? Do you know if it has been fixed in staging?
> I noticed this fix which seems to be very relevant:
>
> https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>
> but it is already included in 4.12.
>
> Thank you,
>
> Stefano
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-24 18:47 ` [Xen-devel] PCI Passthrough bug with x86 HVM Stefano Stabellini
@ 2019-06-25 7:44 ` Roger Pau Monné
2019-06-25 22:14 ` Stefano Stabellini
2019-06-26 6:17 ` Juergen Gross
1 sibling, 1 reply; 11+ messages in thread
From: Roger Pau Monné @ 2019-06-25 7:44 UTC (permalink / raw)
To: Stefano Stabellini
Cc: jgross, wl, andrew.cooper3, jbeulich, xen-devel, boris.ostrovsky,
chao.gao
On Mon, Jun 24, 2019 at 11:47:09AM -0700, Stefano Stabellini wrote:
> + xen-devel
>
> On Mon, 24 Jun 2019, Stefano Stabellini wrote:
> > Hi all,
> >
> > I might have found a bug with PCI passthrough to a Linux HVM guest on
> > x86 with Xen 4.12. It is not easy for me to get access, and especially
> > change components, on this particular system, and I don't have access to
> > other x86 boxes at the moment, so apologies for the partial information
> > report. The setup is as follow:
> >
> > - two PCI devices have been assigned to a HVM guest, everything is fine
> > - reboot the guest from inside, i.e. `reboot' in Linux
> > - after the reboot completes, only one device is assigned
Can you provide the xl debug log of the whole process?
> > Before the reboot, I see all the appropriate xenstore entries for both
> > devices. Everything is fine. After the reboot, I can only see the
> > xenstore entries of one device. It is as if the other device
> > "disappeared" without throwing any errors.
So there are no errors on the hypervisor dmesg or the xl logs at all?
> > Have you seen this before? Do you know if it has been fixed in staging?
> > I noticed this fix which seems to be very relevant:
> >
> > https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
> >
> > but it is already included in 4.12.
AFAICT your issue seems related to xl/libxl not properly re-adding the
devices on reboot. The fix above had to do with leaving devices in a
broken state under some circumstances (ie: they where always attached
to the guest, just not working properly).
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-25 7:44 ` Roger Pau Monné
@ 2019-06-25 22:14 ` Stefano Stabellini
0 siblings, 0 replies; 11+ messages in thread
From: Stefano Stabellini @ 2019-06-25 22:14 UTC (permalink / raw)
To: Roger Pau Monné
Cc: jgross, Stefano Stabellini, wl, andrew.cooper3, jbeulich,
xen-devel, boris.ostrovsky, chao.gao
[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]
On Tue, 25 Jun 2019, Roger Pau Monné wrote:
> On Mon, Jun 24, 2019 at 11:47:09AM -0700, Stefano Stabellini wrote:
> > + xen-devel
> >
> > On Mon, 24 Jun 2019, Stefano Stabellini wrote:
> > > Hi all,
> > >
> > > I might have found a bug with PCI passthrough to a Linux HVM guest on
> > > x86 with Xen 4.12. It is not easy for me to get access, and especially
> > > change components, on this particular system, and I don't have access to
> > > other x86 boxes at the moment, so apologies for the partial information
> > > report. The setup is as follow:
> > >
> > > - two PCI devices have been assigned to a HVM guest, everything is fine
> > > - reboot the guest from inside, i.e. `reboot' in Linux
> > > - after the reboot completes, only one device is assigned
>
> Can you provide the xl debug log of the whole process?
See attached.
> > > Before the reboot, I see all the appropriate xenstore entries for both
> > > devices. Everything is fine. After the reboot, I can only see the
> > > xenstore entries of one device. It is as if the other device
> > > "disappeared" without throwing any errors.
>
> So there are no errors on the hypervisor dmesg or the xl logs at all?
Nope. Only:
[445257.718590] xen_pciback: vpci: 0000:00:0e.0: assign to virtual slot 0
[445257.733048] pciback 0000:00:0e.0: registering for 4
[445257.741257] xen_pciback: vpci: 0000:03:00.0: assign to virtual slot 1
[445257.758836] pciback 0000:03:00.0: registering for 4
[445340.380219] xen_pciback: vpci: 0000:00:0e.0: assign to virtual slot 0
[445340.391755] pciback 0000:00:0e.0: registering for 5
> > > Have you seen this before? Do you know if it has been fixed in staging?
> > > I noticed this fix which seems to be very relevant:
> > >
> > > https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
> > >
> > > but it is already included in 4.12.
>
> AFAICT your issue seems related to xl/libxl not properly re-adding the
> devices on reboot. The fix above had to do with leaving devices in a
> broken state under some circumstances (ie: they where always attached
> to the guest, just not working properly).
Yes, it looks like it is as you describe.
[-- Attachment #2: Type: text/plain, Size: 27715 bytes --]
Waiting for domain test-multi-adapters.1 (domid 4) to die [pid 1536]
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/0: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] nentries=1 rc=1 4..4
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] got=domaininfos[0] got->domain=4
libxl: debug: libxl_domain.c:804:domain_death_xswatch_callback: Domain 4:Exists shutdown_reported=0 dominf.flags=ffff0102
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] nentries=1 rc=1 4..4
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] got=domaininfos[0] got->domain=4
libxl: debug: libxl_domain.c:804:domain_death_xswatch_callback: Domain 4:Exists shutdown_reported=0 dominf.flags=10106
libxl: debug: libxl_domain.c:816:domain_death_xswatch_callback: shutdown reporting
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
Domain 4 has shut down, reason code 1 0x1
Action for shutdown reason code 1 is restart
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 4:connected to /var/run/xen/qmp-libxl-4
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 4:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"query-cpus","id":2}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_domain.c:1749:libxl_retrieve_domain_configuration: Domain 4:No vtpm from xenstore
libxl: debug: libxl_domain.c:1749:libxl_retrieve_domain_configuration: Domain 4:No vusb from xenstore
libxl: debug: libxl_domain.c:1749:libxl_retrieve_domain_configuration: Domain 4:No vusb from xenstore
libxl: warning: libxl_domain.c:1767:libxl_retrieve_domain_configuration: Domain 4:Device present in JSON but not in xenstore, ignored
libxl: debug: libxl_domain.c:1749:libxl_retrieve_domain_configuration: Domain 4:No vdispl from xenstore
libxl: debug: libxl_domain.c:1749:libxl_retrieve_domain_configuration: Domain 4:No vsnd from xenstore
Domain 4 needs to be cleaned up: destroying the domain
libxl: debug: libxl_domain.c:902:libxl_domain_destroy: Domain 4:ao 0x7f67c0724f60: create: how=0 callback=0 poller=0x7f67c0e09680
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 4:connected to /var/run/xen/qmp-libxl-4
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 4:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"device_del","id":2,"arguments":{"id":"pci-pt-00_0e.0"}}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 4:connected to /var/run/xen/qmp-libxl-4
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 4:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 4:next qmp command: '{"execute":"device_del","id":2,"arguments":{"id":"pci-pt-03_00.0"}}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 4:message type: return
libxl: debug: libxl_dm.c:3008:libxl__destroy_device_model: Domain 4:Didn't find dm UID; destroying by pid
libxl: debug: libxl_dm.c:2877:kill_device_model: Device Model signaled
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x7f67bfcadd90 wpath=/local/domain/0/backend/vif/4/0/state token=2/1: register slotnum=2
libxl: debug: libxl_domain.c:911:libxl_domain_destroy: Domain 4:ao 0x7f67c0724f60: inprogress: poller=0x7f67c0e09680, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67bfcadd90 wpath=/local/domain/0/backend/vif/4/0/state token=2/1: event epath=/local/domain/0/backend/vif/4/0/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vif/4/0/state wanted state 6 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x7f67bfcadd90 wpath=/local/domain/0/backend/vif/4/0/state token=2/1: deregister slotnum=2
libxl: debug: libxl_device.c:1117:device_backend_callback: Domain 4:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfcadd90: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 4:calling hotplug script: /etc/xen/scripts/vif-bridge offline
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 4:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 4: type_if=vif
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 4:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: script: /etc/xen/scripts/vif-bridge
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_TYPE: vif
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_PATH: backend/vif/4/0
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_BASE_PATH: backend
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: netdev:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: INTERFACE: vif4.0-emu
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: vif: vif4.0
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-bridge offline
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfcade90: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 4:calling hotplug script: /etc/xen/scripts/vif-bridge remove
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 4:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 4: type_if=tap
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 4:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: script: /etc/xen/scripts/vif-bridge
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_TYPE: vif
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_PATH: backend/vif/4/0
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: XENBUS_BASE_PATH: backend
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: netdev:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: INTERFACE: vif4.0-emu
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 4: vif: vif4.0
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-bridge remove
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfcade90: deregister unregistered
libxl: debug: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 4:num_exec 2, not running hotplug scripts
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 4:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfcade90: deregister unregistered
libxl: debug: libxl_linux.c:235:libxl__get_hotplug_script_info: Domain 4:backend_kind 3, no need to execute scripts
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 4:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67c0724db0: deregister unregistered
libxl: debug: libxl_linux.c:235:libxl__get_hotplug_script_info: Domain 4:backend_kind 6, no need to execute scripts
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 4:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfcad950: deregister unregistered
libxl: debug: libxl_domain.c:1194:devices_destroy_cb: Domain 4:Forked pid 2217 for destroy of domain
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/0: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] nentries=1 rc=1 4..4
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 4:[evg=0x7f67c0e09a20] got=domaininfos[0] got->domain=4
libxl: debug: libxl_domain.c:804:domain_death_xswatch_callback: Domain 4:Exists shutdown_reported=1 dominf.flags=1010f
libxl: debug: libxl_domain.c:724:domain_death_occurred: Domain 4:dying
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
libxl: debug: libxl_event.c:1873:libxl__ao_complete: ao 0x7f67c0724f60: complete, rc=0
libxl: debug: libxl_event.c:1842:libxl__ao__destroy: ao 0x7f67c0724f60: destroy
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/0: deregister slotnum=3
Done. Rebooting now
libxl: debug: libxl_create.c:1693:do_domain_create: Domain 0:ao 0x7f67c0927a00: create: how=0 callback=0 poller=0x7f67c0e09680
libxl: debug: libxl_device.c:397:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:360:disk_try_backend: Disk vdev=xvda, backend phy unsuitable due to format qcow2
libxl: debug: libxl_device.c:432:libxl__device_disk_set_backend: Disk vdev=xvda, using backend qdisk
libxl: debug: libxl_create.c:1018:initiate_domain_create: Domain 5:running bootloader
libxl: debug: libxl_bootloader.c:328:libxl__bootloader_run: Domain 5:not a PV/PVH domain, skipping bootloader
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x55a956584bb8: deregister unregistered
domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
domainbuilder: detail: xc_dom_kernel_file: filename="/usr/lib/xen/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap : 179 kB
libxl: debug: libxl_dom.c:988:libxl__load_hvm_firmware_module: Loading BIOS: /usr/lib/xen/boot/seabios.bin
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.12, caps xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x34fc4
xc: detail: ELF: memory: 0x100000 -> 0x134fc4
domainbuilder: detail: xc_dom_mem_init: mem 2040 MB, pages 0x7f800 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x7f800 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: range: start=0x0 end=0x7f800000
domainbuilder: detail: xc_dom_malloc : 4080 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail: 4KB PAGES: 0x0000000000000200
xc: detail: 2MB PAGES: 0x00000000000003fb
xc: detail: 1GB PAGES: 0x0000000000000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x100+0x35 at 0x7f67c0fe0000
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x100000 -> 0x135000 (pfn 0x100 + 0x35 pages)
xc: detail: ELF: phdr 0 at 0x7f67c0fab000 -> 0x7f67c0fd6540
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x135+0x40 at 0x7f67c0fa0000
domainbuilder: detail: xc_dom_alloc_segment: System Firmware module : 0x135000 -> 0x175000 (pfn 0x135 + 0x40 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x175+0x1 at 0x7f67c0f9f000
domainbuilder: detail: xc_dom_alloc_segment: HVM start info : 0x175000 -> 0x176000 (pfn 0x175 + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x176000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: hvm-3.0-x86_64
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 4085 kB
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 179 kB
domainbuilder: detail: domU mmap : 472 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[0] -> d0 0xfefff
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[1] -> d0 0xfeffc
domainbuilder: detail: xc_dom_release: called
libxl: debug: libxl_device.c:397:libxl__device_disk_set_backend: Disk vdev=xvda spec.backend=qdisk
libxl: debug: libxl_linux.c:235:libxl__get_hotplug_script_info: Domain 5:backend_kind 3, no need to execute scripts
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 5:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bec66a50: deregister unregistered
libxl: debug: libxl_dm.c:178:libxl__domain_get_device_model_uid: Domain 5:dm_restrict disabled, starting QEMU as root
libxl: debug: libxl_disk.c:928:libxl__device_disk_find_local_path: Directly accessing local QDISK target /persist/img/C13945BF7C385D51F4AF24676E502BAA608AA50871FC9A7A4C88FF3DE5FA95B7-197b9eef-281e-4883-a6ed-3878d2cb7548.qcow2
libxl: debug: libxl_dm.c:2602:libxl__spawn_local_dm: Domain 5:Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: /usr/lib/xen/bin/qemu-system-i386
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -xen-domid
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: 5
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -chardev
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-5,server,nowait
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -no-shutdown
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -mon
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -chardev
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-5,server,nowait
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -mon
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -nodefaults
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -no-user-config
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -name
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: test-multi-adapters.1
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -vnc
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: 0.0.0.0:0,to=99
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -display
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: none
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -serial
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: pty
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -device
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: cirrus-vga,vgamem_mb=8
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -boot
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: order=dc
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -usb
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -usbdevice
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: tablet
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -smp
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: 2,maxcpus=2
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -device
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: rtl8139,id=nic0,netdev=net0,mac=00:16:3e:00:01:01
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -netdev
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: type=tap,id=net0,ifname=vif5.0-emu,script=no,downscript=no
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -machine
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: xenfv
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -m
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: 2040
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: -drive
libxl: debug: libxl_dm.c:2604:libxl__spawn_local_dm: Domain 5: file=/persist/img/C13945BF7C385D51F4AF24676E502BAA608AA50871FC9A7A4C88FF3DE5FA95B7-197b9eef-281e-4883-a6ed-3878d2cb7548.qcow2,if=ide,index=0,media=disk,format=qcow2,cache=writeback
libxl: debug: libxl_dm.c:2606:libxl__spawn_local_dm: Domain 5:Spawning device-model /usr/lib/xen/bin/qemu-system-i386 with additional environment:
libxl: debug: libxl_dm.c:2608:libxl__spawn_local_dm: Domain 5: XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x55a956584eb0 wpath=/local/domain/0/device-model/5/state token=3/2: register slotnum=3
libxl: debug: libxl_create.c:1730:do_domain_create: Domain 0:ao 0x7f67c0927a00: inprogress: poller=0x7f67c0e09680, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x55a956584eb0 wpath=/local/domain/0/device-model/5/state token=3/2: event epath=/local/domain/0/device-model/5/state
libxl: debug: libxl_exec.c:407:spawn_watch_event: domain 5 device model: spawn watch p=(null)
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x55a956584eb0 wpath=/local/domain/0/device-model/5/state token=3/2: event epath=/local/domain/0/device-model/5/state
libxl: debug: libxl_exec.c:407:spawn_watch_event: domain 5 device model: spawn watch p=running
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x55a956584eb0 wpath=/local/domain/0/device-model/5/state token=3/2: deregister slotnum=3
libxl: debug: libxl_exec.c:129:libxl_report_child_exitstatus: domain 5 device model (dying as expected) [2262] died due to fatal signal Killed
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x55a956584eb0: deregister unregistered
libxl: debug: libxl_qmp.c:2162:libxl__ev_qmp_dispose: Domain 0: ev 0x55a956584ec8
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 5:connected to /var/run/xen/qmp-libxl-5
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 5:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"query-chardev","id":2}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"query-vnc","id":3}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x7f67bfeb8af0 wpath=/local/domain/0/backend/vif/5/0/state token=3/3: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67bfeb8af0 wpath=/local/domain/0/backend/vif/5/0/state token=3/3: event epath=/local/domain/0/backend/vif/5/0/state
libxl: debug: libxl_event.c:881:devstate_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 still waiting state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67bfeb8af0 wpath=/local/domain/0/backend/vif/5/0/state token=3/3: event epath=/local/domain/0/backend/vif/5/0/state
libxl: debug: libxl_event.c:877:devstate_callback: backend /local/domain/0/backend/vif/5/0/state wanted state 2 ok
libxl: debug: libxl_event.c:676:libxl__ev_xswatch_deregister: watch w=0x7f67bfeb8af0 wpath=/local/domain/0/backend/vif/5/0/state token=3/3: deregister slotnum=3
libxl: debug: libxl_device.c:1117:device_backend_callback: Domain 5:calling device_backend_cleanup
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfeb8af0: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 5:calling hotplug script: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 5:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 5: type_if=vif
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 5:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: script: /etc/xen/scripts/vif-bridge
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_TYPE: vif
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_PATH: backend/vif/5/0
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_BASE_PATH: backend
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: netdev:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: INTERFACE: vif5.0-emu
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: vif: vif5.0
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-bridge online
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfeb8bf0: deregister unregistered
libxl: debug: libxl_device.c:1218:device_hotplug: Domain 5:calling hotplug script: /etc/xen/scripts/vif-bridge add
libxl: debug: libxl_device.c:1219:device_hotplug: Domain 5:extra args:
libxl: debug: libxl_device.c:1225:device_hotplug: Domain 5: type_if=tap
libxl: debug: libxl_device.c:1227:device_hotplug: Domain 5:env:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: script: /etc/xen/scripts/vif-bridge
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_TYPE: vif
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_PATH: backend/vif/5/0
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: XENBUS_BASE_PATH: backend
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: netdev:
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: INTERFACE: vif5.0-emu
libxl: debug: libxl_device.c:1234:device_hotplug: Domain 5: vif: vif5.0
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/vif-bridge add
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfeb8bf0: deregister unregistered
libxl: debug: libxl_linux.c:226:libxl__get_hotplug_script_info: Domain 5:num_exec 2, not running hotplug scripts
libxl: debug: libxl_device.c:1203:device_hotplug: Domain 5:No hotplug script to execute
libxl: debug: libxl_event.c:689:libxl__ev_xswatch_deregister: watch w=0x7f67bfeb8bf0: deregister unregistered
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 5:connected to /var/run/xen/qmp-libxl-5
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 5:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"device_add","id":2,"arguments":{"driver":"xen-pci-passthrough","id":"pci-pt-00_0e.0","hostaddr":"0000:00:0e.0"}}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"query-pci","id":3}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_pci.c:89:libxl__create_pci_backend: Domain 5:Creating pci backend
libxl: debug: libxl_event.c:2190:libxl__ao_progress_report: ao 0x7f67c0927a00: progress report: ignored
libxl: debug: libxl_event.c:1873:libxl__ao_complete: ao 0x7f67c0927a00: complete, rc=0
libxl: debug: libxl_event.c:1842:libxl__ao__destroy: ao 0x7f67c0927a00: destroy
libxl: debug: libxl_qmp.c:813:libxl__qmp_initialize: Domain 5:connected to /var/run/xen/qmp-libxl-5
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: qmp
libxl: debug: libxl_qmp.c:365:qmp_handle_response: Domain 5:QEMU version: 3.0.0
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"qmp_capabilities","id":1}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
libxl: debug: libxl_qmp.c:666:qmp_send_prepare: Domain 5:next qmp command: '{"execute":"cont","id":2}
'
libxl: debug: libxl_qmp.c:350:qmp_handle_response: Domain 5:message type: return
Waiting for domain test-multi-adapters.1 (domid 5) to die [pid 1536]
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/4: register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x7f67c0e09880 wpath=@releaseDomain token=3/4: event epath=@releaseDomain
libxl: debug: libxl_domain.c:767:domain_death_xswatch_callback: Domain 5:[evg=0x7f67c0317fc0] nentries=1 rc=1 5..5
libxl: debug: libxl_domain.c:778:domain_death_xswatch_callback: Domain 5:[evg=0x7f67c0317fc0] got=domaininfos[0] got->domain=5
libxl: debug: libxl_domain.c:804:domain_death_xswatch_callback: Domain 5:Exists shutdown_reported=0 dominf.flags=ffff0102
libxl: debug: libxl_domain.c:771:domain_death_xswatch_callback: [evg=0] all reported
libxl: debug: libxl_domain.c:833:domain_death_xswatch_callback: domain death search done
/tmp #
[-- Attachment #3: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
[not found] ` <alpine.DEB.2.21.1906251427440.5851@sstabellini-ThinkPad-T480s>
@ 2019-06-26 5:05 ` Juergen Gross
0 siblings, 0 replies; 11+ messages in thread
From: Juergen Gross @ 2019-06-26 5:05 UTC (permalink / raw)
To: Stefano Stabellini
Cc: wl, andrew.cooper3, jbeulich, xen-devel, boris.ostrovsky,
chao.gao, roger.pau
On 26.06.19 00:10, Stefano Stabellini wrote:
> On Tue, 25 Jun 2019, Juergen Gross wrote:
>> On 24.06.19 20:45, Stefano Stabellini wrote:
>>> Hi all,
>>>
>>> I might have found a bug with PCI passthrough to a Linux HVM guest on
>>> x86 with Xen 4.12. It is not easy for me to get access, and especially
>>> change components, on this particular system, and I don't have access to
>>> other x86 boxes at the moment, so apologies for the partial information
>>> report. The setup is as follow:
>>>
>>> - two PCI devices have been assigned to a HVM guest, everything is fine
>>> - reboot the guest from inside, i.e. `reboot' in Linux
>>> - after the reboot completes, only one device is assigned
>>>
>>> Before the reboot, I see all the appropriate xenstore entries for both
>>> devices. Everything is fine. After the reboot, I can only see the
>>> xenstore entries of one device. It is as if the other device
>>> "disappeared" without throwing any errors.
>>
>> Can you please post the Xenstore entries before the reboot?
>>
>> I think the numbering scheme of PCI devices in Xenstore isn't like that
>> of other devices...
>
> See attached. The domid goes from 3 to 5, because I shutdown domid 3
> normally the first time around, instead of rebooting.
>
As I thought. Working on a patch.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-24 18:47 ` [Xen-devel] PCI Passthrough bug with x86 HVM Stefano Stabellini
2019-06-25 7:44 ` Roger Pau Monné
@ 2019-06-26 6:17 ` Juergen Gross
2019-06-26 12:21 ` Chao Gao
1 sibling, 1 reply; 11+ messages in thread
From: Juergen Gross @ 2019-06-26 6:17 UTC (permalink / raw)
To: Stefano Stabellini
Cc: wl, andrew.cooper3, jbeulich, xen-devel, boris.ostrovsky,
chao.gao, roger.pau
[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]
On 24.06.19 20:47, Stefano Stabellini wrote:
> + xen-devel
>
> On Mon, 24 Jun 2019, Stefano Stabellini wrote:
>> Hi all,
>>
>> I might have found a bug with PCI passthrough to a Linux HVM guest on
>> x86 with Xen 4.12. It is not easy for me to get access, and especially
>> change components, on this particular system, and I don't have access to
>> other x86 boxes at the moment, so apologies for the partial information
>> report. The setup is as follow:
>>
>> - two PCI devices have been assigned to a HVM guest, everything is fine
>> - reboot the guest from inside, i.e. `reboot' in Linux
>> - after the reboot completes, only one device is assigned
>>
>> Before the reboot, I see all the appropriate xenstore entries for both
>> devices. Everything is fine. After the reboot, I can only see the
>> xenstore entries of one device. It is as if the other device
>> "disappeared" without throwing any errors.
>>
>> Have you seen this before? Do you know if it has been fixed in staging?
>> I noticed this fix which seems to be very relevant:
>>
>> https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>>
>> but it is already included in 4.12.
Stefano, could you please try the attached patch? It is only compile
tested for now.
Juergen
[-- Attachment #2: 0001-libxl-fix-pci-device-re-assigning-after-domain-reboo.patch --]
[-- Type: text/x-patch, Size: 6977 bytes --]
From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
To: xen-devel@lists.xenproject.org
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wl@xen.org>
Date: Wed, 26 Jun 2019 08:15:28 +0200
Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
After a reboot of a guest only the first pci device configuration will
be retrieved from Xenstore resulting in loss of any further assigned
passed through pci devices.
The main reason is that all passed through pci devices reside under a
common root device "0" in Xenstore. So when the device list is rebuilt
from Xenstore after a reboot the sub-devices below that root device
need to be selected instead of using the root device number as a
selector.
Fix that by adding a new member to struct libxl_device_type which when
set is used to get the number of devices. Add such a member for pci to
get the correct number of pci devices instead of implying it from the
number of pci root devices (which will always be 1).
While at it fix the type of libxl__device_pci_from_xs_be() to match
the one of the .from_xenstore member of struct libxl_device_type. This
fixes a latent bug checking the return value of a function returning
void.
Signed-off-by: Juergen Gross <jgross@suse.com>
---
tools/libxl/libxl_device.c | 24 +++++++++++++++++++-----
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/libxl_pci.c | 35 ++++++++++++++++++++++++++---------
3 files changed, 47 insertions(+), 14 deletions(-)
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index db6c0203b7..a2569102ee 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -2026,6 +2026,7 @@ void *libxl__device_list(libxl__gc *gc, const struct libxl_device_type *dt,
char *libxl_path;
char **dir = NULL;
unsigned int ndirs = 0;
+ unsigned int ndevs = 0;
int rc;
*num = 0;
@@ -2037,21 +2038,34 @@ void *libxl__device_list(libxl__gc *gc, const struct libxl_device_type *dt,
dir = libxl__xs_directory(gc, XBT_NULL, libxl_path, &ndirs);
if (dir && ndirs) {
- list = libxl__malloc(NOGC, dt->dev_elem_size * ndirs);
+ if (dt->get_num) {
+ if (ndirs != 1) {
+ LOGD(ERROR, domid, "multiple entries in %s\n", libxl_path);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ rc = dt->get_num(gc, GCSPRINTF("%s/%s", libxl_path, *dir), &ndevs);
+ if (rc) goto out;
+ } else {
+ ndevs = ndirs;
+ }
+ list = libxl__malloc(NOGC, dt->dev_elem_size * ndevs);
item = list;
- while (*num < ndirs) {
+ while (*num < ndevs) {
dt->init(item);
- ++(*num);
if (dt->from_xenstore) {
+ int nr = dt->get_num ? *num : atoi(*dir);
char *device_libxl_path = GCSPRINTF("%s/%s", libxl_path, *dir);
- rc = dt->from_xenstore(gc, device_libxl_path, atoi(*dir), item);
+ rc = dt->from_xenstore(gc, device_libxl_path, nr, item);
if (rc) goto out;
}
item = (uint8_t *)item + dt->dev_elem_size;
- ++dir;
+ ++(*num);
+ if (!dt->get_num)
+ ++dir;
}
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3be5c644c1..a3102871f3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3707,6 +3707,7 @@ typedef void (*device_merge_fn_t)(libxl_ctx *, void *, void *);
typedef int (*device_dm_needed_fn_t)(void *, unsigned);
typedef void (*device_update_config_fn_t)(libxl__gc *, void *, void *);
typedef int (*device_update_devid_fn_t)(libxl__gc *, uint32_t, void *);
+typedef int (*device_get_num_fn_t)(libxl__gc *, const char *, unsigned int *);
typedef int (*device_from_xenstore_fn_t)(libxl__gc *, const char *,
libxl_devid, void *);
typedef int (*device_set_xenstore_config_fn_t)(libxl__gc *, uint32_t, void *,
@@ -3730,6 +3731,7 @@ struct libxl_device_type {
device_dm_needed_fn_t dm_needed;
device_update_config_fn_t update_config;
device_update_devid_fn_t update_devid;
+ device_get_num_fn_t get_num;
device_from_xenstore_fn_t from_xenstore;
device_set_xenstore_config_fn_t set_xenstore_config;
};
diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c
index 4ec6872798..03beb865d9 100644
--- a/tools/libxl/libxl_pci.c
+++ b/tools/libxl/libxl_pci.c
@@ -1547,12 +1547,13 @@ int libxl_device_pci_destroy(libxl_ctx *ctx, uint32_t domid,
return AO_INPROGRESS;
}
-static void libxl__device_pci_from_xs_be(libxl__gc *gc,
- const char *be_path,
- int nr, libxl_device_pci *pci)
+static int libxl__device_pci_from_xs_be(libxl__gc *gc,
+ const char *be_path,
+ libxl_devid nr, void *data)
{
char *s;
unsigned int domain = 0, bus = 0, dev = 0, func = 0, vdevfn = 0;
+ libxl_device_pci *pci = data;
s = libxl__xs_read(gc, XBT_NULL, GCSPRINTF("%s/dev-%d", be_path, nr));
sscanf(s, PCI_BDF, &domain, &bus, &dev, &func);
@@ -1582,24 +1583,39 @@ static void libxl__device_pci_from_xs_be(libxl__gc *gc,
}
} while ((p = strtok_r(NULL, ",=", &saveptr)) != NULL);
}
+
+ return 0;
+}
+
+static int libxl__device_pci_get_num(libxl__gc *gc, const char *be_path,
+ unsigned int *num)
+{
+ char *num_devs;
+ int rc = 0;
+
+ num_devs = libxl__xs_read(gc, XBT_NULL, GCSPRINTF("%s/num_devs", be_path));
+ if (!num_devs)
+ rc = ERROR_FAIL;
+ else
+ *num = atoi(num_devs);
+
+ return rc;
}
libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid, int *num)
{
GC_INIT(ctx);
- char *be_path, *num_devs;
- int n, i;
+ char *be_path;
+ unsigned int n, i;
libxl_device_pci *pcidevs = NULL;
*num = 0;
be_path = libxl__domain_device_backend_path(gc, 0, domid, 0,
LIBXL__DEVICE_KIND_PCI);
- num_devs = libxl__xs_read(gc, XBT_NULL, GCSPRINTF("%s/num_devs", be_path));
- if (!num_devs)
+ if (libxl__device_pci_get_num(gc, be_path, &n))
goto out;
- n = atoi(num_devs);
pcidevs = calloc(n, sizeof(libxl_device_pci));
for (i = 0; i < n; i++)
@@ -1688,7 +1704,8 @@ static int libxl_device_pci_compare(const libxl_device_pci *d1,
#define libxl__device_pci_update_devid NULL
DEFINE_DEVICE_TYPE_STRUCT_X(pcidev, pci, PCI,
- .from_xenstore = (device_from_xenstore_fn_t)libxl__device_pci_from_xs_be,
+ .get_num = libxl__device_pci_get_num,
+ .from_xenstore = libxl__device_pci_from_xs_be,
);
/*
--
2.16.4
[-- Attachment #3: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 6:17 ` Juergen Gross
@ 2019-06-26 12:21 ` Chao Gao
2019-06-26 13:36 ` Juergen Gross
0 siblings, 1 reply; 11+ messages in thread
From: Chao Gao @ 2019-06-26 12:21 UTC (permalink / raw)
To: Juergen Gross
Cc: Stefano Stabellini, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau
On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
>On 24.06.19 20:47, Stefano Stabellini wrote:
>>+ xen-devel
>>
>>On Mon, 24 Jun 2019, Stefano Stabellini wrote:
>>>Hi all,
>>>
>>>I might have found a bug with PCI passthrough to a Linux HVM guest on
>>>x86 with Xen 4.12. It is not easy for me to get access, and especially
>>>change components, on this particular system, and I don't have access to
>>>other x86 boxes at the moment, so apologies for the partial information
>>>report. The setup is as follow:
>>>
>>>- two PCI devices have been assigned to a HVM guest, everything is fine
>>>- reboot the guest from inside, i.e. `reboot' in Linux
>>>- after the reboot completes, only one device is assigned
>>>
>>>Before the reboot, I see all the appropriate xenstore entries for both
>>>devices. Everything is fine. After the reboot, I can only see the
>>>xenstore entries of one device. It is as if the other device
>>>"disappeared" without throwing any errors.
>>>
>>>Have you seen this before? Do you know if it has been fixed in staging?
>>>I noticed this fix which seems to be very relevant:
>>>
>>>https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>>>
>>>but it is already included in 4.12.
>
>Stefano, could you please try the attached patch? It is only compile
>tested for now.
>
>
>Juergen
>From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
>From: Juergen Gross <jgross@suse.com>
>To: xen-devel@lists.xenproject.org
>Cc: Ian Jackson <ian.jackson@eu.citrix.com>
>Cc: Wei Liu <wl@xen.org>
>Date: Wed, 26 Jun 2019 08:15:28 +0200
>Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
>
>After a reboot of a guest only the first pci device configuration will
>be retrieved from Xenstore resulting in loss of any further assigned
>passed through pci devices.
>
>The main reason is that all passed through pci devices reside under a
>common root device "0" in Xenstore. So when the device list is rebuilt
>from Xenstore after a reboot the sub-devices below that root device
>need to be selected instead of using the root device number as a
>selector.
>
>Fix that by adding a new member to struct libxl_device_type which when
>set is used to get the number of devices. Add such a member for pci to
>get the correct number of pci devices instead of implying it from the
>number of pci root devices (which will always be 1).
>
>While at it fix the type of libxl__device_pci_from_xs_be() to match
>the one of the .from_xenstore member of struct libxl_device_type. This
>fixes a latent bug checking the return value of a function returning
>void.
>
>Signed-off-by: Juergen Gross <jgross@suse.com>
Tested-by: Chao Gao <chao.gao@intel.com>
Note that I just tested it on RELEASE-4.12.0, not staging.
I also found USB device would miss across reboot. Is there someone
willing to fix it too?
Thanks
Chao
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 12:21 ` Chao Gao
@ 2019-06-26 13:36 ` Juergen Gross
2019-06-26 14:34 ` Chao Gao
2019-06-26 15:18 ` Stefano Stabellini
0 siblings, 2 replies; 11+ messages in thread
From: Juergen Gross @ 2019-06-26 13:36 UTC (permalink / raw)
To: Chao Gao
Cc: Stefano Stabellini, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau
On 26.06.19 14:21, Chao Gao wrote:
> On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
>> On 24.06.19 20:47, Stefano Stabellini wrote:
>>> + xen-devel
>>>
>>> On Mon, 24 Jun 2019, Stefano Stabellini wrote:
>>>> Hi all,
>>>>
>>>> I might have found a bug with PCI passthrough to a Linux HVM guest on
>>>> x86 with Xen 4.12. It is not easy for me to get access, and especially
>>>> change components, on this particular system, and I don't have access to
>>>> other x86 boxes at the moment, so apologies for the partial information
>>>> report. The setup is as follow:
>>>>
>>>> - two PCI devices have been assigned to a HVM guest, everything is fine
>>>> - reboot the guest from inside, i.e. `reboot' in Linux
>>>> - after the reboot completes, only one device is assigned
>>>>
>>>> Before the reboot, I see all the appropriate xenstore entries for both
>>>> devices. Everything is fine. After the reboot, I can only see the
>>>> xenstore entries of one device. It is as if the other device
>>>> "disappeared" without throwing any errors.
>>>>
>>>> Have you seen this before? Do you know if it has been fixed in staging?
>>>> I noticed this fix which seems to be very relevant:
>>>>
>>>> https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>>>>
>>>> but it is already included in 4.12.
>>
>> Stefano, could you please try the attached patch? It is only compile
>> tested for now.
>>
>>
>> Juergen
>
>>From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
>> From: Juergen Gross <jgross@suse.com>
>> To: xen-devel@lists.xenproject.org
>> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
>> Cc: Wei Liu <wl@xen.org>
>> Date: Wed, 26 Jun 2019 08:15:28 +0200
>> Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
>>
>> After a reboot of a guest only the first pci device configuration will
>> be retrieved from Xenstore resulting in loss of any further assigned
>> passed through pci devices.
>>
>> The main reason is that all passed through pci devices reside under a
>> common root device "0" in Xenstore. So when the device list is rebuilt
>>from Xenstore after a reboot the sub-devices below that root device
>> need to be selected instead of using the root device number as a
>> selector.
>>
>> Fix that by adding a new member to struct libxl_device_type which when
>> set is used to get the number of devices. Add such a member for pci to
>> get the correct number of pci devices instead of implying it from the
>> number of pci root devices (which will always be 1).
>>
>> While at it fix the type of libxl__device_pci_from_xs_be() to match
>> the one of the .from_xenstore member of struct libxl_device_type. This
>> fixes a latent bug checking the return value of a function returning
>> void.
>>
>> Signed-off-by: Juergen Gross <jgross@suse.com>
>
> Tested-by: Chao Gao <chao.gao@intel.com>
Thanks!
>
> Note that I just tested it on RELEASE-4.12.0, not staging.
>
> I also found USB device would miss across reboot. Is there someone
> willing to fix it too?
I'll have a look.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 13:36 ` Juergen Gross
@ 2019-06-26 14:34 ` Chao Gao
2019-06-26 14:44 ` Juergen Gross
2019-06-26 15:18 ` Stefano Stabellini
1 sibling, 1 reply; 11+ messages in thread
From: Chao Gao @ 2019-06-26 14:34 UTC (permalink / raw)
To: Juergen Gross
Cc: Stefano Stabellini, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau
[-- Attachment #1: Type: text/plain, Size: 3388 bytes --]
On Wed, Jun 26, 2019 at 03:36:35PM +0200, Juergen Gross wrote:
>On 26.06.19 14:21, Chao Gao wrote:
>>On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
>>>On 24.06.19 20:47, Stefano Stabellini wrote:
>>>>+ xen-devel
>>>>
>>>>On Mon, 24 Jun 2019, Stefano Stabellini wrote:
>>>>>Hi all,
>>>>>
>>>>>I might have found a bug with PCI passthrough to a Linux HVM guest on
>>>>>x86 with Xen 4.12. It is not easy for me to get access, and especially
>>>>>change components, on this particular system, and I don't have access to
>>>>>other x86 boxes at the moment, so apologies for the partial information
>>>>>report. The setup is as follow:
>>>>>
>>>>>- two PCI devices have been assigned to a HVM guest, everything is fine
>>>>>- reboot the guest from inside, i.e. `reboot' in Linux
>>>>>- after the reboot completes, only one device is assigned
>>>>>
>>>>>Before the reboot, I see all the appropriate xenstore entries for both
>>>>>devices. Everything is fine. After the reboot, I can only see the
>>>>>xenstore entries of one device. It is as if the other device
>>>>>"disappeared" without throwing any errors.
>>>>>
>>>>>Have you seen this before? Do you know if it has been fixed in staging?
>>>>>I noticed this fix which seems to be very relevant:
>>>>>
>>>>>https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>>>>>
>>>>>but it is already included in 4.12.
>>>
>>>Stefano, could you please try the attached patch? It is only compile
>>>tested for now.
>>>
>>>
>>>Juergen
>>
>>>From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
>>>From: Juergen Gross <jgross@suse.com>
>>>To: xen-devel@lists.xenproject.org
>>>Cc: Ian Jackson <ian.jackson@eu.citrix.com>
>>>Cc: Wei Liu <wl@xen.org>
>>>Date: Wed, 26 Jun 2019 08:15:28 +0200
>>>Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
>>>
>>>After a reboot of a guest only the first pci device configuration will
>>>be retrieved from Xenstore resulting in loss of any further assigned
>>>passed through pci devices.
>>>
>>>The main reason is that all passed through pci devices reside under a
>>>common root device "0" in Xenstore. So when the device list is rebuilt
>>>from Xenstore after a reboot the sub-devices below that root device
>>>need to be selected instead of using the root device number as a
>>>selector.
>>>
>>>Fix that by adding a new member to struct libxl_device_type which when
>>>set is used to get the number of devices. Add such a member for pci to
>>>get the correct number of pci devices instead of implying it from the
>>>number of pci root devices (which will always be 1).
>>>
>>>While at it fix the type of libxl__device_pci_from_xs_be() to match
>>>the one of the .from_xenstore member of struct libxl_device_type. This
>>>fixes a latent bug checking the return value of a function returning
>>>void.
>>>
>>>Signed-off-by: Juergen Gross <jgross@suse.com>
>>
>>Tested-by: Chao Gao <chao.gao@intel.com>
>
>Thanks!
>
>>
>>Note that I just tested it on RELEASE-4.12.0, not staging.
>>
>>I also found USB device would miss across reboot. Is there someone
>>willing to fix it too?
>
>I'll have a look.
>
In guest configuration file, it has following two lines:
usbctrl=['type=devicemodel,version=1']
usbdev=['type=hostdev,hostbus=1,hostaddr=3,controller=0,port=1']
Attachments are output of 'xenstore-ls' before and after reboot
Thanks
Chao
[-- Attachment #2: xenstore-before --]
[-- Type: text/plain, Size: 7968 bytes --]
tool = ""
xenstored = ""
local = ""
domain = ""
0 = ""
control = ""
feature-poweroff = "1"
feature-reboot = "1"
feature-suspend = "1"
domid = "0"
name = "Domain-0"
device-model = ""
0 = ""
backends = ""
console = ""
vkbd = ""
qdisk = ""
9pfs = ""
qusb = ""
vfb = ""
qnic = ""
state = "running"
37 = ""
backends = ""
console = ""
vkbd = ""
9pfs = ""
qusb = ""
state = "running"
device = ""
vbd = ""
51712 = ""
backend = "/local/domain/0/backend/qdisk/0/51712"
backend-id = "0"
state = "3"
virtual-device = "51712"
device-type = "disk"
protocol = "x86_64-abi"
ring-ref = "8"
event-channel = "76"
feature-persistent = "1"
backend = ""
qdisk = ""
0 = ""
51712 = ""
frontend = "/local/domain/0/device/vbd/51712"
params = "qcow2:centos.qcow2"
frontend-id = "0"
online = "1"
removable = "0"
bootable = "1"
state = "2"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
feature-flush-cache = "1"
info = "0"
max-ring-page-order = "4"
feature-discard = "1"
hotplug-status = "connected"
37 = ""
51712 = ""
frontend = "/local/domain/37/device/vbd/51712"
params = "qcow2:/home/nervalusr/chao/centos.qcow2"
frontend-id = "37"
online = "1"
removable = "0"
bootable = "1"
state = "4"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
feature-discard = "1"
discard-granularity = "512"
feature-flush-cache = "1"
max-ring-page-order = "4"
info = "0"
sector-size = "512"
sectors = "62914560"
hotplug-status = "connected"
console = ""
37 = ""
0 = ""
frontend = "/local/domain/37/console"
frontend-id = "37"
online = "1"
state = "1"
protocol = "vt100"
vkbd = ""
37 = ""
0 = ""
frontend = "/local/domain/37/device/vkbd/0"
frontend-id = "37"
online = "1"
state = "4"
backend-type = "qemu"
feature-abs-pointer = "1"
feature-raw-pointer = "1"
hotplug-status = "connected"
vif = ""
37 = ""
0 = ""
frontend = "/local/domain/37/device/vif/0"
frontend-id = "37"
online = "1"
state = "4"
script = "/etc/xen/scripts/vif-bridge"
mac = "22:16:3e:00:9f:a7"
bridge = "xenbr0"
handle = "0"
type = "vif_ioemu"
feature-sg = "1"
feature-gso-tcpv4 = "1"
feature-gso-tcpv6 = "1"
feature-ipv6-csum-offload = "1"
feature-rx-copy = "1"
feature-rx-flip = "0"
feature-multicast-control = "1"
feature-dynamic-multicast-control = "1"
feature-split-event-channels = "1"
multi-queue-max-queues = "4"
feature-ctrl-ring = "1"
hotplug-status = "connected"
37 = ""
vm = "/vm/9a6857f8-05a0-4c8d-92c3-1ea04bc28c32"
name = "guest_centos7"
cpu = ""
0 = ""
availability = "online"
1 = ""
availability = "online"
2 = ""
availability = "online"
3 = ""
availability = "online"
memory = ""
static-max = "16777216"
target = "16760832"
videoram = "16384"
device = ""
suspend = ""
event-channel = ""
vbd = ""
51712 = ""
backend = "/local/domain/0/backend/qdisk/37/51712"
backend-id = "0"
state = "4"
virtual-device = "51712"
device-type = "disk"
ring-ref = "2312"
event-channel = "46"
protocol = "x86_64-abi"
feature-persistent = "1"
vkbd = ""
0 = ""
backend = "/local/domain/0/backend/vkbd/37/0"
backend-id = "0"
state = "4"
request-abs-pointer = "1"
page-ref = "4156386"
page-gref = "270"
event-channel = "47"
vif = ""
0 = ""
backend = "/local/domain/0/backend/vif/37/0"
backend-id = "0"
state = "4"
handle = "0"
mac = "22:16:3e:00:9f:a7"
multi-queue-num-queues = "4"
queue-0 = ""
tx-ring-ref = "2304"
rx-ring-ref = "2305"
event-channel-tx = "38"
event-channel-rx = "39"
queue-1 = ""
tx-ring-ref = "2306"
rx-ring-ref = "2307"
event-channel-tx = "40"
event-channel-rx = "41"
queue-2 = ""
tx-ring-ref = "2308"
rx-ring-ref = "2309"
event-channel-tx = "42"
event-channel-rx = "43"
queue-3 = ""
tx-ring-ref = "2310"
rx-ring-ref = "2311"
event-channel-tx = "44"
event-channel-rx = "45"
request-rx-copy = "1"
feature-rx-notify = "1"
feature-sg = "1"
feature-gso-tcpv4 = "1"
feature-gso-tcpv6 = "1"
feature-ipv6-csum-offload = "1"
control = ""
shutdown = ""
feature-poweroff = ""
feature-reboot = ""
feature-suspend = ""
feature-s3 = ""
feature-s4 = ""
sysrq = ""
platform-feature-multiprocessor-suspend = "1"
platform-feature-xs_reset_watches = "1"
hvmloader = ""
bios = "seabios"
allow-memory-relocate = "0"
data = ""
drivers = ""
feature = ""
attr = ""
domid = "37"
store = ""
port = "1"
ring-ref = "1044476"
platform = ""
acpi = "1"
acpi_s3 = "1"
acpi_s4 = "1"
acpi_laptop_slate = "0"
device-model = "qemu_xen"
console = ""
backend = "/local/domain/0/backend/console/37/0"
backend-id = "0"
limit = "1048576"
type = "xenconsoled"
output = "pty"
tty = "/dev/pts/4"
port = "2"
ring-ref = "1044479"
vnc-listen = "127.0.0.1"
vnc-port = "5900"
image = ""
device-model-pid = "8409"
serial = ""
0 = ""
tty = "/dev/pts/6"
vm = ""
9a6857f8-05a0-4c8d-92c3-1ea04bc28c32 = ""
name = "guest_centos7"
uuid = "9a6857f8-05a0-4c8d-92c3-1ea04bc28c32"
rtc = ""
timeoffset = ""
image = ""
ostype = "hvm"
start_time = "1561550075.84"
libxl = ""
pciback = ""
0000-88-00-0 = ""
driver_path = "/sys/bus/pci/drivers/intel_nnp"
0 = ""
device = ""
vbd = ""
51712 = ""
frontend = "/local/domain/0/device/vbd/51712"
backend = "/local/domain/0/backend/qdisk/0/51712"
params = "qcow2:centos.qcow2"
frontend-id = "0"
online = "1"
removable = "0"
bootable = "1"
state = "1"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
37 = ""
device = ""
vbd = ""
51712 = ""
frontend = "/local/domain/37/device/vbd/51712"
backend = "/local/domain/0/backend/qdisk/37/51712"
params = "qcow2:/home/nervalusr/chao/centos.qcow2"
frontend-id = "37"
online = "1"
removable = "0"
bootable = "1"
state = "1"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
console = ""
0 = ""
frontend = "/local/domain/37/console"
backend = "/local/domain/0/backend/console/37/0"
frontend-id = "37"
online = "1"
state = "1"
protocol = "vt100"
vkbd = ""
0 = ""
frontend = "/local/domain/37/device/vkbd/0"
backend = "/local/domain/0/backend/vkbd/37/0"
frontend-id = "37"
online = "1"
state = "1"
backend-type = "qemu"
vif = ""
0 = ""
frontend = "/local/domain/37/device/vif/0"
backend = "/local/domain/0/backend/vif/37/0"
frontend-id = "37"
online = "1"
state = "1"
script = "/etc/xen/scripts/vif-bridge"
mac = "22:16:3e:00:9f:a7"
bridge = "xenbr0"
handle = "0"
type = "vif_ioemu"
vusb = ""
0 = ""
type = "devicemodel"
usb-ver = "1"
num-ports = "2"
port = ""
1 = "1-4"
2 = ""
type = "hvm"
dm-version = "qemu_xen"
[-- Attachment #3: xenstore-after --]
[-- Type: text/plain, Size: 7833 bytes --]
tool = ""
xenstored = ""
local = ""
domain = ""
0 = ""
control = ""
feature-poweroff = "1"
feature-reboot = "1"
feature-suspend = "1"
domid = "0"
name = "Domain-0"
device-model = ""
0 = ""
backends = ""
console = ""
vkbd = ""
qdisk = ""
9pfs = ""
qusb = ""
vfb = ""
qnic = ""
state = "running"
38 = ""
backends = ""
console = ""
vkbd = ""
9pfs = ""
qusb = ""
state = "running"
device = ""
vbd = ""
51712 = ""
backend = "/local/domain/0/backend/qdisk/0/51712"
backend-id = "0"
state = "3"
virtual-device = "51712"
device-type = "disk"
protocol = "x86_64-abi"
ring-ref = "8"
event-channel = "76"
feature-persistent = "1"
backend = ""
qdisk = ""
0 = ""
51712 = ""
frontend = "/local/domain/0/device/vbd/51712"
params = "qcow2:centos.qcow2"
frontend-id = "0"
online = "1"
removable = "0"
bootable = "1"
state = "2"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
feature-flush-cache = "1"
info = "0"
max-ring-page-order = "4"
feature-discard = "1"
hotplug-status = "connected"
38 = ""
51712 = ""
frontend = "/local/domain/38/device/vbd/51712"
params = "qcow2:/home/nervalusr/chao/centos.qcow2"
frontend-id = "38"
online = "1"
removable = "0"
bootable = "1"
state = "4"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
feature-discard = "1"
discard-granularity = "512"
feature-flush-cache = "1"
max-ring-page-order = "4"
info = "0"
sector-size = "512"
sectors = "62914560"
hotplug-status = "connected"
console = ""
38 = ""
0 = ""
frontend = "/local/domain/38/console"
frontend-id = "38"
online = "1"
state = "1"
protocol = "vt100"
vkbd = ""
38 = ""
0 = ""
frontend = "/local/domain/38/device/vkbd/0"
frontend-id = "38"
online = "1"
state = "4"
backend-type = "qemu"
feature-abs-pointer = "1"
feature-raw-pointer = "1"
hotplug-status = "connected"
vif = ""
38 = ""
0 = ""
frontend = "/local/domain/38/device/vif/0"
frontend-id = "38"
online = "1"
state = "4"
script = "/etc/xen/scripts/vif-bridge"
mac = "22:16:3e:00:9f:a7"
bridge = "xenbr0"
handle = "0"
type = "vif_ioemu"
feature-sg = "1"
feature-gso-tcpv4 = "1"
feature-gso-tcpv6 = "1"
feature-ipv6-csum-offload = "1"
feature-rx-copy = "1"
feature-rx-flip = "0"
feature-multicast-control = "1"
feature-dynamic-multicast-control = "1"
feature-split-event-channels = "1"
multi-queue-max-queues = "4"
feature-ctrl-ring = "1"
hotplug-status = "connected"
38 = ""
vm = "/vm/9a6857f8-05a0-4c8d-92c3-1ea04bc28c32"
name = "guest_centos7"
cpu = ""
0 = ""
availability = "online"
1 = ""
availability = "online"
2 = ""
availability = "online"
3 = ""
availability = "online"
memory = ""
static-max = "16777216"
target = "16760832"
videoram = "16384"
device = ""
suspend = ""
event-channel = ""
vbd = ""
51712 = ""
backend = "/local/domain/0/backend/qdisk/38/51712"
backend-id = "0"
state = "4"
virtual-device = "51712"
device-type = "disk"
ring-ref = "2312"
event-channel = "45"
protocol = "x86_64-abi"
feature-persistent = "1"
vkbd = ""
0 = ""
backend = "/local/domain/0/backend/vkbd/38/0"
backend-id = "0"
state = "4"
request-abs-pointer = "1"
page-ref = "222612"
page-gref = "343"
event-channel = "46"
vif = ""
0 = ""
backend = "/local/domain/0/backend/vif/38/0"
backend-id = "0"
state = "4"
handle = "0"
mac = "22:16:3e:00:9f:a7"
multi-queue-num-queues = "4"
queue-0 = ""
tx-ring-ref = "2304"
rx-ring-ref = "2305"
event-channel-tx = "37"
event-channel-rx = "38"
queue-1 = ""
tx-ring-ref = "2306"
rx-ring-ref = "2307"
event-channel-tx = "39"
event-channel-rx = "40"
queue-2 = ""
tx-ring-ref = "2308"
rx-ring-ref = "2309"
event-channel-tx = "41"
event-channel-rx = "42"
queue-3 = ""
tx-ring-ref = "2310"
rx-ring-ref = "2311"
event-channel-tx = "43"
event-channel-rx = "44"
request-rx-copy = "1"
feature-rx-notify = "1"
feature-sg = "1"
feature-gso-tcpv4 = "1"
feature-gso-tcpv6 = "1"
feature-ipv6-csum-offload = "1"
control = ""
shutdown = ""
feature-poweroff = ""
feature-reboot = ""
feature-suspend = ""
feature-s3 = ""
feature-s4 = ""
sysrq = ""
platform-feature-multiprocessor-suspend = "1"
platform-feature-xs_reset_watches = "1"
hvmloader = ""
bios = "seabios"
allow-memory-relocate = "0"
data = ""
drivers = ""
feature = ""
attr = ""
domid = "38"
store = ""
port = "1"
ring-ref = "1044476"
platform = ""
acpi = "1"
acpi_s3 = "1"
acpi_s4 = "1"
acpi_laptop_slate = "0"
device-model = "qemu_xen"
console = ""
backend = "/local/domain/0/backend/console/38/0"
backend-id = "0"
limit = "1048576"
type = "xenconsoled"
output = "pty"
tty = "/dev/pts/3"
port = "2"
ring-ref = "1044479"
vnc-listen = "127.0.0.1"
vnc-port = "5900"
image = ""
device-model-pid = "8783"
serial = ""
0 = ""
tty = "/dev/pts/4"
vm = ""
9a6857f8-05a0-4c8d-92c3-1ea04bc28c32 = ""
name = "guest_centos7"
uuid = "9a6857f8-05a0-4c8d-92c3-1ea04bc28c32"
rtc = ""
timeoffset = ""
image = ""
ostype = "hvm"
start_time = "1561550310.23"
libxl = ""
pciback = ""
0000-88-00-0 = ""
driver_path = "/sys/bus/pci/drivers/intel_nnp"
0 = ""
device = ""
vbd = ""
51712 = ""
frontend = "/local/domain/0/device/vbd/51712"
backend = "/local/domain/0/backend/qdisk/0/51712"
params = "qcow2:centos.qcow2"
frontend-id = "0"
online = "1"
removable = "0"
bootable = "1"
state = "1"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
38 = ""
device = ""
vbd = ""
51712 = ""
frontend = "/local/domain/38/device/vbd/51712"
backend = "/local/domain/0/backend/qdisk/38/51712"
params = "qcow2:/home/nervalusr/chao/centos.qcow2"
frontend-id = "38"
online = "1"
removable = "0"
bootable = "1"
state = "1"
dev = "xvda"
type = "qdisk"
mode = "w"
device-type = "disk"
discard-enable = "1"
console = ""
0 = ""
frontend = "/local/domain/38/console"
backend = "/local/domain/0/backend/console/38/0"
frontend-id = "38"
online = "1"
state = "1"
protocol = "vt100"
vkbd = ""
0 = ""
frontend = "/local/domain/38/device/vkbd/0"
backend = "/local/domain/0/backend/vkbd/38/0"
frontend-id = "38"
online = "1"
state = "1"
backend-type = "qemu"
vif = ""
0 = ""
frontend = "/local/domain/38/device/vif/0"
backend = "/local/domain/0/backend/vif/38/0"
frontend-id = "38"
online = "1"
state = "1"
script = "/etc/xen/scripts/vif-bridge"
mac = "22:16:3e:00:9f:a7"
bridge = "xenbr0"
handle = "0"
type = "vif_ioemu"
type = "hvm"
dm-version = "qemu_xen"
[-- Attachment #4: Type: text/plain, Size: 157 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 14:34 ` Chao Gao
@ 2019-06-26 14:44 ` Juergen Gross
0 siblings, 0 replies; 11+ messages in thread
From: Juergen Gross @ 2019-06-26 14:44 UTC (permalink / raw)
To: Chao Gao
Cc: Stefano Stabellini, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau
On 26.06.19 16:34, Chao Gao wrote:
> On Wed, Jun 26, 2019 at 03:36:35PM +0200, Juergen Gross wrote:
>> On 26.06.19 14:21, Chao Gao wrote:
>>> On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
>>>> On 24.06.19 20:47, Stefano Stabellini wrote:
>>>>> + xen-devel
>>>>>
>>>>> On Mon, 24 Jun 2019, Stefano Stabellini wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I might have found a bug with PCI passthrough to a Linux HVM guest on
>>>>>> x86 with Xen 4.12. It is not easy for me to get access, and especially
>>>>>> change components, on this particular system, and I don't have access to
>>>>>> other x86 boxes at the moment, so apologies for the partial information
>>>>>> report. The setup is as follow:
>>>>>>
>>>>>> - two PCI devices have been assigned to a HVM guest, everything is fine
>>>>>> - reboot the guest from inside, i.e. `reboot' in Linux
>>>>>> - after the reboot completes, only one device is assigned
>>>>>>
>>>>>> Before the reboot, I see all the appropriate xenstore entries for both
>>>>>> devices. Everything is fine. After the reboot, I can only see the
>>>>>> xenstore entries of one device. It is as if the other device
>>>>>> "disappeared" without throwing any errors.
>>>>>>
>>>>>> Have you seen this before? Do you know if it has been fixed in staging?
>>>>>> I noticed this fix which seems to be very relevant:
>>>>>>
>>>>>> https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
>>>>>>
>>>>>> but it is already included in 4.12.
>>>>
>>>> Stefano, could you please try the attached patch? It is only compile
>>>> tested for now.
>>>>
>>>>
>>>> Juergen
>>>
>>> >From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
>>>> From: Juergen Gross <jgross@suse.com>
>>>> To: xen-devel@lists.xenproject.org
>>>> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
>>>> Cc: Wei Liu <wl@xen.org>
>>>> Date: Wed, 26 Jun 2019 08:15:28 +0200
>>>> Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
>>>>
>>>> After a reboot of a guest only the first pci device configuration will
>>>> be retrieved from Xenstore resulting in loss of any further assigned
>>>> passed through pci devices.
>>>>
>>>> The main reason is that all passed through pci devices reside under a
>>>> common root device "0" in Xenstore. So when the device list is rebuilt
>>> >from Xenstore after a reboot the sub-devices below that root device
>>>> need to be selected instead of using the root device number as a
>>>> selector.
>>>>
>>>> Fix that by adding a new member to struct libxl_device_type which when
>>>> set is used to get the number of devices. Add such a member for pci to
>>>> get the correct number of pci devices instead of implying it from the
>>>> number of pci root devices (which will always be 1).
>>>>
>>>> While at it fix the type of libxl__device_pci_from_xs_be() to match
>>>> the one of the .from_xenstore member of struct libxl_device_type. This
>>>> fixes a latent bug checking the return value of a function returning
>>>> void.
>>>>
>>>> Signed-off-by: Juergen Gross <jgross@suse.com>
>>>
>>> Tested-by: Chao Gao <chao.gao@intel.com>
>>
>> Thanks!
>>
>>>
>>> Note that I just tested it on RELEASE-4.12.0, not staging.
>>>
>>> I also found USB device would miss across reboot. Is there someone
>>> willing to fix it too?
>>
>> I'll have a look.
>>
>
> In guest configuration file, it has following two lines:
>
> usbctrl=['type=devicemodel,version=1']
> usbdev=['type=hostdev,hostbus=1,hostaddr=3,controller=0,port=1']
>
> Attachments are output of 'xenstore-ls' before and after reboot
Oh, this seems to be an issue related to qemu parameters. The USB
device is completely emulated by the device model, it is not a PV
device.
Not my area of expertise, I guess.
Juergen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 13:36 ` Juergen Gross
2019-06-26 14:34 ` Chao Gao
@ 2019-06-26 15:18 ` Stefano Stabellini
2019-07-19 19:57 ` Stefano Stabellini
1 sibling, 1 reply; 11+ messages in thread
From: Stefano Stabellini @ 2019-06-26 15:18 UTC (permalink / raw)
To: Juergen Gross
Cc: Stefano Stabellini, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau, Chao Gao
On Wed, 26 Jun 2019, Juergen Gross wrote:
> On 26.06.19 14:21, Chao Gao wrote:
> > On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
> > > On 24.06.19 20:47, Stefano Stabellini wrote:
> > > > + xen-devel
> > > >
> > > > On Mon, 24 Jun 2019, Stefano Stabellini wrote:
> > > > > Hi all,
> > > > >
> > > > > I might have found a bug with PCI passthrough to a Linux HVM guest on
> > > > > x86 with Xen 4.12. It is not easy for me to get access, and especially
> > > > > change components, on this particular system, and I don't have access
> > > > > to
> > > > > other x86 boxes at the moment, so apologies for the partial
> > > > > information
> > > > > report. The setup is as follow:
> > > > >
> > > > > - two PCI devices have been assigned to a HVM guest, everything is
> > > > > fine
> > > > > - reboot the guest from inside, i.e. `reboot' in Linux
> > > > > - after the reboot completes, only one device is assigned
> > > > >
> > > > > Before the reboot, I see all the appropriate xenstore entries for both
> > > > > devices. Everything is fine. After the reboot, I can only see the
> > > > > xenstore entries of one device. It is as if the other device
> > > > > "disappeared" without throwing any errors.
> > > > >
> > > > > Have you seen this before? Do you know if it has been fixed in
> > > > > staging?
> > > > > I noticed this fix which seems to be very relevant:
> > > > >
> > > > > https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
> > > > >
> > > > > but it is already included in 4.12.
> > >
> > > Stefano, could you please try the attached patch? It is only compile
> > > tested for now.
> > >
> > >
> > > Juergen
> >
> > > From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
> > > From: Juergen Gross <jgross@suse.com>
> > > To: xen-devel@lists.xenproject.org
> > > Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> > > Cc: Wei Liu <wl@xen.org>
> > > Date: Wed, 26 Jun 2019 08:15:28 +0200
> > > Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
> > >
> > > After a reboot of a guest only the first pci device configuration will
> > > be retrieved from Xenstore resulting in loss of any further assigned
> > > passed through pci devices.
> > >
> > > The main reason is that all passed through pci devices reside under a
> > > common root device "0" in Xenstore. So when the device list is rebuilt
> > > from Xenstore after a reboot the sub-devices below that root device
> > > need to be selected instead of using the root device number as a
> > > selector.
> > >
> > > Fix that by adding a new member to struct libxl_device_type which when
> > > set is used to get the number of devices. Add such a member for pci to
> > > get the correct number of pci devices instead of implying it from the
> > > number of pci root devices (which will always be 1).
> > >
> > > While at it fix the type of libxl__device_pci_from_xs_be() to match
> > > the one of the .from_xenstore member of struct libxl_device_type. This
> > > fixes a latent bug checking the return value of a function returning
> > > void.
> > >
> > > Signed-off-by: Juergen Gross <jgross@suse.com>
> >
> > Tested-by: Chao Gao <chao.gao@intel.com>
>
> Thanks!
Thank you very much both of you! I'll let you know if it works.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [Xen-devel] PCI Passthrough bug with x86 HVM
2019-06-26 15:18 ` Stefano Stabellini
@ 2019-07-19 19:57 ` Stefano Stabellini
0 siblings, 0 replies; 11+ messages in thread
From: Stefano Stabellini @ 2019-07-19 19:57 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Juergen Gross, wl, andrew.cooper3, jbeulich, xen-devel,
boris.ostrovsky, roger.pau, Chao Gao
On Wed, 26 Jun 2019, Stefano Stabellini wrote:
> On Wed, 26 Jun 2019, Juergen Gross wrote:
> > On 26.06.19 14:21, Chao Gao wrote:
> > > On Wed, Jun 26, 2019 at 08:17:50AM +0200, Juergen Gross wrote:
> > > > On 24.06.19 20:47, Stefano Stabellini wrote:
> > > > > + xen-devel
> > > > >
> > > > > On Mon, 24 Jun 2019, Stefano Stabellini wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > I might have found a bug with PCI passthrough to a Linux HVM guest on
> > > > > > x86 with Xen 4.12. It is not easy for me to get access, and especially
> > > > > > change components, on this particular system, and I don't have access
> > > > > > to
> > > > > > other x86 boxes at the moment, so apologies for the partial
> > > > > > information
> > > > > > report. The setup is as follow:
> > > > > >
> > > > > > - two PCI devices have been assigned to a HVM guest, everything is
> > > > > > fine
> > > > > > - reboot the guest from inside, i.e. `reboot' in Linux
> > > > > > - after the reboot completes, only one device is assigned
> > > > > >
> > > > > > Before the reboot, I see all the appropriate xenstore entries for both
> > > > > > devices. Everything is fine. After the reboot, I can only see the
> > > > > > xenstore entries of one device. It is as if the other device
> > > > > > "disappeared" without throwing any errors.
> > > > > >
> > > > > > Have you seen this before? Do you know if it has been fixed in
> > > > > > staging?
> > > > > > I noticed this fix which seems to be very relevant:
> > > > > >
> > > > > > https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg01616.html
> > > > > >
> > > > > > but it is already included in 4.12.
> > > >
> > > > Stefano, could you please try the attached patch? It is only compile
> > > > tested for now.
> > > >
> > > >
> > > > Juergen
> > >
> > > > From ea95dcdfc60a895cc43baf34c8e0fb088e10008d Mon Sep 17 00:00:00 2001
> > > > From: Juergen Gross <jgross@suse.com>
> > > > To: xen-devel@lists.xenproject.org
> > > > Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> > > > Cc: Wei Liu <wl@xen.org>
> > > > Date: Wed, 26 Jun 2019 08:15:28 +0200
> > > > Subject: [PATCH] libxl: fix pci device re-assigning after domain reboot
> > > >
> > > > After a reboot of a guest only the first pci device configuration will
> > > > be retrieved from Xenstore resulting in loss of any further assigned
> > > > passed through pci devices.
> > > >
> > > > The main reason is that all passed through pci devices reside under a
> > > > common root device "0" in Xenstore. So when the device list is rebuilt
> > > > from Xenstore after a reboot the sub-devices below that root device
> > > > need to be selected instead of using the root device number as a
> > > > selector.
> > > >
> > > > Fix that by adding a new member to struct libxl_device_type which when
> > > > set is used to get the number of devices. Add such a member for pci to
> > > > get the correct number of pci devices instead of implying it from the
> > > > number of pci root devices (which will always be 1).
> > > >
> > > > While at it fix the type of libxl__device_pci_from_xs_be() to match
> > > > the one of the .from_xenstore member of struct libxl_device_type. This
> > > > fixes a latent bug checking the return value of a function returning
> > > > void.
> > > >
> > > > Signed-off-by: Juergen Gross <jgross@suse.com>
> > >
> > > Tested-by: Chao Gao <chao.gao@intel.com>
> >
> > Thanks!
>
> Thank you very much both of you! I'll let you know if it works.
Tested-by: Stefano Stabellini <sstabellini@kernel.org>
Let's get it in the tree, thanks!
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-07-19 19:58 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <alpine.DEB.2.21.1906241135080.2468@sstabellini-ThinkPad-T480s>
2019-06-24 18:47 ` [Xen-devel] PCI Passthrough bug with x86 HVM Stefano Stabellini
2019-06-25 7:44 ` Roger Pau Monné
2019-06-25 22:14 ` Stefano Stabellini
2019-06-26 6:17 ` Juergen Gross
2019-06-26 12:21 ` Chao Gao
2019-06-26 13:36 ` Juergen Gross
2019-06-26 14:34 ` Chao Gao
2019-06-26 14:44 ` Juergen Gross
2019-06-26 15:18 ` Stefano Stabellini
2019-07-19 19:57 ` Stefano Stabellini
[not found] ` <f1b992ab-9e1d-0e37-ebb4-37fc609cfb5d@suse.com>
[not found] ` <alpine.DEB.2.21.1906251427440.5851@sstabellini-ThinkPad-T480s>
2019-06-26 5:05 ` Juergen Gross
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).