* [Xen-devel] Reset pass-thru devices in a VM @ 2019-08-09 8:38 Chao Gao 2019-08-09 8:49 ` Jan Beulich ` (2 more replies) 0 siblings, 3 replies; 9+ messages in thread From: Chao Gao @ 2019-08-09 8:38 UTC (permalink / raw) To: xen-devel Hi everyone, I have a device which only supports secondary bus reset. After being assigned to a VM, it would be placed under host bridge. For devices under host bridge, secondary bus reset is not applicable. Thus, a VM has no way to reset this device. This device's usage would be limited without PCI reset (for example, its driver cannot re-initialize the device properly without PCI reset, which means in VM device won't be usable after unloading the driver), it would be much better if there is a way available to VMs to reset the device. In my mind, a straightfoward solution is to create a virtual bridge for a VM and place the pass-thru device under a virtual bridge. But it isn't supported in Xen (KVM/QEMU supports) and enabling it looks need a lot of efforts. Alternatively, emulating FLR (Function Level Reset) capability for this device might be a feasible way and only needs relatively few changes. I am planning to enable an opt-in feature (like 'permissive') to allow qemu to expose FLR capability to guest for pass-thru devices as long as this device is resetable on dom0 (i.e. the device has 'reset' attribute under its sysfs). And when guest initiates an FLR, qemu just echo 1 to the 'reset' attribute on dom0. Do you think emulating FLR capability is doable? Thanks Chao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 8:38 [Xen-devel] Reset pass-thru devices in a VM Chao Gao @ 2019-08-09 8:49 ` Jan Beulich 2019-08-09 13:24 ` Chao Gao 2019-08-09 12:42 ` Roger Pau Monné 2019-08-26 21:17 ` Pasi Kärkkäinen 2 siblings, 1 reply; 9+ messages in thread From: Jan Beulich @ 2019-08-09 8:49 UTC (permalink / raw) To: Chao Gao; +Cc: xen-devel On 09.08.2019 10:38, Chao Gao wrote: > I have a device which only supports secondary bus reset. After being > assigned to a VM, it would be placed under host bridge. For devices > under host bridge, secondary bus reset is not applicable. Thus, a VM > has no way to reset this device. > > This device's usage would be limited without PCI reset (for example, its > driver cannot re-initialize the device properly without PCI reset, which > means in VM device won't be usable after unloading the driver), it would > be much better if there is a way available to VMs to reset the device. > > In my mind, a straightfoward solution is to create a virtual bridge > for a VM and place the pass-thru device under a virtual bridge. But it > isn't supported in Xen (KVM/QEMU supports) and enabling it looks need > a lot of efforts. Meanwhile I think a couple of years ago there was some initial effort to get a newer chipset (Q35 iirc) emulated for HVM guests. > Alternatively, emulating FLR (Function Level Reset) > capability for this device might be a feasible way and only needs > relatively few changes. I am planning to enable an opt-in feature > (like 'permissive') to allow qemu to expose FLR capability to guest for > pass-thru devices as long as this device is resetable on dom0 (i.e. the > device has 'reset' attribute under its sysfs). And when guest initiates > an FLR, qemu just echo 1 to the 'reset' attribute on dom0. > > Do you think emulating FLR capability is doable? Wouldn't a such emulated guest initiated reset affect other devices (likely not under control of this guest) as well? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 8:49 ` Jan Beulich @ 2019-08-09 13:24 ` Chao Gao 2019-08-09 13:23 ` Jan Beulich 0 siblings, 1 reply; 9+ messages in thread From: Chao Gao @ 2019-08-09 13:24 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote: >On 09.08.2019 10:38, Chao Gao wrote: >>I have a device which only supports secondary bus reset. After being >>assigned to a VM, it would be placed under host bridge. For devices >>under host bridge, secondary bus reset is not applicable. Thus, a VM >>has no way to reset this device. >> >>This device's usage would be limited without PCI reset (for example, its >>driver cannot re-initialize the device properly without PCI reset, which >>means in VM device won't be usable after unloading the driver), it would >>be much better if there is a way available to VMs to reset the device. >> >>In my mind, a straightfoward solution is to create a virtual bridge >>for a VM and place the pass-thru device under a virtual bridge. But it >>isn't supported in Xen (KVM/QEMU supports) and enabling it looks need >>a lot of efforts. > >Meanwhile I think a couple of years ago there was some initial effort >to get a newer chipset (Q35 iirc) emulated for HVM guests. Yes. But it seems that no one is working on this feature now. > >>Alternatively, emulating FLR (Function Level Reset) >>capability for this device might be a feasible way and only needs >>relatively few changes. I am planning to enable an opt-in feature >>(like 'permissive') to allow qemu to expose FLR capability to guest for >>pass-thru devices as long as this device is resetable on dom0 (i.e. the >>device has 'reset' attribute under its sysfs). And when guest initiates >>an FLR, qemu just echo 1 to the 'reset' attribute on dom0. >> >>Do you think emulating FLR capability is doable? > >Wouldn't a such emulated guest initiated reset affect other devices >(likely not under control of this guest) as well? No. Linux kernel guarantees that reset to a device won't affect other devices. Otherwise, such device cannot be reset and no 'reset' attribute will be created under device's sysfs. Specfically, the invocation of pci_dev_reset_slot_function() and pci_parent_bus_reset() in pci_probe_reset_function() will check whether the device (function) is the only one under the slot or bus respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is created only if dev->reset_fn is not zero. Thanks Chao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 13:24 ` Chao Gao @ 2019-08-09 13:23 ` Jan Beulich 2019-08-09 14:13 ` Chao Gao 0 siblings, 1 reply; 9+ messages in thread From: Jan Beulich @ 2019-08-09 13:23 UTC (permalink / raw) To: Chao Gao; +Cc: xen-devel On 09.08.2019 15:24, Chao Gao wrote: > On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote: >> On 09.08.2019 10:38, Chao Gao wrote: >>> Alternatively, emulating FLR (Function Level Reset) >>> capability for this device might be a feasible way and only needs >>> relatively few changes. I am planning to enable an opt-in feature >>> (like 'permissive') to allow qemu to expose FLR capability to guest for >>> pass-thru devices as long as this device is resetable on dom0 (i.e. the >>> device has 'reset' attribute under its sysfs). And when guest initiates >>> an FLR, qemu just echo 1 to the 'reset' attribute on dom0. >>> >>> Do you think emulating FLR capability is doable? >> >> Wouldn't a such emulated guest initiated reset affect other devices >> (likely not under control of this guest) as well? > > No. Linux kernel guarantees that reset to a device won't affect > other devices. Otherwise, such device cannot be reset and no > 'reset' attribute will be created under device's sysfs. > Specfically, the invocation of pci_dev_reset_slot_function() and > pci_parent_bus_reset() in pci_probe_reset_function() will check whether > the device (function) is the only one under the slot or bus > respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is > created only if dev->reset_fn is not zero. Ah, good. But then the opposite question arises: How would your proposed change help if the device shares a bus with others? Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 13:23 ` Jan Beulich @ 2019-08-09 14:13 ` Chao Gao 0 siblings, 0 replies; 9+ messages in thread From: Chao Gao @ 2019-08-09 14:13 UTC (permalink / raw) To: Jan Beulich; +Cc: xen-devel On Fri, Aug 09, 2019 at 03:23:59PM +0200, Jan Beulich wrote: >On 09.08.2019 15:24, Chao Gao wrote: >>On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote: >>>On 09.08.2019 10:38, Chao Gao wrote: >>>>Alternatively, emulating FLR (Function Level Reset) >>>>capability for this device might be a feasible way and only needs >>>>relatively few changes. I am planning to enable an opt-in feature >>>>(like 'permissive') to allow qemu to expose FLR capability to guest for >>>>pass-thru devices as long as this device is resetable on dom0 (i.e. the >>>>device has 'reset' attribute under its sysfs). And when guest initiates >>>>an FLR, qemu just echo 1 to the 'reset' attribute on dom0. >>>> >>>>Do you think emulating FLR capability is doable? >>> >>>Wouldn't a such emulated guest initiated reset affect other devices >>>(likely not under control of this guest) as well? >> >>No. Linux kernel guarantees that reset to a device won't affect >>other devices. Otherwise, such device cannot be reset and no >>'reset' attribute will be created under device's sysfs. >>Specfically, the invocation of pci_dev_reset_slot_function() and >>pci_parent_bus_reset() in pci_probe_reset_function() will check whether >>the device (function) is the only one under the slot or bus >>respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is >>created only if dev->reset_fn is not zero. > >Ah, good. But then the opposite question arises: How would your >proposed change help if the device shares a bus with others? It wouldn't. If the device supports any way to reset it in dom0, this change would help. If even in dom0 there is no way to reset a device, it won't help. But I think for such device, it cannot be safely assigned to a VM because we rely on PCI reset to clean up sensitive data in the device programmed by the previous owner. Thanks Chao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 8:38 [Xen-devel] Reset pass-thru devices in a VM Chao Gao 2019-08-09 8:49 ` Jan Beulich @ 2019-08-09 12:42 ` Roger Pau Monné 2019-08-09 13:57 ` Chao Gao 2019-08-26 21:17 ` Pasi Kärkkäinen 2 siblings, 1 reply; 9+ messages in thread From: Roger Pau Monné @ 2019-08-09 12:42 UTC (permalink / raw) To: Chao Gao; +Cc: xen-devel On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote: > Hi everyone, > > I have a device which only supports secondary bus reset. After being > assigned to a VM, it would be placed under host bridge. For devices > under host bridge, secondary bus reset is not applicable. Thus, a VM > has no way to reset this device. I think in general we don't allow guests to perform any kind of reset of PCI devices, that's always in control of the hardware domain. How are for example BARs going to be placed after such reset? > This device's usage would be limited without PCI reset (for example, its > driver cannot re-initialize the device properly without PCI reset, which > means in VM device won't be usable after unloading the driver), it would > be much better if there is a way available to VMs to reset the device. Is this something common (ie: requiring device reset functionality) for drivers to work correctly? So far we seem to have managed to get away without it. > In my mind, a straightfoward solution is to create a virtual bridge > for a VM and place the pass-thru device under a virtual bridge. But it > isn't supported in Xen (KVM/QEMU supports) and enabling it looks need > a lot of efforts. Alternatively, emulating FLR (Function Level Reset) > capability for this device might be a feasible way and only needs > relatively few changes. I am planning to enable an opt-in feature > (like 'permissive') to allow qemu to expose FLR capability to guest for > pass-thru devices as long as this device is resetable on dom0 (i.e. the > device has 'reset' attribute under its sysfs). And when guest initiates > an FLR, qemu just echo 1 to the 'reset' attribute on dom0. So you would expose the device as FLR capable and just implement it as a secondary bus reset on the device model? That seems feasible, but as noted above I would be worried about the resources owned by the device, and how they are going to be placed after such reset. Note you would also have to notify Xen somehow of such reset, so it tears down all the state related to the device. Thanks, Roger. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 12:42 ` Roger Pau Monné @ 2019-08-09 13:57 ` Chao Gao 0 siblings, 0 replies; 9+ messages in thread From: Chao Gao @ 2019-08-09 13:57 UTC (permalink / raw) To: Roger Pau Monné; +Cc: xen-devel On Fri, Aug 09, 2019 at 02:42:09PM +0200, Roger Pau Monné wrote: >On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote: >> Hi everyone, >> >> I have a device which only supports secondary bus reset. After being >> assigned to a VM, it would be placed under host bridge. For devices >> under host bridge, secondary bus reset is not applicable. Thus, a VM >> has no way to reset this device. > >I think in general we don't allow guests to perform any kind of reset >of PCI devices, that's always in control of the hardware domain. But reset is trapped and performed by the hardware domain. I don't think in this way, guest could access registers or gain more permissions over some registers than it should be. > >How are for example BARs going to be placed after such reset? I don't know BARs are relocated after reset. I will figure it out. Considering KVM/Qemu do support device reset, I think there should be some means. > >> This device's usage would be limited without PCI reset (for example, its >> driver cannot re-initialize the device properly without PCI reset, which >> means in VM device won't be usable after unloading the driver), it would >> be much better if there is a way available to VMs to reset the device. > >Is this something common (ie: requiring device reset functionality) >for drivers to work correctly? I don't think it is common. I am not sure whether it is required for GPU pass-thru in some cases. But I believe that I saw some online materials about GPU pass-thru mentioned how to enable FLR for a VM. > >So far we seem to have managed to get away without it. > >> In my mind, a straightfoward solution is to create a virtual bridge >> for a VM and place the pass-thru device under a virtual bridge. But it >> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need >> a lot of efforts. Alternatively, emulating FLR (Function Level Reset) >> capability for this device might be a feasible way and only needs >> relatively few changes. I am planning to enable an opt-in feature >> (like 'permissive') to allow qemu to expose FLR capability to guest for >> pass-thru devices as long as this device is resetable on dom0 (i.e. the >> device has 'reset' attribute under its sysfs). And when guest initiates >> an FLR, qemu just echo 1 to the 'reset' attribute on dom0. > >So you would expose the device as FLR capable and just implement it as >a secondary bus reset on the device model? For the device I mentioned, yes. Actually, for other devices, it can be any supported reset method. There are several ways to reset a function; secondary bus reset is the last choice. > >That seems feasible, but as noted above I would be worried about the >resources owned by the device, and how they are going to be placed >after such reset. Note you would also have to notify Xen somehow of >such reset, so it tears down all the state related to the device. I will figure out what should be done in Xen side. Thanks Chao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-09 8:38 [Xen-devel] Reset pass-thru devices in a VM Chao Gao 2019-08-09 8:49 ` Jan Beulich 2019-08-09 12:42 ` Roger Pau Monné @ 2019-08-26 21:17 ` Pasi Kärkkäinen 2019-08-27 4:40 ` Chao Gao 2 siblings, 1 reply; 9+ messages in thread From: Pasi Kärkkäinen @ 2019-08-26 21:17 UTC (permalink / raw) To: Chao Gao; +Cc: xen-devel Hi Chao, On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote: > Hi everyone, > > I have a device which only supports secondary bus reset. After being > assigned to a VM, it would be placed under host bridge. For devices > under host bridge, secondary bus reset is not applicable. Thus, a VM > has no way to reset this device. > > This device's usage would be limited without PCI reset (for example, its > driver cannot re-initialize the device properly without PCI reset, which > means in VM device won't be usable after unloading the driver), it would > be much better if there is a way available to VMs to reset the device. > > In my mind, a straightfoward solution is to create a virtual bridge > for a VM and place the pass-thru device under a virtual bridge. But it > isn't supported in Xen (KVM/QEMU supports) and enabling it looks need > a lot of efforts. Alternatively, emulating FLR (Function Level Reset) > capability for this device might be a feasible way and only needs > relatively few changes. I am planning to enable an opt-in feature > (like 'permissive') to allow qemu to expose FLR capability to guest for > pass-thru devices as long as this device is resetable on dom0 (i.e. the > device has 'reset' attribute under its sysfs). And when guest initiates > an FLR, qemu just echo 1 to the 'reset' attribute on dom0. > > Do you think emulating FLR capability is doable? > I wonder if these patches from another thread help with your reset issue: https://lists.xen.org/archives/html/xen-devel/2019-08/msg02304.html Thanks, -- Pasi > Thanks > Chao > _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Xen-devel] Reset pass-thru devices in a VM 2019-08-26 21:17 ` Pasi Kärkkäinen @ 2019-08-27 4:40 ` Chao Gao 0 siblings, 0 replies; 9+ messages in thread From: Chao Gao @ 2019-08-27 4:40 UTC (permalink / raw) To: Pasi Kärkkäinen; +Cc: xen-devel On Tue, Aug 27, 2019 at 12:17:28AM +0300, Pasi Kärkkäinen wrote: >Hi Chao, > >On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote: >> Hi everyone, >> >> I have a device which only supports secondary bus reset. After being >> assigned to a VM, it would be placed under host bridge. For devices >> under host bridge, secondary bus reset is not applicable. Thus, a VM >> has no way to reset this device. >> >> This device's usage would be limited without PCI reset (for example, its >> driver cannot re-initialize the device properly without PCI reset, which >> means in VM device won't be usable after unloading the driver), it would >> be much better if there is a way available to VMs to reset the device. >> >> In my mind, a straightfoward solution is to create a virtual bridge >> for a VM and place the pass-thru device under a virtual bridge. But it >> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need >> a lot of efforts. Alternatively, emulating FLR (Function Level Reset) >> capability for this device might be a feasible way and only needs >> relatively few changes. I am planning to enable an opt-in feature >> (like 'permissive') to allow qemu to expose FLR capability to guest for >> pass-thru devices as long as this device is resetable on dom0 (i.e. the >> device has 'reset' attribute under its sysfs). And when guest initiates >> an FLR, qemu just echo 1 to the 'reset' attribute on dom0. >> >> Do you think emulating FLR capability is doable? >> > >I wonder if these patches from another thread help with your reset issue: > >https://lists.xen.org/archives/html/xen-devel/2019-08/msg02304.html Thanks for your attention. The link you provides seems about how host resets a device. Emulating FLR capability is to expose FLR capability to guest such that guest can reset assigned devices. Definitely, qemu would intercept guest's initiating an FLR and redirect it into a device reset on host. Thanks Chao _______________________________________________ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2019-08-27 4:36 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-08-09 8:38 [Xen-devel] Reset pass-thru devices in a VM Chao Gao 2019-08-09 8:49 ` Jan Beulich 2019-08-09 13:24 ` Chao Gao 2019-08-09 13:23 ` Jan Beulich 2019-08-09 14:13 ` Chao Gao 2019-08-09 12:42 ` Roger Pau Monné 2019-08-09 13:57 ` Chao Gao 2019-08-26 21:17 ` Pasi Kärkkäinen 2019-08-27 4:40 ` Chao Gao
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).