[Xen-devel] Reset pass-thru devices in a VM

[Xen-devel] Reset pass-thru devices in a VM

[Chao Gao]

Hi everyone,

I have a device which only supports secondary bus reset. After being
assigned to a VM, it would be placed under host bridge. For devices
under host bridge, secondary bus reset is not applicable. Thus, a VM
has no way to reset this device.

This device's usage would be limited without PCI reset (for example, its
driver cannot re-initialize the device properly without PCI reset, which
means in VM device won't be usable after unloading the driver), it would
be much better if there is a way available to VMs to reset the device.

In my mind, a straightfoward solution is to create a virtual bridge
for a VM and place the pass-thru device under a virtual bridge. But it
isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
a lot of efforts. Alternatively, emulating FLR (Function Level Reset)
capability for this device might be a feasible way and only needs
relatively few changes. I am planning to enable an opt-in feature
(like 'permissive') to allow qemu to expose FLR capability to guest for
pass-thru devices as long as this device is resetable on dom0 (i.e. the
device has 'reset' attribute under its sysfs). And when guest initiates
an FLR, qemu just echo 1 to the 'reset' attribute on dom0.

Do you think emulating FLR capability is doable?

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Jan Beulich]

On 09.08.2019 10:38, Chao Gao wrote:
> I have a device which only supports secondary bus reset. After being
> assigned to a VM, it would be placed under host bridge. For devices
> under host bridge, secondary bus reset is not applicable. Thus, a VM
> has no way to reset this device.
> 
> This device's usage would be limited without PCI reset (for example, its
> driver cannot re-initialize the device properly without PCI reset, which
> means in VM device won't be usable after unloading the driver), it would
> be much better if there is a way available to VMs to reset the device.
> 
> In my mind, a straightfoward solution is to create a virtual bridge
> for a VM and place the pass-thru device under a virtual bridge. But it
> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
> a lot of efforts.

Meanwhile I think a couple of years ago there was some initial effort
to get a newer chipset (Q35 iirc) emulated for HVM guests.

> Alternatively, emulating FLR (Function Level Reset)
> capability for this device might be a feasible way and only needs
> relatively few changes. I am planning to enable an opt-in feature
> (like 'permissive') to allow qemu to expose FLR capability to guest for
> pass-thru devices as long as this device is resetable on dom0 (i.e. the
> device has 'reset' attribute under its sysfs). And when guest initiates
> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
> 
> Do you think emulating FLR capability is doable?

Wouldn't a such emulated guest initiated reset affect other devices
(likely not under control of this guest) as well?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Chao Gao]

On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote:
>On 09.08.2019 10:38, Chao Gao wrote:
>>I have a device which only supports secondary bus reset. After being
>>assigned to a VM, it would be placed under host bridge. For devices
>>under host bridge, secondary bus reset is not applicable. Thus, a VM
>>has no way to reset this device.
>>
>>This device's usage would be limited without PCI reset (for example, its
>>driver cannot re-initialize the device properly without PCI reset, which
>>means in VM device won't be usable after unloading the driver), it would
>>be much better if there is a way available to VMs to reset the device.
>>
>>In my mind, a straightfoward solution is to create a virtual bridge
>>for a VM and place the pass-thru device under a virtual bridge. But it
>>isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
>>a lot of efforts.
>
>Meanwhile I think a couple of years ago there was some initial effort
>to get a newer chipset (Q35 iirc) emulated for HVM guests.

Yes. But it seems that no one is working on this feature now.

>
>>Alternatively, emulating FLR (Function Level Reset)
>>capability for this device might be a feasible way and only needs
>>relatively few changes. I am planning to enable an opt-in feature
>>(like 'permissive') to allow qemu to expose FLR capability to guest for
>>pass-thru devices as long as this device is resetable on dom0 (i.e. the
>>device has 'reset' attribute under its sysfs). And when guest initiates
>>an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
>>
>>Do you think emulating FLR capability is doable?
>
>Wouldn't a such emulated guest initiated reset affect other devices
>(likely not under control of this guest) as well?

No. Linux kernel guarantees that reset to a device won't affect
other devices. Otherwise, such device cannot be reset and no
'reset' attribute will be created under device's sysfs.
Specfically, the invocation of pci_dev_reset_slot_function() and
pci_parent_bus_reset() in pci_probe_reset_function() will check whether
the device (function) is the only one under the slot or bus
respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is
created only if dev->reset_fn is not zero. 

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Jan Beulich]

On 09.08.2019 15:24, Chao Gao wrote:
> On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote:
>> On 09.08.2019 10:38, Chao Gao wrote:
>>> Alternatively, emulating FLR (Function Level Reset)
>>> capability for this device might be a feasible way and only needs
>>> relatively few changes. I am planning to enable an opt-in feature
>>> (like 'permissive') to allow qemu to expose FLR capability to guest for
>>> pass-thru devices as long as this device is resetable on dom0 (i.e. the
>>> device has 'reset' attribute under its sysfs). And when guest initiates
>>> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
>>>
>>> Do you think emulating FLR capability is doable?
>>
>> Wouldn't a such emulated guest initiated reset affect other devices
>> (likely not under control of this guest) as well?
> 
> No. Linux kernel guarantees that reset to a device won't affect
> other devices. Otherwise, such device cannot be reset and no
> 'reset' attribute will be created under device's sysfs.
> Specfically, the invocation of pci_dev_reset_slot_function() and
> pci_parent_bus_reset() in pci_probe_reset_function() will check whether
> the device (function) is the only one under the slot or bus
> respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is
> created only if dev->reset_fn is not zero.

Ah, good. But then the opposite question arises: How would your
proposed change help if the device shares a bus with others?

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Chao Gao]

On Fri, Aug 09, 2019 at 03:23:59PM +0200, Jan Beulich wrote:
>On 09.08.2019 15:24, Chao Gao wrote:
>>On Fri, Aug 09, 2019 at 10:49:32AM +0200, Jan Beulich wrote:
>>>On 09.08.2019 10:38, Chao Gao wrote:
>>>>Alternatively, emulating FLR (Function Level Reset)
>>>>capability for this device might be a feasible way and only needs
>>>>relatively few changes. I am planning to enable an opt-in feature
>>>>(like 'permissive') to allow qemu to expose FLR capability to guest for
>>>>pass-thru devices as long as this device is resetable on dom0 (i.e. the
>>>>device has 'reset' attribute under its sysfs). And when guest initiates
>>>>an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
>>>>
>>>>Do you think emulating FLR capability is doable?
>>>
>>>Wouldn't a such emulated guest initiated reset affect other devices
>>>(likely not under control of this guest) as well?
>>
>>No. Linux kernel guarantees that reset to a device won't affect
>>other devices. Otherwise, such device cannot be reset and no
>>'reset' attribute will be created under device's sysfs.
>>Specfically, the invocation of pci_dev_reset_slot_function() and
>>pci_parent_bus_reset() in pci_probe_reset_function() will check whether
>>the device (function) is the only one under the slot or bus
>>respectively. In pci_create_capabilities_sysfs(), 'reset' attribute is
>>created only if dev->reset_fn is not zero.
>
>Ah, good. But then the opposite question arises: How would your
>proposed change help if the device shares a bus with others?

It wouldn't. If the device supports any way to reset it in dom0, this
change would help. If even in dom0 there is no way to reset a device,
it won't help. But I think for such device, it cannot be safely assigned
to a VM because we rely on PCI reset to clean up sensitive data in the
device programmed by the previous owner.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Roger Pau Monné]

On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote:
> Hi everyone,
> 
> I have a device which only supports secondary bus reset. After being
> assigned to a VM, it would be placed under host bridge. For devices
> under host bridge, secondary bus reset is not applicable. Thus, a VM
> has no way to reset this device.

I think in general we don't allow guests to perform any kind of reset
of PCI devices, that's always in control of the hardware domain.

How are for example BARs going to be placed after such reset?

> This device's usage would be limited without PCI reset (for example, its
> driver cannot re-initialize the device properly without PCI reset, which
> means in VM device won't be usable after unloading the driver), it would
> be much better if there is a way available to VMs to reset the device.

Is this something common (ie: requiring device reset functionality)
for drivers to work correctly?

So far we seem to have managed to get away without it.

> In my mind, a straightfoward solution is to create a virtual bridge
> for a VM and place the pass-thru device under a virtual bridge. But it
> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
> a lot of efforts. Alternatively, emulating FLR (Function Level Reset)
> capability for this device might be a feasible way and only needs
> relatively few changes. I am planning to enable an opt-in feature
> (like 'permissive') to allow qemu to expose FLR capability to guest for
> pass-thru devices as long as this device is resetable on dom0 (i.e. the
> device has 'reset' attribute under its sysfs). And when guest initiates
> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.

So you would expose the device as FLR capable and just implement it as
a secondary bus reset on the device model?

That seems feasible, but as noted above I would be worried about the
resources owned by the device, and how they are going to be placed
after such reset. Note you would also have to notify Xen somehow of
such reset, so it tears down all the state related to the device.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Chao Gao]

On Fri, Aug 09, 2019 at 02:42:09PM +0200, Roger Pau Monné wrote:
>On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote:
>> Hi everyone,
>> 
>> I have a device which only supports secondary bus reset. After being
>> assigned to a VM, it would be placed under host bridge. For devices
>> under host bridge, secondary bus reset is not applicable. Thus, a VM
>> has no way to reset this device.
>
>I think in general we don't allow guests to perform any kind of reset
>of PCI devices, that's always in control of the hardware domain.

But reset is trapped and performed by the hardware domain. I don't think
in this way, guest could access registers or gain more permissions over
some registers than it should be.

>
>How are for example BARs going to be placed after such reset?

I don't know BARs are relocated after reset. I will figure it out.
Considering KVM/Qemu do support device reset, I think there
should be some means.

>
>> This device's usage would be limited without PCI reset (for example, its
>> driver cannot re-initialize the device properly without PCI reset, which
>> means in VM device won't be usable after unloading the driver), it would
>> be much better if there is a way available to VMs to reset the device.
>
>Is this something common (ie: requiring device reset functionality)
>for drivers to work correctly?

I don't think it is common. I am not sure whether it is required for GPU
pass-thru in some cases. But I believe that I saw some online materials
about GPU pass-thru mentioned how to enable FLR for a VM.

>
>So far we seem to have managed to get away without it.
>
>> In my mind, a straightfoward solution is to create a virtual bridge
>> for a VM and place the pass-thru device under a virtual bridge. But it
>> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
>> a lot of efforts. Alternatively, emulating FLR (Function Level Reset)
>> capability for this device might be a feasible way and only needs
>> relatively few changes. I am planning to enable an opt-in feature
>> (like 'permissive') to allow qemu to expose FLR capability to guest for
>> pass-thru devices as long as this device is resetable on dom0 (i.e. the
>> device has 'reset' attribute under its sysfs). And when guest initiates
>> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
>
>So you would expose the device as FLR capable and just implement it as
>a secondary bus reset on the device model?

For the device I mentioned, yes. Actually, for other devices, it can be
any supported reset method. There are several ways to reset a function;
secondary bus reset is the last choice.

>
>That seems feasible, but as noted above I would be worried about the
>resources owned by the device, and how they are going to be placed
>after such reset. Note you would also have to notify Xen somehow of
>such reset, so it tears down all the state related to the device.

I will figure out what should be done in Xen side.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Pasi Kärkkäinen]

Hi Chao,

On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote:
> Hi everyone,
> 
> I have a device which only supports secondary bus reset. After being
> assigned to a VM, it would be placed under host bridge. For devices
> under host bridge, secondary bus reset is not applicable. Thus, a VM
> has no way to reset this device.
> 
> This device's usage would be limited without PCI reset (for example, its
> driver cannot re-initialize the device properly without PCI reset, which
> means in VM device won't be usable after unloading the driver), it would
> be much better if there is a way available to VMs to reset the device.
> 
> In my mind, a straightfoward solution is to create a virtual bridge
> for a VM and place the pass-thru device under a virtual bridge. But it
> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
> a lot of efforts. Alternatively, emulating FLR (Function Level Reset)
> capability for this device might be a feasible way and only needs
> relatively few changes. I am planning to enable an opt-in feature
> (like 'permissive') to allow qemu to expose FLR capability to guest for
> pass-thru devices as long as this device is resetable on dom0 (i.e. the
> device has 'reset' attribute under its sysfs). And when guest initiates
> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
> 
> Do you think emulating FLR capability is doable?
> 

I wonder if these patches from another thread help with your reset issue:

https://lists.xen.org/archives/html/xen-devel/2019-08/msg02304.html


Thanks,

-- Pasi


> Thanks
> Chao
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Reset pass-thru devices in a VM

[Chao Gao]

On Tue, Aug 27, 2019 at 12:17:28AM +0300, Pasi Kärkkäinen wrote:
>Hi Chao,
>
>On Fri, Aug 09, 2019 at 04:38:33PM +0800, Chao Gao wrote:
>> Hi everyone,
>> 
>> I have a device which only supports secondary bus reset. After being
>> assigned to a VM, it would be placed under host bridge. For devices
>> under host bridge, secondary bus reset is not applicable. Thus, a VM
>> has no way to reset this device.
>> 
>> This device's usage would be limited without PCI reset (for example, its
>> driver cannot re-initialize the device properly without PCI reset, which
>> means in VM device won't be usable after unloading the driver), it would
>> be much better if there is a way available to VMs to reset the device.
>> 
>> In my mind, a straightfoward solution is to create a virtual bridge
>> for a VM and place the pass-thru device under a virtual bridge. But it
>> isn't supported in Xen (KVM/QEMU supports) and enabling it looks need
>> a lot of efforts. Alternatively, emulating FLR (Function Level Reset)
>> capability for this device might be a feasible way and only needs
>> relatively few changes. I am planning to enable an opt-in feature
>> (like 'permissive') to allow qemu to expose FLR capability to guest for
>> pass-thru devices as long as this device is resetable on dom0 (i.e. the
>> device has 'reset' attribute under its sysfs). And when guest initiates
>> an FLR, qemu just echo 1 to the 'reset' attribute on dom0.
>> 
>> Do you think emulating FLR capability is doable?
>> 
>
>I wonder if these patches from another thread help with your reset issue:
>
>https://lists.xen.org/archives/html/xen-devel/2019-08/msg02304.html

Thanks for your attention.

The link you provides seems about how host resets a device. Emulating FLR
capability is to expose FLR capability to guest such that guest can
reset assigned devices. Definitely, qemu would intercept guest's
initiating an FLR and redirect it into a device reset on host.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel