[Question] Indefinitely block in the host when remove the PF driver

[Question] Indefinitely block in the host when remove the PF driver

[Yicong Yang]

When I try to remove the PF driver in the host, the process will be blocked
if the related VF of the device is added in the Qemu as an iEP.

here's what I got in the host:

[root@localhost 0000:75:00.0]# rmmod hisi_zip
[99760.571352] vfio-pci 0000:75:00.1: Relaying device request to user (#0)
[99862.992099] vfio-pci 0000:75:00.1: Relaying device request to user (#10)
[...]

and in the Qemu:

estuary:/$ lspci -tv
-[0000:00]-+-00.0  Device 1b36:0008
           +-01.0  Device 1af4:1000
           +-02.0  Device 1af4:1009
           \-03.0  Device 19e5:a251 <----- the related VF device
estuary:/$ qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
[...]

The rmmod process will be blocked until I kill the Qemu process. That's the only way if I
want to end the rmmod.

So my question is: is such block reasonable? If the VF devcie is occupied or doesn't
support hotplug in the Qemu, shouldn't we fail the rmmod and return something like -EBUSY
rather than make the host blocked indefinitely?

Add the VF under a pcie root port will avoid this. Is it encouraged to always
add the VF under a pcie root port rather than directly add it as an iEP?

Thanks,
Yicong

Re: [Question] Indefinitely block in the host when remove the PF driver

[Alex Williamson]

On Fri, 30 Apr 2021 15:57:47 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:

> When I try to remove the PF driver in the host, the process will be blocked
> if the related VF of the device is added in the Qemu as an iEP.
> 
> here's what I got in the host:
> 
> [root@localhost 0000:75:00.0]# rmmod hisi_zip
> [99760.571352] vfio-pci 0000:75:00.1: Relaying device request to user (#0)
> [99862.992099] vfio-pci 0000:75:00.1: Relaying device request to user (#10)
> [...]
> 
> and in the Qemu:
> 
> estuary:/$ lspci -tv
> -[0000:00]-+-00.0  Device 1b36:0008
>            +-01.0  Device 1af4:1000
>            +-02.0  Device 1af4:1009
>            \-03.0  Device 19e5:a251 <----- the related VF device
> estuary:/$ qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> [...]
> 
> The rmmod process will be blocked until I kill the Qemu process. That's the only way if I
> want to end the rmmod.
> 
> So my question is: is such block reasonable? If the VF devcie is occupied or doesn't
> support hotplug in the Qemu, shouldn't we fail the rmmod and return something like -EBUSY
> rather than make the host blocked indefinitely?

Where would we return -EBUSY?  pci_driver.remove() returns void.
Without blocking, I think our only option would be to kill the user
process.
 
> Add the VF under a pcie root port will avoid this. Is it encouraged to always
> add the VF under a pcie root port rather than directly add it as an iEP?

Releasing a device via the vfio request interrupt is always a
cooperative process currently, the VM needs to be configured such that
the device is capable of being unplugged and the guest needs to respond
to the ejection request.  Thanks,

Alex

Re: [Question] Indefinitely block in the host when remove the PF driver

[Yicong Yang]

[ +qemu-devel ]

On 2021/4/30 22:29, Alex Williamson wrote:
> On Fri, 30 Apr 2021 15:57:47 +0800
> Yicong Yang <yangyicong@hisilicon.com> wrote:
> 
>> When I try to remove the PF driver in the host, the process will be blocked
>> if the related VF of the device is added in the Qemu as an iEP.
>>
>> here's what I got in the host:
>>
>> [root@localhost 0000:75:00.0]# rmmod hisi_zip
>> [99760.571352] vfio-pci 0000:75:00.1: Relaying device request to user (#0)
>> [99862.992099] vfio-pci 0000:75:00.1: Relaying device request to user (#10)
>> [...]
>>
>> and in the Qemu:
>>
>> estuary:/$ lspci -tv
>> -[0000:00]-+-00.0  Device 1b36:0008
>>            +-01.0  Device 1af4:1000
>>            +-02.0  Device 1af4:1009
>>            \-03.0  Device 19e5:a251 <----- the related VF device
>> estuary:/$ qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
>> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
>> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
>> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
>> [...]
>>
>> The rmmod process will be blocked until I kill the Qemu process. That's the only way if I
>> want to end the rmmod.
>>
>> So my question is: is such block reasonable? If the VF devcie is occupied or doesn't
>> support hotplug in the Qemu, shouldn't we fail the rmmod and return something like -EBUSY
>> rather than make the host blocked indefinitely?
> 
> Where would we return -EBUSY?  pci_driver.remove() returns void.
> Without blocking, I think our only option would be to kill the user
> process.
>  

yes. the remove() callback of pci_driver doesn't provide a way to abort the process.

>> Add the VF under a pcie root port will avoid this. Is it encouraged to always
>> add the VF under a pcie root port rather than directly add it as an iEP?
> 
> Releasing a device via the vfio request interrupt is always a
> cooperative process currently, the VM needs to be configured such that
> the device is capable of being unplugged and the guest needs to respond
> to the ejection request.  Thanks,
> 

Does it make sense to abort the VM creation and give some warnings if user try to
pass a vfio pci device to the Qemu and doesn't attach it to a hotpluggable
bridge? Currently I think there isn't such a mechanism in Qemu.

Thanks,
Yicong

Re: [Question] Indefinitely block in the host when remove the PF driver

[Alex Williamson]

On Tue, 11 May 2021 11:44:49 +0800
Yicong Yang <yangyicong@hisilicon.com> wrote:

> [ +qemu-devel ]
> 
> On 2021/4/30 22:29, Alex Williamson wrote:
> > On Fri, 30 Apr 2021 15:57:47 +0800
> > Yicong Yang <yangyicong@hisilicon.com> wrote:
> >   
> >> When I try to remove the PF driver in the host, the process will be blocked
> >> if the related VF of the device is added in the Qemu as an iEP.
> >>
> >> here's what I got in the host:
> >>
> >> [root@localhost 0000:75:00.0]# rmmod hisi_zip
> >> [99760.571352] vfio-pci 0000:75:00.1: Relaying device request to user (#0)
> >> [99862.992099] vfio-pci 0000:75:00.1: Relaying device request to user (#10)
> >> [...]
> >>
> >> and in the Qemu:
> >>
> >> estuary:/$ lspci -tv
> >> -[0000:00]-+-00.0  Device 1b36:0008
> >>            +-01.0  Device 1af4:1000
> >>            +-02.0  Device 1af4:1009
> >>            \-03.0  Device 19e5:a251 <----- the related VF device
> >> estuary:/$ qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> >> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> >> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> >> qemu-system-aarch64: warning: vfio 0000:75:00.1: Bus 'pcie.0' does not support hotplugging
> >> [...]
> >>
> >> The rmmod process will be blocked until I kill the Qemu process. That's the only way if I
> >> want to end the rmmod.
> >>
> >> So my question is: is such block reasonable? If the VF devcie is occupied or doesn't
> >> support hotplug in the Qemu, shouldn't we fail the rmmod and return something like -EBUSY
> >> rather than make the host blocked indefinitely?  
> > 
> > Where would we return -EBUSY?  pci_driver.remove() returns void.
> > Without blocking, I think our only option would be to kill the user
> > process.
> >    
> 
> yes. the remove() callback of pci_driver doesn't provide a way to abort the process.
> 
> >> Add the VF under a pcie root port will avoid this. Is it encouraged to always
> >> add the VF under a pcie root port rather than directly add it as an iEP?  
> > 
> > Releasing a device via the vfio request interrupt is always a
> > cooperative process currently, the VM needs to be configured such that
> > the device is capable of being unplugged and the guest needs to respond
> > to the ejection request.  Thanks,
> >   
> 
> Does it make sense to abort the VM creation and give some warnings if user try to
> pass a vfio pci device to the Qemu and doesn't attach it to a hotpluggable
> bridge? Currently I think there isn't such a mechanism in Qemu.

You're essentially trying to define a usage policy and pick somewhere
to impose it.  I think QEMU is not the right place.  There are plenty
of valid assigned device configurations where the device is not
hotpluggable.  You therefore either need to look up in the stack if
your environment demands that VM configurations should always be able
to release devices at the request of the kernel, or down in the stack
if you believe the kernel has an obligation to take that device if the
user fails to respond to a device request.  We've shied away from the
latter because it generally involves killing the holding process,
either directly or by closing off access to the device, where in the
case of mmaps to the device, ongoing access would result in a SIGBUS to
the process anyway.  I wouldn't object to the kernel having a right to
do this, but it's not something that has reached a high priority.
Thanks,

Alex