Re: [PATCH RFC v1 0/3] Introduce vfio-pci-core subsystem

[Max Gurtovoy <mgurtovoy@nvidia.com>]

On 1/28/2021 11:02 PM, Alex Williamson wrote:
> On Thu, 28 Jan 2021 17:29:30 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
>
>> On Tue, 26 Jan 2021 15:27:43 +0200
>> Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>> On 1/26/2021 5:34 AM, Alex Williamson wrote:
>>>> On Mon, 25 Jan 2021 20:45:22 -0400
>>>> Jason Gunthorpe <jgg@nvidia.com> wrote:
>>>>     
>>>>> On Mon, Jan 25, 2021 at 04:31:51PM -0700, Alex Williamson wrote:
>>>>>> extensions potentially break vendor drivers, etc.  We're only even hand
>>>>>> waving that existing device specific support could be farmed out to new
>>>>>> device specific drivers without even going to the effort to prove that.
>>>>> This is a RFC, not a complete patch series. The RFC is to get feedback
>>>>> on the general design before everyone comits alot of resources and
>>>>> positions get dug in.
>>>>>
>>>>> Do you really think the existing device specific support would be a
>>>>> problem to lift? It already looks pretty clean with the
>>>>> vfio_pci_regops, looks easy enough to lift to the parent.
>>>>>     
>>>>>> So far the TODOs rather mask the dirty little secrets of the
>>>>>> extension rather than showing how a vendor derived driver needs to
>>>>>> root around in struct vfio_pci_device to do something useful, so
>>>>>> probably porting actual device specific support rather than further
>>>>>> hand waving would be more helpful.
>>>>> It would be helpful to get actual feedback on the high level design -
>>>>> someting like this was already tried in May and didn't go anywhere -
>>>>> are you surprised that we are reluctant to commit alot of resources
>>>>> doing a complete job just to have it go nowhere again?
>>>> That's not really what I'm getting from your feedback, indicating
>>>> vfio-pci is essentially done, the mlx stub driver should be enough to
>>>> see the direction, and additional concerns can be handled with TODO
>>>> comments.  Sorry if this is not construed as actual feedback, I think
>>>> both Connie and I are making an effort to understand this and being
>>>> hampered by lack of a clear api or a vendor driver that's anything more
>>>> than vfio-pci plus an aux bus interface.  Thanks,
>>> I think I got the main idea and I'll try to summarize it:
>>>
>>> The separation to vfio-pci.ko and vfio-pci-core.ko is acceptable, and we
>>> do need it to be able to create vendor-vfio-pci.ko driver in the future
>>> to include vendor special souse inside.
>> One other thing I'd like to bring up: What needs to be done in
>> userspace? Does a userspace driver like QEMU need changes to actually
>> exploit this? Does management software like libvirt need to be involved
>> in decision making, or does it just need to provide the knobs to make
>> the driver configurable?
> I'm still pretty nervous about the userspace aspect of this as well.
> QEMU and other actual vfio drivers are probably the least affected,
> at least for QEMU, it'll happily open any device that has a pointer to
> an IOMMU group that's reflected as a vfio group device.  Tools like
> libvirt, on the other hand, actually do driver binding and we need to
> consider how they make driver decisions. Jason suggested that the
> vfio-pci driver ought to be only spec compliant behavior, which sounds
> like some deprecation process of splitting out the IGD, NVLink, zpci,
> etc. features into sub-drivers and eventually removing that device
> specific support from vfio-pci.  Would we expect libvirt to know, "this
> is an 8086 graphics device, try to bind it to vfio-pci-igd" or "uname
> -m says we're running on s390, try to bind it to vfio-zpci"?  Maybe we
> expect derived drivers to only bind to devices they recognize, so
> libvirt could blindly try a whole chain of drivers, ending in vfio-pci.
> Obviously if we have competing drivers that support the same device in
> different ways, that quickly falls apart.

I think we can leave common arch specific stuff, such as s390 (IIUC) in 
the core driver. And only create vfio_pci drivers for 
vendor/device/subvendor specific stuff.

Also, the competing drivers issue can also happen today, right ? after 
adding new_id to vfio_pci I don't know how linux will behave if we'll 
plug new device with same id to the system. which driver will probe it ?

I don't really afraid of competing drivers since we can ask from vendor 
vfio pci_drivers to add vendor_id, device_id, subsystem_vendor and 
subsystem_device so we won't have this problem. I don't think that there 
will be 2 drivers that drive the same device with these 4 ids.

Userspace tool can have a map of ids to drivers and bind the device to 
the right vfio-pci vendor driver if it has one. if not, bind to vfio_pci.ko.

>
> Libvirt could also expand its available driver models for the user to
> specify a variant, I'd support that for overriding a choice that libvirt
> might make otherwise, but forcing the user to know this information is
> just passing the buck.

We can add a code to libvirt as mentioned above.

>
> Some derived drivers could probably actually include device IDs rather
> than only relying on dynamic ids, but then we get into the problem that
> we're competing with native host driver for a device.  The aux bus
> example here is essentially the least troublesome variation since it
> works in conjunction with the native host driver rather than replacing
> it.  Thanks,

same competition after we add new_id to vfio_pci, right ?

>
> Alex

A pointer to needed additions to libvirt will be awsome (or any other hint).

I'll send the V2 soon and then move to libvirt.

-Max.