Re: [PATCH v6 00/20] Add VFIO mediated device support and DEV-MSI support for the idxd driver

[Dave Jiang <dave.jiang@intel.com>]

On 6/8/2021 9:02 AM, Dave Jiang wrote:
>
> On 6/7/2021 12:11 PM, Jason Gunthorpe wrote:
>> On Mon, Jun 07, 2021 at 11:13:04AM -0700, Dave Jiang wrote:
>>
>>> So in step 1, we 'tag' the wq to be dedicated to guest usage and put 
>>> the
>>> hardware wq into enable state. For a dedicated mode wq, we can 
>>> definitely
>>> just register directly and skip the mdev step. For a shared wq mode, 
>>> we can
>>> have multiple mdev running on top of a single wq. So we need some 
>>> way to
>>> create more mdevs. We can either go with the existing established 
>>> creation
>>> path by mdev, or invent something custom for the driver as Jason 
>>> suggested
>>> to accomodate additional virtual devices for guests. We implemented 
>>> the mdev
>>> path originally with consideration of mdev is established and has a 
>>> known
>>> interface already.
>> It sounds like you could just as easially have a 'create new vfio'
>> file under the idxd sysfs.. Especially since you already have a bus
>> and dynamic vfio specific things being created on this bus.
>
> Will explore this and using of 'struct vfio_device' without mdev.
>
Hi Jason. I hacked the idxd driver to remove mdev association and use 
vfio_device directly. Ran into some issues. Specifically mdev does some 
special handling when it comes to iommu domain. When we hit 
vfio_iommu_type1_attach_group(), there's a branch in there for 
mdev_bus_type. It sets the group with mdev_group flag, which later has 
effect of special handling for iommu_attach_group. And in addition, it 
ends up switching the bus to pci_bus_type before iommu_domain_alloc() is 
called.  Do we need to provide similar type of handling for vfio_device 
that are not backed by an entire PCI device like vfio_pci? Not sure it's 
the right thing to do to attach these devices to pci_bus_type directly.