Re: [PATCH V1 vfio 0/9] Introduce a vfio driver over virtio devices

[Jason Gunthorpe <jgg@nvidia.com>]

On Mon, Oct 23, 2023 at 10:45:48AM -0600, Alex Williamson wrote:
> On Mon, 23 Oct 2023 13:20:43 -0300
> Jason Gunthorpe <jgg@nvidia.com> wrote:
> 
> > On Mon, Oct 23, 2023 at 10:09:13AM -0600, Alex Williamson wrote:
> > > On Mon, 23 Oct 2023 12:42:57 -0300
> > > Jason Gunthorpe <jgg@nvidia.com> wrote:
> > >   
> > > > On Mon, Oct 23, 2023 at 09:33:23AM -0600, Alex Williamson wrote:
> > > >   
> > > > > > Alex,
> > > > > > Are you fine to leave the provisioning of the VF including the control 
> > > > > > of its transitional capability in the device hands as was suggested by 
> > > > > > Jason ?    
> > > > > 
> > > > > If this is the standard we're going to follow, ie. profiling of a
> > > > > device is expected to occur prior to the probe of the vfio-pci variant
> > > > > driver, then we should get the out-of-tree NVIDIA vGPU driver on board
> > > > > with this too.    
> > > > 
> > > > Those GPU drivers are using mdev not vfio-pci..  
> > > 
> > > The SR-IOV mdev vGPUs rely on the IOMMU backing device support which
> > > was removed from upstream.    
> > 
> > It wasn't, but it changed forms.
> > 
> > mdev is a sysfs framework for managing lifecycle with GUIDs only.
> > 
> > The thing using mdev can call vfio_register_emulated_iommu_dev() or
> > vfio_register_group_dev(). 
> > 
> > It doesn't matter to the mdev stuff.
> > 
> > The thing using mdev is responsible to get the struct device to pass
> > to vfio_register_group_dev()
> 
> Are we describing what can be done (possibly limited to out-of-tree
> drivers) or what should be done and would be accepted upstream?

Beyond disliking mdev, I'm not really set on how we should try to
setup an extensively mediated PCI SRIOV driver. There is quite a lot
of similarity to SIOV, so it may be the right answer is to put SIOV
and this special mediated SRIOV case on the same, new, infrastructure.

SIOV can't use variant vfio PCI drivers.

mdev guid lifecycle is really ugly and quite limited anyhow.

So I've been thinking we need something else.

> I'm under the impression that mdev has been redefined to be more
> narrowly focused for emulated IOMMU devices and that devices based
> around a PCI VF should be making use of a vfio-pci variant driver.

I've been viewing mdev as legacy, just let it die off with the S390
drivers and Intel GPU as the only users, ever.

When we solve the SIOV issue we should come with something that can
absorb what S390/GPU need too.

At the end of the day we need an API to create /dev/vfioXX on demand,
to configure them before creating them, and the destroy them. It
doesn't matter at all how the driver that owns vfioXX operates, it
will call the right iommufd APIs for RID/PASID/access/etc to do
whatever its thing is.

It would be wonderful if we could get to the point where the new
interface can also create/destroy SRIOV vfios directly too.

> Are you suggesting it's the vendor's choice based on whether they want
> the mdev lifecycle support?

So, in tree I would like to discourage new mdev drivers. Out of tree,
I don't care, the APIs exist if people want to build things with them
then they get the usual out of tree cavet.

> We've defined certain aspects of the vfio-mdev interface as only
> available for emulated IOMMU devices, ex. page pinning.  Thanks,

Did we?

iommufd made it up to the driver to decide what to do, and a driver
can certainly create a concurrent iommufd_access and iommufd_device if
it wants.

AFAICT the container stuff doesn't check, drivers can do both
concurrently?

Jason