Re: [PATCH v2 0/9] Introduce vfio-pci-core subsystem

From: Christoph Hellwig <hch@infradead.org>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Tian, Kevin" <kevin.tian@intel.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"liranl@nvidia.com" <liranl@nvidia.com>,
	"oren@nvidia.com" <oren@nvidia.com>,
	"tzahio@nvidia.com" <tzahio@nvidia.com>,
	"leonro@nvidia.com" <leonro@nvidia.com>,
	"yarong@nvidia.com" <yarong@nvidia.com>,
	"aviadye@nvidia.com" <aviadye@nvidia.com>,
	"shahafs@nvidia.com" <shahafs@nvidia.com>,
	"artemp@nvidia.com" <artemp@nvidia.com>,
	"kwankhede@nvidia.com" <kwankhede@nvidia.com>,
	"ACurrid@nvidia.com" <ACurrid@nvidia.com>,
	"gmataev@nvidia.com" <gmataev@nvidia.com>,
	"cjia@nvidia.com" <cjia@nvidia.com>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"yishaih@nvidia.com" <yishaih@nvidia.com>,
	"aik@ozlabs.ru" <aik@ozlabs.ru>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>
Subject: Re: [PATCH v2 0/9] Introduce vfio-pci-core subsystem
Date: Thu, 11 Feb 2021 08:36:21 +0000	[thread overview]
Message-ID: <20210211083621.GA2378134@infradead.org> (raw)
In-Reply-To: <20210210133452.GW4247@nvidia.com>

On Wed, Feb 10, 2021 at 09:34:52AM -0400, Jason Gunthorpe wrote:
> > I'm a bit confused about the change from v1 to v2, especially about
> > how to inject module specific operations. From live migration p.o.v
> > it may requires two hook points at least for some devices (e.g. i40e 
> > in original Yan's example):
> 
> IMHO, it was too soon to give up on putting the vfio_device_ops in the
> final driver- we should try to define a reasonable public/private
> split of vfio_pci_device as is the norm in the kernel. No reason we
> can't achieve that.
> 
> >  register a migration region and intercept guest writes to specific
> > registers. [PATCH 4/9] demonstrates the former but not the latter
> > (which is allowed in v1).
> 
> And this is why, the ROI to wrapper every vfio op in a PCI op just to
> keep vfio_pci_device completely private is poor :(

Yes.  If Alex has a strong preference to keep some values private
a split between vfio_pci_device vfio_pci_device_priv might be doable,
but it is somewhat silly.

> > Then another question. Once we have this framework in place, do we 
> > mandate this approach for any vendor specific tweak or still allow
> > doing it as vfio_pci_core extensions (such as igd and zdev in this
> > series)?
> 
> I would say no to any further vfio_pci_core extensions that are tied
> to specific PCI devices. Things like zdev are platform features, they
> are not tied to specific PCI devices

Yes, ZDEV is just a special case of exposing extra information for any
PCI device on s390.  It does not fit any split up vfio_pci framework.
In fact I wonder why it even has its own config option.

> > vfio-mdev is just the channel to bring VFIO APIs through mdev core
> > to underlying vendor specific mdev device driver, which is already
> > granted flexibility to tweak whatever needs through mdev_parent_ops.
> 
> This is the second thing, and it could just be deleted. The actual
> final mdev driver can just use vfio_device_ops directly. The
> redirection shim in vfio_mdev.c doesn't add value.

Yes, that would simplify a lot of things.