Re: [PATCH v5 09/19] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET

From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Alex Williamson <alex.williamson@redhat.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"joro@8bytes.org" <joro@8bytes.org>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"cohuck@redhat.com" <cohuck@redhat.com>,
	"eric.auger@redhat.com" <eric.auger@redhat.com>,
	"nicolinc@nvidia.com" <nicolinc@nvidia.com>,
	"kvm@vger.kernel.org" <kvm@vger.kernel.org>,
	"mjrosato@linux.ibm.com" <mjrosato@linux.ibm.com>,
	"chao.p.peng@linux.intel.com" <chao.p.peng@linux.intel.com>,
	"yi.y.sun@linux.intel.com" <yi.y.sun@linux.intel.com>,
	"peterx@redhat.com" <peterx@redhat.com>,
	"jasowang@redhat.com" <jasowang@redhat.com>,
	"shameerali.kolothum.thodi@huawei.com" 
	<shameerali.kolothum.thodi@huawei.com>,
	"lulu@redhat.com" <lulu@redhat.com>,
	"suravee.suthikulpanit@amd.com" <suravee.suthikulpanit@amd.com>,
	"intel-gvt-dev@lists.freedesktop.org" 
	<intel-gvt-dev@lists.freedesktop.org>,
	"intel-gfx@lists.freedesktop.org"
	<intel-gfx@lists.freedesktop.org>,
	"linux-s390@vger.kernel.org" <linux-s390@vger.kernel.org>,
	"Hao, Xudong" <xudong.hao@intel.com>,
	"Zhao, Yan Y" <yan.y.zhao@intel.com>,
	"Xu, Terrence" <terrence.xu@intel.com>
Subject: Re: [PATCH v5 09/19] vfio/pci: Allow passing zero-length fd array in VFIO_DEVICE_PCI_HOT_RESET
Date: Tue, 7 Mar 2023 08:36:30 -0400	[thread overview]
Message-ID: <ZAcvzvhkt9QhCmdi@nvidia.com> (raw)
In-Reply-To: <BN9PR11MB52760ABC93BCE7FB53A131038CB79@BN9PR11MB5276.namprd11.prod.outlook.com>

On Tue, Mar 07, 2023 at 02:31:11AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Monday, March 6, 2023 9:17 PM
> > 
> > On Fri, Mar 03, 2023 at 09:55:42AM -0700, Alex Williamson wrote:
> > 
> > > I can't think of a reason DPDK couldn't use hot-reset.  If we want to
> > > make it a policy, it should be enforced by code, but creating that
> > > policy based on a difficulty in supporting that mode with iommufd isn't
> > > great.
> > 
> > On the other hand adding code to allow device FDs in the hot reset
> > path that is never used and never tested isn't great either..
> > 
> > hot-reset does work for DPDK, it just doesn't work in the case where
> > DPDK would have many VFIO devices open and they have overlapping
> > device sets. Which, again, is something it doesn't do.
> > 
> > IMHO we should leave it out of the kernel and wait for a no-iommu user
> > to come forward that wants hot-reset of many devices. Then we can add
> > and test the device FD part. Most likely such a thing will never come
> > at this point.
> > 
> 
> I think we don't need to have this tradeoff if following Yi's last proposal
> which requires every opened device in the set to be covered by the
> device fd array. with dev_set->lock held in the reset/open path this is
> a safe measure and fully contained in vfio-pci w/o need of further
> checking noiommu or iommufd.

I really prefer the 'use the iommufd option' still exist, it is so
much cleaner and easier for the actual users of this API. We've lost
the point by worrying about no iommu.

Jason