RE: Virtualizing MSI-X on IMS via VFIO

From: "Tian, Kevin" <kevin.tian@intel.com>
To: Alex Williamson <alex.williamson@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: Jason Gunthorpe <jgg@nvidia.com>,
	"Dey, Megha" <megha.dey@intel.com>,
	"Raj, Ashok" <ashok.raj@intel.com>,
	"Pan, Jacob jun" <jacob.jun.pan@intel.com>,
	"Jiang, Dave" <dave.jiang@intel.com>,
	"Liu, Yi L" <yi.l.liu@intel.com>,
	"Lu, Baolu" <baolu.lu@intel.com>,
	"Williams, Dan J" <dan.j.williams@intel.com>,
	"Luck, Tony" <tony.luck@intel.com>,
	"Kumar, Sanjay K" <sanjay.k.kumar@intel.com>,
	LKML <linux-kernel@vger.kernel.org>, KVM <kvm@vger.kernel.org>,
	Kirti Wankhede <kwankhede@nvidia.com>,
	"Peter Zijlstra" <peterz@infradead.org>,
	Marc Zyngier <maz@kernel.org>,
	"Bjorn Helgaas" <helgaas@kernel.org>
Subject: RE: Virtualizing MSI-X on IMS via VFIO
Date: Thu, 24 Jun 2021 00:00:37 +0000	[thread overview]
Message-ID: <MWHPR11MB18864420ACE88E060203F7818C079@MWHPR11MB1886.namprd11.prod.outlook.com> (raw)
In-Reply-To: <20210623091935.3ab3e378.alex.williamson@redhat.com>

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Wednesday, June 23, 2021 11:20 PM
>
[...]
 > > So the only downside today of allocating more MSI-X vectors than
> > necessary is memory consumption for the irq descriptors.
> 
> As above, this is a QEMU policy of essentially trying to be a good
> citizen and allocate only what we can infer the guest is using.  What's
> a good way for QEMU, or any userspace, to know it's running on a host
> where vector exhaustion is not an issue?

In my proposal a new command (VFIO_DEVICE_ALLOC_IRQS) is
introduced to separate allocation from enabling. The availability
of this command could be the indicator whether vector 
exhaustion is not an issue now?

> >
> > So no, we are not going to proliferate this complete ignorance of how
> > MSI-X actually works and just cram another "feature" into code which is
> > known to be incorrect.
> 
> Some of the issues of virtualizing MSI-X are unsolvable without
> creating a new paravirtual interface, but obviously we want to work
> with existing drivers and unmodified guests, so that's not an option.
> 
> To work with what we've got, the vfio API describes the limitation of
> the host interfaces via the VFIO_IRQ_INFO_NORESIZE flag.  QEMU then
> makes a choice in an attempt to better reflect what we can infer of the
> guest programming of the device to incrementally enable vectors.  We

It's a surprise to me that Qemu even doesn't look at this flag today after
searching its code...

> could a) work to provide host kernel interfaces that allow us to remove
> that noresize flag and b) decide whether QEMU's usage policy can be
> improved on kernels where vector exhaustion is no longer an issue.

Thomas can help confirm but looks noresize limitation is still there. 
b) makes more sense since Thomas thinks vector exhaustion is not 
an issue now (except one minor open about irte).

Thanks
Kevin