RE: [PATCH 2/5] Introduce VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER

[Parav Pandit <parav@nvidia.com>]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, January 18, 2022 8:39 PM
> 
> On Tue, Jan 18, 2022 at 10:50:52AM +0000, Parav Pandit wrote:
> >
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, January 18, 2022 4:09 PM
> > >
> > > On Tue, Jan 18, 2022 at 07:30:34AM +0000, Parav Pandit wrote:
> > > > An unrelated command to AQ in Jason's proposal [1] is about " The
> > > management driver MUST create a managed device by allocating".
> > > > We see that creator of the subfunction is often not the only
> > > > entity managing
> > > it.
> > > > They being same in new era finding less and less users.
> > > > So this piece needs more discussion whenever we address that.
> > > >
> > > > [1]
> > > > https://lists.oasis-open.org/archives/virtio-comment/202108/msg001
> > > > 36.h
> > > > tml
> > >
> > > This reminds me. How do AQ commands interact with VF lifecycle?
> > VF device usage is controlled by the same system which is configuring the VF
> via its parent PF device.
> > So VF device shouldn't be in use. Any configuration change while VF device is
> in use will result in failing the AQ command.
> >
> > > E.g. can one change number of vectors for an active VF?
> > > Need to specify this.
> > >
> > > Also, I started worrying about compatibility here.
> > > Let's say the msix capability in a VF specifies 16 vectors.
> > > Can PF specify 32? If yes how will driver program them?
> > Yes, PF can change to 32. When VF driver queries the PCI capability, it will
> reflect 32 instead of 16.
> > > Can PF specify 8? If yes how do we make sure driver does not attempt
> > > to use 16? And what happens if it does?
> > PF is programming the VF msix capability in the device. So virtio pci driver
> operating the PCI VF device cannot access vectors beyond the max value
> programmed by the PF driver.
> 
> Um. Interesting. This means that the msix capability of the VF changes?
Yes.
> Is that in fact spec compliant? Could some OSes cache the value of the
> capability even if the device is not in active use? E.g. I can see how this might
> happen in order to map the MSIX tables even before loading the driver.
> 
PCI subsystem can catch the value before the device driver can load.
Generally a device support intx/msix or intx/msi. So PCI subsystem is not aware what will be used by its upper layer device drivers.
So it usually differs such initialization to a later stage until it is actually used.

Whichever OS driver which implements msix configuration, will have to either not cache it or flush+ rebuild the cache.

> The spec says:
> 	Depending upon system software policy, system software, device driver
> software, or each at
> 	different times or environments may configure a function’s MSI-X
> capability and table
> 	structures with suitable vectors.
> 
> So MSIX canfiguation might not be up to the driver.
> 
> We actually ask driver to read back any vector assigned to a VQ so it's possible
> to fail vector assignment. Maybe that's better.
> 
Virtio driver should not incur any additional complexity in re-reading vector etc.
All the msix config should happen much before drivers gets loaded for the VF.
It is PCI layer of the HV to provide a stable device to virtio device driver which is not undergoing msix table changes, when virtio device driver is operating on it.