[virtio-dev] Re: [virtio-comment] Re: [PATCH 09/11] transport-pci: Describe PCI MMR dev config registers

[Jason Wang <jasowang@redhat.com>]

On Tue, Apr 18, 2023 at 1:23 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Sunday, April 16, 2023 11:23 PM
> >
> > On Fri, Apr 14, 2023 at 11:51 AM Parav Pandit <parav@nvidia.com> wrote:
> > >
> > >
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Thursday, April 13, 2023 11:38 PM
> > >
> > > > > > 1) spec doesn't enforce the size of a specific structure
> > > > > Spec will be extended in coming time.
> > > >
> > > > It's too late to do any new restriction without introducing a flag.
> > > We are really diverging from the topic.
> > > I don’t think it is late. The work in this area of PCI VF has not even begun fully.
> >
> > I meant it needs a new feature flag.
> >
> Ok.
> > >
> > > > Mandating size may easily end up with a architecte specific solution.
> > > >
> > > Unlikely. Other standard device types are also expanding this way.
> >
> > I think we are talking about software technologies instead of device design
> > here.
> >
> Isn't the size of BAR and its cap_len exposed by the device?

Somehow, it's more about how the hypervisor is going to use this,
memory mapped or trapping. For either case, the hypervisor needs to
have virtio knowledge in order to finish this.

>
> > For devices it works.But for hypervisor, it needs to deal with the size that
> > doesn't match arch's page size.
> >
> PCI BAR size of the VF can learn the system page size being different for x86 (4K) and arm (64K).
> PCI transport seems to support it.

Yes this is for SR-IOV but not for other cases. We could invent new
facilities for sure but the hypervisor can not have this assumption.

>
> PCI PF on bare-metal has to understand the highest page size anyway if for some reason bare-metal host wants to map this PF to the VM.
>
> A hypevisor mediating and emulating needs to learn the system page size anyway.
> If underlying device page size is smaller, hypevisor may end up mediating it.

Exactly. So what I want to say is, for whatever case, a hypervisor
needs to have virtio knowledge in order to achieve these.

>
> > I meant you can't have recommendations in features and config.
> Sure. There is none.
>
> > What's more,
> > assuming you have two generations of device
> >
> > gen1: features x,y
> > gen2: features x,y,z
> >
> > You won't be able to do migration between gen1 and gen2 without mediation.
> Gen1 can easily migrate to gen2, because gen1 has smaller subset than gen2.
> When gen2 device is composed, feature z is disabled.

Sure, but this requires a lot of features that do not exist in the
spec. E.g it assumes the device could be composed on demand which
seems to fit the idea of transport virtqueue. So it adds dependencies
for migration where a simple mediation could be used to solve this
without bothering the spec.

>
> Gen2 to gen1 migration can do software-based migration anyway or through mediation.
> But because gen2 may need to migrate to gen1, hence gen2 to gen2 migration also should be done through mediation, doesn’t make sense to me.

It really depends on the design:

1) if you want to expose any features that is done by admin virtqueue
to a guest, mediation is a must (e.g if you want do live migration for
L1)
2) mediation is a must for the idea of transport virtqueue

>
> > Such technologies have been used by cpu features for years.
> > I am not sure why it became a problem for you.
> >
> > > Apart from it some of the PCI device layout compat checks will be covered
> > too.
> > >
> > > > And what you proposed is to allow the management to know the exact
> > > > hardware layout in order to check the compatibility? And the
> > > > management needs to evolve as new structures are added.
> > > Mostly not mgmt stack may not need to evolve a lot.
> > > Because most layouts should be growing within the device context and not at
> > the PCI capabilities etc area.
> > >
> > > And even if it does, its fine as large part of it standard PCI spec definitions.
> >
> > So as mentioned in another thread, this is a PCI specific solution:
> >
> > 1) feature and config are basic virtio facility
> > 2) capability is not but specific to PCI transport
> >
> So any LM solution will have transport specific checks and virtio level checks.

So here's the model that is used by Qemu currently:

1) Device is emulated, it's the charge of the libvirt to launch Qemu
and present a stable ABI for guests.
2) Datapath doesn't need to care about the hardware details since the
hardware layout is invisible from guest

You can see, it's more than sufficient for libvirt to check
features/config space, it doesn't need to care about the hardware BAR
layout. Migration is much easier in this way. And we can use transport
other than PCI in the guest in this case for live migration.

>
> > Checking PCI capability layout in the virtio management is a layer violation
> > which can't work for future transport like SIOV or adminq.
> Virtio management that will have transport level checks is not a violation.
> SIOV will define its own transport anyway. Not to mix with ccw/mmio or pci.
>
> > Management should only see virtio device otherwise the solution becomes
> > transport specific.
> >
> Solution needs to cover transport as well as transport is integral part of the virtio spec.
> Each transport layer will implement feature/config/cap in its own way.

If we can avoid those hardware details to be checked, we should not go
for that. It's a great ease of the management layer.

Thanks

>
> > >
> > > > This complicates the work of
> > > > management's furtherly which I'm not sure it can work.
> > > >
> > > Well, once we work towards it, it can work. :)
> > >
> > > > >
> > > > > > Hypervisor needs to start from a mediation method and do BAR
> > > > > > assignment only when possible.
> > > > > >
> > > > > Not necessarily.
> > > > >
> > > > > > > Cons:
> > > > > > > a. More AQ commands work in sw
> > > > > >
> > > > > > Note that this needs to be done on top of the transport virtqueue.
> > > > > > And we need to carefully design the command sets since they
> > > > > > could be mutually exclusive.
> > > > > >
> > > > > Not sure what more to expect out of transport virtqueue compared to AQ.
> > > > > I didn’t follow, which part could be mutually exclusive?
> > > >
> > > > Transport VQ allows a modern device to be transported via adminq.
> > > >
> > > May be for devices it can work. Hypervisor mediation with CC on horizon for
> > new capabilities is being reduced.
> > > So we don’t see transport vq may not be path forward.
> > >
> > > > And you want to add commands to transport for legacy devices.
> > > >
> > > Yes only legacy emulation who do not care about hypervisor mediation.
> > >
> > > > Can a driver use both the modern transport commands as well as the
> > > > legacy transport commands?
> > > Hard to answer, I likely do not understand as driver namespace is unclear.
> >
> > Things might be simplified if we use separate queues for admin, transport and
> > legacy.
> >
> Do you mean say we have three AQs, AQ_1, AQ_2, and AQ_3;
> AQ_1 of the PF used by admin work as SIOV device create, SRIOV MSIX configuration.
> AQ_2 of the PF used for transporting legacy config access of the PCI VF
> AQ_3 of the PF for some transport work.
>
> If yes, sounds find to me.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org