kvm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* (proposal) RE: [PATCH v7 00/16] vfio: expose virtual Shared Virtual Addressing to VMs
@ 2020-10-12  8:38 Tian, Kevin
  2020-10-13  6:22 ` Jason Wang
                   ` (3 more replies)
  0 siblings, 4 replies; 55+ messages in thread
From: Tian, Kevin @ 2020-10-12  8:38 UTC (permalink / raw)
  To: Jason Wang, Liu, Yi L, alex.williamson, eric.auger, baolu.lu, joro
  Cc: jacob.jun.pan, Raj, Ashok, Tian, Jun J, Sun, Yi Y, jean-philippe,
	peterx, Wu, Hao, stefanha, iommu, kvm, Jason Gunthorpe,
	Michael S. Tsirkin

> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, September 14, 2020 12:20 PM
>
[...]
 > If it's possible, I would suggest a generic uAPI instead of a VFIO
> specific one.
> 
> Jason suggest something like /dev/sva. There will be a lot of other
> subsystems that could benefit from this (e.g vDPA).
> 
> Have you ever considered this approach?
> 

Hi, Jason,

We did some study on this approach and below is the output. It's a
long writing but I didn't find a way to further abstract w/o losing 
necessary context. Sorry about that.

Overall the real purpose of this series is to enable IOMMU nested
translation capability with vSVA as one major usage, through
below new uAPIs:
	1) Report/enable IOMMU nested translation capability;
	2) Allocate/free PASID;
	3) Bind/unbind guest page table;
	4) Invalidate IOMMU cache;
	5) Handle IOMMU page request/response (not in this series);
1/3/4) is the minimal set for using IOMMU nested translation, with 
the other two optional. For example, the guest may enable vSVA on 
a device without using PASID. Or, it may bind its gIOVA page table 
which doesn't require page fault support. Finally, all operations can 
be applied to either physical device or subdevice.

Then we evaluated each uAPI whether generalizing it is a good thing 
both in concept and regarding to complexity.

First, unlike other uAPIs which are all backed by iommu_ops, PASID 
allocation/free is through the IOASID sub-system. From this angle
we feel generalizing PASID management does make some sense. 
First, PASID is just a number and not related to any device before 
it's bound to a page table and IOMMU domain. Second, PASID is a 
global resource (at least on Intel VT-d), while having separate VFIO/
VDPA allocation interfaces may easily cause confusion in userspace,
e.g. which interface to be used if both VFIO/VDPA devices exist. 
Moreover, an unified interface allows centralized control over how 
many PASIDs are allowed per process.

One unclear part with this generalization is about the permission.
Do we open this interface to any process or only to those which
have assigned devices? If the latter, what would be the mechanism
to coordinate between this new interface and specific passthrough 
frameworks? A more tricky case, vSVA support on ARM (Eric/Jean
please correct me) plans to do per-device PASID namespace which
is built on a bind_pasid_table iommu callback to allow guest fully 
manage its PASIDs on a given passthrough device. I'm not sure 
how such requirement can be unified w/o involving passthrough
frameworks, or whether ARM could also switch to global PASID 
style...

Second, IOMMU nested translation is a per IOMMU domain
capability. Since IOMMU domains are managed by VFIO/VDPA
 (alloc/free domain, attach/detach device, set/get domain attribute,
etc.), reporting/enabling the nesting capability is an natural 
extension to the domain uAPI of existing passthrough frameworks. 
Actually, VFIO already includes a nesting enable interface even 
before this series. So it doesn't make sense to generalize this uAPI 
out.

Then the tricky part comes with the remaining operations (3/4/5),
which are all backed by iommu_ops thus effective only within an 
IOMMU domain. To generalize them, the first thing is to find a way 
to associate the sva_FD (opened through generic /dev/sva) with an 
IOMMU domain that is created by VFIO/VDPA. The second thing is 
to replicate {domain<->device/subdevice} association in /dev/sva 
path because some operations (e.g. page fault) is triggered/handled 
per device/subdevice. Therefore, /dev/sva must provide both per-
domain and per-device uAPIs similar to what VFIO/VDPA already 
does. Moreover, mapping page fault to subdevice requires pre-
registering subdevice fault data to IOMMU layer when binding 
guest page table, while such fault data can be only retrieved from 
parent driver through VFIO/VDPA. 

However, we failed to find a good way even at the 1st step about
domain association. The iommu domains are not exposed to the
userspace, and there is no 1:1 mapping between domain and device.
In VFIO, all devices within the same VFIO container share the address
space but they may be organized in multiple IOMMU domains based
on their bus type. How (should we let) the userspace know the
domain information and open an sva_FD for each domain is the main
problem here.

In the end we just realized that doing such generalization doesn't
really lead to a clear design and instead requires tight coordination 
between /dev/sva and VFIO/VDPA for almost every new uAPI 
(especially about synchronization when the domain/device 
association is changed or when the device/subdevice is being reset/
drained). Finally it may become a usability burden to the userspace
on proper use of the two interfaces on the assigned device.
 
Based on above analysis we feel that just generalizing PASID mgmt.
might be a good thing to look at while the remaining operations are 
better being VFIO/VDPA specific uAPIs. anyway in concept those are 
just a subset of the page table management capabilities that an 
IOMMU domain affords. Since all other aspects of the IOMMU domain 
is managed by VFIO/VDPA already, continuing this path for new nesting
capability sounds natural. There is another option by generalizing the 
entire IOMMU domain management (sort of the entire vfio_iommu_
type1), but it's unclear whether such intrusive change is worthwhile 
(especially when VFIO/VDPA already goes different route even in legacy
mapping uAPI: map/unmap vs. IOTLB).

Thoughts?

Thanks
Kevin

^ permalink raw reply	[flat|nested] 55+ messages in thread

end of thread, other threads:[~2020-11-04 19:29 UTC | newest]

Thread overview: 55+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-12  8:38 (proposal) RE: [PATCH v7 00/16] vfio: expose virtual Shared Virtual Addressing to VMs Tian, Kevin
2020-10-13  6:22 ` Jason Wang
2020-10-14  3:08   ` Tian, Kevin
2020-10-14 23:10     ` Alex Williamson
2020-10-15  7:02       ` Jason Wang
2020-10-15  6:52     ` Jason Wang
2020-10-15  7:58       ` Tian, Kevin
2020-10-15  8:40         ` Jason Wang
2020-10-15 10:14           ` Liu, Yi L
2020-10-20  6:18             ` Jason Wang
2020-10-20  8:19               ` Liu, Yi L
2020-10-20  9:19                 ` Jason Wang
2020-10-20  9:40                   ` Liu, Yi L
2020-10-20 13:54                     ` Jason Gunthorpe
2020-10-20 14:00                       ` Liu, Yi L
2020-10-20 14:05                         ` Jason Gunthorpe
2020-10-20 14:09                           ` Liu, Yi L
2020-10-13 10:27 ` Jean-Philippe Brucker
2020-10-14  2:11   ` Tian, Kevin
2020-10-14  3:16 ` Tian, Kevin
2020-10-16 15:36   ` Jason Gunthorpe
2020-10-19  8:39     ` Liu, Yi L
2020-10-19 14:25       ` Jason Gunthorpe
2020-10-20 10:21         ` Liu, Yi L
2020-10-20 14:02           ` Jason Gunthorpe
2020-10-20 14:19             ` Liu, Yi L
2020-10-21  2:21               ` Jason Wang
2020-10-20 16:24             ` Raj, Ashok
2020-10-20 17:03               ` Jason Gunthorpe
2020-10-20 19:51                 ` Raj, Ashok
2020-10-20 19:55                   ` Jason Gunthorpe
2020-10-20 20:08                     ` Raj, Ashok
2020-10-20 20:14                       ` Jason Gunthorpe
2020-10-20 20:27                         ` Raj, Ashok
2020-10-21 11:48                           ` Jason Gunthorpe
2020-10-21 17:51                             ` Raj, Ashok
2020-10-21 18:24                               ` Jason Gunthorpe
2020-10-21 20:03                                 ` Raj, Ashok
2020-10-21 23:32                                   ` Jason Gunthorpe
2020-10-21 23:53                                     ` Raj, Ashok
2020-10-22  2:55                               ` Jason Wang
2020-10-22  3:54                                 ` Liu, Yi L
2020-10-22  4:38                                   ` Jason Wang
2020-11-03  9:52 ` joro
2020-11-03 12:56   ` Jason Gunthorpe
2020-11-03 13:18     ` joro
2020-11-03 13:23       ` Jason Gunthorpe
2020-11-03 14:03         ` joro
2020-11-03 14:06           ` Jason Gunthorpe
2020-11-03 14:35             ` joro
2020-11-03 15:22               ` Jason Gunthorpe
2020-11-03 16:55                 ` joro
2020-11-03 17:48                   ` Jason Gunthorpe
2020-11-03 19:14                     ` joro
2020-11-04 19:29                       ` Jason Gunthorpe

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).