linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Jason Wang <jasowang@redhat.com>,
	Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Alex Williamson <alex.williamson@redhat.com>,
	"Raj, Ashok" <ashok.raj@intel.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Jean-Philippe Brucker <jean-philippe@linaro.com>,
	LKML <linux-kernel@vger.kernel.org>,
	"Jiang, Dave" <dave.jiang@intel.com>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>, Li Zefan <lizefan@huawei.com>,
	Johannes Weiner <hannes@cmpxchg.org>, Tejun Heo <tj@kernel.org>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
	"Wu, Hao" <hao.wu@intel.com>,
	David Woodhouse <dwmw2@infradead.org>
Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Date: Wed, 7 Apr 2021 08:58:26 -0300	[thread overview]
Message-ID: <20210407115826.GE7405@nvidia.com> (raw)
In-Reply-To: <MWHPR11MB1886A7E4C6F3E3A81240517B8C759@MWHPR11MB1886.namprd11.prod.outlook.com>

On Wed, Apr 07, 2021 at 08:17:50AM +0000, Tian, Kevin wrote:
> > From: Jason Gunthorpe
> > Sent: Tuesday, April 6, 2021 8:43 PM
> > 
> > On Tue, Apr 06, 2021 at 09:35:17AM +0800, Jason Wang wrote:
> > 
> > > > VFIO and VDPA has no buisness having map/unmap interfaces once we
> > have
> > > > /dev/ioasid. That all belongs in the iosaid side.
> > > >
> > > > I know they have those interfaces today, but that doesn't mean we have
> > > > to keep using them for PASID use cases, they should be replaced with a
> > > > 'do dma from this pasid on /dev/ioasid' interface certainly not a
> > > > 'here is a pasid from /dev/ioasid, go ahead and configure it youself'
> > > > interface
> > >
> > > So it looks like the PASID was bound to SVA in this design. I think it's not
> > > necessairly the case:
> > 
> > No, I wish people would stop talking about SVA.
> > 
> > SVA and vSVA are a very special narrow configuration of a PASID. There
> > are lots of other PASID configurations! That is the whole point, a
> > PASID is complicated, there are many configuration scenarios, they
> > need to be in one place with a very clearly defined uAPI
> > 
> 
> I feel it also makes sense to allow a subsystem to specify which configurations
> are permitted when allowing a PASID on its device

huh? why?

> e.g. excluding things like
> GPA mappings that existing subsystems (VFIO/VDPA) already handle well:

They don't "handle well", they have some historical baggage that is no
longer suitable for the complexity this area has in the modern world.

Forget about the existing APIs and replace them in /dev/ioasid.

> - Share GPA mappings between multiple devices (w/ or w/o PASID) for 
> better IOTLB efficiency;
>
> - Share GPA mappings between transactions w/ PASID and transactions w/o
> PASID from the same device (e.g. GPU) for better IOTLB efficiency;
> 
> - Use the same page table for GPA mappings before and after the guest 
> turns on/off the PASID capability;

All of these are cases you need to design the /dev/ioasid to handle.

It is pretty clear to me that you'll need non-PASID IOASID's as
well.

Ideally a generic IOASID would just be a page table and it doesn't
crystalize into a RID or RID,PASID routing until devices are attached
to it.

Since IOASID can be nested the only thing that makes any sense is for
each level of the nest to be visible under /dev/ioasid. 

What a complete mess it would be if vfio-pci owns the GPA table,
/dev/ioasid has a nested PASID, and vfio-mdev is running a mdev on top
of that PASID.

> All above are given as long as we continue to let VFIO/VDPA manage the
> iommu domain and associated GPA mappings for PASID.

So don't do that. Don't I keep saying this weird split is making a
horrible mess?

You can't reasonably build the complex PASID scenarios you talk about
above unless the entire translation path is owned by one entity:
/dev/ioasid.

You need to focus on figuring out what that looks like then figure out
how to move VDPA and VFIO to consume /dev/ioasid for all of their
translation instead of open-coding half-baked internal versions.

Jason

  reply	other threads:[~2021-04-07 11:58 UTC|newest]

Thread overview: 269+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-27 22:01 [PATCH V4 00/18] IOASID extensions for guest SVA Jacob Pan
2021-02-27 22:01 ` [PATCH V4 01/18] docs: Document IO Address Space ID (IOASID) APIs Jacob Pan
2021-02-27 22:01 ` [PATCH V4 02/18] iommu/ioasid: Rename ioasid_set_data() Jacob Pan
2021-02-27 22:01 ` [PATCH V4 03/18] iommu/ioasid: Add a separate function for detach data Jacob Pan
2021-02-27 22:01 ` [PATCH V4 04/18] iommu/ioasid: Support setting system-wide capacity Jacob Pan
2021-02-27 22:01 ` [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs Jacob Pan
2021-03-19  0:22   ` Jacob Pan
2021-03-19  9:58     ` Jean-Philippe Brucker
2021-03-19 12:46       ` Jason Gunthorpe
2021-03-19 13:41         ` Jean-Philippe Brucker
2021-03-19 13:54           ` Jason Gunthorpe
2021-03-19 18:22             ` Jacob Pan
2021-03-22  9:24               ` Jean-Philippe Brucker
2021-03-24 17:02                 ` Jacob Pan
2021-03-24 17:03                   ` Jason Gunthorpe
2021-03-24 22:12                     ` Jacob Pan
2021-03-25 10:21                       ` Jean-Philippe Brucker
2021-03-25 17:02                         ` Jacob Pan
2021-03-25 17:16                           ` Jason Gunthorpe
2021-03-25 18:23                             ` Jacob Pan
2021-03-26  8:06                             ` Jean-Philippe Brucker
2021-03-30 13:07                               ` Jason Gunthorpe
2021-03-30 13:42                                 ` Jean-Philippe Brucker
2021-03-30 13:46                                   ` Jason Gunthorpe
2021-03-25 10:26                   ` Jean-Philippe Brucker
2021-03-22 12:03               ` Jason Gunthorpe
2021-03-24 19:05                 ` Jacob Pan
2021-03-29 16:31                   ` Jason Gunthorpe
2021-03-29 22:55                     ` Jacob Pan
2021-03-30 13:43                       ` Jason Gunthorpe
2021-03-31  0:10                         ` Jacob Pan
2021-03-31 12:28                           ` Jason Gunthorpe
2021-03-31 16:34                             ` Jacob Pan
2021-03-31 17:31                               ` Jason Gunthorpe
2021-03-31 18:20                                 ` Jacob Pan
2021-03-31 18:33                                   ` Jason Gunthorpe
2021-03-31 21:50                                     ` Jacob Pan
2021-03-31  8:38                         ` Liu, Yi L
2021-03-30  1:37                     ` Tian, Kevin
2021-03-30 13:28                       ` Jason Gunthorpe
2021-03-31  7:38                         ` Liu, Yi L
2021-03-31 12:40                           ` Jason Gunthorpe
2021-04-01  4:38                             ` Liu, Yi L
2021-04-01  7:04                               ` Liu, Yi L
2021-04-01 11:54                                 ` Jason Gunthorpe
2021-04-02 12:46                                   ` Liu, Yi L
2021-04-01 12:05                                 ` Jean-Philippe Brucker
2021-04-01 12:12                                   ` Jason Gunthorpe
2021-04-01 13:38                                   ` Liu, Yi L
2021-04-01 13:42                                     ` Jason Gunthorpe
2021-04-01 14:08                                       ` Liu, Yi L
2021-04-01 16:03                                         ` Jason Gunthorpe
2021-04-02  7:30                                           ` Tian, Kevin
2021-04-05 23:35                                             ` Jason Gunthorpe
2021-04-06  0:37                                               ` Tian, Kevin
2021-04-06 12:15                                                 ` Jason Gunthorpe
2021-04-15 13:11                                           ` Auger Eric
2021-04-15 23:07                                             ` Jason Gunthorpe
2021-04-16 13:12                                               ` Jacob Pan
2021-04-16 15:45                                                 ` Alex Williamson
2021-04-16 17:23                                                   ` Jacob Pan
2021-04-16 17:54                                                     ` Jason Gunthorpe
2021-04-21 13:18                                                   ` Liu, Yi L
2021-04-21 16:23                                                     ` Jason Gunthorpe
2021-04-21 16:54                                                       ` Alex Williamson
2021-04-21 17:52                                                         ` Jason Gunthorpe
2021-04-21 19:33                                                           ` Alex Williamson
2021-04-21 23:03                                                             ` Jason Gunthorpe
2021-04-22  8:34                                                               ` Tian, Kevin
2021-04-22 12:10                                                                 ` Jason Gunthorpe
2021-04-23  9:06                                                                   ` Tian, Kevin
2021-04-23 11:49                                                                     ` Jason Gunthorpe
2021-04-25  9:24                                                                       ` Tian, Kevin
2021-04-26 12:38                                                                         ` Jason Gunthorpe
2021-04-28  6:34                                                                           ` Tian, Kevin
2021-04-28 15:06                                                                             ` Alex Williamson
2021-05-07  7:36                                                                               ` Tian, Kevin
2021-05-07 11:56                                                                                 ` Jason Gunthorpe
2021-05-07 17:06                                                                                 ` Alex Williamson
2021-05-07 17:10                                                                                   ` Jason Gunthorpe
2021-05-08  6:08                                                                                     ` Tian, Kevin
2021-05-08  7:31                                                                                   ` Tian, Kevin
2021-05-10  2:56                                                                                     ` Lu Baolu
2021-04-28 20:46                                                                             ` Jason Gunthorpe
2021-05-04 16:22                                                                               ` Jacob Pan
2021-05-04 16:31                                                                                 ` Jason Gunthorpe
2021-05-08  5:46                                                                               ` Tian, Kevin
2021-05-04 15:41                                                                             ` Jacob Pan
2021-05-04 18:00                                                                               ` Jason Gunthorpe
2021-05-04 22:11                                                                                 ` Jacob Pan
2021-05-04 23:15                                                                                   ` Jason Gunthorpe
2021-05-05 17:22                                                                                     ` Jacob Pan
2021-05-05 18:00                                                                                       ` Jason Gunthorpe
2021-05-05 20:04                                                                                         ` Jacob Pan
2021-05-05 22:21                                                                                           ` Jason Gunthorpe
2021-05-05 23:23                                                                                             ` Raj, Ashok
2021-05-06 12:22                                                                                               ` Jason Gunthorpe
2021-05-08  7:06                                                                                             ` Liu Yi L
2021-05-06  7:23                                                                                           ` Jean-Philippe Brucker
2021-05-06 12:27                                                                                             ` Jason Gunthorpe
2021-05-06 16:32                                                                                               ` Raj, Ashok
2021-05-07 17:20                                                                                                 ` Jason Gunthorpe
2021-05-07 18:14                                                                                                   ` Raj, Ashok
2021-05-07 18:20                                                                                                     ` Jason Gunthorpe
2021-05-07 19:23                                                                                                       ` Raj, Ashok
2021-05-07 19:28                                                                                                         ` Jason Gunthorpe
2021-05-07 22:15                                                                                                           ` Jacob Pan
2021-05-08  9:56                                                                                                 ` Tian, Kevin
2021-05-10 12:37                                                                                                   ` Jason Gunthorpe
2021-05-10 15:25                                                                                                     ` Raj, Ashok
2021-05-10 15:31                                                                                                       ` Jason Gunthorpe
2021-05-10 16:22                                                                                                         ` Raj, Ashok
2021-05-10 16:39                                                                                                           ` Jason Gunthorpe
2021-05-10 22:28                                                                                                             ` Jacob Pan
2021-05-10 23:45                                                                                                               ` Jason Gunthorpe
2021-05-11  3:56                                                                                                                 ` Jacob Pan
2021-05-11  9:10                                                                                                     ` Tian, Kevin
2021-05-11 13:24                                                                                                       ` Liu Yi L
2021-05-11 22:52                                                                                                         ` Tian, Kevin
2021-05-11 14:38                                                                                                       ` Jason Gunthorpe
2021-05-11 22:51                                                                                                         ` Tian, Kevin
2021-05-11 23:39                                                                                                           ` Jason Gunthorpe
2021-05-12  0:21                                                                                                             ` Tian, Kevin
2021-05-12  0:25                                                                                                               ` Jason Gunthorpe
2021-05-12  0:40                                                                                                                 ` Tian, Kevin
2021-04-29  8:54                                                                       ` Auger Eric
2021-04-29  8:55                                                                   ` Auger Eric
2021-04-29 13:26                                                                   ` Auger Eric
2021-04-29 20:04                                                                     ` Jason Gunthorpe
2021-05-05  9:10                                                                       ` Auger Eric
2021-04-22 17:13                                                               ` Alex Williamson
2021-04-22 17:57                                                                 ` Jason Gunthorpe
2021-04-22 19:37                                                                   ` Alex Williamson
2021-04-22 20:00                                                                     ` Jason Gunthorpe
2021-04-22 22:38                                                                       ` Alex Williamson
2021-04-22 23:39                                                                         ` Jason Gunthorpe
2021-04-23 10:31                                                                           ` Tian, Kevin
2021-04-23 11:57                                                                             ` Jason Gunthorpe
2021-04-27  5:11                                                                             ` David Gibson
2021-04-27 16:39                                                                               ` Jason Gunthorpe
2021-04-28  0:49                                                                                 ` David Gibson
2021-04-23 16:38                                                                           ` Alex Williamson
2021-04-23 22:28                                                                             ` Jason Gunthorpe
2021-04-27  5:15                                                                               ` David Gibson
2021-04-27  5:08                                                                           ` David Gibson
2021-04-27 17:12                                                                             ` Jason Gunthorpe
2021-04-28  0:58                                                                               ` David Gibson
2021-04-28 14:56                                                                                 ` Jason Gunthorpe
2021-04-29  3:04                                                                                   ` David Gibson
2021-05-03 16:15                                                                                     ` Jason Gunthorpe
2021-05-13  5:48                                                                                       ` David Gibson
2021-05-13 13:59                                                                                         ` Jason Gunthorpe
2021-05-24  7:52                                                                                           ` David Gibson
2021-05-24 23:37                                                                                             ` Jason Gunthorpe
2021-05-25 19:26                                                                                               ` Kirti Wankhede
2021-05-25 19:52                                                                                                 ` Jason Gunthorpe
2021-05-25 21:18                                                                                                   ` Kirti Wankhede
2021-05-27  5:00                                                                                                     ` David Gibson
2021-05-27 18:25                                                                                                       ` Kirti Wankhede
2021-06-01  3:45                                                                                                         ` David Gibson
2021-05-27  4:58                                                                                                   ` David Gibson
2021-05-27 18:48                                                                                                     ` Jason Gunthorpe
2021-06-01  4:03                                                                                                       ` David Gibson
2021-06-01 12:57                                                                                                         ` Jason Gunthorpe
2021-06-08  0:44                                                                                                           ` David Gibson
2021-06-08 18:34                                                                                                             ` Jason Gunthorpe
2021-05-25 22:52                                                                                                 ` Alex Williamson
2021-05-26 18:10                                                                                                   ` Kirti Wankhede
2021-05-26 18:59                                                                                                     ` Alex Williamson
2021-05-26 19:13                                                                                                       ` Jason Gunthorpe
2021-05-27  4:53                                                                                               ` David Gibson
2021-05-27 19:06                                                                                                 ` Jason Gunthorpe
2021-06-01  4:27                                                                                                   ` David Gibson
2021-04-28  6:58                                                                               ` Tian, Kevin
2021-05-04 17:12                                                                                 ` Jason Gunthorpe
2021-05-07  8:09                                                                                   ` Tian, Kevin
2021-04-28  7:47                                                                               ` Tian, Kevin
2021-04-28 18:41                                                                                 ` Jason Gunthorpe
2021-04-27  4:50                                                                 ` David Gibson
2021-04-27 17:24                                                                   ` Jason Gunthorpe
2021-04-28  1:23                                                                     ` David Gibson
2021-04-29  0:21                                                                       ` Jason Gunthorpe
2021-04-29  3:20                                                                         ` David Gibson
2021-05-03 16:05                                                                           ` Jason Gunthorpe
2021-05-04  3:54                                                                             ` David Gibson
2021-05-04 18:15                                                                               ` Jason Gunthorpe
2021-05-05  4:28                                                                                 ` Alexey Kardashevskiy
2021-05-05 16:39                                                                                   ` Jason Gunthorpe
2021-05-13  6:07                                                                                     ` David Gibson
2021-05-13 13:50                                                                                       ` Jason Gunthorpe
2021-05-24  7:56                                                                                         ` David Gibson
2021-05-13  6:01                                                                                 ` David Gibson
2021-05-13  6:52                                                                                   ` Tian, Kevin
2021-05-13 13:47                                                                                   ` Jason Gunthorpe
2021-04-22 12:55                                                             ` Liu Yi L
2021-04-16 13:38                                               ` Auger Eric
2021-04-16 14:05                                                 ` Jason Gunthorpe
2021-04-16 14:26                                                   ` Auger Eric
2021-04-16 14:34                                                     ` Jason Gunthorpe
2021-04-16 15:00                                                       ` Auger Eric
2021-04-01 11:46                               ` Jason Gunthorpe
2021-04-01 13:10                                 ` Liu, Yi L
2021-04-01 13:15                                   ` Jason Gunthorpe
2021-04-01 13:43                                     ` Liu, Yi L
2021-04-01 13:46                                       ` Jason Gunthorpe
2021-04-02  7:58                                         ` Tian, Kevin
2021-04-05 23:39                                           ` Jason Gunthorpe
2021-04-06  1:02                                             ` Tian, Kevin
2021-04-06 12:21                                               ` Jason Gunthorpe
2021-04-07  2:23                                                 ` Tian, Kevin
     [not found]                                             ` <MWHPR11MB188628BDB37A4EE36F3D99338C769@MWHPR11MB1886.namprd11.prod.outlook.com>
2021-04-06  2:08                                               ` Tian, Kevin
2021-04-02 10:01                                         ` Tian, Kevin
2021-04-02  8:22                         ` Tian, Kevin
2021-04-05 23:42                           ` Jason Gunthorpe
2021-04-06  1:27                             ` Tian, Kevin
2021-04-06 12:34                               ` Jason Gunthorpe
2021-04-07  2:08                                 ` Tian, Kevin
2021-04-07 12:20                                   ` Jason Gunthorpe
2021-04-07 23:50                                     ` Tian, Kevin
2021-04-08 11:41                                       ` Jason Gunthorpe
2021-04-06  1:35                             ` Jason Wang
2021-04-06 12:42                               ` Jason Gunthorpe
2021-04-07  2:06                                 ` Jason Wang
2021-04-07  8:17                                 ` Tian, Kevin
2021-04-07 11:58                                   ` Jason Gunthorpe [this message]
2021-04-07 18:43                                   ` Jean-Philippe Brucker
2021-04-07 19:36                                     ` Jason Gunthorpe
2021-04-08  9:37                                       ` Jean-Philippe Brucker
2021-03-30  2:24                     ` Tian, Kevin
2021-03-30 13:24                       ` Jason Gunthorpe
2021-03-30  4:14                     ` Tian, Kevin
2021-03-30 13:27                       ` Jason Gunthorpe
2021-03-31  7:41                         ` Liu, Yi L
2021-03-31 12:38                           ` Jason Gunthorpe
2021-03-31 23:46                             ` Jacob Pan
2021-04-01  0:37                               ` Jason Gunthorpe
2021-04-01 17:23                                 ` Jacob Pan
2021-04-01 17:26                                   ` Jason Gunthorpe
2021-03-19 17:14       ` Jacob Pan
2021-02-27 22:01 ` [PATCH V4 06/18] iommu/ioasid: Add free function and states Jacob Pan
2021-02-27 22:01 ` [PATCH V4 07/18] iommu/ioasid: Add ioasid_set iterator helper functions Jacob Pan
2021-02-27 22:01 ` [PATCH V4 08/18] iommu/ioasid: Introduce ioasid_set private ID Jacob Pan
2021-02-27 22:01 ` [PATCH V4 09/18] iommu/ioasid: Introduce notification APIs Jacob Pan
2021-02-27 22:01 ` [PATCH V4 10/18] iommu/ioasid: Support mm token type ioasid_set notifications Jacob Pan
2021-02-27 22:01 ` [PATCH V4 11/18] iommu/ioasid: Add ownership check in guest bind Jacob Pan
2021-02-27 22:01 ` [PATCH V4 12/18] iommu/vt-d: Remove mm reference for guest SVA Jacob Pan
2021-02-27 22:01 ` [PATCH V4 13/18] iommu/ioasid: Add a workqueue for cleanup work Jacob Pan
2021-02-27 22:01 ` [PATCH V4 14/18] iommu/vt-d: Listen to IOASID notifications Jacob Pan
2021-02-27 22:01 ` [RFC PATCH 15/18] cgroup: Introduce ioasids controller Jacob Pan
2021-03-03 15:44   ` Tejun Heo
2021-03-03 21:17     ` Jacob Pan
2021-03-04  0:02       ` Jacob Pan
2021-03-04  0:23         ` Jason Gunthorpe
2021-03-04  9:49         ` Jean-Philippe Brucker
2021-03-04 17:46           ` Jacob Pan
2021-03-04 17:54             ` Jason Gunthorpe
2021-03-04 19:01               ` Jacob Pan
2021-03-04 19:02                 ` Jason Gunthorpe
2021-03-04 21:28                   ` Jacob Pan
2021-03-05  8:30             ` Jean-Philippe Brucker
2021-03-05 17:16               ` Jean-Philippe Brucker
2021-03-05 18:20               ` Jacob Pan
2021-02-27 22:01 ` [RFC PATCH 16/18] iommu/ioasid: Consult IOASIDs cgroup for allocation Jacob Pan
2021-02-27 22:01 ` [RFC PATCH 17/18] docs: cgroup-v1: Add IOASIDs controller Jacob Pan
2021-02-27 22:01 ` [RFC PATCH 18/18] ioasid: Add /dev/ioasid for userspace Jacob Pan
2021-03-10 19:23   ` Jason Gunthorpe
2021-03-11 22:55     ` Jacob Pan
2021-03-12 14:54       ` Jason Gunthorpe
2021-03-02 12:58 ` [PATCH V4 00/18] IOASID extensions for guest SVA Liu, Yi L

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210407115826.GE7405@nvidia.com \
    --to=jgg@nvidia.com \
    --cc=alex.williamson@redhat.com \
    --cc=ashok.raj@intel.com \
    --cc=cgroups@vger.kernel.org \
    --cc=corbet@lwn.net \
    --cc=dave.jiang@intel.com \
    --cc=dwmw2@infradead.org \
    --cc=hannes@cmpxchg.org \
    --cc=hao.wu@intel.com \
    --cc=iommu@lists.linux-foundation.org \
    --cc=jasowang@redhat.com \
    --cc=jean-philippe@linaro.com \
    --cc=jean-philippe@linaro.org \
    --cc=kevin.tian@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lizefan@huawei.com \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).