From: Jason Gunthorpe <email@example.com> To: "Tian, Kevin" <firstname.lastname@example.org> Cc: "Raj, Ashok" <email@example.com>, "Jiang, Dave" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "email@example.com" <firstname.lastname@example.org>, "Pan, Jacob jun" <email@example.com>, "Liu, Yi L" <firstname.lastname@example.org>, "Lu, Baolu" <email@example.com>, "Kumar, Sanjay K" <firstname.lastname@example.org>, "Luck, Tony" <email@example.com>, "Lin, Jing" <firstname.lastname@example.org>, "Williams, Dan J" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com>, "firstname.lastname@example.org" <email@example.com> Subject: Re: [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver. Date: Fri, 24 Apr 2020 15:12:03 -0300 [thread overview] Message-ID: <20200424181203.GU13640@mellanox.com> (raw) In-Reply-To: <AADFC41AFE54684AB9EE6CBC0274A5D19D8A808B@SHSMSX104.ccr.corp.intel.com> On Fri, Apr 24, 2020 at 04:25:56PM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Friday, April 24, 2020 8:45 PM > > > > On Fri, Apr 24, 2020 at 03:27:41AM +0000, Tian, Kevin wrote: > > > > > > > That by itself doesn't translate to what a guest typically does > > > > > with a VDEV. There are other control paths that need to be serviced > > > > > from the kernel code via VFIO. For speed path operations like > > > > > ringing doorbells and such they are directly managed from guest. > > > > > > > > You don't need vfio to mmap BAR pages to userspace. The unique thing > > > > that vfio gives is it provides a way to program the classic non-PASID > > > > iommu, which you are not using here. > > > > > > That unique thing is indeed used here. Please note sharing CPU virtual > > > address space with device (what SVA API is invented for) is not the > > > purpose of this series. We still rely on classic non-PASID iommu > > programming, > > > i.e. mapping/unmapping IOVA->HPA per iommu_domain. Although > > > we do use PASID to tag ADI, the PASID is contained within iommu_domain > > > and invisible to VFIO. From userspace p.o.v, this is a device passthrough > > > usage instead of PASID-based address space binding. > > > > So you have PASID support but don't use it? Why? PASID is much better > > than classic VFIO iommu, it doesn't require page pinning... > > PASID and I/O page fault (through ATS/PRI) are orthogonal things. Don't > draw the equation between them. The host driver can tag PASID to > ADI so every DMA request out of that ADI has a PASID prefix, allowing VT-d > to do PASID-granular DMA isolation. However I/O page fault cannot be > taken for granted. A scalable IOV device may support PASID while without > ATS/PRI. Even when ATS/PRI is supported, the tolerance of I/O page fault > is decided by the work queue mode that is configured by the guest. For > example, if the guest put the work queue in non-faultable transaction > mode, the device doesn't do PRI and simply report error if no valid IOMMU > mapping. Okay, that makes sense, I wasn't aware people were doing PASID without ATS at this point.. > > > idxd is just the first device that supports Scalable IOV. We have a > > > lot more coming later, in different types. Then putting such > > > emulation in user space means that Qemu needs to support all those > > > vendor specific interfaces for every new device which supports > > > > It would be very sad to see an endless amount of device emulation code > > crammed into the kernel. Userspace is where device emulation is > > supposed to live. For security > > I think providing an unified abstraction to userspace is also important, > which is what VFIO provides today. The merit of using one set of VFIO > API to manage all kinds of mediated devices and VF devices is a major > gain. Instead, inventing a new vDPA-like interface for every Scalable-IOV > or equivalent device is just overkill and doesn't scale. Also the actual > emulation code in idxd driver is actually small, if putting aside the PCI > config space part for which I already explained most logic could be shared > between mdev device drivers. If it was just config space you might have an argument, VFIO already does some config space mangling, but emulating BAR space is out of scope of VFIO, IMHO. I also think it is disingenuous to pretend this is similar to SR-IOV. SR-IOV is self contained and the BAR does not require emulation. What you have here sounds like it is just an ordinary multi-queue device with the ability to PASID tag queues for IOMMU handling. This is absolutely not SRIOV - it is much closer to VDPA, which isn't using mdev. Further, I disagree with your assessment that this doesn't scale. You already said you plan a normal user interface for idxd, so instead of having a single sane user interface (ala VDPA) idxd now needs *two*. If this is the general pattern of things to come, it is a bad path. The only thing we get out of this is someone doesn't have to write a idxd emulation driver in qemu, instead they have to write it in the kernel. I don't see how that is a win for the ecosystem. Jason
next prev parent reply other threads:[~2020-04-24 18:12 UTC|newest] Thread overview: 89+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-04-21 23:33 Dave Jiang 2020-04-21 23:33 ` [PATCH RFC 01/15] drivers/base: Introduce platform_msi_ops Dave Jiang 2020-04-26 7:01 ` Greg KH 2020-04-27 21:38 ` Dave Jiang 2020-04-28 7:34 ` Greg KH 2020-04-21 23:33 ` [PATCH RFC 02/15] drivers/base: Introduce a new platform-msi list Dave Jiang 2020-04-25 21:13 ` Thomas Gleixner 2020-05-04 0:08 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 03/15] drivers/base: Allocate/free platform-msi interrupts by group Dave Jiang 2020-04-25 21:23 ` Thomas Gleixner 2020-05-04 0:08 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 04/15] drivers/base: Add support for a new IMS irq domain Dave Jiang 2020-04-23 20:11 ` Jason Gunthorpe 2020-05-01 22:30 ` Dey, Megha 2020-05-03 22:25 ` Jason Gunthorpe 2020-05-03 22:40 ` Dey, Megha 2020-05-03 22:46 ` Jason Gunthorpe 2020-05-04 0:25 ` Dey, Megha 2020-05-04 12:14 ` Jason Gunthorpe 2020-05-06 10:27 ` Tian, Kevin 2020-04-25 21:38 ` Thomas Gleixner 2020-05-04 0:11 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 05/15] ims-msi: Add mask/unmask routines Dave Jiang 2020-04-25 21:49 ` Thomas Gleixner 2020-05-04 0:16 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 06/15] ims-msi: Enable IMS interrupts Dave Jiang 2020-04-25 22:13 ` Thomas Gleixner 2020-05-04 0:17 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 07/15] Documentation: Interrupt Message store Dave Jiang 2020-04-23 20:04 ` Jason Gunthorpe 2020-05-01 22:32 ` Dey, Megha 2020-05-03 22:28 ` Jason Gunthorpe 2020-05-03 22:41 ` Dey, Megha 2020-04-21 23:34 ` [PATCH RFC 08/15] vfio/mdev: Add a member for iommu domain in mdev_device Dave Jiang 2020-04-21 23:34 ` [PATCH RFC 09/15] vfio/type1: Save domain when attach domain to mdev Dave Jiang 2020-04-21 23:34 ` [PATCH RFC 10/15] dmaengine: idxd: add config support for readonly devices Dave Jiang 2020-04-21 23:34 ` [PATCH RFC 11/15] dmaengine: idxd: add IMS support in base driver Dave Jiang 2020-04-21 23:35 ` [PATCH RFC 12/15] dmaengine: idxd: add device support functions in prep for mdev Dave Jiang 2020-04-21 23:35 ` [PATCH RFC 13/15] dmaengine: idxd: add support for VFIO mediated device Dave Jiang 2020-04-21 23:35 ` [PATCH RFC 14/15] dmaengine: idxd: add error notification from host driver to " Dave Jiang 2020-04-21 23:35 ` [PATCH RFC 15/15] dmaengine: idxd: add ABI documentation for mediated device support Dave Jiang 2020-04-21 23:54 ` [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver Jason Gunthorpe 2020-04-22 0:53 ` Tian, Kevin 2020-04-22 11:50 ` Jason Gunthorpe 2020-04-22 21:14 ` Raj, Ashok 2020-04-23 19:12 ` Jason Gunthorpe 2020-04-24 3:27 ` Tian, Kevin 2020-04-24 12:44 ` Jason Gunthorpe 2020-04-24 16:25 ` Tian, Kevin 2020-04-24 18:12 ` Jason Gunthorpe [this message] 2020-04-26 5:18 ` Tian, Kevin 2020-04-26 19:13 ` Jason Gunthorpe 2020-04-27 3:43 ` Alex Williamson 2020-04-27 11:58 ` Jason Gunthorpe 2020-04-27 13:19 ` Alex Williamson 2020-04-27 13:22 ` Jason Gunthorpe 2020-04-27 14:18 ` Alex Williamson 2020-04-27 14:25 ` Jason Gunthorpe 2020-04-27 15:41 ` Alex Williamson 2020-04-27 16:16 ` Jason Gunthorpe 2020-04-27 16:25 ` Dave Jiang 2020-04-27 21:56 ` Jason Gunthorpe 2020-04-29 9:42 ` Tian, Kevin 2020-05-08 20:47 ` Raj, Ashok 2020-05-08 23:16 ` Jason Gunthorpe 2020-05-08 23:52 ` Dave Jiang 2020-05-09 0:09 ` Raj, Ashok 2020-05-09 12:21 ` Jason Gunthorpe 2020-05-13 2:29 ` Jason Wang 2020-05-13 8:30 ` Tian, Kevin 2020-05-13 12:40 ` Jason Gunthorpe 2020-04-27 12:13 ` Tian, Kevin 2020-04-27 12:55 ` Jason Gunthorpe 2020-04-22 21:24 ` Dan Williams 2020-04-23 19:17 ` Dan Williams 2020-04-23 19:49 ` Jason Gunthorpe 2020-05-01 22:31 ` Dey, Megha 2020-05-03 22:21 ` Jason Gunthorpe 2020-05-03 22:32 ` Dey, Megha 2020-04-23 19:18 ` Jason Gunthorpe 2020-05-01 22:31 ` Dey, Megha 2020-05-03 22:22 ` Jason Gunthorpe 2020-05-03 22:31 ` Dey, Megha 2020-05-03 22:36 ` Jason Gunthorpe 2020-05-04 0:20 ` Dey, Megha 2020-04-22 23:04 ` Dey, Megha 2020-04-23 19:44 ` Jason Gunthorpe 2020-05-01 22:32 ` Dey, Megha 2020-04-24 6:31 ` Jason Wang
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200424181203.GU13640@mellanox.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Re: [PATCH RFC 00/15] Add VFIO mediated device support and IMS support for the idxd driver.' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).