Re: [PATCH v2 04/17] driver core: platform: Add driver dma ownership management

From: Jason Gunthorpe via iommu <iommu@lists.linux-foundation.org>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Stuart Yoder <stuyoder@gmail.com>,
	rafael@kernel.org, David Airlie <airlied@linux.ie>,
	linux-pci@vger.kernel.org,
	Thierry Reding <thierry.reding@gmail.com>,
	Diana Craciun <diana.craciun@oss.nxp.com>,
	Will Deacon <will@kernel.org>, Ashok Raj <ashok.raj@intel.com>,
	Jonathan Hunter <jonathanh@nvidia.com>,
	Christoph Hellwig <hch@infradead.org>,
	Kevin Tian <kevin.tian@intel.com>,
	Chaitanya Kulkarni <kch@nvidia.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	kvm@vger.kernel.org, Bjorn Helgaas <bhelgaas@google.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Cornelia Huck <cohuck@redhat.com>,
	linux-kernel@vger.kernel.org, Li Yang <leoyang.li@nxp.com>,
	iommu@lists.linux-foundation.org,
	Jacob jun Pan <jacob.jun.pan@intel.com>,
	Daniel Vetter <daniel@ffwll.ch>,
	Robin Murphy <robin.murphy@arm.com>
Subject: Re: [PATCH v2 04/17] driver core: platform: Add driver dma ownership management
Date: Mon, 29 Nov 2021 08:59:58 -0400	[thread overview]
Message-ID: <20211129125958.GW4670@nvidia.com> (raw)
In-Reply-To: <YaSsv5Z1WS7ldgu3@kroah.com>

On Mon, Nov 29, 2021 at 11:34:39AM +0100, Greg Kroah-Hartman wrote:
> On Sun, Nov 28, 2021 at 07:15:09PM -0400, Jason Gunthorpe wrote:
> > On Sun, Nov 28, 2021 at 09:10:14AM +0100, Greg Kroah-Hartman wrote:
> > > On Sun, Nov 28, 2021 at 10:50:38AM +0800, Lu Baolu wrote:
> > > > Multiple platform devices may be placed in the same IOMMU group because
> > > > they cannot be isolated from each other. These devices must either be
> > > > entirely under kernel control or userspace control, never a mixture. This
> > > > checks and sets DMA ownership during driver binding, and release the
> > > > ownership during driver unbinding.
> > > > 
> > > > Driver may set a new flag (suppress_auto_claim_dma_owner) to disable auto
> > > > claiming DMA_OWNER_DMA_API ownership in the binding process. For instance,
> > > > the userspace framework drivers (vfio etc.) which need to manually claim
> > > > DMA_OWNER_PRIVATE_DOMAIN_USER when assigning a device to userspace.
> > > 
> > > Why would any vfio driver be a platform driver?  
> > 
> > Why not? VFIO implements drivers for most physical device types
> > these days. Why wouldn't platform be included?
> 
> Because "platform" is not a real device type.  It's a catch-all for
> devices that are only described by firmware, so why would you have a
> virtual device for that?  Why would that be needed?

Why does it matter how a physical device is enumerated?
PCI/DT/ACPI/setup.c - it doesn't matter. There is still a physical
device with physical DMA and MMIO.

As long as people are making ethernet controllers and other
interesting devices enumerated through platform_device there will be
need to expose them to userspace through VFIO too.

> Ok, nevermind, you do have a virtual platform device, which personally,
> I find crazy as why would firmware export a "virtual device"?

Why do you keep saying "virtual device"?

The "VF" in vfio refers to language in the PCI-SIG SRIOV
specification. You are better to think of the V as meaning "for
virtualization".

Today vfio is just a nonsense acronym. The subsystem's job is to allow
user space to fully operate a physical HW device, including using its
DMA and interrupts. With the advent of DPDK/SPDK/etc it isn't even
related to virtualization use-cases any more.

It is a lot like UIO except VFIO can operate devices that use DMA too.

> > @@ -76,6 +76,7 @@ static struct platform_driver vfio_platform_driver = {
> >         .driver = {
> >                 .name   = "vfio-platform",
> >         },
> > +       .suppress_auto_claim_dma_owner = true,
> >  };
> > 
> > Which is how VFIO provides support to DPDK for some Ethernet
> > controllers embedded in a few ARM SOCs.
> 
> Ick.  Where does the DT file for these devices live that describe a
> "virtual device" to match with this driver?

The DT describes a physical ethernet device that would normally load
it's netdev driver. If the admin wishes to use that physical ethernet
device in userspace, say with DPDK, then the admin switches the driver
from netdev to vfio.

The OF compatible string 'calxeda,hb-xgmac' is one example:
 Documentation/devicetree/bindings/net/calxeda-xgmac.yaml
 arch/arm/boot/dts/ecx-common.dtsi
 drivers/net/ethernet/calxeda/xgmac.c
 drivers/vfio/platform/reset/vfio_platform_calxedaxgmac.c

We all recently went over the switching mechanim in a lot of detail
when you Ack'd this patch series:

https://lore.kernel.org/kvm/20210721161609.68223-1-yishaih@nvidia.com/

(though vfio platform has not yet been revised to use this mechanism,
it still uses the sysfs driver_override)

> > It is also used in patch 17 in five tegra platform_drivers to make
> > their sharing of an iommu group between possibly related
> > platform_driver's safer.
> 
> Safer how?

tegra makes assumptions that the DT configures the IOMMU in a certain
way. If the DT does something else then the kernel will probably corrupt
memory with wild DMAs. After this series the conflicting IOMMU usages
will be detected and blocked instead.

Thanks,
Jason
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu