Re: Question about reserved_regions w/ Intel IOMMU

From: Jason Gunthorpe <jgg@nvidia.com>
To: "Tian, Kevin" <kevin.tian@intel.com>
Cc: Robin Murphy <robin.murphy@arm.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Baolu Lu <baolu.lu@linux.intel.com>,
	Alexander Duyck <alexander.duyck@gmail.com>,
	LKML <linux-kernel@vger.kernel.org>,
	linux-pci <linux-pci@vger.kernel.org>,
	"iommu@lists.linux.dev" <iommu@lists.linux.dev>
Subject: Re: Question about reserved_regions w/ Intel IOMMU
Date: Fri, 16 Jun 2023 09:20:40 -0300	[thread overview]
Message-ID: <ZIxTmGU4a5dniEY3@nvidia.com> (raw)
In-Reply-To: <BN9PR11MB52765C24405D2475CF3CBEBE8C58A@BN9PR11MB5276.namprd11.prod.outlook.com>

On Fri, Jun 16, 2023 at 08:39:46AM +0000, Tian, Kevin wrote:
> +Alex
> 
> > From: Jason Gunthorpe <jgg@nvidia.com>
> > Sent: Tuesday, June 13, 2023 11:54 PM
> > 
> > On Thu, Jun 08, 2023 at 04:28:24PM +0100, Robin Murphy wrote:
> > 
> > > > The iova_reserve_pci_windows() you've seen is for kernel DMA interfaces
> > > > which is not related to peer-to-peer accesses.
> > >
> > > Right, in general the IOMMU driver cannot be held responsible for
> > whatever
> > > might happen upstream of the IOMMU input.
> > 
> > The driver yes, but..
> > 
> > > The DMA layer carves PCI windows out of its IOVA space
> > > unconditionally because we know that they *might* be problematic,
> > > and we don't have any specific constraints on our IOVA layout so
> > > it's no big deal to just sacrifice some space for simplicity.
> > 
> > This is a problem for everything using UNMANAGED domains. If the iommu
> > API user picks an IOVA it should be able to expect it to work. If the
> > intereconnect fails to allow it to work then this has to be discovered
> > otherwise UNAMANGED domains are not usable at all.
> > 
> > Eg vfio and iommufd are also in trouble on these configurations.
> > 
> 
> If those PCI windows are problematic e.g. due to ACS they belong to
> a single iommu group. If a vfio user opens all the devices in that group
> then it can discover and reserve those windows in its IOVA space. 

How? We don't even exclude the single device's BAR if there is no ACS?

> The problem is that the user may not open all the devices then
> currently there is no way for it to know the windows on those
> unopened devices.
> 
> Curious why nobody complains about this gap before this thread...

Probably because it only matters if you have a real PCIe switch in the
system, which is pretty rare.

Jason