Re: [PATCH RFCv2 15/24] iommufd: Add a flag to skip clearing of IOPTE dirty

From: Jason Gunthorpe <jgg@nvidia.com>
To: Joao Martins <joao.m.martins@oracle.com>
Cc: iommu@lists.linux.dev, Kevin Tian <kevin.tian@intel.com>,
	Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>,
	Lu Baolu <baolu.lu@linux.intel.com>, Yi Liu <yi.l.liu@intel.com>,
	Yi Y Sun <yi.y.sun@intel.com>, Eric Auger <eric.auger@redhat.com>,
	Nicolin Chen <nicolinc@nvidia.com>,
	Joerg Roedel <joro@8bytes.org>,
	Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>,
	Will Deacon <will@kernel.org>,
	Robin Murphy <robin.murphy@arm.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	kvm@vger.kernel.org
Subject: Re: [PATCH RFCv2 15/24] iommufd: Add a flag to skip clearing of IOPTE dirty
Date: Fri, 19 May 2023 10:54:13 -0300	[thread overview]
Message-ID: <ZGd/he0eQtLM+mfe@nvidia.com> (raw)
In-Reply-To: <20230518204650.14541-16-joao.m.martins@oracle.com>

On Thu, May 18, 2023 at 09:46:41PM +0100, Joao Martins wrote:
> VFIO has an operation where you an unmap an IOVA while returning a bitmap
> with the dirty data. In reality the operation doesn't quite query the IO
> pagetables that the the PTE was dirty or not, but it marks as dirty in the
> bitmap anything that was mapped all in one operation.
> 
> In IOMMUFD this equivalent can be done in two operations by querying with
> GET_DIRTY_IOVA followed by UNMAP_IOVA. However, this would incur two TLB
> flushes given that after clearing dirty bits IOMMU implementations require
> invalidating their IOTLB, plus another invalidation needed for the UNMAP.
> To allow dirty bits to be queried faster, we add a flag
> (IOMMU_GET_DIRTY_IOVA_NO_CLEAR) that requests to not clear the dirty bits
> from the PTE (but just reading them), under the expectation that the next
> operation is the unmap. An alternative is to unmap and just perpectually
> mark as dirty as that's the same behaviour as today. So here equivalent
> functionally can be provided, and if real dirty info is required we
> amortize the cost while querying.
> 
> There's still a race against DMA where in theory the unmap of the IOVA
> (when the guest invalidates the IOTLB via emulated iommu) would race
> against the VF performing DMA on the same IOVA being invalidated which
> would be marking the PTE as dirty but losing the update in the
> unmap-related IOTLB flush. The way to actually prevent the race would be to
> write-protect the IOPTE, then query dirty bits and flush the IOTLB in the
> unmap after.  However, this remains an issue that is so far theoretically
> possible, but lacks an use case or whether the race is relevant in the
> first place that justifies such complexity.
> 
> Link:
> https://lore.kernel.org/linux-iommu/20220502185239.GR8364@nvidia.com/

I think you should clip the explanation from the email into the commit
message - eg that we are accepting to resolve this race as throwing
away the DMA and it doesn't matter if it hit physical DRAM or not, the
VM can't tell if we threw it away because the DMA was blocked or
because we failed to copy the DRAM.

Jason