From: Nadav Amit <nadav.amit@gmail.com> To: Joerg Roedel <joro@8bytes.org> Cc: John Garry <john.garry@huawei.com>, Nadav Amit <namit@vmware.com>, Will Deacon <will@kernel.org>, Jiajun Cao <caojiajun@vmware.com>, Lu Baolu <baolu.lu@linux.intel.com>, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Robin Murphy <robin.murphy@arm.com> Subject: [PATCH v6 7/7] iommu/amd: Use only natural aligned flushes in a VM Date: Fri, 23 Jul 2021 02:32:09 -0700 [thread overview] Message-ID: <20210723093209.714328-8-namit@vmware.com> (raw) In-Reply-To: <20210723093209.714328-1-namit@vmware.com> From: Nadav Amit <namit@vmware.com> When running on an AMD vIOMMU, it is better to avoid TLB flushes of unmodified PTEs. vIOMMUs require the hypervisor to synchronize the virtualized IOMMU's PTEs with the physical ones. This process induce overheads. AMD IOMMU allows us to flush any range that is aligned to the power of 2. So when running on top of a vIOMMU, break the range into sub-ranges that are naturally aligned, and flush each one separately. This apporach is better when running with a vIOMMU, but on physical IOMMUs, the penalty of IOTLB misses due to unnecessary flushed entries is likely to be low. Repurpose (i.e., keeping the name, changing the logic) domain_flush_pages() so it is used to choose whether to perform one flush of the whole range or multiple ones to avoid flushing unnecessary ranges. Use NpCache, as usual, to infer whether the IOMMU is physical or virtual. Cc: Joerg Roedel <joro@8bytes.org> Cc: Will Deacon <will@kernel.org> Cc: Jiajun Cao <caojiajun@vmware.com> Cc: Lu Baolu <baolu.lu@linux.intel.com> Cc: iommu@lists.linux-foundation.org Cc: linux-kernel@vger.kernel.org Suggested-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Nadav Amit <namit@vmware.com> --- drivers/iommu/amd/iommu.c | 47 ++++++++++++++++++++++++++++++++++----- 1 file changed, 42 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 7846fcb1e92b..3f6428aa68cd 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -1261,15 +1261,52 @@ static void __domain_flush_pages(struct protection_domain *domain, } static void domain_flush_pages(struct protection_domain *domain, - u64 address, size_t size) + u64 address, size_t size, int pde) { - __domain_flush_pages(domain, address, size, 0); + if (likely(!amd_iommu_np_cache)) { + __domain_flush_pages(domain, address, size, pde); + return; + } + + /* + * When NpCache is on, we infer that we run in a VM and use a vIOMMU. + * In such setups it is best to avoid flushes of ranges which are not + * naturally aligned, since it would lead to flushes of unmodified + * PTEs. Such flushes would require the hypervisor to do more work than + * necessary. Therefore, perform repeated flushes of aligned ranges + * until you cover the range. Each iteration flushes the smaller + * between the natural alignment of the address that we flush and the + * greatest naturally aligned region that fits in the range. + */ + while (size != 0) { + int addr_alignment = __ffs(address); + int size_alignment = __fls(size); + int min_alignment; + size_t flush_size; + + /* + * size is always non-zero, but address might be zero, causing + * addr_alignment to be negative. As the casting of the + * argument in __ffs(address) to long might trim the high bits + * of the address on x86-32, cast to long when doing the check. + */ + if (likely((unsigned long)address != 0)) + min_alignment = min(addr_alignment, size_alignment); + else + min_alignment = size_alignment; + + flush_size = 1ul << min_alignment; + + __domain_flush_pages(domain, address, flush_size, pde); + address += flush_size; + size -= flush_size; + } } /* Flush the whole IO/TLB for a given protection domain - including PDE */ void amd_iommu_domain_flush_tlb_pde(struct protection_domain *domain) { - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 1); + domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 1); } void amd_iommu_domain_flush_complete(struct protection_domain *domain) @@ -1296,7 +1333,7 @@ static void domain_flush_np_cache(struct protection_domain *domain, unsigned long flags; spin_lock_irqsave(&domain->lock, flags); - domain_flush_pages(domain, iova, size); + domain_flush_pages(domain, iova, size, 1); amd_iommu_domain_flush_complete(domain); spin_unlock_irqrestore(&domain->lock, flags); } @@ -2200,7 +2237,7 @@ static void amd_iommu_iotlb_sync(struct iommu_domain *domain, unsigned long flags; spin_lock_irqsave(&dom->lock, flags); - __domain_flush_pages(dom, gather->start, gather->end - gather->start, 1); + domain_flush_pages(dom, gather->start, gather->end - gather->start, 1); amd_iommu_domain_flush_complete(dom); spin_unlock_irqrestore(&dom->lock, flags); } -- 2.25.1
next prev parent reply other threads:[~2021-07-23 17:02 UTC|newest] Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-07-23 9:32 [PATCH v6 0/7] iommu/amd: Enable page-selective flushes Nadav Amit 2021-07-23 9:32 ` [PATCH v6 1/7] iommu/amd: Selective flush on unmap Nadav Amit 2021-07-23 9:32 ` [PATCH v6 2/7] iommu/amd: Do not use flush-queue when NpCache is on Nadav Amit 2021-07-23 9:32 ` [PATCH v6 3/7] iommu: Improve iommu_iotlb_gather helpers Nadav Amit 2021-07-23 9:32 ` [PATCH v6 4/7] iommu: Factor iommu_iotlb_gather_is_disjoint() out Nadav Amit 2021-07-23 9:32 ` [PATCH v6 5/7] iommu/amd: Tailored gather logic for AMD Nadav Amit 2021-07-23 9:32 ` [PATCH v6 6/7] iommu/amd: Sync once for scatter-gather operations Nadav Amit 2021-07-23 9:32 ` Nadav Amit [this message] 2021-07-26 12:31 ` [PATCH v6 0/7] iommu/amd: Enable page-selective flushes Joerg Roedel 2021-08-02 9:26 ` Joerg Roedel
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20210723093209.714328-8-namit@vmware.com \ --to=nadav.amit@gmail.com \ --cc=baolu.lu@linux.intel.com \ --cc=caojiajun@vmware.com \ --cc=iommu@lists.linux-foundation.org \ --cc=john.garry@huawei.com \ --cc=joro@8bytes.org \ --cc=linux-kernel@vger.kernel.org \ --cc=namit@vmware.com \ --cc=robin.murphy@arm.com \ --cc=will@kernel.org \ --subject='Re: [PATCH v6 7/7] iommu/amd: Use only natural aligned flushes in a VM' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).