From: Keqian Zhu <zhukeqian1@huawei.com>
To: <linux-kernel@vger.kernel.org>,
<linux-arm-kernel@lists.infradead.org>, <kvm@vger.kernel.org>,
<kvmarm@lists.cs.columbia.edu>,
<iommu@lists.linux-foundation.org>, Will Deacon <will@kernel.org>,
"Alex Williamson" <alex.williamson@redhat.com>,
Marc Zyngier <maz@kernel.org>,
Catalin Marinas <catalin.marinas@arm.com>
Cc: Cornelia Huck <cohuck@redhat.com>,
lushenming@huawei.com, Kirti Wankhede <kwankhede@nvidia.com>,
Robin Murphy <robin.murphy@arm.com>
Subject: [RFC PATCH 00/11] vfio/iommu_type1: Implement dirty log tracking based on smmuv3 HTTU
Date: Thu, 28 Jan 2021 23:17:31 +0800 [thread overview]
Message-ID: <20210128151742.18840-1-zhukeqian1@huawei.com> (raw)
Hi all,
This patch series implement a new dirty log tracking method for vfio dma.
Intention:
As we know, vfio live migration is an important and valuable feature, but there
are still many hurdles to solve, including migration of interrupt, device state,
DMA dirty log tracking, and etc.
For now, the only dirty log tracking interface is pinning. It has some drawbacks:
1. Only smart vendor drivers are aware of this.
2. It's coarse-grained, the pinned-scope is generally bigger than what the device actually access.
3. It can't track dirty continuously and precisely, vfio populates all pinned-scope as dirty.
So it doesn't work well with iteratively dirty log handling.
About SMMU HTTU:
HTTU (Hardware Translation Table Update) is a feature of ARM SMMUv3, it can update
access flag or/and dirty state of the TTD (Translation Table Descriptor) by hardware.
With HTTU, stage1 TTD is classified into 3 types:
DBM bit AP[2](readonly bit)
1. writable_clean 1 1
2. writable_dirty 1 0
3. readonly 0 1
If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from writable_clean to
writable_dirty. Then software can scan TTD to sync dirty state into dirty bitmap. With
this feature, we can track the dirty log of DMA continuously and precisely.
About this series:
Patch 1-3: Add feature detection for smmu HTTU and enable HTTU for smmu stage1 mapping.
And add feature detection for smmu BBML. We need to split block mapping when
start dirty log tracking and merge page mapping when stop dirty log tracking,
which requires break-before-make procedure. But it might cause problems when the
TTD is alive. The I/O streams might not tolerate translation faults. So BBML
should be used.
Patch 4-7: Add four interfaces (split_block, merge_page, sync_dirty_log and clear_dirty_log)
in IOMMU layer, they are essential to implement dma dirty log tracking for vfio.
We implement these interfaces for arm smmuv3.
Patch 8: Add HWDBM (Hardware Dirty Bit Management) device feature reporting in IOMMU layer.
Patch9-11: Implement a new dirty log tracking method for vfio based on iommu hwdbm. A new
ioctl operation named VFIO_DIRTY_LOG_MANUAL_CLEAR is added, which can eliminate
some redundant dirty handling of userspace.
Optimizations TO Do:
1. We recognized that each smmu_domain (a vfio_container may has several smmu_domain) has its
own stage1 mapping, and we must scan all these mapping to sync dirty state. We plan to refactor
smmu_domain to support more than one smmu in one smmu_domain, then these smmus can share a same
stage1 mapping.
2. We also recognized that scan TTD is a hotspot of performance. Recently, I have implement a
SW/HW conbined dirty log tracking at MMU side [1], which can effectively solve this problem.
This idea can be applied to smmu side too.
Thanks,
Keqian
[1] https://lore.kernel.org/linux-arm-kernel/20210126124444.27136-1-zhukeqian1@huawei.com/
jiangkunkun (11):
iommu/arm-smmu-v3: Add feature detection for HTTU
iommu/arm-smmu-v3: Enable HTTU for SMMU stage1 mapping
iommu/arm-smmu-v3: Add feature detection for BBML
iommu/arm-smmu-v3: Split block descriptor to a span of page
iommu/arm-smmu-v3: Merge a span of page to block descriptor
iommu/arm-smmu-v3: Scan leaf TTD to sync hardware dirty log
iommu/arm-smmu-v3: Clear dirty log according to bitmap
iommu/arm-smmu-v3: Add HWDBM device feature reporting
vfio/iommu_type1: Add HWDBM status maintanance
vfio/iommu_type1: Optimize dirty bitmap population based on iommu
HWDBM
vfio/iommu_type1: Add support for manual dirty log clear
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 138 ++++++-
drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.h | 14 +
drivers/iommu/io-pgtable-arm.c | 392 +++++++++++++++++++-
drivers/iommu/iommu.c | 227 ++++++++++++
drivers/vfio/vfio_iommu_type1.c | 235 +++++++++++-
include/linux/io-pgtable.h | 14 +
include/linux/iommu.h | 55 +++
include/uapi/linux/vfio.h | 28 +-
8 files changed, 1093 insertions(+), 10 deletions(-)
--
2.19.1
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
next reply other threads:[~2021-01-28 15:18 UTC|newest]
Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-28 15:17 Keqian Zhu [this message]
2021-01-28 15:17 ` [RFC PATCH 01/11] iommu/arm-smmu-v3: Add feature detection for HTTU Keqian Zhu
2021-02-04 19:50 ` Robin Murphy
2021-02-05 9:13 ` Keqian Zhu
2021-02-05 9:51 ` Jean-Philippe Brucker
2021-02-07 1:42 ` Keqian Zhu
2021-02-05 11:48 ` Robin Murphy
2021-02-05 16:11 ` Robin Murphy
2021-02-07 1:56 ` Keqian Zhu
2021-02-07 2:19 ` Keqian Zhu
2021-03-02 7:42 ` Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 02/11] iommu/arm-smmu-v3: Enable HTTU for SMMU stage1 mapping Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 03/11] iommu/arm-smmu-v3: Add feature detection for BBML Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 04/11] iommu/arm-smmu-v3: Split block descriptor to a span of page Keqian Zhu
2021-02-04 19:51 ` Robin Murphy
2021-02-07 8:18 ` Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 05/11] iommu/arm-smmu-v3: Merge a span of page to block descriptor Keqian Zhu
2021-02-04 19:52 ` Robin Murphy
2021-02-07 12:13 ` Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 06/11] iommu/arm-smmu-v3: Scan leaf TTD to sync hardware dirty log Keqian Zhu
2021-02-04 19:52 ` Robin Murphy
2021-02-07 12:41 ` Keqian Zhu
2021-02-08 1:17 ` Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 07/11] iommu/arm-smmu-v3: Clear dirty log according to bitmap Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 08/11] iommu/arm-smmu-v3: Add HWDBM device feature reporting Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 09/11] vfio/iommu_type1: Add HWDBM status maintanance Keqian Zhu
2021-01-28 15:17 ` [RFC PATCH 10/11] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM Keqian Zhu
2021-02-07 9:56 ` Yi Sun
2021-02-07 10:40 ` Keqian Zhu
2021-02-09 11:57 ` Yi Sun
2021-02-09 12:08 ` Robin Murphy
2021-02-10 6:15 ` zhukeqian
2021-02-18 1:17 ` Keqian Zhu
2021-02-09 11:16 ` Robin Murphy
2021-02-09 12:02 ` Yi Sun
2021-01-28 15:17 ` [RFC PATCH 11/11] vfio/iommu_type1: Add support for manual dirty log clear Keqian Zhu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210128151742.18840-1-zhukeqian1@huawei.com \
--to=zhukeqian1@huawei.com \
--cc=alex.williamson@redhat.com \
--cc=catalin.marinas@arm.com \
--cc=cohuck@redhat.com \
--cc=iommu@lists.linux-foundation.org \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=kwankhede@nvidia.com \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lushenming@huawei.com \
--cc=maz@kernel.org \
--cc=robin.murphy@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).