* [RFC PATCH v2 0/3] vfio/iommu_type1: Implement dirty log tracking based on IOMMU HWDBM
@ 2021-05-07 10:36 Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Keqian Zhu @ 2021-05-07 10:36 UTC (permalink / raw)
To: linux-kernel, kvm, Alex Williamson, Kirti Wankhede,
Cornelia Huck, Yi Sun, Tian Kevin
Cc: Jonathan Cameron, Robin Murphy, Will Deacon, Joerg Roedel,
Jean-Philippe Brucker, Lu Baolu, wanghaibin.wang, jiangkunkun,
yuzenghui, lushenming
Hi Alex and everyone,
This patch series implement vfio dma dirty log tracking based on IOMMU HWDBM (hardware
dirty bit management, such as SMMU with HTTU or intel IOMMU with SLADE).
This patch series is split from the series[1] that containes both IOMMU part and
VFIO part. Please refer the new IOMMU part v4[2] to review or test.
Changelog:
v2:
- Use separate ioctl to get dirty log without clear it automatically. (Alex)
- Implement based on new iommu dirty tracking framework.
- Track hwdbm status at domain level.
- Bugfix: When get_no_clear, we should recover dirty bitmap too.
- Bugfix: When switch from full dirty policy to iommu hwdbm policy, we should populate full dirty.
Intention:
As we know, vfio live migration is an important and valuable feature, but there
are still many hurdles to solve, including migration of interrupt, device state,
DMA dirty log tracking, and etc.
For now, the only dirty log tracking interface is pinning. It has some drawbacks:
1. Only smart vendor drivers are aware of this.
2. It's coarse-grained, the pinned-scope is generally bigger than what the device actually access.
3. It can't track dirty continuously and precisely, vfio populates all pinned-scope as dirty.
So it doesn't work well with iteratively dirty log handling.
About this series:
Implement a new dirty log tracking method for vfio based on iommu hwdbm. A new
ioctl operation named VFIO_DIRTY_LOG_MANUAL_CLEAR is added, which can eliminate
some redundant dirty handling of userspace.
Optimizations Todo:
1. We recognized that each smmu_domain (a vfio_container may has several smmu_domain) has its
own stage1 mapping, and we must scan all these mapping to sync dirty state. We plan to refactor
smmu_domain to support more than one smmu in one smmu_domain, then these smmus can share a same
stage1 mapping.
2. We also recognized that scan TTD is a hotspot of performance. Recently, I have implement a
SW/HW conbined dirty log tracking at MMU side[3], which can effectively solve this problem.
This idea can be applied to smmu side too.
Thanks,
Keqian
[1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/
[2] https://lore.kernel.org/linux-iommu/20210507102211.8836-1-zhukeqian1@huawei.com/
[3] https://lore.kernel.org/linux-arm-kernel/20210126124444.27136-1-zhukeqian1@huawei.com/
Kunkun Jiang (3):
vfio/iommu_type1: Add HWDBM status maintenance
vfio/iommu_type1: Optimize dirty bitmap population based on iommu
HWDBM
vfio/iommu_type1: Add support for manual dirty log clear
drivers/vfio/vfio_iommu_type1.c | 315 ++++++++++++++++++++++++++++++--
include/uapi/linux/vfio.h | 36 +++-
2 files changed, 337 insertions(+), 14 deletions(-)
--
2.19.1
^ permalink raw reply [flat|nested] 7+ messages in thread
* [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance
2021-05-07 10:36 [RFC PATCH v2 0/3] vfio/iommu_type1: Implement dirty log tracking based on IOMMU HWDBM Keqian Zhu
@ 2021-05-07 10:36 ` Keqian Zhu
2021-05-07 16:15 ` kernel test robot
2021-05-07 18:25 ` kernel test robot
2021-05-07 10:36 ` [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 3/3] vfio/iommu_type1: Add support for manual dirty log clear Keqian Zhu
2 siblings, 2 replies; 7+ messages in thread
From: Keqian Zhu @ 2021-05-07 10:36 UTC (permalink / raw)
To: linux-kernel, kvm, Alex Williamson, Kirti Wankhede,
Cornelia Huck, Yi Sun, Tian Kevin
Cc: Jonathan Cameron, Robin Murphy, Will Deacon, Joerg Roedel,
Jean-Philippe Brucker, Lu Baolu, wanghaibin.wang, jiangkunkun,
yuzenghui, lushenming
From: Kunkun Jiang <jiangkunkun@huawei.com>
We are going to optimize dirty log tracking based on iommu dirty
log tracking, but the dirty log from iommu is useful only when
all iommu backed domains support it.
This maintains a counter in vfio_iommu, which is used for dirty
bitmap population in next patch.
This also maintains a boolean flag in vfio_domain, which is used
in the policy of switch dirty log in next patch.
Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Kunkun Jiang <jiangkunkun@huawei.com>
---
drivers/vfio/vfio_iommu_type1.c | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index a0747c35a778..146aaf95589c 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -73,6 +73,7 @@ struct vfio_iommu {
unsigned int vaddr_invalid_count;
uint64_t pgsize_bitmap;
uint64_t num_non_pinned_groups;
+ uint64_t num_non_hwdbm_domains;
wait_queue_head_t vaddr_wait;
bool v2;
bool nesting;
@@ -86,6 +87,7 @@ struct vfio_domain {
struct list_head group_list;
int prot; /* IOMMU_CACHE */
bool fgsp; /* Fine-grained super pages */
+ bool iommu_hwdbm; /* Hardware dirty management */
};
struct vfio_dma {
@@ -2238,6 +2240,26 @@ static void vfio_iommu_iova_insert_copy(struct vfio_iommu *iommu,
list_splice_tail(iova_copy, iova);
}
+/*
+ * Called after a new group is added to the iommu_domain, or an old group is
+ * removed from the iommu_domain. Update the HWDBM status of vfio_domain and
+ * vfio_iommu.
+ */
+static void vfio_iommu_update_hwdbm(struct vfio_iommu *iommu,
+ struct vfio_domain *domain,
+ bool attach)
+{
+ bool old_hwdbm = domain->iommu_hwdbm;
+ bool new_hwdbm = iommu_support_dirty_log(domain->domain);
+
+ if (old_hwdbm && !new_hwdbm && attach) {
+ iommu->num_non_hwdbm_domains++;
+ } else if (!old_hwdbm && new_hwdbm && !attach) {
+ iommu->num_non_hwdbm_domains--;
+ }
+ domain->iommu_hwdbm = new_hwdbm;
+}
+
static int vfio_iommu_type1_attach_group(void *iommu_data,
struct iommu_group *iommu_group)
{
@@ -2391,6 +2413,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
vfio_iommu_detach_group(domain, group);
if (!vfio_iommu_attach_group(d, group)) {
list_add(&group->next, &d->group_list);
+ vfio_iommu_update_hwdbm(iommu, d, true);
iommu_domain_free(domain->domain);
kfree(domain);
goto done;
@@ -2417,6 +2440,7 @@ static int vfio_iommu_type1_attach_group(void *iommu_data,
list_add(&domain->next, &iommu->domain_list);
vfio_update_pgsize_bitmap(iommu);
+ vfio_iommu_update_hwdbm(iommu, domain, true);
done:
/* Delete the old one and insert new iova list */
vfio_iommu_iova_insert_copy(iommu, &iova_copy);
@@ -2599,6 +2623,7 @@ static void vfio_iommu_type1_detach_group(void *iommu_data,
continue;
vfio_iommu_detach_group(domain, group);
+ vfio_iommu_update_hwdbm(iommu, domain, false);
update_dirty_scope = !group->pinned_page_dirty_scope;
list_del(&group->next);
kfree(group);
--
2.19.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM
2021-05-07 10:36 [RFC PATCH v2 0/3] vfio/iommu_type1: Implement dirty log tracking based on IOMMU HWDBM Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
@ 2021-05-07 10:36 ` Keqian Zhu
2021-05-08 1:03 ` kernel test robot
2021-05-07 10:36 ` [RFC PATCH v2 3/3] vfio/iommu_type1: Add support for manual dirty log clear Keqian Zhu
2 siblings, 1 reply; 7+ messages in thread
From: Keqian Zhu @ 2021-05-07 10:36 UTC (permalink / raw)
To: linux-kernel, kvm, Alex Williamson, Kirti Wankhede,
Cornelia Huck, Yi Sun, Tian Kevin
Cc: Jonathan Cameron, Robin Murphy, Will Deacon, Joerg Roedel,
Jean-Philippe Brucker, Lu Baolu, wanghaibin.wang, jiangkunkun,
yuzenghui, lushenming
From: Kunkun Jiang <jiangkunkun@huawei.com>
In the past if vfio_iommu is not of pinned_page_dirty_scope and
vfio_dma is iommu_mapped, we populate full dirty bitmap for this
vfio_dma. Now we can try to get dirty log from iommu before make
the lousy decision.
The new dirty bitmap population policy:
In detail, if all vfio_group are of pinned_page_dirty_scope, the
dirty bitmap population is not affected. If there are vfio_groups
not of pinned_page_dirty_scope and all domains support HWDBM, we
can try to get dirty log from IOMMU. Otherwise, lead to full dirty
bitmap.
Consider DMA and group hotplug:
Start dirty log for newly added DMA range, and stop dirty log for
DMA range going to remove.
If a domain don't support HWDBM at start, but can support it after
hotplug some groups (attach a first group with HWDBM or detach all
groups without HWDBM). If a domain support HWDBM at start, but do
not support it after hotplug some groups (attach a group without
HWDBM or detach all groups without HWDBM). So our policy is that
switch dirty log for domains dynamically.
Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Kunkun Jiang <jiangkunkun@huawei.com>
---
drivers/vfio/vfio_iommu_type1.c | 180 ++++++++++++++++++++++++++++++--
1 file changed, 172 insertions(+), 8 deletions(-)
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 146aaf95589c..12fe1cf9113e 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -1200,6 +1200,46 @@ static void vfio_update_pgsize_bitmap(struct vfio_iommu *iommu)
}
}
+static int vfio_iommu_dirty_log_clear(struct vfio_iommu *iommu,
+ dma_addr_t start_iova, size_t size,
+ unsigned long *bitmap_buffer,
+ dma_addr_t base_iova,
+ unsigned long pgshift)
+{
+ struct vfio_domain *d;
+ int ret = 0;
+
+ list_for_each_entry(d, &iommu->domain_list, next) {
+ ret = iommu_clear_dirty_log(d->domain, start_iova, size,
+ bitmap_buffer, base_iova, pgshift);
+ if (ret) {
+ pr_warn("vfio_iommu dirty log clear failed!\n");
+ break;
+ }
+ }
+
+ return ret;
+}
+
+static int vfio_iommu_dirty_log_sync(struct vfio_iommu *iommu,
+ struct vfio_dma *dma,
+ unsigned long pgshift)
+{
+ struct vfio_domain *d;
+ int ret = 0;
+
+ list_for_each_entry(d, &iommu->domain_list, next) {
+ ret = iommu_sync_dirty_log(d->domain, dma->iova, dma->size,
+ dma->bitmap, dma->iova, pgshift);
+ if (ret) {
+ pr_warn("vfio_iommu dirty log sync failed!\n");
+ break;
+ }
+ }
+
+ return ret;
+}
+
static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
struct vfio_dma *dma, dma_addr_t base_iova,
size_t pgsize)
@@ -1210,13 +1250,24 @@ static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
unsigned long copy_offset = bit_offset / BITS_PER_LONG;
unsigned long shift = bit_offset % BITS_PER_LONG;
unsigned long leftover;
+ bool iommu_hwdbm_dirty = false;
+ int ret;
- /*
- * mark all pages dirty if any IOMMU capable device is not able
- * to report dirty pages and all pages are pinned and mapped.
- */
- if (iommu->num_non_pinned_groups && dma->iommu_mapped)
+ if (!iommu->num_non_pinned_groups || !dma->iommu_mapped) {
+ /* nothing to do */
+ } else if (!iommu->num_non_hwdbm_domains) {
+ /* try to get dirty log from IOMMU */
+ iommu_hwdbm_dirty = true;
+ ret = vfio_iommu_dirty_log_sync(iommu, dma, pgshift);
+ if (ret)
+ return ret;
+ } else {
+ /*
+ * mark all pages dirty if any IOMMU capable device is not able
+ * to report dirty pages and all pages are pinned and mapped.
+ */
bitmap_set(dma->bitmap, 0, nbits);
+ }
if (shift) {
bitmap_shift_left(dma->bitmap, dma->bitmap, shift,
@@ -1234,6 +1285,11 @@ static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
DIRTY_BITMAP_BYTES(nbits + shift)))
return -EFAULT;
+ /* Recover the bitmap if it'll be used to clear hardware dirty log */
+ if (shift && iommu_hwdbm_dirty)
+ bitmap_shift_right(dma->bitmap, dma->bitmap, shift,
+ nbits + shift);
+
return 0;
}
@@ -1272,6 +1328,16 @@ static int vfio_iova_dirty_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
if (ret)
return ret;
+ /* Clear iommu dirty log to re-enable dirty log tracking */
+ if (iommu->num_non_pinned_groups && dma->iommu_mapped &&
+ !iommu->num_non_hwdbm_domains) {
+ ret = vfio_iommu_dirty_log_clear(iommu, dma->iova,
+ dma->size, dma->bitmap, dma->iova,
+ pgshift);
+ if (ret)
+ return ret;
+ }
+
/*
* Re-populate bitmap to include all pinned pages which are
* considered as dirty but exclude pages which are unpinned and
@@ -1292,6 +1358,22 @@ static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size)
return 0;
}
+static void vfio_dma_dirty_log_switch(struct vfio_iommu *iommu,
+ struct vfio_dma *dma, bool enable)
+{
+ struct vfio_domain *d;
+
+ if (!dma->iommu_mapped)
+ return;
+
+ list_for_each_entry(d, &iommu->domain_list, next) {
+ if (!d->iommu_hwdbm)
+ continue;
+ WARN_ON(iommu_switch_dirty_log(d->domain, enable, dma->iova,
+ dma->size, d->prot | dma->prot));
+ }
+}
+
static int vfio_dma_do_unmap(struct vfio_iommu *iommu,
struct vfio_iommu_type1_dma_unmap *unmap,
struct vfio_bitmap *bitmap)
@@ -1444,6 +1526,10 @@ static int vfio_dma_do_unmap(struct vfio_iommu *iommu,
break;
}
+ /* Stop log for removed dma */
+ if (iommu->dirty_page_tracking)
+ vfio_dma_dirty_log_switch(iommu, dma, false);
+
unmapped += dma->size;
n = rb_next(n);
vfio_remove_dma(iommu, dma);
@@ -1675,8 +1761,13 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu,
if (!ret && iommu->dirty_page_tracking) {
ret = vfio_dma_bitmap_alloc(dma, pgsize);
- if (ret)
+ if (ret) {
vfio_remove_dma(iommu, dma);
+ goto out_unlock;
+ }
+
+ /* Start dirty log for newly added dma */
+ vfio_dma_dirty_log_switch(iommu, dma, true);
}
out_unlock:
@@ -2240,6 +2331,21 @@ static void vfio_iommu_iova_insert_copy(struct vfio_iommu *iommu,
list_splice_tail(iova_copy, iova);
}
+static void vfio_domain_dirty_log_switch(struct vfio_iommu *iommu,
+ struct vfio_domain *d, bool enable)
+{
+ struct rb_node *n;
+ struct vfio_dma *dma;
+
+ for (n = rb_first(&iommu->dma_list); n; n = rb_next(n)) {
+ dma = rb_entry(n, struct vfio_dma, node);
+ if (!dma->iommu_mapped)
+ continue;
+ WARN_ON(iommu_switch_dirty_log(d->domain, enable, dma->iova,
+ dma->size, d->prot | dma->prot));
+ }
+}
+
/*
* Called after a new group is added to the iommu_domain, or an old group is
* removed from the iommu_domain. Update the HWDBM status of vfio_domain and
@@ -2251,13 +2357,48 @@ static void vfio_iommu_update_hwdbm(struct vfio_iommu *iommu,
{
bool old_hwdbm = domain->iommu_hwdbm;
bool new_hwdbm = iommu_support_dirty_log(domain->domain);
+ bool singular = list_is_singular(&domain->group_list);
+ bool num_non_hwdbm_zeroed = false;
+ bool log_enabled, should_enable;
if (old_hwdbm && !new_hwdbm && attach) {
iommu->num_non_hwdbm_domains++;
} else if (!old_hwdbm && new_hwdbm && !attach) {
iommu->num_non_hwdbm_domains--;
+ if (!iommu->num_non_hwdbm_domains)
+ num_non_hwdbm_zeroed = true;
}
domain->iommu_hwdbm = new_hwdbm;
+
+ if (!iommu->dirty_page_tracking)
+ return;
+
+ /*
+ * When switch the dirty policy from full dirty to iommu hwdbm, we must
+ * populate full dirty now to avoid losing dirty.
+ */
+ if (iommu->num_non_pinned_groups && num_non_hwdbm_zeroed)
+ vfio_iommu_populate_bitmap_full(iommu);
+
+ /*
+ * The vfio_domain can switch dirty log tracking dynamically due to
+ * group attach/detach. The basic idea is to convert current dirty log
+ * status to desired dirty log status.
+ *
+ * If old_hwdbm is true then dirty log has been enabled. One exception
+ * is that this is the first group attached to a domain.
+ *
+ * If new_hwdbm is true then dirty log should be enabled. One exception
+ * is that this is the last group detached from a domain.
+ */
+ log_enabled = old_hwdbm && !(attach && singular);
+ should_enable = new_hwdbm && !(!attach && singular);
+
+ /* Switch dirty log tracking when status changed */
+ if (should_enable && !log_enabled)
+ vfio_domain_dirty_log_switch(iommu, domain, true);
+ else if (!should_enable && log_enabled)
+ vfio_domain_dirty_log_switch(iommu, domain, false);
}
static int vfio_iommu_type1_attach_group(void *iommu_data,
@@ -2664,7 +2805,11 @@ static void vfio_iommu_type1_detach_group(void *iommu_data,
*/
if (update_dirty_scope) {
iommu->num_non_pinned_groups--;
- if (iommu->dirty_page_tracking)
+ /*
+ * When switch the dirty policy from full dirty to pinned scope,
+ * we must populate full dirty now to avoid losing dirty.
+ */
+ if (iommu->dirty_page_tracking && iommu->num_non_hwdbm_domains)
vfio_iommu_populate_bitmap_full(iommu);
}
mutex_unlock(&iommu->lock);
@@ -3008,6 +3153,22 @@ static int vfio_iommu_type1_unmap_dma(struct vfio_iommu *iommu,
-EFAULT : 0;
}
+static void vfio_iommu_dirty_log_switch(struct vfio_iommu *iommu, bool enable)
+{
+ struct vfio_domain *d;
+
+ /*
+ * We enable dirty log tracking for these vfio_domains that support
+ * HWDBM. Even if all iommu domains don't support HWDBM for now. They
+ * may support it after detach some groups.
+ */
+ list_for_each_entry(d, &iommu->domain_list, next) {
+ if (!d->iommu_hwdbm)
+ continue;
+ vfio_domain_dirty_log_switch(iommu, d, enable);
+ }
+}
+
static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
unsigned long arg)
{
@@ -3040,8 +3201,10 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
pgsize = 1 << __ffs(iommu->pgsize_bitmap);
if (!iommu->dirty_page_tracking) {
ret = vfio_dma_bitmap_alloc_all(iommu, pgsize);
- if (!ret)
+ if (!ret) {
iommu->dirty_page_tracking = true;
+ vfio_iommu_dirty_log_switch(iommu, true);
+ }
}
mutex_unlock(&iommu->lock);
return ret;
@@ -3050,6 +3213,7 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
if (iommu->dirty_page_tracking) {
iommu->dirty_page_tracking = false;
vfio_dma_bitmap_free_all(iommu);
+ vfio_iommu_dirty_log_switch(iommu, false);
}
mutex_unlock(&iommu->lock);
return 0;
--
2.19.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [RFC PATCH v2 3/3] vfio/iommu_type1: Add support for manual dirty log clear
2021-05-07 10:36 [RFC PATCH v2 0/3] vfio/iommu_type1: Implement dirty log tracking based on IOMMU HWDBM Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM Keqian Zhu
@ 2021-05-07 10:36 ` Keqian Zhu
2 siblings, 0 replies; 7+ messages in thread
From: Keqian Zhu @ 2021-05-07 10:36 UTC (permalink / raw)
To: linux-kernel, kvm, Alex Williamson, Kirti Wankhede,
Cornelia Huck, Yi Sun, Tian Kevin
Cc: Jonathan Cameron, Robin Murphy, Will Deacon, Joerg Roedel,
Jean-Philippe Brucker, Lu Baolu, wanghaibin.wang, jiangkunkun,
yuzenghui, lushenming
From: Kunkun Jiang <jiangkunkun@huawei.com>
In the past, we clear dirty log immediately after sync dirty
log to userspace. This may cause redundant dirty handling if
userspace handles dirty log iteratively:
After vfio clears dirty log, new dirty log starts to generate.
These new dirty log will be reported to userspace even if they
are generated before userspace handles the same dirty page.
That's to say, we should minimize the time gap of dirty log
clearing and dirty log handling.
This adds two user interfaces. Note that user should clear dirty
log before handle corresponding dirty pages.
1. GET_BITMAP_NOCLEAR: get dirty log without clear.
2. CLEAR_BITMAP: manually clear dirty log.
Co-developed-by: Keqian Zhu <zhukeqian1@huawei.com>
Signed-off-by: Kunkun Jiang <jiangkunkun@huawei.com>
---
drivers/vfio/vfio_iommu_type1.c | 114 ++++++++++++++++++++++++++++++--
include/uapi/linux/vfio.h | 36 +++++++++-
2 files changed, 142 insertions(+), 8 deletions(-)
diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index 12fe1cf9113e..9efd2586f1d0 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -78,6 +78,7 @@ struct vfio_iommu {
bool v2;
bool nesting;
bool dirty_page_tracking;
+ bool dirty_log_get_no_clear;
bool container_open;
};
@@ -1240,6 +1241,76 @@ static int vfio_iommu_dirty_log_sync(struct vfio_iommu *iommu,
return ret;
}
+static int vfio_iova_dirty_log_clear(u64 __user *bitmap,
+ struct vfio_iommu *iommu,
+ dma_addr_t iova, size_t size,
+ size_t pgsize)
+{
+ struct vfio_dma *dma;
+ struct rb_node *n;
+ dma_addr_t start_iova, end_iova, riova;
+ unsigned long pgshift = __ffs(pgsize);
+ unsigned long bitmap_size;
+ unsigned long *bitmap_buffer = NULL;
+ bool clear_valid;
+ int rs, re, start, end, dma_offset;
+ int ret = 0;
+
+ bitmap_size = DIRTY_BITMAP_BYTES(size >> pgshift);
+ bitmap_buffer = kvmalloc(bitmap_size, GFP_KERNEL);
+ if (!bitmap_buffer) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ if (copy_from_user(bitmap_buffer, bitmap, bitmap_size)) {
+ ret = -EFAULT;
+ goto out;
+ }
+
+ for (n = rb_first(&iommu->dma_list); n; n = rb_next(n)) {
+ dma = rb_entry(n, struct vfio_dma, node);
+ if ((dma->iova + dma->size - 1) < iova)
+ continue;
+ if (dma->iova > iova + size - 1)
+ break;
+
+ start_iova = max(iova, dma->iova);
+ end_iova = min(iova + size, dma->iova + dma->size);
+
+ /* Similar logic as the tail of vfio_iova_dirty_bitmap */
+
+ clear_valid = false;
+ start = (start_iova - iova) >> pgshift;
+ end = (end_iova - iova) >> pgshift;
+ bitmap_for_each_set_region(bitmap_buffer, rs, re, start, end) {
+ clear_valid = true;
+ riova = iova + (rs << pgshift);
+ dma_offset = (riova - dma->iova) >> pgshift;
+ bitmap_clear(dma->bitmap, dma_offset, re - rs);
+ }
+
+ if (clear_valid)
+ vfio_dma_populate_bitmap(dma, pgsize);
+
+ if (clear_valid && iommu->num_non_pinned_groups &&
+ dma->iommu_mapped && !iommu->num_non_hwdbm_domains) {
+ ret = vfio_iommu_dirty_log_clear(iommu, start_iova,
+ end_iova - start_iova, bitmap_buffer,
+ iova, pgshift);
+ if (ret) {
+ pr_warn("dma dirty log clear failed!\n");
+ goto out;
+ }
+ }
+
+ }
+
+out:
+ kfree(bitmap_buffer);
+ return ret;
+}
+
static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
struct vfio_dma *dma, dma_addr_t base_iova,
size_t pgsize)
@@ -1285,8 +1356,11 @@ static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
DIRTY_BITMAP_BYTES(nbits + shift)))
return -EFAULT;
- /* Recover the bitmap if it'll be used to clear hardware dirty log */
- if (shift && iommu_hwdbm_dirty)
+ /*
+ * Recover the bitmap if it'll be used to clear hardware dirty log, or
+ * user wants to clear the dirty bitmap manually.
+ */
+ if (shift && (iommu_hwdbm_dirty || iommu->dirty_log_get_no_clear))
bitmap_shift_right(dma->bitmap, dma->bitmap, shift,
nbits + shift);
@@ -1328,6 +1402,10 @@ static int vfio_iova_dirty_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
if (ret)
return ret;
+ /* Do not clear dirty automatically when require no clear */
+ if (iommu->dirty_log_get_no_clear)
+ continue;
+
/* Clear iommu dirty log to re-enable dirty log tracking */
if (iommu->num_non_pinned_groups && dma->iommu_mapped &&
!iommu->num_non_hwdbm_domains) {
@@ -2920,6 +2998,10 @@ static int vfio_iommu_type1_check_extension(struct vfio_iommu *iommu,
if (!iommu)
return 0;
return vfio_domains_have_iommu_cache(iommu);
+ case VFIO_DIRTY_LOG_MANUAL_CLEAR:
+ if (!iommu)
+ return 0;
+ return 1;
default:
return 0;
}
@@ -3175,7 +3257,9 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
struct vfio_iommu_type1_dirty_bitmap dirty;
uint32_t mask = VFIO_IOMMU_DIRTY_PAGES_FLAG_START |
VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP |
- VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP;
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP |
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR |
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP;
unsigned long minsz;
int ret = 0;
@@ -3217,7 +3301,9 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
}
mutex_unlock(&iommu->lock);
return 0;
- } else if (dirty.flags & VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP) {
+ } else if (dirty.flags & (VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP |
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR |
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP)) {
struct vfio_iommu_type1_dirty_bitmap_get range;
unsigned long pgshift;
size_t data_size = dirty.argsz - minsz;
@@ -3260,13 +3346,27 @@ static int vfio_iommu_type1_dirty_pages(struct vfio_iommu *iommu,
goto out_unlock;
}
- if (iommu->dirty_page_tracking)
+ if (!iommu->dirty_page_tracking) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
+ if (dirty.flags & (VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP |
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR)) {
+
+ iommu->dirty_log_get_no_clear = !!(dirty.flags &
+ VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR);
+
ret = vfio_iova_dirty_bitmap(range.bitmap.data,
iommu, range.iova,
range.size,
range.bitmap.pgsize);
- else
- ret = -EINVAL;
+ } else {
+ ret = vfio_iova_dirty_log_clear(range.bitmap.data,
+ iommu, range.iova,
+ range.size,
+ range.bitmap.pgsize);
+ }
out_unlock:
mutex_unlock(&iommu->lock);
diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h
index 34b1f53a3901..0d4281fe9e41 100644
--- a/include/uapi/linux/vfio.h
+++ b/include/uapi/linux/vfio.h
@@ -52,6 +52,16 @@
/* Supports the vaddr flag for DMA map and unmap */
#define VFIO_UPDATE_VADDR 10
+/*
+ * The vfio_iommu driver may support user clears dirty log manually, which means
+ * dirty log can be requested to not cleared automatically after dirty log is
+ * copied to userspace, it's user's duty to clear dirty log.
+ *
+ * Note: please refer to VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR and
+ * VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP.
+ */
+#define VFIO_DIRTY_LOG_MANUAL_CLEAR 11
+
/*
* The IOCTL interface is designed for extensibility by embedding the
* structure length (argsz) and flags into structures passed between
@@ -1158,8 +1168,30 @@ struct vfio_iommu_type1_dma_unmap {
* actual bitmap. If dirty pages logging is not enabled, an error will be
* returned.
*
- * Only one of the flags _START, _STOP and _GET may be specified at a time.
+ * The VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR flag is almost same as
+ * VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP, except that it requires underlying
+ * dirty bitmap is not cleared automatically. The user can clear it manually by
+ * calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP flag set.
*
+ * Calling the IOCTL with VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP flag set,
+ * instructs the IOMMU driver to clear the dirty status of pages in a bitmap
+ * for IOMMU container for a given IOVA range. The user must specify the IOVA
+ * range, the bitmap and the pgsize through the structure
+ * vfio_iommu_type1_dirty_bitmap_get in the data[] portion. This interface
+ * supports clearing a bitmap of the smallest supported pgsize only and can be
+ * modified in future to clear a bitmap of any specified supported pgsize. The
+ * user must provide a memory area for the bitmap memory and specify its size
+ * in bitmap.size. One bit is used to represent one page consecutively starting
+ * from iova offset. The user should provide page size in bitmap.pgsize field.
+ * A bit set in the bitmap indicates that the page at that offset from iova is
+ * cleared the dirty status, and dirty tracking is re-enabled for that page. The
+ * caller must set argsz to a value including the size of structure
+ * vfio_iommu_dirty_bitmap_get, but excluing the size of the actual bitmap. If
+ * dirty pages logging is not enabled, an error will be returned. Note: user
+ * should clear dirty log before handle corresponding dirty pages.
+ *
+ * Only one of the flags _START, _STOP, _GET, _GET_NOCLEAR_, and _CLEAR may be
+ * specified at a time.
*/
struct vfio_iommu_type1_dirty_bitmap {
__u32 argsz;
@@ -1167,6 +1199,8 @@ struct vfio_iommu_type1_dirty_bitmap {
#define VFIO_IOMMU_DIRTY_PAGES_FLAG_START (1 << 0)
#define VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP (1 << 1)
#define VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP (1 << 2)
+#define VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP_NOCLEAR (1 << 3)
+#define VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP (1 << 4)
__u8 data[];
};
--
2.19.1
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
@ 2021-05-07 16:15 ` kernel test robot
2021-05-07 18:25 ` kernel test robot
1 sibling, 0 replies; 7+ messages in thread
From: kernel test robot @ 2021-05-07 16:15 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2873 bytes --]
Hi Keqian,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on vfio/next]
[also build test ERROR on linux/master linus/master v5.12 next-20210507]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
base: https://github.com/awilliam/linux-vfio.git next
config: arm-randconfig-r013-20210507 (attached as .config)
compiler: arm-linux-gnueabi-gcc (GCC) 9.3.0
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/be47d8bac24aa1054b1c5719a2b6ac8928ea43e1
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
git checkout be47d8bac24aa1054b1c5719a2b6ac8928ea43e1
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross W=1 ARCH=arm
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
drivers/vfio/vfio_iommu_type1.c: In function 'vfio_iommu_update_hwdbm':
>> drivers/vfio/vfio_iommu_type1.c:2253:19: error: implicit declaration of function 'iommu_support_dirty_log' [-Werror=implicit-function-declaration]
2253 | bool new_hwdbm = iommu_support_dirty_log(domain->domain);
| ^~~~~~~~~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/iommu_support_dirty_log +2253 drivers/vfio/vfio_iommu_type1.c
2242
2243 /*
2244 * Called after a new group is added to the iommu_domain, or an old group is
2245 * removed from the iommu_domain. Update the HWDBM status of vfio_domain and
2246 * vfio_iommu.
2247 */
2248 static void vfio_iommu_update_hwdbm(struct vfio_iommu *iommu,
2249 struct vfio_domain *domain,
2250 bool attach)
2251 {
2252 bool old_hwdbm = domain->iommu_hwdbm;
> 2253 bool new_hwdbm = iommu_support_dirty_log(domain->domain);
2254
2255 if (old_hwdbm && !new_hwdbm && attach) {
2256 iommu->num_non_hwdbm_domains++;
2257 } else if (!old_hwdbm && new_hwdbm && !attach) {
2258 iommu->num_non_hwdbm_domains--;
2259 }
2260 domain->iommu_hwdbm = new_hwdbm;
2261 }
2262
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 35137 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
2021-05-07 16:15 ` kernel test robot
@ 2021-05-07 18:25 ` kernel test robot
1 sibling, 0 replies; 7+ messages in thread
From: kernel test robot @ 2021-05-07 18:25 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2936 bytes --]
Hi Keqian,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on vfio/next]
[also build test ERROR on linux/master linus/master v5.12 next-20210507]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
base: https://github.com/awilliam/linux-vfio.git next
config: x86_64-randconfig-a014-20210507 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project a3a8a1a15b524d91b5308db68e9d293b34cd88dd)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/be47d8bac24aa1054b1c5719a2b6ac8928ea43e1
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
git checkout be47d8bac24aa1054b1c5719a2b6ac8928ea43e1
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> drivers/vfio/vfio_iommu_type1.c:2253:19: error: implicit declaration of function 'iommu_support_dirty_log' [-Werror,-Wimplicit-function-declaration]
bool new_hwdbm = iommu_support_dirty_log(domain->domain);
^
1 error generated.
vim +/iommu_support_dirty_log +2253 drivers/vfio/vfio_iommu_type1.c
2242
2243 /*
2244 * Called after a new group is added to the iommu_domain, or an old group is
2245 * removed from the iommu_domain. Update the HWDBM status of vfio_domain and
2246 * vfio_iommu.
2247 */
2248 static void vfio_iommu_update_hwdbm(struct vfio_iommu *iommu,
2249 struct vfio_domain *domain,
2250 bool attach)
2251 {
2252 bool old_hwdbm = domain->iommu_hwdbm;
> 2253 bool new_hwdbm = iommu_support_dirty_log(domain->domain);
2254
2255 if (old_hwdbm && !new_hwdbm && attach) {
2256 iommu->num_non_hwdbm_domains++;
2257 } else if (!old_hwdbm && new_hwdbm && !attach) {
2258 iommu->num_non_hwdbm_domains--;
2259 }
2260 domain->iommu_hwdbm = new_hwdbm;
2261 }
2262
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 33077 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM
2021-05-07 10:36 ` [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM Keqian Zhu
@ 2021-05-08 1:03 ` kernel test robot
0 siblings, 0 replies; 7+ messages in thread
From: kernel test robot @ 2021-05-08 1:03 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 9303 bytes --]
Hi Keqian,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on vfio/next]
[also build test ERROR on linux/master linus/master v5.12 next-20210507]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]
url: https://github.com/0day-ci/linux/commits/Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
base: https://github.com/awilliam/linux-vfio.git next
config: x86_64-randconfig-a006-20210506 (attached as .config)
compiler: clang version 13.0.0 (https://github.com/llvm/llvm-project a3a8a1a15b524d91b5308db68e9d293b34cd88dd)
reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# install x86_64 cross compiling tool for clang build
# apt-get install binutils-x86-64-linux-gnu
# https://github.com/0day-ci/linux/commit/53385033942861d447f335f3b605adef5f2ec4cd
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Keqian-Zhu/vfio-iommu_type1-Implement-dirty-log-tracking-based-on-IOMMU-HWDBM/20210507-183832
git checkout 53385033942861d447f335f3b605adef5f2ec4cd
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross W=1 ARCH=x86_64
If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
>> drivers/vfio/vfio_iommu_type1.c:1213:9: error: implicit declaration of function 'iommu_clear_dirty_log' [-Werror,-Wimplicit-function-declaration]
ret = iommu_clear_dirty_log(d->domain, start_iova, size,
^
>> drivers/vfio/vfio_iommu_type1.c:1232:9: error: implicit declaration of function 'iommu_sync_dirty_log' [-Werror,-Wimplicit-function-declaration]
ret = iommu_sync_dirty_log(d->domain, dma->iova, dma->size,
^
>> drivers/vfio/vfio_iommu_type1.c:1372:11: error: implicit declaration of function 'iommu_switch_dirty_log' [-Werror,-Wimplicit-function-declaration]
WARN_ON(iommu_switch_dirty_log(d->domain, enable, dma->iova,
^
drivers/vfio/vfio_iommu_type1.c:2344:11: error: implicit declaration of function 'iommu_switch_dirty_log' [-Werror,-Wimplicit-function-declaration]
WARN_ON(iommu_switch_dirty_log(d->domain, enable, dma->iova,
^
drivers/vfio/vfio_iommu_type1.c:2359:19: error: implicit declaration of function 'iommu_support_dirty_log' [-Werror,-Wimplicit-function-declaration]
bool new_hwdbm = iommu_support_dirty_log(domain->domain);
^
5 errors generated.
vim +/iommu_clear_dirty_log +1213 drivers/vfio/vfio_iommu_type1.c
1202
1203 static int vfio_iommu_dirty_log_clear(struct vfio_iommu *iommu,
1204 dma_addr_t start_iova, size_t size,
1205 unsigned long *bitmap_buffer,
1206 dma_addr_t base_iova,
1207 unsigned long pgshift)
1208 {
1209 struct vfio_domain *d;
1210 int ret = 0;
1211
1212 list_for_each_entry(d, &iommu->domain_list, next) {
> 1213 ret = iommu_clear_dirty_log(d->domain, start_iova, size,
1214 bitmap_buffer, base_iova, pgshift);
1215 if (ret) {
1216 pr_warn("vfio_iommu dirty log clear failed!\n");
1217 break;
1218 }
1219 }
1220
1221 return ret;
1222 }
1223
1224 static int vfio_iommu_dirty_log_sync(struct vfio_iommu *iommu,
1225 struct vfio_dma *dma,
1226 unsigned long pgshift)
1227 {
1228 struct vfio_domain *d;
1229 int ret = 0;
1230
1231 list_for_each_entry(d, &iommu->domain_list, next) {
> 1232 ret = iommu_sync_dirty_log(d->domain, dma->iova, dma->size,
1233 dma->bitmap, dma->iova, pgshift);
1234 if (ret) {
1235 pr_warn("vfio_iommu dirty log sync failed!\n");
1236 break;
1237 }
1238 }
1239
1240 return ret;
1241 }
1242
1243 static int update_user_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
1244 struct vfio_dma *dma, dma_addr_t base_iova,
1245 size_t pgsize)
1246 {
1247 unsigned long pgshift = __ffs(pgsize);
1248 unsigned long nbits = dma->size >> pgshift;
1249 unsigned long bit_offset = (dma->iova - base_iova) >> pgshift;
1250 unsigned long copy_offset = bit_offset / BITS_PER_LONG;
1251 unsigned long shift = bit_offset % BITS_PER_LONG;
1252 unsigned long leftover;
1253 bool iommu_hwdbm_dirty = false;
1254 int ret;
1255
1256 if (!iommu->num_non_pinned_groups || !dma->iommu_mapped) {
1257 /* nothing to do */
1258 } else if (!iommu->num_non_hwdbm_domains) {
1259 /* try to get dirty log from IOMMU */
1260 iommu_hwdbm_dirty = true;
1261 ret = vfio_iommu_dirty_log_sync(iommu, dma, pgshift);
1262 if (ret)
1263 return ret;
1264 } else {
1265 /*
1266 * mark all pages dirty if any IOMMU capable device is not able
1267 * to report dirty pages and all pages are pinned and mapped.
1268 */
1269 bitmap_set(dma->bitmap, 0, nbits);
1270 }
1271
1272 if (shift) {
1273 bitmap_shift_left(dma->bitmap, dma->bitmap, shift,
1274 nbits + shift);
1275
1276 if (copy_from_user(&leftover,
1277 (void __user *)(bitmap + copy_offset),
1278 sizeof(leftover)))
1279 return -EFAULT;
1280
1281 bitmap_or(dma->bitmap, dma->bitmap, &leftover, shift);
1282 }
1283
1284 if (copy_to_user((void __user *)(bitmap + copy_offset), dma->bitmap,
1285 DIRTY_BITMAP_BYTES(nbits + shift)))
1286 return -EFAULT;
1287
1288 /* Recover the bitmap if it'll be used to clear hardware dirty log */
1289 if (shift && iommu_hwdbm_dirty)
1290 bitmap_shift_right(dma->bitmap, dma->bitmap, shift,
1291 nbits + shift);
1292
1293 return 0;
1294 }
1295
1296 static int vfio_iova_dirty_bitmap(u64 __user *bitmap, struct vfio_iommu *iommu,
1297 dma_addr_t iova, size_t size, size_t pgsize)
1298 {
1299 struct vfio_dma *dma;
1300 struct rb_node *n;
1301 unsigned long pgshift = __ffs(pgsize);
1302 int ret;
1303
1304 /*
1305 * GET_BITMAP request must fully cover vfio_dma mappings. Multiple
1306 * vfio_dma mappings may be clubbed by specifying large ranges, but
1307 * there must not be any previous mappings bisected by the range.
1308 * An error will be returned if these conditions are not met.
1309 */
1310 dma = vfio_find_dma(iommu, iova, 1);
1311 if (dma && dma->iova != iova)
1312 return -EINVAL;
1313
1314 dma = vfio_find_dma(iommu, iova + size - 1, 0);
1315 if (dma && dma->iova + dma->size != iova + size)
1316 return -EINVAL;
1317
1318 for (n = rb_first(&iommu->dma_list); n; n = rb_next(n)) {
1319 struct vfio_dma *dma = rb_entry(n, struct vfio_dma, node);
1320
1321 if (dma->iova < iova)
1322 continue;
1323
1324 if (dma->iova > iova + size - 1)
1325 break;
1326
1327 ret = update_user_bitmap(bitmap, iommu, dma, iova, pgsize);
1328 if (ret)
1329 return ret;
1330
1331 /* Clear iommu dirty log to re-enable dirty log tracking */
1332 if (iommu->num_non_pinned_groups && dma->iommu_mapped &&
1333 !iommu->num_non_hwdbm_domains) {
1334 ret = vfio_iommu_dirty_log_clear(iommu, dma->iova,
1335 dma->size, dma->bitmap, dma->iova,
1336 pgshift);
1337 if (ret)
1338 return ret;
1339 }
1340
1341 /*
1342 * Re-populate bitmap to include all pinned pages which are
1343 * considered as dirty but exclude pages which are unpinned and
1344 * pages which are marked dirty by vfio_dma_rw()
1345 */
1346 bitmap_clear(dma->bitmap, 0, dma->size >> pgshift);
1347 vfio_dma_populate_bitmap(dma, pgsize);
1348 }
1349 return 0;
1350 }
1351
1352 static int verify_bitmap_size(uint64_t npages, uint64_t bitmap_size)
1353 {
1354 if (!npages || !bitmap_size || (bitmap_size > DIRTY_BITMAP_SIZE_MAX) ||
1355 (bitmap_size < DIRTY_BITMAP_BYTES(npages)))
1356 return -EINVAL;
1357
1358 return 0;
1359 }
1360
1361 static void vfio_dma_dirty_log_switch(struct vfio_iommu *iommu,
1362 struct vfio_dma *dma, bool enable)
1363 {
1364 struct vfio_domain *d;
1365
1366 if (!dma->iommu_mapped)
1367 return;
1368
1369 list_for_each_entry(d, &iommu->domain_list, next) {
1370 if (!d->iommu_hwdbm)
1371 continue;
> 1372 WARN_ON(iommu_switch_dirty_log(d->domain, enable, dma->iova,
1373 dma->size, d->prot | dma->prot));
1374 }
1375 }
1376
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 36507 bytes --]
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-05-08 1:03 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-07 10:36 [RFC PATCH v2 0/3] vfio/iommu_type1: Implement dirty log tracking based on IOMMU HWDBM Keqian Zhu
2021-05-07 10:36 ` [RFC PATCH v2 1/3] vfio/iommu_type1: Add HWDBM status maintenance Keqian Zhu
2021-05-07 16:15 ` kernel test robot
2021-05-07 18:25 ` kernel test robot
2021-05-07 10:36 ` [RFC PATCH v2 2/3] vfio/iommu_type1: Optimize dirty bitmap population based on iommu HWDBM Keqian Zhu
2021-05-08 1:03 ` kernel test robot
2021-05-07 10:36 ` [RFC PATCH v2 3/3] vfio/iommu_type1: Add support for manual dirty log clear Keqian Zhu
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.