* [Intel-gfx] [PATCH v3 00/12] drm/i915: Move LMEM (VRAM) management over to TTM
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
This is an initial patch series to move discrete memory management over to
TTM. It will be followed up shortly with adding more functionality.
The buddy allocator is temporarily removed along with its selftests and
It is replaced with the TTM range manager and some selftests are adjusted
to account for introduced fragmentation. Work is ongoing to reintroduce the
buddy allocator as a TTM resource manager.
A new memcpy ttm move is introduced that uses kmap_local() functionality
rather than vmap(). Among other things stated in the patch commit message
it helps us deal with page-pased LMEM memory. It is generic enough to replace
the ttm memcpy move with some additional work if so desired. On x86 it also
enables prefetching reads from write-combined memory.
Finally the old i915 gem object LMEM backend is replaced with a
i915 gem object TTM backend and some additional i915 gem object ops are
introduced to support the added functionality.
Currently it is used only to support management and eviction of the LMEM
region, but work is underway to extend the support to system memory. In this
way we use TTM the way it was originally intended, having the GPU binding
taken care of by driver code.
Intention is to follow up with
- System memory support
- Pipelined accelerated moves / migration
- Re-added buddy allocator in the TTM framework
v2:
- Add patches to move pagefaulting over to TTM
- Break out TTM changes to separate patches
- Address various review comments as detailed in the affected patches
v3:
- Drop TTM pagefaulting patches for now due changing approach due to a NAK.
- Address feedback on TTM patches
- Move the new TTM memcpy functionality into TTM.
- Move fast WC memcpy to drm
- Various fixes all over the place as shown in patch commit messages.
Cc: Christian König <christian.koenig@amd.com>
Thomas Hellström (12):
drm/i915: Untangle the vma pages_mutex
drm/i915: Don't free shared locks while shared
drm/i915: Fix i915_sg_page_sizes to record dma segments rather than
physical pages
drm/i915/ttm Initialize the ttm device and memory managers
drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
drm/ttm: Add a generic TTM memcpy move for page-based iomem
drm, drm/i915: Move the memcpy_from_wc functionality to core drm
drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
drm/ttm, drm/amdgpu: Allow the driver some control over swapping
drm/i915/ttm: Introduce a TTM i915 gem object backend
drm/i915/lmem: Verify checks for lmem residency
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +
drivers/gpu/drm/drm_drv.c | 2 +
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 +-
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 4 +-
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 71 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_object.c | 154 +++-
drivers/gpu/drm/i915/gem/i915_gem_object.h | 13 +
.../gpu/drm/i915/gem/i915_gem_object_types.h | 49 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 126 +--
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 531 ++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 50 ++
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +-
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 30 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 +-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 15 +-
drivers/gpu/drm/i915/i915_drv.h | 7 +-
drivers/gpu/drm/i915/i915_gem.c | 6 +-
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +-
drivers/gpu/drm/i915/i915_memcpy.h | 34 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 20 +-
drivers/gpu/drm/i915/i915_vma.c | 29 +-
drivers/gpu/drm/i915/intel_memory_region.c | 181 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 45 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 332 ++++++++
drivers/gpu/drm/i915/intel_region_ttm.h | 38 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
.../drm/i915/selftests/intel_memory_region.c | 140 +---
drivers/gpu/drm/i915/selftests/mock_region.c | 52 +-
drivers/gpu/drm/ttm/ttm_bo.c | 63 +-
drivers/gpu/drm/ttm/ttm_bo_util.c | 337 ++++----
drivers/gpu/drm/ttm/ttm_module.c | 35 +
drivers/gpu/drm/ttm/ttm_resource.c | 166 ++++
drivers/gpu/drm/ttm/ttm_tt.c | 50 ++
include/drm/drm_memcpy.h | 47 ++
include/drm/ttm/ttm_bo_driver.h | 28 +
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 ++
include/drm/ttm/ttm_resource.h | 61 ++
include/drm/ttm/ttm_tt.h | 26 +
65 files changed, 2322 insertions(+), 2163 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
create mode 100644 include/drm/drm_memcpy.h
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* [PATCH v3 01/12] drm/i915: Untangle the vma pages_mutex
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Any sleeping dma_resv lock taken while the vma pages_mutex is held
will cause a lockdep splat.
Move the i915_gem_object_pin_pages() call out of the pages_mutex
critical section.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/i915_vma.c | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index a6cd0fa62847..f2b5912fc542 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -800,32 +800,37 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
static int vma_get_pages(struct i915_vma *vma)
{
int err = 0;
+ bool pinned_pages = false;
if (atomic_add_unless(&vma->pages_count, 1, 0))
return 0;
+ if (vma->obj) {
+ err = i915_gem_object_pin_pages(vma->obj);
+ if (err)
+ return err;
+ pinned_pages = true;
+ }
+
/* Allocations ahoy! */
- if (mutex_lock_interruptible(&vma->pages_mutex))
- return -EINTR;
+ if (mutex_lock_interruptible(&vma->pages_mutex)) {
+ err = -EINTR;
+ goto unpin;
+ }
if (!atomic_read(&vma->pages_count)) {
- if (vma->obj) {
- err = i915_gem_object_pin_pages(vma->obj);
- if (err)
- goto unlock;
- }
-
err = vma->ops->set_pages(vma);
- if (err) {
- if (vma->obj)
- i915_gem_object_unpin_pages(vma->obj);
+ if (err)
goto unlock;
- }
+ pinned_pages = false;
}
atomic_inc(&vma->pages_count);
unlock:
mutex_unlock(&vma->pages_mutex);
+unpin:
+ if (pinned_pages)
+ __i915_gem_object_unpin_pages(vma->obj);
return err;
}
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 01/12] drm/i915: Untangle the vma pages_mutex
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Any sleeping dma_resv lock taken while the vma pages_mutex is held
will cause a lockdep splat.
Move the i915_gem_object_pin_pages() call out of the pages_mutex
critical section.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/i915_vma.c | 29 +++++++++++++++++------------
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index a6cd0fa62847..f2b5912fc542 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -800,32 +800,37 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
static int vma_get_pages(struct i915_vma *vma)
{
int err = 0;
+ bool pinned_pages = false;
if (atomic_add_unless(&vma->pages_count, 1, 0))
return 0;
+ if (vma->obj) {
+ err = i915_gem_object_pin_pages(vma->obj);
+ if (err)
+ return err;
+ pinned_pages = true;
+ }
+
/* Allocations ahoy! */
- if (mutex_lock_interruptible(&vma->pages_mutex))
- return -EINTR;
+ if (mutex_lock_interruptible(&vma->pages_mutex)) {
+ err = -EINTR;
+ goto unpin;
+ }
if (!atomic_read(&vma->pages_count)) {
- if (vma->obj) {
- err = i915_gem_object_pin_pages(vma->obj);
- if (err)
- goto unlock;
- }
-
err = vma->ops->set_pages(vma);
- if (err) {
- if (vma->obj)
- i915_gem_object_unpin_pages(vma->obj);
+ if (err)
goto unlock;
- }
+ pinned_pages = false;
}
atomic_inc(&vma->pages_count);
unlock:
mutex_unlock(&vma->pages_mutex);
+unpin:
+ if (pinned_pages)
+ __i915_gem_object_unpin_pages(vma->obj);
return err;
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 02/12] drm/i915: Don't free shared locks while shared
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
We are currently sharing the VM reservation locks across a number of
gem objects with page-table memory. Since TTM will individiualize the
reservation locks when freeing objects, including accessing the shared
locks, make sure that the shared locks are not freed until that is done.
For PPGTT we add an additional refcount, for GGTT we take additional
measures to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
v2: Try harder to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown.
v3: Use a pointer to the vm to indicate that an object shares a reservation
object from that vm, rather than a pointer to the reservation object itself.
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 3 ++
.../gpu/drm/i915/gem/i915_gem_object_types.h | 4 ++
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 ++++++--
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +++++++++++++++----
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +++++++++++-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/i915_drv.c | 5 +++
7 files changed, 93 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 28144410df86..2be6109d0093 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -252,6 +252,9 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915,
if (obj->mm.n_placements > 1)
kfree(obj->mm.placements);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
cond_resched();
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0727d0c76aa0..0415f99b6b95 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -149,6 +149,10 @@ struct drm_i915_gem_object {
* when i915_gem_ww_ctx_backoff() or i915_gem_ww_ctx_fini() are called.
*/
struct list_head obj_link;
+ /**
+ * @shared_resv_from: The object shares the resv from this vm.
+ */
+ struct i915_address_space *shares_resv_from;
union {
struct rcu_head rcu;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index 35069ca5d7de..10c23a749a95 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -746,7 +746,6 @@ static void ggtt_cleanup_hw(struct i915_ggtt *ggtt)
mutex_unlock(&ggtt->vm.mutex);
i915_address_space_fini(&ggtt->vm);
- dma_resv_fini(&ggtt->vm.resv);
arch_phys_wc_del(ggtt->mtrr);
@@ -768,6 +767,19 @@ void i915_ggtt_driver_release(struct drm_i915_private *i915)
ggtt_cleanup_hw(ggtt);
}
+/**
+ * i915_ggtt_driver_late_release - Cleanup of GGTT that needs to be done after
+ * all free objects have been drained.
+ * @i915: i915 device
+ */
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915)
+{
+ struct i915_ggtt *ggtt = &i915->ggtt;
+
+ GEM_WARN_ON(kref_read(&ggtt->vm.resv_ref) != 1);
+ dma_resv_fini(&ggtt->vm._resv);
+}
+
static unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
{
snb_gmch_ctl >>= SNB_GMCH_GGMS_SHIFT;
@@ -829,6 +841,7 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
return -ENOMEM;
}
+ kref_init(&ggtt->vm.resv_ref);
ret = setup_scratch_page(&ggtt->vm);
if (ret) {
drm_err(&i915->drm, "Scratch setup failed\n");
@@ -1135,7 +1148,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
ggtt->vm.gt = gt;
ggtt->vm.i915 = i915;
ggtt->vm.dma = i915->drm.dev;
- dma_resv_init(&ggtt->vm.resv);
+ dma_resv_init(&ggtt->vm._resv);
if (INTEL_GEN(i915) <= 5)
ret = i915_gmch_probe(ggtt);
@@ -1144,7 +1157,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
else
ret = gen8_gmch_probe(ggtt);
if (ret) {
- dma_resv_fini(&ggtt->vm.resv);
+ dma_resv_fini(&ggtt->vm._resv);
return ret;
}
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index 9b98f9d9faa3..94849567143d 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -22,8 +22,11 @@ struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -40,8 +43,11 @@ struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -102,7 +108,7 @@ void __i915_vm_close(struct i915_address_space *vm)
int i915_vm_lock_objects(struct i915_address_space *vm,
struct i915_gem_ww_ctx *ww)
{
- if (vm->scratch[0]->base.resv == &vm->resv) {
+ if (vm->scratch[0]->base.resv == &vm->_resv) {
return i915_gem_object_lock(vm->scratch[0], ww);
} else {
struct i915_ppgtt *ppgtt = i915_vm_to_ppgtt(vm);
@@ -118,6 +124,22 @@ void i915_address_space_fini(struct i915_address_space *vm)
mutex_destroy(&vm->mutex);
}
+/**
+ * i915_vm_resv_release - Final struct i915_address_space destructor
+ * @kref: Pointer to the &i915_address_space.resv_ref member.
+ *
+ * This function is called when the last lock sharer no longer shares the
+ * &i915_address_space._resv lock.
+ */
+void i915_vm_resv_release(struct kref *kref)
+{
+ struct i915_address_space *vm =
+ container_of(kref, typeof(*vm), resv_ref);
+
+ dma_resv_fini(&vm->_resv);
+ kfree(vm);
+}
+
static void __i915_vm_release(struct work_struct *work)
{
struct i915_address_space *vm =
@@ -125,9 +147,8 @@ static void __i915_vm_release(struct work_struct *work)
vm->cleanup(vm);
i915_address_space_fini(vm);
- dma_resv_fini(&vm->resv);
- kfree(vm);
+ i915_vm_resv_put(vm);
}
void i915_vm_release(struct kref *kref)
@@ -144,6 +165,14 @@ void i915_vm_release(struct kref *kref)
void i915_address_space_init(struct i915_address_space *vm, int subclass)
{
kref_init(&vm->ref);
+
+ /*
+ * Special case for GGTT that has already done an early
+ * kref_init here.
+ */
+ if (!kref_read(&vm->resv_ref))
+ kref_init(&vm->resv_ref);
+
INIT_RCU_WORK(&vm->rcu, __i915_vm_release);
atomic_set(&vm->open, 1);
@@ -170,7 +199,7 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass)
might_alloc(GFP_KERNEL);
mutex_release(&vm->mutex.dep_map, _THIS_IP_);
}
- dma_resv_init(&vm->resv);
+ dma_resv_init(&vm->_resv);
GEM_BUG_ON(!vm->total);
drm_mm_init(&vm->mm, 0, vm->total);
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h
index ca00b45827b7..f39be66e84f6 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.h
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.h
@@ -245,7 +245,9 @@ struct i915_address_space {
atomic_t open;
struct mutex mutex; /* protects vma and our lists */
- struct dma_resv resv; /* reservation lock for all pd objects, and buffer pool */
+
+ struct kref resv_ref; /* kref to keep the reservation lock alive. */
+ struct dma_resv _resv; /* reservation lock for all pd objects, and buffer pool */
#define VM_CLASS_GGTT 0
#define VM_CLASS_PPGTT 1
#define VM_CLASS_DPT 2
@@ -404,13 +406,36 @@ i915_vm_get(struct i915_address_space *vm)
return vm;
}
+/**
+ * i915_vm_resv_get - Obtain a reference on the vm's reservation lock
+ * @vm: The vm whose reservation lock we want to share.
+ *
+ * Return: A pointer to the vm's reservation lock.
+ */
+static inline struct dma_resv *i915_vm_resv_get(struct i915_address_space *vm)
+{
+ kref_get(&vm->resv_ref);
+ return &vm->_resv;
+}
+
void i915_vm_release(struct kref *kref);
+void i915_vm_resv_release(struct kref *kref);
+
static inline void i915_vm_put(struct i915_address_space *vm)
{
kref_put(&vm->ref, i915_vm_release);
}
+/**
+ * i915_vm_resv_put - Release a reference on the vm's reservation lock
+ * @resv: Pointer to a reservation lock obtained from i915_vm_resv_get()
+ */
+static inline void i915_vm_resv_put(struct i915_address_space *vm)
+{
+ kref_put(&vm->resv_ref, i915_vm_resv_release);
+}
+
static inline struct i915_address_space *
i915_vm_open(struct i915_address_space *vm)
{
@@ -506,6 +531,7 @@ void i915_ggtt_enable_guc(struct i915_ggtt *ggtt);
void i915_ggtt_disable_guc(struct i915_ggtt *ggtt);
int i915_init_ggtt(struct drm_i915_private *i915);
void i915_ggtt_driver_release(struct drm_i915_private *i915);
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915);
static inline bool i915_ggtt_has_aperture(const struct i915_ggtt *ggtt)
{
diff --git a/drivers/gpu/drm/i915/gt/intel_ppgtt.c b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
index 4e3d80c2295c..aee3a8929245 100644
--- a/drivers/gpu/drm/i915/gt/intel_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
@@ -307,7 +307,7 @@ void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
ppgtt->vm.dma = i915->drm.dev;
ppgtt->vm.total = BIT_ULL(INTEL_INFO(i915)->ppgtt_size);
- dma_resv_init(&ppgtt->vm.resv);
+ dma_resv_init(&ppgtt->vm._resv);
i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 5118dc8386b2..92bccc5623a8 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -631,6 +631,8 @@ static int i915_driver_hw_probe(struct drm_i915_private *dev_priv)
intel_memory_regions_driver_release(dev_priv);
err_ggtt:
i915_ggtt_driver_release(dev_priv);
+ i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
err_perf:
i915_perf_fini(dev_priv);
return ret;
@@ -880,6 +882,8 @@ int i915_driver_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
i915_driver_hw_remove(i915);
intel_memory_regions_driver_release(i915);
i915_ggtt_driver_release(i915);
+ i915_gem_drain_freed_objects(i915);
+ i915_ggtt_driver_late_release(i915);
out_cleanup_mmio:
i915_driver_mmio_release(i915);
out_runtime_pm_put:
@@ -936,6 +940,7 @@ static void i915_driver_release(struct drm_device *dev)
intel_memory_regions_driver_release(dev_priv);
i915_ggtt_driver_release(dev_priv);
i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
i915_driver_mmio_release(dev_priv);
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 02/12] drm/i915: Don't free shared locks while shared
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
We are currently sharing the VM reservation locks across a number of
gem objects with page-table memory. Since TTM will individiualize the
reservation locks when freeing objects, including accessing the shared
locks, make sure that the shared locks are not freed until that is done.
For PPGTT we add an additional refcount, for GGTT we take additional
measures to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
v2: Try harder to make sure objects sharing the GGTT reservation lock are
freed at GGTT takedown.
v3: Use a pointer to the vm to indicate that an object shares a reservation
object from that vm, rather than a pointer to the reservation object itself.
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 3 ++
.../gpu/drm/i915/gem/i915_gem_object_types.h | 4 ++
drivers/gpu/drm/i915/gt/intel_ggtt.c | 19 ++++++--
drivers/gpu/drm/i915/gt/intel_gtt.c | 45 +++++++++++++++----
drivers/gpu/drm/i915/gt/intel_gtt.h | 28 +++++++++++-
drivers/gpu/drm/i915/gt/intel_ppgtt.c | 2 +-
drivers/gpu/drm/i915/i915_drv.c | 5 +++
7 files changed, 93 insertions(+), 13 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 28144410df86..2be6109d0093 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -252,6 +252,9 @@ static void __i915_gem_free_objects(struct drm_i915_private *i915,
if (obj->mm.n_placements > 1)
kfree(obj->mm.placements);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
cond_resched();
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0727d0c76aa0..0415f99b6b95 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -149,6 +149,10 @@ struct drm_i915_gem_object {
* when i915_gem_ww_ctx_backoff() or i915_gem_ww_ctx_fini() are called.
*/
struct list_head obj_link;
+ /**
+ * @shared_resv_from: The object shares the resv from this vm.
+ */
+ struct i915_address_space *shares_resv_from;
union {
struct rcu_head rcu;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index 35069ca5d7de..10c23a749a95 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -746,7 +746,6 @@ static void ggtt_cleanup_hw(struct i915_ggtt *ggtt)
mutex_unlock(&ggtt->vm.mutex);
i915_address_space_fini(&ggtt->vm);
- dma_resv_fini(&ggtt->vm.resv);
arch_phys_wc_del(ggtt->mtrr);
@@ -768,6 +767,19 @@ void i915_ggtt_driver_release(struct drm_i915_private *i915)
ggtt_cleanup_hw(ggtt);
}
+/**
+ * i915_ggtt_driver_late_release - Cleanup of GGTT that needs to be done after
+ * all free objects have been drained.
+ * @i915: i915 device
+ */
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915)
+{
+ struct i915_ggtt *ggtt = &i915->ggtt;
+
+ GEM_WARN_ON(kref_read(&ggtt->vm.resv_ref) != 1);
+ dma_resv_fini(&ggtt->vm._resv);
+}
+
static unsigned int gen6_get_total_gtt_size(u16 snb_gmch_ctl)
{
snb_gmch_ctl >>= SNB_GMCH_GGMS_SHIFT;
@@ -829,6 +841,7 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
return -ENOMEM;
}
+ kref_init(&ggtt->vm.resv_ref);
ret = setup_scratch_page(&ggtt->vm);
if (ret) {
drm_err(&i915->drm, "Scratch setup failed\n");
@@ -1135,7 +1148,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
ggtt->vm.gt = gt;
ggtt->vm.i915 = i915;
ggtt->vm.dma = i915->drm.dev;
- dma_resv_init(&ggtt->vm.resv);
+ dma_resv_init(&ggtt->vm._resv);
if (INTEL_GEN(i915) <= 5)
ret = i915_gmch_probe(ggtt);
@@ -1144,7 +1157,7 @@ static int ggtt_probe_hw(struct i915_ggtt *ggtt, struct intel_gt *gt)
else
ret = gen8_gmch_probe(ggtt);
if (ret) {
- dma_resv_fini(&ggtt->vm.resv);
+ dma_resv_fini(&ggtt->vm._resv);
return ret;
}
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index 9b98f9d9faa3..94849567143d 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -22,8 +22,11 @@ struct drm_i915_gem_object *alloc_pt_lmem(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -40,8 +43,11 @@ struct drm_i915_gem_object *alloc_pt_dma(struct i915_address_space *vm, int sz)
* object underneath, with the idea that one object_lock() will lock
* them all at once.
*/
- if (!IS_ERR(obj))
- obj->base.resv = &vm->resv;
+ if (!IS_ERR(obj)) {
+ obj->base.resv = i915_vm_resv_get(vm);
+ obj->shares_resv_from = vm;
+ }
+
return obj;
}
@@ -102,7 +108,7 @@ void __i915_vm_close(struct i915_address_space *vm)
int i915_vm_lock_objects(struct i915_address_space *vm,
struct i915_gem_ww_ctx *ww)
{
- if (vm->scratch[0]->base.resv == &vm->resv) {
+ if (vm->scratch[0]->base.resv == &vm->_resv) {
return i915_gem_object_lock(vm->scratch[0], ww);
} else {
struct i915_ppgtt *ppgtt = i915_vm_to_ppgtt(vm);
@@ -118,6 +124,22 @@ void i915_address_space_fini(struct i915_address_space *vm)
mutex_destroy(&vm->mutex);
}
+/**
+ * i915_vm_resv_release - Final struct i915_address_space destructor
+ * @kref: Pointer to the &i915_address_space.resv_ref member.
+ *
+ * This function is called when the last lock sharer no longer shares the
+ * &i915_address_space._resv lock.
+ */
+void i915_vm_resv_release(struct kref *kref)
+{
+ struct i915_address_space *vm =
+ container_of(kref, typeof(*vm), resv_ref);
+
+ dma_resv_fini(&vm->_resv);
+ kfree(vm);
+}
+
static void __i915_vm_release(struct work_struct *work)
{
struct i915_address_space *vm =
@@ -125,9 +147,8 @@ static void __i915_vm_release(struct work_struct *work)
vm->cleanup(vm);
i915_address_space_fini(vm);
- dma_resv_fini(&vm->resv);
- kfree(vm);
+ i915_vm_resv_put(vm);
}
void i915_vm_release(struct kref *kref)
@@ -144,6 +165,14 @@ void i915_vm_release(struct kref *kref)
void i915_address_space_init(struct i915_address_space *vm, int subclass)
{
kref_init(&vm->ref);
+
+ /*
+ * Special case for GGTT that has already done an early
+ * kref_init here.
+ */
+ if (!kref_read(&vm->resv_ref))
+ kref_init(&vm->resv_ref);
+
INIT_RCU_WORK(&vm->rcu, __i915_vm_release);
atomic_set(&vm->open, 1);
@@ -170,7 +199,7 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass)
might_alloc(GFP_KERNEL);
mutex_release(&vm->mutex.dep_map, _THIS_IP_);
}
- dma_resv_init(&vm->resv);
+ dma_resv_init(&vm->_resv);
GEM_BUG_ON(!vm->total);
drm_mm_init(&vm->mm, 0, vm->total);
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h
index ca00b45827b7..f39be66e84f6 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.h
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.h
@@ -245,7 +245,9 @@ struct i915_address_space {
atomic_t open;
struct mutex mutex; /* protects vma and our lists */
- struct dma_resv resv; /* reservation lock for all pd objects, and buffer pool */
+
+ struct kref resv_ref; /* kref to keep the reservation lock alive. */
+ struct dma_resv _resv; /* reservation lock for all pd objects, and buffer pool */
#define VM_CLASS_GGTT 0
#define VM_CLASS_PPGTT 1
#define VM_CLASS_DPT 2
@@ -404,13 +406,36 @@ i915_vm_get(struct i915_address_space *vm)
return vm;
}
+/**
+ * i915_vm_resv_get - Obtain a reference on the vm's reservation lock
+ * @vm: The vm whose reservation lock we want to share.
+ *
+ * Return: A pointer to the vm's reservation lock.
+ */
+static inline struct dma_resv *i915_vm_resv_get(struct i915_address_space *vm)
+{
+ kref_get(&vm->resv_ref);
+ return &vm->_resv;
+}
+
void i915_vm_release(struct kref *kref);
+void i915_vm_resv_release(struct kref *kref);
+
static inline void i915_vm_put(struct i915_address_space *vm)
{
kref_put(&vm->ref, i915_vm_release);
}
+/**
+ * i915_vm_resv_put - Release a reference on the vm's reservation lock
+ * @resv: Pointer to a reservation lock obtained from i915_vm_resv_get()
+ */
+static inline void i915_vm_resv_put(struct i915_address_space *vm)
+{
+ kref_put(&vm->resv_ref, i915_vm_resv_release);
+}
+
static inline struct i915_address_space *
i915_vm_open(struct i915_address_space *vm)
{
@@ -506,6 +531,7 @@ void i915_ggtt_enable_guc(struct i915_ggtt *ggtt);
void i915_ggtt_disable_guc(struct i915_ggtt *ggtt);
int i915_init_ggtt(struct drm_i915_private *i915);
void i915_ggtt_driver_release(struct drm_i915_private *i915);
+void i915_ggtt_driver_late_release(struct drm_i915_private *i915);
static inline bool i915_ggtt_has_aperture(const struct i915_ggtt *ggtt)
{
diff --git a/drivers/gpu/drm/i915/gt/intel_ppgtt.c b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
index 4e3d80c2295c..aee3a8929245 100644
--- a/drivers/gpu/drm/i915/gt/intel_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
@@ -307,7 +307,7 @@ void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
ppgtt->vm.dma = i915->drm.dev;
ppgtt->vm.total = BIT_ULL(INTEL_INFO(i915)->ppgtt_size);
- dma_resv_init(&ppgtt->vm.resv);
+ dma_resv_init(&ppgtt->vm._resv);
i915_address_space_init(&ppgtt->vm, VM_CLASS_PPGTT);
ppgtt->vm.vma_ops.bind_vma = ppgtt_bind_vma;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 5118dc8386b2..92bccc5623a8 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -631,6 +631,8 @@ static int i915_driver_hw_probe(struct drm_i915_private *dev_priv)
intel_memory_regions_driver_release(dev_priv);
err_ggtt:
i915_ggtt_driver_release(dev_priv);
+ i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
err_perf:
i915_perf_fini(dev_priv);
return ret;
@@ -880,6 +882,8 @@ int i915_driver_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
i915_driver_hw_remove(i915);
intel_memory_regions_driver_release(i915);
i915_ggtt_driver_release(i915);
+ i915_gem_drain_freed_objects(i915);
+ i915_ggtt_driver_late_release(i915);
out_cleanup_mmio:
i915_driver_mmio_release(i915);
out_runtime_pm_put:
@@ -936,6 +940,7 @@ static void i915_driver_release(struct drm_device *dev)
intel_memory_regions_driver_release(dev_priv);
i915_ggtt_driver_release(dev_priv);
i915_gem_drain_freed_objects(dev_priv);
+ i915_ggtt_driver_late_release(dev_priv);
i915_driver_mmio_release(dev_priv);
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 03/12] drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
All users of this function actually want the dma segment sizes, but that's
not what's calculated. Fix that and rename the function to
i915_sg_dma_sizes to reflect what's calculated.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
drivers/gpu/drm/i915/i915_scatterlist.h | 16 ++++++++++++----
4 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index ccede73c6465..616c3a2f1baf 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -209,7 +209,7 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
if (IS_ERR(pages))
return PTR_ERR(pages);
- sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
index 81dc2bf59bc3..36f373dc493c 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
@@ -208,7 +208,7 @@ static int i915_gem_object_shmem_to_phys(struct drm_i915_gem_object *obj)
err_xfer:
if (!IS_ERR_OR_NULL(pages)) {
- unsigned int sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ unsigned int sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
index a657b99ec760..602f0ed983ec 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
@@ -173,7 +173,7 @@ static int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
goto err;
}
- sg_page_sizes = i915_sg_page_sizes(st->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(st->sgl);
__i915_gem_object_set_pages(obj, st, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index 9cb26a224034..b96baad66a3a 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -101,15 +101,23 @@ static inline struct scatterlist *__sg_next(struct scatterlist *sg)
(((__iter).curr += PAGE_SIZE) >= (__iter).max) ? \
(__iter) = __sgt_iter(__sg_next((__iter).sgp), false), 0 : 0)
-static inline unsigned int i915_sg_page_sizes(struct scatterlist *sg)
+/**
+ * i915_sg_dma_sizes - Record the dma segment sizes of a scatterlist
+ * @sg: The scatterlist
+ *
+ * Return: An unsigned int with segment sizes logically or'ed together.
+ * A caller can use this information to determine what hardware page table
+ * entry sizes can be used to map the memory represented by the scatterlist.
+ */
+static inline unsigned int i915_sg_dma_sizes(struct scatterlist *sg)
{
unsigned int page_sizes;
page_sizes = 0;
- while (sg) {
+ while (sg && sg_dma_len(sg)) {
GEM_BUG_ON(sg->offset);
- GEM_BUG_ON(!IS_ALIGNED(sg->length, PAGE_SIZE));
- page_sizes |= sg->length;
+ GEM_BUG_ON(!IS_ALIGNED(sg_dma_len(sg), PAGE_SIZE));
+ page_sizes |= sg_dma_len(sg);
sg = __sg_next(sg);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 03/12] drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
All users of this function actually want the dma segment sizes, but that's
not what's calculated. Fix that and rename the function to
i915_sg_dma_sizes to reflect what's calculated.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_phys.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 2 +-
drivers/gpu/drm/i915/i915_scatterlist.h | 16 ++++++++++++----
4 files changed, 15 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
index ccede73c6465..616c3a2f1baf 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_dmabuf.c
@@ -209,7 +209,7 @@ static int i915_gem_object_get_pages_dmabuf(struct drm_i915_gem_object *obj)
if (IS_ERR(pages))
return PTR_ERR(pages);
- sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_phys.c b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
index 81dc2bf59bc3..36f373dc493c 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_phys.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_phys.c
@@ -208,7 +208,7 @@ static int i915_gem_object_shmem_to_phys(struct drm_i915_gem_object *obj)
err_xfer:
if (!IS_ERR_OR_NULL(pages)) {
- unsigned int sg_page_sizes = i915_sg_page_sizes(pages->sgl);
+ unsigned int sg_page_sizes = i915_sg_dma_sizes(pages->sgl);
__i915_gem_object_set_pages(obj, pages, sg_page_sizes);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
index a657b99ec760..602f0ed983ec 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_userptr.c
@@ -173,7 +173,7 @@ static int i915_gem_userptr_get_pages(struct drm_i915_gem_object *obj)
goto err;
}
- sg_page_sizes = i915_sg_page_sizes(st->sgl);
+ sg_page_sizes = i915_sg_dma_sizes(st->sgl);
__i915_gem_object_set_pages(obj, st, sg_page_sizes);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index 9cb26a224034..b96baad66a3a 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -101,15 +101,23 @@ static inline struct scatterlist *__sg_next(struct scatterlist *sg)
(((__iter).curr += PAGE_SIZE) >= (__iter).max) ? \
(__iter) = __sgt_iter(__sg_next((__iter).sgp), false), 0 : 0)
-static inline unsigned int i915_sg_page_sizes(struct scatterlist *sg)
+/**
+ * i915_sg_dma_sizes - Record the dma segment sizes of a scatterlist
+ * @sg: The scatterlist
+ *
+ * Return: An unsigned int with segment sizes logically or'ed together.
+ * A caller can use this information to determine what hardware page table
+ * entry sizes can be used to map the memory represented by the scatterlist.
+ */
+static inline unsigned int i915_sg_dma_sizes(struct scatterlist *sg)
{
unsigned int page_sizes;
page_sizes = 0;
- while (sg) {
+ while (sg && sg_dma_len(sg)) {
GEM_BUG_ON(sg->offset);
- GEM_BUG_ON(!IS_ALIGNED(sg->length, PAGE_SIZE));
- page_sizes |= sg->length;
+ GEM_BUG_ON(!IS_ALIGNED(sg_dma_len(sg), PAGE_SIZE));
+ page_sizes |= sg_dma_len(sg);
sg = __sg_next(sg);
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 04/12] drm/i915/ttm Initialize the ttm device and memory managers
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Temporarily remove the buddy allocator and related selftests
and hook up the TTM range manager for i915 regions.
Also modify the mock region selftests somewhat to account for a
fragmenting manager.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com> #v2
---
v2:
- Fix an error unwind in lmem_get_pages() (Reported by Matthew Auld)
- Break out and modify usage of i915_sg_dma_sizes() (Reported by Mattew Auld)
- Break out TTM changes to a separate patch (Reported by Christian König)
v3:
- Fix the same error unwind in mock_region_get_pages()
(Reported by Matthew Auld)
- Don't rely on new TTM functionality, but create a mock TTM device,
(Reported by Christian König)
---
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 59 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 120 ---
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 27 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_drv.c | 8 +
drivers/gpu/drm/i915/i915_drv.h | 7 +-
drivers/gpu/drm/i915/i915_gem.c | 1 +
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 4 +
drivers/gpu/drm/i915/intel_memory_region.c | 180 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 44 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 334 ++++++++
drivers/gpu/drm/i915/intel_region_ttm.h | 38 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
.../drm/i915/selftests/intel_memory_region.c | 133 +--
drivers/gpu/drm/i915/selftests/mock_region.c | 52 +-
29 files changed, 722 insertions(+), 1754 deletions(-)
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
diff --git a/drivers/gpu/drm/i915/Kconfig b/drivers/gpu/drm/i915/Kconfig
index 1e1cb245fca7..b63d374dff23 100644
--- a/drivers/gpu/drm/i915/Kconfig
+++ b/drivers/gpu/drm/i915/Kconfig
@@ -26,6 +26,7 @@ config DRM_I915
select SND_HDA_I915 if SND_HDA_CORE
select CEC_CORE if CEC_NOTIFIER
select VMAP_PFN
+ select DRM_TTM
help
Choose this option if you have a system that has "Intel Graphics
Media Accelerator" or "HD Graphics" integrated graphics,
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index d0d936d9137b..cb8823570996 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -50,6 +50,7 @@ i915-y += i915_drv.o \
intel_memory_region.o \
intel_pch.o \
intel_pm.o \
+ intel_region_ttm.o \
intel_runtime_pm.o \
intel_sideband.o \
intel_step.o \
@@ -160,7 +161,6 @@ gem-y += \
i915-y += \
$(gem-y) \
i915_active.o \
- i915_buddy.o \
i915_cmd_parser.o \
i915_gem_evict.o \
i915_gem_gtt.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index f44bdd08f7cb..3b4aa28a076d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,16 +4,71 @@
*/
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
+static void lmem_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ obj->mm.dirty = false;
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int lmem_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ /* Range manager is always contigous */
+ if (obj->mm.region->is_range_manager)
+ obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
+ void __iomem *vaddr =
+ i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
+
+ if (!vaddr) {
+ struct sg_table *pages =
+ __i915_gem_object_unset_pages(obj);
+
+ if (!IS_ERR_OR_NULL(pages))
+ lmem_put_pages(obj, pages);
+ }
+
+ memset_io(vaddr, 0, obj->base.size);
+ io_mapping_unmap(vaddr);
+ }
+
+ return 0;
+}
+
const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
.name = "i915_gem_object_lmem",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = lmem_get_pages,
+ .put_pages = lmem_put_pages,
.release = i915_gem_object_release_memory_region,
};
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0415f99b6b95..f5b46d11e6e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -235,10 +235,12 @@ struct drm_i915_gem_object {
* Memory region for this object.
*/
struct intel_memory_region *region;
+
/**
- * List of memory region blocks allocated for this object.
+ * Memory manager node allocated for this object.
*/
- struct list_head blocks;
+ void *st_mm_node;
+
/**
* Element within memory_region->objects or region->purgeable
* if the object is marked as DONTNEED. Access is protected by
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index aed8a37ccdc9..62ee2185a41b 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -473,7 +473,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
might_sleep();
GEM_BUG_ON(n >= obj->base.size >> PAGE_SHIFT);
- GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj));
+ if (!i915_gem_object_has_pinned_pages(obj))
+ assert_object_held(obj);
/* As we iterate forward through the sg, we record each entry in a
* radixtree for quick repeated (backwards) lookups. If we have seen
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index ce8fcfc54079..f25e6646c5b7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -8,129 +8,9 @@
#include "i915_drv.h"
#include "i915_trace.h"
-void
-i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- __intel_memory_region_put_pages_buddy(obj->mm.region, &obj->mm.blocks);
-
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-int
-i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
-{
- const u64 max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = obj->mm.region;
- struct list_head *blocks = &obj->mm.blocks;
- resource_size_t size = obj->base.size;
- resource_size_t prev_end;
- struct i915_buddy_block *block;
- unsigned int flags;
- struct sg_table *st;
- struct scatterlist *sg;
- unsigned int sg_page_sizes;
- int ret;
-
- st = kmalloc(sizeof(*st), GFP_KERNEL);
- if (!st)
- return -ENOMEM;
-
- if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
- kfree(st);
- return -ENOMEM;
- }
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, blocks);
- if (ret)
- goto err_free_sg;
-
- GEM_BUG_ON(list_empty(blocks));
-
- sg = st->sgl;
- st->nents = 0;
- sg_page_sizes = 0;
- prev_end = (resource_size_t)-1;
-
- list_for_each_entry(block, blocks, link) {
- u64 block_size, offset;
-
- block_size = min_t(u64, size,
- i915_buddy_block_size(&mem->mm, block));
- offset = i915_buddy_block_offset(block);
-
- while (block_size) {
- u64 len;
-
- if (offset != prev_end || sg->length >= max_segment) {
- if (st->nents) {
- sg_page_sizes |= sg->length;
- sg = __sg_next(sg);
- }
-
- sg_dma_address(sg) = mem->region.start + offset;
- sg_dma_len(sg) = 0;
- sg->length = 0;
- st->nents++;
- }
-
- len = min(block_size, max_segment - sg->length);
- sg->length += len;
- sg_dma_len(sg) += len;
-
- offset += len;
- block_size -= len;
-
- prev_end = offset;
- }
- }
-
- sg_page_sizes |= sg->length;
- sg_mark_end(sg);
- i915_sg_trim(st);
-
- /* Intended for kernel internal use only */
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- struct scatterlist *sg;
- unsigned long i;
-
- for_each_sg(st->sgl, sg, st->nents, i) {
- unsigned int length;
- void __iomem *vaddr;
- dma_addr_t daddr;
-
- daddr = sg_dma_address(sg);
- daddr -= mem->region.start;
- length = sg_dma_len(sg);
-
- vaddr = io_mapping_map_wc(&mem->iomap, daddr, length);
- memset64((void __force *)vaddr, 0, length / sizeof(u64));
- io_mapping_unmap(vaddr);
- }
-
- wmb();
- }
-
- __i915_gem_object_set_pages(obj, st, sg_page_sizes);
-
- return 0;
-
-err_free_sg:
- sg_free_table(st);
- kfree(st);
- return ret;
-}
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem)
{
- INIT_LIST_HEAD(&obj->mm.blocks);
obj->mm.region = intel_memory_region_get(mem);
if (obj->base.size <= mem->min_page_size)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.h b/drivers/gpu/drm/i915/gem/i915_gem_region.h
index ebddc86d78f7..84fcb3297400 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.h
@@ -12,10 +12,6 @@ struct intel_memory_region;
struct drm_i915_gem_object;
struct sg_table;
-int i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj);
-void i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages);
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem);
void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index a9bfa66c8da1..5d16c4462fda 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -628,11 +628,13 @@ static const struct intel_memory_region_ops shmem_region_ops = {
.init_object = shmem_object_init,
};
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915)
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance)
{
return intel_memory_region_create(i915, 0,
totalram_pages() << PAGE_SHIFT,
PAGE_SIZE, 0,
+ type, instance,
&shmem_region_ops);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
index 293f640faa0a..c42abee206da 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
@@ -770,7 +770,8 @@ static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = {
};
struct intel_memory_region *
-i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_uncore *uncore = &i915->uncore;
struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
@@ -788,6 +789,7 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
mem = intel_memory_region_create(i915, lmem_base, lmem_size,
I915_GTT_PAGE_SIZE_4K, io_start,
+ type, instance,
&i915_region_stolen_lmem_ops);
if (IS_ERR(mem))
return mem;
@@ -809,14 +811,15 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
}
struct intel_memory_region*
-i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_memory_region *mem;
mem = intel_memory_region_create(i915,
intel_graphics_stolen_res.start,
resource_size(&intel_graphics_stolen_res),
- PAGE_SIZE, 0,
+ PAGE_SIZE, 0, type, instance,
&i915_region_stolen_smem_ops);
if (IS_ERR(mem))
return mem;
@@ -824,7 +827,6 @@ i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
intel_memory_region_set_name(mem, "stolen-system");
mem->private = true;
-
return mem;
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
index 2bec6c367b9c..ccdf7befc571 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
@@ -21,8 +21,13 @@ int i915_gem_stolen_insert_node_in_range(struct drm_i915_private *dev_priv,
u64 end);
void i915_gem_stolen_remove_node(struct drm_i915_private *dev_priv,
struct drm_mm_node *node);
-struct intel_memory_region *i915_gem_stolen_smem_setup(struct drm_i915_private *i915);
-struct intel_memory_region *i915_gem_stolen_lmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+struct intel_memory_region *
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+
struct drm_i915_gem_object *
i915_gem_object_create_stolen(struct drm_i915_private *dev_priv,
resource_size_t size);
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 8d77dcbad059..3f88ecdee031 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -68,8 +68,6 @@ int intel_gt_probe_lmem(struct intel_gt *gt)
id = INTEL_REGION_LMEM;
mem->id = id;
- mem->type = INTEL_MEMORY_LOCAL;
- mem->instance = 0;
intel_memory_region_set_name(mem, "local%u", mem->instance);
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index 73fceb0c25fc..f7366b054f8e 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -5,6 +5,8 @@
#include "i915_drv.h"
#include "intel_memory_region.h"
+#include "intel_region_lmem.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "intel_region_lmem.h"
@@ -66,9 +68,9 @@ static void release_fake_lmem_bar(struct intel_memory_region *mem)
static void
region_lmem_release(struct intel_memory_region *mem)
{
- release_fake_lmem_bar(mem);
+ intel_region_ttm_fini(mem);
io_mapping_fini(&mem->iomap);
- intel_memory_region_release_buddy(mem);
+ release_fake_lmem_bar(mem);
}
static int
@@ -83,12 +85,21 @@ region_lmem_init(struct intel_memory_region *mem)
if (!io_mapping_init_wc(&mem->iomap,
mem->io_start,
- resource_size(&mem->region)))
- return -EIO;
+ resource_size(&mem->region))) {
+ ret = -EIO;
+ goto out_no_io;
+ }
- ret = intel_memory_region_init_buddy(mem);
+ ret = intel_region_ttm_init(mem);
if (ret)
- io_mapping_fini(&mem->iomap);
+ goto out_no_buddy;
+
+ return 0;
+
+out_no_buddy:
+ io_mapping_fini(&mem->iomap);
+out_no_io:
+ release_fake_lmem_bar(mem);
return ret;
}
@@ -127,6 +138,8 @@ intel_gt_setup_fake_lmem(struct intel_gt *gt)
mappable_end,
PAGE_SIZE,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (!IS_ERR(mem)) {
drm_info(&i915->drm, "Intel graphics fake LMEM: %pR\n",
@@ -198,6 +211,8 @@ static struct intel_memory_region *setup_lmem(struct intel_gt *gt)
lmem_size,
I915_GTT_PAGE_SIZE_4K,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (IS_ERR(mem))
return mem;
diff --git a/drivers/gpu/drm/i915/i915_buddy.c b/drivers/gpu/drm/i915/i915_buddy.c
deleted file mode 100644
index 3a2f6eecb2fc..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.c
+++ /dev/null
@@ -1,435 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/kmemleak.h>
-#include <linux/slab.h>
-
-#include "i915_buddy.h"
-
-#include "i915_gem.h"
-#include "i915_globals.h"
-#include "i915_utils.h"
-
-static struct i915_global_block {
- struct i915_global base;
- struct kmem_cache *slab_blocks;
-} global;
-
-static void i915_global_buddy_shrink(void)
-{
- kmem_cache_shrink(global.slab_blocks);
-}
-
-static void i915_global_buddy_exit(void)
-{
- kmem_cache_destroy(global.slab_blocks);
-}
-
-static struct i915_global_block global = { {
- .shrink = i915_global_buddy_shrink,
- .exit = i915_global_buddy_exit,
-} };
-
-int __init i915_global_buddy_init(void)
-{
- global.slab_blocks = KMEM_CACHE(i915_buddy_block, SLAB_HWCACHE_ALIGN);
- if (!global.slab_blocks)
- return -ENOMEM;
-
- i915_global_register(&global.base);
- return 0;
-}
-
-static struct i915_buddy_block *i915_block_alloc(struct i915_buddy_block *parent,
- unsigned int order,
- u64 offset)
-{
- struct i915_buddy_block *block;
-
- GEM_BUG_ON(order > I915_BUDDY_MAX_ORDER);
-
- block = kmem_cache_zalloc(global.slab_blocks, GFP_KERNEL);
- if (!block)
- return NULL;
-
- block->header = offset;
- block->header |= order;
- block->parent = parent;
-
- GEM_BUG_ON(block->header & I915_BUDDY_HEADER_UNUSED);
- return block;
-}
-
-static void i915_block_free(struct i915_buddy_block *block)
-{
- kmem_cache_free(global.slab_blocks, block);
-}
-
-static void mark_allocated(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_ALLOCATED;
-
- list_del(&block->link);
-}
-
-static void mark_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_FREE;
-
- list_add(&block->link,
- &mm->free_list[i915_buddy_block_order(block)]);
-}
-
-static void mark_split(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_SPLIT;
-
- list_del(&block->link);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size)
-{
- unsigned int i;
- u64 offset;
-
- if (size < chunk_size)
- return -EINVAL;
-
- if (chunk_size < PAGE_SIZE)
- return -EINVAL;
-
- if (!is_power_of_2(chunk_size))
- return -EINVAL;
-
- size = round_down(size, chunk_size);
-
- mm->size = size;
- mm->chunk_size = chunk_size;
- mm->max_order = ilog2(size) - ilog2(chunk_size);
-
- GEM_BUG_ON(mm->max_order > I915_BUDDY_MAX_ORDER);
-
- mm->free_list = kmalloc_array(mm->max_order + 1,
- sizeof(struct list_head),
- GFP_KERNEL);
- if (!mm->free_list)
- return -ENOMEM;
-
- for (i = 0; i <= mm->max_order; ++i)
- INIT_LIST_HEAD(&mm->free_list[i]);
-
- mm->n_roots = hweight64(size);
-
- mm->roots = kmalloc_array(mm->n_roots,
- sizeof(struct i915_buddy_block *),
- GFP_KERNEL);
- if (!mm->roots)
- goto out_free_list;
-
- offset = 0;
- i = 0;
-
- /*
- * Split into power-of-two blocks, in case we are given a size that is
- * not itself a power-of-two.
- */
- do {
- struct i915_buddy_block *root;
- unsigned int order;
- u64 root_size;
-
- root_size = rounddown_pow_of_two(size);
- order = ilog2(root_size) - ilog2(chunk_size);
-
- root = i915_block_alloc(NULL, order, offset);
- if (!root)
- goto out_free_roots;
-
- mark_free(mm, root);
-
- GEM_BUG_ON(i > mm->max_order);
- GEM_BUG_ON(i915_buddy_block_size(mm, root) < chunk_size);
-
- mm->roots[i] = root;
-
- offset += root_size;
- size -= root_size;
- i++;
- } while (size);
-
- return 0;
-
-out_free_roots:
- while (i--)
- i915_block_free(mm->roots[i]);
- kfree(mm->roots);
-out_free_list:
- kfree(mm->free_list);
- return -ENOMEM;
-}
-
-void i915_buddy_fini(struct i915_buddy_mm *mm)
-{
- int i;
-
- for (i = 0; i < mm->n_roots; ++i) {
- GEM_WARN_ON(!i915_buddy_block_is_free(mm->roots[i]));
- i915_block_free(mm->roots[i]);
- }
-
- kfree(mm->roots);
- kfree(mm->free_list);
-}
-
-static int split_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- unsigned int block_order = i915_buddy_block_order(block) - 1;
- u64 offset = i915_buddy_block_offset(block);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
- GEM_BUG_ON(!i915_buddy_block_order(block));
-
- block->left = i915_block_alloc(block, block_order, offset);
- if (!block->left)
- return -ENOMEM;
-
- block->right = i915_block_alloc(block, block_order,
- offset + (mm->chunk_size << block_order));
- if (!block->right) {
- i915_block_free(block->left);
- return -ENOMEM;
- }
-
- mark_free(mm, block->left);
- mark_free(mm, block->right);
-
- mark_split(block);
-
- return 0;
-}
-
-static struct i915_buddy_block *
-get_buddy(struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- parent = block->parent;
- if (!parent)
- return NULL;
-
- if (parent->left == block)
- return parent->right;
-
- return parent->left;
-}
-
-static void __i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- while ((parent = block->parent)) {
- struct i915_buddy_block *buddy;
-
- buddy = get_buddy(block);
-
- if (!i915_buddy_block_is_free(buddy))
- break;
-
- list_del(&buddy->link);
-
- i915_block_free(block);
- i915_block_free(buddy);
-
- block = parent;
- }
-
- mark_free(mm, block);
-}
-
-void i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- GEM_BUG_ON(!i915_buddy_block_is_allocated(block));
- __i915_buddy_free(mm, block);
-}
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects)
-{
- struct i915_buddy_block *block, *on;
-
- list_for_each_entry_safe(block, on, objects, link) {
- i915_buddy_free(mm, block);
- cond_resched();
- }
- INIT_LIST_HEAD(objects);
-}
-
-/*
- * Allocate power-of-two block. The order value here translates to:
- *
- * 0 = 2^0 * mm->chunk_size
- * 1 = 2^1 * mm->chunk_size
- * 2 = 2^2 * mm->chunk_size
- * ...
- */
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order)
-{
- struct i915_buddy_block *block = NULL;
- unsigned int i;
- int err;
-
- for (i = order; i <= mm->max_order; ++i) {
- block = list_first_entry_or_null(&mm->free_list[i],
- struct i915_buddy_block,
- link);
- if (block)
- break;
- }
-
- if (!block)
- return ERR_PTR(-ENOSPC);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
-
- while (i != order) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto out_free;
-
- /* Go low */
- block = block->left;
- i--;
- }
-
- mark_allocated(block);
- kmemleak_update_trace(block);
- return block;
-
-out_free:
- if (i != order)
- __i915_buddy_free(mm, block);
- return ERR_PTR(err);
-}
-
-static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= e2 && e1 >= s2;
-}
-
-static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= s2 && e1 >= e2;
-}
-
-/*
- * Allocate range. Note that it's safe to chain together multiple alloc_ranges
- * with the same blocks list.
- *
- * Intended for pre-allocating portions of the address space, for example to
- * reserve a block for the initial framebuffer or similar, hence the expectation
- * here is that i915_buddy_alloc() is still the main vehicle for
- * allocations, so if that's not the case then the drm_mm range allocator is
- * probably a much better fit, and so you should probably go use that instead.
- */
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *buddy;
- LIST_HEAD(allocated);
- LIST_HEAD(dfs);
- u64 end;
- int err;
- int i;
-
- if (size < mm->chunk_size)
- return -EINVAL;
-
- if (!IS_ALIGNED(size | start, mm->chunk_size))
- return -EINVAL;
-
- if (range_overflows(start, size, mm->size))
- return -EINVAL;
-
- for (i = 0; i < mm->n_roots; ++i)
- list_add_tail(&mm->roots[i]->tmp_link, &dfs);
-
- end = start + size - 1;
-
- do {
- u64 block_start;
- u64 block_end;
-
- block = list_first_entry_or_null(&dfs,
- struct i915_buddy_block,
- tmp_link);
- if (!block)
- break;
-
- list_del(&block->tmp_link);
-
- block_start = i915_buddy_block_offset(block);
- block_end = block_start + i915_buddy_block_size(mm, block) - 1;
-
- if (!overlaps(start, end, block_start, block_end))
- continue;
-
- if (i915_buddy_block_is_allocated(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- if (contains(start, end, block_start, block_end)) {
- if (!i915_buddy_block_is_free(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- mark_allocated(block);
- list_add_tail(&block->link, &allocated);
- continue;
- }
-
- if (!i915_buddy_block_is_split(block)) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto err_undo;
- }
-
- list_add(&block->right->tmp_link, &dfs);
- list_add(&block->left->tmp_link, &dfs);
- } while (1);
-
- list_splice_tail(&allocated, blocks);
- return 0;
-
-err_undo:
- /*
- * We really don't want to leave around a bunch of split blocks, since
- * bigger is better, so make sure we merge everything back before we
- * free the allocated blocks.
- */
- buddy = get_buddy(block);
- if (buddy &&
- (i915_buddy_block_is_free(block) &&
- i915_buddy_block_is_free(buddy)))
- __i915_buddy_free(mm, block);
-
-err_free:
- i915_buddy_free_list(mm, &allocated);
- return err;
-}
-
-#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
-#include "selftests/i915_buddy.c"
-#endif
diff --git a/drivers/gpu/drm/i915/i915_buddy.h b/drivers/gpu/drm/i915/i915_buddy.h
deleted file mode 100644
index 9ce5200f4001..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_BUDDY_H__
-#define __I915_BUDDY_H__
-
-#include <linux/bitops.h>
-#include <linux/list.h>
-
-struct i915_buddy_block {
-#define I915_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
-#define I915_BUDDY_HEADER_STATE GENMASK_ULL(11, 10)
-#define I915_BUDDY_ALLOCATED (1 << 10)
-#define I915_BUDDY_FREE (2 << 10)
-#define I915_BUDDY_SPLIT (3 << 10)
-/* Free to be used, if needed in the future */
-#define I915_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
-#define I915_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
- u64 header;
-
- struct i915_buddy_block *left;
- struct i915_buddy_block *right;
- struct i915_buddy_block *parent;
-
- void *private; /* owned by creator */
-
- /*
- * While the block is allocated by the user through i915_buddy_alloc*,
- * the user has ownership of the link, for example to maintain within
- * a list, if so desired. As soon as the block is freed with
- * i915_buddy_free* ownership is given back to the mm.
- */
- struct list_head link;
- struct list_head tmp_link;
-};
-
-/* Order-zero must be at least PAGE_SIZE */
-#define I915_BUDDY_MAX_ORDER (63 - PAGE_SHIFT)
-
-/*
- * Binary Buddy System.
- *
- * Locking should be handled by the user, a simple mutex around
- * i915_buddy_alloc* and i915_buddy_free* should suffice.
- */
-struct i915_buddy_mm {
- /* Maintain a free list for each order. */
- struct list_head *free_list;
-
- /*
- * Maintain explicit binary tree(s) to track the allocation of the
- * address space. This gives us a simple way of finding a buddy block
- * and performing the potentially recursive merge step when freeing a
- * block. Nodes are either allocated or free, in which case they will
- * also exist on the respective free list.
- */
- struct i915_buddy_block **roots;
-
- /*
- * Anything from here is public, and remains static for the lifetime of
- * the mm. Everything above is considered do-not-touch.
- */
- unsigned int n_roots;
- unsigned int max_order;
-
- /* Must be at least PAGE_SIZE */
- u64 chunk_size;
- u64 size;
-};
-
-static inline u64
-i915_buddy_block_offset(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_OFFSET;
-}
-
-static inline unsigned int
-i915_buddy_block_order(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_ORDER;
-}
-
-static inline unsigned int
-i915_buddy_block_state(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_STATE;
-}
-
-static inline bool
-i915_buddy_block_is_allocated(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_ALLOCATED;
-}
-
-static inline bool
-i915_buddy_block_is_free(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_FREE;
-}
-
-static inline bool
-i915_buddy_block_is_split(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_SPLIT;
-}
-
-static inline u64
-i915_buddy_block_size(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- return mm->chunk_size << i915_buddy_block_order(block);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size);
-
-void i915_buddy_fini(struct i915_buddy_mm *mm);
-
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order);
-
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size);
-
-void i915_buddy_free(struct i915_buddy_mm *mm, struct i915_buddy_block *block);
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects);
-
-#endif
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 92bccc5623a8..122dd297b6af 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -84,6 +84,7 @@
#include "intel_gvt.h"
#include "intel_memory_region.h"
#include "intel_pm.h"
+#include "intel_region_ttm.h"
#include "intel_sideband.h"
#include "vlv_suspend.h"
@@ -335,6 +336,10 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
if (ret < 0)
goto err_workqueues;
+ ret = intel_region_ttm_device_init(dev_priv);
+ if (ret)
+ goto err_ttm;
+
intel_wopcm_init_early(&dev_priv->wopcm);
intel_gt_init_early(&dev_priv->gt, dev_priv);
@@ -359,6 +364,8 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
err_gem:
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
+err_ttm:
vlv_suspend_cleanup(dev_priv);
err_workqueues:
i915_workqueues_cleanup(dev_priv);
@@ -376,6 +383,7 @@ static void i915_driver_late_release(struct drm_i915_private *dev_priv)
intel_power_domains_cleanup(dev_priv);
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
vlv_suspend_cleanup(dev_priv);
i915_workqueues_cleanup(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 128198e8b4d0..1276a2f60740 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -60,6 +60,7 @@
#include <drm/drm_atomic.h>
#include <drm/drm_connector.h>
#include <drm/i915_mei_hdcp_interface.h>
+#include <drm/ttm/ttm_device.h>
#include "i915_params.h"
#include "i915_reg.h"
@@ -1166,6 +1167,9 @@ struct drm_i915_private {
/* Mutex to protect the above hdcp component related values. */
struct mutex hdcp_comp_mutex;
+ /* The TTM device structure. */
+ struct ttm_device bdev;
+
I915_SELFTEST_DECLARE(struct i915_selftest_stash selftest;)
/*
@@ -1751,7 +1755,8 @@ void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv);
void i915_gem_init_early(struct drm_i915_private *dev_priv);
void i915_gem_cleanup_early(struct drm_i915_private *dev_priv);
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance);
static inline void i915_gem_drain_freed_objects(struct drm_i915_private *i915)
{
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d0018c5f88bd..180f6e9107d4 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1109,6 +1109,7 @@ int i915_gem_init(struct drm_i915_private *dev_priv)
}
i915_gem_drain_freed_objects(dev_priv);
+
return ret;
}
diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c
index 3aa213684293..77f1911c463b 100644
--- a/drivers/gpu/drm/i915/i915_globals.c
+++ b/drivers/gpu/drm/i915/i915_globals.c
@@ -87,7 +87,6 @@ static void __i915_globals_cleanup(void)
static __initconst int (* const initfn[])(void) = {
i915_global_active_init,
- i915_global_buddy_init,
i915_global_context_init,
i915_global_gem_context_init,
i915_global_objects_init,
diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h
index b2f5cd9b9b1a..2d199f411a4a 100644
--- a/drivers/gpu/drm/i915/i915_globals.h
+++ b/drivers/gpu/drm/i915/i915_globals.h
@@ -27,7 +27,6 @@ void i915_globals_exit(void);
/* constructors */
int i915_global_active_init(void);
-int i915_global_buddy_init(void);
int i915_global_context_init(void);
int i915_global_gem_context_init(void);
int i915_global_objects_init(void);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.c b/drivers/gpu/drm/i915/i915_scatterlist.c
index cc6b3846a8c7..69e9e6c3135e 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.c
+++ b/drivers/gpu/drm/i915/i915_scatterlist.c
@@ -6,6 +6,10 @@
#include "i915_scatterlist.h"
+#include <drm/drm_mm.h>
+
+#include <linux/slab.h>
+
bool i915_sg_trim(struct sg_table *orig_st)
{
struct sg_table new_st;
@@ -34,6 +38,72 @@ bool i915_sg_trim(struct sg_table *orig_st)
return true;
}
+/**
+ * i915_sg_from_mm_node - Create an sg_table from a struct drm_mm_node
+ * @node: The drm_mm_node.
+ * @region_start: An offset to add to the dma addresses of the sg list.
+ *
+ * Create a struct sg_table, initializing it from a struct drm_mm_node,
+ * taking a maximum segment length into account, splitting into segments
+ * if necessary.
+ *
+ * Return: A pointer to a kmalloced struct sg_table on success, negative
+ * error code cast to an error pointer on failure.
+ */
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start)
+{
+ const u64 max_segment = SZ_1G; /* Do we have a limit on this? */
+ u64 segment_pages = max_segment >> PAGE_SHIFT;
+ u64 block_size, offset, prev_end;
+ struct sg_table *st;
+ struct scatterlist *sg;
+
+ st = kmalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ if (sg_alloc_table(st, DIV_ROUND_UP(node->size, segment_pages),
+ GFP_KERNEL)) {
+ kfree(st);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ sg = st->sgl;
+ st->nents = 0;
+ prev_end = (resource_size_t)-1;
+ block_size = node->size << PAGE_SHIFT;
+ offset = node->start << PAGE_SHIFT;
+
+ while (block_size) {
+ u64 len;
+
+ if (offset != prev_end || sg->length >= max_segment) {
+ if (st->nents)
+ sg = __sg_next(sg);
+
+ sg_dma_address(sg) = region_start + offset;
+ sg_dma_len(sg) = 0;
+ sg->length = 0;
+ st->nents++;
+ }
+
+ len = min(block_size, max_segment - sg->length);
+ sg->length += len;
+ sg_dma_len(sg) += len;
+
+ offset += len;
+ block_size -= len;
+
+ prev_end = offset;
+ }
+
+ sg_mark_end(sg);
+ i915_sg_trim(st);
+
+ return st;
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/scatterlist.c"
#endif
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index b96baad66a3a..5acca45ea981 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -13,6 +13,8 @@
#include "i915_gem.h"
+struct drm_mm_node;
+
/*
* Optimised SGL iterator for GEM objects
*/
@@ -141,4 +143,6 @@ static inline unsigned int i915_sg_segment_size(void)
bool i915_sg_trim(struct sg_table *orig_st);
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start);
#endif
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index d98e8b81d322..4092cc987679 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -28,6 +28,11 @@ static const struct {
},
};
+struct intel_region_reserve {
+ struct list_head link;
+ void *node;
+};
+
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance)
@@ -58,146 +63,61 @@ intel_memory_region_by_type(struct drm_i915_private *i915,
return NULL;
}
-static u64
-intel_memory_region_free_pages(struct intel_memory_region *mem,
- struct list_head *blocks)
+/**
+ * intel_memory_region_unreserve - Unreserve all previously reserved
+ * ranges
+ * @mem: The region containing the reserved ranges.
+ */
+void intel_memory_region_unreserve(struct intel_memory_region *mem)
{
- struct i915_buddy_block *block, *on;
- u64 size = 0;
+ struct intel_region_reserve *reserve, *next;
- list_for_each_entry_safe(block, on, blocks, link) {
- size += i915_buddy_block_size(&mem->mm, block);
- i915_buddy_free(&mem->mm, block);
- }
- INIT_LIST_HEAD(blocks);
+ if (!mem->priv_ops || !mem->priv_ops->free)
+ return;
- return size;
-}
-
-void
-__intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks)
-{
mutex_lock(&mem->mm_lock);
- mem->avail += intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
-}
-
-void
-__intel_memory_region_put_block_buddy(struct i915_buddy_block *block)
-{
- struct list_head blocks;
-
- INIT_LIST_HEAD(&blocks);
- list_add(&block->link, &blocks);
- __intel_memory_region_put_pages_buddy(block->private, &blocks);
-}
-
-int
-__intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks)
-{
- unsigned int min_order = 0;
- unsigned long n_pages;
-
- GEM_BUG_ON(!IS_ALIGNED(size, mem->mm.chunk_size));
- GEM_BUG_ON(!list_empty(blocks));
-
- if (flags & I915_ALLOC_MIN_PAGE_SIZE) {
- min_order = ilog2(mem->min_page_size) -
- ilog2(mem->mm.chunk_size);
- }
-
- if (flags & I915_ALLOC_CONTIGUOUS) {
- size = roundup_pow_of_two(size);
- min_order = ilog2(size) - ilog2(mem->mm.chunk_size);
+ list_for_each_entry_safe(reserve, next, &mem->reserved, link) {
+ list_del(&reserve->link);
+ mem->priv_ops->free(mem, reserve->node);
+ kfree(reserve);
}
-
- if (size > mem->mm.size)
- return -E2BIG;
-
- n_pages = size >> ilog2(mem->mm.chunk_size);
-
- mutex_lock(&mem->mm_lock);
-
- do {
- struct i915_buddy_block *block;
- unsigned int order;
-
- order = fls(n_pages) - 1;
- GEM_BUG_ON(order > mem->mm.max_order);
- GEM_BUG_ON(order < min_order);
-
- do {
- block = i915_buddy_alloc(&mem->mm, order);
- if (!IS_ERR(block))
- break;
-
- if (order-- == min_order)
- goto err_free_blocks;
- } while (1);
-
- n_pages -= BIT(order);
-
- block->private = mem;
- list_add_tail(&block->link, blocks);
-
- if (!n_pages)
- break;
- } while (1);
-
- mem->avail -= size;
mutex_unlock(&mem->mm_lock);
- return 0;
-
-err_free_blocks:
- intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
- return -ENXIO;
}
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags)
+/**
+ * intel_memory_region_reserve - Reserve a memory range
+ * @mem: The region for which we want to reserve a range.
+ * @offset: Start of the range to reserve.
+ * @size: The size of the range to reserve.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
{
- struct i915_buddy_block *block;
- LIST_HEAD(blocks);
int ret;
+ struct intel_region_reserve *reserve;
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, &blocks);
- if (ret)
- return ERR_PTR(ret);
+ if (!mem->priv_ops || !mem->priv_ops->reserve)
+ return -EINVAL;
- block = list_first_entry(&blocks, typeof(*block), link);
- list_del_init(&block->link);
- return block;
-}
+ reserve = kzalloc(sizeof(*reserve), GFP_KERNEL);
+ if (!reserve)
+ return -ENOMEM;
-int intel_memory_region_init_buddy(struct intel_memory_region *mem)
-{
- return i915_buddy_init(&mem->mm, resource_size(&mem->region),
- PAGE_SIZE);
-}
-
-void intel_memory_region_release_buddy(struct intel_memory_region *mem)
-{
- i915_buddy_free_list(&mem->mm, &mem->reserved);
- i915_buddy_fini(&mem->mm);
-}
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size)
-{
- int ret;
+ reserve->node = mem->priv_ops->reserve(mem, offset, size);
+ if (IS_ERR(reserve->node)) {
+ ret = PTR_ERR(reserve->node);
+ kfree(reserve);
+ return ret;
+ }
mutex_lock(&mem->mm_lock);
- ret = i915_buddy_alloc_range(&mem->mm, &mem->reserved, offset, size);
+ list_add_tail(&reserve->link, &mem->reserved);
mutex_unlock(&mem->mm_lock);
- return ret;
+ return 0;
}
struct intel_memory_region *
@@ -206,6 +126,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops)
{
struct intel_memory_region *mem;
@@ -222,6 +144,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
mem->ops = ops;
mem->total = size;
mem->avail = mem->total;
+ mem->type = type;
+ mem->instance = instance;
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
@@ -259,6 +183,7 @@ static void __intel_memory_region_destroy(struct kref *kref)
struct intel_memory_region *mem =
container_of(kref, typeof(*mem), kref);
+ intel_memory_region_unreserve(mem);
if (mem->ops->release)
mem->ops->release(mem);
@@ -296,15 +221,15 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
instance = intel_region_map[i].instance;
switch (type) {
case INTEL_MEMORY_SYSTEM:
- mem = i915_gem_shmem_setup(i915);
+ mem = i915_gem_shmem_setup(i915, type, instance);
break;
case INTEL_MEMORY_STOLEN_LOCAL:
- mem = i915_gem_stolen_lmem_setup(i915);
+ mem = i915_gem_stolen_lmem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
case INTEL_MEMORY_STOLEN_SYSTEM:
- mem = i915_gem_stolen_smem_setup(i915);
+ mem = i915_gem_stolen_smem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
@@ -321,9 +246,6 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
}
mem->id = i;
- mem->type = type;
- mem->instance = instance;
-
i915->mm.regions[i] = mem;
}
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index d24ce5a0b30b..e69cde13daf2 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -13,8 +13,6 @@
#include <drm/drm_mm.h>
#include <drm/i915_drm.h>
-#include "i915_buddy.h"
-
struct drm_i915_private;
struct drm_i915_gem_object;
struct intel_memory_region;
@@ -25,6 +23,7 @@ enum intel_memory_type {
INTEL_MEMORY_LOCAL = I915_MEMORY_CLASS_DEVICE,
INTEL_MEMORY_STOLEN_SYSTEM,
INTEL_MEMORY_STOLEN_LOCAL,
+ INTEL_MEMORY_MOCK,
};
enum intel_region_id {
@@ -59,10 +58,19 @@ struct intel_memory_region_ops {
unsigned int flags);
};
+struct intel_memory_region_private_ops {
+ void *(*reserve)(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
+ void (*free)(struct intel_memory_region *mem,
+ void *node);
+};
+
struct intel_memory_region {
struct drm_i915_private *i915;
const struct intel_memory_region_ops *ops;
+ const struct intel_memory_region_private_ops *priv_ops;
struct io_mapping iomap;
struct resource region;
@@ -70,7 +78,6 @@ struct intel_memory_region {
/* For fake LMEM */
struct drm_mm_node fake_mappable;
- struct i915_buddy_mm mm;
struct mutex mm_lock;
struct kref kref;
@@ -95,36 +102,26 @@ struct intel_memory_region {
struct list_head list;
struct list_head purgeable;
} objects;
+
+ size_t chunk_size;
+ unsigned int max_order;
+ bool is_range_manager;
+
+ void *region_private;
};
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance);
-int intel_memory_region_init_buddy(struct intel_memory_region *mem);
-void intel_memory_region_release_buddy(struct intel_memory_region *mem);
-
-int __intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks);
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-void __intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks);
-void __intel_memory_region_put_block_buddy(struct i915_buddy_block *block);
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size);
-
struct intel_memory_region *
intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t start,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops);
struct intel_memory_region *
@@ -141,4 +138,9 @@ __printf(2, 3) void
intel_memory_region_set_name(struct intel_memory_region *mem,
const char *fmt, ...);
+void intel_memory_region_unreserve(struct intel_memory_region *mem);
+
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
new file mode 100644
index 000000000000..69c98298326b
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -0,0 +1,334 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_device.h>
+
+#include "i915_drv.h"
+#include "i915_scatterlist.h"
+
+#include "intel_region_ttm.h"
+
+/**
+ * DOC: TTM support structure
+ *
+ * The code in this file deals with setting up memory managers for TTM
+ * LMEM and MOCK regions and converting the output from
+ * the managers to struct sg_table, Basically providing the mapping from
+ * i915 GEM regions to TTM memory types and resource managers.
+ */
+
+/**
+ * struct intel_region_ttm_private - Private info for TTM regions
+ * @man: The TTM resource manager used to manage this region
+ * @mock_bdev: A mock zero-initialized TTM device for this region
+ */
+struct intel_region_ttm_private {
+ struct ttm_resource_manager *man;
+
+ I915_SELFTEST_DECLARE(struct ttm_device mock_bdev);
+ I915_SELFTEST_DECLARE(struct drm_vma_offset_manager mock_vma);
+};
+
+/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
+static struct ttm_device_funcs i915_ttm_bo_driver;
+
+static struct intel_region_ttm_private *to_ttm_private(void *data)
+{
+ return (struct intel_region_ttm_private *)data;
+}
+
+/**
+ * intel_region_ttm_device_init - Initialize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
+{
+ struct drm_device *drm = &dev_priv->drm;
+
+ return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ drm->dev, drm->anon_inode->i_mapping,
+ drm->vma_offset_manager, false, false);
+}
+
+/**
+ * intel_region_ttm_device_fini - Finalize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ */
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv)
+{
+ ttm_device_fini(&dev_priv->bdev);
+}
+
+/*
+ * Map the i915 memory regions to TTM memory types. We use the
+ * driver-private types for now, reserving TTM_PL_VRAM for stolen
+ * memory and TTM_PL_TT for GGTT use if decided to implement this.
+ */
+static int intel_region_to_ttm_type(struct intel_memory_region *mem)
+{
+ int type;
+
+ GEM_BUG_ON(mem->type != INTEL_MEMORY_LOCAL &&
+ mem->type != INTEL_MEMORY_MOCK);
+
+ type = mem->instance + TTM_PL_PRIV;
+ GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES);
+
+ return type;
+}
+
+static void *intel_region_ttm_node_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * Having to use a mock_bo is unfortunate but stems from some
+ * drivers having private managers that insist to know what the
+ * allocate memory is intended for, using it to send private
+ * data to the manager. Also recently the bo has been used to send
+ * alignment info to the manager. Assume that apart from the latter,
+ * none of the managers we use will ever access the buffer object
+ * members, hoping we can pass the alignment info in the
+ * struct ttm_place in the future.
+ */
+
+ place.fpfn = offset >> PAGE_SHIFT;
+ place.lpfn = place.fpfn + (size >> PAGE_SHIFT);
+ res.num_pages = size >> PAGE_SHIFT;
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+/**
+ * intel_region_ttm_node_free - Free a node allocated from a resource manager
+ * @mem: The region the node was allocated from.
+ * @node: The opaque node representing an allocation.
+ */
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_resource res = {};
+
+ res.mm_node = node;
+ man->func->free(man, &res);
+}
+
+static const struct intel_memory_region_private_ops priv_ops = {
+ .reserve = intel_region_ttm_node_reserve,
+ .free = intel_region_ttm_node_free,
+};
+
+static int __intel_region_ttm_init(struct intel_memory_region *mem,
+ struct intel_region_ttm_private *priv,
+ struct ttm_device *bdev)
+{
+ int mem_type = intel_region_to_ttm_type(mem);
+ int ret;
+
+ ret = ttm_range_man_init(bdev, mem_type, false,
+ resource_size(&mem->region) >> PAGE_SHIFT);
+ if (ret)
+ return ret;
+
+ priv->man = ttm_manager_type(bdev, mem_type);
+ mem->chunk_size = PAGE_SIZE;
+ mem->max_order =
+ get_order(rounddown_pow_of_two(resource_size(&mem->region)));
+ mem->is_range_manager = true;
+ mem->priv_ops = &priv_ops;
+ mem->region_private = priv;
+
+ return 0;
+}
+
+/**
+ * intel_region_ttm_init - Initialize a memory region for TTM.
+ * @mem: The region to initialize.
+ *
+ * This function initializes a suitable TTM resource manager for the
+ * region, attaches it to the TTM device.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_init(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv;
+ int ret;
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ ret = __intel_region_ttm_init(mem, priv, &mem->i915->bdev);
+ if (ret)
+ kfree(priv);
+ return ret;
+}
+
+/**
+ * intel_region_ttm_fini - Finalize a TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration.
+ */
+void intel_region_ttm_fini(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv =
+ to_ttm_private(mem->region_private);
+ int ret;
+
+ ret = ttm_range_man_fini(&mem->i915->bdev, intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ mem->region_private = NULL;
+ kfree(priv);
+}
+
+/**
+ * intel_region_ttm_node_to_st - Convert an opaque TTM resource manager node
+ * to an sg_table.
+ * @mem: The memory region.
+ * @node: The resource manager node obtained from the TTM resource manager.
+ *
+ * The gem backends typically use sg-tables for operations on the underlying
+ * io_memory. So provide a way for the backends to translate the
+ * nodes they are handed from TTM to sg-tables.
+ *
+ * Return: A malloced sg_table on success, an error pointer on failure.
+ */
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node)
+{
+ return i915_sg_from_mm_node(node, mem->region.start);
+}
+
+/**
+ * intel_region_ttm_node_alloc - Allocate memory resources from a region
+ * @mem: The memory region,
+ * @size: The requested size in bytes
+ * @flags: Allocation flags
+ *
+ * This functionality is provided only for callers that need to allocate
+ * memory from standalone TTM range managers, without the TTM eviction
+ * functionality. Don't use if you are not completely sure that's the
+ * case. The returned opaque node can be converted to an sg_table using
+ * intel_region_ttm_node_to_st(), and can be freed using
+ * intel_region_ttm_node_free().
+ *
+ * Return: A valid pointer on success, an error pointer on failure.
+ */
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * We ignore the flags for now since we're using the range
+ * manager and contigous and min page size would be fulfilled
+ * by default if size is min page size aligned.
+ */
+ res.num_pages = size >> PAGE_SHIFT;
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ mock_bo.page_alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ mock_bo.page_alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ mock_bo.page_alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+#ifdef CONFIG_DRM_I915_SELFTEST
+/**
+ * intel_region_ttm_init_mock - Initialize a mock memory region for TTM.
+ * @mem: The region to initialize.
+ *
+ * This function initializes a suitable TTM resource manager for the
+ * region, attaches it to the TTM device.
+ * This code is for mock selftests only.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_init_mock(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv;
+ int ret;
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ drm_vma_offset_manager_init(&priv->mock_vma,
+ DRM_FILE_PAGE_OFFSET_START,
+ DRM_FILE_PAGE_OFFSET_SIZE);
+ ret = ttm_device_init(&priv->mock_bdev, &i915_ttm_bo_driver,
+ NULL, NULL, &priv->mock_vma, false, false);
+ if (ret)
+ goto out_device;
+
+ ret = __intel_region_ttm_init(mem, priv, &priv->mock_bdev);
+ if (ret)
+ goto out_region;
+
+ return 0;
+
+out_region:
+ ttm_device_fini(&priv->mock_bdev);
+ drm_vma_offset_manager_destroy(&priv->mock_vma);
+out_device:
+ kfree(priv);
+ return ret;
+}
+
+/**
+ * intel_region_ttm_fini_mock - Finalize a mock TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration. Finally destroys the mock TTM device and
+ * vma manager. This code is for mock selftests only.
+ */
+void intel_region_ttm_fini_mock(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv =
+ to_ttm_private(mem->region_private);
+ int ret;
+
+ ret = ttm_range_man_fini(&priv->mock_bdev, intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ ttm_device_fini(&priv->mock_bdev);
+ drm_vma_offset_manager_destroy(&priv->mock_vma);
+ mem->region_private = NULL;
+ kfree(priv);
+}
+#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
new file mode 100644
index 000000000000..23c6c2d4b1cd
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _INTEL_REGION_TTM_H_
+#define _INTEL_REGION_TTM_H_
+
+#include <linux/types.h>
+
+#include "i915_selftest.h"
+
+struct drm_i915_private;
+struct intel_memory_region;
+
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
+
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv);
+
+int intel_region_ttm_init(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini(struct intel_memory_region *mem);
+
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node);
+
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+
+#ifdef CONFIG_DRM_I915_SELFTEST
+int intel_region_ttm_init_mock(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini_mock(struct intel_memory_region *mem);
+#endif
+#endif /* _INTEL_REGION_TTM_H_ */
diff --git a/drivers/gpu/drm/i915/selftests/i915_buddy.c b/drivers/gpu/drm/i915/selftests/i915_buddy.c
deleted file mode 100644
index f0f5c4df8dbc..000000000000
--- a/drivers/gpu/drm/i915/selftests/i915_buddy.c
+++ /dev/null
@@ -1,789 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/prime_numbers.h>
-
-#include "../i915_selftest.h"
-#include "i915_random.h"
-
-static void __igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block,
- bool buddy)
-{
- pr_err("block info: header=%llx, state=%u, order=%d, offset=%llx size=%llx root=%s buddy=%s\n",
- block->header,
- i915_buddy_block_state(block),
- i915_buddy_block_order(block),
- i915_buddy_block_offset(block),
- i915_buddy_block_size(mm, block),
- yesno(!block->parent),
- yesno(buddy));
-}
-
-static void igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
-
- __igt_dump_block(mm, block, false);
-
- buddy = get_buddy(block);
- if (buddy)
- __igt_dump_block(mm, buddy, true);
-}
-
-static int igt_check_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
- unsigned int block_state;
- u64 block_size;
- u64 offset;
- int err = 0;
-
- block_state = i915_buddy_block_state(block);
-
- if (block_state != I915_BUDDY_ALLOCATED &&
- block_state != I915_BUDDY_FREE &&
- block_state != I915_BUDDY_SPLIT) {
- pr_err("block state mismatch\n");
- err = -EINVAL;
- }
-
- block_size = i915_buddy_block_size(mm, block);
- offset = i915_buddy_block_offset(block);
-
- if (block_size < mm->chunk_size) {
- pr_err("block size smaller than min size\n");
- err = -EINVAL;
- }
-
- if (!is_power_of_2(block_size)) {
- pr_err("block size not power of two\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(block_size, mm->chunk_size)) {
- pr_err("block size not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, mm->chunk_size)) {
- pr_err("block offset not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, block_size)) {
- pr_err("block offset not aligned to block size\n");
- err = -EINVAL;
- }
-
- buddy = get_buddy(block);
-
- if (!buddy && block->parent) {
- pr_err("buddy has gone fishing\n");
- err = -EINVAL;
- }
-
- if (buddy) {
- if (i915_buddy_block_offset(buddy) != (offset ^ block_size)) {
- pr_err("buddy has wrong offset\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_size(mm, buddy) != block_size) {
- pr_err("buddy size mismatch\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_state(buddy) == block_state &&
- block_state == I915_BUDDY_FREE) {
- pr_err("block and its buddy are free\n");
- err = -EINVAL;
- }
- }
-
- return err;
-}
-
-static int igt_check_blocks(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 expected_size,
- bool is_contiguous)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *prev;
- u64 total;
- int err = 0;
-
- block = NULL;
- prev = NULL;
- total = 0;
-
- list_for_each_entry(block, blocks, link) {
- err = igt_check_block(mm, block);
-
- if (!i915_buddy_block_is_allocated(block)) {
- pr_err("block not allocated\n"),
- err = -EINVAL;
- }
-
- if (is_contiguous && prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(block);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("block offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- if (err)
- break;
-
- total += i915_buddy_block_size(mm, block);
- prev = block;
- }
-
- if (!err) {
- if (total != expected_size) {
- pr_err("size mismatch, expected=%llx, found=%llx\n",
- expected_size, total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev block, dump:\n");
- igt_dump_block(mm, prev);
- }
-
- if (block) {
- pr_err("bad block, dump:\n");
- igt_dump_block(mm, block);
- }
-
- return err;
-}
-
-static int igt_check_mm(struct i915_buddy_mm *mm)
-{
- struct i915_buddy_block *root;
- struct i915_buddy_block *prev;
- unsigned int i;
- u64 total;
- int err = 0;
-
- if (!mm->n_roots) {
- pr_err("n_roots is zero\n");
- return -EINVAL;
- }
-
- if (mm->n_roots != hweight64(mm->size)) {
- pr_err("n_roots mismatch, n_roots=%u, expected=%lu\n",
- mm->n_roots, hweight64(mm->size));
- return -EINVAL;
- }
-
- root = NULL;
- prev = NULL;
- total = 0;
-
- for (i = 0; i < mm->n_roots; ++i) {
- struct i915_buddy_block *block;
- unsigned int order;
-
- root = mm->roots[i];
- if (!root) {
- pr_err("root(%u) is NULL\n", i);
- err = -EINVAL;
- break;
- }
-
- err = igt_check_block(mm, root);
-
- if (!i915_buddy_block_is_free(root)) {
- pr_err("root not free\n");
- err = -EINVAL;
- }
-
- order = i915_buddy_block_order(root);
-
- if (!i) {
- if (order != mm->max_order) {
- pr_err("max order root missing\n");
- err = -EINVAL;
- }
- }
-
- if (prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(root);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("root offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- block = list_first_entry_or_null(&mm->free_list[order],
- struct i915_buddy_block,
- link);
- if (block != root) {
- pr_err("root mismatch at order=%u\n", order);
- err = -EINVAL;
- }
-
- if (err)
- break;
-
- prev = root;
- total += i915_buddy_block_size(mm, root);
- }
-
- if (!err) {
- if (total != mm->size) {
- pr_err("expected mm size=%llx, found=%llx\n", mm->size,
- total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev root(%u), dump:\n", i - 1);
- igt_dump_block(mm, prev);
- }
-
- if (root) {
- pr_err("bad root(%u), dump:\n", i);
- igt_dump_block(mm, root);
- }
-
- return err;
-}
-
-static void igt_mm_config(u64 *size, u64 *chunk_size)
-{
- I915_RND_STATE(prng);
- u32 s, ms;
-
- /* Nothing fancy, just try to get an interesting bit pattern */
-
- prandom_seed_state(&prng, i915_selftest.random_seed);
-
- /* Let size be a random number of pages up to 8 GB (2M pages) */
- s = 1 + i915_prandom_u32_max_state((BIT(33 - 12)) - 1, &prng);
- /* Let the chunk size be a random power of 2 less than size */
- ms = BIT(i915_prandom_u32_max_state(ilog2(s), &prng));
- /* Round size down to the chunk size */
- s &= -ms;
-
- /* Convert from pages to bytes */
- *chunk_size = (u64)ms << 12;
- *size = (u64)s << 12;
-}
-
-static int igt_buddy_alloc_smoke(void *arg)
-{
- struct i915_buddy_mm mm;
- IGT_TIMEOUT(end_time);
- I915_RND_STATE(prng);
- u64 chunk_size;
- u64 mm_size;
- int *order;
- int err, i;
-
- igt_mm_config(&mm_size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", mm_size, chunk_size);
-
- err = i915_buddy_init(&mm, mm_size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- order = i915_random_order(mm.max_order + 1, &prng);
- if (!order)
- goto out_fini;
-
- for (i = 0; i <= mm.max_order; ++i) {
- struct i915_buddy_block *block;
- int max_order = order[i];
- bool timeout = false;
- LIST_HEAD(blocks);
- int order;
- u64 total;
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort\n");
- break;
- }
-
- pr_info("filling from max_order=%u\n", max_order);
-
- order = max_order;
- total = 0;
-
- do {
-retry:
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- if (err == -ENOMEM) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- } else {
- if (order--) {
- err = 0;
- goto retry;
- }
-
- pr_err("buddy_alloc with order=%d failed(%d)\n",
- order, err);
- }
-
- break;
- }
-
- list_add_tail(&block->link, &blocks);
-
- if (i915_buddy_block_order(block) != order) {
- pr_err("buddy_alloc order mismatch\n");
- err = -EINVAL;
- break;
- }
-
- total += i915_buddy_block_size(&mm, block);
-
- if (__igt_timeout(end_time, NULL)) {
- timeout = true;
- break;
- }
- } while (total < mm.size);
-
- if (!err)
- err = igt_check_blocks(&mm, &blocks, total, false);
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
- if (err || timeout)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- kfree(order);
-out_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_pessimistic(void *arg)
-{
- const unsigned int max_order = 16;
- struct i915_buddy_block *block, *bn;
- struct i915_buddy_mm mm;
- unsigned int order;
- LIST_HEAD(blocks);
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order < max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* And now the last remaining block available */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM on final alloc\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
-
- /* Should be completely full! */
- for (order = max_order; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- block = list_last_entry(&blocks, typeof(*block), link);
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- /* As we free in increasing size, we make available larger blocks */
- order = 1;
- list_for_each_entry_safe(block, bn, &blocks, link) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
- order++;
- }
-
- /* To confirm, now the whole mm should be available */
- block = i915_buddy_alloc(&mm, max_order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- max_order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_optimistic(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- int order;
- int err;
-
- /*
- * Create a mm with one block of each order available, and
- * try to allocate them all.
- */
-
- err = i915_buddy_init(&mm,
- PAGE_SIZE * ((1 << (max_order + 1)) - 1),
- PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* Should be completely full! */
- block = i915_buddy_alloc(&mm, 0);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded, it should be full!");
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_pathological(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- LIST_HEAD(holes);
- int order, top;
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left. Free the largest block, then whittle down again.
- * Eventually we will have a fully 50% fragmented mm.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (top = max_order; top; top--) {
- /* Make room by freeing the largest allocated block */
- block = list_first_entry_or_null(&blocks, typeof(*block), link);
- if (block) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
- }
-
- for (order = top; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d, top=%d\n",
- order, top);
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
- }
-
- /* There should be one final page for this sub-allocation */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM for hole\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &holes);
-
- block = i915_buddy_alloc(&mm, top);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at top-order %d/%d, it should be full!",
- top, max_order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- i915_buddy_free_list(&mm, &holes);
-
- /* Nothing larger than blocks of chunk_size now available */
- for (order = 1; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
-err:
- list_splice_tail(&holes, &blocks);
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_range(void *arg)
-{
- struct i915_buddy_mm mm;
- unsigned long page_num;
- LIST_HEAD(blocks);
- u64 chunk_size;
- u64 offset;
- u64 size;
- u64 rem;
- int err;
-
- igt_mm_config(&size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", size, chunk_size);
-
- err = i915_buddy_init(&mm, size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort, abort, abort!\n");
- goto err_fini;
- }
-
- rem = mm.size;
- offset = 0;
-
- for_each_prime_number_from(page_num, 1, ULONG_MAX - 1) {
- struct i915_buddy_block *block;
- LIST_HEAD(tmp);
-
- size = min(page_num * mm.chunk_size, rem);
-
- err = i915_buddy_alloc_range(&mm, &tmp, offset, size);
- if (err) {
- if (err == -ENOMEM) {
- pr_info("alloc_range hit -ENOMEM with size=%llx\n",
- size);
- } else {
- pr_err("alloc_range with offset=%llx, size=%llx failed(%d)\n",
- offset, size, err);
- }
-
- break;
- }
-
- block = list_first_entry_or_null(&tmp,
- struct i915_buddy_block,
- link);
- if (!block) {
- pr_err("alloc_range has no blocks\n");
- err = -EINVAL;
- break;
- }
-
- if (i915_buddy_block_offset(block) != offset) {
- pr_err("alloc_range start offset mismatch, found=%llx, expected=%llx\n",
- i915_buddy_block_offset(block), offset);
- err = -EINVAL;
- }
-
- if (!err)
- err = igt_check_blocks(&mm, &tmp, size, true);
-
- list_splice_tail(&tmp, &blocks);
-
- if (err)
- break;
-
- offset += size;
-
- rem -= size;
- if (!rem)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
-err_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_limit(void *arg)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- const u64 size = U64_MAX;
- int err;
-
- err = i915_buddy_init(&mm, size, PAGE_SIZE);
- if (err)
- return err;
-
- if (mm.max_order != I915_BUDDY_MAX_ORDER) {
- pr_err("mm.max_order(%d) != %d\n",
- mm.max_order, I915_BUDDY_MAX_ORDER);
- err = -EINVAL;
- goto out_fini;
- }
-
- block = i915_buddy_alloc(&mm, mm.max_order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- goto out_fini;
- }
-
- if (i915_buddy_block_order(block) != mm.max_order) {
- pr_err("block order(%d) != %d\n",
- i915_buddy_block_order(block), mm.max_order);
- err = -EINVAL;
- goto out_free;
- }
-
- if (i915_buddy_block_size(&mm, block) !=
- BIT_ULL(mm.max_order) * PAGE_SIZE) {
- pr_err("block size(%llu) != %llu\n",
- i915_buddy_block_size(&mm, block),
- BIT_ULL(mm.max_order) * PAGE_SIZE);
- err = -EINVAL;
- goto out_free;
- }
-
-out_free:
- i915_buddy_free(&mm, block);
-out_fini:
- i915_buddy_fini(&mm);
- return err;
-}
-
-int i915_buddy_mock_selftests(void)
-{
- static const struct i915_subtest tests[] = {
- SUBTEST(igt_buddy_alloc_pessimistic),
- SUBTEST(igt_buddy_alloc_optimistic),
- SUBTEST(igt_buddy_alloc_pathological),
- SUBTEST(igt_buddy_alloc_smoke),
- SUBTEST(igt_buddy_alloc_range),
- SUBTEST(igt_buddy_alloc_limit),
- };
-
- return i915_subtests(tests, NULL);
-}
diff --git a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
index 3db34d3eea58..34e5caf38093 100644
--- a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
+++ b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
@@ -33,5 +33,4 @@ selftest(evict, i915_gem_evict_mock_selftests)
selftest(gtt, i915_gem_gtt_mock_selftests)
selftest(hugepages, i915_gem_huge_page_mock_selftests)
selftest(contexts, i915_gem_context_mock_selftests)
-selftest(buddy, i915_buddy_mock_selftests)
selftest(memory_region, intel_memory_region_mock_selftests)
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index f85fd8cbfbf5..c85d516b85cd 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -57,9 +57,10 @@ static int igt_mock_fill(void *arg)
LIST_HEAD(objects);
int err = 0;
- page_size = mem->mm.chunk_size;
- max_pages = div64_u64(total, page_size);
+ page_size = mem->chunk_size;
rem = total;
+retry:
+ max_pages = div64_u64(rem, page_size);
for_each_prime_number_from(page_num, 1, max_pages) {
resource_size_t size = page_num * page_size;
@@ -85,6 +86,11 @@ static int igt_mock_fill(void *arg)
err = 0;
if (err == -ENXIO) {
if (page_num * page_size <= rem) {
+ if (mem->is_range_manager && max_pages > 1) {
+ max_pages >>= 1;
+ goto retry;
+ }
+
pr_err("%s failed, space still left in region\n",
__func__);
err = -EINVAL;
@@ -199,12 +205,18 @@ static int igt_mock_reserve(void *arg)
do {
u32 size = i915_prandom_u32_max_state(cur_avail, &prng);
+retry:
size = max_t(u32, round_up(size, PAGE_SIZE), PAGE_SIZE);
obj = igt_object_create(mem, &objects, size, 0);
if (IS_ERR(obj)) {
- if (PTR_ERR(obj) == -ENXIO)
+ if (PTR_ERR(obj) == -ENXIO) {
+ if (mem->is_range_manager &&
+ size > mem->chunk_size) {
+ size >>= 1;
+ goto retry;
+ }
break;
-
+ }
err = PTR_ERR(obj);
goto out_close;
}
@@ -220,7 +232,7 @@ static int igt_mock_reserve(void *arg)
out_close:
kfree(order);
close_objects(mem, &objects);
- i915_buddy_free_list(&mem->mm, &mem->reserved);
+ intel_memory_region_unreserve(mem);
return err;
}
@@ -240,7 +252,7 @@ static int igt_mock_contiguous(void *arg)
total = resource_size(&mem->region);
/* Min size */
- obj = igt_object_create(mem, &objects, mem->mm.chunk_size,
+ obj = igt_object_create(mem, &objects, mem->chunk_size,
I915_BO_ALLOC_CONTIGUOUS);
if (IS_ERR(obj))
return PTR_ERR(obj);
@@ -321,14 +333,16 @@ static int igt_mock_contiguous(void *arg)
min = target;
target = total >> 1;
- /* Make sure we can still allocate all the fragmented space */
- obj = igt_object_create(mem, &objects, target, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto err_close_objects;
- }
+ if (!mem->is_range_manager) {
+ /* Make sure we can still allocate all the fragmented space */
+ obj = igt_object_create(mem, &objects, target, 0);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto err_close_objects;
+ }
- igt_object_release(obj);
+ igt_object_release(obj);
+ }
/*
* Even though we have enough free space, we don't have a big enough
@@ -348,7 +362,7 @@ static int igt_mock_contiguous(void *arg)
}
target >>= 1;
- } while (target >= mem->mm.chunk_size);
+ } while (target >= mem->chunk_size);
err_close_objects:
list_splice_tail(&holes, &objects);
@@ -368,7 +382,7 @@ static int igt_mock_splintered_region(void *arg)
/*
* Sanity check we can still allocate everything even if the
- * mm.max_order != mm.size. i.e our starting address space size is not a
+ * max_order != mm.size. i.e our starting address space size is not a
* power-of-two.
*/
@@ -377,17 +391,10 @@ static int igt_mock_splintered_region(void *arg)
if (IS_ERR(mem))
return PTR_ERR(mem);
- if (mem->mm.size != size) {
- pr_err("%s size mismatch(%llu != %llu)\n",
- __func__, mem->mm.size, size);
- err = -EINVAL;
- goto out_put;
- }
-
expected_order = get_order(rounddown_pow_of_two(size));
- if (mem->mm.max_order != expected_order) {
+ if (mem->max_order != expected_order) {
pr_err("%s order mismatch(%u != %u)\n",
- __func__, mem->mm.max_order, expected_order);
+ __func__, mem->max_order, expected_order);
err = -EINVAL;
goto out_put;
}
@@ -408,12 +415,15 @@ static int igt_mock_splintered_region(void *arg)
* sure that does indeed hold true.
*/
- obj = igt_object_create(mem, &objects, size, I915_BO_ALLOC_CONTIGUOUS);
- if (!IS_ERR(obj)) {
- pr_err("%s too large contiguous allocation was not rejected\n",
- __func__);
- err = -EINVAL;
- goto out_close;
+ if (!mem->is_range_manager) {
+ obj = igt_object_create(mem, &objects, size,
+ I915_BO_ALLOC_CONTIGUOUS);
+ if (!IS_ERR(obj)) {
+ pr_err("%s too large contiguous allocation was not rejected\n",
+ __func__);
+ err = -EINVAL;
+ goto out_close;
+ }
}
obj = igt_object_create(mem, &objects, rounddown_pow_of_two(size),
@@ -432,68 +442,6 @@ static int igt_mock_splintered_region(void *arg)
return err;
}
-#ifndef SZ_8G
-#define SZ_8G BIT_ULL(33)
-#endif
-
-static int igt_mock_max_segment(void *arg)
-{
- const unsigned int max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = arg;
- struct drm_i915_private *i915 = mem->i915;
- struct drm_i915_gem_object *obj;
- struct i915_buddy_block *block;
- struct scatterlist *sg;
- LIST_HEAD(objects);
- u64 size;
- int err = 0;
-
- /*
- * While we may create very large contiguous blocks, we may need
- * to break those down for consumption elsewhere. In particular,
- * dma-mapping with scatterlist elements have an implicit limit of
- * UINT_MAX on each element.
- */
-
- size = SZ_8G;
- mem = mock_region_create(i915, 0, size, PAGE_SIZE, 0);
- if (IS_ERR(mem))
- return PTR_ERR(mem);
-
- obj = igt_object_create(mem, &objects, size, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto out_put;
- }
-
- size = 0;
- list_for_each_entry(block, &obj->mm.blocks, link) {
- if (i915_buddy_block_size(&mem->mm, block) > size)
- size = i915_buddy_block_size(&mem->mm, block);
- }
- if (size < max_segment) {
- pr_err("%s: Failed to create a huge contiguous block [> %u], largest block %lld\n",
- __func__, max_segment, size);
- err = -EINVAL;
- goto out_close;
- }
-
- for (sg = obj->mm.pages->sgl; sg; sg = sg_next(sg)) {
- if (sg->length > max_segment) {
- pr_err("%s: Created an oversized scatterlist entry, %u > %u\n",
- __func__, sg->length, max_segment);
- err = -EINVAL;
- goto out_close;
- }
- }
-
-out_close:
- close_objects(mem, &objects);
-out_put:
- intel_memory_region_put(mem);
- return err;
-}
-
static int igt_gpu_write_dw(struct intel_context *ce,
struct i915_vma *vma,
u32 dword,
@@ -1098,7 +1046,6 @@ int intel_memory_region_mock_selftests(void)
SUBTEST(igt_mock_fill),
SUBTEST(igt_mock_contiguous),
SUBTEST(igt_mock_splintered_region),
- SUBTEST(igt_mock_max_segment),
};
struct intel_memory_region *mem;
struct drm_i915_private *i915;
diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c
index 5d2d010a1e22..d61387fda945 100644
--- a/drivers/gpu/drm/i915/selftests/mock_region.c
+++ b/drivers/gpu/drm/i915/selftests/mock_region.c
@@ -1,17 +1,54 @@
// SPDX-License-Identifier: MIT
/*
- * Copyright © 2019 Intel Corporation
+ * Copyright © 2019-2021 Intel Corporation
*/
+#include <linux/scatterlist.h>
+
#include "gem/i915_gem_region.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_region.h"
+static void mock_region_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int mock_region_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ return 0;
+}
+
static const struct drm_i915_gem_object_ops mock_region_obj_ops = {
.name = "mock-region",
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = mock_region_get_pages,
+ .put_pages = mock_region_put_pages,
.release = i915_gem_object_release_memory_region,
};
@@ -23,7 +60,7 @@ static int mock_object_init(struct intel_memory_region *mem,
static struct lock_class_key lock_class;
struct drm_i915_private *i915 = mem->i915;
- if (size > mem->mm.size)
+ if (size > resource_size(&mem->region))
return -E2BIG;
drm_gem_private_object_init(&i915->drm, &obj->base, size);
@@ -39,8 +76,8 @@ static int mock_object_init(struct intel_memory_region *mem,
}
static const struct intel_memory_region_ops mock_region_ops = {
- .init = intel_memory_region_init_buddy,
- .release = intel_memory_region_release_buddy,
+ .init = intel_region_ttm_init_mock,
+ .release = intel_region_ttm_fini_mock,
.init_object = mock_object_init,
};
@@ -52,5 +89,6 @@ mock_region_create(struct drm_i915_private *i915,
resource_size_t io_start)
{
return intel_memory_region_create(i915, start, size, min_page_size,
- io_start, &mock_region_ops);
+ io_start, INTEL_MEMORY_MOCK, 0,
+ &mock_region_ops);
}
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 04/12] drm/i915/ttm Initialize the ttm device and memory managers
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Temporarily remove the buddy allocator and related selftests
and hook up the TTM range manager for i915 regions.
Also modify the mock region selftests somewhat to account for a
fragmenting manager.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com> #v2
---
v2:
- Fix an error unwind in lmem_get_pages() (Reported by Matthew Auld)
- Break out and modify usage of i915_sg_dma_sizes() (Reported by Mattew Auld)
- Break out TTM changes to a separate patch (Reported by Christian König)
v3:
- Fix the same error unwind in mock_region_get_pages()
(Reported by Matthew Auld)
- Don't rely on new TTM functionality, but create a mock TTM device,
(Reported by Christian König)
---
drivers/gpu/drm/i915/Kconfig | 1 +
drivers/gpu/drm/i915/Makefile | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 59 +-
.../gpu/drm/i915/gem/i915_gem_object_types.h | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_pages.c | 3 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 120 ---
drivers/gpu/drm/i915/gem/i915_gem_region.h | 4 -
drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.c | 10 +-
drivers/gpu/drm/i915/gem/i915_gem_stolen.h | 9 +-
drivers/gpu/drm/i915/gt/intel_gt.c | 2 -
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 27 +-
drivers/gpu/drm/i915/i915_buddy.c | 435 ----------
drivers/gpu/drm/i915/i915_buddy.h | 131 ---
drivers/gpu/drm/i915/i915_drv.c | 8 +
drivers/gpu/drm/i915/i915_drv.h | 7 +-
drivers/gpu/drm/i915/i915_gem.c | 1 +
drivers/gpu/drm/i915/i915_globals.c | 1 -
drivers/gpu/drm/i915/i915_globals.h | 1 -
drivers/gpu/drm/i915/i915_scatterlist.c | 70 ++
drivers/gpu/drm/i915/i915_scatterlist.h | 4 +
drivers/gpu/drm/i915/intel_memory_region.c | 180 ++--
drivers/gpu/drm/i915/intel_memory_region.h | 44 +-
drivers/gpu/drm/i915/intel_region_ttm.c | 334 ++++++++
drivers/gpu/drm/i915/intel_region_ttm.h | 38 +
drivers/gpu/drm/i915/selftests/i915_buddy.c | 789 ------------------
.../drm/i915/selftests/i915_mock_selftests.h | 1 -
.../drm/i915/selftests/intel_memory_region.c | 133 +--
drivers/gpu/drm/i915/selftests/mock_region.c | 52 +-
29 files changed, 722 insertions(+), 1754 deletions(-)
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.c
delete mode 100644 drivers/gpu/drm/i915/i915_buddy.h
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.c
create mode 100644 drivers/gpu/drm/i915/intel_region_ttm.h
delete mode 100644 drivers/gpu/drm/i915/selftests/i915_buddy.c
diff --git a/drivers/gpu/drm/i915/Kconfig b/drivers/gpu/drm/i915/Kconfig
index 1e1cb245fca7..b63d374dff23 100644
--- a/drivers/gpu/drm/i915/Kconfig
+++ b/drivers/gpu/drm/i915/Kconfig
@@ -26,6 +26,7 @@ config DRM_I915
select SND_HDA_I915 if SND_HDA_CORE
select CEC_CORE if CEC_NOTIFIER
select VMAP_PFN
+ select DRM_TTM
help
Choose this option if you have a system that has "Intel Graphics
Media Accelerator" or "HD Graphics" integrated graphics,
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index d0d936d9137b..cb8823570996 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -50,6 +50,7 @@ i915-y += i915_drv.o \
intel_memory_region.o \
intel_pch.o \
intel_pm.o \
+ intel_region_ttm.o \
intel_runtime_pm.o \
intel_sideband.o \
intel_step.o \
@@ -160,7 +161,6 @@ gem-y += \
i915-y += \
$(gem-y) \
i915_active.o \
- i915_buddy.o \
i915_cmd_parser.o \
i915_gem_evict.o \
i915_gem_gtt.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index f44bdd08f7cb..3b4aa28a076d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,16 +4,71 @@
*/
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
+static void lmem_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ obj->mm.dirty = false;
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int lmem_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ /* Range manager is always contigous */
+ if (obj->mm.region->is_range_manager)
+ obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
+ void __iomem *vaddr =
+ i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
+
+ if (!vaddr) {
+ struct sg_table *pages =
+ __i915_gem_object_unset_pages(obj);
+
+ if (!IS_ERR_OR_NULL(pages))
+ lmem_put_pages(obj, pages);
+ }
+
+ memset_io(vaddr, 0, obj->base.size);
+ io_mapping_unmap(vaddr);
+ }
+
+ return 0;
+}
+
const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
.name = "i915_gem_object_lmem",
.flags = I915_GEM_OBJECT_HAS_IOMEM,
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = lmem_get_pages,
+ .put_pages = lmem_put_pages,
.release = i915_gem_object_release_memory_region,
};
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index 0415f99b6b95..f5b46d11e6e6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -235,10 +235,12 @@ struct drm_i915_gem_object {
* Memory region for this object.
*/
struct intel_memory_region *region;
+
/**
- * List of memory region blocks allocated for this object.
+ * Memory manager node allocated for this object.
*/
- struct list_head blocks;
+ void *st_mm_node;
+
/**
* Element within memory_region->objects or region->purgeable
* if the object is marked as DONTNEED. Access is protected by
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index aed8a37ccdc9..62ee2185a41b 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -473,7 +473,8 @@ __i915_gem_object_get_sg(struct drm_i915_gem_object *obj,
might_sleep();
GEM_BUG_ON(n >= obj->base.size >> PAGE_SHIFT);
- GEM_BUG_ON(!i915_gem_object_has_pinned_pages(obj));
+ if (!i915_gem_object_has_pinned_pages(obj))
+ assert_object_held(obj);
/* As we iterate forward through the sg, we record each entry in a
* radixtree for quick repeated (backwards) lookups. If we have seen
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index ce8fcfc54079..f25e6646c5b7 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -8,129 +8,9 @@
#include "i915_drv.h"
#include "i915_trace.h"
-void
-i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- __intel_memory_region_put_pages_buddy(obj->mm.region, &obj->mm.blocks);
-
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-int
-i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj)
-{
- const u64 max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = obj->mm.region;
- struct list_head *blocks = &obj->mm.blocks;
- resource_size_t size = obj->base.size;
- resource_size_t prev_end;
- struct i915_buddy_block *block;
- unsigned int flags;
- struct sg_table *st;
- struct scatterlist *sg;
- unsigned int sg_page_sizes;
- int ret;
-
- st = kmalloc(sizeof(*st), GFP_KERNEL);
- if (!st)
- return -ENOMEM;
-
- if (sg_alloc_table(st, size >> PAGE_SHIFT, GFP_KERNEL)) {
- kfree(st);
- return -ENOMEM;
- }
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, blocks);
- if (ret)
- goto err_free_sg;
-
- GEM_BUG_ON(list_empty(blocks));
-
- sg = st->sgl;
- st->nents = 0;
- sg_page_sizes = 0;
- prev_end = (resource_size_t)-1;
-
- list_for_each_entry(block, blocks, link) {
- u64 block_size, offset;
-
- block_size = min_t(u64, size,
- i915_buddy_block_size(&mem->mm, block));
- offset = i915_buddy_block_offset(block);
-
- while (block_size) {
- u64 len;
-
- if (offset != prev_end || sg->length >= max_segment) {
- if (st->nents) {
- sg_page_sizes |= sg->length;
- sg = __sg_next(sg);
- }
-
- sg_dma_address(sg) = mem->region.start + offset;
- sg_dma_len(sg) = 0;
- sg->length = 0;
- st->nents++;
- }
-
- len = min(block_size, max_segment - sg->length);
- sg->length += len;
- sg_dma_len(sg) += len;
-
- offset += len;
- block_size -= len;
-
- prev_end = offset;
- }
- }
-
- sg_page_sizes |= sg->length;
- sg_mark_end(sg);
- i915_sg_trim(st);
-
- /* Intended for kernel internal use only */
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- struct scatterlist *sg;
- unsigned long i;
-
- for_each_sg(st->sgl, sg, st->nents, i) {
- unsigned int length;
- void __iomem *vaddr;
- dma_addr_t daddr;
-
- daddr = sg_dma_address(sg);
- daddr -= mem->region.start;
- length = sg_dma_len(sg);
-
- vaddr = io_mapping_map_wc(&mem->iomap, daddr, length);
- memset64((void __force *)vaddr, 0, length / sizeof(u64));
- io_mapping_unmap(vaddr);
- }
-
- wmb();
- }
-
- __i915_gem_object_set_pages(obj, st, sg_page_sizes);
-
- return 0;
-
-err_free_sg:
- sg_free_table(st);
- kfree(st);
- return ret;
-}
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem)
{
- INIT_LIST_HEAD(&obj->mm.blocks);
obj->mm.region = intel_memory_region_get(mem);
if (obj->base.size <= mem->min_page_size)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.h b/drivers/gpu/drm/i915/gem/i915_gem_region.h
index ebddc86d78f7..84fcb3297400 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.h
@@ -12,10 +12,6 @@ struct intel_memory_region;
struct drm_i915_gem_object;
struct sg_table;
-int i915_gem_object_get_pages_buddy(struct drm_i915_gem_object *obj);
-void i915_gem_object_put_pages_buddy(struct drm_i915_gem_object *obj,
- struct sg_table *pages);
-
void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
struct intel_memory_region *mem);
void i915_gem_object_release_memory_region(struct drm_i915_gem_object *obj);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
index a9bfa66c8da1..5d16c4462fda 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shmem.c
@@ -628,11 +628,13 @@ static const struct intel_memory_region_ops shmem_region_ops = {
.init_object = shmem_object_init,
};
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915)
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance)
{
return intel_memory_region_create(i915, 0,
totalram_pages() << PAGE_SHIFT,
PAGE_SIZE, 0,
+ type, instance,
&shmem_region_ops);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
index 293f640faa0a..c42abee206da 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.c
@@ -770,7 +770,8 @@ static const struct intel_memory_region_ops i915_region_stolen_lmem_ops = {
};
struct intel_memory_region *
-i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_uncore *uncore = &i915->uncore;
struct pci_dev *pdev = to_pci_dev(i915->drm.dev);
@@ -788,6 +789,7 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
mem = intel_memory_region_create(i915, lmem_base, lmem_size,
I915_GTT_PAGE_SIZE_4K, io_start,
+ type, instance,
&i915_region_stolen_lmem_ops);
if (IS_ERR(mem))
return mem;
@@ -809,14 +811,15 @@ i915_gem_stolen_lmem_setup(struct drm_i915_private *i915)
}
struct intel_memory_region*
-i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance)
{
struct intel_memory_region *mem;
mem = intel_memory_region_create(i915,
intel_graphics_stolen_res.start,
resource_size(&intel_graphics_stolen_res),
- PAGE_SIZE, 0,
+ PAGE_SIZE, 0, type, instance,
&i915_region_stolen_smem_ops);
if (IS_ERR(mem))
return mem;
@@ -824,7 +827,6 @@ i915_gem_stolen_smem_setup(struct drm_i915_private *i915)
intel_memory_region_set_name(mem, "stolen-system");
mem->private = true;
-
return mem;
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
index 2bec6c367b9c..ccdf7befc571 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_stolen.h
@@ -21,8 +21,13 @@ int i915_gem_stolen_insert_node_in_range(struct drm_i915_private *dev_priv,
u64 end);
void i915_gem_stolen_remove_node(struct drm_i915_private *dev_priv,
struct drm_mm_node *node);
-struct intel_memory_region *i915_gem_stolen_smem_setup(struct drm_i915_private *i915);
-struct intel_memory_region *i915_gem_stolen_lmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *
+i915_gem_stolen_smem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+struct intel_memory_region *
+i915_gem_stolen_lmem_setup(struct drm_i915_private *i915, u16 type,
+ u16 instance);
+
struct drm_i915_gem_object *
i915_gem_object_create_stolen(struct drm_i915_private *dev_priv,
resource_size_t size);
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 8d77dcbad059..3f88ecdee031 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -68,8 +68,6 @@ int intel_gt_probe_lmem(struct intel_gt *gt)
id = INTEL_REGION_LMEM;
mem->id = id;
- mem->type = INTEL_MEMORY_LOCAL;
- mem->instance = 0;
intel_memory_region_set_name(mem, "local%u", mem->instance);
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index 73fceb0c25fc..f7366b054f8e 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -5,6 +5,8 @@
#include "i915_drv.h"
#include "intel_memory_region.h"
+#include "intel_region_lmem.h"
+#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
#include "intel_region_lmem.h"
@@ -66,9 +68,9 @@ static void release_fake_lmem_bar(struct intel_memory_region *mem)
static void
region_lmem_release(struct intel_memory_region *mem)
{
- release_fake_lmem_bar(mem);
+ intel_region_ttm_fini(mem);
io_mapping_fini(&mem->iomap);
- intel_memory_region_release_buddy(mem);
+ release_fake_lmem_bar(mem);
}
static int
@@ -83,12 +85,21 @@ region_lmem_init(struct intel_memory_region *mem)
if (!io_mapping_init_wc(&mem->iomap,
mem->io_start,
- resource_size(&mem->region)))
- return -EIO;
+ resource_size(&mem->region))) {
+ ret = -EIO;
+ goto out_no_io;
+ }
- ret = intel_memory_region_init_buddy(mem);
+ ret = intel_region_ttm_init(mem);
if (ret)
- io_mapping_fini(&mem->iomap);
+ goto out_no_buddy;
+
+ return 0;
+
+out_no_buddy:
+ io_mapping_fini(&mem->iomap);
+out_no_io:
+ release_fake_lmem_bar(mem);
return ret;
}
@@ -127,6 +138,8 @@ intel_gt_setup_fake_lmem(struct intel_gt *gt)
mappable_end,
PAGE_SIZE,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (!IS_ERR(mem)) {
drm_info(&i915->drm, "Intel graphics fake LMEM: %pR\n",
@@ -198,6 +211,8 @@ static struct intel_memory_region *setup_lmem(struct intel_gt *gt)
lmem_size,
I915_GTT_PAGE_SIZE_4K,
io_start,
+ INTEL_MEMORY_LOCAL,
+ 0,
&intel_region_lmem_ops);
if (IS_ERR(mem))
return mem;
diff --git a/drivers/gpu/drm/i915/i915_buddy.c b/drivers/gpu/drm/i915/i915_buddy.c
deleted file mode 100644
index 3a2f6eecb2fc..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.c
+++ /dev/null
@@ -1,435 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/kmemleak.h>
-#include <linux/slab.h>
-
-#include "i915_buddy.h"
-
-#include "i915_gem.h"
-#include "i915_globals.h"
-#include "i915_utils.h"
-
-static struct i915_global_block {
- struct i915_global base;
- struct kmem_cache *slab_blocks;
-} global;
-
-static void i915_global_buddy_shrink(void)
-{
- kmem_cache_shrink(global.slab_blocks);
-}
-
-static void i915_global_buddy_exit(void)
-{
- kmem_cache_destroy(global.slab_blocks);
-}
-
-static struct i915_global_block global = { {
- .shrink = i915_global_buddy_shrink,
- .exit = i915_global_buddy_exit,
-} };
-
-int __init i915_global_buddy_init(void)
-{
- global.slab_blocks = KMEM_CACHE(i915_buddy_block, SLAB_HWCACHE_ALIGN);
- if (!global.slab_blocks)
- return -ENOMEM;
-
- i915_global_register(&global.base);
- return 0;
-}
-
-static struct i915_buddy_block *i915_block_alloc(struct i915_buddy_block *parent,
- unsigned int order,
- u64 offset)
-{
- struct i915_buddy_block *block;
-
- GEM_BUG_ON(order > I915_BUDDY_MAX_ORDER);
-
- block = kmem_cache_zalloc(global.slab_blocks, GFP_KERNEL);
- if (!block)
- return NULL;
-
- block->header = offset;
- block->header |= order;
- block->parent = parent;
-
- GEM_BUG_ON(block->header & I915_BUDDY_HEADER_UNUSED);
- return block;
-}
-
-static void i915_block_free(struct i915_buddy_block *block)
-{
- kmem_cache_free(global.slab_blocks, block);
-}
-
-static void mark_allocated(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_ALLOCATED;
-
- list_del(&block->link);
-}
-
-static void mark_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_FREE;
-
- list_add(&block->link,
- &mm->free_list[i915_buddy_block_order(block)]);
-}
-
-static void mark_split(struct i915_buddy_block *block)
-{
- block->header &= ~I915_BUDDY_HEADER_STATE;
- block->header |= I915_BUDDY_SPLIT;
-
- list_del(&block->link);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size)
-{
- unsigned int i;
- u64 offset;
-
- if (size < chunk_size)
- return -EINVAL;
-
- if (chunk_size < PAGE_SIZE)
- return -EINVAL;
-
- if (!is_power_of_2(chunk_size))
- return -EINVAL;
-
- size = round_down(size, chunk_size);
-
- mm->size = size;
- mm->chunk_size = chunk_size;
- mm->max_order = ilog2(size) - ilog2(chunk_size);
-
- GEM_BUG_ON(mm->max_order > I915_BUDDY_MAX_ORDER);
-
- mm->free_list = kmalloc_array(mm->max_order + 1,
- sizeof(struct list_head),
- GFP_KERNEL);
- if (!mm->free_list)
- return -ENOMEM;
-
- for (i = 0; i <= mm->max_order; ++i)
- INIT_LIST_HEAD(&mm->free_list[i]);
-
- mm->n_roots = hweight64(size);
-
- mm->roots = kmalloc_array(mm->n_roots,
- sizeof(struct i915_buddy_block *),
- GFP_KERNEL);
- if (!mm->roots)
- goto out_free_list;
-
- offset = 0;
- i = 0;
-
- /*
- * Split into power-of-two blocks, in case we are given a size that is
- * not itself a power-of-two.
- */
- do {
- struct i915_buddy_block *root;
- unsigned int order;
- u64 root_size;
-
- root_size = rounddown_pow_of_two(size);
- order = ilog2(root_size) - ilog2(chunk_size);
-
- root = i915_block_alloc(NULL, order, offset);
- if (!root)
- goto out_free_roots;
-
- mark_free(mm, root);
-
- GEM_BUG_ON(i > mm->max_order);
- GEM_BUG_ON(i915_buddy_block_size(mm, root) < chunk_size);
-
- mm->roots[i] = root;
-
- offset += root_size;
- size -= root_size;
- i++;
- } while (size);
-
- return 0;
-
-out_free_roots:
- while (i--)
- i915_block_free(mm->roots[i]);
- kfree(mm->roots);
-out_free_list:
- kfree(mm->free_list);
- return -ENOMEM;
-}
-
-void i915_buddy_fini(struct i915_buddy_mm *mm)
-{
- int i;
-
- for (i = 0; i < mm->n_roots; ++i) {
- GEM_WARN_ON(!i915_buddy_block_is_free(mm->roots[i]));
- i915_block_free(mm->roots[i]);
- }
-
- kfree(mm->roots);
- kfree(mm->free_list);
-}
-
-static int split_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- unsigned int block_order = i915_buddy_block_order(block) - 1;
- u64 offset = i915_buddy_block_offset(block);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
- GEM_BUG_ON(!i915_buddy_block_order(block));
-
- block->left = i915_block_alloc(block, block_order, offset);
- if (!block->left)
- return -ENOMEM;
-
- block->right = i915_block_alloc(block, block_order,
- offset + (mm->chunk_size << block_order));
- if (!block->right) {
- i915_block_free(block->left);
- return -ENOMEM;
- }
-
- mark_free(mm, block->left);
- mark_free(mm, block->right);
-
- mark_split(block);
-
- return 0;
-}
-
-static struct i915_buddy_block *
-get_buddy(struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- parent = block->parent;
- if (!parent)
- return NULL;
-
- if (parent->left == block)
- return parent->right;
-
- return parent->left;
-}
-
-static void __i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *parent;
-
- while ((parent = block->parent)) {
- struct i915_buddy_block *buddy;
-
- buddy = get_buddy(block);
-
- if (!i915_buddy_block_is_free(buddy))
- break;
-
- list_del(&buddy->link);
-
- i915_block_free(block);
- i915_block_free(buddy);
-
- block = parent;
- }
-
- mark_free(mm, block);
-}
-
-void i915_buddy_free(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- GEM_BUG_ON(!i915_buddy_block_is_allocated(block));
- __i915_buddy_free(mm, block);
-}
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects)
-{
- struct i915_buddy_block *block, *on;
-
- list_for_each_entry_safe(block, on, objects, link) {
- i915_buddy_free(mm, block);
- cond_resched();
- }
- INIT_LIST_HEAD(objects);
-}
-
-/*
- * Allocate power-of-two block. The order value here translates to:
- *
- * 0 = 2^0 * mm->chunk_size
- * 1 = 2^1 * mm->chunk_size
- * 2 = 2^2 * mm->chunk_size
- * ...
- */
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order)
-{
- struct i915_buddy_block *block = NULL;
- unsigned int i;
- int err;
-
- for (i = order; i <= mm->max_order; ++i) {
- block = list_first_entry_or_null(&mm->free_list[i],
- struct i915_buddy_block,
- link);
- if (block)
- break;
- }
-
- if (!block)
- return ERR_PTR(-ENOSPC);
-
- GEM_BUG_ON(!i915_buddy_block_is_free(block));
-
- while (i != order) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto out_free;
-
- /* Go low */
- block = block->left;
- i--;
- }
-
- mark_allocated(block);
- kmemleak_update_trace(block);
- return block;
-
-out_free:
- if (i != order)
- __i915_buddy_free(mm, block);
- return ERR_PTR(err);
-}
-
-static inline bool overlaps(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= e2 && e1 >= s2;
-}
-
-static inline bool contains(u64 s1, u64 e1, u64 s2, u64 e2)
-{
- return s1 <= s2 && e1 >= e2;
-}
-
-/*
- * Allocate range. Note that it's safe to chain together multiple alloc_ranges
- * with the same blocks list.
- *
- * Intended for pre-allocating portions of the address space, for example to
- * reserve a block for the initial framebuffer or similar, hence the expectation
- * here is that i915_buddy_alloc() is still the main vehicle for
- * allocations, so if that's not the case then the drm_mm range allocator is
- * probably a much better fit, and so you should probably go use that instead.
- */
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *buddy;
- LIST_HEAD(allocated);
- LIST_HEAD(dfs);
- u64 end;
- int err;
- int i;
-
- if (size < mm->chunk_size)
- return -EINVAL;
-
- if (!IS_ALIGNED(size | start, mm->chunk_size))
- return -EINVAL;
-
- if (range_overflows(start, size, mm->size))
- return -EINVAL;
-
- for (i = 0; i < mm->n_roots; ++i)
- list_add_tail(&mm->roots[i]->tmp_link, &dfs);
-
- end = start + size - 1;
-
- do {
- u64 block_start;
- u64 block_end;
-
- block = list_first_entry_or_null(&dfs,
- struct i915_buddy_block,
- tmp_link);
- if (!block)
- break;
-
- list_del(&block->tmp_link);
-
- block_start = i915_buddy_block_offset(block);
- block_end = block_start + i915_buddy_block_size(mm, block) - 1;
-
- if (!overlaps(start, end, block_start, block_end))
- continue;
-
- if (i915_buddy_block_is_allocated(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- if (contains(start, end, block_start, block_end)) {
- if (!i915_buddy_block_is_free(block)) {
- err = -ENOSPC;
- goto err_free;
- }
-
- mark_allocated(block);
- list_add_tail(&block->link, &allocated);
- continue;
- }
-
- if (!i915_buddy_block_is_split(block)) {
- err = split_block(mm, block);
- if (unlikely(err))
- goto err_undo;
- }
-
- list_add(&block->right->tmp_link, &dfs);
- list_add(&block->left->tmp_link, &dfs);
- } while (1);
-
- list_splice_tail(&allocated, blocks);
- return 0;
-
-err_undo:
- /*
- * We really don't want to leave around a bunch of split blocks, since
- * bigger is better, so make sure we merge everything back before we
- * free the allocated blocks.
- */
- buddy = get_buddy(block);
- if (buddy &&
- (i915_buddy_block_is_free(block) &&
- i915_buddy_block_is_free(buddy)))
- __i915_buddy_free(mm, block);
-
-err_free:
- i915_buddy_free_list(mm, &allocated);
- return err;
-}
-
-#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
-#include "selftests/i915_buddy.c"
-#endif
diff --git a/drivers/gpu/drm/i915/i915_buddy.h b/drivers/gpu/drm/i915/i915_buddy.h
deleted file mode 100644
index 9ce5200f4001..000000000000
--- a/drivers/gpu/drm/i915/i915_buddy.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_BUDDY_H__
-#define __I915_BUDDY_H__
-
-#include <linux/bitops.h>
-#include <linux/list.h>
-
-struct i915_buddy_block {
-#define I915_BUDDY_HEADER_OFFSET GENMASK_ULL(63, 12)
-#define I915_BUDDY_HEADER_STATE GENMASK_ULL(11, 10)
-#define I915_BUDDY_ALLOCATED (1 << 10)
-#define I915_BUDDY_FREE (2 << 10)
-#define I915_BUDDY_SPLIT (3 << 10)
-/* Free to be used, if needed in the future */
-#define I915_BUDDY_HEADER_UNUSED GENMASK_ULL(9, 6)
-#define I915_BUDDY_HEADER_ORDER GENMASK_ULL(5, 0)
- u64 header;
-
- struct i915_buddy_block *left;
- struct i915_buddy_block *right;
- struct i915_buddy_block *parent;
-
- void *private; /* owned by creator */
-
- /*
- * While the block is allocated by the user through i915_buddy_alloc*,
- * the user has ownership of the link, for example to maintain within
- * a list, if so desired. As soon as the block is freed with
- * i915_buddy_free* ownership is given back to the mm.
- */
- struct list_head link;
- struct list_head tmp_link;
-};
-
-/* Order-zero must be at least PAGE_SIZE */
-#define I915_BUDDY_MAX_ORDER (63 - PAGE_SHIFT)
-
-/*
- * Binary Buddy System.
- *
- * Locking should be handled by the user, a simple mutex around
- * i915_buddy_alloc* and i915_buddy_free* should suffice.
- */
-struct i915_buddy_mm {
- /* Maintain a free list for each order. */
- struct list_head *free_list;
-
- /*
- * Maintain explicit binary tree(s) to track the allocation of the
- * address space. This gives us a simple way of finding a buddy block
- * and performing the potentially recursive merge step when freeing a
- * block. Nodes are either allocated or free, in which case they will
- * also exist on the respective free list.
- */
- struct i915_buddy_block **roots;
-
- /*
- * Anything from here is public, and remains static for the lifetime of
- * the mm. Everything above is considered do-not-touch.
- */
- unsigned int n_roots;
- unsigned int max_order;
-
- /* Must be at least PAGE_SIZE */
- u64 chunk_size;
- u64 size;
-};
-
-static inline u64
-i915_buddy_block_offset(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_OFFSET;
-}
-
-static inline unsigned int
-i915_buddy_block_order(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_ORDER;
-}
-
-static inline unsigned int
-i915_buddy_block_state(struct i915_buddy_block *block)
-{
- return block->header & I915_BUDDY_HEADER_STATE;
-}
-
-static inline bool
-i915_buddy_block_is_allocated(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_ALLOCATED;
-}
-
-static inline bool
-i915_buddy_block_is_free(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_FREE;
-}
-
-static inline bool
-i915_buddy_block_is_split(struct i915_buddy_block *block)
-{
- return i915_buddy_block_state(block) == I915_BUDDY_SPLIT;
-}
-
-static inline u64
-i915_buddy_block_size(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- return mm->chunk_size << i915_buddy_block_order(block);
-}
-
-int i915_buddy_init(struct i915_buddy_mm *mm, u64 size, u64 chunk_size);
-
-void i915_buddy_fini(struct i915_buddy_mm *mm);
-
-struct i915_buddy_block *
-i915_buddy_alloc(struct i915_buddy_mm *mm, unsigned int order);
-
-int i915_buddy_alloc_range(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 start, u64 size);
-
-void i915_buddy_free(struct i915_buddy_mm *mm, struct i915_buddy_block *block);
-
-void i915_buddy_free_list(struct i915_buddy_mm *mm, struct list_head *objects);
-
-#endif
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 92bccc5623a8..122dd297b6af 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -84,6 +84,7 @@
#include "intel_gvt.h"
#include "intel_memory_region.h"
#include "intel_pm.h"
+#include "intel_region_ttm.h"
#include "intel_sideband.h"
#include "vlv_suspend.h"
@@ -335,6 +336,10 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
if (ret < 0)
goto err_workqueues;
+ ret = intel_region_ttm_device_init(dev_priv);
+ if (ret)
+ goto err_ttm;
+
intel_wopcm_init_early(&dev_priv->wopcm);
intel_gt_init_early(&dev_priv->gt, dev_priv);
@@ -359,6 +364,8 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
err_gem:
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
+err_ttm:
vlv_suspend_cleanup(dev_priv);
err_workqueues:
i915_workqueues_cleanup(dev_priv);
@@ -376,6 +383,7 @@ static void i915_driver_late_release(struct drm_i915_private *dev_priv)
intel_power_domains_cleanup(dev_priv);
i915_gem_cleanup_early(dev_priv);
intel_gt_driver_late_release(&dev_priv->gt);
+ intel_region_ttm_device_fini(dev_priv);
vlv_suspend_cleanup(dev_priv);
i915_workqueues_cleanup(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 128198e8b4d0..1276a2f60740 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -60,6 +60,7 @@
#include <drm/drm_atomic.h>
#include <drm/drm_connector.h>
#include <drm/i915_mei_hdcp_interface.h>
+#include <drm/ttm/ttm_device.h>
#include "i915_params.h"
#include "i915_reg.h"
@@ -1166,6 +1167,9 @@ struct drm_i915_private {
/* Mutex to protect the above hdcp component related values. */
struct mutex hdcp_comp_mutex;
+ /* The TTM device structure. */
+ struct ttm_device bdev;
+
I915_SELFTEST_DECLARE(struct i915_selftest_stash selftest;)
/*
@@ -1751,7 +1755,8 @@ void i915_gem_cleanup_userptr(struct drm_i915_private *dev_priv);
void i915_gem_init_early(struct drm_i915_private *dev_priv);
void i915_gem_cleanup_early(struct drm_i915_private *dev_priv);
-struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915);
+struct intel_memory_region *i915_gem_shmem_setup(struct drm_i915_private *i915,
+ u16 type, u16 instance);
static inline void i915_gem_drain_freed_objects(struct drm_i915_private *i915)
{
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index d0018c5f88bd..180f6e9107d4 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1109,6 +1109,7 @@ int i915_gem_init(struct drm_i915_private *dev_priv)
}
i915_gem_drain_freed_objects(dev_priv);
+
return ret;
}
diff --git a/drivers/gpu/drm/i915/i915_globals.c b/drivers/gpu/drm/i915/i915_globals.c
index 3aa213684293..77f1911c463b 100644
--- a/drivers/gpu/drm/i915/i915_globals.c
+++ b/drivers/gpu/drm/i915/i915_globals.c
@@ -87,7 +87,6 @@ static void __i915_globals_cleanup(void)
static __initconst int (* const initfn[])(void) = {
i915_global_active_init,
- i915_global_buddy_init,
i915_global_context_init,
i915_global_gem_context_init,
i915_global_objects_init,
diff --git a/drivers/gpu/drm/i915/i915_globals.h b/drivers/gpu/drm/i915/i915_globals.h
index b2f5cd9b9b1a..2d199f411a4a 100644
--- a/drivers/gpu/drm/i915/i915_globals.h
+++ b/drivers/gpu/drm/i915/i915_globals.h
@@ -27,7 +27,6 @@ void i915_globals_exit(void);
/* constructors */
int i915_global_active_init(void);
-int i915_global_buddy_init(void);
int i915_global_context_init(void);
int i915_global_gem_context_init(void);
int i915_global_objects_init(void);
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.c b/drivers/gpu/drm/i915/i915_scatterlist.c
index cc6b3846a8c7..69e9e6c3135e 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.c
+++ b/drivers/gpu/drm/i915/i915_scatterlist.c
@@ -6,6 +6,10 @@
#include "i915_scatterlist.h"
+#include <drm/drm_mm.h>
+
+#include <linux/slab.h>
+
bool i915_sg_trim(struct sg_table *orig_st)
{
struct sg_table new_st;
@@ -34,6 +38,72 @@ bool i915_sg_trim(struct sg_table *orig_st)
return true;
}
+/**
+ * i915_sg_from_mm_node - Create an sg_table from a struct drm_mm_node
+ * @node: The drm_mm_node.
+ * @region_start: An offset to add to the dma addresses of the sg list.
+ *
+ * Create a struct sg_table, initializing it from a struct drm_mm_node,
+ * taking a maximum segment length into account, splitting into segments
+ * if necessary.
+ *
+ * Return: A pointer to a kmalloced struct sg_table on success, negative
+ * error code cast to an error pointer on failure.
+ */
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start)
+{
+ const u64 max_segment = SZ_1G; /* Do we have a limit on this? */
+ u64 segment_pages = max_segment >> PAGE_SHIFT;
+ u64 block_size, offset, prev_end;
+ struct sg_table *st;
+ struct scatterlist *sg;
+
+ st = kmalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ if (sg_alloc_table(st, DIV_ROUND_UP(node->size, segment_pages),
+ GFP_KERNEL)) {
+ kfree(st);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ sg = st->sgl;
+ st->nents = 0;
+ prev_end = (resource_size_t)-1;
+ block_size = node->size << PAGE_SHIFT;
+ offset = node->start << PAGE_SHIFT;
+
+ while (block_size) {
+ u64 len;
+
+ if (offset != prev_end || sg->length >= max_segment) {
+ if (st->nents)
+ sg = __sg_next(sg);
+
+ sg_dma_address(sg) = region_start + offset;
+ sg_dma_len(sg) = 0;
+ sg->length = 0;
+ st->nents++;
+ }
+
+ len = min(block_size, max_segment - sg->length);
+ sg->length += len;
+ sg_dma_len(sg) += len;
+
+ offset += len;
+ block_size -= len;
+
+ prev_end = offset;
+ }
+
+ sg_mark_end(sg);
+ i915_sg_trim(st);
+
+ return st;
+}
+
#if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
#include "selftests/scatterlist.c"
#endif
diff --git a/drivers/gpu/drm/i915/i915_scatterlist.h b/drivers/gpu/drm/i915/i915_scatterlist.h
index b96baad66a3a..5acca45ea981 100644
--- a/drivers/gpu/drm/i915/i915_scatterlist.h
+++ b/drivers/gpu/drm/i915/i915_scatterlist.h
@@ -13,6 +13,8 @@
#include "i915_gem.h"
+struct drm_mm_node;
+
/*
* Optimised SGL iterator for GEM objects
*/
@@ -141,4 +143,6 @@ static inline unsigned int i915_sg_segment_size(void)
bool i915_sg_trim(struct sg_table *orig_st);
+struct sg_table *i915_sg_from_mm_node(const struct drm_mm_node *node,
+ u64 region_start);
#endif
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index d98e8b81d322..4092cc987679 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -28,6 +28,11 @@ static const struct {
},
};
+struct intel_region_reserve {
+ struct list_head link;
+ void *node;
+};
+
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance)
@@ -58,146 +63,61 @@ intel_memory_region_by_type(struct drm_i915_private *i915,
return NULL;
}
-static u64
-intel_memory_region_free_pages(struct intel_memory_region *mem,
- struct list_head *blocks)
+/**
+ * intel_memory_region_unreserve - Unreserve all previously reserved
+ * ranges
+ * @mem: The region containing the reserved ranges.
+ */
+void intel_memory_region_unreserve(struct intel_memory_region *mem)
{
- struct i915_buddy_block *block, *on;
- u64 size = 0;
+ struct intel_region_reserve *reserve, *next;
- list_for_each_entry_safe(block, on, blocks, link) {
- size += i915_buddy_block_size(&mem->mm, block);
- i915_buddy_free(&mem->mm, block);
- }
- INIT_LIST_HEAD(blocks);
+ if (!mem->priv_ops || !mem->priv_ops->free)
+ return;
- return size;
-}
-
-void
-__intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks)
-{
mutex_lock(&mem->mm_lock);
- mem->avail += intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
-}
-
-void
-__intel_memory_region_put_block_buddy(struct i915_buddy_block *block)
-{
- struct list_head blocks;
-
- INIT_LIST_HEAD(&blocks);
- list_add(&block->link, &blocks);
- __intel_memory_region_put_pages_buddy(block->private, &blocks);
-}
-
-int
-__intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks)
-{
- unsigned int min_order = 0;
- unsigned long n_pages;
-
- GEM_BUG_ON(!IS_ALIGNED(size, mem->mm.chunk_size));
- GEM_BUG_ON(!list_empty(blocks));
-
- if (flags & I915_ALLOC_MIN_PAGE_SIZE) {
- min_order = ilog2(mem->min_page_size) -
- ilog2(mem->mm.chunk_size);
- }
-
- if (flags & I915_ALLOC_CONTIGUOUS) {
- size = roundup_pow_of_two(size);
- min_order = ilog2(size) - ilog2(mem->mm.chunk_size);
+ list_for_each_entry_safe(reserve, next, &mem->reserved, link) {
+ list_del(&reserve->link);
+ mem->priv_ops->free(mem, reserve->node);
+ kfree(reserve);
}
-
- if (size > mem->mm.size)
- return -E2BIG;
-
- n_pages = size >> ilog2(mem->mm.chunk_size);
-
- mutex_lock(&mem->mm_lock);
-
- do {
- struct i915_buddy_block *block;
- unsigned int order;
-
- order = fls(n_pages) - 1;
- GEM_BUG_ON(order > mem->mm.max_order);
- GEM_BUG_ON(order < min_order);
-
- do {
- block = i915_buddy_alloc(&mem->mm, order);
- if (!IS_ERR(block))
- break;
-
- if (order-- == min_order)
- goto err_free_blocks;
- } while (1);
-
- n_pages -= BIT(order);
-
- block->private = mem;
- list_add_tail(&block->link, blocks);
-
- if (!n_pages)
- break;
- } while (1);
-
- mem->avail -= size;
mutex_unlock(&mem->mm_lock);
- return 0;
-
-err_free_blocks:
- intel_memory_region_free_pages(mem, blocks);
- mutex_unlock(&mem->mm_lock);
- return -ENXIO;
}
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags)
+/**
+ * intel_memory_region_reserve - Reserve a memory range
+ * @mem: The region for which we want to reserve a range.
+ * @offset: Start of the range to reserve.
+ * @size: The size of the range to reserve.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
{
- struct i915_buddy_block *block;
- LIST_HEAD(blocks);
int ret;
+ struct intel_region_reserve *reserve;
- ret = __intel_memory_region_get_pages_buddy(mem, size, flags, &blocks);
- if (ret)
- return ERR_PTR(ret);
+ if (!mem->priv_ops || !mem->priv_ops->reserve)
+ return -EINVAL;
- block = list_first_entry(&blocks, typeof(*block), link);
- list_del_init(&block->link);
- return block;
-}
+ reserve = kzalloc(sizeof(*reserve), GFP_KERNEL);
+ if (!reserve)
+ return -ENOMEM;
-int intel_memory_region_init_buddy(struct intel_memory_region *mem)
-{
- return i915_buddy_init(&mem->mm, resource_size(&mem->region),
- PAGE_SIZE);
-}
-
-void intel_memory_region_release_buddy(struct intel_memory_region *mem)
-{
- i915_buddy_free_list(&mem->mm, &mem->reserved);
- i915_buddy_fini(&mem->mm);
-}
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size)
-{
- int ret;
+ reserve->node = mem->priv_ops->reserve(mem, offset, size);
+ if (IS_ERR(reserve->node)) {
+ ret = PTR_ERR(reserve->node);
+ kfree(reserve);
+ return ret;
+ }
mutex_lock(&mem->mm_lock);
- ret = i915_buddy_alloc_range(&mem->mm, &mem->reserved, offset, size);
+ list_add_tail(&reserve->link, &mem->reserved);
mutex_unlock(&mem->mm_lock);
- return ret;
+ return 0;
}
struct intel_memory_region *
@@ -206,6 +126,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops)
{
struct intel_memory_region *mem;
@@ -222,6 +144,8 @@ intel_memory_region_create(struct drm_i915_private *i915,
mem->ops = ops;
mem->total = size;
mem->avail = mem->total;
+ mem->type = type;
+ mem->instance = instance;
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
@@ -259,6 +183,7 @@ static void __intel_memory_region_destroy(struct kref *kref)
struct intel_memory_region *mem =
container_of(kref, typeof(*mem), kref);
+ intel_memory_region_unreserve(mem);
if (mem->ops->release)
mem->ops->release(mem);
@@ -296,15 +221,15 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
instance = intel_region_map[i].instance;
switch (type) {
case INTEL_MEMORY_SYSTEM:
- mem = i915_gem_shmem_setup(i915);
+ mem = i915_gem_shmem_setup(i915, type, instance);
break;
case INTEL_MEMORY_STOLEN_LOCAL:
- mem = i915_gem_stolen_lmem_setup(i915);
+ mem = i915_gem_stolen_lmem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
case INTEL_MEMORY_STOLEN_SYSTEM:
- mem = i915_gem_stolen_smem_setup(i915);
+ mem = i915_gem_stolen_smem_setup(i915, type, instance);
if (!IS_ERR(mem))
i915->mm.stolen_region = mem;
break;
@@ -321,9 +246,6 @@ int intel_memory_regions_hw_probe(struct drm_i915_private *i915)
}
mem->id = i;
- mem->type = type;
- mem->instance = instance;
-
i915->mm.regions[i] = mem;
}
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index d24ce5a0b30b..e69cde13daf2 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -13,8 +13,6 @@
#include <drm/drm_mm.h>
#include <drm/i915_drm.h>
-#include "i915_buddy.h"
-
struct drm_i915_private;
struct drm_i915_gem_object;
struct intel_memory_region;
@@ -25,6 +23,7 @@ enum intel_memory_type {
INTEL_MEMORY_LOCAL = I915_MEMORY_CLASS_DEVICE,
INTEL_MEMORY_STOLEN_SYSTEM,
INTEL_MEMORY_STOLEN_LOCAL,
+ INTEL_MEMORY_MOCK,
};
enum intel_region_id {
@@ -59,10 +58,19 @@ struct intel_memory_region_ops {
unsigned int flags);
};
+struct intel_memory_region_private_ops {
+ void *(*reserve)(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
+ void (*free)(struct intel_memory_region *mem,
+ void *node);
+};
+
struct intel_memory_region {
struct drm_i915_private *i915;
const struct intel_memory_region_ops *ops;
+ const struct intel_memory_region_private_ops *priv_ops;
struct io_mapping iomap;
struct resource region;
@@ -70,7 +78,6 @@ struct intel_memory_region {
/* For fake LMEM */
struct drm_mm_node fake_mappable;
- struct i915_buddy_mm mm;
struct mutex mm_lock;
struct kref kref;
@@ -95,36 +102,26 @@ struct intel_memory_region {
struct list_head list;
struct list_head purgeable;
} objects;
+
+ size_t chunk_size;
+ unsigned int max_order;
+ bool is_range_manager;
+
+ void *region_private;
};
struct intel_memory_region *
intel_memory_region_lookup(struct drm_i915_private *i915,
u16 class, u16 instance);
-int intel_memory_region_init_buddy(struct intel_memory_region *mem);
-void intel_memory_region_release_buddy(struct intel_memory_region *mem);
-
-int __intel_memory_region_get_pages_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags,
- struct list_head *blocks);
-struct i915_buddy_block *
-__intel_memory_region_get_block_buddy(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-void __intel_memory_region_put_pages_buddy(struct intel_memory_region *mem,
- struct list_head *blocks);
-void __intel_memory_region_put_block_buddy(struct i915_buddy_block *block);
-
-int intel_memory_region_reserve(struct intel_memory_region *mem,
- u64 offset, u64 size);
-
struct intel_memory_region *
intel_memory_region_create(struct drm_i915_private *i915,
resource_size_t start,
resource_size_t size,
resource_size_t min_page_size,
resource_size_t io_start,
+ u16 type,
+ u16 instance,
const struct intel_memory_region_ops *ops);
struct intel_memory_region *
@@ -141,4 +138,9 @@ __printf(2, 3) void
intel_memory_region_set_name(struct intel_memory_region *mem,
const char *fmt, ...);
+void intel_memory_region_unreserve(struct intel_memory_region *mem);
+
+int intel_memory_region_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size);
#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
new file mode 100644
index 000000000000..69c98298326b
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -0,0 +1,334 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_device.h>
+
+#include "i915_drv.h"
+#include "i915_scatterlist.h"
+
+#include "intel_region_ttm.h"
+
+/**
+ * DOC: TTM support structure
+ *
+ * The code in this file deals with setting up memory managers for TTM
+ * LMEM and MOCK regions and converting the output from
+ * the managers to struct sg_table, Basically providing the mapping from
+ * i915 GEM regions to TTM memory types and resource managers.
+ */
+
+/**
+ * struct intel_region_ttm_private - Private info for TTM regions
+ * @man: The TTM resource manager used to manage this region
+ * @mock_bdev: A mock zero-initialized TTM device for this region
+ */
+struct intel_region_ttm_private {
+ struct ttm_resource_manager *man;
+
+ I915_SELFTEST_DECLARE(struct ttm_device mock_bdev);
+ I915_SELFTEST_DECLARE(struct drm_vma_offset_manager mock_vma);
+};
+
+/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
+static struct ttm_device_funcs i915_ttm_bo_driver;
+
+static struct intel_region_ttm_private *to_ttm_private(void *data)
+{
+ return (struct intel_region_ttm_private *)data;
+}
+
+/**
+ * intel_region_ttm_device_init - Initialize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv)
+{
+ struct drm_device *drm = &dev_priv->drm;
+
+ return ttm_device_init(&dev_priv->bdev, &i915_ttm_bo_driver,
+ drm->dev, drm->anon_inode->i_mapping,
+ drm->vma_offset_manager, false, false);
+}
+
+/**
+ * intel_region_ttm_device_fini - Finalize a TTM device
+ * @dev_priv: Pointer to an i915 device private structure.
+ */
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv)
+{
+ ttm_device_fini(&dev_priv->bdev);
+}
+
+/*
+ * Map the i915 memory regions to TTM memory types. We use the
+ * driver-private types for now, reserving TTM_PL_VRAM for stolen
+ * memory and TTM_PL_TT for GGTT use if decided to implement this.
+ */
+static int intel_region_to_ttm_type(struct intel_memory_region *mem)
+{
+ int type;
+
+ GEM_BUG_ON(mem->type != INTEL_MEMORY_LOCAL &&
+ mem->type != INTEL_MEMORY_MOCK);
+
+ type = mem->instance + TTM_PL_PRIV;
+ GEM_BUG_ON(type >= TTM_NUM_MEM_TYPES);
+
+ return type;
+}
+
+static void *intel_region_ttm_node_reserve(struct intel_memory_region *mem,
+ resource_size_t offset,
+ resource_size_t size)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * Having to use a mock_bo is unfortunate but stems from some
+ * drivers having private managers that insist to know what the
+ * allocate memory is intended for, using it to send private
+ * data to the manager. Also recently the bo has been used to send
+ * alignment info to the manager. Assume that apart from the latter,
+ * none of the managers we use will ever access the buffer object
+ * members, hoping we can pass the alignment info in the
+ * struct ttm_place in the future.
+ */
+
+ place.fpfn = offset >> PAGE_SHIFT;
+ place.lpfn = place.fpfn + (size >> PAGE_SHIFT);
+ res.num_pages = size >> PAGE_SHIFT;
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+/**
+ * intel_region_ttm_node_free - Free a node allocated from a resource manager
+ * @mem: The region the node was allocated from.
+ * @node: The opaque node representing an allocation.
+ */
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_resource res = {};
+
+ res.mm_node = node;
+ man->func->free(man, &res);
+}
+
+static const struct intel_memory_region_private_ops priv_ops = {
+ .reserve = intel_region_ttm_node_reserve,
+ .free = intel_region_ttm_node_free,
+};
+
+static int __intel_region_ttm_init(struct intel_memory_region *mem,
+ struct intel_region_ttm_private *priv,
+ struct ttm_device *bdev)
+{
+ int mem_type = intel_region_to_ttm_type(mem);
+ int ret;
+
+ ret = ttm_range_man_init(bdev, mem_type, false,
+ resource_size(&mem->region) >> PAGE_SHIFT);
+ if (ret)
+ return ret;
+
+ priv->man = ttm_manager_type(bdev, mem_type);
+ mem->chunk_size = PAGE_SIZE;
+ mem->max_order =
+ get_order(rounddown_pow_of_two(resource_size(&mem->region)));
+ mem->is_range_manager = true;
+ mem->priv_ops = &priv_ops;
+ mem->region_private = priv;
+
+ return 0;
+}
+
+/**
+ * intel_region_ttm_init - Initialize a memory region for TTM.
+ * @mem: The region to initialize.
+ *
+ * This function initializes a suitable TTM resource manager for the
+ * region, attaches it to the TTM device.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_init(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv;
+ int ret;
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ ret = __intel_region_ttm_init(mem, priv, &mem->i915->bdev);
+ if (ret)
+ kfree(priv);
+ return ret;
+}
+
+/**
+ * intel_region_ttm_fini - Finalize a TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration.
+ */
+void intel_region_ttm_fini(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv =
+ to_ttm_private(mem->region_private);
+ int ret;
+
+ ret = ttm_range_man_fini(&mem->i915->bdev, intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ mem->region_private = NULL;
+ kfree(priv);
+}
+
+/**
+ * intel_region_ttm_node_to_st - Convert an opaque TTM resource manager node
+ * to an sg_table.
+ * @mem: The memory region.
+ * @node: The resource manager node obtained from the TTM resource manager.
+ *
+ * The gem backends typically use sg-tables for operations on the underlying
+ * io_memory. So provide a way for the backends to translate the
+ * nodes they are handed from TTM to sg-tables.
+ *
+ * Return: A malloced sg_table on success, an error pointer on failure.
+ */
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node)
+{
+ return i915_sg_from_mm_node(node, mem->region.start);
+}
+
+/**
+ * intel_region_ttm_node_alloc - Allocate memory resources from a region
+ * @mem: The memory region,
+ * @size: The requested size in bytes
+ * @flags: Allocation flags
+ *
+ * This functionality is provided only for callers that need to allocate
+ * memory from standalone TTM range managers, without the TTM eviction
+ * functionality. Don't use if you are not completely sure that's the
+ * case. The returned opaque node can be converted to an sg_table using
+ * intel_region_ttm_node_to_st(), and can be freed using
+ * intel_region_ttm_node_free().
+ *
+ * Return: A valid pointer on success, an error pointer on failure.
+ */
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags)
+{
+ struct ttm_resource_manager *man =
+ to_ttm_private(mem->region_private)->man;
+ struct ttm_place place = {};
+ struct ttm_resource res = {};
+ struct ttm_buffer_object mock_bo = {};
+ int ret;
+
+ /*
+ * We ignore the flags for now since we're using the range
+ * manager and contigous and min page size would be fulfilled
+ * by default if size is min page size aligned.
+ */
+ res.num_pages = size >> PAGE_SHIFT;
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ mock_bo.page_alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ mock_bo.page_alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ mock_bo.page_alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ ret = man->func->alloc(man, &mock_bo, &place, &res);
+ if (ret == -ENOSPC)
+ ret = -ENXIO;
+ return ret ? ERR_PTR(ret) : res.mm_node;
+}
+
+#ifdef CONFIG_DRM_I915_SELFTEST
+/**
+ * intel_region_ttm_init_mock - Initialize a mock memory region for TTM.
+ * @mem: The region to initialize.
+ *
+ * This function initializes a suitable TTM resource manager for the
+ * region, attaches it to the TTM device.
+ * This code is for mock selftests only.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int intel_region_ttm_init_mock(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv;
+ int ret;
+
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+
+ drm_vma_offset_manager_init(&priv->mock_vma,
+ DRM_FILE_PAGE_OFFSET_START,
+ DRM_FILE_PAGE_OFFSET_SIZE);
+ ret = ttm_device_init(&priv->mock_bdev, &i915_ttm_bo_driver,
+ NULL, NULL, &priv->mock_vma, false, false);
+ if (ret)
+ goto out_device;
+
+ ret = __intel_region_ttm_init(mem, priv, &priv->mock_bdev);
+ if (ret)
+ goto out_region;
+
+ return 0;
+
+out_region:
+ ttm_device_fini(&priv->mock_bdev);
+ drm_vma_offset_manager_destroy(&priv->mock_vma);
+out_device:
+ kfree(priv);
+ return ret;
+}
+
+/**
+ * intel_region_ttm_fini_mock - Finalize a mock TTM region.
+ * @mem: The memory region
+ *
+ * This functions takes down the TTM resource manager associated with the
+ * memory region, and if it was registered with the TTM device,
+ * removes that registration. Finally destroys the mock TTM device and
+ * vma manager. This code is for mock selftests only.
+ */
+void intel_region_ttm_fini_mock(struct intel_memory_region *mem)
+{
+ struct intel_region_ttm_private *priv =
+ to_ttm_private(mem->region_private);
+ int ret;
+
+ ret = ttm_range_man_fini(&priv->mock_bdev, intel_region_to_ttm_type(mem));
+ GEM_WARN_ON(ret);
+ ttm_device_fini(&priv->mock_bdev);
+ drm_vma_offset_manager_destroy(&priv->mock_vma);
+ mem->region_private = NULL;
+ kfree(priv);
+}
+#endif
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
new file mode 100644
index 000000000000..23c6c2d4b1cd
--- /dev/null
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -0,0 +1,38 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _INTEL_REGION_TTM_H_
+#define _INTEL_REGION_TTM_H_
+
+#include <linux/types.h>
+
+#include "i915_selftest.h"
+
+struct drm_i915_private;
+struct intel_memory_region;
+
+int intel_region_ttm_device_init(struct drm_i915_private *dev_priv);
+
+void intel_region_ttm_device_fini(struct drm_i915_private *dev_priv);
+
+int intel_region_ttm_init(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini(struct intel_memory_region *mem);
+
+struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
+ void *node);
+
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
+void intel_region_ttm_node_free(struct intel_memory_region *mem,
+ void *node);
+
+#ifdef CONFIG_DRM_I915_SELFTEST
+int intel_region_ttm_init_mock(struct intel_memory_region *mem);
+
+void intel_region_ttm_fini_mock(struct intel_memory_region *mem);
+#endif
+#endif /* _INTEL_REGION_TTM_H_ */
diff --git a/drivers/gpu/drm/i915/selftests/i915_buddy.c b/drivers/gpu/drm/i915/selftests/i915_buddy.c
deleted file mode 100644
index f0f5c4df8dbc..000000000000
--- a/drivers/gpu/drm/i915/selftests/i915_buddy.c
+++ /dev/null
@@ -1,789 +0,0 @@
-// SPDX-License-Identifier: MIT
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#include <linux/prime_numbers.h>
-
-#include "../i915_selftest.h"
-#include "i915_random.h"
-
-static void __igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block,
- bool buddy)
-{
- pr_err("block info: header=%llx, state=%u, order=%d, offset=%llx size=%llx root=%s buddy=%s\n",
- block->header,
- i915_buddy_block_state(block),
- i915_buddy_block_order(block),
- i915_buddy_block_offset(block),
- i915_buddy_block_size(mm, block),
- yesno(!block->parent),
- yesno(buddy));
-}
-
-static void igt_dump_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
-
- __igt_dump_block(mm, block, false);
-
- buddy = get_buddy(block);
- if (buddy)
- __igt_dump_block(mm, buddy, true);
-}
-
-static int igt_check_block(struct i915_buddy_mm *mm,
- struct i915_buddy_block *block)
-{
- struct i915_buddy_block *buddy;
- unsigned int block_state;
- u64 block_size;
- u64 offset;
- int err = 0;
-
- block_state = i915_buddy_block_state(block);
-
- if (block_state != I915_BUDDY_ALLOCATED &&
- block_state != I915_BUDDY_FREE &&
- block_state != I915_BUDDY_SPLIT) {
- pr_err("block state mismatch\n");
- err = -EINVAL;
- }
-
- block_size = i915_buddy_block_size(mm, block);
- offset = i915_buddy_block_offset(block);
-
- if (block_size < mm->chunk_size) {
- pr_err("block size smaller than min size\n");
- err = -EINVAL;
- }
-
- if (!is_power_of_2(block_size)) {
- pr_err("block size not power of two\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(block_size, mm->chunk_size)) {
- pr_err("block size not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, mm->chunk_size)) {
- pr_err("block offset not aligned to min size\n");
- err = -EINVAL;
- }
-
- if (!IS_ALIGNED(offset, block_size)) {
- pr_err("block offset not aligned to block size\n");
- err = -EINVAL;
- }
-
- buddy = get_buddy(block);
-
- if (!buddy && block->parent) {
- pr_err("buddy has gone fishing\n");
- err = -EINVAL;
- }
-
- if (buddy) {
- if (i915_buddy_block_offset(buddy) != (offset ^ block_size)) {
- pr_err("buddy has wrong offset\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_size(mm, buddy) != block_size) {
- pr_err("buddy size mismatch\n");
- err = -EINVAL;
- }
-
- if (i915_buddy_block_state(buddy) == block_state &&
- block_state == I915_BUDDY_FREE) {
- pr_err("block and its buddy are free\n");
- err = -EINVAL;
- }
- }
-
- return err;
-}
-
-static int igt_check_blocks(struct i915_buddy_mm *mm,
- struct list_head *blocks,
- u64 expected_size,
- bool is_contiguous)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_block *prev;
- u64 total;
- int err = 0;
-
- block = NULL;
- prev = NULL;
- total = 0;
-
- list_for_each_entry(block, blocks, link) {
- err = igt_check_block(mm, block);
-
- if (!i915_buddy_block_is_allocated(block)) {
- pr_err("block not allocated\n"),
- err = -EINVAL;
- }
-
- if (is_contiguous && prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(block);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("block offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- if (err)
- break;
-
- total += i915_buddy_block_size(mm, block);
- prev = block;
- }
-
- if (!err) {
- if (total != expected_size) {
- pr_err("size mismatch, expected=%llx, found=%llx\n",
- expected_size, total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev block, dump:\n");
- igt_dump_block(mm, prev);
- }
-
- if (block) {
- pr_err("bad block, dump:\n");
- igt_dump_block(mm, block);
- }
-
- return err;
-}
-
-static int igt_check_mm(struct i915_buddy_mm *mm)
-{
- struct i915_buddy_block *root;
- struct i915_buddy_block *prev;
- unsigned int i;
- u64 total;
- int err = 0;
-
- if (!mm->n_roots) {
- pr_err("n_roots is zero\n");
- return -EINVAL;
- }
-
- if (mm->n_roots != hweight64(mm->size)) {
- pr_err("n_roots mismatch, n_roots=%u, expected=%lu\n",
- mm->n_roots, hweight64(mm->size));
- return -EINVAL;
- }
-
- root = NULL;
- prev = NULL;
- total = 0;
-
- for (i = 0; i < mm->n_roots; ++i) {
- struct i915_buddy_block *block;
- unsigned int order;
-
- root = mm->roots[i];
- if (!root) {
- pr_err("root(%u) is NULL\n", i);
- err = -EINVAL;
- break;
- }
-
- err = igt_check_block(mm, root);
-
- if (!i915_buddy_block_is_free(root)) {
- pr_err("root not free\n");
- err = -EINVAL;
- }
-
- order = i915_buddy_block_order(root);
-
- if (!i) {
- if (order != mm->max_order) {
- pr_err("max order root missing\n");
- err = -EINVAL;
- }
- }
-
- if (prev) {
- u64 prev_block_size;
- u64 prev_offset;
- u64 offset;
-
- prev_offset = i915_buddy_block_offset(prev);
- prev_block_size = i915_buddy_block_size(mm, prev);
- offset = i915_buddy_block_offset(root);
-
- if (offset != (prev_offset + prev_block_size)) {
- pr_err("root offset mismatch\n");
- err = -EINVAL;
- }
- }
-
- block = list_first_entry_or_null(&mm->free_list[order],
- struct i915_buddy_block,
- link);
- if (block != root) {
- pr_err("root mismatch at order=%u\n", order);
- err = -EINVAL;
- }
-
- if (err)
- break;
-
- prev = root;
- total += i915_buddy_block_size(mm, root);
- }
-
- if (!err) {
- if (total != mm->size) {
- pr_err("expected mm size=%llx, found=%llx\n", mm->size,
- total);
- err = -EINVAL;
- }
- return err;
- }
-
- if (prev) {
- pr_err("prev root(%u), dump:\n", i - 1);
- igt_dump_block(mm, prev);
- }
-
- if (root) {
- pr_err("bad root(%u), dump:\n", i);
- igt_dump_block(mm, root);
- }
-
- return err;
-}
-
-static void igt_mm_config(u64 *size, u64 *chunk_size)
-{
- I915_RND_STATE(prng);
- u32 s, ms;
-
- /* Nothing fancy, just try to get an interesting bit pattern */
-
- prandom_seed_state(&prng, i915_selftest.random_seed);
-
- /* Let size be a random number of pages up to 8 GB (2M pages) */
- s = 1 + i915_prandom_u32_max_state((BIT(33 - 12)) - 1, &prng);
- /* Let the chunk size be a random power of 2 less than size */
- ms = BIT(i915_prandom_u32_max_state(ilog2(s), &prng));
- /* Round size down to the chunk size */
- s &= -ms;
-
- /* Convert from pages to bytes */
- *chunk_size = (u64)ms << 12;
- *size = (u64)s << 12;
-}
-
-static int igt_buddy_alloc_smoke(void *arg)
-{
- struct i915_buddy_mm mm;
- IGT_TIMEOUT(end_time);
- I915_RND_STATE(prng);
- u64 chunk_size;
- u64 mm_size;
- int *order;
- int err, i;
-
- igt_mm_config(&mm_size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", mm_size, chunk_size);
-
- err = i915_buddy_init(&mm, mm_size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- order = i915_random_order(mm.max_order + 1, &prng);
- if (!order)
- goto out_fini;
-
- for (i = 0; i <= mm.max_order; ++i) {
- struct i915_buddy_block *block;
- int max_order = order[i];
- bool timeout = false;
- LIST_HEAD(blocks);
- int order;
- u64 total;
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort\n");
- break;
- }
-
- pr_info("filling from max_order=%u\n", max_order);
-
- order = max_order;
- total = 0;
-
- do {
-retry:
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- if (err == -ENOMEM) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- } else {
- if (order--) {
- err = 0;
- goto retry;
- }
-
- pr_err("buddy_alloc with order=%d failed(%d)\n",
- order, err);
- }
-
- break;
- }
-
- list_add_tail(&block->link, &blocks);
-
- if (i915_buddy_block_order(block) != order) {
- pr_err("buddy_alloc order mismatch\n");
- err = -EINVAL;
- break;
- }
-
- total += i915_buddy_block_size(&mm, block);
-
- if (__igt_timeout(end_time, NULL)) {
- timeout = true;
- break;
- }
- } while (total < mm.size);
-
- if (!err)
- err = igt_check_blocks(&mm, &blocks, total, false);
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
- if (err || timeout)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- kfree(order);
-out_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_pessimistic(void *arg)
-{
- const unsigned int max_order = 16;
- struct i915_buddy_block *block, *bn;
- struct i915_buddy_mm mm;
- unsigned int order;
- LIST_HEAD(blocks);
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order < max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* And now the last remaining block available */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM on final alloc\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
-
- /* Should be completely full! */
- for (order = max_order; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- block = list_last_entry(&blocks, typeof(*block), link);
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- /* As we free in increasing size, we make available larger blocks */
- order = 1;
- list_for_each_entry_safe(block, bn, &blocks, link) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
-
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
- order++;
- }
-
- /* To confirm, now the whole mm should be available */
- block = i915_buddy_alloc(&mm, max_order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc (realloc) hit -ENOMEM with order=%d\n",
- max_order);
- err = PTR_ERR(block);
- goto err;
- }
- i915_buddy_free(&mm, block);
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_optimistic(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- int order;
- int err;
-
- /*
- * Create a mm with one block of each order available, and
- * try to allocate them all.
- */
-
- err = i915_buddy_init(&mm,
- PAGE_SIZE * ((1 << (max_order + 1)) - 1),
- PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (order = 0; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d\n",
- order);
- err = PTR_ERR(block);
- goto err;
- }
-
- list_add_tail(&block->link, &blocks);
- }
-
- /* Should be completely full! */
- block = i915_buddy_alloc(&mm, 0);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded, it should be full!");
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
-
-err:
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_pathological(void *arg)
-{
- const int max_order = 16;
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- LIST_HEAD(blocks);
- LIST_HEAD(holes);
- int order, top;
- int err;
-
- /*
- * Create a pot-sized mm, then allocate one of each possible
- * order within. This should leave the mm with exactly one
- * page left. Free the largest block, then whittle down again.
- * Eventually we will have a fully 50% fragmented mm.
- */
-
- err = i915_buddy_init(&mm, PAGE_SIZE << max_order, PAGE_SIZE);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
- GEM_BUG_ON(mm.max_order != max_order);
-
- for (top = max_order; top; top--) {
- /* Make room by freeing the largest allocated block */
- block = list_first_entry_or_null(&blocks, typeof(*block), link);
- if (block) {
- list_del(&block->link);
- i915_buddy_free(&mm, block);
- }
-
- for (order = top; order--; ) {
- block = i915_buddy_alloc(&mm, order);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM with order=%d, top=%d\n",
- order, top);
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &blocks);
- }
-
- /* There should be one final page for this sub-allocation */
- block = i915_buddy_alloc(&mm, 0);
- if (IS_ERR(block)) {
- pr_info("buddy_alloc hit -ENOMEM for hole\n");
- err = PTR_ERR(block);
- goto err;
- }
- list_add_tail(&block->link, &holes);
-
- block = i915_buddy_alloc(&mm, top);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at top-order %d/%d, it should be full!",
- top, max_order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
- i915_buddy_free_list(&mm, &holes);
-
- /* Nothing larger than blocks of chunk_size now available */
- for (order = 1; order <= max_order; order++) {
- block = i915_buddy_alloc(&mm, order);
- if (!IS_ERR(block)) {
- pr_info("buddy_alloc unexpectedly succeeded at order %d, it should be full!",
- order);
- list_add_tail(&block->link, &blocks);
- err = -EINVAL;
- goto err;
- }
- }
-
-err:
- list_splice_tail(&holes, &blocks);
- i915_buddy_free_list(&mm, &blocks);
- i915_buddy_fini(&mm);
- return err;
-}
-
-static int igt_buddy_alloc_range(void *arg)
-{
- struct i915_buddy_mm mm;
- unsigned long page_num;
- LIST_HEAD(blocks);
- u64 chunk_size;
- u64 offset;
- u64 size;
- u64 rem;
- int err;
-
- igt_mm_config(&size, &chunk_size);
-
- pr_info("buddy_init with size=%llx, chunk_size=%llx\n", size, chunk_size);
-
- err = i915_buddy_init(&mm, size, chunk_size);
- if (err) {
- pr_err("buddy_init failed(%d)\n", err);
- return err;
- }
-
- err = igt_check_mm(&mm);
- if (err) {
- pr_err("pre-mm check failed, abort, abort, abort!\n");
- goto err_fini;
- }
-
- rem = mm.size;
- offset = 0;
-
- for_each_prime_number_from(page_num, 1, ULONG_MAX - 1) {
- struct i915_buddy_block *block;
- LIST_HEAD(tmp);
-
- size = min(page_num * mm.chunk_size, rem);
-
- err = i915_buddy_alloc_range(&mm, &tmp, offset, size);
- if (err) {
- if (err == -ENOMEM) {
- pr_info("alloc_range hit -ENOMEM with size=%llx\n",
- size);
- } else {
- pr_err("alloc_range with offset=%llx, size=%llx failed(%d)\n",
- offset, size, err);
- }
-
- break;
- }
-
- block = list_first_entry_or_null(&tmp,
- struct i915_buddy_block,
- link);
- if (!block) {
- pr_err("alloc_range has no blocks\n");
- err = -EINVAL;
- break;
- }
-
- if (i915_buddy_block_offset(block) != offset) {
- pr_err("alloc_range start offset mismatch, found=%llx, expected=%llx\n",
- i915_buddy_block_offset(block), offset);
- err = -EINVAL;
- }
-
- if (!err)
- err = igt_check_blocks(&mm, &tmp, size, true);
-
- list_splice_tail(&tmp, &blocks);
-
- if (err)
- break;
-
- offset += size;
-
- rem -= size;
- if (!rem)
- break;
-
- cond_resched();
- }
-
- if (err == -ENOMEM)
- err = 0;
-
- i915_buddy_free_list(&mm, &blocks);
-
- if (!err) {
- err = igt_check_mm(&mm);
- if (err)
- pr_err("post-mm check failed\n");
- }
-
-err_fini:
- i915_buddy_fini(&mm);
-
- return err;
-}
-
-static int igt_buddy_alloc_limit(void *arg)
-{
- struct i915_buddy_block *block;
- struct i915_buddy_mm mm;
- const u64 size = U64_MAX;
- int err;
-
- err = i915_buddy_init(&mm, size, PAGE_SIZE);
- if (err)
- return err;
-
- if (mm.max_order != I915_BUDDY_MAX_ORDER) {
- pr_err("mm.max_order(%d) != %d\n",
- mm.max_order, I915_BUDDY_MAX_ORDER);
- err = -EINVAL;
- goto out_fini;
- }
-
- block = i915_buddy_alloc(&mm, mm.max_order);
- if (IS_ERR(block)) {
- err = PTR_ERR(block);
- goto out_fini;
- }
-
- if (i915_buddy_block_order(block) != mm.max_order) {
- pr_err("block order(%d) != %d\n",
- i915_buddy_block_order(block), mm.max_order);
- err = -EINVAL;
- goto out_free;
- }
-
- if (i915_buddy_block_size(&mm, block) !=
- BIT_ULL(mm.max_order) * PAGE_SIZE) {
- pr_err("block size(%llu) != %llu\n",
- i915_buddy_block_size(&mm, block),
- BIT_ULL(mm.max_order) * PAGE_SIZE);
- err = -EINVAL;
- goto out_free;
- }
-
-out_free:
- i915_buddy_free(&mm, block);
-out_fini:
- i915_buddy_fini(&mm);
- return err;
-}
-
-int i915_buddy_mock_selftests(void)
-{
- static const struct i915_subtest tests[] = {
- SUBTEST(igt_buddy_alloc_pessimistic),
- SUBTEST(igt_buddy_alloc_optimistic),
- SUBTEST(igt_buddy_alloc_pathological),
- SUBTEST(igt_buddy_alloc_smoke),
- SUBTEST(igt_buddy_alloc_range),
- SUBTEST(igt_buddy_alloc_limit),
- };
-
- return i915_subtests(tests, NULL);
-}
diff --git a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
index 3db34d3eea58..34e5caf38093 100644
--- a/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
+++ b/drivers/gpu/drm/i915/selftests/i915_mock_selftests.h
@@ -33,5 +33,4 @@ selftest(evict, i915_gem_evict_mock_selftests)
selftest(gtt, i915_gem_gtt_mock_selftests)
selftest(hugepages, i915_gem_huge_page_mock_selftests)
selftest(contexts, i915_gem_context_mock_selftests)
-selftest(buddy, i915_buddy_mock_selftests)
selftest(memory_region, intel_memory_region_mock_selftests)
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index f85fd8cbfbf5..c85d516b85cd 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -57,9 +57,10 @@ static int igt_mock_fill(void *arg)
LIST_HEAD(objects);
int err = 0;
- page_size = mem->mm.chunk_size;
- max_pages = div64_u64(total, page_size);
+ page_size = mem->chunk_size;
rem = total;
+retry:
+ max_pages = div64_u64(rem, page_size);
for_each_prime_number_from(page_num, 1, max_pages) {
resource_size_t size = page_num * page_size;
@@ -85,6 +86,11 @@ static int igt_mock_fill(void *arg)
err = 0;
if (err == -ENXIO) {
if (page_num * page_size <= rem) {
+ if (mem->is_range_manager && max_pages > 1) {
+ max_pages >>= 1;
+ goto retry;
+ }
+
pr_err("%s failed, space still left in region\n",
__func__);
err = -EINVAL;
@@ -199,12 +205,18 @@ static int igt_mock_reserve(void *arg)
do {
u32 size = i915_prandom_u32_max_state(cur_avail, &prng);
+retry:
size = max_t(u32, round_up(size, PAGE_SIZE), PAGE_SIZE);
obj = igt_object_create(mem, &objects, size, 0);
if (IS_ERR(obj)) {
- if (PTR_ERR(obj) == -ENXIO)
+ if (PTR_ERR(obj) == -ENXIO) {
+ if (mem->is_range_manager &&
+ size > mem->chunk_size) {
+ size >>= 1;
+ goto retry;
+ }
break;
-
+ }
err = PTR_ERR(obj);
goto out_close;
}
@@ -220,7 +232,7 @@ static int igt_mock_reserve(void *arg)
out_close:
kfree(order);
close_objects(mem, &objects);
- i915_buddy_free_list(&mem->mm, &mem->reserved);
+ intel_memory_region_unreserve(mem);
return err;
}
@@ -240,7 +252,7 @@ static int igt_mock_contiguous(void *arg)
total = resource_size(&mem->region);
/* Min size */
- obj = igt_object_create(mem, &objects, mem->mm.chunk_size,
+ obj = igt_object_create(mem, &objects, mem->chunk_size,
I915_BO_ALLOC_CONTIGUOUS);
if (IS_ERR(obj))
return PTR_ERR(obj);
@@ -321,14 +333,16 @@ static int igt_mock_contiguous(void *arg)
min = target;
target = total >> 1;
- /* Make sure we can still allocate all the fragmented space */
- obj = igt_object_create(mem, &objects, target, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto err_close_objects;
- }
+ if (!mem->is_range_manager) {
+ /* Make sure we can still allocate all the fragmented space */
+ obj = igt_object_create(mem, &objects, target, 0);
+ if (IS_ERR(obj)) {
+ err = PTR_ERR(obj);
+ goto err_close_objects;
+ }
- igt_object_release(obj);
+ igt_object_release(obj);
+ }
/*
* Even though we have enough free space, we don't have a big enough
@@ -348,7 +362,7 @@ static int igt_mock_contiguous(void *arg)
}
target >>= 1;
- } while (target >= mem->mm.chunk_size);
+ } while (target >= mem->chunk_size);
err_close_objects:
list_splice_tail(&holes, &objects);
@@ -368,7 +382,7 @@ static int igt_mock_splintered_region(void *arg)
/*
* Sanity check we can still allocate everything even if the
- * mm.max_order != mm.size. i.e our starting address space size is not a
+ * max_order != mm.size. i.e our starting address space size is not a
* power-of-two.
*/
@@ -377,17 +391,10 @@ static int igt_mock_splintered_region(void *arg)
if (IS_ERR(mem))
return PTR_ERR(mem);
- if (mem->mm.size != size) {
- pr_err("%s size mismatch(%llu != %llu)\n",
- __func__, mem->mm.size, size);
- err = -EINVAL;
- goto out_put;
- }
-
expected_order = get_order(rounddown_pow_of_two(size));
- if (mem->mm.max_order != expected_order) {
+ if (mem->max_order != expected_order) {
pr_err("%s order mismatch(%u != %u)\n",
- __func__, mem->mm.max_order, expected_order);
+ __func__, mem->max_order, expected_order);
err = -EINVAL;
goto out_put;
}
@@ -408,12 +415,15 @@ static int igt_mock_splintered_region(void *arg)
* sure that does indeed hold true.
*/
- obj = igt_object_create(mem, &objects, size, I915_BO_ALLOC_CONTIGUOUS);
- if (!IS_ERR(obj)) {
- pr_err("%s too large contiguous allocation was not rejected\n",
- __func__);
- err = -EINVAL;
- goto out_close;
+ if (!mem->is_range_manager) {
+ obj = igt_object_create(mem, &objects, size,
+ I915_BO_ALLOC_CONTIGUOUS);
+ if (!IS_ERR(obj)) {
+ pr_err("%s too large contiguous allocation was not rejected\n",
+ __func__);
+ err = -EINVAL;
+ goto out_close;
+ }
}
obj = igt_object_create(mem, &objects, rounddown_pow_of_two(size),
@@ -432,68 +442,6 @@ static int igt_mock_splintered_region(void *arg)
return err;
}
-#ifndef SZ_8G
-#define SZ_8G BIT_ULL(33)
-#endif
-
-static int igt_mock_max_segment(void *arg)
-{
- const unsigned int max_segment = i915_sg_segment_size();
- struct intel_memory_region *mem = arg;
- struct drm_i915_private *i915 = mem->i915;
- struct drm_i915_gem_object *obj;
- struct i915_buddy_block *block;
- struct scatterlist *sg;
- LIST_HEAD(objects);
- u64 size;
- int err = 0;
-
- /*
- * While we may create very large contiguous blocks, we may need
- * to break those down for consumption elsewhere. In particular,
- * dma-mapping with scatterlist elements have an implicit limit of
- * UINT_MAX on each element.
- */
-
- size = SZ_8G;
- mem = mock_region_create(i915, 0, size, PAGE_SIZE, 0);
- if (IS_ERR(mem))
- return PTR_ERR(mem);
-
- obj = igt_object_create(mem, &objects, size, 0);
- if (IS_ERR(obj)) {
- err = PTR_ERR(obj);
- goto out_put;
- }
-
- size = 0;
- list_for_each_entry(block, &obj->mm.blocks, link) {
- if (i915_buddy_block_size(&mem->mm, block) > size)
- size = i915_buddy_block_size(&mem->mm, block);
- }
- if (size < max_segment) {
- pr_err("%s: Failed to create a huge contiguous block [> %u], largest block %lld\n",
- __func__, max_segment, size);
- err = -EINVAL;
- goto out_close;
- }
-
- for (sg = obj->mm.pages->sgl; sg; sg = sg_next(sg)) {
- if (sg->length > max_segment) {
- pr_err("%s: Created an oversized scatterlist entry, %u > %u\n",
- __func__, sg->length, max_segment);
- err = -EINVAL;
- goto out_close;
- }
- }
-
-out_close:
- close_objects(mem, &objects);
-out_put:
- intel_memory_region_put(mem);
- return err;
-}
-
static int igt_gpu_write_dw(struct intel_context *ce,
struct i915_vma *vma,
u32 dword,
@@ -1098,7 +1046,6 @@ int intel_memory_region_mock_selftests(void)
SUBTEST(igt_mock_fill),
SUBTEST(igt_mock_contiguous),
SUBTEST(igt_mock_splintered_region),
- SUBTEST(igt_mock_max_segment),
};
struct intel_memory_region *mem;
struct drm_i915_private *i915;
diff --git a/drivers/gpu/drm/i915/selftests/mock_region.c b/drivers/gpu/drm/i915/selftests/mock_region.c
index 5d2d010a1e22..d61387fda945 100644
--- a/drivers/gpu/drm/i915/selftests/mock_region.c
+++ b/drivers/gpu/drm/i915/selftests/mock_region.c
@@ -1,17 +1,54 @@
// SPDX-License-Identifier: MIT
/*
- * Copyright © 2019 Intel Corporation
+ * Copyright © 2019-2021 Intel Corporation
*/
+#include <linux/scatterlist.h>
+
#include "gem/i915_gem_region.h"
#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
#include "mock_region.h"
+static void mock_region_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *pages)
+{
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ sg_free_table(pages);
+ kfree(pages);
+}
+
+static int mock_region_get_pages(struct drm_i915_gem_object *obj)
+{
+ unsigned int flags;
+ struct sg_table *pages;
+
+ flags = I915_ALLOC_MIN_PAGE_SIZE;
+ if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
+ flags |= I915_ALLOC_CONTIGUOUS;
+
+ obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
+ obj->base.size,
+ flags);
+ if (IS_ERR(obj->mm.st_mm_node))
+ return PTR_ERR(obj->mm.st_mm_node);
+
+ pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
+ if (IS_ERR(pages)) {
+ intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
+ return PTR_ERR(pages);
+ }
+
+ __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
+
+ return 0;
+}
+
static const struct drm_i915_gem_object_ops mock_region_obj_ops = {
.name = "mock-region",
- .get_pages = i915_gem_object_get_pages_buddy,
- .put_pages = i915_gem_object_put_pages_buddy,
+ .get_pages = mock_region_get_pages,
+ .put_pages = mock_region_put_pages,
.release = i915_gem_object_release_memory_region,
};
@@ -23,7 +60,7 @@ static int mock_object_init(struct intel_memory_region *mem,
static struct lock_class_key lock_class;
struct drm_i915_private *i915 = mem->i915;
- if (size > mem->mm.size)
+ if (size > resource_size(&mem->region))
return -E2BIG;
drm_gem_private_object_init(&i915->drm, &obj->base, size);
@@ -39,8 +76,8 @@ static int mock_object_init(struct intel_memory_region *mem,
}
static const struct intel_memory_region_ops mock_region_ops = {
- .init = intel_memory_region_init_buddy,
- .release = intel_memory_region_release_buddy,
+ .init = intel_region_ttm_init_mock,
+ .release = intel_region_ttm_fini_mock,
.init_object = mock_object_init,
};
@@ -52,5 +89,6 @@ mock_region_create(struct drm_i915_private *i915,
resource_size_t io_start)
{
return intel_memory_region_create(i915, start, size, min_page_size,
- io_start, &mock_region_ops);
+ io_start, INTEL_MEMORY_MOCK, 0,
+ &mock_region_ops);
}
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 05/12] drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Embed a struct ttm_buffer_object into the i915 gem object, making sure
we alias the gem object part. It's a bit unfortunate that the
struct ttm_buffer_ojbect embeds a gem object since we otherwise could
make the TTM part private to the TTM backend, and use the usual
i915 gem object for the other backends.
To make this a bit more storage efficient for the other backends,
we'd have to use a pointer for the gem object which would require
a lot of changes in the driver. We postpone that for later.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 7 +++++++
drivers/gpu/drm/i915/gem/i915_gem_object_types.h | 12 +++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 2be6109d0093..5706d471692d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -62,6 +62,13 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_object_ops *ops,
struct lock_class_key *key, unsigned flags)
{
+ /*
+ * A gem object is embedded both in a struct ttm_buffer_object :/ and
+ * in a drm_i915_gem_object. Make sure they are aliased.
+ */
+ BUILD_BUG_ON(offsetof(typeof(*obj), base) !=
+ offsetof(typeof(*obj), __do_not_access.base));
+
spin_lock_init(&obj->vma.lock);
INIT_LIST_HEAD(&obj->vma.list);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index f5b46d11e6e6..d047ea126029 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -10,6 +10,7 @@
#include <linux/mmu_notifier.h>
#include <drm/drm_gem.h>
+#include <drm/ttm/ttm_bo_api.h>
#include <uapi/drm/i915_drm.h>
#include "i915_active.h"
@@ -99,7 +100,16 @@ struct i915_gem_object_page_iter {
};
struct drm_i915_gem_object {
- struct drm_gem_object base;
+ /*
+ * We might have reason to revisit the below since it wastes
+ * a lot of space for non-ttm gem objects.
+ * In any case, always use the accessors for the ttm_buffer_object
+ * when accessing it.
+ */
+ union {
+ struct drm_gem_object base;
+ struct ttm_buffer_object __do_not_access;
+ };
const struct drm_i915_gem_object_ops *ops;
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 05/12] drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström
Embed a struct ttm_buffer_object into the i915 gem object, making sure
we alias the gem object part. It's a bit unfortunate that the
struct ttm_buffer_ojbect embeds a gem object since we otherwise could
make the TTM part private to the TTM backend, and use the usual
i915 gem object for the other backends.
To make this a bit more storage efficient for the other backends,
we'd have to use a pointer for the gem object which would require
a lot of changes in the driver. We postpone that for later.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Acked-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
drivers/gpu/drm/i915/gem/i915_gem_object.c | 7 +++++++
drivers/gpu/drm/i915/gem/i915_gem_object_types.h | 12 +++++++++++-
2 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 2be6109d0093..5706d471692d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -62,6 +62,13 @@ void i915_gem_object_init(struct drm_i915_gem_object *obj,
const struct drm_i915_gem_object_ops *ops,
struct lock_class_key *key, unsigned flags)
{
+ /*
+ * A gem object is embedded both in a struct ttm_buffer_object :/ and
+ * in a drm_i915_gem_object. Make sure they are aliased.
+ */
+ BUILD_BUG_ON(offsetof(typeof(*obj), base) !=
+ offsetof(typeof(*obj), __do_not_access.base));
+
spin_lock_init(&obj->vma.lock);
INIT_LIST_HEAD(&obj->vma.list);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index f5b46d11e6e6..d047ea126029 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -10,6 +10,7 @@
#include <linux/mmu_notifier.h>
#include <drm/drm_gem.h>
+#include <drm/ttm/ttm_bo_api.h>
#include <uapi/drm/i915_drm.h>
#include "i915_active.h"
@@ -99,7 +100,16 @@ struct i915_gem_object_page_iter {
};
struct drm_i915_gem_object {
- struct drm_gem_object base;
+ /*
+ * We might have reason to revisit the below since it wastes
+ * a lot of space for non-ttm gem objects.
+ * In any case, always use the accessors for the ttm_buffer_object
+ * when accessing it.
+ */
+ union {
+ struct drm_gem_object base;
+ struct ttm_buffer_object __do_not_access;
+ };
const struct drm_i915_gem_object_ops *ops;
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
The internal ttm_bo_util memcpy uses ioremap functionality, and while it
probably might be possible to use it for copying in- and out of
sglist represented io memory, using io_mem_reserve() / io_mem_free()
callbacks, that would cause problems with fault().
Instead, implement a method mapping page-by-page using kmap_local()
semantics. As an additional benefit we then avoid the occasional global
TLB flushes of ioremap() and consuming ioremap space, elimination of a
critical point of failure and with a slight change of semantics we could
also push the memcpy out async for testing and async driver development
purposes.
A special linear iomem iterator is introduced internally to mimic the
old ioremap behaviour for code-paths that can't immediately be ported
over. This adds to the code size and should be considered a temporary
solution.
Looking at the code we have a lot of checks for iomap tagged pointers.
Ideally we should extend the core memremap functions to also accept
uncached memory and kmap_local functionality. Then we could strip a
lot of code.
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Split up in various TTM files and addressed review comments by
Christian König. Tested and fixed legacy iomap memcpy path on i915.
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
include/drm/ttm/ttm_bo_driver.h | 28 +++
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
include/drm/ttm/ttm_resource.h | 61 +++++++
include/drm/ttm/ttm_tt.h | 16 ++
9 files changed, 508 insertions(+), 181 deletions(-)
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ae8b61460724..912cbe8e60a2 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
mem->bus.addr = NULL;
}
-static int ttm_resource_ioremap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void **virtual)
+/**
+ * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
+ * @bo: The struct ttm_buffer_object.
+ * @new_mem: The struct ttm_resource we're moving to (copy destination).
+ * @new_iter: A struct ttm_kmap_iter representing the destination resource.
+ * @src_iter: A struct ttm_kmap_iter representing the source resource.
+ *
+ * This function is intended to be able to move out async under a
+ * dma-fence if desired.
+ */
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ struct ttm_resource *dst_mem,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter)
{
- int ret;
- void *addr;
-
- *virtual = NULL;
- ret = ttm_mem_io_reserve(bdev, mem);
- if (ret || !mem->bus.is_iomem)
- return ret;
+ const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
+ const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
+ struct ttm_tt *ttm = bo->ttm;
+ struct dma_buf_map src_map, dst_map;
+ pgoff_t i;
- if (mem->bus.addr) {
- addr = mem->bus.addr;
- } else {
- size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+ /* Single TTM move. NOP */
+ if (dst_ops->maps_tt && src_ops->maps_tt)
+ return;
- if (mem->bus.caching == ttm_write_combined)
- addr = ioremap_wc(mem->bus.offset, bus_size);
-#ifdef CONFIG_X86
- else if (mem->bus.caching == ttm_cached)
- addr = ioremap_cache(mem->bus.offset, bus_size);
-#endif
- else
- addr = ioremap(mem->bus.offset, bus_size);
- if (!addr) {
- ttm_mem_io_free(bdev, mem);
- return -ENOMEM;
+ /* Don't move nonexistent data. Clear destination instead. */
+ if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
+ if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
+ return;
+
+ for (i = 0; i < dst_mem->num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ if (dst_map.is_iomem)
+ memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
+ else
+ memset(dst_map.vaddr, 0, PAGE_SIZE);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
}
+ return;
}
- *virtual = addr;
- return 0;
-}
-
-static void ttm_resource_iounmap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void *virtual)
-{
- if (virtual && mem->bus.addr == NULL)
- iounmap(virtual);
- ttm_mem_io_free(bdev, mem);
-}
-
-static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
-{
- uint32_t *dstP =
- (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
- uint32_t *srcP =
- (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
-
- int i;
- for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
- iowrite32(ioread32(srcP++), dstP++);
- return 0;
-}
-
-static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *d = ttm->pages[page];
- void *dst;
-
- if (!d)
- return -ENOMEM;
-
- src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
- dst = kmap_atomic_prot(d, prot);
- if (!dst)
- return -ENOMEM;
-
- memcpy_fromio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(dst);
-
- return 0;
-}
-
-static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *s = ttm->pages[page];
- void *src;
-
- if (!s)
- return -ENOMEM;
-
- dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
- src = kmap_atomic_prot(s, prot);
- if (!src)
- return -ENOMEM;
- memcpy_toio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(src);
+ for (i = 0; i < dst_mem->num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ src_ops->map_local(src_iter, &src_map, i);
+
+ if (!src_map.is_iomem && !dst_map.is_iomem) {
+ memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
+ } else if (!src_map.is_iomem) {
+ dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
+ PAGE_SIZE);
+ } else if (!dst_map.is_iomem) {
+ memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
+ PAGE_SIZE);
+ } else {
+ int j;
+ u32 __iomem *src = src_map.vaddr_iomem;
+ u32 __iomem *dst = dst_map.vaddr_iomem;
- return 0;
+ for (j = 0; j < (PAGE_SIZE >> 2); ++j)
+ iowrite32(ioread32(src++), dst++);
+ }
+ if (src_ops->unmap_local)
+ src_ops->unmap_local(src_iter, &src_map);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
+ }
}
+EXPORT_SYMBOL(ttm_move_memcpy);
int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
struct ttm_operation_ctx *ctx,
- struct ttm_resource *new_mem)
+ struct ttm_resource *dst_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
struct ttm_tt *ttm = bo->ttm;
- struct ttm_resource *old_mem = &bo->mem;
- struct ttm_resource old_copy = *old_mem;
- void *old_iomap;
- void *new_iomap;
+ struct ttm_resource *src_mem = &bo->mem;
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bdev, src_mem->mem_type);
+ struct ttm_resource src_copy = *src_mem;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_linear_io io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
int ret;
- unsigned long i;
- ret = ttm_bo_wait_ctx(bo, ctx);
- if (ret)
- return ret;
-
- ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
- if (ret)
- return ret;
- ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
- if (ret)
- goto out;
-
- /*
- * Single TTM move. NOP.
- */
- if (old_iomap == NULL && new_iomap == NULL)
- goto out2;
-
- /*
- * Don't move nonexistent data. Clear destination instead.
- */
- if (old_iomap == NULL &&
- (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
- !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
- memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
- goto out2;
- }
-
- /*
- * TTM might be null for moves within the same region.
- */
- if (ttm) {
+ if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
+ dst_man->use_tt)) {
ret = ttm_tt_populate(bdev, ttm, ctx);
if (ret)
- goto out1;
+ return ret;
}
- for (i = 0; i < new_mem->num_pages; ++i) {
- if (old_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
- ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
- prot);
- } else if (new_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
- ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
- prot);
- } else {
- ret = ttm_copy_io_page(new_iomap, old_iomap, i);
- }
- if (ret)
- goto out1;
+ dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
+ if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
+ dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
+ if (IS_ERR(dst_iter))
+ return PTR_ERR(dst_iter);
+
+ src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
+ if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
+ src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
+ if (IS_ERR(src_iter)) {
+ ret = PTR_ERR(src_iter);
+ goto out_src_iter;
}
- mb();
-out2:
- old_copy = *old_mem;
- ttm_bo_assign_mem(bo, new_mem);
-
- if (!man->use_tt)
- ttm_bo_tt_destroy(bo);
+ ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
+ src_copy = *src_mem;
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
-out1:
- ttm_resource_iounmap(bdev, old_mem, new_iomap);
-out:
- ttm_resource_iounmap(bdev, &old_copy, old_iomap);
+ if (!src_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
+out_src_iter:
+ if (!dst_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
- /*
- * On error, keep the mm node!
- */
- if (!ret)
- ttm_resource_free(bo, &old_copy);
return ret;
}
EXPORT_SYMBOL(ttm_bo_move_memcpy);
@@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
man = ttm_manager_type(bo->bdev, res->mem_type);
caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
- /* Cached mappings need no adjustment */
- if (caching == ttm_cached)
- return tmp;
-
-#if defined(__i386__) || defined(__x86_64__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else if (boot_cpu_data.x86 > 3)
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
- defined(__powerpc__) || defined(__mips__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__sparc__)
- tmp = pgprot_noncached(tmp);
-#endif
- return tmp;
+ return ttm_prot_from_caching(caching, tmp);
}
EXPORT_SYMBOL(ttm_io_prot);
diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
index 56b0efdba1a9..997c458f68a9 100644
--- a/drivers/gpu/drm/ttm/ttm_module.c
+++ b/drivers/gpu/drm/ttm/ttm_module.c
@@ -31,12 +31,47 @@
*/
#include <linux/module.h>
#include <linux/device.h>
+#include <linux/pgtable.h>
#include <linux/sched.h>
#include <linux/debugfs.h>
#include <drm/drm_sysfs.h>
+#include <drm/ttm/ttm_caching.h>
#include "ttm_module.h"
+/**
+ * ttm_prot_from_caching - Modify the page protection according to the
+ * ttm cacing mode
+ * @caching: The ttm caching mode
+ * @tmp: The original page protection
+ *
+ * Return: The modified page protection
+ */
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
+{
+ /* Cached mappings need no adjustment */
+ if (caching == ttm_cached)
+ return tmp;
+
+#if defined(__i386__) || defined(__x86_64__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else if (boot_cpu_data.x86 > 3)
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
+ defined(__powerpc__) || defined(__mips__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__sparc__)
+ tmp = pgprot_noncached(tmp);
+#endif
+ return tmp;
+}
+
struct dentry *ttm_debugfs_root;
static int __init ttm_init(void)
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 59e2b7157e41..e05ae7e3d477 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -22,6 +22,10 @@
* Authors: Christian König
*/
+#include <linux/dma-buf-map.h>
+#include <linux/io-mapping.h>
+#include <linux/scatterlist.h>
+
#include <drm/ttm/ttm_resource.h>
#include <drm/ttm/ttm_bo_driver.h>
@@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
man->func->debug(man, p);
}
EXPORT_SYMBOL(ttm_resource_manager_debug);
+
+static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_iomap *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+ void __iomem *addr;
+
+retry:
+ while (i >= iter_io->cache.end) {
+ iter_io->cache.sg = iter_io->cache.sg ?
+ sg_next(iter_io->cache.sg) : iter_io->st->sgl;
+ iter_io->cache.i = iter_io->cache.end;
+ iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
+ PAGE_SHIFT;
+ iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
+ iter_io->start;
+ }
+
+ if (i < iter_io->cache.i) {
+ iter_io->cache.end = 0;
+ iter_io->cache.sg = NULL;
+ goto retry;
+ }
+
+ addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
+ (((resource_size_t)i - iter_io->cache.i)
+ << PAGE_SHIFT));
+ dma_buf_map_set_vaddr_iomem(dmap, addr);
+}
+
+static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ io_mapping_unmap_local(map->vaddr_iomem);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
+ .map_local = ttm_kmap_iter_iomap_map_local,
+ .unmap_local = ttm_kmap_iter_iomap_unmap_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
+ * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
+ * @iomap: The struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct
+ * ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start)
+{
+ iter_io->base.ops = &ttm_kmap_iter_io_ops;
+ iter_io->iomap = iomap;
+ iter_io->st = st;
+ iter_io->start = start;
+ memset(&iter_io->cache, 0, sizeof(iter_io->cache));
+
+ return &iter_io->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
+
+/**
+ * DOC: Linear io iterator
+ *
+ * This code should die in the not too near future. Best would be if we could
+ * make io-mapping use memremap for all io memory, and have memremap
+ * implement a kmap_local functionality. We could then strip a huge amount of
+ * code. These linear io iterators are implemented to mimic old functionality,
+ * and they don't use kmap_local semantics at all internally. Rather ioremap or
+ * friends, and at least on 32-bit they add global TLB flushes and points
+ * of failure.
+ */
+
+static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_linear_io *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+
+ *dmap = iter_io->dmap;
+ dma_buf_map_incr(dmap, i * PAGE_SIZE);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
+ .map_local = ttm_kmap_iter_linear_io_map_local,
+ .maps_tt = false,
+};
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ int ret;
+
+ ret = ttm_mem_io_reserve(bdev, mem);
+ if (ret)
+ goto out_err;
+ if (!mem->bus.is_iomem) {
+ ret = -EINVAL;
+ goto out_io_free;
+ }
+
+ if (mem->bus.addr) {
+ dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
+ iter_io->needs_unmap = false;
+ } else {
+ size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+
+ iter_io->needs_unmap = true;
+ if (mem->bus.caching == ttm_write_combined)
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap_wc(mem->bus.offset,
+ bus_size));
+ else if (mem->bus.caching == ttm_cached)
+ dma_buf_map_set_vaddr(&iter_io->dmap,
+ memremap(mem->bus.offset, bus_size,
+ MEMREMAP_WB));
+ else
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap(mem->bus.offset,
+ bus_size));
+ if (dma_buf_map_is_null(&iter_io->dmap)) {
+ ret = -ENOMEM;
+ goto out_io_free;
+ }
+ }
+
+ iter_io->base.ops = &ttm_kmap_iter_linear_io_ops;
+ return &iter_io->base;
+
+out_io_free:
+ ttm_mem_io_free(bdev, mem);
+out_err:
+ return ERR_PTR(ret);
+}
+
+void
+ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ if (iter_io->needs_unmap && dma_buf_map_is_set(&iter_io->dmap)) {
+ if (iter_io->dmap.is_iomem)
+ iounmap(iter_io->dmap.vaddr_iomem);
+ else
+ memunmap(iter_io->dmap.vaddr);
+ }
+
+ ttm_mem_io_free(bdev, mem);
+}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 539e0232cb3b..0e41227116b1 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -433,3 +433,45 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages)
if (!ttm_dma32_pages_limit)
ttm_dma32_pages_limit = num_dma32_pages;
}
+
+static void ttm_kmap_iter_tt_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_tt *iter_tt =
+ container_of(iter, typeof(*iter_tt), base);
+
+ dma_buf_map_set_vaddr(dmap, kmap_local_page_prot(iter_tt->tt->pages[i],
+ iter_tt->prot));
+}
+
+static void ttm_kmap_iter_tt_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ kunmap_local(map->vaddr);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_tt_ops = {
+ .map_local = ttm_kmap_iter_tt_map_local,
+ .unmap_local = ttm_kmap_iter_tt_unmap_local,
+ .maps_tt = true,
+};
+
+/**
+ * ttm_kmap_iter_tt_init - Initialize a struct ttm_kmap_iter_tt
+ * @iter_tt: The struct ttm_kmap_iter_tt to initialize.
+ * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt)
+{
+ iter_tt->base.ops = &ttm_kmap_iter_tt_ops;
+ iter_tt->tt = tt;
+ iter_tt->prot = ttm_prot_from_caching(tt->caching, PAGE_KERNEL);
+
+ return &iter_tt->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index dbccac957f8f..61fa648ee9e4 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -40,6 +40,7 @@
#include <drm/ttm/ttm_device.h>
#include "ttm_bo_api.h"
+#include "ttm_kmap_iter.h"
#include "ttm_placement.h"
#include "ttm_tt.h"
#include "ttm_pool.h"
@@ -272,6 +273,23 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
bool pipeline,
struct ttm_resource *new_mem);
+/**
+ * ttm_bo_move_accel_cleanup.
+ *
+ * @bo: A pointer to a struct ttm_buffer_object.
+ * @new_mem: struct ttm_resource indicating where to move.
+ *
+ * Special case of ttm_bo_move_accel_cleanup where the bo is guaranteed
+ * by the caller to be idle. Typically used after memcpy buffer moves.
+ */
+static inline void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem)
+{
+ int ret = ttm_bo_move_accel_cleanup(bo, NULL, true, false, new_mem);
+
+ WARN_ON(ret);
+}
+
/**
* ttm_bo_pipeline_gutting.
*
@@ -332,4 +350,14 @@ int ttm_range_man_init(struct ttm_device *bdev,
int ttm_range_man_fini(struct ttm_device *bdev,
unsigned type);
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter);
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
#endif
diff --git a/include/drm/ttm/ttm_caching.h b/include/drm/ttm/ttm_caching.h
index a0b4a49fa432..3c9dd65f5aaf 100644
--- a/include/drm/ttm/ttm_caching.h
+++ b/include/drm/ttm/ttm_caching.h
@@ -33,4 +33,6 @@ enum ttm_caching {
ttm_cached
};
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp);
+
#endif
diff --git a/include/drm/ttm/ttm_kmap_iter.h b/include/drm/ttm/ttm_kmap_iter.h
new file mode 100644
index 000000000000..8bb00fd39d6c
--- /dev/null
+++ b/include/drm/ttm/ttm_kmap_iter.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef __TTM_KMAP_ITER_H__
+#define __TTM_KMAP_ITER_H__
+
+#include <linux/types.h>
+
+struct ttm_kmap_iter;
+struct dma_buf_map;
+
+/**
+ * struct ttm_kmap_iter_ops - Ops structure for a struct
+ * ttm_kmap_iter.
+ * @maps_tt: Whether the iterator maps TT memory directly, as opposed
+ * mapping a TT through an aperture. Both these modes have
+ * struct ttm_resource_manager::use_tt set, but the latter typically
+ * returns is_iomem == true from ttm_mem_io_reserve.
+ */
+struct ttm_kmap_iter_ops {
+ /**
+ * kmap_local() - Map a PAGE_SIZE part of the resource using
+ * kmap_local semantics.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ * @i: The location within the resource to map. PAGE_SIZE granularity.
+ */
+ void (*map_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap, pgoff_t i);
+ /**
+ * unmap_local() - Unmap a PAGE_SIZE part of the resource previously
+ * mapped using kmap_local.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ */
+ void (*unmap_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap);
+ bool maps_tt;
+};
+
+/**
+ * struct ttm_kmap_iter - Iterator for kmap_local type operations on a
+ * resource.
+ * @ops: Pointer to the operations struct.
+ *
+ * This struct is intended to be embedded in a resource-specific specialization
+ * implementing operations for the resource.
+ *
+ * Nothing stops us from extending the operations to vmap, vmap_pfn etc,
+ * replacing some or parts of the ttm_bo_util. cpu-map functionality.
+ */
+struct ttm_kmap_iter {
+ const struct ttm_kmap_iter_ops *ops;
+};
+
+#endif /* __TTM_KMAP_ITER_H__ */
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 890b9d369519..b8dc0bdb0da5 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -27,9 +27,11 @@
#include <linux/types.h>
#include <linux/mutex.h>
+#include <linux/dma-buf-map.h>
#include <linux/dma-fence.h>
#include <drm/drm_print.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
#define TTM_MAX_BO_PRIORITY 4U
@@ -38,6 +40,10 @@ struct ttm_resource_manager;
struct ttm_resource;
struct ttm_place;
struct ttm_buffer_object;
+struct dma_buf_map;
+struct io_mapping;
+struct sg_table;
+struct scatterlist;
struct ttm_resource_manager_func {
/**
@@ -176,6 +182,45 @@ struct ttm_resource {
struct ttm_bus_placement bus;
};
+/**
+ * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
+ * struct sg_table backed struct ttm_resource.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface.
+ * @iomap: struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ * @cache: Scatterlist traversal cache for fast lookups.
+ * @cache.sg: Pointer to the currently cached scatterlist segment.
+ * @cache.i: First index of @sg. PAGE_SIZE granularity.
+ * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity.
+ * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity.
+ */
+struct ttm_kmap_iter_iomap {
+ struct ttm_kmap_iter base;
+ struct io_mapping *iomap;
+ struct sg_table *st;
+ resource_size_t start;
+ struct {
+ struct scatterlist *sg;
+ pgoff_t i;
+ pgoff_t end;
+ pgoff_t offs;
+ } cache;
+};
+
+/**
+ * struct ttm_kmap_iter_linear_io - Iterator specialization for linear io
+ * @base: The base iterator
+ * @dmap: Points to the starting address of the region
+ * @needs_unmap: Whether we need to unmap on fini
+ */
+struct ttm_kmap_iter_linear_io {
+ struct ttm_kmap_iter base;
+ struct dma_buf_map dmap;
+ bool needs_unmap;
+};
+
/**
* ttm_resource_manager_set_used
*
@@ -237,4 +282,20 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
void ttm_resource_manager_debug(struct ttm_resource_manager *man,
struct drm_printer *p);
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
+
+struct ttm_kmap_iter_linear_io;
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
+
+void ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
#endif
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 134d09ef7766..3102059db726 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -29,6 +29,7 @@
#include <linux/types.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
struct ttm_bo_device;
struct ttm_tt;
@@ -69,6 +70,18 @@ struct ttm_tt {
enum ttm_caching caching;
};
+/**
+ * struct ttm_kmap_iter_tt - Specialization of a mappig iterator for a tt.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface
+ * @tt: Cached struct ttm_tt.
+ * @prot: Cached page protection for mapping.
+ */
+struct ttm_kmap_iter_tt {
+ struct ttm_kmap_iter base;
+ struct ttm_tt *tt;
+ pgprot_t prot;
+};
+
static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
{
return tt->page_flags & TTM_PAGE_FLAG_PRIV_POPULATED;
@@ -159,6 +172,9 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
+struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt);
+
#if IS_ENABLED(CONFIG_AGP)
#include <linux/agp_backend.h>
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
The internal ttm_bo_util memcpy uses ioremap functionality, and while it
probably might be possible to use it for copying in- and out of
sglist represented io memory, using io_mem_reserve() / io_mem_free()
callbacks, that would cause problems with fault().
Instead, implement a method mapping page-by-page using kmap_local()
semantics. As an additional benefit we then avoid the occasional global
TLB flushes of ioremap() and consuming ioremap space, elimination of a
critical point of failure and with a slight change of semantics we could
also push the memcpy out async for testing and async driver development
purposes.
A special linear iomem iterator is introduced internally to mimic the
old ioremap behaviour for code-paths that can't immediately be ported
over. This adds to the code size and should be considered a temporary
solution.
Looking at the code we have a lot of checks for iomap tagged pointers.
Ideally we should extend the core memremap functions to also accept
uncached memory and kmap_local functionality. Then we could strip a
lot of code.
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Split up in various TTM files and addressed review comments by
Christian König. Tested and fixed legacy iomap memcpy path on i915.
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
include/drm/ttm/ttm_bo_driver.h | 28 +++
include/drm/ttm/ttm_caching.h | 2 +
include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
include/drm/ttm/ttm_resource.h | 61 +++++++
include/drm/ttm/ttm_tt.h | 16 ++
9 files changed, 508 insertions(+), 181 deletions(-)
create mode 100644 include/drm/ttm/ttm_kmap_iter.h
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index ae8b61460724..912cbe8e60a2 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
mem->bus.addr = NULL;
}
-static int ttm_resource_ioremap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void **virtual)
+/**
+ * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
+ * @bo: The struct ttm_buffer_object.
+ * @new_mem: The struct ttm_resource we're moving to (copy destination).
+ * @new_iter: A struct ttm_kmap_iter representing the destination resource.
+ * @src_iter: A struct ttm_kmap_iter representing the source resource.
+ *
+ * This function is intended to be able to move out async under a
+ * dma-fence if desired.
+ */
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ struct ttm_resource *dst_mem,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter)
{
- int ret;
- void *addr;
-
- *virtual = NULL;
- ret = ttm_mem_io_reserve(bdev, mem);
- if (ret || !mem->bus.is_iomem)
- return ret;
+ const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
+ const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
+ struct ttm_tt *ttm = bo->ttm;
+ struct dma_buf_map src_map, dst_map;
+ pgoff_t i;
- if (mem->bus.addr) {
- addr = mem->bus.addr;
- } else {
- size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+ /* Single TTM move. NOP */
+ if (dst_ops->maps_tt && src_ops->maps_tt)
+ return;
- if (mem->bus.caching == ttm_write_combined)
- addr = ioremap_wc(mem->bus.offset, bus_size);
-#ifdef CONFIG_X86
- else if (mem->bus.caching == ttm_cached)
- addr = ioremap_cache(mem->bus.offset, bus_size);
-#endif
- else
- addr = ioremap(mem->bus.offset, bus_size);
- if (!addr) {
- ttm_mem_io_free(bdev, mem);
- return -ENOMEM;
+ /* Don't move nonexistent data. Clear destination instead. */
+ if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
+ if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
+ return;
+
+ for (i = 0; i < dst_mem->num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ if (dst_map.is_iomem)
+ memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
+ else
+ memset(dst_map.vaddr, 0, PAGE_SIZE);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
}
+ return;
}
- *virtual = addr;
- return 0;
-}
-
-static void ttm_resource_iounmap(struct ttm_device *bdev,
- struct ttm_resource *mem,
- void *virtual)
-{
- if (virtual && mem->bus.addr == NULL)
- iounmap(virtual);
- ttm_mem_io_free(bdev, mem);
-}
-
-static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
-{
- uint32_t *dstP =
- (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
- uint32_t *srcP =
- (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
-
- int i;
- for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
- iowrite32(ioread32(srcP++), dstP++);
- return 0;
-}
-
-static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *d = ttm->pages[page];
- void *dst;
-
- if (!d)
- return -ENOMEM;
-
- src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
- dst = kmap_atomic_prot(d, prot);
- if (!dst)
- return -ENOMEM;
-
- memcpy_fromio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(dst);
-
- return 0;
-}
-
-static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
- unsigned long page,
- pgprot_t prot)
-{
- struct page *s = ttm->pages[page];
- void *src;
-
- if (!s)
- return -ENOMEM;
-
- dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
- src = kmap_atomic_prot(s, prot);
- if (!src)
- return -ENOMEM;
- memcpy_toio(dst, src, PAGE_SIZE);
-
- kunmap_atomic(src);
+ for (i = 0; i < dst_mem->num_pages; ++i) {
+ dst_ops->map_local(dst_iter, &dst_map, i);
+ src_ops->map_local(src_iter, &src_map, i);
+
+ if (!src_map.is_iomem && !dst_map.is_iomem) {
+ memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
+ } else if (!src_map.is_iomem) {
+ dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
+ PAGE_SIZE);
+ } else if (!dst_map.is_iomem) {
+ memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
+ PAGE_SIZE);
+ } else {
+ int j;
+ u32 __iomem *src = src_map.vaddr_iomem;
+ u32 __iomem *dst = dst_map.vaddr_iomem;
- return 0;
+ for (j = 0; j < (PAGE_SIZE >> 2); ++j)
+ iowrite32(ioread32(src++), dst++);
+ }
+ if (src_ops->unmap_local)
+ src_ops->unmap_local(src_iter, &src_map);
+ if (dst_ops->unmap_local)
+ dst_ops->unmap_local(dst_iter, &dst_map);
+ }
}
+EXPORT_SYMBOL(ttm_move_memcpy);
int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
struct ttm_operation_ctx *ctx,
- struct ttm_resource *new_mem)
+ struct ttm_resource *dst_mem)
{
struct ttm_device *bdev = bo->bdev;
- struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
struct ttm_tt *ttm = bo->ttm;
- struct ttm_resource *old_mem = &bo->mem;
- struct ttm_resource old_copy = *old_mem;
- void *old_iomap;
- void *new_iomap;
+ struct ttm_resource *src_mem = &bo->mem;
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bdev, src_mem->mem_type);
+ struct ttm_resource src_copy = *src_mem;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_linear_io io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
int ret;
- unsigned long i;
- ret = ttm_bo_wait_ctx(bo, ctx);
- if (ret)
- return ret;
-
- ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
- if (ret)
- return ret;
- ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
- if (ret)
- goto out;
-
- /*
- * Single TTM move. NOP.
- */
- if (old_iomap == NULL && new_iomap == NULL)
- goto out2;
-
- /*
- * Don't move nonexistent data. Clear destination instead.
- */
- if (old_iomap == NULL &&
- (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
- !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
- memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
- goto out2;
- }
-
- /*
- * TTM might be null for moves within the same region.
- */
- if (ttm) {
+ if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
+ dst_man->use_tt)) {
ret = ttm_tt_populate(bdev, ttm, ctx);
if (ret)
- goto out1;
+ return ret;
}
- for (i = 0; i < new_mem->num_pages; ++i) {
- if (old_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
- ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
- prot);
- } else if (new_iomap == NULL) {
- pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
- ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
- prot);
- } else {
- ret = ttm_copy_io_page(new_iomap, old_iomap, i);
- }
- if (ret)
- goto out1;
+ dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
+ if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
+ dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
+ if (IS_ERR(dst_iter))
+ return PTR_ERR(dst_iter);
+
+ src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
+ if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
+ src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
+ if (IS_ERR(src_iter)) {
+ ret = PTR_ERR(src_iter);
+ goto out_src_iter;
}
- mb();
-out2:
- old_copy = *old_mem;
- ttm_bo_assign_mem(bo, new_mem);
-
- if (!man->use_tt)
- ttm_bo_tt_destroy(bo);
+ ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
+ src_copy = *src_mem;
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
-out1:
- ttm_resource_iounmap(bdev, old_mem, new_iomap);
-out:
- ttm_resource_iounmap(bdev, &old_copy, old_iomap);
+ if (!src_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
+out_src_iter:
+ if (!dst_iter->ops->maps_tt)
+ ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
- /*
- * On error, keep the mm node!
- */
- if (!ret)
- ttm_resource_free(bo, &old_copy);
return ret;
}
EXPORT_SYMBOL(ttm_bo_move_memcpy);
@@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
man = ttm_manager_type(bo->bdev, res->mem_type);
caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
- /* Cached mappings need no adjustment */
- if (caching == ttm_cached)
- return tmp;
-
-#if defined(__i386__) || defined(__x86_64__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else if (boot_cpu_data.x86 > 3)
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
- defined(__powerpc__) || defined(__mips__)
- if (caching == ttm_write_combined)
- tmp = pgprot_writecombine(tmp);
- else
- tmp = pgprot_noncached(tmp);
-#endif
-#if defined(__sparc__)
- tmp = pgprot_noncached(tmp);
-#endif
- return tmp;
+ return ttm_prot_from_caching(caching, tmp);
}
EXPORT_SYMBOL(ttm_io_prot);
diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
index 56b0efdba1a9..997c458f68a9 100644
--- a/drivers/gpu/drm/ttm/ttm_module.c
+++ b/drivers/gpu/drm/ttm/ttm_module.c
@@ -31,12 +31,47 @@
*/
#include <linux/module.h>
#include <linux/device.h>
+#include <linux/pgtable.h>
#include <linux/sched.h>
#include <linux/debugfs.h>
#include <drm/drm_sysfs.h>
+#include <drm/ttm/ttm_caching.h>
#include "ttm_module.h"
+/**
+ * ttm_prot_from_caching - Modify the page protection according to the
+ * ttm cacing mode
+ * @caching: The ttm caching mode
+ * @tmp: The original page protection
+ *
+ * Return: The modified page protection
+ */
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
+{
+ /* Cached mappings need no adjustment */
+ if (caching == ttm_cached)
+ return tmp;
+
+#if defined(__i386__) || defined(__x86_64__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else if (boot_cpu_data.x86 > 3)
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
+ defined(__powerpc__) || defined(__mips__)
+ if (caching == ttm_write_combined)
+ tmp = pgprot_writecombine(tmp);
+ else
+ tmp = pgprot_noncached(tmp);
+#endif
+#if defined(__sparc__)
+ tmp = pgprot_noncached(tmp);
+#endif
+ return tmp;
+}
+
struct dentry *ttm_debugfs_root;
static int __init ttm_init(void)
diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
index 59e2b7157e41..e05ae7e3d477 100644
--- a/drivers/gpu/drm/ttm/ttm_resource.c
+++ b/drivers/gpu/drm/ttm/ttm_resource.c
@@ -22,6 +22,10 @@
* Authors: Christian König
*/
+#include <linux/dma-buf-map.h>
+#include <linux/io-mapping.h>
+#include <linux/scatterlist.h>
+
#include <drm/ttm/ttm_resource.h>
#include <drm/ttm/ttm_bo_driver.h>
@@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
man->func->debug(man, p);
}
EXPORT_SYMBOL(ttm_resource_manager_debug);
+
+static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_iomap *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+ void __iomem *addr;
+
+retry:
+ while (i >= iter_io->cache.end) {
+ iter_io->cache.sg = iter_io->cache.sg ?
+ sg_next(iter_io->cache.sg) : iter_io->st->sgl;
+ iter_io->cache.i = iter_io->cache.end;
+ iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
+ PAGE_SHIFT;
+ iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
+ iter_io->start;
+ }
+
+ if (i < iter_io->cache.i) {
+ iter_io->cache.end = 0;
+ iter_io->cache.sg = NULL;
+ goto retry;
+ }
+
+ addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
+ (((resource_size_t)i - iter_io->cache.i)
+ << PAGE_SHIFT));
+ dma_buf_map_set_vaddr_iomem(dmap, addr);
+}
+
+static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ io_mapping_unmap_local(map->vaddr_iomem);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
+ .map_local = ttm_kmap_iter_iomap_map_local,
+ .unmap_local = ttm_kmap_iter_iomap_unmap_local,
+ .maps_tt = false,
+};
+
+/**
+ * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
+ * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
+ * @iomap: The struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct
+ * ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start)
+{
+ iter_io->base.ops = &ttm_kmap_iter_io_ops;
+ iter_io->iomap = iomap;
+ iter_io->st = st;
+ iter_io->start = start;
+ memset(&iter_io->cache, 0, sizeof(iter_io->cache));
+
+ return &iter_io->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
+
+/**
+ * DOC: Linear io iterator
+ *
+ * This code should die in the not too near future. Best would be if we could
+ * make io-mapping use memremap for all io memory, and have memremap
+ * implement a kmap_local functionality. We could then strip a huge amount of
+ * code. These linear io iterators are implemented to mimic old functionality,
+ * and they don't use kmap_local semantics at all internally. Rather ioremap or
+ * friends, and at least on 32-bit they add global TLB flushes and points
+ * of failure.
+ */
+
+static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_linear_io *iter_io =
+ container_of(iter, typeof(*iter_io), base);
+
+ *dmap = iter_io->dmap;
+ dma_buf_map_incr(dmap, i * PAGE_SIZE);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
+ .map_local = ttm_kmap_iter_linear_io_map_local,
+ .maps_tt = false,
+};
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ int ret;
+
+ ret = ttm_mem_io_reserve(bdev, mem);
+ if (ret)
+ goto out_err;
+ if (!mem->bus.is_iomem) {
+ ret = -EINVAL;
+ goto out_io_free;
+ }
+
+ if (mem->bus.addr) {
+ dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
+ iter_io->needs_unmap = false;
+ } else {
+ size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
+
+ iter_io->needs_unmap = true;
+ if (mem->bus.caching == ttm_write_combined)
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap_wc(mem->bus.offset,
+ bus_size));
+ else if (mem->bus.caching == ttm_cached)
+ dma_buf_map_set_vaddr(&iter_io->dmap,
+ memremap(mem->bus.offset, bus_size,
+ MEMREMAP_WB));
+ else
+ dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
+ ioremap(mem->bus.offset,
+ bus_size));
+ if (dma_buf_map_is_null(&iter_io->dmap)) {
+ ret = -ENOMEM;
+ goto out_io_free;
+ }
+ }
+
+ iter_io->base.ops = &ttm_kmap_iter_linear_io_ops;
+ return &iter_io->base;
+
+out_io_free:
+ ttm_mem_io_free(bdev, mem);
+out_err:
+ return ERR_PTR(ret);
+}
+
+void
+ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem)
+{
+ if (iter_io->needs_unmap && dma_buf_map_is_set(&iter_io->dmap)) {
+ if (iter_io->dmap.is_iomem)
+ iounmap(iter_io->dmap.vaddr_iomem);
+ else
+ memunmap(iter_io->dmap.vaddr);
+ }
+
+ ttm_mem_io_free(bdev, mem);
+}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 539e0232cb3b..0e41227116b1 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -433,3 +433,45 @@ void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages)
if (!ttm_dma32_pages_limit)
ttm_dma32_pages_limit = num_dma32_pages;
}
+
+static void ttm_kmap_iter_tt_map_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *dmap,
+ pgoff_t i)
+{
+ struct ttm_kmap_iter_tt *iter_tt =
+ container_of(iter, typeof(*iter_tt), base);
+
+ dma_buf_map_set_vaddr(dmap, kmap_local_page_prot(iter_tt->tt->pages[i],
+ iter_tt->prot));
+}
+
+static void ttm_kmap_iter_tt_unmap_local(struct ttm_kmap_iter *iter,
+ struct dma_buf_map *map)
+{
+ kunmap_local(map->vaddr);
+}
+
+static const struct ttm_kmap_iter_ops ttm_kmap_iter_tt_ops = {
+ .map_local = ttm_kmap_iter_tt_map_local,
+ .unmap_local = ttm_kmap_iter_tt_unmap_local,
+ .maps_tt = true,
+};
+
+/**
+ * ttm_kmap_iter_tt_init - Initialize a struct ttm_kmap_iter_tt
+ * @iter_tt: The struct ttm_kmap_iter_tt to initialize.
+ * @tt: Struct ttm_tt holding page pointers of the struct ttm_resource.
+ *
+ * Return: Pointer to the embedded struct ttm_kmap_iter.
+ */
+struct ttm_kmap_iter *
+ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt)
+{
+ iter_tt->base.ops = &ttm_kmap_iter_tt_ops;
+ iter_tt->tt = tt;
+ iter_tt->prot = ttm_prot_from_caching(tt->caching, PAGE_KERNEL);
+
+ return &iter_tt->base;
+}
+EXPORT_SYMBOL(ttm_kmap_iter_tt_init);
diff --git a/include/drm/ttm/ttm_bo_driver.h b/include/drm/ttm/ttm_bo_driver.h
index dbccac957f8f..61fa648ee9e4 100644
--- a/include/drm/ttm/ttm_bo_driver.h
+++ b/include/drm/ttm/ttm_bo_driver.h
@@ -40,6 +40,7 @@
#include <drm/ttm/ttm_device.h>
#include "ttm_bo_api.h"
+#include "ttm_kmap_iter.h"
#include "ttm_placement.h"
#include "ttm_tt.h"
#include "ttm_pool.h"
@@ -272,6 +273,23 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
bool pipeline,
struct ttm_resource *new_mem);
+/**
+ * ttm_bo_move_accel_cleanup.
+ *
+ * @bo: A pointer to a struct ttm_buffer_object.
+ * @new_mem: struct ttm_resource indicating where to move.
+ *
+ * Special case of ttm_bo_move_accel_cleanup where the bo is guaranteed
+ * by the caller to be idle. Typically used after memcpy buffer moves.
+ */
+static inline void ttm_bo_move_sync_cleanup(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem)
+{
+ int ret = ttm_bo_move_accel_cleanup(bo, NULL, true, false, new_mem);
+
+ WARN_ON(ret);
+}
+
/**
* ttm_bo_pipeline_gutting.
*
@@ -332,4 +350,14 @@ int ttm_range_man_init(struct ttm_device *bdev,
int ttm_range_man_fini(struct ttm_device *bdev,
unsigned type);
+void ttm_move_memcpy(struct ttm_buffer_object *bo,
+ struct ttm_resource *new_mem,
+ struct ttm_kmap_iter *dst_iter,
+ struct ttm_kmap_iter *src_iter);
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
#endif
diff --git a/include/drm/ttm/ttm_caching.h b/include/drm/ttm/ttm_caching.h
index a0b4a49fa432..3c9dd65f5aaf 100644
--- a/include/drm/ttm/ttm_caching.h
+++ b/include/drm/ttm/ttm_caching.h
@@ -33,4 +33,6 @@ enum ttm_caching {
ttm_cached
};
+pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp);
+
#endif
diff --git a/include/drm/ttm/ttm_kmap_iter.h b/include/drm/ttm/ttm_kmap_iter.h
new file mode 100644
index 000000000000..8bb00fd39d6c
--- /dev/null
+++ b/include/drm/ttm/ttm_kmap_iter.h
@@ -0,0 +1,61 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef __TTM_KMAP_ITER_H__
+#define __TTM_KMAP_ITER_H__
+
+#include <linux/types.h>
+
+struct ttm_kmap_iter;
+struct dma_buf_map;
+
+/**
+ * struct ttm_kmap_iter_ops - Ops structure for a struct
+ * ttm_kmap_iter.
+ * @maps_tt: Whether the iterator maps TT memory directly, as opposed
+ * mapping a TT through an aperture. Both these modes have
+ * struct ttm_resource_manager::use_tt set, but the latter typically
+ * returns is_iomem == true from ttm_mem_io_reserve.
+ */
+struct ttm_kmap_iter_ops {
+ /**
+ * kmap_local() - Map a PAGE_SIZE part of the resource using
+ * kmap_local semantics.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ * @i: The location within the resource to map. PAGE_SIZE granularity.
+ */
+ void (*map_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap, pgoff_t i);
+ /**
+ * unmap_local() - Unmap a PAGE_SIZE part of the resource previously
+ * mapped using kmap_local.
+ * @res_iter: Pointer to the struct ttm_kmap_iter representing
+ * the resource.
+ * @dmap: The struct dma_buf_map holding the virtual address after
+ * the operation.
+ */
+ void (*unmap_local)(struct ttm_kmap_iter *res_iter,
+ struct dma_buf_map *dmap);
+ bool maps_tt;
+};
+
+/**
+ * struct ttm_kmap_iter - Iterator for kmap_local type operations on a
+ * resource.
+ * @ops: Pointer to the operations struct.
+ *
+ * This struct is intended to be embedded in a resource-specific specialization
+ * implementing operations for the resource.
+ *
+ * Nothing stops us from extending the operations to vmap, vmap_pfn etc,
+ * replacing some or parts of the ttm_bo_util. cpu-map functionality.
+ */
+struct ttm_kmap_iter {
+ const struct ttm_kmap_iter_ops *ops;
+};
+
+#endif /* __TTM_KMAP_ITER_H__ */
diff --git a/include/drm/ttm/ttm_resource.h b/include/drm/ttm/ttm_resource.h
index 890b9d369519..b8dc0bdb0da5 100644
--- a/include/drm/ttm/ttm_resource.h
+++ b/include/drm/ttm/ttm_resource.h
@@ -27,9 +27,11 @@
#include <linux/types.h>
#include <linux/mutex.h>
+#include <linux/dma-buf-map.h>
#include <linux/dma-fence.h>
#include <drm/drm_print.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
#define TTM_MAX_BO_PRIORITY 4U
@@ -38,6 +40,10 @@ struct ttm_resource_manager;
struct ttm_resource;
struct ttm_place;
struct ttm_buffer_object;
+struct dma_buf_map;
+struct io_mapping;
+struct sg_table;
+struct scatterlist;
struct ttm_resource_manager_func {
/**
@@ -176,6 +182,45 @@ struct ttm_resource {
struct ttm_bus_placement bus;
};
+/**
+ * struct ttm_kmap_iter_iomap - Specialization for a struct io_mapping +
+ * struct sg_table backed struct ttm_resource.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface.
+ * @iomap: struct io_mapping representing the underlying linear io_memory.
+ * @st: sg_table into @iomap, representing the memory of the struct ttm_resource.
+ * @start: Offset that needs to be subtracted from @st to make
+ * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
+ * @cache: Scatterlist traversal cache for fast lookups.
+ * @cache.sg: Pointer to the currently cached scatterlist segment.
+ * @cache.i: First index of @sg. PAGE_SIZE granularity.
+ * @cache.end: Last index + 1 of @sg. PAGE_SIZE granularity.
+ * @cache.offs: First offset into @iomap of @sg. PAGE_SIZE granularity.
+ */
+struct ttm_kmap_iter_iomap {
+ struct ttm_kmap_iter base;
+ struct io_mapping *iomap;
+ struct sg_table *st;
+ resource_size_t start;
+ struct {
+ struct scatterlist *sg;
+ pgoff_t i;
+ pgoff_t end;
+ pgoff_t offs;
+ } cache;
+};
+
+/**
+ * struct ttm_kmap_iter_linear_io - Iterator specialization for linear io
+ * @base: The base iterator
+ * @dmap: Points to the starting address of the region
+ * @needs_unmap: Whether we need to unmap on fini
+ */
+struct ttm_kmap_iter_linear_io {
+ struct ttm_kmap_iter base;
+ struct dma_buf_map dmap;
+ bool needs_unmap;
+};
+
/**
* ttm_resource_manager_set_used
*
@@ -237,4 +282,20 @@ int ttm_resource_manager_evict_all(struct ttm_device *bdev,
void ttm_resource_manager_debug(struct ttm_resource_manager *man,
struct drm_printer *p);
+struct ttm_kmap_iter *
+ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
+ struct io_mapping *iomap,
+ struct sg_table *st,
+ resource_size_t start);
+
+struct ttm_kmap_iter_linear_io;
+
+struct ttm_kmap_iter *
+ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
+
+void ttm_kmap_iter_linear_io_fini(struct ttm_kmap_iter_linear_io *iter_io,
+ struct ttm_device *bdev,
+ struct ttm_resource *mem);
#endif
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 134d09ef7766..3102059db726 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -29,6 +29,7 @@
#include <linux/types.h>
#include <drm/ttm/ttm_caching.h>
+#include <drm/ttm/ttm_kmap_iter.h>
struct ttm_bo_device;
struct ttm_tt;
@@ -69,6 +70,18 @@ struct ttm_tt {
enum ttm_caching caching;
};
+/**
+ * struct ttm_kmap_iter_tt - Specialization of a mappig iterator for a tt.
+ * @base: Embedded struct ttm_kmap_iter providing the usage interface
+ * @tt: Cached struct ttm_tt.
+ * @prot: Cached page protection for mapping.
+ */
+struct ttm_kmap_iter_tt {
+ struct ttm_kmap_iter base;
+ struct ttm_tt *tt;
+ pgprot_t prot;
+};
+
static inline bool ttm_tt_is_populated(struct ttm_tt *tt)
{
return tt->page_flags & TTM_PAGE_FLAG_PRIV_POPULATED;
@@ -159,6 +172,9 @@ void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
+struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
+ struct ttm_tt *tt);
+
#if IS_ENABLED(CONFIG_AGP)
#include <linux/agp_backend.h>
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-25 9:18 ` Matthew Auld
-1 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 9:18 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
> probably might be possible to use it for copying in- and out of
> sglist represented io memory, using io_mem_reserve() / io_mem_free()
> callbacks, that would cause problems with fault().
> Instead, implement a method mapping page-by-page using kmap_local()
> semantics. As an additional benefit we then avoid the occasional global
> TLB flushes of ioremap() and consuming ioremap space, elimination of a
> critical point of failure and with a slight change of semantics we could
> also push the memcpy out async for testing and async driver development
> purposes.
>
> A special linear iomem iterator is introduced internally to mimic the
> old ioremap behaviour for code-paths that can't immediately be ported
> over. This adds to the code size and should be considered a temporary
> solution.
>
> Looking at the code we have a lot of checks for iomap tagged pointers.
> Ideally we should extend the core memremap functions to also accept
> uncached memory and kmap_local functionality. Then we could strip a
> lot of code.
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v3:
> - Split up in various TTM files and addressed review comments by
> Christian König. Tested and fixed legacy iomap memcpy path on i915.
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> include/drm/ttm/ttm_bo_driver.h | 28 +++
> include/drm/ttm/ttm_caching.h | 2 +
> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> include/drm/ttm/ttm_resource.h | 61 +++++++
> include/drm/ttm/ttm_tt.h | 16 ++
> 9 files changed, 508 insertions(+), 181 deletions(-)
> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index ae8b61460724..912cbe8e60a2 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
> mem->bus.addr = NULL;
> }
>
> -static int ttm_resource_ioremap(struct ttm_device *bdev,
> - struct ttm_resource *mem,
> - void **virtual)
> +/**
> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
> + * @bo: The struct ttm_buffer_object.
> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
> + *
> + * This function is intended to be able to move out async under a
> + * dma-fence if desired.
> + */
> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> + struct ttm_resource *dst_mem,
> + struct ttm_kmap_iter *dst_iter,
> + struct ttm_kmap_iter *src_iter)
> {
> - int ret;
> - void *addr;
> -
> - *virtual = NULL;
> - ret = ttm_mem_io_reserve(bdev, mem);
> - if (ret || !mem->bus.is_iomem)
> - return ret;
> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> + struct ttm_tt *ttm = bo->ttm;
> + struct dma_buf_map src_map, dst_map;
> + pgoff_t i;
>
> - if (mem->bus.addr) {
> - addr = mem->bus.addr;
> - } else {
> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> + /* Single TTM move. NOP */
> + if (dst_ops->maps_tt && src_ops->maps_tt)
> + return;
>
> - if (mem->bus.caching == ttm_write_combined)
> - addr = ioremap_wc(mem->bus.offset, bus_size);
> -#ifdef CONFIG_X86
> - else if (mem->bus.caching == ttm_cached)
> - addr = ioremap_cache(mem->bus.offset, bus_size);
> -#endif
> - else
> - addr = ioremap(mem->bus.offset, bus_size);
> - if (!addr) {
> - ttm_mem_io_free(bdev, mem);
> - return -ENOMEM;
> + /* Don't move nonexistent data. Clear destination instead. */
> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
> + return;
> +
> + for (i = 0; i < dst_mem->num_pages; ++i) {
> + dst_ops->map_local(dst_iter, &dst_map, i);
> + if (dst_map.is_iomem)
> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
> + else
> + memset(dst_map.vaddr, 0, PAGE_SIZE);
> + if (dst_ops->unmap_local)
> + dst_ops->unmap_local(dst_iter, &dst_map);
> }
> + return;
> }
> - *virtual = addr;
> - return 0;
> -}
> -
> -static void ttm_resource_iounmap(struct ttm_device *bdev,
> - struct ttm_resource *mem,
> - void *virtual)
> -{
> - if (virtual && mem->bus.addr == NULL)
> - iounmap(virtual);
> - ttm_mem_io_free(bdev, mem);
> -}
> -
> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
> -{
> - uint32_t *dstP =
> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
> - uint32_t *srcP =
> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
> -
> - int i;
> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> - iowrite32(ioread32(srcP++), dstP++);
> - return 0;
> -}
> -
> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> - unsigned long page,
> - pgprot_t prot)
> -{
> - struct page *d = ttm->pages[page];
> - void *dst;
> -
> - if (!d)
> - return -ENOMEM;
> -
> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
> - dst = kmap_atomic_prot(d, prot);
> - if (!dst)
> - return -ENOMEM;
> -
> - memcpy_fromio(dst, src, PAGE_SIZE);
> -
> - kunmap_atomic(dst);
> -
> - return 0;
> -}
> -
> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> - unsigned long page,
> - pgprot_t prot)
> -{
> - struct page *s = ttm->pages[page];
> - void *src;
> -
> - if (!s)
> - return -ENOMEM;
> -
> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
> - src = kmap_atomic_prot(s, prot);
> - if (!src)
> - return -ENOMEM;
>
> - memcpy_toio(dst, src, PAGE_SIZE);
> -
> - kunmap_atomic(src);
> + for (i = 0; i < dst_mem->num_pages; ++i) {
> + dst_ops->map_local(dst_iter, &dst_map, i);
> + src_ops->map_local(src_iter, &src_map, i);
> +
> + if (!src_map.is_iomem && !dst_map.is_iomem) {
> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> + } else if (!src_map.is_iomem) {
> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> + PAGE_SIZE);
> + } else if (!dst_map.is_iomem) {
> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
> + PAGE_SIZE);
> + } else {
> + int j;
> + u32 __iomem *src = src_map.vaddr_iomem;
> + u32 __iomem *dst = dst_map.vaddr_iomem;
>
> - return 0;
> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
IMO PAGE_SIZE / sizeof(u32) is easier to understand.
> + iowrite32(ioread32(src++), dst++);
> + }
> + if (src_ops->unmap_local)
> + src_ops->unmap_local(src_iter, &src_map);
> + if (dst_ops->unmap_local)
> + dst_ops->unmap_local(dst_iter, &dst_map);
> + }
> }
> +EXPORT_SYMBOL(ttm_move_memcpy);
>
> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> struct ttm_operation_ctx *ctx,
> - struct ttm_resource *new_mem)
> + struct ttm_resource *dst_mem)
> {
> struct ttm_device *bdev = bo->bdev;
> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
> + struct ttm_resource_manager *dst_man =
> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> struct ttm_tt *ttm = bo->ttm;
> - struct ttm_resource *old_mem = &bo->mem;
> - struct ttm_resource old_copy = *old_mem;
> - void *old_iomap;
> - void *new_iomap;
> + struct ttm_resource *src_mem = &bo->mem;
> + struct ttm_resource_manager *src_man =
> + ttm_manager_type(bdev, src_mem->mem_type);
> + struct ttm_resource src_copy = *src_mem;
> + union {
> + struct ttm_kmap_iter_tt tt;
> + struct ttm_kmap_iter_linear_io io;
> + } _dst_iter, _src_iter;
> + struct ttm_kmap_iter *dst_iter, *src_iter;
> int ret;
> - unsigned long i;
>
> - ret = ttm_bo_wait_ctx(bo, ctx);
> - if (ret)
> - return ret;
> -
> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> - if (ret)
> - return ret;
> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> - if (ret)
> - goto out;
> -
> - /*
> - * Single TTM move. NOP.
> - */
> - if (old_iomap == NULL && new_iomap == NULL)
> - goto out2;
> -
> - /*
> - * Don't move nonexistent data. Clear destination instead.
> - */
> - if (old_iomap == NULL &&
> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
> - goto out2;
> - }
> -
> - /*
> - * TTM might be null for moves within the same region.
> - */
> - if (ttm) {
> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
> + dst_man->use_tt)) {
> ret = ttm_tt_populate(bdev, ttm, ctx);
> if (ret)
> - goto out1;
> + return ret;
> }
>
> - for (i = 0; i < new_mem->num_pages; ++i) {
> - if (old_iomap == NULL) {
> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
> - prot);
> - } else if (new_iomap == NULL) {
> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
> - prot);
> - } else {
> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
> - }
> - if (ret)
> - goto out1;
> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
> + if (IS_ERR(dst_iter))
> + return PTR_ERR(dst_iter);
> +
> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
> + if (IS_ERR(src_iter)) {
> + ret = PTR_ERR(src_iter);
> + goto out_src_iter;
> }
> - mb();
> -out2:
> - old_copy = *old_mem;
>
> - ttm_bo_assign_mem(bo, new_mem);
> -
> - if (!man->use_tt)
> - ttm_bo_tt_destroy(bo);
> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> + src_copy = *src_mem;
> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>
> -out1:
> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> -out:
> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> + if (!src_iter->ops->maps_tt)
> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
> +out_src_iter:
> + if (!dst_iter->ops->maps_tt)
> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
>
> - /*
> - * On error, keep the mm node!
> - */
> - if (!ret)
> - ttm_resource_free(bo, &old_copy);
> return ret;
> }
> EXPORT_SYMBOL(ttm_bo_move_memcpy);
> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
> man = ttm_manager_type(bo->bdev, res->mem_type);
> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
>
> - /* Cached mappings need no adjustment */
> - if (caching == ttm_cached)
> - return tmp;
> -
> -#if defined(__i386__) || defined(__x86_64__)
> - if (caching == ttm_write_combined)
> - tmp = pgprot_writecombine(tmp);
> - else if (boot_cpu_data.x86 > 3)
> - tmp = pgprot_noncached(tmp);
> -#endif
> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> - defined(__powerpc__) || defined(__mips__)
> - if (caching == ttm_write_combined)
> - tmp = pgprot_writecombine(tmp);
> - else
> - tmp = pgprot_noncached(tmp);
> -#endif
> -#if defined(__sparc__)
> - tmp = pgprot_noncached(tmp);
> -#endif
> - return tmp;
> + return ttm_prot_from_caching(caching, tmp);
> }
> EXPORT_SYMBOL(ttm_io_prot);
>
> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
> index 56b0efdba1a9..997c458f68a9 100644
> --- a/drivers/gpu/drm/ttm/ttm_module.c
> +++ b/drivers/gpu/drm/ttm/ttm_module.c
> @@ -31,12 +31,47 @@
> */
> #include <linux/module.h>
> #include <linux/device.h>
> +#include <linux/pgtable.h>
> #include <linux/sched.h>
> #include <linux/debugfs.h>
> #include <drm/drm_sysfs.h>
> +#include <drm/ttm/ttm_caching.h>
>
> #include "ttm_module.h"
>
> +/**
> + * ttm_prot_from_caching - Modify the page protection according to the
> + * ttm cacing mode
> + * @caching: The ttm caching mode
> + * @tmp: The original page protection
> + *
> + * Return: The modified page protection
> + */
> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
> +{
> + /* Cached mappings need no adjustment */
> + if (caching == ttm_cached)
> + return tmp;
> +
> +#if defined(__i386__) || defined(__x86_64__)
> + if (caching == ttm_write_combined)
> + tmp = pgprot_writecombine(tmp);
> + else if (boot_cpu_data.x86 > 3)
> + tmp = pgprot_noncached(tmp);
> +#endif
> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> + defined(__powerpc__) || defined(__mips__)
> + if (caching == ttm_write_combined)
> + tmp = pgprot_writecombine(tmp);
> + else
> + tmp = pgprot_noncached(tmp);
> +#endif
> +#if defined(__sparc__)
> + tmp = pgprot_noncached(tmp);
> +#endif
> + return tmp;
> +}
> +
> struct dentry *ttm_debugfs_root;
>
> static int __init ttm_init(void)
> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
> index 59e2b7157e41..e05ae7e3d477 100644
> --- a/drivers/gpu/drm/ttm/ttm_resource.c
> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> @@ -22,6 +22,10 @@
> * Authors: Christian König
> */
>
> +#include <linux/dma-buf-map.h>
> +#include <linux/io-mapping.h>
> +#include <linux/scatterlist.h>
> +
> #include <drm/ttm/ttm_resource.h>
> #include <drm/ttm/ttm_bo_driver.h>
>
> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
> man->func->debug(man, p);
> }
> EXPORT_SYMBOL(ttm_resource_manager_debug);
> +
> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *dmap,
> + pgoff_t i)
> +{
> + struct ttm_kmap_iter_iomap *iter_io =
> + container_of(iter, typeof(*iter_io), base);
> + void __iomem *addr;
> +
> +retry:
> + while (i >= iter_io->cache.end) {
> + iter_io->cache.sg = iter_io->cache.sg ?
> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
> + iter_io->cache.i = iter_io->cache.end;
> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
> + PAGE_SHIFT;
> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
> + iter_io->start;
> + }
> +
> + if (i < iter_io->cache.i) {
> + iter_io->cache.end = 0;
> + iter_io->cache.sg = NULL;
> + goto retry;
> + }
> +
> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
> + (((resource_size_t)i - iter_io->cache.i)
> + << PAGE_SHIFT));
> + dma_buf_map_set_vaddr_iomem(dmap, addr);
> +}
> +
> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *map)
> +{
> + io_mapping_unmap_local(map->vaddr_iomem);
> +}
> +
> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> + .map_local = ttm_kmap_iter_iomap_map_local,
> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> + .maps_tt = false,
> +};
> +
> +/**
> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
> + * @st: sg_table into @iomap, representing the memory of the struct
> + * ttm_resource.
> + * @start: Offset that needs to be subtracted from @st to make
> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> + *
> + * Return: Pointer to the embedded struct ttm_kmap_iter.
> + */
> +struct ttm_kmap_iter *
> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> + struct io_mapping *iomap,
> + struct sg_table *st,
> + resource_size_t start)
> +{
> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> + iter_io->iomap = iomap;
> + iter_io->st = st;
> + iter_io->start = start;
> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> +
> + return &iter_io->base;
> +}
> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> +
> +/**
> + * DOC: Linear io iterator
> + *
> + * This code should die in the not too near future. Best would be if we could
> + * make io-mapping use memremap for all io memory, and have memremap
> + * implement a kmap_local functionality. We could then strip a huge amount of
> + * code. These linear io iterators are implemented to mimic old functionality,
> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
> + * friends, and at least on 32-bit they add global TLB flushes and points
> + * of failure.
> + */
> +
> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *dmap,
> + pgoff_t i)
> +{
> + struct ttm_kmap_iter_linear_io *iter_io =
> + container_of(iter, typeof(*iter_io), base);
> +
> + *dmap = iter_io->dmap;
> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> +}
> +
> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
> + .map_local = ttm_kmap_iter_linear_io_map_local,
> + .maps_tt = false,
> +};
> +
> +struct ttm_kmap_iter *
> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
> + struct ttm_device *bdev,
> + struct ttm_resource *mem)
> +{
> + int ret;
> +
> + ret = ttm_mem_io_reserve(bdev, mem);
> + if (ret)
> + goto out_err;
> + if (!mem->bus.is_iomem) {
> + ret = -EINVAL;
> + goto out_io_free;
> + }
> +
> + if (mem->bus.addr) {
> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
> + iter_io->needs_unmap = false;
> + } else {
> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> +
> + iter_io->needs_unmap = true;
> + if (mem->bus.caching == ttm_write_combined)
> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
> + ioremap_wc(mem->bus.offset,
> + bus_size));
> + else if (mem->bus.caching == ttm_cached)
> + dma_buf_map_set_vaddr(&iter_io->dmap,
> + memremap(mem->bus.offset, bus_size,
> + MEMREMAP_WB));
The comments in set_vaddr suggest that this is meant for
system-memory. Does that actually matter or is it just about not
losing the __iomem annotation on platforms where it matters?
Apparently cached device local is a thing. Also should this not be
wrapped in CONFIG_X86?
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-25 9:18 ` Matthew Auld
0 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 9:18 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
> probably might be possible to use it for copying in- and out of
> sglist represented io memory, using io_mem_reserve() / io_mem_free()
> callbacks, that would cause problems with fault().
> Instead, implement a method mapping page-by-page using kmap_local()
> semantics. As an additional benefit we then avoid the occasional global
> TLB flushes of ioremap() and consuming ioremap space, elimination of a
> critical point of failure and with a slight change of semantics we could
> also push the memcpy out async for testing and async driver development
> purposes.
>
> A special linear iomem iterator is introduced internally to mimic the
> old ioremap behaviour for code-paths that can't immediately be ported
> over. This adds to the code size and should be considered a temporary
> solution.
>
> Looking at the code we have a lot of checks for iomap tagged pointers.
> Ideally we should extend the core memremap functions to also accept
> uncached memory and kmap_local functionality. Then we could strip a
> lot of code.
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> v3:
> - Split up in various TTM files and addressed review comments by
> Christian König. Tested and fixed legacy iomap memcpy path on i915.
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> include/drm/ttm/ttm_bo_driver.h | 28 +++
> include/drm/ttm/ttm_caching.h | 2 +
> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> include/drm/ttm/ttm_resource.h | 61 +++++++
> include/drm/ttm/ttm_tt.h | 16 ++
> 9 files changed, 508 insertions(+), 181 deletions(-)
> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index ae8b61460724..912cbe8e60a2 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
> mem->bus.addr = NULL;
> }
>
> -static int ttm_resource_ioremap(struct ttm_device *bdev,
> - struct ttm_resource *mem,
> - void **virtual)
> +/**
> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
> + * @bo: The struct ttm_buffer_object.
> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
> + *
> + * This function is intended to be able to move out async under a
> + * dma-fence if desired.
> + */
> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> + struct ttm_resource *dst_mem,
> + struct ttm_kmap_iter *dst_iter,
> + struct ttm_kmap_iter *src_iter)
> {
> - int ret;
> - void *addr;
> -
> - *virtual = NULL;
> - ret = ttm_mem_io_reserve(bdev, mem);
> - if (ret || !mem->bus.is_iomem)
> - return ret;
> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> + struct ttm_tt *ttm = bo->ttm;
> + struct dma_buf_map src_map, dst_map;
> + pgoff_t i;
>
> - if (mem->bus.addr) {
> - addr = mem->bus.addr;
> - } else {
> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> + /* Single TTM move. NOP */
> + if (dst_ops->maps_tt && src_ops->maps_tt)
> + return;
>
> - if (mem->bus.caching == ttm_write_combined)
> - addr = ioremap_wc(mem->bus.offset, bus_size);
> -#ifdef CONFIG_X86
> - else if (mem->bus.caching == ttm_cached)
> - addr = ioremap_cache(mem->bus.offset, bus_size);
> -#endif
> - else
> - addr = ioremap(mem->bus.offset, bus_size);
> - if (!addr) {
> - ttm_mem_io_free(bdev, mem);
> - return -ENOMEM;
> + /* Don't move nonexistent data. Clear destination instead. */
> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
> + return;
> +
> + for (i = 0; i < dst_mem->num_pages; ++i) {
> + dst_ops->map_local(dst_iter, &dst_map, i);
> + if (dst_map.is_iomem)
> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
> + else
> + memset(dst_map.vaddr, 0, PAGE_SIZE);
> + if (dst_ops->unmap_local)
> + dst_ops->unmap_local(dst_iter, &dst_map);
> }
> + return;
> }
> - *virtual = addr;
> - return 0;
> -}
> -
> -static void ttm_resource_iounmap(struct ttm_device *bdev,
> - struct ttm_resource *mem,
> - void *virtual)
> -{
> - if (virtual && mem->bus.addr == NULL)
> - iounmap(virtual);
> - ttm_mem_io_free(bdev, mem);
> -}
> -
> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
> -{
> - uint32_t *dstP =
> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
> - uint32_t *srcP =
> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
> -
> - int i;
> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> - iowrite32(ioread32(srcP++), dstP++);
> - return 0;
> -}
> -
> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> - unsigned long page,
> - pgprot_t prot)
> -{
> - struct page *d = ttm->pages[page];
> - void *dst;
> -
> - if (!d)
> - return -ENOMEM;
> -
> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
> - dst = kmap_atomic_prot(d, prot);
> - if (!dst)
> - return -ENOMEM;
> -
> - memcpy_fromio(dst, src, PAGE_SIZE);
> -
> - kunmap_atomic(dst);
> -
> - return 0;
> -}
> -
> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> - unsigned long page,
> - pgprot_t prot)
> -{
> - struct page *s = ttm->pages[page];
> - void *src;
> -
> - if (!s)
> - return -ENOMEM;
> -
> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
> - src = kmap_atomic_prot(s, prot);
> - if (!src)
> - return -ENOMEM;
>
> - memcpy_toio(dst, src, PAGE_SIZE);
> -
> - kunmap_atomic(src);
> + for (i = 0; i < dst_mem->num_pages; ++i) {
> + dst_ops->map_local(dst_iter, &dst_map, i);
> + src_ops->map_local(src_iter, &src_map, i);
> +
> + if (!src_map.is_iomem && !dst_map.is_iomem) {
> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> + } else if (!src_map.is_iomem) {
> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> + PAGE_SIZE);
> + } else if (!dst_map.is_iomem) {
> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
> + PAGE_SIZE);
> + } else {
> + int j;
> + u32 __iomem *src = src_map.vaddr_iomem;
> + u32 __iomem *dst = dst_map.vaddr_iomem;
>
> - return 0;
> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
IMO PAGE_SIZE / sizeof(u32) is easier to understand.
> + iowrite32(ioread32(src++), dst++);
> + }
> + if (src_ops->unmap_local)
> + src_ops->unmap_local(src_iter, &src_map);
> + if (dst_ops->unmap_local)
> + dst_ops->unmap_local(dst_iter, &dst_map);
> + }
> }
> +EXPORT_SYMBOL(ttm_move_memcpy);
>
> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> struct ttm_operation_ctx *ctx,
> - struct ttm_resource *new_mem)
> + struct ttm_resource *dst_mem)
> {
> struct ttm_device *bdev = bo->bdev;
> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
> + struct ttm_resource_manager *dst_man =
> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> struct ttm_tt *ttm = bo->ttm;
> - struct ttm_resource *old_mem = &bo->mem;
> - struct ttm_resource old_copy = *old_mem;
> - void *old_iomap;
> - void *new_iomap;
> + struct ttm_resource *src_mem = &bo->mem;
> + struct ttm_resource_manager *src_man =
> + ttm_manager_type(bdev, src_mem->mem_type);
> + struct ttm_resource src_copy = *src_mem;
> + union {
> + struct ttm_kmap_iter_tt tt;
> + struct ttm_kmap_iter_linear_io io;
> + } _dst_iter, _src_iter;
> + struct ttm_kmap_iter *dst_iter, *src_iter;
> int ret;
> - unsigned long i;
>
> - ret = ttm_bo_wait_ctx(bo, ctx);
> - if (ret)
> - return ret;
> -
> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> - if (ret)
> - return ret;
> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> - if (ret)
> - goto out;
> -
> - /*
> - * Single TTM move. NOP.
> - */
> - if (old_iomap == NULL && new_iomap == NULL)
> - goto out2;
> -
> - /*
> - * Don't move nonexistent data. Clear destination instead.
> - */
> - if (old_iomap == NULL &&
> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
> - goto out2;
> - }
> -
> - /*
> - * TTM might be null for moves within the same region.
> - */
> - if (ttm) {
> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
> + dst_man->use_tt)) {
> ret = ttm_tt_populate(bdev, ttm, ctx);
> if (ret)
> - goto out1;
> + return ret;
> }
>
> - for (i = 0; i < new_mem->num_pages; ++i) {
> - if (old_iomap == NULL) {
> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
> - prot);
> - } else if (new_iomap == NULL) {
> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
> - prot);
> - } else {
> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
> - }
> - if (ret)
> - goto out1;
> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
> + if (IS_ERR(dst_iter))
> + return PTR_ERR(dst_iter);
> +
> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
> + if (IS_ERR(src_iter)) {
> + ret = PTR_ERR(src_iter);
> + goto out_src_iter;
> }
> - mb();
> -out2:
> - old_copy = *old_mem;
>
> - ttm_bo_assign_mem(bo, new_mem);
> -
> - if (!man->use_tt)
> - ttm_bo_tt_destroy(bo);
> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> + src_copy = *src_mem;
> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>
> -out1:
> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> -out:
> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> + if (!src_iter->ops->maps_tt)
> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
> +out_src_iter:
> + if (!dst_iter->ops->maps_tt)
> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
>
> - /*
> - * On error, keep the mm node!
> - */
> - if (!ret)
> - ttm_resource_free(bo, &old_copy);
> return ret;
> }
> EXPORT_SYMBOL(ttm_bo_move_memcpy);
> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
> man = ttm_manager_type(bo->bdev, res->mem_type);
> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
>
> - /* Cached mappings need no adjustment */
> - if (caching == ttm_cached)
> - return tmp;
> -
> -#if defined(__i386__) || defined(__x86_64__)
> - if (caching == ttm_write_combined)
> - tmp = pgprot_writecombine(tmp);
> - else if (boot_cpu_data.x86 > 3)
> - tmp = pgprot_noncached(tmp);
> -#endif
> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> - defined(__powerpc__) || defined(__mips__)
> - if (caching == ttm_write_combined)
> - tmp = pgprot_writecombine(tmp);
> - else
> - tmp = pgprot_noncached(tmp);
> -#endif
> -#if defined(__sparc__)
> - tmp = pgprot_noncached(tmp);
> -#endif
> - return tmp;
> + return ttm_prot_from_caching(caching, tmp);
> }
> EXPORT_SYMBOL(ttm_io_prot);
>
> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
> index 56b0efdba1a9..997c458f68a9 100644
> --- a/drivers/gpu/drm/ttm/ttm_module.c
> +++ b/drivers/gpu/drm/ttm/ttm_module.c
> @@ -31,12 +31,47 @@
> */
> #include <linux/module.h>
> #include <linux/device.h>
> +#include <linux/pgtable.h>
> #include <linux/sched.h>
> #include <linux/debugfs.h>
> #include <drm/drm_sysfs.h>
> +#include <drm/ttm/ttm_caching.h>
>
> #include "ttm_module.h"
>
> +/**
> + * ttm_prot_from_caching - Modify the page protection according to the
> + * ttm cacing mode
> + * @caching: The ttm caching mode
> + * @tmp: The original page protection
> + *
> + * Return: The modified page protection
> + */
> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
> +{
> + /* Cached mappings need no adjustment */
> + if (caching == ttm_cached)
> + return tmp;
> +
> +#if defined(__i386__) || defined(__x86_64__)
> + if (caching == ttm_write_combined)
> + tmp = pgprot_writecombine(tmp);
> + else if (boot_cpu_data.x86 > 3)
> + tmp = pgprot_noncached(tmp);
> +#endif
> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> + defined(__powerpc__) || defined(__mips__)
> + if (caching == ttm_write_combined)
> + tmp = pgprot_writecombine(tmp);
> + else
> + tmp = pgprot_noncached(tmp);
> +#endif
> +#if defined(__sparc__)
> + tmp = pgprot_noncached(tmp);
> +#endif
> + return tmp;
> +}
> +
> struct dentry *ttm_debugfs_root;
>
> static int __init ttm_init(void)
> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
> index 59e2b7157e41..e05ae7e3d477 100644
> --- a/drivers/gpu/drm/ttm/ttm_resource.c
> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> @@ -22,6 +22,10 @@
> * Authors: Christian König
> */
>
> +#include <linux/dma-buf-map.h>
> +#include <linux/io-mapping.h>
> +#include <linux/scatterlist.h>
> +
> #include <drm/ttm/ttm_resource.h>
> #include <drm/ttm/ttm_bo_driver.h>
>
> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
> man->func->debug(man, p);
> }
> EXPORT_SYMBOL(ttm_resource_manager_debug);
> +
> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *dmap,
> + pgoff_t i)
> +{
> + struct ttm_kmap_iter_iomap *iter_io =
> + container_of(iter, typeof(*iter_io), base);
> + void __iomem *addr;
> +
> +retry:
> + while (i >= iter_io->cache.end) {
> + iter_io->cache.sg = iter_io->cache.sg ?
> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
> + iter_io->cache.i = iter_io->cache.end;
> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
> + PAGE_SHIFT;
> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
> + iter_io->start;
> + }
> +
> + if (i < iter_io->cache.i) {
> + iter_io->cache.end = 0;
> + iter_io->cache.sg = NULL;
> + goto retry;
> + }
> +
> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
> + (((resource_size_t)i - iter_io->cache.i)
> + << PAGE_SHIFT));
> + dma_buf_map_set_vaddr_iomem(dmap, addr);
> +}
> +
> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *map)
> +{
> + io_mapping_unmap_local(map->vaddr_iomem);
> +}
> +
> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> + .map_local = ttm_kmap_iter_iomap_map_local,
> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> + .maps_tt = false,
> +};
> +
> +/**
> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
> + * @st: sg_table into @iomap, representing the memory of the struct
> + * ttm_resource.
> + * @start: Offset that needs to be subtracted from @st to make
> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> + *
> + * Return: Pointer to the embedded struct ttm_kmap_iter.
> + */
> +struct ttm_kmap_iter *
> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> + struct io_mapping *iomap,
> + struct sg_table *st,
> + resource_size_t start)
> +{
> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> + iter_io->iomap = iomap;
> + iter_io->st = st;
> + iter_io->start = start;
> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> +
> + return &iter_io->base;
> +}
> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> +
> +/**
> + * DOC: Linear io iterator
> + *
> + * This code should die in the not too near future. Best would be if we could
> + * make io-mapping use memremap for all io memory, and have memremap
> + * implement a kmap_local functionality. We could then strip a huge amount of
> + * code. These linear io iterators are implemented to mimic old functionality,
> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
> + * friends, and at least on 32-bit they add global TLB flushes and points
> + * of failure.
> + */
> +
> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
> + struct dma_buf_map *dmap,
> + pgoff_t i)
> +{
> + struct ttm_kmap_iter_linear_io *iter_io =
> + container_of(iter, typeof(*iter_io), base);
> +
> + *dmap = iter_io->dmap;
> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> +}
> +
> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
> + .map_local = ttm_kmap_iter_linear_io_map_local,
> + .maps_tt = false,
> +};
> +
> +struct ttm_kmap_iter *
> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
> + struct ttm_device *bdev,
> + struct ttm_resource *mem)
> +{
> + int ret;
> +
> + ret = ttm_mem_io_reserve(bdev, mem);
> + if (ret)
> + goto out_err;
> + if (!mem->bus.is_iomem) {
> + ret = -EINVAL;
> + goto out_io_free;
> + }
> +
> + if (mem->bus.addr) {
> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
> + iter_io->needs_unmap = false;
> + } else {
> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> +
> + iter_io->needs_unmap = true;
> + if (mem->bus.caching == ttm_write_combined)
> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
> + ioremap_wc(mem->bus.offset,
> + bus_size));
> + else if (mem->bus.caching == ttm_cached)
> + dma_buf_map_set_vaddr(&iter_io->dmap,
> + memremap(mem->bus.offset, bus_size,
> + MEMREMAP_WB));
The comments in set_vaddr suggest that this is meant for
system-memory. Does that actually matter or is it just about not
losing the __iomem annotation on platforms where it matters?
Apparently cached device local is a thing. Also should this not be
wrapped in CONFIG_X86?
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-25 9:18 ` Matthew Auld
@ 2021-05-25 9:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 9:32 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On 5/25/21 11:18 AM, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
>> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
>> probably might be possible to use it for copying in- and out of
>> sglist represented io memory, using io_mem_reserve() / io_mem_free()
>> callbacks, that would cause problems with fault().
>> Instead, implement a method mapping page-by-page using kmap_local()
>> semantics. As an additional benefit we then avoid the occasional global
>> TLB flushes of ioremap() and consuming ioremap space, elimination of a
>> critical point of failure and with a slight change of semantics we could
>> also push the memcpy out async for testing and async driver development
>> purposes.
>>
>> A special linear iomem iterator is introduced internally to mimic the
>> old ioremap behaviour for code-paths that can't immediately be ported
>> over. This adds to the code size and should be considered a temporary
>> solution.
>>
>> Looking at the code we have a lot of checks for iomap tagged pointers.
>> Ideally we should extend the core memremap functions to also accept
>> uncached memory and kmap_local functionality. Then we could strip a
>> lot of code.
>>
>> Cc: Christian König <christian.koenig@amd.com>
>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> ---
>> v3:
>> - Split up in various TTM files and addressed review comments by
>> Christian König. Tested and fixed legacy iomap memcpy path on i915.
>> ---
>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>> include/drm/ttm/ttm_caching.h | 2 +
>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>> include/drm/ttm/ttm_resource.h | 61 +++++++
>> include/drm/ttm/ttm_tt.h | 16 ++
>> 9 files changed, 508 insertions(+), 181 deletions(-)
>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
>> index ae8b61460724..912cbe8e60a2 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
>> mem->bus.addr = NULL;
>> }
>>
>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>> - struct ttm_resource *mem,
>> - void **virtual)
>> +/**
>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
>> + * @bo: The struct ttm_buffer_object.
>> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
>> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
>> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
>> + *
>> + * This function is intended to be able to move out async under a
>> + * dma-fence if desired.
>> + */
>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>> + struct ttm_resource *dst_mem,
>> + struct ttm_kmap_iter *dst_iter,
>> + struct ttm_kmap_iter *src_iter)
>> {
>> - int ret;
>> - void *addr;
>> -
>> - *virtual = NULL;
>> - ret = ttm_mem_io_reserve(bdev, mem);
>> - if (ret || !mem->bus.is_iomem)
>> - return ret;
>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
>> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
>> + struct ttm_tt *ttm = bo->ttm;
>> + struct dma_buf_map src_map, dst_map;
>> + pgoff_t i;
>>
>> - if (mem->bus.addr) {
>> - addr = mem->bus.addr;
>> - } else {
>> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
>> + /* Single TTM move. NOP */
>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>> + return;
>>
>> - if (mem->bus.caching == ttm_write_combined)
>> - addr = ioremap_wc(mem->bus.offset, bus_size);
>> -#ifdef CONFIG_X86
>> - else if (mem->bus.caching == ttm_cached)
>> - addr = ioremap_cache(mem->bus.offset, bus_size);
>> -#endif
>> - else
>> - addr = ioremap(mem->bus.offset, bus_size);
>> - if (!addr) {
>> - ttm_mem_io_free(bdev, mem);
>> - return -ENOMEM;
>> + /* Don't move nonexistent data. Clear destination instead. */
>> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
>> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
>> + return;
>> +
>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>> + dst_ops->map_local(dst_iter, &dst_map, i);
>> + if (dst_map.is_iomem)
>> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
>> + else
>> + memset(dst_map.vaddr, 0, PAGE_SIZE);
>> + if (dst_ops->unmap_local)
>> + dst_ops->unmap_local(dst_iter, &dst_map);
>> }
>> + return;
>> }
>> - *virtual = addr;
>> - return 0;
>> -}
>> -
>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>> - struct ttm_resource *mem,
>> - void *virtual)
>> -{
>> - if (virtual && mem->bus.addr == NULL)
>> - iounmap(virtual);
>> - ttm_mem_io_free(bdev, mem);
>> -}
>> -
>> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
>> -{
>> - uint32_t *dstP =
>> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
>> - uint32_t *srcP =
>> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
>> -
>> - int i;
>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>> - iowrite32(ioread32(srcP++), dstP++);
>> - return 0;
>> -}
>> -
>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>> - unsigned long page,
>> - pgprot_t prot)
>> -{
>> - struct page *d = ttm->pages[page];
>> - void *dst;
>> -
>> - if (!d)
>> - return -ENOMEM;
>> -
>> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
>> - dst = kmap_atomic_prot(d, prot);
>> - if (!dst)
>> - return -ENOMEM;
>> -
>> - memcpy_fromio(dst, src, PAGE_SIZE);
>> -
>> - kunmap_atomic(dst);
>> -
>> - return 0;
>> -}
>> -
>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>> - unsigned long page,
>> - pgprot_t prot)
>> -{
>> - struct page *s = ttm->pages[page];
>> - void *src;
>> -
>> - if (!s)
>> - return -ENOMEM;
>> -
>> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
>> - src = kmap_atomic_prot(s, prot);
>> - if (!src)
>> - return -ENOMEM;
>>
>> - memcpy_toio(dst, src, PAGE_SIZE);
>> -
>> - kunmap_atomic(src);
>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>> + dst_ops->map_local(dst_iter, &dst_map, i);
>> + src_ops->map_local(src_iter, &src_map, i);
>> +
>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
>> + } else if (!src_map.is_iomem) {
>> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
>> + PAGE_SIZE);
>> + } else if (!dst_map.is_iomem) {
>> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
>> + PAGE_SIZE);
>> + } else {
>> + int j;
>> + u32 __iomem *src = src_map.vaddr_iomem;
>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>
>> - return 0;
>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
OK, will fix.
>
>> + iowrite32(ioread32(src++), dst++);
>> + }
>> + if (src_ops->unmap_local)
>> + src_ops->unmap_local(src_iter, &src_map);
>> + if (dst_ops->unmap_local)
>> + dst_ops->unmap_local(dst_iter, &dst_map);
>> + }
>> }
>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>
>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>> struct ttm_operation_ctx *ctx,
>> - struct ttm_resource *new_mem)
>> + struct ttm_resource *dst_mem)
>> {
>> struct ttm_device *bdev = bo->bdev;
>> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
>> + struct ttm_resource_manager *dst_man =
>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>> struct ttm_tt *ttm = bo->ttm;
>> - struct ttm_resource *old_mem = &bo->mem;
>> - struct ttm_resource old_copy = *old_mem;
>> - void *old_iomap;
>> - void *new_iomap;
>> + struct ttm_resource *src_mem = &bo->mem;
>> + struct ttm_resource_manager *src_man =
>> + ttm_manager_type(bdev, src_mem->mem_type);
>> + struct ttm_resource src_copy = *src_mem;
>> + union {
>> + struct ttm_kmap_iter_tt tt;
>> + struct ttm_kmap_iter_linear_io io;
>> + } _dst_iter, _src_iter;
>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>> int ret;
>> - unsigned long i;
>>
>> - ret = ttm_bo_wait_ctx(bo, ctx);
>> - if (ret)
>> - return ret;
>> -
>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>> - if (ret)
>> - return ret;
>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>> - if (ret)
>> - goto out;
>> -
>> - /*
>> - * Single TTM move. NOP.
>> - */
>> - if (old_iomap == NULL && new_iomap == NULL)
>> - goto out2;
>> -
>> - /*
>> - * Don't move nonexistent data. Clear destination instead.
>> - */
>> - if (old_iomap == NULL &&
>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
>> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
>> - goto out2;
>> - }
>> -
>> - /*
>> - * TTM might be null for moves within the same region.
>> - */
>> - if (ttm) {
>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
>> + dst_man->use_tt)) {
>> ret = ttm_tt_populate(bdev, ttm, ctx);
>> if (ret)
>> - goto out1;
>> + return ret;
>> }
>>
>> - for (i = 0; i < new_mem->num_pages; ++i) {
>> - if (old_iomap == NULL) {
>> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
>> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
>> - prot);
>> - } else if (new_iomap == NULL) {
>> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
>> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
>> - prot);
>> - } else {
>> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
>> - }
>> - if (ret)
>> - goto out1;
>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
>> + if (IS_ERR(dst_iter))
>> + return PTR_ERR(dst_iter);
>> +
>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
>> + if (IS_ERR(src_iter)) {
>> + ret = PTR_ERR(src_iter);
>> + goto out_src_iter;
>> }
>> - mb();
>> -out2:
>> - old_copy = *old_mem;
>>
>> - ttm_bo_assign_mem(bo, new_mem);
>> -
>> - if (!man->use_tt)
>> - ttm_bo_tt_destroy(bo);
>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>> + src_copy = *src_mem;
>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>
>> -out1:
>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>> -out:
>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>> + if (!src_iter->ops->maps_tt)
>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
>> +out_src_iter:
>> + if (!dst_iter->ops->maps_tt)
>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
>>
>> - /*
>> - * On error, keep the mm node!
>> - */
>> - if (!ret)
>> - ttm_resource_free(bo, &old_copy);
>> return ret;
>> }
>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
>> man = ttm_manager_type(bo->bdev, res->mem_type);
>> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
>>
>> - /* Cached mappings need no adjustment */
>> - if (caching == ttm_cached)
>> - return tmp;
>> -
>> -#if defined(__i386__) || defined(__x86_64__)
>> - if (caching == ttm_write_combined)
>> - tmp = pgprot_writecombine(tmp);
>> - else if (boot_cpu_data.x86 > 3)
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
>> - defined(__powerpc__) || defined(__mips__)
>> - if (caching == ttm_write_combined)
>> - tmp = pgprot_writecombine(tmp);
>> - else
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> -#if defined(__sparc__)
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> - return tmp;
>> + return ttm_prot_from_caching(caching, tmp);
>> }
>> EXPORT_SYMBOL(ttm_io_prot);
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
>> index 56b0efdba1a9..997c458f68a9 100644
>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>> @@ -31,12 +31,47 @@
>> */
>> #include <linux/module.h>
>> #include <linux/device.h>
>> +#include <linux/pgtable.h>
>> #include <linux/sched.h>
>> #include <linux/debugfs.h>
>> #include <drm/drm_sysfs.h>
>> +#include <drm/ttm/ttm_caching.h>
>>
>> #include "ttm_module.h"
>>
>> +/**
>> + * ttm_prot_from_caching - Modify the page protection according to the
>> + * ttm cacing mode
>> + * @caching: The ttm caching mode
>> + * @tmp: The original page protection
>> + *
>> + * Return: The modified page protection
>> + */
>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
>> +{
>> + /* Cached mappings need no adjustment */
>> + if (caching == ttm_cached)
>> + return tmp;
>> +
>> +#if defined(__i386__) || defined(__x86_64__)
>> + if (caching == ttm_write_combined)
>> + tmp = pgprot_writecombine(tmp);
>> + else if (boot_cpu_data.x86 > 3)
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
>> + defined(__powerpc__) || defined(__mips__)
>> + if (caching == ttm_write_combined)
>> + tmp = pgprot_writecombine(tmp);
>> + else
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> +#if defined(__sparc__)
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> + return tmp;
>> +}
>> +
>> struct dentry *ttm_debugfs_root;
>>
>> static int __init ttm_init(void)
>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
>> index 59e2b7157e41..e05ae7e3d477 100644
>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>> @@ -22,6 +22,10 @@
>> * Authors: Christian König
>> */
>>
>> +#include <linux/dma-buf-map.h>
>> +#include <linux/io-mapping.h>
>> +#include <linux/scatterlist.h>
>> +
>> #include <drm/ttm/ttm_resource.h>
>> #include <drm/ttm/ttm_bo_driver.h>
>>
>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
>> man->func->debug(man, p);
>> }
>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>> +
>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *dmap,
>> + pgoff_t i)
>> +{
>> + struct ttm_kmap_iter_iomap *iter_io =
>> + container_of(iter, typeof(*iter_io), base);
>> + void __iomem *addr;
>> +
>> +retry:
>> + while (i >= iter_io->cache.end) {
>> + iter_io->cache.sg = iter_io->cache.sg ?
>> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
>> + iter_io->cache.i = iter_io->cache.end;
>> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
>> + PAGE_SHIFT;
>> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
>> + iter_io->start;
>> + }
>> +
>> + if (i < iter_io->cache.i) {
>> + iter_io->cache.end = 0;
>> + iter_io->cache.sg = NULL;
>> + goto retry;
>> + }
>> +
>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
>> + (((resource_size_t)i - iter_io->cache.i)
>> + << PAGE_SHIFT));
>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>> +}
>> +
>> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *map)
>> +{
>> + io_mapping_unmap_local(map->vaddr_iomem);
>> +}
>> +
>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>> + .map_local = ttm_kmap_iter_iomap_map_local,
>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>> + .maps_tt = false,
>> +};
>> +
>> +/**
>> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
>> + * @st: sg_table into @iomap, representing the memory of the struct
>> + * ttm_resource.
>> + * @start: Offset that needs to be subtracted from @st to make
>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>> + *
>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>> + */
>> +struct ttm_kmap_iter *
>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>> + struct io_mapping *iomap,
>> + struct sg_table *st,
>> + resource_size_t start)
>> +{
>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>> + iter_io->iomap = iomap;
>> + iter_io->st = st;
>> + iter_io->start = start;
>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>> +
>> + return &iter_io->base;
>> +}
>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>> +
>> +/**
>> + * DOC: Linear io iterator
>> + *
>> + * This code should die in the not too near future. Best would be if we could
>> + * make io-mapping use memremap for all io memory, and have memremap
>> + * implement a kmap_local functionality. We could then strip a huge amount of
>> + * code. These linear io iterators are implemented to mimic old functionality,
>> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
>> + * friends, and at least on 32-bit they add global TLB flushes and points
>> + * of failure.
>> + */
>> +
>> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *dmap,
>> + pgoff_t i)
>> +{
>> + struct ttm_kmap_iter_linear_io *iter_io =
>> + container_of(iter, typeof(*iter_io), base);
>> +
>> + *dmap = iter_io->dmap;
>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>> +}
>> +
>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>> + .maps_tt = false,
>> +};
>> +
>> +struct ttm_kmap_iter *
>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
>> + struct ttm_device *bdev,
>> + struct ttm_resource *mem)
>> +{
>> + int ret;
>> +
>> + ret = ttm_mem_io_reserve(bdev, mem);
>> + if (ret)
>> + goto out_err;
>> + if (!mem->bus.is_iomem) {
>> + ret = -EINVAL;
>> + goto out_io_free;
>> + }
>> +
>> + if (mem->bus.addr) {
>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
>> + iter_io->needs_unmap = false;
>> + } else {
>> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
>> +
>> + iter_io->needs_unmap = true;
>> + if (mem->bus.caching == ttm_write_combined)
>> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
>> + ioremap_wc(mem->bus.offset,
>> + bus_size));
>> + else if (mem->bus.caching == ttm_cached)
>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>> + memremap(mem->bus.offset, bus_size,
>> + MEMREMAP_WB));
> The comments in set_vaddr suggest that this is meant for
> system-memory. Does that actually matter or is it just about not
> losing the __iomem annotation on platforms where it matters?
Yes, it's the latter. dma_buf_map() is relatively new and the author
probably didn't think about the case of cached iomem, which is used by,
for example, vmwgfx.
> Apparently cached device local is a thing. Also should this not be
> wrapped in CONFIG_X86?
Both dma_buf_map() and memremap are generic, I think, I guess memremap
would return NULL if it's not supported.
/Thomas
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-25 9:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 9:32 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On 5/25/21 11:18 AM, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
>> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
>> probably might be possible to use it for copying in- and out of
>> sglist represented io memory, using io_mem_reserve() / io_mem_free()
>> callbacks, that would cause problems with fault().
>> Instead, implement a method mapping page-by-page using kmap_local()
>> semantics. As an additional benefit we then avoid the occasional global
>> TLB flushes of ioremap() and consuming ioremap space, elimination of a
>> critical point of failure and with a slight change of semantics we could
>> also push the memcpy out async for testing and async driver development
>> purposes.
>>
>> A special linear iomem iterator is introduced internally to mimic the
>> old ioremap behaviour for code-paths that can't immediately be ported
>> over. This adds to the code size and should be considered a temporary
>> solution.
>>
>> Looking at the code we have a lot of checks for iomap tagged pointers.
>> Ideally we should extend the core memremap functions to also accept
>> uncached memory and kmap_local functionality. Then we could strip a
>> lot of code.
>>
>> Cc: Christian König <christian.koenig@amd.com>
>> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> ---
>> v3:
>> - Split up in various TTM files and addressed review comments by
>> Christian König. Tested and fixed legacy iomap memcpy path on i915.
>> ---
>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>> include/drm/ttm/ttm_caching.h | 2 +
>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>> include/drm/ttm/ttm_resource.h | 61 +++++++
>> include/drm/ttm/ttm_tt.h | 16 ++
>> 9 files changed, 508 insertions(+), 181 deletions(-)
>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
>> index ae8b61460724..912cbe8e60a2 100644
>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
>> mem->bus.addr = NULL;
>> }
>>
>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>> - struct ttm_resource *mem,
>> - void **virtual)
>> +/**
>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
>> + * @bo: The struct ttm_buffer_object.
>> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
>> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
>> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
>> + *
>> + * This function is intended to be able to move out async under a
>> + * dma-fence if desired.
>> + */
>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>> + struct ttm_resource *dst_mem,
>> + struct ttm_kmap_iter *dst_iter,
>> + struct ttm_kmap_iter *src_iter)
>> {
>> - int ret;
>> - void *addr;
>> -
>> - *virtual = NULL;
>> - ret = ttm_mem_io_reserve(bdev, mem);
>> - if (ret || !mem->bus.is_iomem)
>> - return ret;
>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
>> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
>> + struct ttm_tt *ttm = bo->ttm;
>> + struct dma_buf_map src_map, dst_map;
>> + pgoff_t i;
>>
>> - if (mem->bus.addr) {
>> - addr = mem->bus.addr;
>> - } else {
>> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
>> + /* Single TTM move. NOP */
>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>> + return;
>>
>> - if (mem->bus.caching == ttm_write_combined)
>> - addr = ioremap_wc(mem->bus.offset, bus_size);
>> -#ifdef CONFIG_X86
>> - else if (mem->bus.caching == ttm_cached)
>> - addr = ioremap_cache(mem->bus.offset, bus_size);
>> -#endif
>> - else
>> - addr = ioremap(mem->bus.offset, bus_size);
>> - if (!addr) {
>> - ttm_mem_io_free(bdev, mem);
>> - return -ENOMEM;
>> + /* Don't move nonexistent data. Clear destination instead. */
>> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
>> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
>> + return;
>> +
>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>> + dst_ops->map_local(dst_iter, &dst_map, i);
>> + if (dst_map.is_iomem)
>> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
>> + else
>> + memset(dst_map.vaddr, 0, PAGE_SIZE);
>> + if (dst_ops->unmap_local)
>> + dst_ops->unmap_local(dst_iter, &dst_map);
>> }
>> + return;
>> }
>> - *virtual = addr;
>> - return 0;
>> -}
>> -
>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>> - struct ttm_resource *mem,
>> - void *virtual)
>> -{
>> - if (virtual && mem->bus.addr == NULL)
>> - iounmap(virtual);
>> - ttm_mem_io_free(bdev, mem);
>> -}
>> -
>> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
>> -{
>> - uint32_t *dstP =
>> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
>> - uint32_t *srcP =
>> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
>> -
>> - int i;
>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>> - iowrite32(ioread32(srcP++), dstP++);
>> - return 0;
>> -}
>> -
>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>> - unsigned long page,
>> - pgprot_t prot)
>> -{
>> - struct page *d = ttm->pages[page];
>> - void *dst;
>> -
>> - if (!d)
>> - return -ENOMEM;
>> -
>> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
>> - dst = kmap_atomic_prot(d, prot);
>> - if (!dst)
>> - return -ENOMEM;
>> -
>> - memcpy_fromio(dst, src, PAGE_SIZE);
>> -
>> - kunmap_atomic(dst);
>> -
>> - return 0;
>> -}
>> -
>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>> - unsigned long page,
>> - pgprot_t prot)
>> -{
>> - struct page *s = ttm->pages[page];
>> - void *src;
>> -
>> - if (!s)
>> - return -ENOMEM;
>> -
>> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
>> - src = kmap_atomic_prot(s, prot);
>> - if (!src)
>> - return -ENOMEM;
>>
>> - memcpy_toio(dst, src, PAGE_SIZE);
>> -
>> - kunmap_atomic(src);
>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>> + dst_ops->map_local(dst_iter, &dst_map, i);
>> + src_ops->map_local(src_iter, &src_map, i);
>> +
>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
>> + } else if (!src_map.is_iomem) {
>> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
>> + PAGE_SIZE);
>> + } else if (!dst_map.is_iomem) {
>> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
>> + PAGE_SIZE);
>> + } else {
>> + int j;
>> + u32 __iomem *src = src_map.vaddr_iomem;
>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>
>> - return 0;
>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
OK, will fix.
>
>> + iowrite32(ioread32(src++), dst++);
>> + }
>> + if (src_ops->unmap_local)
>> + src_ops->unmap_local(src_iter, &src_map);
>> + if (dst_ops->unmap_local)
>> + dst_ops->unmap_local(dst_iter, &dst_map);
>> + }
>> }
>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>
>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>> struct ttm_operation_ctx *ctx,
>> - struct ttm_resource *new_mem)
>> + struct ttm_resource *dst_mem)
>> {
>> struct ttm_device *bdev = bo->bdev;
>> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
>> + struct ttm_resource_manager *dst_man =
>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>> struct ttm_tt *ttm = bo->ttm;
>> - struct ttm_resource *old_mem = &bo->mem;
>> - struct ttm_resource old_copy = *old_mem;
>> - void *old_iomap;
>> - void *new_iomap;
>> + struct ttm_resource *src_mem = &bo->mem;
>> + struct ttm_resource_manager *src_man =
>> + ttm_manager_type(bdev, src_mem->mem_type);
>> + struct ttm_resource src_copy = *src_mem;
>> + union {
>> + struct ttm_kmap_iter_tt tt;
>> + struct ttm_kmap_iter_linear_io io;
>> + } _dst_iter, _src_iter;
>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>> int ret;
>> - unsigned long i;
>>
>> - ret = ttm_bo_wait_ctx(bo, ctx);
>> - if (ret)
>> - return ret;
>> -
>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>> - if (ret)
>> - return ret;
>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>> - if (ret)
>> - goto out;
>> -
>> - /*
>> - * Single TTM move. NOP.
>> - */
>> - if (old_iomap == NULL && new_iomap == NULL)
>> - goto out2;
>> -
>> - /*
>> - * Don't move nonexistent data. Clear destination instead.
>> - */
>> - if (old_iomap == NULL &&
>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
>> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
>> - goto out2;
>> - }
>> -
>> - /*
>> - * TTM might be null for moves within the same region.
>> - */
>> - if (ttm) {
>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
>> + dst_man->use_tt)) {
>> ret = ttm_tt_populate(bdev, ttm, ctx);
>> if (ret)
>> - goto out1;
>> + return ret;
>> }
>>
>> - for (i = 0; i < new_mem->num_pages; ++i) {
>> - if (old_iomap == NULL) {
>> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
>> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
>> - prot);
>> - } else if (new_iomap == NULL) {
>> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
>> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
>> - prot);
>> - } else {
>> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
>> - }
>> - if (ret)
>> - goto out1;
>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
>> + if (IS_ERR(dst_iter))
>> + return PTR_ERR(dst_iter);
>> +
>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
>> + if (IS_ERR(src_iter)) {
>> + ret = PTR_ERR(src_iter);
>> + goto out_src_iter;
>> }
>> - mb();
>> -out2:
>> - old_copy = *old_mem;
>>
>> - ttm_bo_assign_mem(bo, new_mem);
>> -
>> - if (!man->use_tt)
>> - ttm_bo_tt_destroy(bo);
>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>> + src_copy = *src_mem;
>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>
>> -out1:
>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>> -out:
>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>> + if (!src_iter->ops->maps_tt)
>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
>> +out_src_iter:
>> + if (!dst_iter->ops->maps_tt)
>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
>>
>> - /*
>> - * On error, keep the mm node!
>> - */
>> - if (!ret)
>> - ttm_resource_free(bo, &old_copy);
>> return ret;
>> }
>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
>> man = ttm_manager_type(bo->bdev, res->mem_type);
>> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
>>
>> - /* Cached mappings need no adjustment */
>> - if (caching == ttm_cached)
>> - return tmp;
>> -
>> -#if defined(__i386__) || defined(__x86_64__)
>> - if (caching == ttm_write_combined)
>> - tmp = pgprot_writecombine(tmp);
>> - else if (boot_cpu_data.x86 > 3)
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
>> - defined(__powerpc__) || defined(__mips__)
>> - if (caching == ttm_write_combined)
>> - tmp = pgprot_writecombine(tmp);
>> - else
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> -#if defined(__sparc__)
>> - tmp = pgprot_noncached(tmp);
>> -#endif
>> - return tmp;
>> + return ttm_prot_from_caching(caching, tmp);
>> }
>> EXPORT_SYMBOL(ttm_io_prot);
>>
>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
>> index 56b0efdba1a9..997c458f68a9 100644
>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>> @@ -31,12 +31,47 @@
>> */
>> #include <linux/module.h>
>> #include <linux/device.h>
>> +#include <linux/pgtable.h>
>> #include <linux/sched.h>
>> #include <linux/debugfs.h>
>> #include <drm/drm_sysfs.h>
>> +#include <drm/ttm/ttm_caching.h>
>>
>> #include "ttm_module.h"
>>
>> +/**
>> + * ttm_prot_from_caching - Modify the page protection according to the
>> + * ttm cacing mode
>> + * @caching: The ttm caching mode
>> + * @tmp: The original page protection
>> + *
>> + * Return: The modified page protection
>> + */
>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
>> +{
>> + /* Cached mappings need no adjustment */
>> + if (caching == ttm_cached)
>> + return tmp;
>> +
>> +#if defined(__i386__) || defined(__x86_64__)
>> + if (caching == ttm_write_combined)
>> + tmp = pgprot_writecombine(tmp);
>> + else if (boot_cpu_data.x86 > 3)
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
>> + defined(__powerpc__) || defined(__mips__)
>> + if (caching == ttm_write_combined)
>> + tmp = pgprot_writecombine(tmp);
>> + else
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> +#if defined(__sparc__)
>> + tmp = pgprot_noncached(tmp);
>> +#endif
>> + return tmp;
>> +}
>> +
>> struct dentry *ttm_debugfs_root;
>>
>> static int __init ttm_init(void)
>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
>> index 59e2b7157e41..e05ae7e3d477 100644
>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>> @@ -22,6 +22,10 @@
>> * Authors: Christian König
>> */
>>
>> +#include <linux/dma-buf-map.h>
>> +#include <linux/io-mapping.h>
>> +#include <linux/scatterlist.h>
>> +
>> #include <drm/ttm/ttm_resource.h>
>> #include <drm/ttm/ttm_bo_driver.h>
>>
>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
>> man->func->debug(man, p);
>> }
>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>> +
>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *dmap,
>> + pgoff_t i)
>> +{
>> + struct ttm_kmap_iter_iomap *iter_io =
>> + container_of(iter, typeof(*iter_io), base);
>> + void __iomem *addr;
>> +
>> +retry:
>> + while (i >= iter_io->cache.end) {
>> + iter_io->cache.sg = iter_io->cache.sg ?
>> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
>> + iter_io->cache.i = iter_io->cache.end;
>> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
>> + PAGE_SHIFT;
>> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
>> + iter_io->start;
>> + }
>> +
>> + if (i < iter_io->cache.i) {
>> + iter_io->cache.end = 0;
>> + iter_io->cache.sg = NULL;
>> + goto retry;
>> + }
>> +
>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
>> + (((resource_size_t)i - iter_io->cache.i)
>> + << PAGE_SHIFT));
>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>> +}
>> +
>> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *map)
>> +{
>> + io_mapping_unmap_local(map->vaddr_iomem);
>> +}
>> +
>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>> + .map_local = ttm_kmap_iter_iomap_map_local,
>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>> + .maps_tt = false,
>> +};
>> +
>> +/**
>> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
>> + * @st: sg_table into @iomap, representing the memory of the struct
>> + * ttm_resource.
>> + * @start: Offset that needs to be subtracted from @st to make
>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>> + *
>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>> + */
>> +struct ttm_kmap_iter *
>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>> + struct io_mapping *iomap,
>> + struct sg_table *st,
>> + resource_size_t start)
>> +{
>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>> + iter_io->iomap = iomap;
>> + iter_io->st = st;
>> + iter_io->start = start;
>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>> +
>> + return &iter_io->base;
>> +}
>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>> +
>> +/**
>> + * DOC: Linear io iterator
>> + *
>> + * This code should die in the not too near future. Best would be if we could
>> + * make io-mapping use memremap for all io memory, and have memremap
>> + * implement a kmap_local functionality. We could then strip a huge amount of
>> + * code. These linear io iterators are implemented to mimic old functionality,
>> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
>> + * friends, and at least on 32-bit they add global TLB flushes and points
>> + * of failure.
>> + */
>> +
>> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
>> + struct dma_buf_map *dmap,
>> + pgoff_t i)
>> +{
>> + struct ttm_kmap_iter_linear_io *iter_io =
>> + container_of(iter, typeof(*iter_io), base);
>> +
>> + *dmap = iter_io->dmap;
>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>> +}
>> +
>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>> + .maps_tt = false,
>> +};
>> +
>> +struct ttm_kmap_iter *
>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
>> + struct ttm_device *bdev,
>> + struct ttm_resource *mem)
>> +{
>> + int ret;
>> +
>> + ret = ttm_mem_io_reserve(bdev, mem);
>> + if (ret)
>> + goto out_err;
>> + if (!mem->bus.is_iomem) {
>> + ret = -EINVAL;
>> + goto out_io_free;
>> + }
>> +
>> + if (mem->bus.addr) {
>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
>> + iter_io->needs_unmap = false;
>> + } else {
>> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
>> +
>> + iter_io->needs_unmap = true;
>> + if (mem->bus.caching == ttm_write_combined)
>> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
>> + ioremap_wc(mem->bus.offset,
>> + bus_size));
>> + else if (mem->bus.caching == ttm_cached)
>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>> + memremap(mem->bus.offset, bus_size,
>> + MEMREMAP_WB));
> The comments in set_vaddr suggest that this is meant for
> system-memory. Does that actually matter or is it just about not
> losing the __iomem annotation on platforms where it matters?
Yes, it's the latter. dma_buf_map() is relatively new and the author
probably didn't think about the case of cached iomem, which is used by,
for example, vmwgfx.
> Apparently cached device local is a thing. Also should this not be
> wrapped in CONFIG_X86?
Both dma_buf_map() and memremap are generic, I think, I guess memremap
would return NULL if it's not supported.
/Thomas
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-25 9:32 ` Thomas Hellström
@ 2021-05-25 9:58 ` Matthew Auld
-1 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 9:58 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 25 May 2021 at 10:32, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
>
> On 5/25/21 11:18 AM, Matthew Auld wrote:
> > On Fri, 21 May 2021 at 16:33, Thomas Hellström
> > <thomas.hellstrom@linux.intel.com> wrote:
> >> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
> >> probably might be possible to use it for copying in- and out of
> >> sglist represented io memory, using io_mem_reserve() / io_mem_free()
> >> callbacks, that would cause problems with fault().
> >> Instead, implement a method mapping page-by-page using kmap_local()
> >> semantics. As an additional benefit we then avoid the occasional global
> >> TLB flushes of ioremap() and consuming ioremap space, elimination of a
> >> critical point of failure and with a slight change of semantics we could
> >> also push the memcpy out async for testing and async driver development
> >> purposes.
> >>
> >> A special linear iomem iterator is introduced internally to mimic the
> >> old ioremap behaviour for code-paths that can't immediately be ported
> >> over. This adds to the code size and should be considered a temporary
> >> solution.
> >>
> >> Looking at the code we have a lot of checks for iomap tagged pointers.
> >> Ideally we should extend the core memremap functions to also accept
> >> uncached memory and kmap_local functionality. Then we could strip a
> >> lot of code.
> >>
> >> Cc: Christian König <christian.koenig@amd.com>
> >> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> >> ---
> >> v3:
> >> - Split up in various TTM files and addressed review comments by
> >> Christian König. Tested and fixed legacy iomap memcpy path on i915.
> >> ---
> >> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
> >> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> >> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> >> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> >> include/drm/ttm/ttm_bo_driver.h | 28 +++
> >> include/drm/ttm/ttm_caching.h | 2 +
> >> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> >> include/drm/ttm/ttm_resource.h | 61 +++++++
> >> include/drm/ttm/ttm_tt.h | 16 ++
> >> 9 files changed, 508 insertions(+), 181 deletions(-)
> >> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
> >>
> >> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> index ae8b61460724..912cbe8e60a2 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
> >> mem->bus.addr = NULL;
> >> }
> >>
> >> -static int ttm_resource_ioremap(struct ttm_device *bdev,
> >> - struct ttm_resource *mem,
> >> - void **virtual)
> >> +/**
> >> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
> >> + * @bo: The struct ttm_buffer_object.
> >> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
> >> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
> >> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
> >> + *
> >> + * This function is intended to be able to move out async under a
> >> + * dma-fence if desired.
> >> + */
> >> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> >> + struct ttm_resource *dst_mem,
> >> + struct ttm_kmap_iter *dst_iter,
> >> + struct ttm_kmap_iter *src_iter)
> >> {
> >> - int ret;
> >> - void *addr;
> >> -
> >> - *virtual = NULL;
> >> - ret = ttm_mem_io_reserve(bdev, mem);
> >> - if (ret || !mem->bus.is_iomem)
> >> - return ret;
> >> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
> >> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> >> + struct ttm_tt *ttm = bo->ttm;
> >> + struct dma_buf_map src_map, dst_map;
> >> + pgoff_t i;
> >>
> >> - if (mem->bus.addr) {
> >> - addr = mem->bus.addr;
> >> - } else {
> >> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> >> + /* Single TTM move. NOP */
> >> + if (dst_ops->maps_tt && src_ops->maps_tt)
> >> + return;
> >>
> >> - if (mem->bus.caching == ttm_write_combined)
> >> - addr = ioremap_wc(mem->bus.offset, bus_size);
> >> -#ifdef CONFIG_X86
> >> - else if (mem->bus.caching == ttm_cached)
> >> - addr = ioremap_cache(mem->bus.offset, bus_size);
> >> -#endif
> >> - else
> >> - addr = ioremap(mem->bus.offset, bus_size);
> >> - if (!addr) {
> >> - ttm_mem_io_free(bdev, mem);
> >> - return -ENOMEM;
> >> + /* Don't move nonexistent data. Clear destination instead. */
> >> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
> >> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
> >> + return;
> >> +
> >> + for (i = 0; i < dst_mem->num_pages; ++i) {
> >> + dst_ops->map_local(dst_iter, &dst_map, i);
> >> + if (dst_map.is_iomem)
> >> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
> >> + else
> >> + memset(dst_map.vaddr, 0, PAGE_SIZE);
> >> + if (dst_ops->unmap_local)
> >> + dst_ops->unmap_local(dst_iter, &dst_map);
> >> }
> >> + return;
> >> }
> >> - *virtual = addr;
> >> - return 0;
> >> -}
> >> -
> >> -static void ttm_resource_iounmap(struct ttm_device *bdev,
> >> - struct ttm_resource *mem,
> >> - void *virtual)
> >> -{
> >> - if (virtual && mem->bus.addr == NULL)
> >> - iounmap(virtual);
> >> - ttm_mem_io_free(bdev, mem);
> >> -}
> >> -
> >> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
> >> -{
> >> - uint32_t *dstP =
> >> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
> >> - uint32_t *srcP =
> >> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
> >> -
> >> - int i;
> >> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> >> - iowrite32(ioread32(srcP++), dstP++);
> >> - return 0;
> >> -}
> >> -
> >> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> >> - unsigned long page,
> >> - pgprot_t prot)
> >> -{
> >> - struct page *d = ttm->pages[page];
> >> - void *dst;
> >> -
> >> - if (!d)
> >> - return -ENOMEM;
> >> -
> >> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
> >> - dst = kmap_atomic_prot(d, prot);
> >> - if (!dst)
> >> - return -ENOMEM;
> >> -
> >> - memcpy_fromio(dst, src, PAGE_SIZE);
> >> -
> >> - kunmap_atomic(dst);
> >> -
> >> - return 0;
> >> -}
> >> -
> >> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> >> - unsigned long page,
> >> - pgprot_t prot)
> >> -{
> >> - struct page *s = ttm->pages[page];
> >> - void *src;
> >> -
> >> - if (!s)
> >> - return -ENOMEM;
> >> -
> >> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
> >> - src = kmap_atomic_prot(s, prot);
> >> - if (!src)
> >> - return -ENOMEM;
> >>
> >> - memcpy_toio(dst, src, PAGE_SIZE);
> >> -
> >> - kunmap_atomic(src);
> >> + for (i = 0; i < dst_mem->num_pages; ++i) {
> >> + dst_ops->map_local(dst_iter, &dst_map, i);
> >> + src_ops->map_local(src_iter, &src_map, i);
> >> +
> >> + if (!src_map.is_iomem && !dst_map.is_iomem) {
> >> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> >> + } else if (!src_map.is_iomem) {
> >> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> >> + PAGE_SIZE);
> >> + } else if (!dst_map.is_iomem) {
> >> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
> >> + PAGE_SIZE);
> >> + } else {
> >> + int j;
> >> + u32 __iomem *src = src_map.vaddr_iomem;
> >> + u32 __iomem *dst = dst_map.vaddr_iomem;
> >>
> >> - return 0;
> >> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> > IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>
> OK, will fix.
>
>
> >
> >> + iowrite32(ioread32(src++), dst++);
> >> + }
> >> + if (src_ops->unmap_local)
> >> + src_ops->unmap_local(src_iter, &src_map);
> >> + if (dst_ops->unmap_local)
> >> + dst_ops->unmap_local(dst_iter, &dst_map);
> >> + }
> >> }
> >> +EXPORT_SYMBOL(ttm_move_memcpy);
> >>
> >> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> >> struct ttm_operation_ctx *ctx,
> >> - struct ttm_resource *new_mem)
> >> + struct ttm_resource *dst_mem)
> >> {
> >> struct ttm_device *bdev = bo->bdev;
> >> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
> >> + struct ttm_resource_manager *dst_man =
> >> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> >> struct ttm_tt *ttm = bo->ttm;
> >> - struct ttm_resource *old_mem = &bo->mem;
> >> - struct ttm_resource old_copy = *old_mem;
> >> - void *old_iomap;
> >> - void *new_iomap;
> >> + struct ttm_resource *src_mem = &bo->mem;
> >> + struct ttm_resource_manager *src_man =
> >> + ttm_manager_type(bdev, src_mem->mem_type);
> >> + struct ttm_resource src_copy = *src_mem;
> >> + union {
> >> + struct ttm_kmap_iter_tt tt;
> >> + struct ttm_kmap_iter_linear_io io;
> >> + } _dst_iter, _src_iter;
> >> + struct ttm_kmap_iter *dst_iter, *src_iter;
> >> int ret;
> >> - unsigned long i;
> >>
> >> - ret = ttm_bo_wait_ctx(bo, ctx);
> >> - if (ret)
> >> - return ret;
> >> -
> >> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> >> - if (ret)
> >> - return ret;
> >> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> >> - if (ret)
> >> - goto out;
> >> -
> >> - /*
> >> - * Single TTM move. NOP.
> >> - */
> >> - if (old_iomap == NULL && new_iomap == NULL)
> >> - goto out2;
> >> -
> >> - /*
> >> - * Don't move nonexistent data. Clear destination instead.
> >> - */
> >> - if (old_iomap == NULL &&
> >> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> >> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
> >> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
> >> - goto out2;
> >> - }
> >> -
> >> - /*
> >> - * TTM might be null for moves within the same region.
> >> - */
> >> - if (ttm) {
> >> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
> >> + dst_man->use_tt)) {
> >> ret = ttm_tt_populate(bdev, ttm, ctx);
> >> if (ret)
> >> - goto out1;
> >> + return ret;
> >> }
> >>
> >> - for (i = 0; i < new_mem->num_pages; ++i) {
> >> - if (old_iomap == NULL) {
> >> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
> >> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
> >> - prot);
> >> - } else if (new_iomap == NULL) {
> >> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
> >> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
> >> - prot);
> >> - } else {
> >> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
> >> - }
> >> - if (ret)
> >> - goto out1;
> >> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
> >> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> >> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
> >> + if (IS_ERR(dst_iter))
> >> + return PTR_ERR(dst_iter);
> >> +
> >> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
> >> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> >> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
> >> + if (IS_ERR(src_iter)) {
> >> + ret = PTR_ERR(src_iter);
> >> + goto out_src_iter;
> >> }
> >> - mb();
> >> -out2:
> >> - old_copy = *old_mem;
> >>
> >> - ttm_bo_assign_mem(bo, new_mem);
> >> -
> >> - if (!man->use_tt)
> >> - ttm_bo_tt_destroy(bo);
> >> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> >> + src_copy = *src_mem;
> >> + ttm_bo_move_sync_cleanup(bo, dst_mem);
> >>
> >> -out1:
> >> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> >> -out:
> >> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> >> + if (!src_iter->ops->maps_tt)
> >> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
> >> +out_src_iter:
> >> + if (!dst_iter->ops->maps_tt)
> >> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
> >>
> >> - /*
> >> - * On error, keep the mm node!
> >> - */
> >> - if (!ret)
> >> - ttm_resource_free(bo, &old_copy);
> >> return ret;
> >> }
> >> EXPORT_SYMBOL(ttm_bo_move_memcpy);
> >> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
> >> man = ttm_manager_type(bo->bdev, res->mem_type);
> >> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
> >>
> >> - /* Cached mappings need no adjustment */
> >> - if (caching == ttm_cached)
> >> - return tmp;
> >> -
> >> -#if defined(__i386__) || defined(__x86_64__)
> >> - if (caching == ttm_write_combined)
> >> - tmp = pgprot_writecombine(tmp);
> >> - else if (boot_cpu_data.x86 > 3)
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> >> - defined(__powerpc__) || defined(__mips__)
> >> - if (caching == ttm_write_combined)
> >> - tmp = pgprot_writecombine(tmp);
> >> - else
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> -#if defined(__sparc__)
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> - return tmp;
> >> + return ttm_prot_from_caching(caching, tmp);
> >> }
> >> EXPORT_SYMBOL(ttm_io_prot);
> >>
> >> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
> >> index 56b0efdba1a9..997c458f68a9 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_module.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_module.c
> >> @@ -31,12 +31,47 @@
> >> */
> >> #include <linux/module.h>
> >> #include <linux/device.h>
> >> +#include <linux/pgtable.h>
> >> #include <linux/sched.h>
> >> #include <linux/debugfs.h>
> >> #include <drm/drm_sysfs.h>
> >> +#include <drm/ttm/ttm_caching.h>
> >>
> >> #include "ttm_module.h"
> >>
> >> +/**
> >> + * ttm_prot_from_caching - Modify the page protection according to the
> >> + * ttm cacing mode
> >> + * @caching: The ttm caching mode
> >> + * @tmp: The original page protection
> >> + *
> >> + * Return: The modified page protection
> >> + */
> >> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
> >> +{
> >> + /* Cached mappings need no adjustment */
> >> + if (caching == ttm_cached)
> >> + return tmp;
> >> +
> >> +#if defined(__i386__) || defined(__x86_64__)
> >> + if (caching == ttm_write_combined)
> >> + tmp = pgprot_writecombine(tmp);
> >> + else if (boot_cpu_data.x86 > 3)
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> >> + defined(__powerpc__) || defined(__mips__)
> >> + if (caching == ttm_write_combined)
> >> + tmp = pgprot_writecombine(tmp);
> >> + else
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> +#if defined(__sparc__)
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> + return tmp;
> >> +}
> >> +
> >> struct dentry *ttm_debugfs_root;
> >>
> >> static int __init ttm_init(void)
> >> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
> >> index 59e2b7157e41..e05ae7e3d477 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_resource.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> >> @@ -22,6 +22,10 @@
> >> * Authors: Christian König
> >> */
> >>
> >> +#include <linux/dma-buf-map.h>
> >> +#include <linux/io-mapping.h>
> >> +#include <linux/scatterlist.h>
> >> +
> >> #include <drm/ttm/ttm_resource.h>
> >> #include <drm/ttm/ttm_bo_driver.h>
> >>
> >> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
> >> man->func->debug(man, p);
> >> }
> >> EXPORT_SYMBOL(ttm_resource_manager_debug);
> >> +
> >> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *dmap,
> >> + pgoff_t i)
> >> +{
> >> + struct ttm_kmap_iter_iomap *iter_io =
> >> + container_of(iter, typeof(*iter_io), base);
> >> + void __iomem *addr;
> >> +
> >> +retry:
> >> + while (i >= iter_io->cache.end) {
> >> + iter_io->cache.sg = iter_io->cache.sg ?
> >> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
> >> + iter_io->cache.i = iter_io->cache.end;
> >> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
> >> + PAGE_SHIFT;
> >> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
> >> + iter_io->start;
> >> + }
> >> +
> >> + if (i < iter_io->cache.i) {
> >> + iter_io->cache.end = 0;
> >> + iter_io->cache.sg = NULL;
> >> + goto retry;
> >> + }
> >> +
> >> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
> >> + (((resource_size_t)i - iter_io->cache.i)
> >> + << PAGE_SHIFT));
> >> + dma_buf_map_set_vaddr_iomem(dmap, addr);
> >> +}
> >> +
> >> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *map)
> >> +{
> >> + io_mapping_unmap_local(map->vaddr_iomem);
> >> +}
> >> +
> >> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> >> + .map_local = ttm_kmap_iter_iomap_map_local,
> >> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> >> + .maps_tt = false,
> >> +};
> >> +
> >> +/**
> >> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
> >> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> >> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
> >> + * @st: sg_table into @iomap, representing the memory of the struct
> >> + * ttm_resource.
> >> + * @start: Offset that needs to be subtracted from @st to make
> >> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> >> + *
> >> + * Return: Pointer to the embedded struct ttm_kmap_iter.
> >> + */
> >> +struct ttm_kmap_iter *
> >> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> >> + struct io_mapping *iomap,
> >> + struct sg_table *st,
> >> + resource_size_t start)
> >> +{
> >> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> >> + iter_io->iomap = iomap;
> >> + iter_io->st = st;
> >> + iter_io->start = start;
> >> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> >> +
> >> + return &iter_io->base;
> >> +}
> >> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> >> +
> >> +/**
> >> + * DOC: Linear io iterator
> >> + *
> >> + * This code should die in the not too near future. Best would be if we could
> >> + * make io-mapping use memremap for all io memory, and have memremap
> >> + * implement a kmap_local functionality. We could then strip a huge amount of
> >> + * code. These linear io iterators are implemented to mimic old functionality,
> >> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
> >> + * friends, and at least on 32-bit they add global TLB flushes and points
> >> + * of failure.
> >> + */
> >> +
> >> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *dmap,
> >> + pgoff_t i)
> >> +{
> >> + struct ttm_kmap_iter_linear_io *iter_io =
> >> + container_of(iter, typeof(*iter_io), base);
> >> +
> >> + *dmap = iter_io->dmap;
> >> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> >> +}
> >> +
> >> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
> >> + .map_local = ttm_kmap_iter_linear_io_map_local,
> >> + .maps_tt = false,
> >> +};
> >> +
> >> +struct ttm_kmap_iter *
> >> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
> >> + struct ttm_device *bdev,
> >> + struct ttm_resource *mem)
> >> +{
> >> + int ret;
> >> +
> >> + ret = ttm_mem_io_reserve(bdev, mem);
> >> + if (ret)
> >> + goto out_err;
> >> + if (!mem->bus.is_iomem) {
> >> + ret = -EINVAL;
> >> + goto out_io_free;
> >> + }
> >> +
> >> + if (mem->bus.addr) {
> >> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
> >> + iter_io->needs_unmap = false;
> >> + } else {
> >> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> >> +
> >> + iter_io->needs_unmap = true;
> >> + if (mem->bus.caching == ttm_write_combined)
> >> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
> >> + ioremap_wc(mem->bus.offset,
> >> + bus_size));
> >> + else if (mem->bus.caching == ttm_cached)
> >> + dma_buf_map_set_vaddr(&iter_io->dmap,
> >> + memremap(mem->bus.offset, bus_size,
> >> + MEMREMAP_WB));
> > The comments in set_vaddr suggest that this is meant for
> > system-memory. Does that actually matter or is it just about not
> > losing the __iomem annotation on platforms where it matters?
>
> Yes, it's the latter. dma_buf_map() is relatively new and the author
> probably didn't think about the case of cached iomem, which is used by,
> for example, vmwgfx.
>
> > Apparently cached device local is a thing. Also should this not be
> > wrapped in CONFIG_X86?
>
> Both dma_buf_map() and memremap are generic, I think, I guess memremap
> would return NULL if it's not supported.
It looks like memremap just wraps ioremap_cache, but since it also
discards the __iomem annotation should we be doing that universally?
Also not sure if ioremap_cache is universally supported, so wrapping
in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
at least that looks like roughly what the previous code was doing? Not
too sure tbh.
>
> /Thomas
>
>
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-25 9:58 ` Matthew Auld
0 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 9:58 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 25 May 2021 at 10:32, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
>
> On 5/25/21 11:18 AM, Matthew Auld wrote:
> > On Fri, 21 May 2021 at 16:33, Thomas Hellström
> > <thomas.hellstrom@linux.intel.com> wrote:
> >> The internal ttm_bo_util memcpy uses ioremap functionality, and while it
> >> probably might be possible to use it for copying in- and out of
> >> sglist represented io memory, using io_mem_reserve() / io_mem_free()
> >> callbacks, that would cause problems with fault().
> >> Instead, implement a method mapping page-by-page using kmap_local()
> >> semantics. As an additional benefit we then avoid the occasional global
> >> TLB flushes of ioremap() and consuming ioremap space, elimination of a
> >> critical point of failure and with a slight change of semantics we could
> >> also push the memcpy out async for testing and async driver development
> >> purposes.
> >>
> >> A special linear iomem iterator is introduced internally to mimic the
> >> old ioremap behaviour for code-paths that can't immediately be ported
> >> over. This adds to the code size and should be considered a temporary
> >> solution.
> >>
> >> Looking at the code we have a lot of checks for iomap tagged pointers.
> >> Ideally we should extend the core memremap functions to also accept
> >> uncached memory and kmap_local functionality. Then we could strip a
> >> lot of code.
> >>
> >> Cc: Christian König <christian.koenig@amd.com>
> >> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> >> ---
> >> v3:
> >> - Split up in various TTM files and addressed review comments by
> >> Christian König. Tested and fixed legacy iomap memcpy path on i915.
> >> ---
> >> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++-------------------
> >> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> >> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> >> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> >> include/drm/ttm/ttm_bo_driver.h | 28 +++
> >> include/drm/ttm/ttm_caching.h | 2 +
> >> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> >> include/drm/ttm/ttm_resource.h | 61 +++++++
> >> include/drm/ttm/ttm_tt.h | 16 ++
> >> 9 files changed, 508 insertions(+), 181 deletions(-)
> >> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
> >>
> >> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> index ae8b61460724..912cbe8e60a2 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> >> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device *bdev,
> >> mem->bus.addr = NULL;
> >> }
> >>
> >> -static int ttm_resource_ioremap(struct ttm_device *bdev,
> >> - struct ttm_resource *mem,
> >> - void **virtual)
> >> +/**
> >> + * ttm_move_memcpy - Helper to perform a memcpy ttm move operation.
> >> + * @bo: The struct ttm_buffer_object.
> >> + * @new_mem: The struct ttm_resource we're moving to (copy destination).
> >> + * @new_iter: A struct ttm_kmap_iter representing the destination resource.
> >> + * @src_iter: A struct ttm_kmap_iter representing the source resource.
> >> + *
> >> + * This function is intended to be able to move out async under a
> >> + * dma-fence if desired.
> >> + */
> >> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> >> + struct ttm_resource *dst_mem,
> >> + struct ttm_kmap_iter *dst_iter,
> >> + struct ttm_kmap_iter *src_iter)
> >> {
> >> - int ret;
> >> - void *addr;
> >> -
> >> - *virtual = NULL;
> >> - ret = ttm_mem_io_reserve(bdev, mem);
> >> - if (ret || !mem->bus.is_iomem)
> >> - return ret;
> >> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter->ops;
> >> + const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> >> + struct ttm_tt *ttm = bo->ttm;
> >> + struct dma_buf_map src_map, dst_map;
> >> + pgoff_t i;
> >>
> >> - if (mem->bus.addr) {
> >> - addr = mem->bus.addr;
> >> - } else {
> >> - size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> >> + /* Single TTM move. NOP */
> >> + if (dst_ops->maps_tt && src_ops->maps_tt)
> >> + return;
> >>
> >> - if (mem->bus.caching == ttm_write_combined)
> >> - addr = ioremap_wc(mem->bus.offset, bus_size);
> >> -#ifdef CONFIG_X86
> >> - else if (mem->bus.caching == ttm_cached)
> >> - addr = ioremap_cache(mem->bus.offset, bus_size);
> >> -#endif
> >> - else
> >> - addr = ioremap(mem->bus.offset, bus_size);
> >> - if (!addr) {
> >> - ttm_mem_io_free(bdev, mem);
> >> - return -ENOMEM;
> >> + /* Don't move nonexistent data. Clear destination instead. */
> >> + if (src_ops->maps_tt && (!ttm || !ttm_tt_is_populated(ttm))) {
> >> + if (ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC))
> >> + return;
> >> +
> >> + for (i = 0; i < dst_mem->num_pages; ++i) {
> >> + dst_ops->map_local(dst_iter, &dst_map, i);
> >> + if (dst_map.is_iomem)
> >> + memset_io(dst_map.vaddr_iomem, 0, PAGE_SIZE);
> >> + else
> >> + memset(dst_map.vaddr, 0, PAGE_SIZE);
> >> + if (dst_ops->unmap_local)
> >> + dst_ops->unmap_local(dst_iter, &dst_map);
> >> }
> >> + return;
> >> }
> >> - *virtual = addr;
> >> - return 0;
> >> -}
> >> -
> >> -static void ttm_resource_iounmap(struct ttm_device *bdev,
> >> - struct ttm_resource *mem,
> >> - void *virtual)
> >> -{
> >> - if (virtual && mem->bus.addr == NULL)
> >> - iounmap(virtual);
> >> - ttm_mem_io_free(bdev, mem);
> >> -}
> >> -
> >> -static int ttm_copy_io_page(void *dst, void *src, unsigned long page)
> >> -{
> >> - uint32_t *dstP =
> >> - (uint32_t *) ((unsigned long)dst + (page << PAGE_SHIFT));
> >> - uint32_t *srcP =
> >> - (uint32_t *) ((unsigned long)src + (page << PAGE_SHIFT));
> >> -
> >> - int i;
> >> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> >> - iowrite32(ioread32(srcP++), dstP++);
> >> - return 0;
> >> -}
> >> -
> >> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> >> - unsigned long page,
> >> - pgprot_t prot)
> >> -{
> >> - struct page *d = ttm->pages[page];
> >> - void *dst;
> >> -
> >> - if (!d)
> >> - return -ENOMEM;
> >> -
> >> - src = (void *)((unsigned long)src + (page << PAGE_SHIFT));
> >> - dst = kmap_atomic_prot(d, prot);
> >> - if (!dst)
> >> - return -ENOMEM;
> >> -
> >> - memcpy_fromio(dst, src, PAGE_SIZE);
> >> -
> >> - kunmap_atomic(dst);
> >> -
> >> - return 0;
> >> -}
> >> -
> >> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> >> - unsigned long page,
> >> - pgprot_t prot)
> >> -{
> >> - struct page *s = ttm->pages[page];
> >> - void *src;
> >> -
> >> - if (!s)
> >> - return -ENOMEM;
> >> -
> >> - dst = (void *)((unsigned long)dst + (page << PAGE_SHIFT));
> >> - src = kmap_atomic_prot(s, prot);
> >> - if (!src)
> >> - return -ENOMEM;
> >>
> >> - memcpy_toio(dst, src, PAGE_SIZE);
> >> -
> >> - kunmap_atomic(src);
> >> + for (i = 0; i < dst_mem->num_pages; ++i) {
> >> + dst_ops->map_local(dst_iter, &dst_map, i);
> >> + src_ops->map_local(src_iter, &src_map, i);
> >> +
> >> + if (!src_map.is_iomem && !dst_map.is_iomem) {
> >> + memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> >> + } else if (!src_map.is_iomem) {
> >> + dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> >> + PAGE_SIZE);
> >> + } else if (!dst_map.is_iomem) {
> >> + memcpy_fromio(dst_map.vaddr, src_map.vaddr_iomem,
> >> + PAGE_SIZE);
> >> + } else {
> >> + int j;
> >> + u32 __iomem *src = src_map.vaddr_iomem;
> >> + u32 __iomem *dst = dst_map.vaddr_iomem;
> >>
> >> - return 0;
> >> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> > IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>
> OK, will fix.
>
>
> >
> >> + iowrite32(ioread32(src++), dst++);
> >> + }
> >> + if (src_ops->unmap_local)
> >> + src_ops->unmap_local(src_iter, &src_map);
> >> + if (dst_ops->unmap_local)
> >> + dst_ops->unmap_local(dst_iter, &dst_map);
> >> + }
> >> }
> >> +EXPORT_SYMBOL(ttm_move_memcpy);
> >>
> >> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> >> struct ttm_operation_ctx *ctx,
> >> - struct ttm_resource *new_mem)
> >> + struct ttm_resource *dst_mem)
> >> {
> >> struct ttm_device *bdev = bo->bdev;
> >> - struct ttm_resource_manager *man = ttm_manager_type(bdev, new_mem->mem_type);
> >> + struct ttm_resource_manager *dst_man =
> >> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> >> struct ttm_tt *ttm = bo->ttm;
> >> - struct ttm_resource *old_mem = &bo->mem;
> >> - struct ttm_resource old_copy = *old_mem;
> >> - void *old_iomap;
> >> - void *new_iomap;
> >> + struct ttm_resource *src_mem = &bo->mem;
> >> + struct ttm_resource_manager *src_man =
> >> + ttm_manager_type(bdev, src_mem->mem_type);
> >> + struct ttm_resource src_copy = *src_mem;
> >> + union {
> >> + struct ttm_kmap_iter_tt tt;
> >> + struct ttm_kmap_iter_linear_io io;
> >> + } _dst_iter, _src_iter;
> >> + struct ttm_kmap_iter *dst_iter, *src_iter;
> >> int ret;
> >> - unsigned long i;
> >>
> >> - ret = ttm_bo_wait_ctx(bo, ctx);
> >> - if (ret)
> >> - return ret;
> >> -
> >> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> >> - if (ret)
> >> - return ret;
> >> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> >> - if (ret)
> >> - goto out;
> >> -
> >> - /*
> >> - * Single TTM move. NOP.
> >> - */
> >> - if (old_iomap == NULL && new_iomap == NULL)
> >> - goto out2;
> >> -
> >> - /*
> >> - * Don't move nonexistent data. Clear destination instead.
> >> - */
> >> - if (old_iomap == NULL &&
> >> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> >> - !(ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)))) {
> >> - memset_io(new_iomap, 0, new_mem->num_pages*PAGE_SIZE);
> >> - goto out2;
> >> - }
> >> -
> >> - /*
> >> - * TTM might be null for moves within the same region.
> >> - */
> >> - if (ttm) {
> >> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED) ||
> >> + dst_man->use_tt)) {
> >> ret = ttm_tt_populate(bdev, ttm, ctx);
> >> if (ret)
> >> - goto out1;
> >> + return ret;
> >> }
> >>
> >> - for (i = 0; i < new_mem->num_pages; ++i) {
> >> - if (old_iomap == NULL) {
> >> - pgprot_t prot = ttm_io_prot(bo, old_mem, PAGE_KERNEL);
> >> - ret = ttm_copy_ttm_io_page(ttm, new_iomap, i,
> >> - prot);
> >> - } else if (new_iomap == NULL) {
> >> - pgprot_t prot = ttm_io_prot(bo, new_mem, PAGE_KERNEL);
> >> - ret = ttm_copy_io_ttm_page(ttm, old_iomap, i,
> >> - prot);
> >> - } else {
> >> - ret = ttm_copy_io_page(new_iomap, old_iomap, i);
> >> - }
> >> - if (ret)
> >> - goto out1;
> >> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io, bdev, dst_mem);
> >> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> >> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm);
> >> + if (IS_ERR(dst_iter))
> >> + return PTR_ERR(dst_iter);
> >> +
> >> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io, bdev, src_mem);
> >> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> >> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm);
> >> + if (IS_ERR(src_iter)) {
> >> + ret = PTR_ERR(src_iter);
> >> + goto out_src_iter;
> >> }
> >> - mb();
> >> -out2:
> >> - old_copy = *old_mem;
> >>
> >> - ttm_bo_assign_mem(bo, new_mem);
> >> -
> >> - if (!man->use_tt)
> >> - ttm_bo_tt_destroy(bo);
> >> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> >> + src_copy = *src_mem;
> >> + ttm_bo_move_sync_cleanup(bo, dst_mem);
> >>
> >> -out1:
> >> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> >> -out:
> >> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> >> + if (!src_iter->ops->maps_tt)
> >> + ttm_kmap_iter_linear_io_fini(&_src_iter.io, bdev, &src_copy);
> >> +out_src_iter:
> >> + if (!dst_iter->ops->maps_tt)
> >> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io, bdev, dst_mem);
> >>
> >> - /*
> >> - * On error, keep the mm node!
> >> - */
> >> - if (!ret)
> >> - ttm_resource_free(bo, &old_copy);
> >> return ret;
> >> }
> >> EXPORT_SYMBOL(ttm_bo_move_memcpy);
> >> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct ttm_buffer_object *bo, struct ttm_resource *res,
> >> man = ttm_manager_type(bo->bdev, res->mem_type);
> >> caching = man->use_tt ? bo->ttm->caching : res->bus.caching;
> >>
> >> - /* Cached mappings need no adjustment */
> >> - if (caching == ttm_cached)
> >> - return tmp;
> >> -
> >> -#if defined(__i386__) || defined(__x86_64__)
> >> - if (caching == ttm_write_combined)
> >> - tmp = pgprot_writecombine(tmp);
> >> - else if (boot_cpu_data.x86 > 3)
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> -#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> >> - defined(__powerpc__) || defined(__mips__)
> >> - if (caching == ttm_write_combined)
> >> - tmp = pgprot_writecombine(tmp);
> >> - else
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> -#if defined(__sparc__)
> >> - tmp = pgprot_noncached(tmp);
> >> -#endif
> >> - return tmp;
> >> + return ttm_prot_from_caching(caching, tmp);
> >> }
> >> EXPORT_SYMBOL(ttm_io_prot);
> >>
> >> diff --git a/drivers/gpu/drm/ttm/ttm_module.c b/drivers/gpu/drm/ttm/ttm_module.c
> >> index 56b0efdba1a9..997c458f68a9 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_module.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_module.c
> >> @@ -31,12 +31,47 @@
> >> */
> >> #include <linux/module.h>
> >> #include <linux/device.h>
> >> +#include <linux/pgtable.h>
> >> #include <linux/sched.h>
> >> #include <linux/debugfs.h>
> >> #include <drm/drm_sysfs.h>
> >> +#include <drm/ttm/ttm_caching.h>
> >>
> >> #include "ttm_module.h"
> >>
> >> +/**
> >> + * ttm_prot_from_caching - Modify the page protection according to the
> >> + * ttm cacing mode
> >> + * @caching: The ttm caching mode
> >> + * @tmp: The original page protection
> >> + *
> >> + * Return: The modified page protection
> >> + */
> >> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching, pgprot_t tmp)
> >> +{
> >> + /* Cached mappings need no adjustment */
> >> + if (caching == ttm_cached)
> >> + return tmp;
> >> +
> >> +#if defined(__i386__) || defined(__x86_64__)
> >> + if (caching == ttm_write_combined)
> >> + tmp = pgprot_writecombine(tmp);
> >> + else if (boot_cpu_data.x86 > 3)
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> +#if defined(__ia64__) || defined(__arm__) || defined(__aarch64__) || \
> >> + defined(__powerpc__) || defined(__mips__)
> >> + if (caching == ttm_write_combined)
> >> + tmp = pgprot_writecombine(tmp);
> >> + else
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> +#if defined(__sparc__)
> >> + tmp = pgprot_noncached(tmp);
> >> +#endif
> >> + return tmp;
> >> +}
> >> +
> >> struct dentry *ttm_debugfs_root;
> >>
> >> static int __init ttm_init(void)
> >> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c b/drivers/gpu/drm/ttm/ttm_resource.c
> >> index 59e2b7157e41..e05ae7e3d477 100644
> >> --- a/drivers/gpu/drm/ttm/ttm_resource.c
> >> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> >> @@ -22,6 +22,10 @@
> >> * Authors: Christian König
> >> */
> >>
> >> +#include <linux/dma-buf-map.h>
> >> +#include <linux/io-mapping.h>
> >> +#include <linux/scatterlist.h>
> >> +
> >> #include <drm/ttm/ttm_resource.h>
> >> #include <drm/ttm/ttm_bo_driver.h>
> >>
> >> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct ttm_resource_manager *man,
> >> man->func->debug(man, p);
> >> }
> >> EXPORT_SYMBOL(ttm_resource_manager_debug);
> >> +
> >> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *dmap,
> >> + pgoff_t i)
> >> +{
> >> + struct ttm_kmap_iter_iomap *iter_io =
> >> + container_of(iter, typeof(*iter_io), base);
> >> + void __iomem *addr;
> >> +
> >> +retry:
> >> + while (i >= iter_io->cache.end) {
> >> + iter_io->cache.sg = iter_io->cache.sg ?
> >> + sg_next(iter_io->cache.sg) : iter_io->st->sgl;
> >> + iter_io->cache.i = iter_io->cache.end;
> >> + iter_io->cache.end += sg_dma_len(iter_io->cache.sg) >>
> >> + PAGE_SHIFT;
> >> + iter_io->cache.offs = sg_dma_address(iter_io->cache.sg) -
> >> + iter_io->start;
> >> + }
> >> +
> >> + if (i < iter_io->cache.i) {
> >> + iter_io->cache.end = 0;
> >> + iter_io->cache.sg = NULL;
> >> + goto retry;
> >> + }
> >> +
> >> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io->cache.offs +
> >> + (((resource_size_t)i - iter_io->cache.i)
> >> + << PAGE_SHIFT));
> >> + dma_buf_map_set_vaddr_iomem(dmap, addr);
> >> +}
> >> +
> >> +static void ttm_kmap_iter_iomap_unmap_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *map)
> >> +{
> >> + io_mapping_unmap_local(map->vaddr_iomem);
> >> +}
> >> +
> >> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> >> + .map_local = ttm_kmap_iter_iomap_map_local,
> >> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> >> + .maps_tt = false,
> >> +};
> >> +
> >> +/**
> >> + * ttm_kmap_iter_iomap_init - Initialize a struct ttm_kmap_iter_iomap
> >> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> >> + * @iomap: The struct io_mapping representing the underlying linear io_memory.
> >> + * @st: sg_table into @iomap, representing the memory of the struct
> >> + * ttm_resource.
> >> + * @start: Offset that needs to be subtracted from @st to make
> >> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> >> + *
> >> + * Return: Pointer to the embedded struct ttm_kmap_iter.
> >> + */
> >> +struct ttm_kmap_iter *
> >> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> >> + struct io_mapping *iomap,
> >> + struct sg_table *st,
> >> + resource_size_t start)
> >> +{
> >> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> >> + iter_io->iomap = iomap;
> >> + iter_io->st = st;
> >> + iter_io->start = start;
> >> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> >> +
> >> + return &iter_io->base;
> >> +}
> >> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> >> +
> >> +/**
> >> + * DOC: Linear io iterator
> >> + *
> >> + * This code should die in the not too near future. Best would be if we could
> >> + * make io-mapping use memremap for all io memory, and have memremap
> >> + * implement a kmap_local functionality. We could then strip a huge amount of
> >> + * code. These linear io iterators are implemented to mimic old functionality,
> >> + * and they don't use kmap_local semantics at all internally. Rather ioremap or
> >> + * friends, and at least on 32-bit they add global TLB flushes and points
> >> + * of failure.
> >> + */
> >> +
> >> +static void ttm_kmap_iter_linear_io_map_local(struct ttm_kmap_iter *iter,
> >> + struct dma_buf_map *dmap,
> >> + pgoff_t i)
> >> +{
> >> + struct ttm_kmap_iter_linear_io *iter_io =
> >> + container_of(iter, typeof(*iter_io), base);
> >> +
> >> + *dmap = iter_io->dmap;
> >> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> >> +}
> >> +
> >> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_linear_io_ops = {
> >> + .map_local = ttm_kmap_iter_linear_io_map_local,
> >> + .maps_tt = false,
> >> +};
> >> +
> >> +struct ttm_kmap_iter *
> >> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io *iter_io,
> >> + struct ttm_device *bdev,
> >> + struct ttm_resource *mem)
> >> +{
> >> + int ret;
> >> +
> >> + ret = ttm_mem_io_reserve(bdev, mem);
> >> + if (ret)
> >> + goto out_err;
> >> + if (!mem->bus.is_iomem) {
> >> + ret = -EINVAL;
> >> + goto out_io_free;
> >> + }
> >> +
> >> + if (mem->bus.addr) {
> >> + dma_buf_map_set_vaddr(&iter_io->dmap, mem->bus.addr);
> >> + iter_io->needs_unmap = false;
> >> + } else {
> >> + size_t bus_size = (size_t)mem->num_pages << PAGE_SHIFT;
> >> +
> >> + iter_io->needs_unmap = true;
> >> + if (mem->bus.caching == ttm_write_combined)
> >> + dma_buf_map_set_vaddr_iomem(&iter_io->dmap,
> >> + ioremap_wc(mem->bus.offset,
> >> + bus_size));
> >> + else if (mem->bus.caching == ttm_cached)
> >> + dma_buf_map_set_vaddr(&iter_io->dmap,
> >> + memremap(mem->bus.offset, bus_size,
> >> + MEMREMAP_WB));
> > The comments in set_vaddr suggest that this is meant for
> > system-memory. Does that actually matter or is it just about not
> > losing the __iomem annotation on platforms where it matters?
>
> Yes, it's the latter. dma_buf_map() is relatively new and the author
> probably didn't think about the case of cached iomem, which is used by,
> for example, vmwgfx.
>
> > Apparently cached device local is a thing. Also should this not be
> > wrapped in CONFIG_X86?
>
> Both dma_buf_map() and memremap are generic, I think, I guess memremap
> would return NULL if it's not supported.
It looks like memremap just wraps ioremap_cache, but since it also
discards the __iomem annotation should we be doing that universally?
Also not sure if ioremap_cache is universally supported, so wrapping
in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
at least that looks like roughly what the previous code was doing? Not
too sure tbh.
>
> /Thomas
>
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-25 9:58 ` Matthew Auld
@ 2021-05-25 10:07 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 10:07 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
> On Tue, 25 May 2021 at 10:32, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> >
> > On 5/25/21 11:18 AM, Matthew Auld wrote:
> > > On Fri, 21 May 2021 at 16:33, Thomas Hellström
> > > <thomas.hellstrom@linux.intel.com> wrote:
> > > > The internal ttm_bo_util memcpy uses ioremap functionality, and
> > > > while it
> > > > probably might be possible to use it for copying in- and out of
> > > > sglist represented io memory, using io_mem_reserve() /
> > > > io_mem_free()
> > > > callbacks, that would cause problems with fault().
> > > > Instead, implement a method mapping page-by-page using
> > > > kmap_local()
> > > > semantics. As an additional benefit we then avoid the
> > > > occasional global
> > > > TLB flushes of ioremap() and consuming ioremap space,
> > > > elimination of a
> > > > critical point of failure and with a slight change of semantics
> > > > we could
> > > > also push the memcpy out async for testing and async driver
> > > > development
> > > > purposes.
> > > >
> > > > A special linear iomem iterator is introduced internally to
> > > > mimic the
> > > > old ioremap behaviour for code-paths that can't immediately be
> > > > ported
> > > > over. This adds to the code size and should be considered a
> > > > temporary
> > > > solution.
> > > >
> > > > Looking at the code we have a lot of checks for iomap tagged
> > > > pointers.
> > > > Ideally we should extend the core memremap functions to also
> > > > accept
> > > > uncached memory and kmap_local functionality. Then we could
> > > > strip a
> > > > lot of code.
> > > >
> > > > Cc: Christian König <christian.koenig@amd.com>
> > > > Signed-off-by: Thomas Hellström <
> > > > thomas.hellstrom@linux.intel.com>
> > > > ---
> > > > v3:
> > > > - Split up in various TTM files and addressed review comments
> > > > by
> > > > Christian König. Tested and fixed legacy iomap memcpy path
> > > > on i915.
> > > > ---
> > > > drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
> > > > ---------
> > > > drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> > > > drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> > > > drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> > > > include/drm/ttm/ttm_bo_driver.h | 28 +++
> > > > include/drm/ttm/ttm_caching.h | 2 +
> > > > include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> > > > include/drm/ttm/ttm_resource.h | 61 +++++++
> > > > include/drm/ttm/ttm_tt.h | 16 ++
> > > > 9 files changed, 508 insertions(+), 181 deletions(-)
> > > > create mode 100644 include/drm/ttm/ttm_kmap_iter.h
> > > >
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > index ae8b61460724..912cbe8e60a2 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
> > > > *bdev,
> > > > mem->bus.addr = NULL;
> > > > }
> > > >
> > > > -static int ttm_resource_ioremap(struct ttm_device *bdev,
> > > > - struct ttm_resource *mem,
> > > > - void **virtual)
> > > > +/**
> > > > + * ttm_move_memcpy - Helper to perform a memcpy ttm move
> > > > operation.
> > > > + * @bo: The struct ttm_buffer_object.
> > > > + * @new_mem: The struct ttm_resource we're moving to (copy
> > > > destination).
> > > > + * @new_iter: A struct ttm_kmap_iter representing the
> > > > destination resource.
> > > > + * @src_iter: A struct ttm_kmap_iter representing the source
> > > > resource.
> > > > + *
> > > > + * This function is intended to be able to move out async
> > > > under a
> > > > + * dma-fence if desired.
> > > > + */
> > > > +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> > > > + struct ttm_resource *dst_mem,
> > > > + struct ttm_kmap_iter *dst_iter,
> > > > + struct ttm_kmap_iter *src_iter)
> > > > {
> > > > - int ret;
> > > > - void *addr;
> > > > -
> > > > - *virtual = NULL;
> > > > - ret = ttm_mem_io_reserve(bdev, mem);
> > > > - if (ret || !mem->bus.is_iomem)
> > > > - return ret;
> > > > + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
> > > > >ops;
> > > > + const struct ttm_kmap_iter_ops *src_ops = src_iter-
> > > > >ops;
> > > > + struct ttm_tt *ttm = bo->ttm;
> > > > + struct dma_buf_map src_map, dst_map;
> > > > + pgoff_t i;
> > > >
> > > > - if (mem->bus.addr) {
> > > > - addr = mem->bus.addr;
> > > > - } else {
> > > > - size_t bus_size = (size_t)mem->num_pages <<
> > > > PAGE_SHIFT;
> > > > + /* Single TTM move. NOP */
> > > > + if (dst_ops->maps_tt && src_ops->maps_tt)
> > > > + return;
> > > >
> > > > - if (mem->bus.caching == ttm_write_combined)
> > > > - addr = ioremap_wc(mem->bus.offset,
> > > > bus_size);
> > > > -#ifdef CONFIG_X86
> > > > - else if (mem->bus.caching == ttm_cached)
> > > > - addr = ioremap_cache(mem->bus.offset,
> > > > bus_size);
> > > > -#endif
> > > > - else
> > > > - addr = ioremap(mem->bus.offset,
> > > > bus_size);
> > > > - if (!addr) {
> > > > - ttm_mem_io_free(bdev, mem);
> > > > - return -ENOMEM;
> > > > + /* Don't move nonexistent data. Clear destination
> > > > instead. */
> > > > + if (src_ops->maps_tt && (!ttm ||
> > > > !ttm_tt_is_populated(ttm))) {
> > > > + if (ttm && !(ttm->page_flags &
> > > > TTM_PAGE_FLAG_ZERO_ALLOC))
> > > > + return;
> > > > +
> > > > + for (i = 0; i < dst_mem->num_pages; ++i) {
> > > > + dst_ops->map_local(dst_iter, &dst_map,
> > > > i);
> > > > + if (dst_map.is_iomem)
> > > > + memset_io(dst_map.vaddr_iomem,
> > > > 0, PAGE_SIZE);
> > > > + else
> > > > + memset(dst_map.vaddr, 0,
> > > > PAGE_SIZE);
> > > > + if (dst_ops->unmap_local)
> > > > + dst_ops->unmap_local(dst_iter,
> > > > &dst_map);
> > > > }
> > > > + return;
> > > > }
> > > > - *virtual = addr;
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static void ttm_resource_iounmap(struct ttm_device *bdev,
> > > > - struct ttm_resource *mem,
> > > > - void *virtual)
> > > > -{
> > > > - if (virtual && mem->bus.addr == NULL)
> > > > - iounmap(virtual);
> > > > - ttm_mem_io_free(bdev, mem);
> > > > -}
> > > > -
> > > > -static int ttm_copy_io_page(void *dst, void *src, unsigned
> > > > long page)
> > > > -{
> > > > - uint32_t *dstP =
> > > > - (uint32_t *) ((unsigned long)dst + (page <<
> > > > PAGE_SHIFT));
> > > > - uint32_t *srcP =
> > > > - (uint32_t *) ((unsigned long)src + (page <<
> > > > PAGE_SHIFT));
> > > > -
> > > > - int i;
> > > > - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> > > > - iowrite32(ioread32(srcP++), dstP++);
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> > > > - unsigned long page,
> > > > - pgprot_t prot)
> > > > -{
> > > > - struct page *d = ttm->pages[page];
> > > > - void *dst;
> > > > -
> > > > - if (!d)
> > > > - return -ENOMEM;
> > > > -
> > > > - src = (void *)((unsigned long)src + (page <<
> > > > PAGE_SHIFT));
> > > > - dst = kmap_atomic_prot(d, prot);
> > > > - if (!dst)
> > > > - return -ENOMEM;
> > > > -
> > > > - memcpy_fromio(dst, src, PAGE_SIZE);
> > > > -
> > > > - kunmap_atomic(dst);
> > > > -
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> > > > - unsigned long page,
> > > > - pgprot_t prot)
> > > > -{
> > > > - struct page *s = ttm->pages[page];
> > > > - void *src;
> > > > -
> > > > - if (!s)
> > > > - return -ENOMEM;
> > > > -
> > > > - dst = (void *)((unsigned long)dst + (page <<
> > > > PAGE_SHIFT));
> > > > - src = kmap_atomic_prot(s, prot);
> > > > - if (!src)
> > > > - return -ENOMEM;
> > > >
> > > > - memcpy_toio(dst, src, PAGE_SIZE);
> > > > -
> > > > - kunmap_atomic(src);
> > > > + for (i = 0; i < dst_mem->num_pages; ++i) {
> > > > + dst_ops->map_local(dst_iter, &dst_map, i);
> > > > + src_ops->map_local(src_iter, &src_map, i);
> > > > +
> > > > + if (!src_map.is_iomem && !dst_map.is_iomem) {
> > > > + memcpy(dst_map.vaddr, src_map.vaddr,
> > > > PAGE_SIZE);
> > > > + } else if (!src_map.is_iomem) {
> > > > + dma_buf_map_memcpy_to(&dst_map,
> > > > src_map.vaddr,
> > > > + PAGE_SIZE);
> > > > + } else if (!dst_map.is_iomem) {
> > > > + memcpy_fromio(dst_map.vaddr,
> > > > src_map.vaddr_iomem,
> > > > + PAGE_SIZE);
> > > > + } else {
> > > > + int j;
> > > > + u32 __iomem *src = src_map.vaddr_iomem;
> > > > + u32 __iomem *dst = dst_map.vaddr_iomem;
> > > >
> > > > - return 0;
> > > > + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> > > IMO PAGE_SIZE / sizeof(u32) is easier to understand.
> >
> > OK, will fix.
> >
> >
> > >
> > > > + iowrite32(ioread32(src++),
> > > > dst++);
> > > > + }
> > > > + if (src_ops->unmap_local)
> > > > + src_ops->unmap_local(src_iter,
> > > > &src_map);
> > > > + if (dst_ops->unmap_local)
> > > > + dst_ops->unmap_local(dst_iter,
> > > > &dst_map);
> > > > + }
> > > > }
> > > > +EXPORT_SYMBOL(ttm_move_memcpy);
> > > >
> > > > int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> > > > struct ttm_operation_ctx *ctx,
> > > > - struct ttm_resource *new_mem)
> > > > + struct ttm_resource *dst_mem)
> > > > {
> > > > struct ttm_device *bdev = bo->bdev;
> > > > - struct ttm_resource_manager *man =
> > > > ttm_manager_type(bdev, new_mem->mem_type);
> > > > + struct ttm_resource_manager *dst_man =
> > > > + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> > > > struct ttm_tt *ttm = bo->ttm;
> > > > - struct ttm_resource *old_mem = &bo->mem;
> > > > - struct ttm_resource old_copy = *old_mem;
> > > > - void *old_iomap;
> > > > - void *new_iomap;
> > > > + struct ttm_resource *src_mem = &bo->mem;
> > > > + struct ttm_resource_manager *src_man =
> > > > + ttm_manager_type(bdev, src_mem->mem_type);
> > > > + struct ttm_resource src_copy = *src_mem;
> > > > + union {
> > > > + struct ttm_kmap_iter_tt tt;
> > > > + struct ttm_kmap_iter_linear_io io;
> > > > + } _dst_iter, _src_iter;
> > > > + struct ttm_kmap_iter *dst_iter, *src_iter;
> > > > int ret;
> > > > - unsigned long i;
> > > >
> > > > - ret = ttm_bo_wait_ctx(bo, ctx);
> > > > - if (ret)
> > > > - return ret;
> > > > -
> > > > - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> > > > - if (ret)
> > > > - return ret;
> > > > - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> > > > - if (ret)
> > > > - goto out;
> > > > -
> > > > - /*
> > > > - * Single TTM move. NOP.
> > > > - */
> > > > - if (old_iomap == NULL && new_iomap == NULL)
> > > > - goto out2;
> > > > -
> > > > - /*
> > > > - * Don't move nonexistent data. Clear destination
> > > > instead.
> > > > - */
> > > > - if (old_iomap == NULL &&
> > > > - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> > > > - !(ttm->page_flags &
> > > > TTM_PAGE_FLAG_SWAPPED)))) {
> > > > - memset_io(new_iomap, 0, new_mem-
> > > > >num_pages*PAGE_SIZE);
> > > > - goto out2;
> > > > - }
> > > > -
> > > > - /*
> > > > - * TTM might be null for moves within the same region.
> > > > - */
> > > > - if (ttm) {
> > > > + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
> > > > ||
> > > > + dst_man->use_tt)) {
> > > > ret = ttm_tt_populate(bdev, ttm, ctx);
> > > > if (ret)
> > > > - goto out1;
> > > > + return ret;
> > > > }
> > > >
> > > > - for (i = 0; i < new_mem->num_pages; ++i) {
> > > > - if (old_iomap == NULL) {
> > > > - pgprot_t prot = ttm_io_prot(bo,
> > > > old_mem, PAGE_KERNEL);
> > > > - ret = ttm_copy_ttm_io_page(ttm,
> > > > new_iomap, i,
> > > > - prot);
> > > > - } else if (new_iomap == NULL) {
> > > > - pgprot_t prot = ttm_io_prot(bo,
> > > > new_mem, PAGE_KERNEL);
> > > > - ret = ttm_copy_io_ttm_page(ttm,
> > > > old_iomap, i,
> > > > - prot);
> > > > - } else {
> > > > - ret = ttm_copy_io_page(new_iomap,
> > > > old_iomap, i);
> > > > - }
> > > > - if (ret)
> > > > - goto out1;
> > > > + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
> > > > bdev, dst_mem);
> > > > + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> > > > + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
> > > > bo->ttm);
> > > > + if (IS_ERR(dst_iter))
> > > > + return PTR_ERR(dst_iter);
> > > > +
> > > > + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
> > > > bdev, src_mem);
> > > > + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> > > > + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
> > > > bo->ttm);
> > > > + if (IS_ERR(src_iter)) {
> > > > + ret = PTR_ERR(src_iter);
> > > > + goto out_src_iter;
> > > > }
> > > > - mb();
> > > > -out2:
> > > > - old_copy = *old_mem;
> > > >
> > > > - ttm_bo_assign_mem(bo, new_mem);
> > > > -
> > > > - if (!man->use_tt)
> > > > - ttm_bo_tt_destroy(bo);
> > > > + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> > > > + src_copy = *src_mem;
> > > > + ttm_bo_move_sync_cleanup(bo, dst_mem);
> > > >
> > > > -out1:
> > > > - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> > > > -out:
> > > > - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> > > > + if (!src_iter->ops->maps_tt)
> > > > + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
> > > > bdev, &src_copy);
> > > > +out_src_iter:
> > > > + if (!dst_iter->ops->maps_tt)
> > > > + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
> > > > bdev, dst_mem);
> > > >
> > > > - /*
> > > > - * On error, keep the mm node!
> > > > - */
> > > > - if (!ret)
> > > > - ttm_resource_free(bo, &old_copy);
> > > > return ret;
> > > > }
> > > > EXPORT_SYMBOL(ttm_bo_move_memcpy);
> > > > @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
> > > > ttm_buffer_object *bo, struct ttm_resource *res,
> > > > man = ttm_manager_type(bo->bdev, res->mem_type);
> > > > caching = man->use_tt ? bo->ttm->caching : res-
> > > > >bus.caching;
> > > >
> > > > - /* Cached mappings need no adjustment */
> > > > - if (caching == ttm_cached)
> > > > - return tmp;
> > > > -
> > > > -#if defined(__i386__) || defined(__x86_64__)
> > > > - if (caching == ttm_write_combined)
> > > > - tmp = pgprot_writecombine(tmp);
> > > > - else if (boot_cpu_data.x86 > 3)
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > -#if defined(__ia64__) || defined(__arm__) ||
> > > > defined(__aarch64__) || \
> > > > - defined(__powerpc__) || defined(__mips__)
> > > > - if (caching == ttm_write_combined)
> > > > - tmp = pgprot_writecombine(tmp);
> > > > - else
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > -#if defined(__sparc__)
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > - return tmp;
> > > > + return ttm_prot_from_caching(caching, tmp);
> > > > }
> > > > EXPORT_SYMBOL(ttm_io_prot);
> > > >
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_module.c
> > > > b/drivers/gpu/drm/ttm/ttm_module.c
> > > > index 56b0efdba1a9..997c458f68a9 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_module.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_module.c
> > > > @@ -31,12 +31,47 @@
> > > > */
> > > > #include <linux/module.h>
> > > > #include <linux/device.h>
> > > > +#include <linux/pgtable.h>
> > > > #include <linux/sched.h>
> > > > #include <linux/debugfs.h>
> > > > #include <drm/drm_sysfs.h>
> > > > +#include <drm/ttm/ttm_caching.h>
> > > >
> > > > #include "ttm_module.h"
> > > >
> > > > +/**
> > > > + * ttm_prot_from_caching - Modify the page protection
> > > > according to the
> > > > + * ttm cacing mode
> > > > + * @caching: The ttm caching mode
> > > > + * @tmp: The original page protection
> > > > + *
> > > > + * Return: The modified page protection
> > > > + */
> > > > +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
> > > > pgprot_t tmp)
> > > > +{
> > > > + /* Cached mappings need no adjustment */
> > > > + if (caching == ttm_cached)
> > > > + return tmp;
> > > > +
> > > > +#if defined(__i386__) || defined(__x86_64__)
> > > > + if (caching == ttm_write_combined)
> > > > + tmp = pgprot_writecombine(tmp);
> > > > + else if (boot_cpu_data.x86 > 3)
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > +#if defined(__ia64__) || defined(__arm__) ||
> > > > defined(__aarch64__) || \
> > > > + defined(__powerpc__) || defined(__mips__)
> > > > + if (caching == ttm_write_combined)
> > > > + tmp = pgprot_writecombine(tmp);
> > > > + else
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > +#if defined(__sparc__)
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > + return tmp;
> > > > +}
> > > > +
> > > > struct dentry *ttm_debugfs_root;
> > > >
> > > > static int __init ttm_init(void)
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
> > > > b/drivers/gpu/drm/ttm/ttm_resource.c
> > > > index 59e2b7157e41..e05ae7e3d477 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_resource.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> > > > @@ -22,6 +22,10 @@
> > > > * Authors: Christian König
> > > > */
> > > >
> > > > +#include <linux/dma-buf-map.h>
> > > > +#include <linux/io-mapping.h>
> > > > +#include <linux/scatterlist.h>
> > > > +
> > > > #include <drm/ttm/ttm_resource.h>
> > > > #include <drm/ttm/ttm_bo_driver.h>
> > > >
> > > > @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
> > > > ttm_resource_manager *man,
> > > > man->func->debug(man, p);
> > > > }
> > > > EXPORT_SYMBOL(ttm_resource_manager_debug);
> > > > +
> > > > +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
> > > > *iter,
> > > > + struct dma_buf_map
> > > > *dmap,
> > > > + pgoff_t i)
> > > > +{
> > > > + struct ttm_kmap_iter_iomap *iter_io =
> > > > + container_of(iter, typeof(*iter_io), base);
> > > > + void __iomem *addr;
> > > > +
> > > > +retry:
> > > > + while (i >= iter_io->cache.end) {
> > > > + iter_io->cache.sg = iter_io->cache.sg ?
> > > > + sg_next(iter_io->cache.sg) : iter_io-
> > > > >st->sgl;
> > > > + iter_io->cache.i = iter_io->cache.end;
> > > > + iter_io->cache.end += sg_dma_len(iter_io-
> > > > >cache.sg) >>
> > > > + PAGE_SHIFT;
> > > > + iter_io->cache.offs = sg_dma_address(iter_io-
> > > > >cache.sg) -
> > > > + iter_io->start;
> > > > + }
> > > > +
> > > > + if (i < iter_io->cache.i) {
> > > > + iter_io->cache.end = 0;
> > > > + iter_io->cache.sg = NULL;
> > > > + goto retry;
> > > > + }
> > > > +
> > > > + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
> > > > >cache.offs +
> > > > + (((resource_size_t)i -
> > > > iter_io->cache.i)
> > > > + << PAGE_SHIFT));
> > > > + dma_buf_map_set_vaddr_iomem(dmap, addr);
> > > > +}
> > > > +
> > > > +static void ttm_kmap_iter_iomap_unmap_local(struct
> > > > ttm_kmap_iter *iter,
> > > > + struct dma_buf_map
> > > > *map)
> > > > +{
> > > > + io_mapping_unmap_local(map->vaddr_iomem);
> > > > +}
> > > > +
> > > > +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> > > > + .map_local = ttm_kmap_iter_iomap_map_local,
> > > > + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> > > > + .maps_tt = false,
> > > > +};
> > > > +
> > > > +/**
> > > > + * ttm_kmap_iter_iomap_init - Initialize a struct
> > > > ttm_kmap_iter_iomap
> > > > + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> > > > + * @iomap: The struct io_mapping representing the underlying
> > > > linear io_memory.
> > > > + * @st: sg_table into @iomap, representing the memory of the
> > > > struct
> > > > + * ttm_resource.
> > > > + * @start: Offset that needs to be subtracted from @st to make
> > > > + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> > > > + *
> > > > + * Return: Pointer to the embedded struct ttm_kmap_iter.
> > > > + */
> > > > +struct ttm_kmap_iter *
> > > > +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> > > > + struct io_mapping *iomap,
> > > > + struct sg_table *st,
> > > > + resource_size_t start)
> > > > +{
> > > > + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> > > > + iter_io->iomap = iomap;
> > > > + iter_io->st = st;
> > > > + iter_io->start = start;
> > > > + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> > > > +
> > > > + return &iter_io->base;
> > > > +}
> > > > +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> > > > +
> > > > +/**
> > > > + * DOC: Linear io iterator
> > > > + *
> > > > + * This code should die in the not too near future. Best would
> > > > be if we could
> > > > + * make io-mapping use memremap for all io memory, and have
> > > > memremap
> > > > + * implement a kmap_local functionality. We could then strip a
> > > > huge amount of
> > > > + * code. These linear io iterators are implemented to mimic
> > > > old functionality,
> > > > + * and they don't use kmap_local semantics at all internally.
> > > > Rather ioremap or
> > > > + * friends, and at least on 32-bit they add global TLB flushes
> > > > and points
> > > > + * of failure.
> > > > + */
> > > > +
> > > > +static void ttm_kmap_iter_linear_io_map_local(struct
> > > > ttm_kmap_iter *iter,
> > > > + struct
> > > > dma_buf_map *dmap,
> > > > + pgoff_t i)
> > > > +{
> > > > + struct ttm_kmap_iter_linear_io *iter_io =
> > > > + container_of(iter, typeof(*iter_io), base);
> > > > +
> > > > + *dmap = iter_io->dmap;
> > > > + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> > > > +}
> > > > +
> > > > +static const struct ttm_kmap_iter_ops
> > > > ttm_kmap_iter_linear_io_ops = {
> > > > + .map_local = ttm_kmap_iter_linear_io_map_local,
> > > > + .maps_tt = false,
> > > > +};
> > > > +
> > > > +struct ttm_kmap_iter *
> > > > +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
> > > > *iter_io,
> > > > + struct ttm_device *bdev,
> > > > + struct ttm_resource *mem)
> > > > +{
> > > > + int ret;
> > > > +
> > > > + ret = ttm_mem_io_reserve(bdev, mem);
> > > > + if (ret)
> > > > + goto out_err;
> > > > + if (!mem->bus.is_iomem) {
> > > > + ret = -EINVAL;
> > > > + goto out_io_free;
> > > > + }
> > > > +
> > > > + if (mem->bus.addr) {
> > > > + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
> > > > >bus.addr);
> > > > + iter_io->needs_unmap = false;
> > > > + } else {
> > > > + size_t bus_size = (size_t)mem->num_pages <<
> > > > PAGE_SHIFT;
> > > > +
> > > > + iter_io->needs_unmap = true;
> > > > + if (mem->bus.caching == ttm_write_combined)
> > > > + dma_buf_map_set_vaddr_iomem(&iter_io-
> > > > >dmap,
> > > > +
> > > > ioremap_wc(mem->bus.offset,
> > > > +
> > > > bus_size));
> > > > + else if (mem->bus.caching == ttm_cached)
> > > > + dma_buf_map_set_vaddr(&iter_io->dmap,
> > > > + memremap(mem-
> > > > >bus.offset, bus_size,
> > > > +
> > > > MEMREMAP_WB));
> > > The comments in set_vaddr suggest that this is meant for
> > > system-memory. Does that actually matter or is it just about not
> > > losing the __iomem annotation on platforms where it matters?
> >
> > Yes, it's the latter. dma_buf_map() is relatively new and the
> > author
> > probably didn't think about the case of cached iomem, which is used
> > by,
> > for example, vmwgfx.
> >
> > > Apparently cached device local is a thing. Also should this not
> > > be
> > > wrapped in CONFIG_X86?
> >
> > Both dma_buf_map() and memremap are generic, I think, I guess
> > memremap
> > would return NULL if it's not supported.
>
> It looks like memremap just wraps ioremap_cache, but since it also
> discards the __iomem annotation should we be doing that universally?
> Also not sure if ioremap_cache is universally supported, so wrapping
> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
> at least that looks like roughly what the previous code was doing?
> Not
> too sure tbh.
>
I think the long term goal is to use memremap all over the place, to
just not have to bother with the __iomem annotation. But to do that io-
mapping.h needs to support memremap. But for now we need to be strict
about __iomem unless we're in arch specific code. That's why that
dma_buf_map thing was created, but TTM memcpy was never fully adapted.
As for limited arch support for memremap cached, It looks like we only
need to or in "backup" mapping modes in the memremap flags, and we'd
mimic the previous behaviour.
/Thomas
> >
> > /Thomas
> >
> >
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-25 10:07 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 10:07 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
> On Tue, 25 May 2021 at 10:32, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> >
> > On 5/25/21 11:18 AM, Matthew Auld wrote:
> > > On Fri, 21 May 2021 at 16:33, Thomas Hellström
> > > <thomas.hellstrom@linux.intel.com> wrote:
> > > > The internal ttm_bo_util memcpy uses ioremap functionality, and
> > > > while it
> > > > probably might be possible to use it for copying in- and out of
> > > > sglist represented io memory, using io_mem_reserve() /
> > > > io_mem_free()
> > > > callbacks, that would cause problems with fault().
> > > > Instead, implement a method mapping page-by-page using
> > > > kmap_local()
> > > > semantics. As an additional benefit we then avoid the
> > > > occasional global
> > > > TLB flushes of ioremap() and consuming ioremap space,
> > > > elimination of a
> > > > critical point of failure and with a slight change of semantics
> > > > we could
> > > > also push the memcpy out async for testing and async driver
> > > > development
> > > > purposes.
> > > >
> > > > A special linear iomem iterator is introduced internally to
> > > > mimic the
> > > > old ioremap behaviour for code-paths that can't immediately be
> > > > ported
> > > > over. This adds to the code size and should be considered a
> > > > temporary
> > > > solution.
> > > >
> > > > Looking at the code we have a lot of checks for iomap tagged
> > > > pointers.
> > > > Ideally we should extend the core memremap functions to also
> > > > accept
> > > > uncached memory and kmap_local functionality. Then we could
> > > > strip a
> > > > lot of code.
> > > >
> > > > Cc: Christian König <christian.koenig@amd.com>
> > > > Signed-off-by: Thomas Hellström <
> > > > thomas.hellstrom@linux.intel.com>
> > > > ---
> > > > v3:
> > > > - Split up in various TTM files and addressed review comments
> > > > by
> > > > Christian König. Tested and fixed legacy iomap memcpy path
> > > > on i915.
> > > > ---
> > > > drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
> > > > ---------
> > > > drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
> > > > drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
> > > > drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
> > > > include/drm/ttm/ttm_bo_driver.h | 28 +++
> > > > include/drm/ttm/ttm_caching.h | 2 +
> > > > include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
> > > > include/drm/ttm/ttm_resource.h | 61 +++++++
> > > > include/drm/ttm/ttm_tt.h | 16 ++
> > > > 9 files changed, 508 insertions(+), 181 deletions(-)
> > > > create mode 100644 include/drm/ttm/ttm_kmap_iter.h
> > > >
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > index ae8b61460724..912cbe8e60a2 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > > > @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
> > > > *bdev,
> > > > mem->bus.addr = NULL;
> > > > }
> > > >
> > > > -static int ttm_resource_ioremap(struct ttm_device *bdev,
> > > > - struct ttm_resource *mem,
> > > > - void **virtual)
> > > > +/**
> > > > + * ttm_move_memcpy - Helper to perform a memcpy ttm move
> > > > operation.
> > > > + * @bo: The struct ttm_buffer_object.
> > > > + * @new_mem: The struct ttm_resource we're moving to (copy
> > > > destination).
> > > > + * @new_iter: A struct ttm_kmap_iter representing the
> > > > destination resource.
> > > > + * @src_iter: A struct ttm_kmap_iter representing the source
> > > > resource.
> > > > + *
> > > > + * This function is intended to be able to move out async
> > > > under a
> > > > + * dma-fence if desired.
> > > > + */
> > > > +void ttm_move_memcpy(struct ttm_buffer_object *bo,
> > > > + struct ttm_resource *dst_mem,
> > > > + struct ttm_kmap_iter *dst_iter,
> > > > + struct ttm_kmap_iter *src_iter)
> > > > {
> > > > - int ret;
> > > > - void *addr;
> > > > -
> > > > - *virtual = NULL;
> > > > - ret = ttm_mem_io_reserve(bdev, mem);
> > > > - if (ret || !mem->bus.is_iomem)
> > > > - return ret;
> > > > + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
> > > > >ops;
> > > > + const struct ttm_kmap_iter_ops *src_ops = src_iter-
> > > > >ops;
> > > > + struct ttm_tt *ttm = bo->ttm;
> > > > + struct dma_buf_map src_map, dst_map;
> > > > + pgoff_t i;
> > > >
> > > > - if (mem->bus.addr) {
> > > > - addr = mem->bus.addr;
> > > > - } else {
> > > > - size_t bus_size = (size_t)mem->num_pages <<
> > > > PAGE_SHIFT;
> > > > + /* Single TTM move. NOP */
> > > > + if (dst_ops->maps_tt && src_ops->maps_tt)
> > > > + return;
> > > >
> > > > - if (mem->bus.caching == ttm_write_combined)
> > > > - addr = ioremap_wc(mem->bus.offset,
> > > > bus_size);
> > > > -#ifdef CONFIG_X86
> > > > - else if (mem->bus.caching == ttm_cached)
> > > > - addr = ioremap_cache(mem->bus.offset,
> > > > bus_size);
> > > > -#endif
> > > > - else
> > > > - addr = ioremap(mem->bus.offset,
> > > > bus_size);
> > > > - if (!addr) {
> > > > - ttm_mem_io_free(bdev, mem);
> > > > - return -ENOMEM;
> > > > + /* Don't move nonexistent data. Clear destination
> > > > instead. */
> > > > + if (src_ops->maps_tt && (!ttm ||
> > > > !ttm_tt_is_populated(ttm))) {
> > > > + if (ttm && !(ttm->page_flags &
> > > > TTM_PAGE_FLAG_ZERO_ALLOC))
> > > > + return;
> > > > +
> > > > + for (i = 0; i < dst_mem->num_pages; ++i) {
> > > > + dst_ops->map_local(dst_iter, &dst_map,
> > > > i);
> > > > + if (dst_map.is_iomem)
> > > > + memset_io(dst_map.vaddr_iomem,
> > > > 0, PAGE_SIZE);
> > > > + else
> > > > + memset(dst_map.vaddr, 0,
> > > > PAGE_SIZE);
> > > > + if (dst_ops->unmap_local)
> > > > + dst_ops->unmap_local(dst_iter,
> > > > &dst_map);
> > > > }
> > > > + return;
> > > > }
> > > > - *virtual = addr;
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static void ttm_resource_iounmap(struct ttm_device *bdev,
> > > > - struct ttm_resource *mem,
> > > > - void *virtual)
> > > > -{
> > > > - if (virtual && mem->bus.addr == NULL)
> > > > - iounmap(virtual);
> > > > - ttm_mem_io_free(bdev, mem);
> > > > -}
> > > > -
> > > > -static int ttm_copy_io_page(void *dst, void *src, unsigned
> > > > long page)
> > > > -{
> > > > - uint32_t *dstP =
> > > > - (uint32_t *) ((unsigned long)dst + (page <<
> > > > PAGE_SHIFT));
> > > > - uint32_t *srcP =
> > > > - (uint32_t *) ((unsigned long)src + (page <<
> > > > PAGE_SHIFT));
> > > > -
> > > > - int i;
> > > > - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
> > > > - iowrite32(ioread32(srcP++), dstP++);
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
> > > > - unsigned long page,
> > > > - pgprot_t prot)
> > > > -{
> > > > - struct page *d = ttm->pages[page];
> > > > - void *dst;
> > > > -
> > > > - if (!d)
> > > > - return -ENOMEM;
> > > > -
> > > > - src = (void *)((unsigned long)src + (page <<
> > > > PAGE_SHIFT));
> > > > - dst = kmap_atomic_prot(d, prot);
> > > > - if (!dst)
> > > > - return -ENOMEM;
> > > > -
> > > > - memcpy_fromio(dst, src, PAGE_SIZE);
> > > > -
> > > > - kunmap_atomic(dst);
> > > > -
> > > > - return 0;
> > > > -}
> > > > -
> > > > -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
> > > > - unsigned long page,
> > > > - pgprot_t prot)
> > > > -{
> > > > - struct page *s = ttm->pages[page];
> > > > - void *src;
> > > > -
> > > > - if (!s)
> > > > - return -ENOMEM;
> > > > -
> > > > - dst = (void *)((unsigned long)dst + (page <<
> > > > PAGE_SHIFT));
> > > > - src = kmap_atomic_prot(s, prot);
> > > > - if (!src)
> > > > - return -ENOMEM;
> > > >
> > > > - memcpy_toio(dst, src, PAGE_SIZE);
> > > > -
> > > > - kunmap_atomic(src);
> > > > + for (i = 0; i < dst_mem->num_pages; ++i) {
> > > > + dst_ops->map_local(dst_iter, &dst_map, i);
> > > > + src_ops->map_local(src_iter, &src_map, i);
> > > > +
> > > > + if (!src_map.is_iomem && !dst_map.is_iomem) {
> > > > + memcpy(dst_map.vaddr, src_map.vaddr,
> > > > PAGE_SIZE);
> > > > + } else if (!src_map.is_iomem) {
> > > > + dma_buf_map_memcpy_to(&dst_map,
> > > > src_map.vaddr,
> > > > + PAGE_SIZE);
> > > > + } else if (!dst_map.is_iomem) {
> > > > + memcpy_fromio(dst_map.vaddr,
> > > > src_map.vaddr_iomem,
> > > > + PAGE_SIZE);
> > > > + } else {
> > > > + int j;
> > > > + u32 __iomem *src = src_map.vaddr_iomem;
> > > > + u32 __iomem *dst = dst_map.vaddr_iomem;
> > > >
> > > > - return 0;
> > > > + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
> > > IMO PAGE_SIZE / sizeof(u32) is easier to understand.
> >
> > OK, will fix.
> >
> >
> > >
> > > > + iowrite32(ioread32(src++),
> > > > dst++);
> > > > + }
> > > > + if (src_ops->unmap_local)
> > > > + src_ops->unmap_local(src_iter,
> > > > &src_map);
> > > > + if (dst_ops->unmap_local)
> > > > + dst_ops->unmap_local(dst_iter,
> > > > &dst_map);
> > > > + }
> > > > }
> > > > +EXPORT_SYMBOL(ttm_move_memcpy);
> > > >
> > > > int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
> > > > struct ttm_operation_ctx *ctx,
> > > > - struct ttm_resource *new_mem)
> > > > + struct ttm_resource *dst_mem)
> > > > {
> > > > struct ttm_device *bdev = bo->bdev;
> > > > - struct ttm_resource_manager *man =
> > > > ttm_manager_type(bdev, new_mem->mem_type);
> > > > + struct ttm_resource_manager *dst_man =
> > > > + ttm_manager_type(bo->bdev, dst_mem->mem_type);
> > > > struct ttm_tt *ttm = bo->ttm;
> > > > - struct ttm_resource *old_mem = &bo->mem;
> > > > - struct ttm_resource old_copy = *old_mem;
> > > > - void *old_iomap;
> > > > - void *new_iomap;
> > > > + struct ttm_resource *src_mem = &bo->mem;
> > > > + struct ttm_resource_manager *src_man =
> > > > + ttm_manager_type(bdev, src_mem->mem_type);
> > > > + struct ttm_resource src_copy = *src_mem;
> > > > + union {
> > > > + struct ttm_kmap_iter_tt tt;
> > > > + struct ttm_kmap_iter_linear_io io;
> > > > + } _dst_iter, _src_iter;
> > > > + struct ttm_kmap_iter *dst_iter, *src_iter;
> > > > int ret;
> > > > - unsigned long i;
> > > >
> > > > - ret = ttm_bo_wait_ctx(bo, ctx);
> > > > - if (ret)
> > > > - return ret;
> > > > -
> > > > - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
> > > > - if (ret)
> > > > - return ret;
> > > > - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
> > > > - if (ret)
> > > > - goto out;
> > > > -
> > > > - /*
> > > > - * Single TTM move. NOP.
> > > > - */
> > > > - if (old_iomap == NULL && new_iomap == NULL)
> > > > - goto out2;
> > > > -
> > > > - /*
> > > > - * Don't move nonexistent data. Clear destination
> > > > instead.
> > > > - */
> > > > - if (old_iomap == NULL &&
> > > > - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
> > > > - !(ttm->page_flags &
> > > > TTM_PAGE_FLAG_SWAPPED)))) {
> > > > - memset_io(new_iomap, 0, new_mem-
> > > > >num_pages*PAGE_SIZE);
> > > > - goto out2;
> > > > - }
> > > > -
> > > > - /*
> > > > - * TTM might be null for moves within the same region.
> > > > - */
> > > > - if (ttm) {
> > > > + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
> > > > ||
> > > > + dst_man->use_tt)) {
> > > > ret = ttm_tt_populate(bdev, ttm, ctx);
> > > > if (ret)
> > > > - goto out1;
> > > > + return ret;
> > > > }
> > > >
> > > > - for (i = 0; i < new_mem->num_pages; ++i) {
> > > > - if (old_iomap == NULL) {
> > > > - pgprot_t prot = ttm_io_prot(bo,
> > > > old_mem, PAGE_KERNEL);
> > > > - ret = ttm_copy_ttm_io_page(ttm,
> > > > new_iomap, i,
> > > > - prot);
> > > > - } else if (new_iomap == NULL) {
> > > > - pgprot_t prot = ttm_io_prot(bo,
> > > > new_mem, PAGE_KERNEL);
> > > > - ret = ttm_copy_io_ttm_page(ttm,
> > > > old_iomap, i,
> > > > - prot);
> > > > - } else {
> > > > - ret = ttm_copy_io_page(new_iomap,
> > > > old_iomap, i);
> > > > - }
> > > > - if (ret)
> > > > - goto out1;
> > > > + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
> > > > bdev, dst_mem);
> > > > + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
> > > > + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
> > > > bo->ttm);
> > > > + if (IS_ERR(dst_iter))
> > > > + return PTR_ERR(dst_iter);
> > > > +
> > > > + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
> > > > bdev, src_mem);
> > > > + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
> > > > + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
> > > > bo->ttm);
> > > > + if (IS_ERR(src_iter)) {
> > > > + ret = PTR_ERR(src_iter);
> > > > + goto out_src_iter;
> > > > }
> > > > - mb();
> > > > -out2:
> > > > - old_copy = *old_mem;
> > > >
> > > > - ttm_bo_assign_mem(bo, new_mem);
> > > > -
> > > > - if (!man->use_tt)
> > > > - ttm_bo_tt_destroy(bo);
> > > > + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
> > > > + src_copy = *src_mem;
> > > > + ttm_bo_move_sync_cleanup(bo, dst_mem);
> > > >
> > > > -out1:
> > > > - ttm_resource_iounmap(bdev, old_mem, new_iomap);
> > > > -out:
> > > > - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
> > > > + if (!src_iter->ops->maps_tt)
> > > > + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
> > > > bdev, &src_copy);
> > > > +out_src_iter:
> > > > + if (!dst_iter->ops->maps_tt)
> > > > + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
> > > > bdev, dst_mem);
> > > >
> > > > - /*
> > > > - * On error, keep the mm node!
> > > > - */
> > > > - if (!ret)
> > > > - ttm_resource_free(bo, &old_copy);
> > > > return ret;
> > > > }
> > > > EXPORT_SYMBOL(ttm_bo_move_memcpy);
> > > > @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
> > > > ttm_buffer_object *bo, struct ttm_resource *res,
> > > > man = ttm_manager_type(bo->bdev, res->mem_type);
> > > > caching = man->use_tt ? bo->ttm->caching : res-
> > > > >bus.caching;
> > > >
> > > > - /* Cached mappings need no adjustment */
> > > > - if (caching == ttm_cached)
> > > > - return tmp;
> > > > -
> > > > -#if defined(__i386__) || defined(__x86_64__)
> > > > - if (caching == ttm_write_combined)
> > > > - tmp = pgprot_writecombine(tmp);
> > > > - else if (boot_cpu_data.x86 > 3)
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > -#if defined(__ia64__) || defined(__arm__) ||
> > > > defined(__aarch64__) || \
> > > > - defined(__powerpc__) || defined(__mips__)
> > > > - if (caching == ttm_write_combined)
> > > > - tmp = pgprot_writecombine(tmp);
> > > > - else
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > -#if defined(__sparc__)
> > > > - tmp = pgprot_noncached(tmp);
> > > > -#endif
> > > > - return tmp;
> > > > + return ttm_prot_from_caching(caching, tmp);
> > > > }
> > > > EXPORT_SYMBOL(ttm_io_prot);
> > > >
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_module.c
> > > > b/drivers/gpu/drm/ttm/ttm_module.c
> > > > index 56b0efdba1a9..997c458f68a9 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_module.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_module.c
> > > > @@ -31,12 +31,47 @@
> > > > */
> > > > #include <linux/module.h>
> > > > #include <linux/device.h>
> > > > +#include <linux/pgtable.h>
> > > > #include <linux/sched.h>
> > > > #include <linux/debugfs.h>
> > > > #include <drm/drm_sysfs.h>
> > > > +#include <drm/ttm/ttm_caching.h>
> > > >
> > > > #include "ttm_module.h"
> > > >
> > > > +/**
> > > > + * ttm_prot_from_caching - Modify the page protection
> > > > according to the
> > > > + * ttm cacing mode
> > > > + * @caching: The ttm caching mode
> > > > + * @tmp: The original page protection
> > > > + *
> > > > + * Return: The modified page protection
> > > > + */
> > > > +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
> > > > pgprot_t tmp)
> > > > +{
> > > > + /* Cached mappings need no adjustment */
> > > > + if (caching == ttm_cached)
> > > > + return tmp;
> > > > +
> > > > +#if defined(__i386__) || defined(__x86_64__)
> > > > + if (caching == ttm_write_combined)
> > > > + tmp = pgprot_writecombine(tmp);
> > > > + else if (boot_cpu_data.x86 > 3)
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > +#if defined(__ia64__) || defined(__arm__) ||
> > > > defined(__aarch64__) || \
> > > > + defined(__powerpc__) || defined(__mips__)
> > > > + if (caching == ttm_write_combined)
> > > > + tmp = pgprot_writecombine(tmp);
> > > > + else
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > +#if defined(__sparc__)
> > > > + tmp = pgprot_noncached(tmp);
> > > > +#endif
> > > > + return tmp;
> > > > +}
> > > > +
> > > > struct dentry *ttm_debugfs_root;
> > > >
> > > > static int __init ttm_init(void)
> > > > diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
> > > > b/drivers/gpu/drm/ttm/ttm_resource.c
> > > > index 59e2b7157e41..e05ae7e3d477 100644
> > > > --- a/drivers/gpu/drm/ttm/ttm_resource.c
> > > > +++ b/drivers/gpu/drm/ttm/ttm_resource.c
> > > > @@ -22,6 +22,10 @@
> > > > * Authors: Christian König
> > > > */
> > > >
> > > > +#include <linux/dma-buf-map.h>
> > > > +#include <linux/io-mapping.h>
> > > > +#include <linux/scatterlist.h>
> > > > +
> > > > #include <drm/ttm/ttm_resource.h>
> > > > #include <drm/ttm/ttm_bo_driver.h>
> > > >
> > > > @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
> > > > ttm_resource_manager *man,
> > > > man->func->debug(man, p);
> > > > }
> > > > EXPORT_SYMBOL(ttm_resource_manager_debug);
> > > > +
> > > > +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
> > > > *iter,
> > > > + struct dma_buf_map
> > > > *dmap,
> > > > + pgoff_t i)
> > > > +{
> > > > + struct ttm_kmap_iter_iomap *iter_io =
> > > > + container_of(iter, typeof(*iter_io), base);
> > > > + void __iomem *addr;
> > > > +
> > > > +retry:
> > > > + while (i >= iter_io->cache.end) {
> > > > + iter_io->cache.sg = iter_io->cache.sg ?
> > > > + sg_next(iter_io->cache.sg) : iter_io-
> > > > >st->sgl;
> > > > + iter_io->cache.i = iter_io->cache.end;
> > > > + iter_io->cache.end += sg_dma_len(iter_io-
> > > > >cache.sg) >>
> > > > + PAGE_SHIFT;
> > > > + iter_io->cache.offs = sg_dma_address(iter_io-
> > > > >cache.sg) -
> > > > + iter_io->start;
> > > > + }
> > > > +
> > > > + if (i < iter_io->cache.i) {
> > > > + iter_io->cache.end = 0;
> > > > + iter_io->cache.sg = NULL;
> > > > + goto retry;
> > > > + }
> > > > +
> > > > + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
> > > > >cache.offs +
> > > > + (((resource_size_t)i -
> > > > iter_io->cache.i)
> > > > + << PAGE_SHIFT));
> > > > + dma_buf_map_set_vaddr_iomem(dmap, addr);
> > > > +}
> > > > +
> > > > +static void ttm_kmap_iter_iomap_unmap_local(struct
> > > > ttm_kmap_iter *iter,
> > > > + struct dma_buf_map
> > > > *map)
> > > > +{
> > > > + io_mapping_unmap_local(map->vaddr_iomem);
> > > > +}
> > > > +
> > > > +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
> > > > + .map_local = ttm_kmap_iter_iomap_map_local,
> > > > + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
> > > > + .maps_tt = false,
> > > > +};
> > > > +
> > > > +/**
> > > > + * ttm_kmap_iter_iomap_init - Initialize a struct
> > > > ttm_kmap_iter_iomap
> > > > + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
> > > > + * @iomap: The struct io_mapping representing the underlying
> > > > linear io_memory.
> > > > + * @st: sg_table into @iomap, representing the memory of the
> > > > struct
> > > > + * ttm_resource.
> > > > + * @start: Offset that needs to be subtracted from @st to make
> > > > + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
> > > > + *
> > > > + * Return: Pointer to the embedded struct ttm_kmap_iter.
> > > > + */
> > > > +struct ttm_kmap_iter *
> > > > +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
> > > > + struct io_mapping *iomap,
> > > > + struct sg_table *st,
> > > > + resource_size_t start)
> > > > +{
> > > > + iter_io->base.ops = &ttm_kmap_iter_io_ops;
> > > > + iter_io->iomap = iomap;
> > > > + iter_io->st = st;
> > > > + iter_io->start = start;
> > > > + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
> > > > +
> > > > + return &iter_io->base;
> > > > +}
> > > > +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
> > > > +
> > > > +/**
> > > > + * DOC: Linear io iterator
> > > > + *
> > > > + * This code should die in the not too near future. Best would
> > > > be if we could
> > > > + * make io-mapping use memremap for all io memory, and have
> > > > memremap
> > > > + * implement a kmap_local functionality. We could then strip a
> > > > huge amount of
> > > > + * code. These linear io iterators are implemented to mimic
> > > > old functionality,
> > > > + * and they don't use kmap_local semantics at all internally.
> > > > Rather ioremap or
> > > > + * friends, and at least on 32-bit they add global TLB flushes
> > > > and points
> > > > + * of failure.
> > > > + */
> > > > +
> > > > +static void ttm_kmap_iter_linear_io_map_local(struct
> > > > ttm_kmap_iter *iter,
> > > > + struct
> > > > dma_buf_map *dmap,
> > > > + pgoff_t i)
> > > > +{
> > > > + struct ttm_kmap_iter_linear_io *iter_io =
> > > > + container_of(iter, typeof(*iter_io), base);
> > > > +
> > > > + *dmap = iter_io->dmap;
> > > > + dma_buf_map_incr(dmap, i * PAGE_SIZE);
> > > > +}
> > > > +
> > > > +static const struct ttm_kmap_iter_ops
> > > > ttm_kmap_iter_linear_io_ops = {
> > > > + .map_local = ttm_kmap_iter_linear_io_map_local,
> > > > + .maps_tt = false,
> > > > +};
> > > > +
> > > > +struct ttm_kmap_iter *
> > > > +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
> > > > *iter_io,
> > > > + struct ttm_device *bdev,
> > > > + struct ttm_resource *mem)
> > > > +{
> > > > + int ret;
> > > > +
> > > > + ret = ttm_mem_io_reserve(bdev, mem);
> > > > + if (ret)
> > > > + goto out_err;
> > > > + if (!mem->bus.is_iomem) {
> > > > + ret = -EINVAL;
> > > > + goto out_io_free;
> > > > + }
> > > > +
> > > > + if (mem->bus.addr) {
> > > > + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
> > > > >bus.addr);
> > > > + iter_io->needs_unmap = false;
> > > > + } else {
> > > > + size_t bus_size = (size_t)mem->num_pages <<
> > > > PAGE_SHIFT;
> > > > +
> > > > + iter_io->needs_unmap = true;
> > > > + if (mem->bus.caching == ttm_write_combined)
> > > > + dma_buf_map_set_vaddr_iomem(&iter_io-
> > > > >dmap,
> > > > +
> > > > ioremap_wc(mem->bus.offset,
> > > > +
> > > > bus_size));
> > > > + else if (mem->bus.caching == ttm_cached)
> > > > + dma_buf_map_set_vaddr(&iter_io->dmap,
> > > > + memremap(mem-
> > > > >bus.offset, bus_size,
> > > > +
> > > > MEMREMAP_WB));
> > > The comments in set_vaddr suggest that this is meant for
> > > system-memory. Does that actually matter or is it just about not
> > > losing the __iomem annotation on platforms where it matters?
> >
> > Yes, it's the latter. dma_buf_map() is relatively new and the
> > author
> > probably didn't think about the case of cached iomem, which is used
> > by,
> > for example, vmwgfx.
> >
> > > Apparently cached device local is a thing. Also should this not
> > > be
> > > wrapped in CONFIG_X86?
> >
> > Both dma_buf_map() and memremap are generic, I think, I guess
> > memremap
> > would return NULL if it's not supported.
>
> It looks like memremap just wraps ioremap_cache, but since it also
> discards the __iomem annotation should we be doing that universally?
> Also not sure if ioremap_cache is universally supported, so wrapping
> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
> at least that looks like roughly what the previous code was doing?
> Not
> too sure tbh.
>
I think the long term goal is to use memremap all over the place, to
just not have to bother with the __iomem annotation. But to do that io-
mapping.h needs to support memremap. But for now we need to be strict
about __iomem unless we're in arch specific code. That's why that
dma_buf_map thing was created, but TTM memcpy was never fully adapted.
As for limited arch support for memremap cached, It looks like we only
need to or in "backup" mapping modes in the memremap flags, and we'd
mimic the previous behaviour.
/Thomas
> >
> > /Thomas
> >
> >
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-25 10:07 ` Thomas Hellström
@ 2021-05-25 15:48 ` Christian König
-1 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-25 15:48 UTC (permalink / raw)
To: Thomas Hellström, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
Am 25.05.21 um 12:07 schrieb Thomas Hellström:
> On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
>> On Tue, 25 May 2021 at 10:32, Thomas Hellström
>> <thomas.hellstrom@linux.intel.com> wrote:
>>>
>>> On 5/25/21 11:18 AM, Matthew Auld wrote:
>>>> On Fri, 21 May 2021 at 16:33, Thomas Hellström
>>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>> The internal ttm_bo_util memcpy uses ioremap functionality, and
>>>>> while it
>>>>> probably might be possible to use it for copying in- and out of
>>>>> sglist represented io memory, using io_mem_reserve() /
>>>>> io_mem_free()
>>>>> callbacks, that would cause problems with fault().
>>>>> Instead, implement a method mapping page-by-page using
>>>>> kmap_local()
>>>>> semantics. As an additional benefit we then avoid the
>>>>> occasional global
>>>>> TLB flushes of ioremap() and consuming ioremap space,
>>>>> elimination of a
>>>>> critical point of failure and with a slight change of semantics
>>>>> we could
>>>>> also push the memcpy out async for testing and async driver
>>>>> development
>>>>> purposes.
>>>>>
>>>>> A special linear iomem iterator is introduced internally to
>>>>> mimic the
>>>>> old ioremap behaviour for code-paths that can't immediately be
>>>>> ported
>>>>> over. This adds to the code size and should be considered a
>>>>> temporary
>>>>> solution.
>>>>>
>>>>> Looking at the code we have a lot of checks for iomap tagged
>>>>> pointers.
>>>>> Ideally we should extend the core memremap functions to also
>>>>> accept
>>>>> uncached memory and kmap_local functionality. Then we could
>>>>> strip a
>>>>> lot of code.
>>>>>
>>>>> Cc: Christian König <christian.koenig@amd.com>
>>>>> Signed-off-by: Thomas Hellström <
>>>>> thomas.hellstrom@linux.intel.com>
>>>>> ---
>>>>> v3:
>>>>> - Split up in various TTM files and addressed review comments
>>>>> by
>>>>> Christian König. Tested and fixed legacy iomap memcpy path
>>>>> on i915.
>>>>> ---
>>>>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
>>>>> ---------
>>>>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>>>>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>>>>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>>>>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>>>>> include/drm/ttm/ttm_caching.h | 2 +
>>>>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>>>>> include/drm/ttm/ttm_resource.h | 61 +++++++
>>>>> include/drm/ttm/ttm_tt.h | 16 ++
>>>>> 9 files changed, 508 insertions(+), 181 deletions(-)
>>>>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>>>>
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> index ae8b61460724..912cbe8e60a2 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
>>>>> *bdev,
>>>>> mem->bus.addr = NULL;
>>>>> }
>>>>>
>>>>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>>>>> - struct ttm_resource *mem,
>>>>> - void **virtual)
>>>>> +/**
>>>>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move
>>>>> operation.
>>>>> + * @bo: The struct ttm_buffer_object.
>>>>> + * @new_mem: The struct ttm_resource we're moving to (copy
>>>>> destination).
>>>>> + * @new_iter: A struct ttm_kmap_iter representing the
>>>>> destination resource.
>>>>> + * @src_iter: A struct ttm_kmap_iter representing the source
>>>>> resource.
>>>>> + *
>>>>> + * This function is intended to be able to move out async
>>>>> under a
>>>>> + * dma-fence if desired.
>>>>> + */
>>>>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>>>>> + struct ttm_resource *dst_mem,
>>>>> + struct ttm_kmap_iter *dst_iter,
>>>>> + struct ttm_kmap_iter *src_iter)
>>>>> {
>>>>> - int ret;
>>>>> - void *addr;
>>>>> -
>>>>> - *virtual = NULL;
>>>>> - ret = ttm_mem_io_reserve(bdev, mem);
>>>>> - if (ret || !mem->bus.is_iomem)
>>>>> - return ret;
>>>>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
>>>>>> ops;
>>>>> + const struct ttm_kmap_iter_ops *src_ops = src_iter-
>>>>>> ops;
>>>>> + struct ttm_tt *ttm = bo->ttm;
>>>>> + struct dma_buf_map src_map, dst_map;
>>>>> + pgoff_t i;
>>>>>
>>>>> - if (mem->bus.addr) {
>>>>> - addr = mem->bus.addr;
>>>>> - } else {
>>>>> - size_t bus_size = (size_t)mem->num_pages <<
>>>>> PAGE_SHIFT;
>>>>> + /* Single TTM move. NOP */
>>>>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>>>>> + return;
>>>>>
>>>>> - if (mem->bus.caching == ttm_write_combined)
>>>>> - addr = ioremap_wc(mem->bus.offset,
>>>>> bus_size);
>>>>> -#ifdef CONFIG_X86
>>>>> - else if (mem->bus.caching == ttm_cached)
>>>>> - addr = ioremap_cache(mem->bus.offset,
>>>>> bus_size);
>>>>> -#endif
>>>>> - else
>>>>> - addr = ioremap(mem->bus.offset,
>>>>> bus_size);
>>>>> - if (!addr) {
>>>>> - ttm_mem_io_free(bdev, mem);
>>>>> - return -ENOMEM;
>>>>> + /* Don't move nonexistent data. Clear destination
>>>>> instead. */
>>>>> + if (src_ops->maps_tt && (!ttm ||
>>>>> !ttm_tt_is_populated(ttm))) {
>>>>> + if (ttm && !(ttm->page_flags &
>>>>> TTM_PAGE_FLAG_ZERO_ALLOC))
>>>>> + return;
>>>>> +
>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>> + dst_ops->map_local(dst_iter, &dst_map,
>>>>> i);
>>>>> + if (dst_map.is_iomem)
>>>>> + memset_io(dst_map.vaddr_iomem,
>>>>> 0, PAGE_SIZE);
>>>>> + else
>>>>> + memset(dst_map.vaddr, 0,
>>>>> PAGE_SIZE);
>>>>> + if (dst_ops->unmap_local)
>>>>> + dst_ops->unmap_local(dst_iter,
>>>>> &dst_map);
>>>>> }
>>>>> + return;
>>>>> }
>>>>> - *virtual = addr;
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>>>>> - struct ttm_resource *mem,
>>>>> - void *virtual)
>>>>> -{
>>>>> - if (virtual && mem->bus.addr == NULL)
>>>>> - iounmap(virtual);
>>>>> - ttm_mem_io_free(bdev, mem);
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_io_page(void *dst, void *src, unsigned
>>>>> long page)
>>>>> -{
>>>>> - uint32_t *dstP =
>>>>> - (uint32_t *) ((unsigned long)dst + (page <<
>>>>> PAGE_SHIFT));
>>>>> - uint32_t *srcP =
>>>>> - (uint32_t *) ((unsigned long)src + (page <<
>>>>> PAGE_SHIFT));
>>>>> -
>>>>> - int i;
>>>>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>>>>> - iowrite32(ioread32(srcP++), dstP++);
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>>>>> - unsigned long page,
>>>>> - pgprot_t prot)
>>>>> -{
>>>>> - struct page *d = ttm->pages[page];
>>>>> - void *dst;
>>>>> -
>>>>> - if (!d)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - src = (void *)((unsigned long)src + (page <<
>>>>> PAGE_SHIFT));
>>>>> - dst = kmap_atomic_prot(d, prot);
>>>>> - if (!dst)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - memcpy_fromio(dst, src, PAGE_SIZE);
>>>>> -
>>>>> - kunmap_atomic(dst);
>>>>> -
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>>>>> - unsigned long page,
>>>>> - pgprot_t prot)
>>>>> -{
>>>>> - struct page *s = ttm->pages[page];
>>>>> - void *src;
>>>>> -
>>>>> - if (!s)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - dst = (void *)((unsigned long)dst + (page <<
>>>>> PAGE_SHIFT));
>>>>> - src = kmap_atomic_prot(s, prot);
>>>>> - if (!src)
>>>>> - return -ENOMEM;
>>>>>
>>>>> - memcpy_toio(dst, src, PAGE_SIZE);
>>>>> -
>>>>> - kunmap_atomic(src);
>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>> + dst_ops->map_local(dst_iter, &dst_map, i);
>>>>> + src_ops->map_local(src_iter, &src_map, i);
>>>>> +
>>>>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>>>>> + memcpy(dst_map.vaddr, src_map.vaddr,
>>>>> PAGE_SIZE);
>>>>> + } else if (!src_map.is_iomem) {
>>>>> + dma_buf_map_memcpy_to(&dst_map,
>>>>> src_map.vaddr,
>>>>> + PAGE_SIZE);
>>>>> + } else if (!dst_map.is_iomem) {
>>>>> + memcpy_fromio(dst_map.vaddr,
>>>>> src_map.vaddr_iomem,
>>>>> + PAGE_SIZE);
>>>>> + } else {
>>>>> + int j;
>>>>> + u32 __iomem *src = src_map.vaddr_iomem;
>>>>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>>>>
>>>>> - return 0;
>>>>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
>>>> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>>> OK, will fix.
>>>
>>>
>>>>> + iowrite32(ioread32(src++),
>>>>> dst++);
>>>>> + }
>>>>> + if (src_ops->unmap_local)
>>>>> + src_ops->unmap_local(src_iter,
>>>>> &src_map);
>>>>> + if (dst_ops->unmap_local)
>>>>> + dst_ops->unmap_local(dst_iter,
>>>>> &dst_map);
>>>>> + }
>>>>> }
>>>>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>>>>
>>>>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>>>>> struct ttm_operation_ctx *ctx,
>>>>> - struct ttm_resource *new_mem)
>>>>> + struct ttm_resource *dst_mem)
>>>>> {
>>>>> struct ttm_device *bdev = bo->bdev;
>>>>> - struct ttm_resource_manager *man =
>>>>> ttm_manager_type(bdev, new_mem->mem_type);
>>>>> + struct ttm_resource_manager *dst_man =
>>>>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>>>>> struct ttm_tt *ttm = bo->ttm;
>>>>> - struct ttm_resource *old_mem = &bo->mem;
>>>>> - struct ttm_resource old_copy = *old_mem;
>>>>> - void *old_iomap;
>>>>> - void *new_iomap;
>>>>> + struct ttm_resource *src_mem = &bo->mem;
>>>>> + struct ttm_resource_manager *src_man =
>>>>> + ttm_manager_type(bdev, src_mem->mem_type);
>>>>> + struct ttm_resource src_copy = *src_mem;
>>>>> + union {
>>>>> + struct ttm_kmap_iter_tt tt;
>>>>> + struct ttm_kmap_iter_linear_io io;
>>>>> + } _dst_iter, _src_iter;
>>>>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>>>>> int ret;
>>>>> - unsigned long i;
>>>>>
>>>>> - ret = ttm_bo_wait_ctx(bo, ctx);
>>>>> - if (ret)
>>>>> - return ret;
>>>>> -
>>>>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>>>>> - if (ret)
>>>>> - return ret;
>>>>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>>>>> - if (ret)
>>>>> - goto out;
>>>>> -
>>>>> - /*
>>>>> - * Single TTM move. NOP.
>>>>> - */
>>>>> - if (old_iomap == NULL && new_iomap == NULL)
>>>>> - goto out2;
>>>>> -
>>>>> - /*
>>>>> - * Don't move nonexistent data. Clear destination
>>>>> instead.
>>>>> - */
>>>>> - if (old_iomap == NULL &&
>>>>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>>>>> - !(ttm->page_flags &
>>>>> TTM_PAGE_FLAG_SWAPPED)))) {
>>>>> - memset_io(new_iomap, 0, new_mem-
>>>>>> num_pages*PAGE_SIZE);
>>>>> - goto out2;
>>>>> - }
>>>>> -
>>>>> - /*
>>>>> - * TTM might be null for moves within the same region.
>>>>> - */
>>>>> - if (ttm) {
>>>>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
>>>>> ||
>>>>> + dst_man->use_tt)) {
>>>>> ret = ttm_tt_populate(bdev, ttm, ctx);
>>>>> if (ret)
>>>>> - goto out1;
>>>>> + return ret;
>>>>> }
>>>>>
>>>>> - for (i = 0; i < new_mem->num_pages; ++i) {
>>>>> - if (old_iomap == NULL) {
>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>> old_mem, PAGE_KERNEL);
>>>>> - ret = ttm_copy_ttm_io_page(ttm,
>>>>> new_iomap, i,
>>>>> - prot);
>>>>> - } else if (new_iomap == NULL) {
>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>> new_mem, PAGE_KERNEL);
>>>>> - ret = ttm_copy_io_ttm_page(ttm,
>>>>> old_iomap, i,
>>>>> - prot);
>>>>> - } else {
>>>>> - ret = ttm_copy_io_page(new_iomap,
>>>>> old_iomap, i);
>>>>> - }
>>>>> - if (ret)
>>>>> - goto out1;
>>>>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
>>>>> bdev, dst_mem);
>>>>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>>>>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
>>>>> bo->ttm);
>>>>> + if (IS_ERR(dst_iter))
>>>>> + return PTR_ERR(dst_iter);
>>>>> +
>>>>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
>>>>> bdev, src_mem);
>>>>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>>>>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
>>>>> bo->ttm);
>>>>> + if (IS_ERR(src_iter)) {
>>>>> + ret = PTR_ERR(src_iter);
>>>>> + goto out_src_iter;
>>>>> }
>>>>> - mb();
>>>>> -out2:
>>>>> - old_copy = *old_mem;
>>>>>
>>>>> - ttm_bo_assign_mem(bo, new_mem);
>>>>> -
>>>>> - if (!man->use_tt)
>>>>> - ttm_bo_tt_destroy(bo);
>>>>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>>>>> + src_copy = *src_mem;
>>>>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>>>>
>>>>> -out1:
>>>>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>>>>> -out:
>>>>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>>>>> + if (!src_iter->ops->maps_tt)
>>>>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
>>>>> bdev, &src_copy);
>>>>> +out_src_iter:
>>>>> + if (!dst_iter->ops->maps_tt)
>>>>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
>>>>> bdev, dst_mem);
>>>>>
>>>>> - /*
>>>>> - * On error, keep the mm node!
>>>>> - */
>>>>> - if (!ret)
>>>>> - ttm_resource_free(bo, &old_copy);
>>>>> return ret;
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>>>>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
>>>>> ttm_buffer_object *bo, struct ttm_resource *res,
>>>>> man = ttm_manager_type(bo->bdev, res->mem_type);
>>>>> caching = man->use_tt ? bo->ttm->caching : res-
>>>>>> bus.caching;
>>>>> - /* Cached mappings need no adjustment */
>>>>> - if (caching == ttm_cached)
>>>>> - return tmp;
>>>>> -
>>>>> -#if defined(__i386__) || defined(__x86_64__)
>>>>> - if (caching == ttm_write_combined)
>>>>> - tmp = pgprot_writecombine(tmp);
>>>>> - else if (boot_cpu_data.x86 > 3)
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> -#if defined(__ia64__) || defined(__arm__) ||
>>>>> defined(__aarch64__) || \
>>>>> - defined(__powerpc__) || defined(__mips__)
>>>>> - if (caching == ttm_write_combined)
>>>>> - tmp = pgprot_writecombine(tmp);
>>>>> - else
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> -#if defined(__sparc__)
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> - return tmp;
>>>>> + return ttm_prot_from_caching(caching, tmp);
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_io_prot);
>>>>>
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c
>>>>> b/drivers/gpu/drm/ttm/ttm_module.c
>>>>> index 56b0efdba1a9..997c458f68a9 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>>>>> @@ -31,12 +31,47 @@
>>>>> */
>>>>> #include <linux/module.h>
>>>>> #include <linux/device.h>
>>>>> +#include <linux/pgtable.h>
>>>>> #include <linux/sched.h>
>>>>> #include <linux/debugfs.h>
>>>>> #include <drm/drm_sysfs.h>
>>>>> +#include <drm/ttm/ttm_caching.h>
>>>>>
>>>>> #include "ttm_module.h"
>>>>>
>>>>> +/**
>>>>> + * ttm_prot_from_caching - Modify the page protection
>>>>> according to the
>>>>> + * ttm cacing mode
>>>>> + * @caching: The ttm caching mode
>>>>> + * @tmp: The original page protection
>>>>> + *
>>>>> + * Return: The modified page protection
>>>>> + */
>>>>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
>>>>> pgprot_t tmp)
>>>>> +{
>>>>> + /* Cached mappings need no adjustment */
>>>>> + if (caching == ttm_cached)
>>>>> + return tmp;
>>>>> +
>>>>> +#if defined(__i386__) || defined(__x86_64__)
>>>>> + if (caching == ttm_write_combined)
>>>>> + tmp = pgprot_writecombine(tmp);
>>>>> + else if (boot_cpu_data.x86 > 3)
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> +#if defined(__ia64__) || defined(__arm__) ||
>>>>> defined(__aarch64__) || \
>>>>> + defined(__powerpc__) || defined(__mips__)
>>>>> + if (caching == ttm_write_combined)
>>>>> + tmp = pgprot_writecombine(tmp);
>>>>> + else
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> +#if defined(__sparc__)
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> + return tmp;
>>>>> +}
>>>>> +
>>>>> struct dentry *ttm_debugfs_root;
>>>>>
>>>>> static int __init ttm_init(void)
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> index 59e2b7157e41..e05ae7e3d477 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> @@ -22,6 +22,10 @@
>>>>> * Authors: Christian König
>>>>> */
>>>>>
>>>>> +#include <linux/dma-buf-map.h>
>>>>> +#include <linux/io-mapping.h>
>>>>> +#include <linux/scatterlist.h>
>>>>> +
>>>>> #include <drm/ttm/ttm_resource.h>
>>>>> #include <drm/ttm/ttm_bo_driver.h>
>>>>>
>>>>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
>>>>> ttm_resource_manager *man,
>>>>> man->func->debug(man, p);
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>>>>> +
>>>>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
>>>>> *iter,
>>>>> + struct dma_buf_map
>>>>> *dmap,
>>>>> + pgoff_t i)
>>>>> +{
>>>>> + struct ttm_kmap_iter_iomap *iter_io =
>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>> + void __iomem *addr;
>>>>> +
>>>>> +retry:
>>>>> + while (i >= iter_io->cache.end) {
>>>>> + iter_io->cache.sg = iter_io->cache.sg ?
>>>>> + sg_next(iter_io->cache.sg) : iter_io-
>>>>>> st->sgl;
>>>>> + iter_io->cache.i = iter_io->cache.end;
>>>>> + iter_io->cache.end += sg_dma_len(iter_io-
>>>>>> cache.sg) >>
>>>>> + PAGE_SHIFT;
>>>>> + iter_io->cache.offs = sg_dma_address(iter_io-
>>>>>> cache.sg) -
>>>>> + iter_io->start;
>>>>> + }
>>>>> +
>>>>> + if (i < iter_io->cache.i) {
>>>>> + iter_io->cache.end = 0;
>>>>> + iter_io->cache.sg = NULL;
>>>>> + goto retry;
>>>>> + }
>>>>> +
>>>>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
>>>>>> cache.offs +
>>>>> + (((resource_size_t)i -
>>>>> iter_io->cache.i)
>>>>> + << PAGE_SHIFT));
>>>>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>>>>> +}
>>>>> +
>>>>> +static void ttm_kmap_iter_iomap_unmap_local(struct
>>>>> ttm_kmap_iter *iter,
>>>>> + struct dma_buf_map
>>>>> *map)
>>>>> +{
>>>>> + io_mapping_unmap_local(map->vaddr_iomem);
>>>>> +}
>>>>> +
>>>>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>>>>> + .map_local = ttm_kmap_iter_iomap_map_local,
>>>>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>>>>> + .maps_tt = false,
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * ttm_kmap_iter_iomap_init - Initialize a struct
>>>>> ttm_kmap_iter_iomap
>>>>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>>>>> + * @iomap: The struct io_mapping representing the underlying
>>>>> linear io_memory.
>>>>> + * @st: sg_table into @iomap, representing the memory of the
>>>>> struct
>>>>> + * ttm_resource.
>>>>> + * @start: Offset that needs to be subtracted from @st to make
>>>>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>>>>> + *
>>>>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>>>>> + */
>>>>> +struct ttm_kmap_iter *
>>>>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>>>>> + struct io_mapping *iomap,
>>>>> + struct sg_table *st,
>>>>> + resource_size_t start)
>>>>> +{
>>>>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>>>>> + iter_io->iomap = iomap;
>>>>> + iter_io->st = st;
>>>>> + iter_io->start = start;
>>>>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>>>>> +
>>>>> + return &iter_io->base;
>>>>> +}
>>>>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>>>>> +
>>>>> +/**
>>>>> + * DOC: Linear io iterator
>>>>> + *
>>>>> + * This code should die in the not too near future. Best would
>>>>> be if we could
>>>>> + * make io-mapping use memremap for all io memory, and have
>>>>> memremap
>>>>> + * implement a kmap_local functionality. We could then strip a
>>>>> huge amount of
>>>>> + * code. These linear io iterators are implemented to mimic
>>>>> old functionality,
>>>>> + * and they don't use kmap_local semantics at all internally.
>>>>> Rather ioremap or
>>>>> + * friends, and at least on 32-bit they add global TLB flushes
>>>>> and points
>>>>> + * of failure.
>>>>> + */
>>>>> +
>>>>> +static void ttm_kmap_iter_linear_io_map_local(struct
>>>>> ttm_kmap_iter *iter,
>>>>> + struct
>>>>> dma_buf_map *dmap,
>>>>> + pgoff_t i)
>>>>> +{
>>>>> + struct ttm_kmap_iter_linear_io *iter_io =
>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>> +
>>>>> + *dmap = iter_io->dmap;
>>>>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>>>>> +}
>>>>> +
>>>>> +static const struct ttm_kmap_iter_ops
>>>>> ttm_kmap_iter_linear_io_ops = {
>>>>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>>>>> + .maps_tt = false,
>>>>> +};
>>>>> +
>>>>> +struct ttm_kmap_iter *
>>>>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
>>>>> *iter_io,
>>>>> + struct ttm_device *bdev,
>>>>> + struct ttm_resource *mem)
>>>>> +{
>>>>> + int ret;
>>>>> +
>>>>> + ret = ttm_mem_io_reserve(bdev, mem);
>>>>> + if (ret)
>>>>> + goto out_err;
>>>>> + if (!mem->bus.is_iomem) {
>>>>> + ret = -EINVAL;
>>>>> + goto out_io_free;
>>>>> + }
>>>>> +
>>>>> + if (mem->bus.addr) {
>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
>>>>>> bus.addr);
>>>>> + iter_io->needs_unmap = false;
>>>>> + } else {
>>>>> + size_t bus_size = (size_t)mem->num_pages <<
>>>>> PAGE_SHIFT;
>>>>> +
>>>>> + iter_io->needs_unmap = true;
>>>>> + if (mem->bus.caching == ttm_write_combined)
>>>>> + dma_buf_map_set_vaddr_iomem(&iter_io-
>>>>>> dmap,
>>>>> +
>>>>> ioremap_wc(mem->bus.offset,
>>>>> +
>>>>> bus_size));
>>>>> + else if (mem->bus.caching == ttm_cached)
>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>>>>> + memremap(mem-
>>>>>> bus.offset, bus_size,
>>>>> +
>>>>> MEMREMAP_WB));
>>>> The comments in set_vaddr suggest that this is meant for
>>>> system-memory. Does that actually matter or is it just about not
>>>> losing the __iomem annotation on platforms where it matters?
>>> Yes, it's the latter. dma_buf_map() is relatively new and the
>>> author
>>> probably didn't think about the case of cached iomem, which is used
>>> by,
>>> for example, vmwgfx.
>>>
>>>> Apparently cached device local is a thing. Also should this not
>>>> be
>>>> wrapped in CONFIG_X86?
>>> Both dma_buf_map() and memremap are generic, I think, I guess
>>> memremap
>>> would return NULL if it's not supported.
>> It looks like memremap just wraps ioremap_cache, but since it also
>> discards the __iomem annotation should we be doing that universally?
>> Also not sure if ioremap_cache is universally supported, so wrapping
>> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
>> at least that looks like roughly what the previous code was doing?
>> Not
>> too sure tbh.
>>
> I think the long term goal is to use memremap all over the place, to
> just not have to bother with the __iomem annotation. But to do that io-
> mapping.h needs to support memremap. But for now we need to be strict
> about __iomem unless we're in arch specific code. That's why that
> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
I don't think that this will work. __iomem annotation is there because
we have architectures where you need to use special CPU instructions for
iomem access.
That won't go away just because we use memremap().
Christian.
>
> As for limited arch support for memremap cached, It looks like we only
> need to or in "backup" mapping modes in the memremap flags, and we'd
> mimic the previous behaviour.
>
> /Thomas
>
>
>>> /Thomas
>>>
>>>
>
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-25 15:48 ` Christian König
0 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-25 15:48 UTC (permalink / raw)
To: Thomas Hellström, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
Am 25.05.21 um 12:07 schrieb Thomas Hellström:
> On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
>> On Tue, 25 May 2021 at 10:32, Thomas Hellström
>> <thomas.hellstrom@linux.intel.com> wrote:
>>>
>>> On 5/25/21 11:18 AM, Matthew Auld wrote:
>>>> On Fri, 21 May 2021 at 16:33, Thomas Hellström
>>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>> The internal ttm_bo_util memcpy uses ioremap functionality, and
>>>>> while it
>>>>> probably might be possible to use it for copying in- and out of
>>>>> sglist represented io memory, using io_mem_reserve() /
>>>>> io_mem_free()
>>>>> callbacks, that would cause problems with fault().
>>>>> Instead, implement a method mapping page-by-page using
>>>>> kmap_local()
>>>>> semantics. As an additional benefit we then avoid the
>>>>> occasional global
>>>>> TLB flushes of ioremap() and consuming ioremap space,
>>>>> elimination of a
>>>>> critical point of failure and with a slight change of semantics
>>>>> we could
>>>>> also push the memcpy out async for testing and async driver
>>>>> development
>>>>> purposes.
>>>>>
>>>>> A special linear iomem iterator is introduced internally to
>>>>> mimic the
>>>>> old ioremap behaviour for code-paths that can't immediately be
>>>>> ported
>>>>> over. This adds to the code size and should be considered a
>>>>> temporary
>>>>> solution.
>>>>>
>>>>> Looking at the code we have a lot of checks for iomap tagged
>>>>> pointers.
>>>>> Ideally we should extend the core memremap functions to also
>>>>> accept
>>>>> uncached memory and kmap_local functionality. Then we could
>>>>> strip a
>>>>> lot of code.
>>>>>
>>>>> Cc: Christian König <christian.koenig@amd.com>
>>>>> Signed-off-by: Thomas Hellström <
>>>>> thomas.hellstrom@linux.intel.com>
>>>>> ---
>>>>> v3:
>>>>> - Split up in various TTM files and addressed review comments
>>>>> by
>>>>> Christian König. Tested and fixed legacy iomap memcpy path
>>>>> on i915.
>>>>> ---
>>>>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
>>>>> ---------
>>>>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>>>>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>>>>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>>>>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>>>>> include/drm/ttm/ttm_caching.h | 2 +
>>>>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>>>>> include/drm/ttm/ttm_resource.h | 61 +++++++
>>>>> include/drm/ttm/ttm_tt.h | 16 ++
>>>>> 9 files changed, 508 insertions(+), 181 deletions(-)
>>>>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>>>>
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> index ae8b61460724..912cbe8e60a2 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
>>>>> *bdev,
>>>>> mem->bus.addr = NULL;
>>>>> }
>>>>>
>>>>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>>>>> - struct ttm_resource *mem,
>>>>> - void **virtual)
>>>>> +/**
>>>>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move
>>>>> operation.
>>>>> + * @bo: The struct ttm_buffer_object.
>>>>> + * @new_mem: The struct ttm_resource we're moving to (copy
>>>>> destination).
>>>>> + * @new_iter: A struct ttm_kmap_iter representing the
>>>>> destination resource.
>>>>> + * @src_iter: A struct ttm_kmap_iter representing the source
>>>>> resource.
>>>>> + *
>>>>> + * This function is intended to be able to move out async
>>>>> under a
>>>>> + * dma-fence if desired.
>>>>> + */
>>>>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>>>>> + struct ttm_resource *dst_mem,
>>>>> + struct ttm_kmap_iter *dst_iter,
>>>>> + struct ttm_kmap_iter *src_iter)
>>>>> {
>>>>> - int ret;
>>>>> - void *addr;
>>>>> -
>>>>> - *virtual = NULL;
>>>>> - ret = ttm_mem_io_reserve(bdev, mem);
>>>>> - if (ret || !mem->bus.is_iomem)
>>>>> - return ret;
>>>>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
>>>>>> ops;
>>>>> + const struct ttm_kmap_iter_ops *src_ops = src_iter-
>>>>>> ops;
>>>>> + struct ttm_tt *ttm = bo->ttm;
>>>>> + struct dma_buf_map src_map, dst_map;
>>>>> + pgoff_t i;
>>>>>
>>>>> - if (mem->bus.addr) {
>>>>> - addr = mem->bus.addr;
>>>>> - } else {
>>>>> - size_t bus_size = (size_t)mem->num_pages <<
>>>>> PAGE_SHIFT;
>>>>> + /* Single TTM move. NOP */
>>>>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>>>>> + return;
>>>>>
>>>>> - if (mem->bus.caching == ttm_write_combined)
>>>>> - addr = ioremap_wc(mem->bus.offset,
>>>>> bus_size);
>>>>> -#ifdef CONFIG_X86
>>>>> - else if (mem->bus.caching == ttm_cached)
>>>>> - addr = ioremap_cache(mem->bus.offset,
>>>>> bus_size);
>>>>> -#endif
>>>>> - else
>>>>> - addr = ioremap(mem->bus.offset,
>>>>> bus_size);
>>>>> - if (!addr) {
>>>>> - ttm_mem_io_free(bdev, mem);
>>>>> - return -ENOMEM;
>>>>> + /* Don't move nonexistent data. Clear destination
>>>>> instead. */
>>>>> + if (src_ops->maps_tt && (!ttm ||
>>>>> !ttm_tt_is_populated(ttm))) {
>>>>> + if (ttm && !(ttm->page_flags &
>>>>> TTM_PAGE_FLAG_ZERO_ALLOC))
>>>>> + return;
>>>>> +
>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>> + dst_ops->map_local(dst_iter, &dst_map,
>>>>> i);
>>>>> + if (dst_map.is_iomem)
>>>>> + memset_io(dst_map.vaddr_iomem,
>>>>> 0, PAGE_SIZE);
>>>>> + else
>>>>> + memset(dst_map.vaddr, 0,
>>>>> PAGE_SIZE);
>>>>> + if (dst_ops->unmap_local)
>>>>> + dst_ops->unmap_local(dst_iter,
>>>>> &dst_map);
>>>>> }
>>>>> + return;
>>>>> }
>>>>> - *virtual = addr;
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>>>>> - struct ttm_resource *mem,
>>>>> - void *virtual)
>>>>> -{
>>>>> - if (virtual && mem->bus.addr == NULL)
>>>>> - iounmap(virtual);
>>>>> - ttm_mem_io_free(bdev, mem);
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_io_page(void *dst, void *src, unsigned
>>>>> long page)
>>>>> -{
>>>>> - uint32_t *dstP =
>>>>> - (uint32_t *) ((unsigned long)dst + (page <<
>>>>> PAGE_SHIFT));
>>>>> - uint32_t *srcP =
>>>>> - (uint32_t *) ((unsigned long)src + (page <<
>>>>> PAGE_SHIFT));
>>>>> -
>>>>> - int i;
>>>>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>>>>> - iowrite32(ioread32(srcP++), dstP++);
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>>>>> - unsigned long page,
>>>>> - pgprot_t prot)
>>>>> -{
>>>>> - struct page *d = ttm->pages[page];
>>>>> - void *dst;
>>>>> -
>>>>> - if (!d)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - src = (void *)((unsigned long)src + (page <<
>>>>> PAGE_SHIFT));
>>>>> - dst = kmap_atomic_prot(d, prot);
>>>>> - if (!dst)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - memcpy_fromio(dst, src, PAGE_SIZE);
>>>>> -
>>>>> - kunmap_atomic(dst);
>>>>> -
>>>>> - return 0;
>>>>> -}
>>>>> -
>>>>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>>>>> - unsigned long page,
>>>>> - pgprot_t prot)
>>>>> -{
>>>>> - struct page *s = ttm->pages[page];
>>>>> - void *src;
>>>>> -
>>>>> - if (!s)
>>>>> - return -ENOMEM;
>>>>> -
>>>>> - dst = (void *)((unsigned long)dst + (page <<
>>>>> PAGE_SHIFT));
>>>>> - src = kmap_atomic_prot(s, prot);
>>>>> - if (!src)
>>>>> - return -ENOMEM;
>>>>>
>>>>> - memcpy_toio(dst, src, PAGE_SIZE);
>>>>> -
>>>>> - kunmap_atomic(src);
>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>> + dst_ops->map_local(dst_iter, &dst_map, i);
>>>>> + src_ops->map_local(src_iter, &src_map, i);
>>>>> +
>>>>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>>>>> + memcpy(dst_map.vaddr, src_map.vaddr,
>>>>> PAGE_SIZE);
>>>>> + } else if (!src_map.is_iomem) {
>>>>> + dma_buf_map_memcpy_to(&dst_map,
>>>>> src_map.vaddr,
>>>>> + PAGE_SIZE);
>>>>> + } else if (!dst_map.is_iomem) {
>>>>> + memcpy_fromio(dst_map.vaddr,
>>>>> src_map.vaddr_iomem,
>>>>> + PAGE_SIZE);
>>>>> + } else {
>>>>> + int j;
>>>>> + u32 __iomem *src = src_map.vaddr_iomem;
>>>>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>>>>
>>>>> - return 0;
>>>>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
>>>> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>>> OK, will fix.
>>>
>>>
>>>>> + iowrite32(ioread32(src++),
>>>>> dst++);
>>>>> + }
>>>>> + if (src_ops->unmap_local)
>>>>> + src_ops->unmap_local(src_iter,
>>>>> &src_map);
>>>>> + if (dst_ops->unmap_local)
>>>>> + dst_ops->unmap_local(dst_iter,
>>>>> &dst_map);
>>>>> + }
>>>>> }
>>>>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>>>>
>>>>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>>>>> struct ttm_operation_ctx *ctx,
>>>>> - struct ttm_resource *new_mem)
>>>>> + struct ttm_resource *dst_mem)
>>>>> {
>>>>> struct ttm_device *bdev = bo->bdev;
>>>>> - struct ttm_resource_manager *man =
>>>>> ttm_manager_type(bdev, new_mem->mem_type);
>>>>> + struct ttm_resource_manager *dst_man =
>>>>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>>>>> struct ttm_tt *ttm = bo->ttm;
>>>>> - struct ttm_resource *old_mem = &bo->mem;
>>>>> - struct ttm_resource old_copy = *old_mem;
>>>>> - void *old_iomap;
>>>>> - void *new_iomap;
>>>>> + struct ttm_resource *src_mem = &bo->mem;
>>>>> + struct ttm_resource_manager *src_man =
>>>>> + ttm_manager_type(bdev, src_mem->mem_type);
>>>>> + struct ttm_resource src_copy = *src_mem;
>>>>> + union {
>>>>> + struct ttm_kmap_iter_tt tt;
>>>>> + struct ttm_kmap_iter_linear_io io;
>>>>> + } _dst_iter, _src_iter;
>>>>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>>>>> int ret;
>>>>> - unsigned long i;
>>>>>
>>>>> - ret = ttm_bo_wait_ctx(bo, ctx);
>>>>> - if (ret)
>>>>> - return ret;
>>>>> -
>>>>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>>>>> - if (ret)
>>>>> - return ret;
>>>>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>>>>> - if (ret)
>>>>> - goto out;
>>>>> -
>>>>> - /*
>>>>> - * Single TTM move. NOP.
>>>>> - */
>>>>> - if (old_iomap == NULL && new_iomap == NULL)
>>>>> - goto out2;
>>>>> -
>>>>> - /*
>>>>> - * Don't move nonexistent data. Clear destination
>>>>> instead.
>>>>> - */
>>>>> - if (old_iomap == NULL &&
>>>>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>>>>> - !(ttm->page_flags &
>>>>> TTM_PAGE_FLAG_SWAPPED)))) {
>>>>> - memset_io(new_iomap, 0, new_mem-
>>>>>> num_pages*PAGE_SIZE);
>>>>> - goto out2;
>>>>> - }
>>>>> -
>>>>> - /*
>>>>> - * TTM might be null for moves within the same region.
>>>>> - */
>>>>> - if (ttm) {
>>>>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
>>>>> ||
>>>>> + dst_man->use_tt)) {
>>>>> ret = ttm_tt_populate(bdev, ttm, ctx);
>>>>> if (ret)
>>>>> - goto out1;
>>>>> + return ret;
>>>>> }
>>>>>
>>>>> - for (i = 0; i < new_mem->num_pages; ++i) {
>>>>> - if (old_iomap == NULL) {
>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>> old_mem, PAGE_KERNEL);
>>>>> - ret = ttm_copy_ttm_io_page(ttm,
>>>>> new_iomap, i,
>>>>> - prot);
>>>>> - } else if (new_iomap == NULL) {
>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>> new_mem, PAGE_KERNEL);
>>>>> - ret = ttm_copy_io_ttm_page(ttm,
>>>>> old_iomap, i,
>>>>> - prot);
>>>>> - } else {
>>>>> - ret = ttm_copy_io_page(new_iomap,
>>>>> old_iomap, i);
>>>>> - }
>>>>> - if (ret)
>>>>> - goto out1;
>>>>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
>>>>> bdev, dst_mem);
>>>>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>>>>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
>>>>> bo->ttm);
>>>>> + if (IS_ERR(dst_iter))
>>>>> + return PTR_ERR(dst_iter);
>>>>> +
>>>>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
>>>>> bdev, src_mem);
>>>>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>>>>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
>>>>> bo->ttm);
>>>>> + if (IS_ERR(src_iter)) {
>>>>> + ret = PTR_ERR(src_iter);
>>>>> + goto out_src_iter;
>>>>> }
>>>>> - mb();
>>>>> -out2:
>>>>> - old_copy = *old_mem;
>>>>>
>>>>> - ttm_bo_assign_mem(bo, new_mem);
>>>>> -
>>>>> - if (!man->use_tt)
>>>>> - ttm_bo_tt_destroy(bo);
>>>>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>>>>> + src_copy = *src_mem;
>>>>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>>>>
>>>>> -out1:
>>>>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>>>>> -out:
>>>>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>>>>> + if (!src_iter->ops->maps_tt)
>>>>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
>>>>> bdev, &src_copy);
>>>>> +out_src_iter:
>>>>> + if (!dst_iter->ops->maps_tt)
>>>>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
>>>>> bdev, dst_mem);
>>>>>
>>>>> - /*
>>>>> - * On error, keep the mm node!
>>>>> - */
>>>>> - if (!ret)
>>>>> - ttm_resource_free(bo, &old_copy);
>>>>> return ret;
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>>>>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
>>>>> ttm_buffer_object *bo, struct ttm_resource *res,
>>>>> man = ttm_manager_type(bo->bdev, res->mem_type);
>>>>> caching = man->use_tt ? bo->ttm->caching : res-
>>>>>> bus.caching;
>>>>> - /* Cached mappings need no adjustment */
>>>>> - if (caching == ttm_cached)
>>>>> - return tmp;
>>>>> -
>>>>> -#if defined(__i386__) || defined(__x86_64__)
>>>>> - if (caching == ttm_write_combined)
>>>>> - tmp = pgprot_writecombine(tmp);
>>>>> - else if (boot_cpu_data.x86 > 3)
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> -#if defined(__ia64__) || defined(__arm__) ||
>>>>> defined(__aarch64__) || \
>>>>> - defined(__powerpc__) || defined(__mips__)
>>>>> - if (caching == ttm_write_combined)
>>>>> - tmp = pgprot_writecombine(tmp);
>>>>> - else
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> -#if defined(__sparc__)
>>>>> - tmp = pgprot_noncached(tmp);
>>>>> -#endif
>>>>> - return tmp;
>>>>> + return ttm_prot_from_caching(caching, tmp);
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_io_prot);
>>>>>
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c
>>>>> b/drivers/gpu/drm/ttm/ttm_module.c
>>>>> index 56b0efdba1a9..997c458f68a9 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>>>>> @@ -31,12 +31,47 @@
>>>>> */
>>>>> #include <linux/module.h>
>>>>> #include <linux/device.h>
>>>>> +#include <linux/pgtable.h>
>>>>> #include <linux/sched.h>
>>>>> #include <linux/debugfs.h>
>>>>> #include <drm/drm_sysfs.h>
>>>>> +#include <drm/ttm/ttm_caching.h>
>>>>>
>>>>> #include "ttm_module.h"
>>>>>
>>>>> +/**
>>>>> + * ttm_prot_from_caching - Modify the page protection
>>>>> according to the
>>>>> + * ttm cacing mode
>>>>> + * @caching: The ttm caching mode
>>>>> + * @tmp: The original page protection
>>>>> + *
>>>>> + * Return: The modified page protection
>>>>> + */
>>>>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
>>>>> pgprot_t tmp)
>>>>> +{
>>>>> + /* Cached mappings need no adjustment */
>>>>> + if (caching == ttm_cached)
>>>>> + return tmp;
>>>>> +
>>>>> +#if defined(__i386__) || defined(__x86_64__)
>>>>> + if (caching == ttm_write_combined)
>>>>> + tmp = pgprot_writecombine(tmp);
>>>>> + else if (boot_cpu_data.x86 > 3)
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> +#if defined(__ia64__) || defined(__arm__) ||
>>>>> defined(__aarch64__) || \
>>>>> + defined(__powerpc__) || defined(__mips__)
>>>>> + if (caching == ttm_write_combined)
>>>>> + tmp = pgprot_writecombine(tmp);
>>>>> + else
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> +#if defined(__sparc__)
>>>>> + tmp = pgprot_noncached(tmp);
>>>>> +#endif
>>>>> + return tmp;
>>>>> +}
>>>>> +
>>>>> struct dentry *ttm_debugfs_root;
>>>>>
>>>>> static int __init ttm_init(void)
>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> index 59e2b7157e41..e05ae7e3d477 100644
>>>>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>> @@ -22,6 +22,10 @@
>>>>> * Authors: Christian König
>>>>> */
>>>>>
>>>>> +#include <linux/dma-buf-map.h>
>>>>> +#include <linux/io-mapping.h>
>>>>> +#include <linux/scatterlist.h>
>>>>> +
>>>>> #include <drm/ttm/ttm_resource.h>
>>>>> #include <drm/ttm/ttm_bo_driver.h>
>>>>>
>>>>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
>>>>> ttm_resource_manager *man,
>>>>> man->func->debug(man, p);
>>>>> }
>>>>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>>>>> +
>>>>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
>>>>> *iter,
>>>>> + struct dma_buf_map
>>>>> *dmap,
>>>>> + pgoff_t i)
>>>>> +{
>>>>> + struct ttm_kmap_iter_iomap *iter_io =
>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>> + void __iomem *addr;
>>>>> +
>>>>> +retry:
>>>>> + while (i >= iter_io->cache.end) {
>>>>> + iter_io->cache.sg = iter_io->cache.sg ?
>>>>> + sg_next(iter_io->cache.sg) : iter_io-
>>>>>> st->sgl;
>>>>> + iter_io->cache.i = iter_io->cache.end;
>>>>> + iter_io->cache.end += sg_dma_len(iter_io-
>>>>>> cache.sg) >>
>>>>> + PAGE_SHIFT;
>>>>> + iter_io->cache.offs = sg_dma_address(iter_io-
>>>>>> cache.sg) -
>>>>> + iter_io->start;
>>>>> + }
>>>>> +
>>>>> + if (i < iter_io->cache.i) {
>>>>> + iter_io->cache.end = 0;
>>>>> + iter_io->cache.sg = NULL;
>>>>> + goto retry;
>>>>> + }
>>>>> +
>>>>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
>>>>>> cache.offs +
>>>>> + (((resource_size_t)i -
>>>>> iter_io->cache.i)
>>>>> + << PAGE_SHIFT));
>>>>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>>>>> +}
>>>>> +
>>>>> +static void ttm_kmap_iter_iomap_unmap_local(struct
>>>>> ttm_kmap_iter *iter,
>>>>> + struct dma_buf_map
>>>>> *map)
>>>>> +{
>>>>> + io_mapping_unmap_local(map->vaddr_iomem);
>>>>> +}
>>>>> +
>>>>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>>>>> + .map_local = ttm_kmap_iter_iomap_map_local,
>>>>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>>>>> + .maps_tt = false,
>>>>> +};
>>>>> +
>>>>> +/**
>>>>> + * ttm_kmap_iter_iomap_init - Initialize a struct
>>>>> ttm_kmap_iter_iomap
>>>>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>>>>> + * @iomap: The struct io_mapping representing the underlying
>>>>> linear io_memory.
>>>>> + * @st: sg_table into @iomap, representing the memory of the
>>>>> struct
>>>>> + * ttm_resource.
>>>>> + * @start: Offset that needs to be subtracted from @st to make
>>>>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>>>>> + *
>>>>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>>>>> + */
>>>>> +struct ttm_kmap_iter *
>>>>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>>>>> + struct io_mapping *iomap,
>>>>> + struct sg_table *st,
>>>>> + resource_size_t start)
>>>>> +{
>>>>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>>>>> + iter_io->iomap = iomap;
>>>>> + iter_io->st = st;
>>>>> + iter_io->start = start;
>>>>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>>>>> +
>>>>> + return &iter_io->base;
>>>>> +}
>>>>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>>>>> +
>>>>> +/**
>>>>> + * DOC: Linear io iterator
>>>>> + *
>>>>> + * This code should die in the not too near future. Best would
>>>>> be if we could
>>>>> + * make io-mapping use memremap for all io memory, and have
>>>>> memremap
>>>>> + * implement a kmap_local functionality. We could then strip a
>>>>> huge amount of
>>>>> + * code. These linear io iterators are implemented to mimic
>>>>> old functionality,
>>>>> + * and they don't use kmap_local semantics at all internally.
>>>>> Rather ioremap or
>>>>> + * friends, and at least on 32-bit they add global TLB flushes
>>>>> and points
>>>>> + * of failure.
>>>>> + */
>>>>> +
>>>>> +static void ttm_kmap_iter_linear_io_map_local(struct
>>>>> ttm_kmap_iter *iter,
>>>>> + struct
>>>>> dma_buf_map *dmap,
>>>>> + pgoff_t i)
>>>>> +{
>>>>> + struct ttm_kmap_iter_linear_io *iter_io =
>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>> +
>>>>> + *dmap = iter_io->dmap;
>>>>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>>>>> +}
>>>>> +
>>>>> +static const struct ttm_kmap_iter_ops
>>>>> ttm_kmap_iter_linear_io_ops = {
>>>>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>>>>> + .maps_tt = false,
>>>>> +};
>>>>> +
>>>>> +struct ttm_kmap_iter *
>>>>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
>>>>> *iter_io,
>>>>> + struct ttm_device *bdev,
>>>>> + struct ttm_resource *mem)
>>>>> +{
>>>>> + int ret;
>>>>> +
>>>>> + ret = ttm_mem_io_reserve(bdev, mem);
>>>>> + if (ret)
>>>>> + goto out_err;
>>>>> + if (!mem->bus.is_iomem) {
>>>>> + ret = -EINVAL;
>>>>> + goto out_io_free;
>>>>> + }
>>>>> +
>>>>> + if (mem->bus.addr) {
>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
>>>>>> bus.addr);
>>>>> + iter_io->needs_unmap = false;
>>>>> + } else {
>>>>> + size_t bus_size = (size_t)mem->num_pages <<
>>>>> PAGE_SHIFT;
>>>>> +
>>>>> + iter_io->needs_unmap = true;
>>>>> + if (mem->bus.caching == ttm_write_combined)
>>>>> + dma_buf_map_set_vaddr_iomem(&iter_io-
>>>>>> dmap,
>>>>> +
>>>>> ioremap_wc(mem->bus.offset,
>>>>> +
>>>>> bus_size));
>>>>> + else if (mem->bus.caching == ttm_cached)
>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>>>>> + memremap(mem-
>>>>>> bus.offset, bus_size,
>>>>> +
>>>>> MEMREMAP_WB));
>>>> The comments in set_vaddr suggest that this is meant for
>>>> system-memory. Does that actually matter or is it just about not
>>>> losing the __iomem annotation on platforms where it matters?
>>> Yes, it's the latter. dma_buf_map() is relatively new and the
>>> author
>>> probably didn't think about the case of cached iomem, which is used
>>> by,
>>> for example, vmwgfx.
>>>
>>>> Apparently cached device local is a thing. Also should this not
>>>> be
>>>> wrapped in CONFIG_X86?
>>> Both dma_buf_map() and memremap are generic, I think, I guess
>>> memremap
>>> would return NULL if it's not supported.
>> It looks like memremap just wraps ioremap_cache, but since it also
>> discards the __iomem annotation should we be doing that universally?
>> Also not sure if ioremap_cache is universally supported, so wrapping
>> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
>> at least that looks like roughly what the previous code was doing?
>> Not
>> too sure tbh.
>>
> I think the long term goal is to use memremap all over the place, to
> just not have to bother with the __iomem annotation. But to do that io-
> mapping.h needs to support memremap. But for now we need to be strict
> about __iomem unless we're in arch specific code. That's why that
> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
I don't think that this will work. __iomem annotation is there because
we have architectures where you need to use special CPU instructions for
iomem access.
That won't go away just because we use memremap().
Christian.
>
> As for limited arch support for memremap cached, It looks like we only
> need to or in "backup" mapping modes in the memremap flags, and we'd
> mimic the previous behaviour.
>
> /Thomas
>
>
>>> /Thomas
>>>
>>>
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-25 15:48 ` Christian König
@ 2021-05-26 7:39 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-26 7:39 UTC (permalink / raw)
To: Christian König, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
On 5/25/21 5:48 PM, Christian König wrote:
>
>
> Am 25.05.21 um 12:07 schrieb Thomas Hellström:
>> On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
>>> On Tue, 25 May 2021 at 10:32, Thomas Hellström
>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>
>>>> On 5/25/21 11:18 AM, Matthew Auld wrote:
>>>>> On Fri, 21 May 2021 at 16:33, Thomas Hellström
>>>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>>> The internal ttm_bo_util memcpy uses ioremap functionality, and
>>>>>> while it
>>>>>> probably might be possible to use it for copying in- and out of
>>>>>> sglist represented io memory, using io_mem_reserve() /
>>>>>> io_mem_free()
>>>>>> callbacks, that would cause problems with fault().
>>>>>> Instead, implement a method mapping page-by-page using
>>>>>> kmap_local()
>>>>>> semantics. As an additional benefit we then avoid the
>>>>>> occasional global
>>>>>> TLB flushes of ioremap() and consuming ioremap space,
>>>>>> elimination of a
>>>>>> critical point of failure and with a slight change of semantics
>>>>>> we could
>>>>>> also push the memcpy out async for testing and async driver
>>>>>> development
>>>>>> purposes.
>>>>>>
>>>>>> A special linear iomem iterator is introduced internally to
>>>>>> mimic the
>>>>>> old ioremap behaviour for code-paths that can't immediately be
>>>>>> ported
>>>>>> over. This adds to the code size and should be considered a
>>>>>> temporary
>>>>>> solution.
>>>>>>
>>>>>> Looking at the code we have a lot of checks for iomap tagged
>>>>>> pointers.
>>>>>> Ideally we should extend the core memremap functions to also
>>>>>> accept
>>>>>> uncached memory and kmap_local functionality. Then we could
>>>>>> strip a
>>>>>> lot of code.
>>>>>>
>>>>>> Cc: Christian König <christian.koenig@amd.com>
>>>>>> Signed-off-by: Thomas Hellström <
>>>>>> thomas.hellstrom@linux.intel.com>
>>>>>> ---
>>>>>> v3:
>>>>>> - Split up in various TTM files and addressed review comments
>>>>>> by
>>>>>> Christian König. Tested and fixed legacy iomap memcpy path
>>>>>> on i915.
>>>>>> ---
>>>>>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
>>>>>> ---------
>>>>>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>>>>>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>>>>>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>>>>>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>>>>>> include/drm/ttm/ttm_caching.h | 2 +
>>>>>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>>>>>> include/drm/ttm/ttm_resource.h | 61 +++++++
>>>>>> include/drm/ttm/ttm_tt.h | 16 ++
>>>>>> 9 files changed, 508 insertions(+), 181 deletions(-)
>>>>>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> index ae8b61460724..912cbe8e60a2 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
>>>>>> *bdev,
>>>>>> mem->bus.addr = NULL;
>>>>>> }
>>>>>>
>>>>>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>>>>>> - struct ttm_resource *mem,
>>>>>> - void **virtual)
>>>>>> +/**
>>>>>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move
>>>>>> operation.
>>>>>> + * @bo: The struct ttm_buffer_object.
>>>>>> + * @new_mem: The struct ttm_resource we're moving to (copy
>>>>>> destination).
>>>>>> + * @new_iter: A struct ttm_kmap_iter representing the
>>>>>> destination resource.
>>>>>> + * @src_iter: A struct ttm_kmap_iter representing the source
>>>>>> resource.
>>>>>> + *
>>>>>> + * This function is intended to be able to move out async
>>>>>> under a
>>>>>> + * dma-fence if desired.
>>>>>> + */
>>>>>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>>>>>> + struct ttm_resource *dst_mem,
>>>>>> + struct ttm_kmap_iter *dst_iter,
>>>>>> + struct ttm_kmap_iter *src_iter)
>>>>>> {
>>>>>> - int ret;
>>>>>> - void *addr;
>>>>>> -
>>>>>> - *virtual = NULL;
>>>>>> - ret = ttm_mem_io_reserve(bdev, mem);
>>>>>> - if (ret || !mem->bus.is_iomem)
>>>>>> - return ret;
>>>>>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
>>>>>>> ops;
>>>>>> + const struct ttm_kmap_iter_ops *src_ops = src_iter-
>>>>>>> ops;
>>>>>> + struct ttm_tt *ttm = bo->ttm;
>>>>>> + struct dma_buf_map src_map, dst_map;
>>>>>> + pgoff_t i;
>>>>>>
>>>>>> - if (mem->bus.addr) {
>>>>>> - addr = mem->bus.addr;
>>>>>> - } else {
>>>>>> - size_t bus_size = (size_t)mem->num_pages <<
>>>>>> PAGE_SHIFT;
>>>>>> + /* Single TTM move. NOP */
>>>>>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>>>>>> + return;
>>>>>>
>>>>>> - if (mem->bus.caching == ttm_write_combined)
>>>>>> - addr = ioremap_wc(mem->bus.offset,
>>>>>> bus_size);
>>>>>> -#ifdef CONFIG_X86
>>>>>> - else if (mem->bus.caching == ttm_cached)
>>>>>> - addr = ioremap_cache(mem->bus.offset,
>>>>>> bus_size);
>>>>>> -#endif
>>>>>> - else
>>>>>> - addr = ioremap(mem->bus.offset,
>>>>>> bus_size);
>>>>>> - if (!addr) {
>>>>>> - ttm_mem_io_free(bdev, mem);
>>>>>> - return -ENOMEM;
>>>>>> + /* Don't move nonexistent data. Clear destination
>>>>>> instead. */
>>>>>> + if (src_ops->maps_tt && (!ttm ||
>>>>>> !ttm_tt_is_populated(ttm))) {
>>>>>> + if (ttm && !(ttm->page_flags &
>>>>>> TTM_PAGE_FLAG_ZERO_ALLOC))
>>>>>> + return;
>>>>>> +
>>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>>> + dst_ops->map_local(dst_iter, &dst_map,
>>>>>> i);
>>>>>> + if (dst_map.is_iomem)
>>>>>> + memset_io(dst_map.vaddr_iomem,
>>>>>> 0, PAGE_SIZE);
>>>>>> + else
>>>>>> + memset(dst_map.vaddr, 0,
>>>>>> PAGE_SIZE);
>>>>>> + if (dst_ops->unmap_local)
>>>>>> + dst_ops->unmap_local(dst_iter,
>>>>>> &dst_map);
>>>>>> }
>>>>>> + return;
>>>>>> }
>>>>>> - *virtual = addr;
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>>>>>> - struct ttm_resource *mem,
>>>>>> - void *virtual)
>>>>>> -{
>>>>>> - if (virtual && mem->bus.addr == NULL)
>>>>>> - iounmap(virtual);
>>>>>> - ttm_mem_io_free(bdev, mem);
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_io_page(void *dst, void *src, unsigned
>>>>>> long page)
>>>>>> -{
>>>>>> - uint32_t *dstP =
>>>>>> - (uint32_t *) ((unsigned long)dst + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - uint32_t *srcP =
>>>>>> - (uint32_t *) ((unsigned long)src + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> -
>>>>>> - int i;
>>>>>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>>>>>> - iowrite32(ioread32(srcP++), dstP++);
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>>>>>> - unsigned long page,
>>>>>> - pgprot_t prot)
>>>>>> -{
>>>>>> - struct page *d = ttm->pages[page];
>>>>>> - void *dst;
>>>>>> -
>>>>>> - if (!d)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - src = (void *)((unsigned long)src + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - dst = kmap_atomic_prot(d, prot);
>>>>>> - if (!dst)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - memcpy_fromio(dst, src, PAGE_SIZE);
>>>>>> -
>>>>>> - kunmap_atomic(dst);
>>>>>> -
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>>>>>> - unsigned long page,
>>>>>> - pgprot_t prot)
>>>>>> -{
>>>>>> - struct page *s = ttm->pages[page];
>>>>>> - void *src;
>>>>>> -
>>>>>> - if (!s)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - dst = (void *)((unsigned long)dst + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - src = kmap_atomic_prot(s, prot);
>>>>>> - if (!src)
>>>>>> - return -ENOMEM;
>>>>>>
>>>>>> - memcpy_toio(dst, src, PAGE_SIZE);
>>>>>> -
>>>>>> - kunmap_atomic(src);
>>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>>> + dst_ops->map_local(dst_iter, &dst_map, i);
>>>>>> + src_ops->map_local(src_iter, &src_map, i);
>>>>>> +
>>>>>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>>>>>> + memcpy(dst_map.vaddr, src_map.vaddr,
>>>>>> PAGE_SIZE);
>>>>>> + } else if (!src_map.is_iomem) {
>>>>>> + dma_buf_map_memcpy_to(&dst_map,
>>>>>> src_map.vaddr,
>>>>>> + PAGE_SIZE);
>>>>>> + } else if (!dst_map.is_iomem) {
>>>>>> + memcpy_fromio(dst_map.vaddr,
>>>>>> src_map.vaddr_iomem,
>>>>>> + PAGE_SIZE);
>>>>>> + } else {
>>>>>> + int j;
>>>>>> + u32 __iomem *src = src_map.vaddr_iomem;
>>>>>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>>>>>
>>>>>> - return 0;
>>>>>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
>>>>> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>>>> OK, will fix.
>>>>
>>>>
>>>>>> + iowrite32(ioread32(src++),
>>>>>> dst++);
>>>>>> + }
>>>>>> + if (src_ops->unmap_local)
>>>>>> + src_ops->unmap_local(src_iter,
>>>>>> &src_map);
>>>>>> + if (dst_ops->unmap_local)
>>>>>> + dst_ops->unmap_local(dst_iter,
>>>>>> &dst_map);
>>>>>> + }
>>>>>> }
>>>>>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>>>>>
>>>>>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>>>>>> struct ttm_operation_ctx *ctx,
>>>>>> - struct ttm_resource *new_mem)
>>>>>> + struct ttm_resource *dst_mem)
>>>>>> {
>>>>>> struct ttm_device *bdev = bo->bdev;
>>>>>> - struct ttm_resource_manager *man =
>>>>>> ttm_manager_type(bdev, new_mem->mem_type);
>>>>>> + struct ttm_resource_manager *dst_man =
>>>>>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>>>>>> struct ttm_tt *ttm = bo->ttm;
>>>>>> - struct ttm_resource *old_mem = &bo->mem;
>>>>>> - struct ttm_resource old_copy = *old_mem;
>>>>>> - void *old_iomap;
>>>>>> - void *new_iomap;
>>>>>> + struct ttm_resource *src_mem = &bo->mem;
>>>>>> + struct ttm_resource_manager *src_man =
>>>>>> + ttm_manager_type(bdev, src_mem->mem_type);
>>>>>> + struct ttm_resource src_copy = *src_mem;
>>>>>> + union {
>>>>>> + struct ttm_kmap_iter_tt tt;
>>>>>> + struct ttm_kmap_iter_linear_io io;
>>>>>> + } _dst_iter, _src_iter;
>>>>>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>>>>>> int ret;
>>>>>> - unsigned long i;
>>>>>>
>>>>>> - ret = ttm_bo_wait_ctx(bo, ctx);
>>>>>> - if (ret)
>>>>>> - return ret;
>>>>>> -
>>>>>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>>>>>> - if (ret)
>>>>>> - return ret;
>>>>>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>>>>>> - if (ret)
>>>>>> - goto out;
>>>>>> -
>>>>>> - /*
>>>>>> - * Single TTM move. NOP.
>>>>>> - */
>>>>>> - if (old_iomap == NULL && new_iomap == NULL)
>>>>>> - goto out2;
>>>>>> -
>>>>>> - /*
>>>>>> - * Don't move nonexistent data. Clear destination
>>>>>> instead.
>>>>>> - */
>>>>>> - if (old_iomap == NULL &&
>>>>>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>>>>>> - !(ttm->page_flags &
>>>>>> TTM_PAGE_FLAG_SWAPPED)))) {
>>>>>> - memset_io(new_iomap, 0, new_mem-
>>>>>>> num_pages*PAGE_SIZE);
>>>>>> - goto out2;
>>>>>> - }
>>>>>> -
>>>>>> - /*
>>>>>> - * TTM might be null for moves within the same region.
>>>>>> - */
>>>>>> - if (ttm) {
>>>>>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
>>>>>> ||
>>>>>> + dst_man->use_tt)) {
>>>>>> ret = ttm_tt_populate(bdev, ttm, ctx);
>>>>>> if (ret)
>>>>>> - goto out1;
>>>>>> + return ret;
>>>>>> }
>>>>>>
>>>>>> - for (i = 0; i < new_mem->num_pages; ++i) {
>>>>>> - if (old_iomap == NULL) {
>>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>>> old_mem, PAGE_KERNEL);
>>>>>> - ret = ttm_copy_ttm_io_page(ttm,
>>>>>> new_iomap, i,
>>>>>> - prot);
>>>>>> - } else if (new_iomap == NULL) {
>>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>>> new_mem, PAGE_KERNEL);
>>>>>> - ret = ttm_copy_io_ttm_page(ttm,
>>>>>> old_iomap, i,
>>>>>> - prot);
>>>>>> - } else {
>>>>>> - ret = ttm_copy_io_page(new_iomap,
>>>>>> old_iomap, i);
>>>>>> - }
>>>>>> - if (ret)
>>>>>> - goto out1;
>>>>>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
>>>>>> bdev, dst_mem);
>>>>>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>>>>>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
>>>>>> bo->ttm);
>>>>>> + if (IS_ERR(dst_iter))
>>>>>> + return PTR_ERR(dst_iter);
>>>>>> +
>>>>>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
>>>>>> bdev, src_mem);
>>>>>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>>>>>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
>>>>>> bo->ttm);
>>>>>> + if (IS_ERR(src_iter)) {
>>>>>> + ret = PTR_ERR(src_iter);
>>>>>> + goto out_src_iter;
>>>>>> }
>>>>>> - mb();
>>>>>> -out2:
>>>>>> - old_copy = *old_mem;
>>>>>>
>>>>>> - ttm_bo_assign_mem(bo, new_mem);
>>>>>> -
>>>>>> - if (!man->use_tt)
>>>>>> - ttm_bo_tt_destroy(bo);
>>>>>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>>>>>> + src_copy = *src_mem;
>>>>>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>>>>>
>>>>>> -out1:
>>>>>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>>>>>> -out:
>>>>>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>>>>>> + if (!src_iter->ops->maps_tt)
>>>>>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
>>>>>> bdev, &src_copy);
>>>>>> +out_src_iter:
>>>>>> + if (!dst_iter->ops->maps_tt)
>>>>>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
>>>>>> bdev, dst_mem);
>>>>>>
>>>>>> - /*
>>>>>> - * On error, keep the mm node!
>>>>>> - */
>>>>>> - if (!ret)
>>>>>> - ttm_resource_free(bo, &old_copy);
>>>>>> return ret;
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>>>>>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
>>>>>> ttm_buffer_object *bo, struct ttm_resource *res,
>>>>>> man = ttm_manager_type(bo->bdev, res->mem_type);
>>>>>> caching = man->use_tt ? bo->ttm->caching : res-
>>>>>>> bus.caching;
>>>>>> - /* Cached mappings need no adjustment */
>>>>>> - if (caching == ttm_cached)
>>>>>> - return tmp;
>>>>>> -
>>>>>> -#if defined(__i386__) || defined(__x86_64__)
>>>>>> - if (caching == ttm_write_combined)
>>>>>> - tmp = pgprot_writecombine(tmp);
>>>>>> - else if (boot_cpu_data.x86 > 3)
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> -#if defined(__ia64__) || defined(__arm__) ||
>>>>>> defined(__aarch64__) || \
>>>>>> - defined(__powerpc__) || defined(__mips__)
>>>>>> - if (caching == ttm_write_combined)
>>>>>> - tmp = pgprot_writecombine(tmp);
>>>>>> - else
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> -#if defined(__sparc__)
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> - return tmp;
>>>>>> + return ttm_prot_from_caching(caching, tmp);
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_io_prot);
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> index 56b0efdba1a9..997c458f68a9 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> @@ -31,12 +31,47 @@
>>>>>> */
>>>>>> #include <linux/module.h>
>>>>>> #include <linux/device.h>
>>>>>> +#include <linux/pgtable.h>
>>>>>> #include <linux/sched.h>
>>>>>> #include <linux/debugfs.h>
>>>>>> #include <drm/drm_sysfs.h>
>>>>>> +#include <drm/ttm/ttm_caching.h>
>>>>>>
>>>>>> #include "ttm_module.h"
>>>>>>
>>>>>> +/**
>>>>>> + * ttm_prot_from_caching - Modify the page protection
>>>>>> according to the
>>>>>> + * ttm cacing mode
>>>>>> + * @caching: The ttm caching mode
>>>>>> + * @tmp: The original page protection
>>>>>> + *
>>>>>> + * Return: The modified page protection
>>>>>> + */
>>>>>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
>>>>>> pgprot_t tmp)
>>>>>> +{
>>>>>> + /* Cached mappings need no adjustment */
>>>>>> + if (caching == ttm_cached)
>>>>>> + return tmp;
>>>>>> +
>>>>>> +#if defined(__i386__) || defined(__x86_64__)
>>>>>> + if (caching == ttm_write_combined)
>>>>>> + tmp = pgprot_writecombine(tmp);
>>>>>> + else if (boot_cpu_data.x86 > 3)
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> +#if defined(__ia64__) || defined(__arm__) ||
>>>>>> defined(__aarch64__) || \
>>>>>> + defined(__powerpc__) || defined(__mips__)
>>>>>> + if (caching == ttm_write_combined)
>>>>>> + tmp = pgprot_writecombine(tmp);
>>>>>> + else
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> +#if defined(__sparc__)
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> + return tmp;
>>>>>> +}
>>>>>> +
>>>>>> struct dentry *ttm_debugfs_root;
>>>>>>
>>>>>> static int __init ttm_init(void)
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> index 59e2b7157e41..e05ae7e3d477 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> @@ -22,6 +22,10 @@
>>>>>> * Authors: Christian König
>>>>>> */
>>>>>>
>>>>>> +#include <linux/dma-buf-map.h>
>>>>>> +#include <linux/io-mapping.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>>> +
>>>>>> #include <drm/ttm/ttm_resource.h>
>>>>>> #include <drm/ttm/ttm_bo_driver.h>
>>>>>>
>>>>>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
>>>>>> ttm_resource_manager *man,
>>>>>> man->func->debug(man, p);
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>>>>>> +
>>>>>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
>>>>>> *iter,
>>>>>> + struct dma_buf_map
>>>>>> *dmap,
>>>>>> + pgoff_t i)
>>>>>> +{
>>>>>> + struct ttm_kmap_iter_iomap *iter_io =
>>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>>> + void __iomem *addr;
>>>>>> +
>>>>>> +retry:
>>>>>> + while (i >= iter_io->cache.end) {
>>>>>> + iter_io->cache.sg = iter_io->cache.sg ?
>>>>>> + sg_next(iter_io->cache.sg) : iter_io-
>>>>>>> st->sgl;
>>>>>> + iter_io->cache.i = iter_io->cache.end;
>>>>>> + iter_io->cache.end += sg_dma_len(iter_io-
>>>>>>> cache.sg) >>
>>>>>> + PAGE_SHIFT;
>>>>>> + iter_io->cache.offs = sg_dma_address(iter_io-
>>>>>>> cache.sg) -
>>>>>> + iter_io->start;
>>>>>> + }
>>>>>> +
>>>>>> + if (i < iter_io->cache.i) {
>>>>>> + iter_io->cache.end = 0;
>>>>>> + iter_io->cache.sg = NULL;
>>>>>> + goto retry;
>>>>>> + }
>>>>>> +
>>>>>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
>>>>>>> cache.offs +
>>>>>> + (((resource_size_t)i -
>>>>>> iter_io->cache.i)
>>>>>> + << PAGE_SHIFT));
>>>>>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>>>>>> +}
>>>>>> +
>>>>>> +static void ttm_kmap_iter_iomap_unmap_local(struct
>>>>>> ttm_kmap_iter *iter,
>>>>>> + struct dma_buf_map
>>>>>> *map)
>>>>>> +{
>>>>>> + io_mapping_unmap_local(map->vaddr_iomem);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>>>>>> + .map_local = ttm_kmap_iter_iomap_map_local,
>>>>>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>>>>>> + .maps_tt = false,
>>>>>> +};
>>>>>> +
>>>>>> +/**
>>>>>> + * ttm_kmap_iter_iomap_init - Initialize a struct
>>>>>> ttm_kmap_iter_iomap
>>>>>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>>>>>> + * @iomap: The struct io_mapping representing the underlying
>>>>>> linear io_memory.
>>>>>> + * @st: sg_table into @iomap, representing the memory of the
>>>>>> struct
>>>>>> + * ttm_resource.
>>>>>> + * @start: Offset that needs to be subtracted from @st to make
>>>>>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>>>>>> + *
>>>>>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>>>>>> + */
>>>>>> +struct ttm_kmap_iter *
>>>>>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>>>>>> + struct io_mapping *iomap,
>>>>>> + struct sg_table *st,
>>>>>> + resource_size_t start)
>>>>>> +{
>>>>>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>>>>>> + iter_io->iomap = iomap;
>>>>>> + iter_io->st = st;
>>>>>> + iter_io->start = start;
>>>>>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>>>>>> +
>>>>>> + return &iter_io->base;
>>>>>> +}
>>>>>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>>>>>> +
>>>>>> +/**
>>>>>> + * DOC: Linear io iterator
>>>>>> + *
>>>>>> + * This code should die in the not too near future. Best would
>>>>>> be if we could
>>>>>> + * make io-mapping use memremap for all io memory, and have
>>>>>> memremap
>>>>>> + * implement a kmap_local functionality. We could then strip a
>>>>>> huge amount of
>>>>>> + * code. These linear io iterators are implemented to mimic
>>>>>> old functionality,
>>>>>> + * and they don't use kmap_local semantics at all internally.
>>>>>> Rather ioremap or
>>>>>> + * friends, and at least on 32-bit they add global TLB flushes
>>>>>> and points
>>>>>> + * of failure.
>>>>>> + */
>>>>>> +
>>>>>> +static void ttm_kmap_iter_linear_io_map_local(struct
>>>>>> ttm_kmap_iter *iter,
>>>>>> + struct
>>>>>> dma_buf_map *dmap,
>>>>>> + pgoff_t i)
>>>>>> +{
>>>>>> + struct ttm_kmap_iter_linear_io *iter_io =
>>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>>> +
>>>>>> + *dmap = iter_io->dmap;
>>>>>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct ttm_kmap_iter_ops
>>>>>> ttm_kmap_iter_linear_io_ops = {
>>>>>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>>>>>> + .maps_tt = false,
>>>>>> +};
>>>>>> +
>>>>>> +struct ttm_kmap_iter *
>>>>>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
>>>>>> *iter_io,
>>>>>> + struct ttm_device *bdev,
>>>>>> + struct ttm_resource *mem)
>>>>>> +{
>>>>>> + int ret;
>>>>>> +
>>>>>> + ret = ttm_mem_io_reserve(bdev, mem);
>>>>>> + if (ret)
>>>>>> + goto out_err;
>>>>>> + if (!mem->bus.is_iomem) {
>>>>>> + ret = -EINVAL;
>>>>>> + goto out_io_free;
>>>>>> + }
>>>>>> +
>>>>>> + if (mem->bus.addr) {
>>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
>>>>>>> bus.addr);
>>>>>> + iter_io->needs_unmap = false;
>>>>>> + } else {
>>>>>> + size_t bus_size = (size_t)mem->num_pages <<
>>>>>> PAGE_SHIFT;
>>>>>> +
>>>>>> + iter_io->needs_unmap = true;
>>>>>> + if (mem->bus.caching == ttm_write_combined)
>>>>>> + dma_buf_map_set_vaddr_iomem(&iter_io-
>>>>>>> dmap,
>>>>>> +
>>>>>> ioremap_wc(mem->bus.offset,
>>>>>> +
>>>>>> bus_size));
>>>>>> + else if (mem->bus.caching == ttm_cached)
>>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>>>>>> + memremap(mem-
>>>>>>> bus.offset, bus_size,
>>>>>> +
>>>>>> MEMREMAP_WB));
>>>>> The comments in set_vaddr suggest that this is meant for
>>>>> system-memory. Does that actually matter or is it just about not
>>>>> losing the __iomem annotation on platforms where it matters?
>>>> Yes, it's the latter. dma_buf_map() is relatively new and the
>>>> author
>>>> probably didn't think about the case of cached iomem, which is used
>>>> by,
>>>> for example, vmwgfx.
>>>>
>>>>> Apparently cached device local is a thing. Also should this not
>>>>> be
>>>>> wrapped in CONFIG_X86?
>>>> Both dma_buf_map() and memremap are generic, I think, I guess
>>>> memremap
>>>> would return NULL if it's not supported.
>>> It looks like memremap just wraps ioremap_cache, but since it also
>>> discards the __iomem annotation should we be doing that universally?
>>> Also not sure if ioremap_cache is universally supported, so wrapping
>>> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
>>> at least that looks like roughly what the previous code was doing?
>>> Not
>>> too sure tbh.
>>>
>> I think the long term goal is to use memremap all over the place, to
>> just not have to bother with the __iomem annotation. But to do that io-
>> mapping.h needs to support memremap. But for now we need to be strict
>> about __iomem unless we're in arch specific code. That's why that
>> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
>
> I don't think that this will work. __iomem annotation is there because
> we have architectures where you need to use special CPU instructions
> for iomem access.
>
> That won't go away just because we use memremap().
That's true, but can we ever support those with TTM, given that we allow
user-space mmaping that transparently may change to an iomap? Given
that, and what's written here
https://lwn.net/Articles/653585/
To me it sounds like if an architecture can't support memremap, we can't
support it with TTM either.
In any case for this particular patch, to avoid potential regressions,
OK if I just add an ioremap() in case the memremap fails?
/Thomas
>
> Christian.
>
>>
>> As for limited arch support for memremap cached, It looks like we only
>> need to or in "backup" mapping modes in the memremap flags, and we'd
>> mimic the previous behaviour.
>>
>> /Thomas
>>
>>
>>>> /Thomas
>>>>
>>>>
>>
>
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-26 7:39 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-26 7:39 UTC (permalink / raw)
To: Christian König, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
On 5/25/21 5:48 PM, Christian König wrote:
>
>
> Am 25.05.21 um 12:07 schrieb Thomas Hellström:
>> On Tue, 2021-05-25 at 10:58 +0100, Matthew Auld wrote:
>>> On Tue, 25 May 2021 at 10:32, Thomas Hellström
>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>
>>>> On 5/25/21 11:18 AM, Matthew Auld wrote:
>>>>> On Fri, 21 May 2021 at 16:33, Thomas Hellström
>>>>> <thomas.hellstrom@linux.intel.com> wrote:
>>>>>> The internal ttm_bo_util memcpy uses ioremap functionality, and
>>>>>> while it
>>>>>> probably might be possible to use it for copying in- and out of
>>>>>> sglist represented io memory, using io_mem_reserve() /
>>>>>> io_mem_free()
>>>>>> callbacks, that would cause problems with fault().
>>>>>> Instead, implement a method mapping page-by-page using
>>>>>> kmap_local()
>>>>>> semantics. As an additional benefit we then avoid the
>>>>>> occasional global
>>>>>> TLB flushes of ioremap() and consuming ioremap space,
>>>>>> elimination of a
>>>>>> critical point of failure and with a slight change of semantics
>>>>>> we could
>>>>>> also push the memcpy out async for testing and async driver
>>>>>> development
>>>>>> purposes.
>>>>>>
>>>>>> A special linear iomem iterator is introduced internally to
>>>>>> mimic the
>>>>>> old ioremap behaviour for code-paths that can't immediately be
>>>>>> ported
>>>>>> over. This adds to the code size and should be considered a
>>>>>> temporary
>>>>>> solution.
>>>>>>
>>>>>> Looking at the code we have a lot of checks for iomap tagged
>>>>>> pointers.
>>>>>> Ideally we should extend the core memremap functions to also
>>>>>> accept
>>>>>> uncached memory and kmap_local functionality. Then we could
>>>>>> strip a
>>>>>> lot of code.
>>>>>>
>>>>>> Cc: Christian König <christian.koenig@amd.com>
>>>>>> Signed-off-by: Thomas Hellström <
>>>>>> thomas.hellstrom@linux.intel.com>
>>>>>> ---
>>>>>> v3:
>>>>>> - Split up in various TTM files and addressed review comments
>>>>>> by
>>>>>> Christian König. Tested and fixed legacy iomap memcpy path
>>>>>> on i915.
>>>>>> ---
>>>>>> drivers/gpu/drm/ttm/ttm_bo_util.c | 278 ++++++++++----------
>>>>>> ---------
>>>>>> drivers/gpu/drm/ttm/ttm_module.c | 35 ++++
>>>>>> drivers/gpu/drm/ttm/ttm_resource.c | 166 +++++++++++++++++
>>>>>> drivers/gpu/drm/ttm/ttm_tt.c | 42 +++++
>>>>>> include/drm/ttm/ttm_bo_driver.h | 28 +++
>>>>>> include/drm/ttm/ttm_caching.h | 2 +
>>>>>> include/drm/ttm/ttm_kmap_iter.h | 61 +++++++
>>>>>> include/drm/ttm/ttm_resource.h | 61 +++++++
>>>>>> include/drm/ttm/ttm_tt.h | 16 ++
>>>>>> 9 files changed, 508 insertions(+), 181 deletions(-)
>>>>>> create mode 100644 include/drm/ttm/ttm_kmap_iter.h
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> index ae8b61460724..912cbe8e60a2 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
>>>>>> @@ -72,190 +72,126 @@ void ttm_mem_io_free(struct ttm_device
>>>>>> *bdev,
>>>>>> mem->bus.addr = NULL;
>>>>>> }
>>>>>>
>>>>>> -static int ttm_resource_ioremap(struct ttm_device *bdev,
>>>>>> - struct ttm_resource *mem,
>>>>>> - void **virtual)
>>>>>> +/**
>>>>>> + * ttm_move_memcpy - Helper to perform a memcpy ttm move
>>>>>> operation.
>>>>>> + * @bo: The struct ttm_buffer_object.
>>>>>> + * @new_mem: The struct ttm_resource we're moving to (copy
>>>>>> destination).
>>>>>> + * @new_iter: A struct ttm_kmap_iter representing the
>>>>>> destination resource.
>>>>>> + * @src_iter: A struct ttm_kmap_iter representing the source
>>>>>> resource.
>>>>>> + *
>>>>>> + * This function is intended to be able to move out async
>>>>>> under a
>>>>>> + * dma-fence if desired.
>>>>>> + */
>>>>>> +void ttm_move_memcpy(struct ttm_buffer_object *bo,
>>>>>> + struct ttm_resource *dst_mem,
>>>>>> + struct ttm_kmap_iter *dst_iter,
>>>>>> + struct ttm_kmap_iter *src_iter)
>>>>>> {
>>>>>> - int ret;
>>>>>> - void *addr;
>>>>>> -
>>>>>> - *virtual = NULL;
>>>>>> - ret = ttm_mem_io_reserve(bdev, mem);
>>>>>> - if (ret || !mem->bus.is_iomem)
>>>>>> - return ret;
>>>>>> + const struct ttm_kmap_iter_ops *dst_ops = dst_iter-
>>>>>>> ops;
>>>>>> + const struct ttm_kmap_iter_ops *src_ops = src_iter-
>>>>>>> ops;
>>>>>> + struct ttm_tt *ttm = bo->ttm;
>>>>>> + struct dma_buf_map src_map, dst_map;
>>>>>> + pgoff_t i;
>>>>>>
>>>>>> - if (mem->bus.addr) {
>>>>>> - addr = mem->bus.addr;
>>>>>> - } else {
>>>>>> - size_t bus_size = (size_t)mem->num_pages <<
>>>>>> PAGE_SHIFT;
>>>>>> + /* Single TTM move. NOP */
>>>>>> + if (dst_ops->maps_tt && src_ops->maps_tt)
>>>>>> + return;
>>>>>>
>>>>>> - if (mem->bus.caching == ttm_write_combined)
>>>>>> - addr = ioremap_wc(mem->bus.offset,
>>>>>> bus_size);
>>>>>> -#ifdef CONFIG_X86
>>>>>> - else if (mem->bus.caching == ttm_cached)
>>>>>> - addr = ioremap_cache(mem->bus.offset,
>>>>>> bus_size);
>>>>>> -#endif
>>>>>> - else
>>>>>> - addr = ioremap(mem->bus.offset,
>>>>>> bus_size);
>>>>>> - if (!addr) {
>>>>>> - ttm_mem_io_free(bdev, mem);
>>>>>> - return -ENOMEM;
>>>>>> + /* Don't move nonexistent data. Clear destination
>>>>>> instead. */
>>>>>> + if (src_ops->maps_tt && (!ttm ||
>>>>>> !ttm_tt_is_populated(ttm))) {
>>>>>> + if (ttm && !(ttm->page_flags &
>>>>>> TTM_PAGE_FLAG_ZERO_ALLOC))
>>>>>> + return;
>>>>>> +
>>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>>> + dst_ops->map_local(dst_iter, &dst_map,
>>>>>> i);
>>>>>> + if (dst_map.is_iomem)
>>>>>> + memset_io(dst_map.vaddr_iomem,
>>>>>> 0, PAGE_SIZE);
>>>>>> + else
>>>>>> + memset(dst_map.vaddr, 0,
>>>>>> PAGE_SIZE);
>>>>>> + if (dst_ops->unmap_local)
>>>>>> + dst_ops->unmap_local(dst_iter,
>>>>>> &dst_map);
>>>>>> }
>>>>>> + return;
>>>>>> }
>>>>>> - *virtual = addr;
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static void ttm_resource_iounmap(struct ttm_device *bdev,
>>>>>> - struct ttm_resource *mem,
>>>>>> - void *virtual)
>>>>>> -{
>>>>>> - if (virtual && mem->bus.addr == NULL)
>>>>>> - iounmap(virtual);
>>>>>> - ttm_mem_io_free(bdev, mem);
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_io_page(void *dst, void *src, unsigned
>>>>>> long page)
>>>>>> -{
>>>>>> - uint32_t *dstP =
>>>>>> - (uint32_t *) ((unsigned long)dst + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - uint32_t *srcP =
>>>>>> - (uint32_t *) ((unsigned long)src + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> -
>>>>>> - int i;
>>>>>> - for (i = 0; i < PAGE_SIZE / sizeof(uint32_t); ++i)
>>>>>> - iowrite32(ioread32(srcP++), dstP++);
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_io_ttm_page(struct ttm_tt *ttm, void *src,
>>>>>> - unsigned long page,
>>>>>> - pgprot_t prot)
>>>>>> -{
>>>>>> - struct page *d = ttm->pages[page];
>>>>>> - void *dst;
>>>>>> -
>>>>>> - if (!d)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - src = (void *)((unsigned long)src + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - dst = kmap_atomic_prot(d, prot);
>>>>>> - if (!dst)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - memcpy_fromio(dst, src, PAGE_SIZE);
>>>>>> -
>>>>>> - kunmap_atomic(dst);
>>>>>> -
>>>>>> - return 0;
>>>>>> -}
>>>>>> -
>>>>>> -static int ttm_copy_ttm_io_page(struct ttm_tt *ttm, void *dst,
>>>>>> - unsigned long page,
>>>>>> - pgprot_t prot)
>>>>>> -{
>>>>>> - struct page *s = ttm->pages[page];
>>>>>> - void *src;
>>>>>> -
>>>>>> - if (!s)
>>>>>> - return -ENOMEM;
>>>>>> -
>>>>>> - dst = (void *)((unsigned long)dst + (page <<
>>>>>> PAGE_SHIFT));
>>>>>> - src = kmap_atomic_prot(s, prot);
>>>>>> - if (!src)
>>>>>> - return -ENOMEM;
>>>>>>
>>>>>> - memcpy_toio(dst, src, PAGE_SIZE);
>>>>>> -
>>>>>> - kunmap_atomic(src);
>>>>>> + for (i = 0; i < dst_mem->num_pages; ++i) {
>>>>>> + dst_ops->map_local(dst_iter, &dst_map, i);
>>>>>> + src_ops->map_local(src_iter, &src_map, i);
>>>>>> +
>>>>>> + if (!src_map.is_iomem && !dst_map.is_iomem) {
>>>>>> + memcpy(dst_map.vaddr, src_map.vaddr,
>>>>>> PAGE_SIZE);
>>>>>> + } else if (!src_map.is_iomem) {
>>>>>> + dma_buf_map_memcpy_to(&dst_map,
>>>>>> src_map.vaddr,
>>>>>> + PAGE_SIZE);
>>>>>> + } else if (!dst_map.is_iomem) {
>>>>>> + memcpy_fromio(dst_map.vaddr,
>>>>>> src_map.vaddr_iomem,
>>>>>> + PAGE_SIZE);
>>>>>> + } else {
>>>>>> + int j;
>>>>>> + u32 __iomem *src = src_map.vaddr_iomem;
>>>>>> + u32 __iomem *dst = dst_map.vaddr_iomem;
>>>>>>
>>>>>> - return 0;
>>>>>> + for (j = 0; j < (PAGE_SIZE >> 2); ++j)
>>>>> IMO PAGE_SIZE / sizeof(u32) is easier to understand.
>>>> OK, will fix.
>>>>
>>>>
>>>>>> + iowrite32(ioread32(src++),
>>>>>> dst++);
>>>>>> + }
>>>>>> + if (src_ops->unmap_local)
>>>>>> + src_ops->unmap_local(src_iter,
>>>>>> &src_map);
>>>>>> + if (dst_ops->unmap_local)
>>>>>> + dst_ops->unmap_local(dst_iter,
>>>>>> &dst_map);
>>>>>> + }
>>>>>> }
>>>>>> +EXPORT_SYMBOL(ttm_move_memcpy);
>>>>>>
>>>>>> int ttm_bo_move_memcpy(struct ttm_buffer_object *bo,
>>>>>> struct ttm_operation_ctx *ctx,
>>>>>> - struct ttm_resource *new_mem)
>>>>>> + struct ttm_resource *dst_mem)
>>>>>> {
>>>>>> struct ttm_device *bdev = bo->bdev;
>>>>>> - struct ttm_resource_manager *man =
>>>>>> ttm_manager_type(bdev, new_mem->mem_type);
>>>>>> + struct ttm_resource_manager *dst_man =
>>>>>> + ttm_manager_type(bo->bdev, dst_mem->mem_type);
>>>>>> struct ttm_tt *ttm = bo->ttm;
>>>>>> - struct ttm_resource *old_mem = &bo->mem;
>>>>>> - struct ttm_resource old_copy = *old_mem;
>>>>>> - void *old_iomap;
>>>>>> - void *new_iomap;
>>>>>> + struct ttm_resource *src_mem = &bo->mem;
>>>>>> + struct ttm_resource_manager *src_man =
>>>>>> + ttm_manager_type(bdev, src_mem->mem_type);
>>>>>> + struct ttm_resource src_copy = *src_mem;
>>>>>> + union {
>>>>>> + struct ttm_kmap_iter_tt tt;
>>>>>> + struct ttm_kmap_iter_linear_io io;
>>>>>> + } _dst_iter, _src_iter;
>>>>>> + struct ttm_kmap_iter *dst_iter, *src_iter;
>>>>>> int ret;
>>>>>> - unsigned long i;
>>>>>>
>>>>>> - ret = ttm_bo_wait_ctx(bo, ctx);
>>>>>> - if (ret)
>>>>>> - return ret;
>>>>>> -
>>>>>> - ret = ttm_resource_ioremap(bdev, old_mem, &old_iomap);
>>>>>> - if (ret)
>>>>>> - return ret;
>>>>>> - ret = ttm_resource_ioremap(bdev, new_mem, &new_iomap);
>>>>>> - if (ret)
>>>>>> - goto out;
>>>>>> -
>>>>>> - /*
>>>>>> - * Single TTM move. NOP.
>>>>>> - */
>>>>>> - if (old_iomap == NULL && new_iomap == NULL)
>>>>>> - goto out2;
>>>>>> -
>>>>>> - /*
>>>>>> - * Don't move nonexistent data. Clear destination
>>>>>> instead.
>>>>>> - */
>>>>>> - if (old_iomap == NULL &&
>>>>>> - (ttm == NULL || (!ttm_tt_is_populated(ttm) &&
>>>>>> - !(ttm->page_flags &
>>>>>> TTM_PAGE_FLAG_SWAPPED)))) {
>>>>>> - memset_io(new_iomap, 0, new_mem-
>>>>>>> num_pages*PAGE_SIZE);
>>>>>> - goto out2;
>>>>>> - }
>>>>>> -
>>>>>> - /*
>>>>>> - * TTM might be null for moves within the same region.
>>>>>> - */
>>>>>> - if (ttm) {
>>>>>> + if (ttm && ((ttm->page_flags & TTM_PAGE_FLAG_SWAPPED)
>>>>>> ||
>>>>>> + dst_man->use_tt)) {
>>>>>> ret = ttm_tt_populate(bdev, ttm, ctx);
>>>>>> if (ret)
>>>>>> - goto out1;
>>>>>> + return ret;
>>>>>> }
>>>>>>
>>>>>> - for (i = 0; i < new_mem->num_pages; ++i) {
>>>>>> - if (old_iomap == NULL) {
>>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>>> old_mem, PAGE_KERNEL);
>>>>>> - ret = ttm_copy_ttm_io_page(ttm,
>>>>>> new_iomap, i,
>>>>>> - prot);
>>>>>> - } else if (new_iomap == NULL) {
>>>>>> - pgprot_t prot = ttm_io_prot(bo,
>>>>>> new_mem, PAGE_KERNEL);
>>>>>> - ret = ttm_copy_io_ttm_page(ttm,
>>>>>> old_iomap, i,
>>>>>> - prot);
>>>>>> - } else {
>>>>>> - ret = ttm_copy_io_page(new_iomap,
>>>>>> old_iomap, i);
>>>>>> - }
>>>>>> - if (ret)
>>>>>> - goto out1;
>>>>>> + dst_iter = ttm_kmap_iter_linear_io_init(&_dst_iter.io,
>>>>>> bdev, dst_mem);
>>>>>> + if (PTR_ERR(dst_iter) == -EINVAL && dst_man->use_tt)
>>>>>> + dst_iter = ttm_kmap_iter_tt_init(&_dst_iter.tt,
>>>>>> bo->ttm);
>>>>>> + if (IS_ERR(dst_iter))
>>>>>> + return PTR_ERR(dst_iter);
>>>>>> +
>>>>>> + src_iter = ttm_kmap_iter_linear_io_init(&_src_iter.io,
>>>>>> bdev, src_mem);
>>>>>> + if (PTR_ERR(src_iter) == -EINVAL && src_man->use_tt)
>>>>>> + src_iter = ttm_kmap_iter_tt_init(&_src_iter.tt,
>>>>>> bo->ttm);
>>>>>> + if (IS_ERR(src_iter)) {
>>>>>> + ret = PTR_ERR(src_iter);
>>>>>> + goto out_src_iter;
>>>>>> }
>>>>>> - mb();
>>>>>> -out2:
>>>>>> - old_copy = *old_mem;
>>>>>>
>>>>>> - ttm_bo_assign_mem(bo, new_mem);
>>>>>> -
>>>>>> - if (!man->use_tt)
>>>>>> - ttm_bo_tt_destroy(bo);
>>>>>> + ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
>>>>>> + src_copy = *src_mem;
>>>>>> + ttm_bo_move_sync_cleanup(bo, dst_mem);
>>>>>>
>>>>>> -out1:
>>>>>> - ttm_resource_iounmap(bdev, old_mem, new_iomap);
>>>>>> -out:
>>>>>> - ttm_resource_iounmap(bdev, &old_copy, old_iomap);
>>>>>> + if (!src_iter->ops->maps_tt)
>>>>>> + ttm_kmap_iter_linear_io_fini(&_src_iter.io,
>>>>>> bdev, &src_copy);
>>>>>> +out_src_iter:
>>>>>> + if (!dst_iter->ops->maps_tt)
>>>>>> + ttm_kmap_iter_linear_io_fini(&_dst_iter.io,
>>>>>> bdev, dst_mem);
>>>>>>
>>>>>> - /*
>>>>>> - * On error, keep the mm node!
>>>>>> - */
>>>>>> - if (!ret)
>>>>>> - ttm_resource_free(bo, &old_copy);
>>>>>> return ret;
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_bo_move_memcpy);
>>>>>> @@ -336,27 +272,7 @@ pgprot_t ttm_io_prot(struct
>>>>>> ttm_buffer_object *bo, struct ttm_resource *res,
>>>>>> man = ttm_manager_type(bo->bdev, res->mem_type);
>>>>>> caching = man->use_tt ? bo->ttm->caching : res-
>>>>>>> bus.caching;
>>>>>> - /* Cached mappings need no adjustment */
>>>>>> - if (caching == ttm_cached)
>>>>>> - return tmp;
>>>>>> -
>>>>>> -#if defined(__i386__) || defined(__x86_64__)
>>>>>> - if (caching == ttm_write_combined)
>>>>>> - tmp = pgprot_writecombine(tmp);
>>>>>> - else if (boot_cpu_data.x86 > 3)
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> -#if defined(__ia64__) || defined(__arm__) ||
>>>>>> defined(__aarch64__) || \
>>>>>> - defined(__powerpc__) || defined(__mips__)
>>>>>> - if (caching == ttm_write_combined)
>>>>>> - tmp = pgprot_writecombine(tmp);
>>>>>> - else
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> -#if defined(__sparc__)
>>>>>> - tmp = pgprot_noncached(tmp);
>>>>>> -#endif
>>>>>> - return tmp;
>>>>>> + return ttm_prot_from_caching(caching, tmp);
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_io_prot);
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> index 56b0efdba1a9..997c458f68a9 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_module.c
>>>>>> @@ -31,12 +31,47 @@
>>>>>> */
>>>>>> #include <linux/module.h>
>>>>>> #include <linux/device.h>
>>>>>> +#include <linux/pgtable.h>
>>>>>> #include <linux/sched.h>
>>>>>> #include <linux/debugfs.h>
>>>>>> #include <drm/drm_sysfs.h>
>>>>>> +#include <drm/ttm/ttm_caching.h>
>>>>>>
>>>>>> #include "ttm_module.h"
>>>>>>
>>>>>> +/**
>>>>>> + * ttm_prot_from_caching - Modify the page protection
>>>>>> according to the
>>>>>> + * ttm cacing mode
>>>>>> + * @caching: The ttm caching mode
>>>>>> + * @tmp: The original page protection
>>>>>> + *
>>>>>> + * Return: The modified page protection
>>>>>> + */
>>>>>> +pgprot_t ttm_prot_from_caching(enum ttm_caching caching,
>>>>>> pgprot_t tmp)
>>>>>> +{
>>>>>> + /* Cached mappings need no adjustment */
>>>>>> + if (caching == ttm_cached)
>>>>>> + return tmp;
>>>>>> +
>>>>>> +#if defined(__i386__) || defined(__x86_64__)
>>>>>> + if (caching == ttm_write_combined)
>>>>>> + tmp = pgprot_writecombine(tmp);
>>>>>> + else if (boot_cpu_data.x86 > 3)
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> +#if defined(__ia64__) || defined(__arm__) ||
>>>>>> defined(__aarch64__) || \
>>>>>> + defined(__powerpc__) || defined(__mips__)
>>>>>> + if (caching == ttm_write_combined)
>>>>>> + tmp = pgprot_writecombine(tmp);
>>>>>> + else
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> +#if defined(__sparc__)
>>>>>> + tmp = pgprot_noncached(tmp);
>>>>>> +#endif
>>>>>> + return tmp;
>>>>>> +}
>>>>>> +
>>>>>> struct dentry *ttm_debugfs_root;
>>>>>>
>>>>>> static int __init ttm_init(void)
>>>>>> diff --git a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> index 59e2b7157e41..e05ae7e3d477 100644
>>>>>> --- a/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> +++ b/drivers/gpu/drm/ttm/ttm_resource.c
>>>>>> @@ -22,6 +22,10 @@
>>>>>> * Authors: Christian König
>>>>>> */
>>>>>>
>>>>>> +#include <linux/dma-buf-map.h>
>>>>>> +#include <linux/io-mapping.h>
>>>>>> +#include <linux/scatterlist.h>
>>>>>> +
>>>>>> #include <drm/ttm/ttm_resource.h>
>>>>>> #include <drm/ttm/ttm_bo_driver.h>
>>>>>>
>>>>>> @@ -147,3 +151,165 @@ void ttm_resource_manager_debug(struct
>>>>>> ttm_resource_manager *man,
>>>>>> man->func->debug(man, p);
>>>>>> }
>>>>>> EXPORT_SYMBOL(ttm_resource_manager_debug);
>>>>>> +
>>>>>> +static void ttm_kmap_iter_iomap_map_local(struct ttm_kmap_iter
>>>>>> *iter,
>>>>>> + struct dma_buf_map
>>>>>> *dmap,
>>>>>> + pgoff_t i)
>>>>>> +{
>>>>>> + struct ttm_kmap_iter_iomap *iter_io =
>>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>>> + void __iomem *addr;
>>>>>> +
>>>>>> +retry:
>>>>>> + while (i >= iter_io->cache.end) {
>>>>>> + iter_io->cache.sg = iter_io->cache.sg ?
>>>>>> + sg_next(iter_io->cache.sg) : iter_io-
>>>>>>> st->sgl;
>>>>>> + iter_io->cache.i = iter_io->cache.end;
>>>>>> + iter_io->cache.end += sg_dma_len(iter_io-
>>>>>>> cache.sg) >>
>>>>>> + PAGE_SHIFT;
>>>>>> + iter_io->cache.offs = sg_dma_address(iter_io-
>>>>>>> cache.sg) -
>>>>>> + iter_io->start;
>>>>>> + }
>>>>>> +
>>>>>> + if (i < iter_io->cache.i) {
>>>>>> + iter_io->cache.end = 0;
>>>>>> + iter_io->cache.sg = NULL;
>>>>>> + goto retry;
>>>>>> + }
>>>>>> +
>>>>>> + addr = io_mapping_map_local_wc(iter_io->iomap, iter_io-
>>>>>>> cache.offs +
>>>>>> + (((resource_size_t)i -
>>>>>> iter_io->cache.i)
>>>>>> + << PAGE_SHIFT));
>>>>>> + dma_buf_map_set_vaddr_iomem(dmap, addr);
>>>>>> +}
>>>>>> +
>>>>>> +static void ttm_kmap_iter_iomap_unmap_local(struct
>>>>>> ttm_kmap_iter *iter,
>>>>>> + struct dma_buf_map
>>>>>> *map)
>>>>>> +{
>>>>>> + io_mapping_unmap_local(map->vaddr_iomem);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct ttm_kmap_iter_ops ttm_kmap_iter_io_ops = {
>>>>>> + .map_local = ttm_kmap_iter_iomap_map_local,
>>>>>> + .unmap_local = ttm_kmap_iter_iomap_unmap_local,
>>>>>> + .maps_tt = false,
>>>>>> +};
>>>>>> +
>>>>>> +/**
>>>>>> + * ttm_kmap_iter_iomap_init - Initialize a struct
>>>>>> ttm_kmap_iter_iomap
>>>>>> + * @iter_io: The struct ttm_kmap_iter_iomap to initialize.
>>>>>> + * @iomap: The struct io_mapping representing the underlying
>>>>>> linear io_memory.
>>>>>> + * @st: sg_table into @iomap, representing the memory of the
>>>>>> struct
>>>>>> + * ttm_resource.
>>>>>> + * @start: Offset that needs to be subtracted from @st to make
>>>>>> + * sg_dma_address(st->sgl) - @start == 0 for @iomap start.
>>>>>> + *
>>>>>> + * Return: Pointer to the embedded struct ttm_kmap_iter.
>>>>>> + */
>>>>>> +struct ttm_kmap_iter *
>>>>>> +ttm_kmap_iter_iomap_init(struct ttm_kmap_iter_iomap *iter_io,
>>>>>> + struct io_mapping *iomap,
>>>>>> + struct sg_table *st,
>>>>>> + resource_size_t start)
>>>>>> +{
>>>>>> + iter_io->base.ops = &ttm_kmap_iter_io_ops;
>>>>>> + iter_io->iomap = iomap;
>>>>>> + iter_io->st = st;
>>>>>> + iter_io->start = start;
>>>>>> + memset(&iter_io->cache, 0, sizeof(iter_io->cache));
>>>>>> +
>>>>>> + return &iter_io->base;
>>>>>> +}
>>>>>> +EXPORT_SYMBOL(ttm_kmap_iter_iomap_init);
>>>>>> +
>>>>>> +/**
>>>>>> + * DOC: Linear io iterator
>>>>>> + *
>>>>>> + * This code should die in the not too near future. Best would
>>>>>> be if we could
>>>>>> + * make io-mapping use memremap for all io memory, and have
>>>>>> memremap
>>>>>> + * implement a kmap_local functionality. We could then strip a
>>>>>> huge amount of
>>>>>> + * code. These linear io iterators are implemented to mimic
>>>>>> old functionality,
>>>>>> + * and they don't use kmap_local semantics at all internally.
>>>>>> Rather ioremap or
>>>>>> + * friends, and at least on 32-bit they add global TLB flushes
>>>>>> and points
>>>>>> + * of failure.
>>>>>> + */
>>>>>> +
>>>>>> +static void ttm_kmap_iter_linear_io_map_local(struct
>>>>>> ttm_kmap_iter *iter,
>>>>>> + struct
>>>>>> dma_buf_map *dmap,
>>>>>> + pgoff_t i)
>>>>>> +{
>>>>>> + struct ttm_kmap_iter_linear_io *iter_io =
>>>>>> + container_of(iter, typeof(*iter_io), base);
>>>>>> +
>>>>>> + *dmap = iter_io->dmap;
>>>>>> + dma_buf_map_incr(dmap, i * PAGE_SIZE);
>>>>>> +}
>>>>>> +
>>>>>> +static const struct ttm_kmap_iter_ops
>>>>>> ttm_kmap_iter_linear_io_ops = {
>>>>>> + .map_local = ttm_kmap_iter_linear_io_map_local,
>>>>>> + .maps_tt = false,
>>>>>> +};
>>>>>> +
>>>>>> +struct ttm_kmap_iter *
>>>>>> +ttm_kmap_iter_linear_io_init(struct ttm_kmap_iter_linear_io
>>>>>> *iter_io,
>>>>>> + struct ttm_device *bdev,
>>>>>> + struct ttm_resource *mem)
>>>>>> +{
>>>>>> + int ret;
>>>>>> +
>>>>>> + ret = ttm_mem_io_reserve(bdev, mem);
>>>>>> + if (ret)
>>>>>> + goto out_err;
>>>>>> + if (!mem->bus.is_iomem) {
>>>>>> + ret = -EINVAL;
>>>>>> + goto out_io_free;
>>>>>> + }
>>>>>> +
>>>>>> + if (mem->bus.addr) {
>>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap, mem-
>>>>>>> bus.addr);
>>>>>> + iter_io->needs_unmap = false;
>>>>>> + } else {
>>>>>> + size_t bus_size = (size_t)mem->num_pages <<
>>>>>> PAGE_SHIFT;
>>>>>> +
>>>>>> + iter_io->needs_unmap = true;
>>>>>> + if (mem->bus.caching == ttm_write_combined)
>>>>>> + dma_buf_map_set_vaddr_iomem(&iter_io-
>>>>>>> dmap,
>>>>>> +
>>>>>> ioremap_wc(mem->bus.offset,
>>>>>> +
>>>>>> bus_size));
>>>>>> + else if (mem->bus.caching == ttm_cached)
>>>>>> + dma_buf_map_set_vaddr(&iter_io->dmap,
>>>>>> + memremap(mem-
>>>>>>> bus.offset, bus_size,
>>>>>> +
>>>>>> MEMREMAP_WB));
>>>>> The comments in set_vaddr suggest that this is meant for
>>>>> system-memory. Does that actually matter or is it just about not
>>>>> losing the __iomem annotation on platforms where it matters?
>>>> Yes, it's the latter. dma_buf_map() is relatively new and the
>>>> author
>>>> probably didn't think about the case of cached iomem, which is used
>>>> by,
>>>> for example, vmwgfx.
>>>>
>>>>> Apparently cached device local is a thing. Also should this not
>>>>> be
>>>>> wrapped in CONFIG_X86?
>>>> Both dma_buf_map() and memremap are generic, I think, I guess
>>>> memremap
>>>> would return NULL if it's not supported.
>>> It looks like memremap just wraps ioremap_cache, but since it also
>>> discards the __iomem annotation should we be doing that universally?
>>> Also not sure if ioremap_cache is universally supported, so wrapping
>>> in CONFIG_X86 and falling back to plain ioremap() might be needed? Or
>>> at least that looks like roughly what the previous code was doing?
>>> Not
>>> too sure tbh.
>>>
>> I think the long term goal is to use memremap all over the place, to
>> just not have to bother with the __iomem annotation. But to do that io-
>> mapping.h needs to support memremap. But for now we need to be strict
>> about __iomem unless we're in arch specific code. That's why that
>> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
>
> I don't think that this will work. __iomem annotation is there because
> we have architectures where you need to use special CPU instructions
> for iomem access.
>
> That won't go away just because we use memremap().
That's true, but can we ever support those with TTM, given that we allow
user-space mmaping that transparently may change to an iomap? Given
that, and what's written here
https://lwn.net/Articles/653585/
To me it sounds like if an architecture can't support memremap, we can't
support it with TTM either.
In any case for this particular patch, to avoid potential regressions,
OK if I just add an ioremap() in case the memremap fails?
/Thomas
>
> Christian.
>
>>
>> As for limited arch support for memremap cached, It looks like we only
>> need to or in "backup" mapping modes in the memremap flags, and we'd
>> mimic the previous behaviour.
>>
>> /Thomas
>>
>>
>>>> /Thomas
>>>>
>>>>
>>
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-26 7:39 ` Thomas Hellström
@ 2021-05-26 10:45 ` Christian König
-1 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-26 10:45 UTC (permalink / raw)
To: Thomas Hellström, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
Am 26.05.21 um 09:39 schrieb Thomas Hellström:
> [SNIP]
>>> I think the long term goal is to use memremap all over the place, to
>>> just not have to bother with the __iomem annotation. But to do that io-
>>> mapping.h needs to support memremap. But for now we need to be strict
>>> about __iomem unless we're in arch specific code. That's why that
>>> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
>>
>> I don't think that this will work. __iomem annotation is there
>> because we have architectures where you need to use special CPU
>> instructions for iomem access.
>>
>> That won't go away just because we use memremap().
>
> That's true, but can we ever support those with TTM, given that we
> allow user-space mmaping that transparently may change to an iomap?
> Given that, and what's written here
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F653585%2F&data=04%7C01%7Cchristian.koenig%40amd.com%7C1cdcfe9d20e740308c9e08d92019785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637576116034492654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e2BFGQJcElwVxrvHcnALDWscHN7ernLekGvXHqWBcwY%3D&reserved=0
>
>
> To me it sounds like if an architecture can't support memremap, we
> can't support it with TTM either.
That was also my argument, but this is unfortunately not true.
We already have architectures where the __iomem approach is mandatory
for kernel mappings, but work fine for userspace (don't ask me how that
works, idk).
That's the reason why we had to fix the UVD firmware upload in the kernel:
commit ba0b2275a6781b2f3919d931d63329b5548f6d5f
Author: Christian König <christian.koenig@amd.com>
Date: Tue Aug 23 11:00:17 2016 +0200
drm/amdgpu: use memcpy_to/fromio for UVD fw upload
>
> In any case for this particular patch, to avoid potential regressions,
> OK if I just add an ioremap() in case the memremap fails?
Well because of the issues outlined above I would actually prefer if we
can keep the __iomem annotation around.
Christian.
>
> /Thomas
>
>
>>
>> Christian.
>>
>>>
>>> As for limited arch support for memremap cached, It looks like we only
>>> need to or in "backup" mapping modes in the memremap flags, and we'd
>>> mimic the previous behaviour.
>>>
>>> /Thomas
>>>
>>>
>>>>> /Thomas
>>>>>
>>>>>
>>>
>>
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-26 10:45 ` Christian König
0 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-26 10:45 UTC (permalink / raw)
To: Thomas Hellström, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
Am 26.05.21 um 09:39 schrieb Thomas Hellström:
> [SNIP]
>>> I think the long term goal is to use memremap all over the place, to
>>> just not have to bother with the __iomem annotation. But to do that io-
>>> mapping.h needs to support memremap. But for now we need to be strict
>>> about __iomem unless we're in arch specific code. That's why that
>>> dma_buf_map thing was created, but TTM memcpy was never fully adapted.
>>
>> I don't think that this will work. __iomem annotation is there
>> because we have architectures where you need to use special CPU
>> instructions for iomem access.
>>
>> That won't go away just because we use memremap().
>
> That's true, but can we ever support those with TTM, given that we
> allow user-space mmaping that transparently may change to an iomap?
> Given that, and what's written here
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F653585%2F&data=04%7C01%7Cchristian.koenig%40amd.com%7C1cdcfe9d20e740308c9e08d92019785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637576116034492654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e2BFGQJcElwVxrvHcnALDWscHN7ernLekGvXHqWBcwY%3D&reserved=0
>
>
> To me it sounds like if an architecture can't support memremap, we
> can't support it with TTM either.
That was also my argument, but this is unfortunately not true.
We already have architectures where the __iomem approach is mandatory
for kernel mappings, but work fine for userspace (don't ask me how that
works, idk).
That's the reason why we had to fix the UVD firmware upload in the kernel:
commit ba0b2275a6781b2f3919d931d63329b5548f6d5f
Author: Christian König <christian.koenig@amd.com>
Date: Tue Aug 23 11:00:17 2016 +0200
drm/amdgpu: use memcpy_to/fromio for UVD fw upload
>
> In any case for this particular patch, to avoid potential regressions,
> OK if I just add an ioremap() in case the memremap fails?
Well because of the issues outlined above I would actually prefer if we
can keep the __iomem annotation around.
Christian.
>
> /Thomas
>
>
>>
>> Christian.
>>
>>>
>>> As for limited arch support for memremap cached, It looks like we only
>>> need to or in "backup" mapping modes in the memremap flags, and we'd
>>> mimic the previous behaviour.
>>>
>>> /Thomas
>>>
>>>
>>>>> /Thomas
>>>>>
>>>>>
>>>
>>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
2021-05-26 10:45 ` Christian König
@ 2021-05-26 10:57 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-26 10:57 UTC (permalink / raw)
To: Christian König, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
On Wed, 2021-05-26 at 12:45 +0200, Christian König wrote:
> Am 26.05.21 um 09:39 schrieb Thomas Hellström:
> > [SNIP]
> > > > I think the long term goal is to use memremap all over the
> > > > place, to
> > > > just not have to bother with the __iomem annotation. But to do
> > > > that io-
> > > > mapping.h needs to support memremap. But for now we need to be
> > > > strict
> > > > about __iomem unless we're in arch specific code. That's why
> > > > that
> > > > dma_buf_map thing was created, but TTM memcpy was never fully
> > > > adapted.
> > >
> > > I don't think that this will work. __iomem annotation is there
> > > because we have architectures where you need to use special CPU
> > > instructions for iomem access.
> > >
> > > That won't go away just because we use memremap().
> >
> > That's true, but can we ever support those with TTM, given that we
> > allow user-space mmaping that transparently may change to an iomap?
> > Given that, and what's written here
> >
> >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F653585%2F&data=04%7C01%7Cchristian.koenig%40amd.com%7C1cdcfe9d20e740308c9e08d92019785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637576116034492654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e2BFGQJcElwVxrvHcnALDWscHN7ernLekGvXHqWBcwY%3D&reserved=0
> >
> >
> >
> > To me it sounds like if an architecture can't support memremap, we
> > can't support it with TTM either.
>
> That was also my argument, but this is unfortunately not true.
>
> We already have architectures where the __iomem approach is mandatory
> for kernel mappings, but work fine for userspace (don't ask me how
> that
> works, idk).
Ugh. :/
>
> That's the reason why we had to fix the UVD firmware upload in the
> kernel:
>
> commit ba0b2275a6781b2f3919d931d63329b5548f6d5f
> Author: Christian König <christian.koenig@amd.com>
> Date: Tue Aug 23 11:00:17 2016 +0200
>
> drm/amdgpu: use memcpy_to/fromio for UVD fw upload
>
> >
> > In any case for this particular patch, to avoid potential
> > regressions,
> > OK if I just add an ioremap() in case the memremap fails?
>
> Well because of the issues outlined above I would actually prefer if
> we
> can keep the __iomem annotation around.
Well, we'd do that. Since we use the dma_buf_map unconditionally.
So what would happen in the above case, would be:
- memremap would fail. (Otherwise I'd be terribly confused)
- we retry with ioremap and the dma-buf-map member is_iomem would thus
be set
- memcpy would do the right thing, based on is_iomem.
/Thomas
>
> Christian.
>
> >
> > /Thomas
> >
> >
> > >
> > > Christian.
> > >
> > > >
> > > > As for limited arch support for memremap cached, It looks like
> > > > we only
> > > > need to or in "backup" mapping modes in the memremap flags, and
> > > > we'd
> > > > mimic the previous behaviour.
> > > >
> > > > /Thomas
> > > >
> > > >
> > > > > > /Thomas
> > > > > >
> > > > > >
> > > >
> > >
>
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [Intel-gfx] [PATCH v3 06/12] drm/ttm: Add a generic TTM memcpy move for page-based iomem
@ 2021-05-26 10:57 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-26 10:57 UTC (permalink / raw)
To: Christian König, Matthew Auld
Cc: Intel Graphics Development, ML dri-devel
On Wed, 2021-05-26 at 12:45 +0200, Christian König wrote:
> Am 26.05.21 um 09:39 schrieb Thomas Hellström:
> > [SNIP]
> > > > I think the long term goal is to use memremap all over the
> > > > place, to
> > > > just not have to bother with the __iomem annotation. But to do
> > > > that io-
> > > > mapping.h needs to support memremap. But for now we need to be
> > > > strict
> > > > about __iomem unless we're in arch specific code. That's why
> > > > that
> > > > dma_buf_map thing was created, but TTM memcpy was never fully
> > > > adapted.
> > >
> > > I don't think that this will work. __iomem annotation is there
> > > because we have architectures where you need to use special CPU
> > > instructions for iomem access.
> > >
> > > That won't go away just because we use memremap().
> >
> > That's true, but can we ever support those with TTM, given that we
> > allow user-space mmaping that transparently may change to an iomap?
> > Given that, and what's written here
> >
> >
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F653585%2F&data=04%7C01%7Cchristian.koenig%40amd.com%7C1cdcfe9d20e740308c9e08d92019785b%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637576116034492654%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=e2BFGQJcElwVxrvHcnALDWscHN7ernLekGvXHqWBcwY%3D&reserved=0
> >
> >
> >
> > To me it sounds like if an architecture can't support memremap, we
> > can't support it with TTM either.
>
> That was also my argument, but this is unfortunately not true.
>
> We already have architectures where the __iomem approach is mandatory
> for kernel mappings, but work fine for userspace (don't ask me how
> that
> works, idk).
Ugh. :/
>
> That's the reason why we had to fix the UVD firmware upload in the
> kernel:
>
> commit ba0b2275a6781b2f3919d931d63329b5548f6d5f
> Author: Christian König <christian.koenig@amd.com>
> Date: Tue Aug 23 11:00:17 2016 +0200
>
> drm/amdgpu: use memcpy_to/fromio for UVD fw upload
>
> >
> > In any case for this particular patch, to avoid potential
> > regressions,
> > OK if I just add an ioremap() in case the memremap fails?
>
> Well because of the issues outlined above I would actually prefer if
> we
> can keep the __iomem annotation around.
Well, we'd do that. Since we use the dma_buf_map unconditionally.
So what would happen in the above case, would be:
- memremap would fail. (Otherwise I'd be terribly confused)
- we retry with ioremap and the dma-buf-map member is_iomem would thus
be set
- memcpy would do the right thing, based on is_iomem.
/Thomas
>
> Christian.
>
> >
> > /Thomas
> >
> >
> > >
> > > Christian.
> > >
> > > >
> > > > As for limited arch support for memremap cached, It looks like
> > > > we only
> > > > need to or in "backup" mapping modes in the memremap flags, and
> > > > we'd
> > > > mimic the previous behaviour.
> > > >
> > > > /Thomas
> > > >
> > > >
> > > > > > /Thomas
> > > > > >
> > > > > >
> > > >
> > >
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Memcpy from wc will be used as well by TTM memcpy.
Move it to core drm, and make the interface do the right thing
even on !X86.
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Dave Airlie <airlied@gmail.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/drm_drv.c | 2 +
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++++-----
drivers/gpu/drm/i915/Makefile | 1 -
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 ++-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 ++--
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 2 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
.../drm/i915/selftests/intel_memory_region.c | 7 ++-
include/drm/drm_memcpy.h | 47 ++++++++++++++
14 files changed, 121 insertions(+), 76 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 include/drm/drm_memcpy.h
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index a91cc7684904..f3ab8586c3d7 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
- drm_managed.o drm_vblank_work.o
+ drm_managed.o drm_vblank_work.o drm_memcpy.o \
drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 3d8d68a98b95..351cc2900cf1 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -40,6 +40,7 @@
#include <drm/drm_drv.h>
#include <drm/drm_file.h>
#include <drm/drm_managed.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_mode_object.h>
#include <drm/drm_print.h>
@@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
drm_connector_ida_init();
idr_init(&drm_minors_idr);
+ drm_memcpy_init_early();
ret = drm_sysfs_init();
if (ret < 0) {
diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
similarity index 70%
rename from drivers/gpu/drm/i915/i915_memcpy.c
rename to drivers/gpu/drm/drm_memcpy.c
index 1b021a4902de..740377749caa 100644
--- a/drivers/gpu/drm/i915/i915_memcpy.c
+++ b/drivers/gpu/drm/drm_memcpy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: MIT
/*
* Copyright © 2016 Intel Corporation
*
@@ -22,16 +23,12 @@
*
*/
+#ifdef CONFIG_X86
+#include <linux/dma-buf-map.h>
#include <linux/kernel.h>
#include <asm/fpu/api.h>
-#include "i915_memcpy.h"
-
-#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
-#define CI_BUG_ON(expr) BUG_ON(expr)
-#else
-#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
-#endif
+#include "drm/drm_memcpy.h"
static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
@@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
}
/**
- * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
+ * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * i915_memcpy_from_wc copies @len bytes from @src to @dst using
+ * drm_memcpy_from_wc copies @len bytes from @src to @dst using
* non-temporal instructions where available. Note that all arguments
* (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
* of 16.
*
* To test whether accelerated reads from WC are supported, use
- * i915_memcpy_from_wc(NULL, NULL, 0);
+ * drm_memcpy_from_wc(NULL, NULL, 0);
+ * This interface is intended for memremapped memory without the __iomem tag.
*
* Returns true if the copy was successful, false if the preconditions
* are not met.
*/
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
return false;
@@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
return false;
}
+EXPORT_SYMBOL(drm_memcpy_from_wc);
/**
- * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
+ * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
+ * struct dma_buf_map arguments.
+ * @dst: destination map
+ * @src: source map
+ * @len: how many bytes to copy
+ *
+ * This is identical to drm_memcpy_from_wc, except it's intended for
+ * potentially ioremapped memory rather than memremapped memory.
+ *
+ * Returns true if the copy was successful, false if the preconditions
+ * are not met.
+ */
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len)
+{
+ /* For X86 we can safely drop __iomem */
+ return drm_memcpy_from_wc(dst->is_iomem ?
+ (void __force *)dst->vaddr_iomem :
+ dst->vaddr,
+ src->is_iomem ?
+ (void const __force *)src->vaddr_iomem :
+ src->vaddr,
+ len);
+}
+EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
+
+/**
+ * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
+ * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
* @src to @dst using * non-temporal instructions where available, but
* accepts that its arguments may not be aligned, but are valid for the
* potential 16-byte read past the end.
+ *
+ * This interface is intended for mremapped memory without the __iomem tag.
*/
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
unsigned long addr;
- CI_BUG_ON(!i915_has_memcpy_from_wc());
-
addr = (unsigned long)src;
if (!IS_ALIGNED(addr, 16)) {
unsigned long x = min(ALIGN(addr, 16) - addr, len);
@@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
if (likely(len))
__memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
}
+EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
-void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
+void drm_memcpy_init_early(void)
{
/*
* Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
@@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
!boot_cpu_has(X86_FEATURE_HYPERVISOR))
static_branch_enable(&has_movntdqa);
}
+#endif
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index cb8823570996..998606b7f49f 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -61,7 +61,6 @@ i915-y += i915_drv.o \
# core library code
i915-y += \
dma_resv_utils.o \
- i915_memcpy.o \
i915_mm.o \
i915_sw_fence.o \
i915_sw_fence_work.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 297143511f99..77285e421fb8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -10,6 +10,7 @@
#include <linux/uaccess.h>
#include <drm/drm_syncobj.h>
+#include <drm/drm_memcpy.h>
#include "display/intel_frontbuffer.h"
@@ -28,7 +29,6 @@
#include "i915_sw_fence_work.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
-#include "i915_memcpy.h"
struct eb_vma {
struct i915_vma *vma;
@@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
!(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
pw->batch_map = ERR_PTR(-ENODEV);
- if (needs_clflush && i915_has_memcpy_from_wc())
+ if (needs_clflush && drm_has_memcpy_from_wc())
pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
if (IS_ERR(pw->batch_map)) {
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 5706d471692d..e9247afb0320 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -24,6 +24,8 @@
#include <linux/sched/mm.h>
+#include <drm/drm_memcpy.h>
+
#include "display/intel_frontbuffer.h"
#include "i915_drv.h"
#include "i915_gem_clflush.h"
@@ -31,7 +33,6 @@
#include "i915_gem_mman.h"
#include "i915_gem_object.h"
#include "i915_globals.h"
-#include "i915_memcpy.h"
#include "i915_trace.h"
static struct i915_global_object {
@@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
PAGE_SIZE);
src_ptr = src_map + offset_in_page(offset);
- if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
+ if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
memcpy_fromio(dst, src_ptr, size);
io_mapping_unmap(src_map);
diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
index 8784257ec808..92ada67a3835 100644
--- a/drivers/gpu/drm/i915/gt/selftest_reset.c
+++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
@@ -5,9 +5,10 @@
#include <linux/crc32.h>
+#include <drm/drm_memcpy.h>
+
#include "gem/i915_gem_stolen.h"
-#include "i915_memcpy.h"
#include "i915_selftest.h"
#include "intel_gpu_commands.h"
#include "selftests/igt_reset.h"
@@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
memset_io(s, STACK_MAGIC, PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
crc[page] = crc32_le(0, in, PAGE_SIZE);
@@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
x = crc32_le(0, in, PAGE_SIZE);
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
index c36d5eb5bbb9..f045e42be6ca 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
@@ -5,9 +5,10 @@
#include <linux/debugfs.h>
+#include <drm/drm_memcpy.h>
+
#include "gt/intel_gt.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
#include "intel_guc_log.h"
static void guc_log_capture_logs(struct intel_guc_log *log);
@@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
/* Just copy the newly written data */
if (read_offset > write_offset) {
- i915_memcpy_from_wc(dst_data, src_data, write_offset);
+ drm_memcpy_from_wc(dst_data, src_data, write_offset);
bytes_to_copy = buffer_size - read_offset;
} else {
bytes_to_copy = write_offset - read_offset;
}
- i915_memcpy_from_wc(dst_data + read_offset,
- src_data + read_offset, bytes_to_copy);
+ drm_memcpy_from_wc(dst_data + read_offset,
+ src_data + read_offset, bytes_to_copy);
src_data += buffer_size;
dst_data += buffer_size;
@@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
* it should be present on the chipsets supporting GuC based
* submisssions.
*/
- if (!i915_has_memcpy_from_wc()) {
+ if (!drm_has_memcpy_from_wc()) {
ret = -ENXIO;
goto out_unlock;
}
diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
index 5b4b2bd46e7c..98653f1a2b1d 100644
--- a/drivers/gpu/drm/i915/i915_cmd_parser.c
+++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
@@ -24,12 +24,12 @@
* Brad Volkin <bradley.d.volkin@intel.com>
*
*/
+#include <drm/drm_memcpy.h>
#include "gt/intel_engine.h"
#include "gt/intel_gpu_commands.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
/**
* DOC: batch buffer command parser
@@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
if (src) {
GEM_BUG_ON(!needs_clflush);
- i915_unaligned_memcpy_from_wc(dst, src + offset, length);
+ drm_unaligned_memcpy_from_wc(dst, src + offset, length);
} else {
struct scatterlist *sg;
void *ptr;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 122dd297b6af..0df9dd62c717 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -72,7 +72,6 @@
#include "i915_drv.h"
#include "i915_ioc32.h"
#include "i915_irq.h"
-#include "i915_memcpy.h"
#include "i915_perf.h"
#include "i915_query.h"
#include "i915_suspend.h"
@@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
mutex_init(&dev_priv->pps_mutex);
mutex_init(&dev_priv->hdcp_comp_mutex);
- i915_memcpy_init_early(dev_priv);
intel_runtime_pm_init_early(&dev_priv->runtime_pm);
ret = i915_workqueues_init(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 99ca242ec13b..ee11920fbea5 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -34,6 +34,7 @@
#include <linux/utsname.h>
#include <linux/zlib.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_print.h>
#include "display/intel_csr.h"
@@ -46,7 +47,6 @@
#include "i915_drv.h"
#include "i915_gpu_error.h"
-#include "i915_memcpy.h"
#include "i915_scatterlist.h"
#define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
@@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
}
c->tmp = NULL;
- if (i915_has_memcpy_from_wc())
+ if (drm_has_memcpy_from_wc())
c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
return true;
@@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
struct z_stream_s *zstream = &c->zstream;
zstream->next_in = src;
- if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
+ if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
zstream->next_in = c->tmp;
zstream->avail_in = PAGE_SIZE;
@@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
if (!ptr)
return -ENOMEM;
- if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
+ if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
memcpy(ptr, src, PAGE_SIZE);
dst->pages[dst->page_count++] = ptr;
cond_resched();
diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
deleted file mode 100644
index 3df063a3293b..000000000000
--- a/drivers/gpu/drm/i915/i915_memcpy.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_MEMCPY_H__
-#define __I915_MEMCPY_H__
-
-#include <linux/types.h>
-
-struct drm_i915_private;
-
-void i915_memcpy_init_early(struct drm_i915_private *i915);
-
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-
-/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
- * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
- * perform the operation. To check beforehand, pass in the parameters to
- * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
- * you only need to pass in the minor offsets, page-aligned pointers are
- * always valid.
- *
- * For just checking for SSE4.1, in the foreknowledge that the future use
- * will be correctly aligned, just use i915_has_memcpy_from_wc().
- */
-#define i915_can_memcpy_from_wc(dst, src, len) \
- i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
-
-#define i915_has_memcpy_from_wc() \
- i915_memcpy_from_wc(NULL, NULL, 0)
-
-#endif /* __I915_MEMCPY_H__ */
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index c85d516b85cd..6bb399e9be78 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -6,6 +6,8 @@
#include <linux/prime_numbers.h>
#include <linux/sort.h>
+#include <drm/drm_memcpy.h>
+
#include "../i915_selftest.h"
#include "mock_drm.h"
@@ -20,7 +22,6 @@
#include "gem/selftests/mock_context.h"
#include "gt/intel_engine_user.h"
#include "gt/intel_gt.h"
-#include "i915_memcpy.h"
#include "selftests/igt_flush_test.h"
#include "selftests/i915_random.h"
@@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
{
- i915_memcpy_from_wc(dst, src, size);
+ drm_memcpy_from_wc(dst, src, size);
}
static int _perf_memcpy(struct intel_memory_region *src_mr,
@@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
{
"memcpy_from_wc",
igt_memcpy_from_wc,
- !i915_has_memcpy_from_wc(),
+ !drm_has_memcpy_from_wc(),
},
};
struct drm_i915_gem_object *src, *dst;
diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
new file mode 100644
index 000000000000..b1d709753b4d
--- /dev/null
+++ b/include/drm/drm_memcpy.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __DRM_MEMCPY_H__
+#define __DRM_MEMCPY_H__
+
+#include <linux/types.h>
+
+struct dma_buf_map;
+
+#ifdef CONFIG_X86
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len);
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+
+/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
+ * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
+ * perform the operation. To check beforehand, pass in the parameters to
+ * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
+ * you only need to pass in the minor offsets, page-aligned pointers are
+ * always valid.
+ *
+ * For just checking for SSE4.1, in the foreknowledge that the future use
+ * will be correctly aligned, just use drm_has_memcpy_from_wc().
+ */
+#define drm_can_memcpy_from_wc(dst, src, len) \
+ drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
+
+#define drm_has_memcpy_from_wc() \
+ drm_memcpy_from_wc(NULL, NULL, 0)
+
+void drm_memcpy_init_early(void);
+
+#else
+
+#define drm_memcpy_from_wc(_dst, _src, _len) (false)
+#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
+#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
+#define drm_has_memcpy_from_wc() (false)
+#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
+#define drm_memcpy_init_early() do {} while (0)
+#endif /* CONFIG_X86 */
+#endif /* __DRM_MEMCPY_H__ */
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Memcpy from wc will be used as well by TTM memcpy.
Move it to core drm, and make the interface do the right thing
even on !X86.
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: Dave Airlie <airlied@gmail.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/Makefile | 2 +-
drivers/gpu/drm/drm_drv.c | 2 +
.../drm/{i915/i915_memcpy.c => drm_memcpy.c} | 63 ++++++++++++++-----
drivers/gpu/drm/i915/Makefile | 1 -
.../gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 +-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 5 +-
drivers/gpu/drm/i915/gt/selftest_reset.c | 7 ++-
drivers/gpu/drm/i915/gt/uc/intel_guc_log.c | 11 ++--
drivers/gpu/drm/i915/i915_cmd_parser.c | 4 +-
drivers/gpu/drm/i915/i915_drv.c | 2 -
drivers/gpu/drm/i915/i915_gpu_error.c | 8 +--
drivers/gpu/drm/i915/i915_memcpy.h | 34 ----------
.../drm/i915/selftests/intel_memory_region.c | 7 ++-
include/drm/drm_memcpy.h | 47 ++++++++++++++
14 files changed, 121 insertions(+), 76 deletions(-)
rename drivers/gpu/drm/{i915/i915_memcpy.c => drm_memcpy.c} (70%)
delete mode 100644 drivers/gpu/drm/i915/i915_memcpy.h
create mode 100644 include/drm/drm_memcpy.h
diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile
index a91cc7684904..f3ab8586c3d7 100644
--- a/drivers/gpu/drm/Makefile
+++ b/drivers/gpu/drm/Makefile
@@ -18,7 +18,7 @@ drm-y := drm_aperture.o drm_auth.o drm_cache.o \
drm_dumb_buffers.o drm_mode_config.o drm_vblank.o \
drm_syncobj.o drm_lease.o drm_writeback.o drm_client.o \
drm_client_modeset.o drm_atomic_uapi.o drm_hdcp.o \
- drm_managed.o drm_vblank_work.o
+ drm_managed.o drm_vblank_work.o drm_memcpy.o \
drm-$(CONFIG_DRM_LEGACY) += drm_agpsupport.o drm_bufs.o drm_context.o drm_dma.o \
drm_legacy_misc.o drm_lock.o drm_memory.o drm_scatter.o \
diff --git a/drivers/gpu/drm/drm_drv.c b/drivers/gpu/drm/drm_drv.c
index 3d8d68a98b95..351cc2900cf1 100644
--- a/drivers/gpu/drm/drm_drv.c
+++ b/drivers/gpu/drm/drm_drv.c
@@ -40,6 +40,7 @@
#include <drm/drm_drv.h>
#include <drm/drm_file.h>
#include <drm/drm_managed.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_mode_object.h>
#include <drm/drm_print.h>
@@ -1041,6 +1042,7 @@ static int __init drm_core_init(void)
drm_connector_ida_init();
idr_init(&drm_minors_idr);
+ drm_memcpy_init_early();
ret = drm_sysfs_init();
if (ret < 0) {
diff --git a/drivers/gpu/drm/i915/i915_memcpy.c b/drivers/gpu/drm/drm_memcpy.c
similarity index 70%
rename from drivers/gpu/drm/i915/i915_memcpy.c
rename to drivers/gpu/drm/drm_memcpy.c
index 1b021a4902de..740377749caa 100644
--- a/drivers/gpu/drm/i915/i915_memcpy.c
+++ b/drivers/gpu/drm/drm_memcpy.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: MIT
/*
* Copyright © 2016 Intel Corporation
*
@@ -22,16 +23,12 @@
*
*/
+#ifdef CONFIG_X86
+#include <linux/dma-buf-map.h>
#include <linux/kernel.h>
#include <asm/fpu/api.h>
-#include "i915_memcpy.h"
-
-#if IS_ENABLED(CONFIG_DRM_I915_DEBUG)
-#define CI_BUG_ON(expr) BUG_ON(expr)
-#else
-#define CI_BUG_ON(expr) BUILD_BUG_ON_INVALID(expr)
-#endif
+#include "drm/drm_memcpy.h"
static DEFINE_STATIC_KEY_FALSE(has_movntdqa);
@@ -94,23 +91,24 @@ static void __memcpy_ntdqu(void *dst, const void *src, unsigned long len)
}
/**
- * i915_memcpy_from_wc: perform an accelerated *aligned* read from WC
+ * drm_memcpy_from_wc: perform an accelerated *aligned* read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * i915_memcpy_from_wc copies @len bytes from @src to @dst using
+ * drm_memcpy_from_wc copies @len bytes from @src to @dst using
* non-temporal instructions where available. Note that all arguments
* (@src, @dst) must be aligned to 16 bytes and @len must be a multiple
* of 16.
*
* To test whether accelerated reads from WC are supported, use
- * i915_memcpy_from_wc(NULL, NULL, 0);
+ * drm_memcpy_from_wc(NULL, NULL, 0);
+ * This interface is intended for memremapped memory without the __iomem tag.
*
* Returns true if the copy was successful, false if the preconditions
* are not met.
*/
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
if (unlikely(((unsigned long)dst | (unsigned long)src | len) & 15))
return false;
@@ -123,24 +121,53 @@ bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len)
return false;
}
+EXPORT_SYMBOL(drm_memcpy_from_wc);
/**
- * i915_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
+ * drm_memcpy_from_wc_dbm: perform an accelerated *aligned* read from WC with
+ * struct dma_buf_map arguments.
+ * @dst: destination map
+ * @src: source map
+ * @len: how many bytes to copy
+ *
+ * This is identical to drm_memcpy_from_wc, except it's intended for
+ * potentially ioremapped memory rather than memremapped memory.
+ *
+ * Returns true if the copy was successful, false if the preconditions
+ * are not met.
+ */
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len)
+{
+ /* For X86 we can safely drop __iomem */
+ return drm_memcpy_from_wc(dst->is_iomem ?
+ (void __force *)dst->vaddr_iomem :
+ dst->vaddr,
+ src->is_iomem ?
+ (void const __force *)src->vaddr_iomem :
+ src->vaddr,
+ len);
+}
+EXPORT_SYMBOL(drm_memcpy_from_wc_dbm);
+
+/**
+ * drm_unaligned_memcpy_from_wc: perform a mostly accelerated read from WC
* @dst: destination pointer
* @src: source pointer
* @len: how many bytes to copy
*
- * Like i915_memcpy_from_wc(), the unaligned variant copies @len bytes from
+ * Like drm_memcpy_from_wc(), the unaligned variant copies @len bytes from
* @src to @dst using * non-temporal instructions where available, but
* accepts that its arguments may not be aligned, but are valid for the
* potential 16-byte read past the end.
+ *
+ * This interface is intended for mremapped memory without the __iomem tag.
*/
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len)
{
unsigned long addr;
- CI_BUG_ON(!i915_has_memcpy_from_wc());
-
addr = (unsigned long)src;
if (!IS_ALIGNED(addr, 16)) {
unsigned long x = min(ALIGN(addr, 16) - addr, len);
@@ -155,8 +182,9 @@ void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len
if (likely(len))
__memcpy_ntdqu(dst, src, DIV_ROUND_UP(len, 16));
}
+EXPORT_SYMBOL(drm_unaligned_memcpy_from_wc);
-void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
+void drm_memcpy_init_early(void)
{
/*
* Some hypervisors (e.g. KVM) don't support VEX-prefix instructions
@@ -166,3 +194,4 @@ void i915_memcpy_init_early(struct drm_i915_private *dev_priv)
!boot_cpu_has(X86_FEATURE_HYPERVISOR))
static_branch_enable(&has_movntdqa);
}
+#endif
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index cb8823570996..998606b7f49f 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -61,7 +61,6 @@ i915-y += i915_drv.o \
# core library code
i915-y += \
dma_resv_utils.o \
- i915_memcpy.o \
i915_mm.o \
i915_sw_fence.o \
i915_sw_fence_work.o \
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 297143511f99..77285e421fb8 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -10,6 +10,7 @@
#include <linux/uaccess.h>
#include <drm/drm_syncobj.h>
+#include <drm/drm_memcpy.h>
#include "display/intel_frontbuffer.h"
@@ -28,7 +29,6 @@
#include "i915_sw_fence_work.h"
#include "i915_trace.h"
#include "i915_user_extensions.h"
-#include "i915_memcpy.h"
struct eb_vma {
struct i915_vma *vma;
@@ -2503,7 +2503,7 @@ static int eb_parse_pipeline(struct i915_execbuffer *eb,
!(batch->cache_coherent & I915_BO_CACHE_COHERENT_FOR_READ);
pw->batch_map = ERR_PTR(-ENODEV);
- if (needs_clflush && i915_has_memcpy_from_wc())
+ if (needs_clflush && drm_has_memcpy_from_wc())
pw->batch_map = i915_gem_object_pin_map(batch, I915_MAP_WC);
if (IS_ERR(pw->batch_map)) {
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 5706d471692d..e9247afb0320 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -24,6 +24,8 @@
#include <linux/sched/mm.h>
+#include <drm/drm_memcpy.h>
+
#include "display/intel_frontbuffer.h"
#include "i915_drv.h"
#include "i915_gem_clflush.h"
@@ -31,7 +33,6 @@
#include "i915_gem_mman.h"
#include "i915_gem_object.h"
#include "i915_globals.h"
-#include "i915_memcpy.h"
#include "i915_trace.h"
static struct i915_global_object {
@@ -374,7 +375,7 @@ i915_gem_object_read_from_page_iomap(struct drm_i915_gem_object *obj, u64 offset
PAGE_SIZE);
src_ptr = src_map + offset_in_page(offset);
- if (!i915_memcpy_from_wc(dst, (void __force *)src_ptr, size))
+ if (!drm_memcpy_from_wc(dst, (void __force *)src_ptr, size))
memcpy_fromio(dst, src_ptr, size);
io_mapping_unmap(src_map);
diff --git a/drivers/gpu/drm/i915/gt/selftest_reset.c b/drivers/gpu/drm/i915/gt/selftest_reset.c
index 8784257ec808..92ada67a3835 100644
--- a/drivers/gpu/drm/i915/gt/selftest_reset.c
+++ b/drivers/gpu/drm/i915/gt/selftest_reset.c
@@ -5,9 +5,10 @@
#include <linux/crc32.h>
+#include <drm/drm_memcpy.h>
+
#include "gem/i915_gem_stolen.h"
-#include "i915_memcpy.h"
#include "i915_selftest.h"
#include "intel_gpu_commands.h"
#include "selftests/igt_reset.h"
@@ -99,7 +100,7 @@ __igt_reset_stolen(struct intel_gt *gt,
memset_io(s, STACK_MAGIC, PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
crc[page] = crc32_le(0, in, PAGE_SIZE);
@@ -135,7 +136,7 @@ __igt_reset_stolen(struct intel_gt *gt,
PAGE_SIZE);
in = (void __force *)s;
- if (i915_memcpy_from_wc(tmp, in, PAGE_SIZE))
+ if (drm_memcpy_from_wc(tmp, in, PAGE_SIZE))
in = tmp;
x = crc32_le(0, in, PAGE_SIZE);
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
index c36d5eb5bbb9..f045e42be6ca 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_log.c
@@ -5,9 +5,10 @@
#include <linux/debugfs.h>
+#include <drm/drm_memcpy.h>
+
#include "gt/intel_gt.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
#include "intel_guc_log.h"
static void guc_log_capture_logs(struct intel_guc_log *log);
@@ -295,13 +296,13 @@ static void guc_read_update_log_buffer(struct intel_guc_log *log)
/* Just copy the newly written data */
if (read_offset > write_offset) {
- i915_memcpy_from_wc(dst_data, src_data, write_offset);
+ drm_memcpy_from_wc(dst_data, src_data, write_offset);
bytes_to_copy = buffer_size - read_offset;
} else {
bytes_to_copy = write_offset - read_offset;
}
- i915_memcpy_from_wc(dst_data + read_offset,
- src_data + read_offset, bytes_to_copy);
+ drm_memcpy_from_wc(dst_data + read_offset,
+ src_data + read_offset, bytes_to_copy);
src_data += buffer_size;
dst_data += buffer_size;
@@ -569,7 +570,7 @@ int intel_guc_log_relay_open(struct intel_guc_log *log)
* it should be present on the chipsets supporting GuC based
* submisssions.
*/
- if (!i915_has_memcpy_from_wc()) {
+ if (!drm_has_memcpy_from_wc()) {
ret = -ENXIO;
goto out_unlock;
}
diff --git a/drivers/gpu/drm/i915/i915_cmd_parser.c b/drivers/gpu/drm/i915/i915_cmd_parser.c
index 5b4b2bd46e7c..98653f1a2b1d 100644
--- a/drivers/gpu/drm/i915/i915_cmd_parser.c
+++ b/drivers/gpu/drm/i915/i915_cmd_parser.c
@@ -24,12 +24,12 @@
* Brad Volkin <bradley.d.volkin@intel.com>
*
*/
+#include <drm/drm_memcpy.h>
#include "gt/intel_engine.h"
#include "gt/intel_gpu_commands.h"
#include "i915_drv.h"
-#include "i915_memcpy.h"
/**
* DOC: batch buffer command parser
@@ -1152,7 +1152,7 @@ static u32 *copy_batch(struct drm_i915_gem_object *dst_obj,
if (src) {
GEM_BUG_ON(!needs_clflush);
- i915_unaligned_memcpy_from_wc(dst, src + offset, length);
+ drm_unaligned_memcpy_from_wc(dst, src + offset, length);
} else {
struct scatterlist *sg;
void *ptr;
diff --git a/drivers/gpu/drm/i915/i915_drv.c b/drivers/gpu/drm/i915/i915_drv.c
index 122dd297b6af..0df9dd62c717 100644
--- a/drivers/gpu/drm/i915/i915_drv.c
+++ b/drivers/gpu/drm/i915/i915_drv.c
@@ -72,7 +72,6 @@
#include "i915_drv.h"
#include "i915_ioc32.h"
#include "i915_irq.h"
-#include "i915_memcpy.h"
#include "i915_perf.h"
#include "i915_query.h"
#include "i915_suspend.h"
@@ -325,7 +324,6 @@ static int i915_driver_early_probe(struct drm_i915_private *dev_priv)
mutex_init(&dev_priv->pps_mutex);
mutex_init(&dev_priv->hdcp_comp_mutex);
- i915_memcpy_init_early(dev_priv);
intel_runtime_pm_init_early(&dev_priv->runtime_pm);
ret = i915_workqueues_init(dev_priv);
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index 99ca242ec13b..ee11920fbea5 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -34,6 +34,7 @@
#include <linux/utsname.h>
#include <linux/zlib.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_print.h>
#include "display/intel_csr.h"
@@ -46,7 +47,6 @@
#include "i915_drv.h"
#include "i915_gpu_error.h"
-#include "i915_memcpy.h"
#include "i915_scatterlist.h"
#define ALLOW_FAIL (GFP_KERNEL | __GFP_RETRY_MAYFAIL | __GFP_NOWARN)
@@ -255,7 +255,7 @@ static bool compress_init(struct i915_vma_compress *c)
}
c->tmp = NULL;
- if (i915_has_memcpy_from_wc())
+ if (drm_has_memcpy_from_wc())
c->tmp = pool_alloc(&c->pool, ALLOW_FAIL);
return true;
@@ -295,7 +295,7 @@ static int compress_page(struct i915_vma_compress *c,
struct z_stream_s *zstream = &c->zstream;
zstream->next_in = src;
- if (wc && c->tmp && i915_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
+ if (wc && c->tmp && drm_memcpy_from_wc(c->tmp, src, PAGE_SIZE))
zstream->next_in = c->tmp;
zstream->avail_in = PAGE_SIZE;
@@ -395,7 +395,7 @@ static int compress_page(struct i915_vma_compress *c,
if (!ptr)
return -ENOMEM;
- if (!(wc && i915_memcpy_from_wc(ptr, src, PAGE_SIZE)))
+ if (!(wc && drm_memcpy_from_wc(ptr, src, PAGE_SIZE)))
memcpy(ptr, src, PAGE_SIZE);
dst->pages[dst->page_count++] = ptr;
cond_resched();
diff --git a/drivers/gpu/drm/i915/i915_memcpy.h b/drivers/gpu/drm/i915/i915_memcpy.h
deleted file mode 100644
index 3df063a3293b..000000000000
--- a/drivers/gpu/drm/i915/i915_memcpy.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: MIT */
-/*
- * Copyright © 2019 Intel Corporation
- */
-
-#ifndef __I915_MEMCPY_H__
-#define __I915_MEMCPY_H__
-
-#include <linux/types.h>
-
-struct drm_i915_private;
-
-void i915_memcpy_init_early(struct drm_i915_private *i915);
-
-bool i915_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-void i915_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
-
-/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
- * as well as SSE4.1 support. i915_memcpy_from_wc() will report if it cannot
- * perform the operation. To check beforehand, pass in the parameters to
- * to i915_can_memcpy_from_wc() - since we only care about the low 4 bits,
- * you only need to pass in the minor offsets, page-aligned pointers are
- * always valid.
- *
- * For just checking for SSE4.1, in the foreknowledge that the future use
- * will be correctly aligned, just use i915_has_memcpy_from_wc().
- */
-#define i915_can_memcpy_from_wc(dst, src, len) \
- i915_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
-
-#define i915_has_memcpy_from_wc() \
- i915_memcpy_from_wc(NULL, NULL, 0)
-
-#endif /* __I915_MEMCPY_H__ */
diff --git a/drivers/gpu/drm/i915/selftests/intel_memory_region.c b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
index c85d516b85cd..6bb399e9be78 100644
--- a/drivers/gpu/drm/i915/selftests/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/selftests/intel_memory_region.c
@@ -6,6 +6,8 @@
#include <linux/prime_numbers.h>
#include <linux/sort.h>
+#include <drm/drm_memcpy.h>
+
#include "../i915_selftest.h"
#include "mock_drm.h"
@@ -20,7 +22,6 @@
#include "gem/selftests/mock_context.h"
#include "gt/intel_engine_user.h"
#include "gt/intel_gt.h"
-#include "i915_memcpy.h"
#include "selftests/igt_flush_test.h"
#include "selftests/i915_random.h"
@@ -901,7 +902,7 @@ static inline void igt_memcpy(void *dst, const void *src, size_t size)
static inline void igt_memcpy_from_wc(void *dst, const void *src, size_t size)
{
- i915_memcpy_from_wc(dst, src, size);
+ drm_memcpy_from_wc(dst, src, size);
}
static int _perf_memcpy(struct intel_memory_region *src_mr,
@@ -925,7 +926,7 @@ static int _perf_memcpy(struct intel_memory_region *src_mr,
{
"memcpy_from_wc",
igt_memcpy_from_wc,
- !i915_has_memcpy_from_wc(),
+ !drm_has_memcpy_from_wc(),
},
};
struct drm_i915_gem_object *src, *dst;
diff --git a/include/drm/drm_memcpy.h b/include/drm/drm_memcpy.h
new file mode 100644
index 000000000000..b1d709753b4d
--- /dev/null
+++ b/include/drm/drm_memcpy.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2019 Intel Corporation
+ */
+
+#ifndef __DRM_MEMCPY_H__
+#define __DRM_MEMCPY_H__
+
+#include <linux/types.h>
+
+struct dma_buf_map;
+
+#ifdef CONFIG_X86
+bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
+ const struct dma_buf_map *src,
+ unsigned long len);
+void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
+
+/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
+ * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
+ * perform the operation. To check beforehand, pass in the parameters to
+ * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
+ * you only need to pass in the minor offsets, page-aligned pointers are
+ * always valid.
+ *
+ * For just checking for SSE4.1, in the foreknowledge that the future use
+ * will be correctly aligned, just use drm_has_memcpy_from_wc().
+ */
+#define drm_can_memcpy_from_wc(dst, src, len) \
+ drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
+
+#define drm_has_memcpy_from_wc() \
+ drm_memcpy_from_wc(NULL, NULL, 0)
+
+void drm_memcpy_init_early(void);
+
+#else
+
+#define drm_memcpy_from_wc(_dst, _src, _len) (false)
+#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
+#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
+#define drm_has_memcpy_from_wc() (false)
+#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
+#define drm_memcpy_init_early() do {} while (0)
+#endif /* CONFIG_X86 */
+#endif /* __DRM_MEMCPY_H__ */
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-24 16:45 ` Matthew Auld
-1 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-24 16:45 UTC (permalink / raw)
To: Thomas Hellström
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> Memcpy from wc will be used as well by TTM memcpy.
> Move it to core drm, and make the interface do the right thing
> even on !X86.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Dave Airlie <airlied@gmail.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
<snip>
> +
> +#ifdef CONFIG_X86
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len);
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +
> +/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
> + * perform the operation. To check beforehand, pass in the parameters to
> + * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
> + * you only need to pass in the minor offsets, page-aligned pointers are
> + * always valid.
> + *
> + * For just checking for SSE4.1, in the foreknowledge that the future use
> + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> + */
> +#define drm_can_memcpy_from_wc(dst, src, len) \
> + drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> +
> +#define drm_has_memcpy_from_wc() \
> + drm_memcpy_from_wc(NULL, NULL, 0)
> +
> +void drm_memcpy_init_early(void);
> +
> +#else
> +
> +#define drm_memcpy_from_wc(_dst, _src, _len) (false)
> +#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
> +#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
> +#define drm_has_memcpy_from_wc() (false)
Does the compiler not complain for these on !x86, if called without
checking the result of the statement? Maybe just make these function
stubs?
Otherwise,
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
> +#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
> +#define drm_memcpy_init_early() do {} while (0)
> +#endif /* CONFIG_X86 */
> +#endif /* __DRM_MEMCPY_H__ */
> --
> 2.31.1
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
@ 2021-05-24 16:45 ` Matthew Auld
0 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-24 16:45 UTC (permalink / raw)
To: Thomas Hellström
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> Memcpy from wc will be used as well by TTM memcpy.
> Move it to core drm, and make the interface do the right thing
> even on !X86.
>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: Dave Airlie <airlied@gmail.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
<snip>
> +
> +#ifdef CONFIG_X86
> +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> + const struct dma_buf_map *src,
> + unsigned long len);
> +void drm_unaligned_memcpy_from_wc(void *dst, const void *src, unsigned long len);
> +
> +/* The movntdqa instructions used for memcpy-from-wc require 16-byte alignment,
> + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if it cannot
> + * perform the operation. To check beforehand, pass in the parameters to
> + * drm_can_memcpy_from_wc() - since we only care about the low 4 bits,
> + * you only need to pass in the minor offsets, page-aligned pointers are
> + * always valid.
> + *
> + * For just checking for SSE4.1, in the foreknowledge that the future use
> + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> + */
> +#define drm_can_memcpy_from_wc(dst, src, len) \
> + drm_memcpy_from_wc((void *)((unsigned long)(dst) | (unsigned long)(src) | (len)), NULL, 0)
> +
> +#define drm_has_memcpy_from_wc() \
> + drm_memcpy_from_wc(NULL, NULL, 0)
> +
> +void drm_memcpy_init_early(void);
> +
> +#else
> +
> +#define drm_memcpy_from_wc(_dst, _src, _len) (false)
> +#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
> +#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
> +#define drm_has_memcpy_from_wc() (false)
Does the compiler not complain for these on !x86, if called without
checking the result of the statement? Maybe just make these function
stubs?
Otherwise,
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
> +#define drm_unaligned_memcpy_from_wc(_dst, _src, _len) WARN_ON(1)
> +#define drm_memcpy_init_early() do {} while (0)
> +#endif /* CONFIG_X86 */
> +#endif /* __DRM_MEMCPY_H__ */
> --
> 2.31.1
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
2021-05-24 16:45 ` Matthew Auld
@ 2021-05-24 18:12 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-24 18:12 UTC (permalink / raw)
To: Matthew Auld
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Mon, 2021-05-24 at 17:45 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > Memcpy from wc will be used as well by TTM memcpy.
> > Move it to core drm, and make the interface do the right thing
> > even on !X86.
> >
> > Cc: Christian König <christian.koenig@amd.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Cc: Dave Airlie <airlied@gmail.com>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
>
> <snip>
>
> > +
> > +#ifdef CONFIG_X86
> > +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long
> > len);
> > +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> > + const struct dma_buf_map *src,
> > + unsigned long len);
> > +void drm_unaligned_memcpy_from_wc(void *dst, const void *src,
> > unsigned long len);
> > +
> > +/* The movntdqa instructions used for memcpy-from-wc require 16-
> > byte alignment,
> > + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if
> > it cannot
> > + * perform the operation. To check beforehand, pass in the
> > parameters to
> > + * drm_can_memcpy_from_wc() - since we only care about the low 4
> > bits,
> > + * you only need to pass in the minor offsets, page-aligned
> > pointers are
> > + * always valid.
> > + *
> > + * For just checking for SSE4.1, in the foreknowledge that the
> > future use
> > + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> > + */
> > +#define drm_can_memcpy_from_wc(dst, src, len) \
> > + drm_memcpy_from_wc((void *)((unsigned long)(dst) |
> > (unsigned long)(src) | (len)), NULL, 0)
> > +
> > +#define drm_has_memcpy_from_wc() \
> > + drm_memcpy_from_wc(NULL, NULL, 0)
> > +
> > +void drm_memcpy_init_early(void);
> > +
> > +#else
> > +
> > +#define drm_memcpy_from_wc(_dst, _src, _len) (false)
> > +#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
> > +#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
> > +#define drm_has_memcpy_from_wc() (false)
>
> Does the compiler not complain for these on !x86, if called without
> checking the result of the statement? Maybe just make these function
> stubs?
>
> Otherwise,
> Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Hmm, you're right. I'll fix. Thanks for reviewing!
/Thomas
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [Intel-gfx] [PATCH v3 07/12] drm, drm/i915: Move the memcpy_from_wc functionality to core drm
@ 2021-05-24 18:12 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-24 18:12 UTC (permalink / raw)
To: Matthew Auld
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Mon, 2021-05-24 at 17:45 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > Memcpy from wc will be used as well by TTM memcpy.
> > Move it to core drm, and make the interface do the right thing
> > even on !X86.
> >
> > Cc: Christian König <christian.koenig@amd.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Cc: Dave Airlie <airlied@gmail.com>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
>
> <snip>
>
> > +
> > +#ifdef CONFIG_X86
> > +bool drm_memcpy_from_wc(void *dst, const void *src, unsigned long
> > len);
> > +bool drm_memcpy_from_wc_dbm(struct dma_buf_map *dst,
> > + const struct dma_buf_map *src,
> > + unsigned long len);
> > +void drm_unaligned_memcpy_from_wc(void *dst, const void *src,
> > unsigned long len);
> > +
> > +/* The movntdqa instructions used for memcpy-from-wc require 16-
> > byte alignment,
> > + * as well as SSE4.1 support. drm_memcpy_from_wc() will report if
> > it cannot
> > + * perform the operation. To check beforehand, pass in the
> > parameters to
> > + * drm_can_memcpy_from_wc() - since we only care about the low 4
> > bits,
> > + * you only need to pass in the minor offsets, page-aligned
> > pointers are
> > + * always valid.
> > + *
> > + * For just checking for SSE4.1, in the foreknowledge that the
> > future use
> > + * will be correctly aligned, just use drm_has_memcpy_from_wc().
> > + */
> > +#define drm_can_memcpy_from_wc(dst, src, len) \
> > + drm_memcpy_from_wc((void *)((unsigned long)(dst) |
> > (unsigned long)(src) | (len)), NULL, 0)
> > +
> > +#define drm_has_memcpy_from_wc() \
> > + drm_memcpy_from_wc(NULL, NULL, 0)
> > +
> > +void drm_memcpy_init_early(void);
> > +
> > +#else
> > +
> > +#define drm_memcpy_from_wc(_dst, _src, _len) (false)
> > +#define drm_memcpy_from_wc_dbm(_dst, _src, _len) (false)
> > +#define drm_can_memcpy_from_wc(_dst, _src, _len) (false)
> > +#define drm_has_memcpy_from_wc() (false)
>
> Does the compiler not complain for these on !x86, if called without
> checking the result of the statement? Maybe just make these function
> stubs?
>
> Otherwise,
> Reviewed-by: Matthew Auld <matthew.auld@intel.com>
Hmm, you're right. I'll fix. Thanks for reviewing!
/Thomas
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Use fast wc memcpy for reading out of wc memory for TTM bo moves.
Cc: Dave Airlie <airlied@gmail.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 912cbe8e60a2..4a7d3d672f9a 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -31,6 +31,7 @@
#include <drm/ttm/ttm_bo_driver.h>
#include <drm/ttm/ttm_placement.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_vma_manager.h>
#include <linux/dma-buf-map.h>
#include <linux/io.h>
@@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
struct ttm_tt *ttm = bo->ttm;
struct dma_buf_map src_map, dst_map;
+ bool wc_memcpy;
pgoff_t i;
/* Single TTM move. NOP */
@@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
return;
}
+ wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
+ drm_has_memcpy_from_wc());
+
for (i = 0; i < dst_mem->num_pages; ++i) {
dst_ops->map_local(dst_iter, &dst_map, i);
src_ops->map_local(src_iter, &src_map, i);
- if (!src_map.is_iomem && !dst_map.is_iomem) {
+ if (wc_memcpy) {
+ drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
+ } else if (!src_map.is_iomem && !dst_map.is_iomem) {
memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
} else if (!src_map.is_iomem) {
dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel
Cc: Thomas Hellström, Christian König, Daniel Vetter
Use fast wc memcpy for reading out of wc memory for TTM bo moves.
Cc: Dave Airlie <airlied@gmail.com>
Cc: Christian König <christian.koenig@amd.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 912cbe8e60a2..4a7d3d672f9a 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -31,6 +31,7 @@
#include <drm/ttm/ttm_bo_driver.h>
#include <drm/ttm/ttm_placement.h>
+#include <drm/drm_memcpy.h>
#include <drm/drm_vma_manager.h>
#include <linux/dma-buf-map.h>
#include <linux/io.h>
@@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
struct ttm_tt *ttm = bo->ttm;
struct dma_buf_map src_map, dst_map;
+ bool wc_memcpy;
pgoff_t i;
/* Single TTM move. NOP */
@@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
return;
}
+ wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
+ drm_has_memcpy_from_wc());
+
for (i = 0; i < dst_mem->num_pages; ++i) {
dst_ops->map_local(dst_iter, &dst_map, i);
src_ops->map_local(src_iter, &src_map, i);
- if (!src_map.is_iomem && !dst_map.is_iomem) {
+ if (wc_memcpy) {
+ drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
+ } else if (!src_map.is_iomem && !dst_map.is_iomem) {
memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
} else if (!src_map.is_iomem) {
dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread* Re: [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-24 18:16 ` Matthew Auld
-1 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-24 18:16 UTC (permalink / raw)
To: Thomas Hellström
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> Use fast wc memcpy for reading out of wc memory for TTM bo moves.
>
> Cc: Dave Airlie <airlied@gmail.com>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 912cbe8e60a2..4a7d3d672f9a 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -31,6 +31,7 @@
>
> #include <drm/ttm/ttm_bo_driver.h>
> #include <drm/ttm/ttm_placement.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_vma_manager.h>
> #include <linux/dma-buf-map.h>
> #include <linux/io.h>
> @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> struct ttm_tt *ttm = bo->ttm;
> struct dma_buf_map src_map, dst_map;
> + bool wc_memcpy;
> pgoff_t i;
>
> /* Single TTM move. NOP */
> @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> return;
> }
>
> + wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
Why do we only consider the caching value for the maps_tt case? Or am
I misreading this?
> + drm_has_memcpy_from_wc());
> +
> for (i = 0; i < dst_mem->num_pages; ++i) {
> dst_ops->map_local(dst_iter, &dst_map, i);
> src_ops->map_local(src_iter, &src_map, i);
>
> - if (!src_map.is_iomem && !dst_map.is_iomem) {
> + if (wc_memcpy) {
> + drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
Do we need to check the return value here? memcpy_from_wc expects
certain address alignment, or is that always guaranteed here? Maybe
throw a warning just for paranoia?
> + } else if (!src_map.is_iomem && !dst_map.is_iomem) {
> memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> } else if (!src_map.is_iomem) {
> dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> --
> 2.31.1
>
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
@ 2021-05-24 18:16 ` Matthew Auld
0 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-24 18:16 UTC (permalink / raw)
To: Thomas Hellström
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> Use fast wc memcpy for reading out of wc memory for TTM bo moves.
>
> Cc: Dave Airlie <airlied@gmail.com>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 912cbe8e60a2..4a7d3d672f9a 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -31,6 +31,7 @@
>
> #include <drm/ttm/ttm_bo_driver.h>
> #include <drm/ttm/ttm_placement.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_vma_manager.h>
> #include <linux/dma-buf-map.h>
> #include <linux/io.h>
> @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> struct ttm_tt *ttm = bo->ttm;
> struct dma_buf_map src_map, dst_map;
> + bool wc_memcpy;
> pgoff_t i;
>
> /* Single TTM move. NOP */
> @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> return;
> }
>
> + wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
Why do we only consider the caching value for the maps_tt case? Or am
I misreading this?
> + drm_has_memcpy_from_wc());
> +
> for (i = 0; i < dst_mem->num_pages; ++i) {
> dst_ops->map_local(dst_iter, &dst_map, i);
> src_ops->map_local(src_iter, &src_map, i);
>
> - if (!src_map.is_iomem && !dst_map.is_iomem) {
> + if (wc_memcpy) {
> + drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
Do we need to check the return value here? memcpy_from_wc expects
certain address alignment, or is that always guaranteed here? Maybe
throw a warning just for paranoia?
> + } else if (!src_map.is_iomem && !dst_map.is_iomem) {
> memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> } else if (!src_map.is_iomem) {
> dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
> --
> 2.31.1
>
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
2021-05-24 18:16 ` [Intel-gfx] " Matthew Auld
@ 2021-05-24 18:47 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-24 18:47 UTC (permalink / raw)
To: Matthew Auld
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Mon, 2021-05-24 at 19:16 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > Use fast wc memcpy for reading out of wc memory for TTM bo moves.
> >
> > Cc: Dave Airlie <airlied@gmail.com>
> > Cc: Christian König <christian.koenig@amd.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
> > drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> > 1 file changed, 8 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 912cbe8e60a2..4a7d3d672f9a 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -31,6 +31,7 @@
> >
> > #include <drm/ttm/ttm_bo_driver.h>
> > #include <drm/ttm/ttm_placement.h>
> > +#include <drm/drm_memcpy.h>
> > #include <drm/drm_vma_manager.h>
> > #include <linux/dma-buf-map.h>
> > #include <linux/io.h>
> > @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object
> > *bo,
> > const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> > struct ttm_tt *ttm = bo->ttm;
> > struct dma_buf_map src_map, dst_map;
> > + bool wc_memcpy;
> > pgoff_t i;
> >
> > /* Single TTM move. NOP */
> > @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object
> > *bo,
> > return;
> > }
> >
> > + wc_memcpy = ((!src_ops->maps_tt || ttm->caching !=
> > ttm_cached) &&
>
> Why do we only consider the caching value for the maps_tt case? Or am
> I misreading this?
Hmm, you're right we should probably check also the iomem caching or
ignore the tt caching. Sometimes these special memcpys tend to be less
cpu cache thrashing and should be used wherever possible, but I guess
we should start out with only using it when source is WC.
>
> > + drm_has_memcpy_from_wc());
> > +
> > for (i = 0; i < dst_mem->num_pages; ++i) {
> > dst_ops->map_local(dst_iter, &dst_map, i);
> > src_ops->map_local(src_iter, &src_map, i);
> >
> > - if (!src_map.is_iomem && !dst_map.is_iomem) {
> > + if (wc_memcpy) {
> > + drm_memcpy_from_wc_dbm(&dst_map, &src_map,
> > PAGE_SIZE);
>
> Do we need to check the return value here? memcpy_from_wc expects
> certain address alignment, or is that always guaranteed here? Maybe
> throw a warning just for paranoia?
It should always be PAGE_SIZE aligned. But I guess it doesn't hurt to
do
if (wc_memcpy && drm_memcpy_from_wc_dbm())
;
>
> > + } else if (!src_map.is_iomem && !dst_map.is_iomem)
> > {
> > memcpy(dst_map.vaddr, src_map.vaddr,
> > PAGE_SIZE);
> > } else if (!src_map.is_iomem) {
> > dma_buf_map_memcpy_to(&dst_map,
> > src_map.vaddr,
> > --
> > 2.31.1
> >
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
@ 2021-05-24 18:47 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-24 18:47 UTC (permalink / raw)
To: Matthew Auld
Cc: Daniel Vetter, Intel Graphics Development, Christian König,
ML dri-devel
On Mon, 2021-05-24 at 19:16 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > Use fast wc memcpy for reading out of wc memory for TTM bo moves.
> >
> > Cc: Dave Airlie <airlied@gmail.com>
> > Cc: Christian König <christian.koenig@amd.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
> > drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> > 1 file changed, 8 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 912cbe8e60a2..4a7d3d672f9a 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -31,6 +31,7 @@
> >
> > #include <drm/ttm/ttm_bo_driver.h>
> > #include <drm/ttm/ttm_placement.h>
> > +#include <drm/drm_memcpy.h>
> > #include <drm/drm_vma_manager.h>
> > #include <linux/dma-buf-map.h>
> > #include <linux/io.h>
> > @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object
> > *bo,
> > const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> > struct ttm_tt *ttm = bo->ttm;
> > struct dma_buf_map src_map, dst_map;
> > + bool wc_memcpy;
> > pgoff_t i;
> >
> > /* Single TTM move. NOP */
> > @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object
> > *bo,
> > return;
> > }
> >
> > + wc_memcpy = ((!src_ops->maps_tt || ttm->caching !=
> > ttm_cached) &&
>
> Why do we only consider the caching value for the maps_tt case? Or am
> I misreading this?
Hmm, you're right we should probably check also the iomem caching or
ignore the tt caching. Sometimes these special memcpys tend to be less
cpu cache thrashing and should be used wherever possible, but I guess
we should start out with only using it when source is WC.
>
> > + drm_has_memcpy_from_wc());
> > +
> > for (i = 0; i < dst_mem->num_pages; ++i) {
> > dst_ops->map_local(dst_iter, &dst_map, i);
> > src_ops->map_local(src_iter, &src_map, i);
> >
> > - if (!src_map.is_iomem && !dst_map.is_iomem) {
> > + if (wc_memcpy) {
> > + drm_memcpy_from_wc_dbm(&dst_map, &src_map,
> > PAGE_SIZE);
>
> Do we need to check the return value here? memcpy_from_wc expects
> certain address alignment, or is that always guaranteed here? Maybe
> throw a warning just for paranoia?
It should always be PAGE_SIZE aligned. But I guess it doesn't hurt to
do
if (wc_memcpy && drm_memcpy_from_wc_dbm())
;
>
> > + } else if (!src_map.is_iomem && !dst_map.is_iomem)
> > {
> > memcpy(dst_map.vaddr, src_map.vaddr,
> > PAGE_SIZE);
> > } else if (!src_map.is_iomem) {
> > dma_buf_map_memcpy_to(&dst_map,
> > src_map.vaddr,
> > --
> > 2.31.1
> >
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-26 12:48 ` Christian König
-1 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-26 12:48 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel; +Cc: Daniel Vetter
Am 21.05.21 um 17:32 schrieb Thomas Hellström:
> Use fast wc memcpy for reading out of wc memory for TTM bo moves.
>
> Cc: Dave Airlie <airlied@gmail.com>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 912cbe8e60a2..4a7d3d672f9a 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -31,6 +31,7 @@
>
> #include <drm/ttm/ttm_bo_driver.h>
> #include <drm/ttm/ttm_placement.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_vma_manager.h>
> #include <linux/dma-buf-map.h>
> #include <linux/io.h>
> @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> struct ttm_tt *ttm = bo->ttm;
> struct dma_buf_map src_map, dst_map;
> + bool wc_memcpy;
> pgoff_t i;
>
> /* Single TTM move. NOP */
> @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> return;
> }
>
> + wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
> + drm_has_memcpy_from_wc());
> +
> for (i = 0; i < dst_mem->num_pages; ++i) {
> dst_ops->map_local(dst_iter, &dst_map, i);
> src_ops->map_local(src_iter, &src_map, i);
>
> - if (!src_map.is_iomem && !dst_map.is_iomem) {
> + if (wc_memcpy) {
> + drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
> + } else if (!src_map.is_iomem && !dst_map.is_iomem) {
> memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> } else if (!src_map.is_iomem) {
> dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 08/12] drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
@ 2021-05-26 12:48 ` Christian König
0 siblings, 0 replies; 60+ messages in thread
From: Christian König @ 2021-05-26 12:48 UTC (permalink / raw)
To: Thomas Hellström, intel-gfx, dri-devel; +Cc: Daniel Vetter
Am 21.05.21 um 17:32 schrieb Thomas Hellström:
> Use fast wc memcpy for reading out of wc memory for TTM bo moves.
>
> Cc: Dave Airlie <airlied@gmail.com>
> Cc: Christian König <christian.koenig@amd.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo_util.c | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 912cbe8e60a2..4a7d3d672f9a 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -31,6 +31,7 @@
>
> #include <drm/ttm/ttm_bo_driver.h>
> #include <drm/ttm/ttm_placement.h>
> +#include <drm/drm_memcpy.h>
> #include <drm/drm_vma_manager.h>
> #include <linux/dma-buf-map.h>
> #include <linux/io.h>
> @@ -91,6 +92,7 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> const struct ttm_kmap_iter_ops *src_ops = src_iter->ops;
> struct ttm_tt *ttm = bo->ttm;
> struct dma_buf_map src_map, dst_map;
> + bool wc_memcpy;
> pgoff_t i;
>
> /* Single TTM move. NOP */
> @@ -114,11 +116,16 @@ void ttm_move_memcpy(struct ttm_buffer_object *bo,
> return;
> }
>
> + wc_memcpy = ((!src_ops->maps_tt || ttm->caching != ttm_cached) &&
> + drm_has_memcpy_from_wc());
> +
> for (i = 0; i < dst_mem->num_pages; ++i) {
> dst_ops->map_local(dst_iter, &dst_map, i);
> src_ops->map_local(src_iter, &src_map, i);
>
> - if (!src_map.is_iomem && !dst_map.is_iomem) {
> + if (wc_memcpy) {
> + drm_memcpy_from_wc_dbm(&dst_map, &src_map, PAGE_SIZE);
> + } else if (!src_map.is_iomem && !dst_map.is_iomem) {
> memcpy(dst_map.vaddr, src_map.vaddr, PAGE_SIZE);
> } else if (!src_map.is_iomem) {
> dma_buf_map_memcpy_to(&dst_map, src_map.vaddr,
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
create a ghost object and push it out to delayed destroy.
Fix this by adding a path for idle, and document the function.
Also avoid having the bo end up in a bad state vulnerable to user-space
triggered kernel BUGs if the call to ttm_tt_create() fails.
Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
drivers/gpu/drm/ttm/ttm_bo_util.c | 52 ++++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
include/drm/ttm/ttm_tt.h | 10 ++++++
4 files changed, 73 insertions(+), 14 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index ca1b098b6a56..a8fa3375b8aa 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bdev->funcs->evict_flags(bo, &placement);
if (!placement.num_placement && !placement.num_busy_placement) {
- ttm_bo_wait(bo, false, false);
+ ret = ttm_bo_wait(bo, true, false);
+ if (ret)
+ return ret;
- ttm_bo_cleanup_memtype_use(bo);
- return ttm_tt_create(bo, false);
+ /*
+ * Since we've already synced, this frees backing store
+ * immediately.
+ */
+ return ttm_bo_pipeline_gutting(bo);
}
ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
@@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
/*
* Remove the backing store if no placement is given.
*/
- if (!placement->num_placement && !placement->num_busy_placement) {
- ret = ttm_bo_pipeline_gutting(bo);
- if (ret)
- return ret;
-
- return ttm_tt_create(bo, false);
- }
+ if (!placement->num_placement && !placement->num_busy_placement)
+ return ttm_bo_pipeline_gutting(bo);
/*
* Check whether we need to move buffer.
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 4a7d3d672f9a..7fa9b3a852eb 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
}
EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
+/**
+ * ttm_bo_pipeline_gutting - purge the contents of a bo
+ * @bo: The buffer object
+ *
+ * Purge the contents of a bo, async if the bo is not idle.
+ * After a successful call, the bo is left unpopulated in
+ * system placement. The function may wait uninterruptible
+ * for idle on OOM.
+ *
+ * Return: 0 if successful, negative error code on failure.
+ */
int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
{
static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
struct ttm_buffer_object *ghost;
+ struct ttm_tt *ttm;
int ret;
- ret = ttm_buffer_object_transfer(bo, &ghost);
+ /* If already idle, no need for ghost object dance. */
+ ret = ttm_bo_wait(bo, false, true);
+ if (ret != -EBUSY) {
+ if (!bo->ttm) {
+ ret = ttm_tt_create(bo, true);
+ if (ret)
+ return ret;
+ } else {
+ ttm_tt_unpopulate(bo->bdev, bo->ttm);
+ if (bo->type == ttm_bo_type_device)
+ ttm_tt_mark_for_clear(bo->ttm);
+ }
+ ttm_resource_free(bo, &bo->mem);
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
+
+ return 0;
+ }
+
+ /*
+ * We need an unpopulated ttm_tt after giving our current one,
+ * if any, to the ghost object. And we can't afford to fail
+ * creating one *after* the operation.
+ */
+
+ ttm = bo->ttm;
+ bo->ttm = NULL;
+ ret = ttm_tt_create(bo, true);
+ swap(bo->ttm, ttm);
if (ret)
return ret;
+ ret = ttm_buffer_object_transfer(bo, &ghost);
+ if (ret) {
+ ttm_tt_destroy(bo->bdev, ttm);
+ return ret;
+ }
+
ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
/* Last resort, wait for the BO to be idle when we are OOM */
if (ret)
ttm_bo_wait(bo, false, false);
- ttm_resource_alloc(bo, &sys_mem, &bo->mem);
- bo->ttm = NULL;
-
dma_resv_unlock(&ghost->base._resv);
ttm_bo_put(ghost);
+ bo->ttm = ttm;
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
return 0;
}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 0e41227116b1..913b330a234b 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
}
EXPORT_SYMBOL(ttm_tt_destroy_common);
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
+{
+ ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+}
+
void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
{
bdev->funcs->ttm_tt_destroy(bdev, ttm);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 3102059db726..daa9c4cf48bb 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
*/
void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
+/**
+ * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
+ *
+ * @ttm: Pointer to the ttm_tt structure
+ *
+ * Marks pages for clearing so that the next time the page vector is
+ * populated, the pages will be cleared.
+ */
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
+
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
create a ghost object and push it out to delayed destroy.
Fix this by adding a path for idle, and document the function.
Also avoid having the bo end up in a bad state vulnerable to user-space
triggered kernel BUGs if the call to ttm_tt_create() fails.
Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
drivers/gpu/drm/ttm/ttm_bo_util.c | 52 ++++++++++++++++++++++++++++---
drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
include/drm/ttm/ttm_tt.h | 10 ++++++
4 files changed, 73 insertions(+), 14 deletions(-)
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index ca1b098b6a56..a8fa3375b8aa 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bdev->funcs->evict_flags(bo, &placement);
if (!placement.num_placement && !placement.num_busy_placement) {
- ttm_bo_wait(bo, false, false);
+ ret = ttm_bo_wait(bo, true, false);
+ if (ret)
+ return ret;
- ttm_bo_cleanup_memtype_use(bo);
- return ttm_tt_create(bo, false);
+ /*
+ * Since we've already synced, this frees backing store
+ * immediately.
+ */
+ return ttm_bo_pipeline_gutting(bo);
}
ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
@@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
/*
* Remove the backing store if no placement is given.
*/
- if (!placement->num_placement && !placement->num_busy_placement) {
- ret = ttm_bo_pipeline_gutting(bo);
- if (ret)
- return ret;
-
- return ttm_tt_create(bo, false);
- }
+ if (!placement->num_placement && !placement->num_busy_placement)
+ return ttm_bo_pipeline_gutting(bo);
/*
* Check whether we need to move buffer.
diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
index 4a7d3d672f9a..7fa9b3a852eb 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_util.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
@@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
}
EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
+/**
+ * ttm_bo_pipeline_gutting - purge the contents of a bo
+ * @bo: The buffer object
+ *
+ * Purge the contents of a bo, async if the bo is not idle.
+ * After a successful call, the bo is left unpopulated in
+ * system placement. The function may wait uninterruptible
+ * for idle on OOM.
+ *
+ * Return: 0 if successful, negative error code on failure.
+ */
int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
{
static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
struct ttm_buffer_object *ghost;
+ struct ttm_tt *ttm;
int ret;
- ret = ttm_buffer_object_transfer(bo, &ghost);
+ /* If already idle, no need for ghost object dance. */
+ ret = ttm_bo_wait(bo, false, true);
+ if (ret != -EBUSY) {
+ if (!bo->ttm) {
+ ret = ttm_tt_create(bo, true);
+ if (ret)
+ return ret;
+ } else {
+ ttm_tt_unpopulate(bo->bdev, bo->ttm);
+ if (bo->type == ttm_bo_type_device)
+ ttm_tt_mark_for_clear(bo->ttm);
+ }
+ ttm_resource_free(bo, &bo->mem);
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
+
+ return 0;
+ }
+
+ /*
+ * We need an unpopulated ttm_tt after giving our current one,
+ * if any, to the ghost object. And we can't afford to fail
+ * creating one *after* the operation.
+ */
+
+ ttm = bo->ttm;
+ bo->ttm = NULL;
+ ret = ttm_tt_create(bo, true);
+ swap(bo->ttm, ttm);
if (ret)
return ret;
+ ret = ttm_buffer_object_transfer(bo, &ghost);
+ if (ret) {
+ ttm_tt_destroy(bo->bdev, ttm);
+ return ret;
+ }
+
ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
/* Last resort, wait for the BO to be idle when we are OOM */
if (ret)
ttm_bo_wait(bo, false, false);
- ttm_resource_alloc(bo, &sys_mem, &bo->mem);
- bo->ttm = NULL;
-
dma_resv_unlock(&ghost->base._resv);
ttm_bo_put(ghost);
+ bo->ttm = ttm;
+ ttm_resource_alloc(bo, &sys_mem, &bo->mem);
return 0;
}
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 0e41227116b1..913b330a234b 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
}
EXPORT_SYMBOL(ttm_tt_destroy_common);
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
+{
+ ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+}
+
void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
{
bdev->funcs->ttm_tt_destroy(bdev, ttm);
diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
index 3102059db726..daa9c4cf48bb 100644
--- a/include/drm/ttm/ttm_tt.h
+++ b/include/drm/ttm/ttm_tt.h
@@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
*/
void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
+/**
+ * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
+ *
+ * @ttm: Pointer to the ttm_tt structure
+ *
+ * Marks pages for clearing so that the next time the page vector is
+ * populated, the pages will be cleared.
+ */
+void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
+
void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-25 11:00 ` Matthew Auld
-1 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 11:00 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
> create a ghost object and push it out to delayed destroy.
> Fix this by adding a path for idle, and document the function.
>
> Also avoid having the bo end up in a bad state vulnerable to user-space
> triggered kernel BUGs if the call to ttm_tt_create() fails.
>
> Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
> drivers/gpu/drm/ttm/ttm_bo_util.c | 52 ++++++++++++++++++++++++++++---
> drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> include/drm/ttm/ttm_tt.h | 10 ++++++
> 4 files changed, 73 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index ca1b098b6a56..a8fa3375b8aa 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bdev->funcs->evict_flags(bo, &placement);
>
> if (!placement.num_placement && !placement.num_busy_placement) {
> - ttm_bo_wait(bo, false, false);
> + ret = ttm_bo_wait(bo, true, false);
> + if (ret)
> + return ret;
>
> - ttm_bo_cleanup_memtype_use(bo);
> - return ttm_tt_create(bo, false);
> + /*
> + * Since we've already synced, this frees backing store
> + * immediately.
> + */
> + return ttm_bo_pipeline_gutting(bo);
> }
>
> ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> @@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
> /*
> * Remove the backing store if no placement is given.
> */
> - if (!placement->num_placement && !placement->num_busy_placement) {
> - ret = ttm_bo_pipeline_gutting(bo);
> - if (ret)
> - return ret;
> -
> - return ttm_tt_create(bo, false);
> - }
> + if (!placement->num_placement && !placement->num_busy_placement)
> + return ttm_bo_pipeline_gutting(bo);
>
> /*
> * Check whether we need to move buffer.
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 4a7d3d672f9a..7fa9b3a852eb 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
> }
> EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
>
> +/**
> + * ttm_bo_pipeline_gutting - purge the contents of a bo
> + * @bo: The buffer object
> + *
> + * Purge the contents of a bo, async if the bo is not idle.
> + * After a successful call, the bo is left unpopulated in
> + * system placement. The function may wait uninterruptible
> + * for idle on OOM.
> + *
> + * Return: 0 if successful, negative error code on failure.
> + */
> int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> {
> static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
> struct ttm_buffer_object *ghost;
> + struct ttm_tt *ttm;
> int ret;
>
> - ret = ttm_buffer_object_transfer(bo, &ghost);
> + /* If already idle, no need for ghost object dance. */
> + ret = ttm_bo_wait(bo, false, true);
> + if (ret != -EBUSY) {
> + if (!bo->ttm) {
> + ret = ttm_tt_create(bo, true);
Why do we now unconditionally add clearing? Below also.
> + if (ret)
> + return ret;
> + } else {
> + ttm_tt_unpopulate(bo->bdev, bo->ttm);
> + if (bo->type == ttm_bo_type_device)
> + ttm_tt_mark_for_clear(bo->ttm);
> + }
> + ttm_resource_free(bo, &bo->mem);
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> +
> + return 0;
> + }
> +
> + /*
> + * We need an unpopulated ttm_tt after giving our current one,
> + * if any, to the ghost object. And we can't afford to fail
> + * creating one *after* the operation.
> + */
> +
> + ttm = bo->ttm;
> + bo->ttm = NULL;
> + ret = ttm_tt_create(bo, true);
> + swap(bo->ttm, ttm);
> if (ret)
> return ret;
>
> + ret = ttm_buffer_object_transfer(bo, &ghost);
> + if (ret) {
> + ttm_tt_destroy(bo->bdev, ttm);
> + return ret;
> + }
> +
> ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
> /* Last resort, wait for the BO to be idle when we are OOM */
> if (ret)
> ttm_bo_wait(bo, false, false);
>
> - ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> - bo->ttm = NULL;
> -
> dma_resv_unlock(&ghost->base._resv);
> ttm_bo_put(ghost);
> + bo->ttm = ttm;
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
>
> return 0;
> }
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 0e41227116b1..913b330a234b 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
> }
> EXPORT_SYMBOL(ttm_tt_destroy_common);
>
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
> +{
> + ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
> +}
> +
> void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
> {
> bdev->funcs->ttm_tt_destroy(bdev, ttm);
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index 3102059db726..daa9c4cf48bb 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
> */
> void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
>
> +/**
> + * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> + *
> + * @ttm: Pointer to the ttm_tt structure
> + *
> + * Marks pages for clearing so that the next time the page vector is
> + * populated, the pages will be cleared.
> + */
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
> +
> void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>
> struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
> --
> 2.31.1
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
@ 2021-05-25 11:00 ` Matthew Auld
0 siblings, 0 replies; 60+ messages in thread
From: Matthew Auld @ 2021-05-25 11:00 UTC (permalink / raw)
To: Thomas Hellström
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Fri, 21 May 2021 at 16:33, Thomas Hellström
<thomas.hellstrom@linux.intel.com> wrote:
>
> If the bo is idle when calling ttm_bo_pipeline_gutting(), we unnecessarily
> create a ghost object and push it out to delayed destroy.
> Fix this by adding a path for idle, and document the function.
>
> Also avoid having the bo end up in a bad state vulnerable to user-space
> triggered kernel BUGs if the call to ttm_tt_create() fails.
>
> Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
>
> Cc: Christian König <christian.koenig@amd.com>
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> ---
> drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
> drivers/gpu/drm/ttm/ttm_bo_util.c | 52 ++++++++++++++++++++++++++++---
> drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> include/drm/ttm/ttm_tt.h | 10 ++++++
> 4 files changed, 73 insertions(+), 14 deletions(-)
>
> diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
> index ca1b098b6a56..a8fa3375b8aa 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
> bdev->funcs->evict_flags(bo, &placement);
>
> if (!placement.num_placement && !placement.num_busy_placement) {
> - ttm_bo_wait(bo, false, false);
> + ret = ttm_bo_wait(bo, true, false);
> + if (ret)
> + return ret;
>
> - ttm_bo_cleanup_memtype_use(bo);
> - return ttm_tt_create(bo, false);
> + /*
> + * Since we've already synced, this frees backing store
> + * immediately.
> + */
> + return ttm_bo_pipeline_gutting(bo);
> }
>
> ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> @@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object *bo,
> /*
> * Remove the backing store if no placement is given.
> */
> - if (!placement->num_placement && !placement->num_busy_placement) {
> - ret = ttm_bo_pipeline_gutting(bo);
> - if (ret)
> - return ret;
> -
> - return ttm_tt_create(bo, false);
> - }
> + if (!placement->num_placement && !placement->num_busy_placement)
> + return ttm_bo_pipeline_gutting(bo);
>
> /*
> * Check whether we need to move buffer.
> diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c b/drivers/gpu/drm/ttm/ttm_bo_util.c
> index 4a7d3d672f9a..7fa9b3a852eb 100644
> --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> @@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct ttm_buffer_object *bo,
> }
> EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
>
> +/**
> + * ttm_bo_pipeline_gutting - purge the contents of a bo
> + * @bo: The buffer object
> + *
> + * Purge the contents of a bo, async if the bo is not idle.
> + * After a successful call, the bo is left unpopulated in
> + * system placement. The function may wait uninterruptible
> + * for idle on OOM.
> + *
> + * Return: 0 if successful, negative error code on failure.
> + */
> int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> {
> static const struct ttm_place sys_mem = { .mem_type = TTM_PL_SYSTEM };
> struct ttm_buffer_object *ghost;
> + struct ttm_tt *ttm;
> int ret;
>
> - ret = ttm_buffer_object_transfer(bo, &ghost);
> + /* If already idle, no need for ghost object dance. */
> + ret = ttm_bo_wait(bo, false, true);
> + if (ret != -EBUSY) {
> + if (!bo->ttm) {
> + ret = ttm_tt_create(bo, true);
Why do we now unconditionally add clearing? Below also.
> + if (ret)
> + return ret;
> + } else {
> + ttm_tt_unpopulate(bo->bdev, bo->ttm);
> + if (bo->type == ttm_bo_type_device)
> + ttm_tt_mark_for_clear(bo->ttm);
> + }
> + ttm_resource_free(bo, &bo->mem);
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> +
> + return 0;
> + }
> +
> + /*
> + * We need an unpopulated ttm_tt after giving our current one,
> + * if any, to the ghost object. And we can't afford to fail
> + * creating one *after* the operation.
> + */
> +
> + ttm = bo->ttm;
> + bo->ttm = NULL;
> + ret = ttm_tt_create(bo, true);
> + swap(bo->ttm, ttm);
> if (ret)
> return ret;
>
> + ret = ttm_buffer_object_transfer(bo, &ghost);
> + if (ret) {
> + ttm_tt_destroy(bo->bdev, ttm);
> + return ret;
> + }
> +
> ret = dma_resv_copy_fences(&ghost->base._resv, bo->base.resv);
> /* Last resort, wait for the BO to be idle when we are OOM */
> if (ret)
> ttm_bo_wait(bo, false, false);
>
> - ttm_resource_alloc(bo, &sys_mem, &bo->mem);
> - bo->ttm = NULL;
> -
> dma_resv_unlock(&ghost->base._resv);
> ttm_bo_put(ghost);
> + bo->ttm = ttm;
> + ttm_resource_alloc(bo, &sys_mem, &bo->mem);
>
> return 0;
> }
> diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
> index 0e41227116b1..913b330a234b 100644
> --- a/drivers/gpu/drm/ttm/ttm_tt.c
> +++ b/drivers/gpu/drm/ttm/ttm_tt.c
> @@ -134,6 +134,11 @@ void ttm_tt_destroy_common(struct ttm_device *bdev, struct ttm_tt *ttm)
> }
> EXPORT_SYMBOL(ttm_tt_destroy_common);
>
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm)
> +{
> + ttm->page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
> +}
> +
> void ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
> {
> bdev->funcs->ttm_tt_destroy(bdev, ttm);
> diff --git a/include/drm/ttm/ttm_tt.h b/include/drm/ttm/ttm_tt.h
> index 3102059db726..daa9c4cf48bb 100644
> --- a/include/drm/ttm/ttm_tt.h
> +++ b/include/drm/ttm/ttm_tt.h
> @@ -170,6 +170,16 @@ int ttm_tt_populate(struct ttm_device *bdev, struct ttm_tt *ttm, struct ttm_oper
> */
> void ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm);
>
> +/**
> + * ttm_tt_mark_for_clear - Mark pages for clearing on populate.
> + *
> + * @ttm: Pointer to the ttm_tt structure
> + *
> + * Marks pages for clearing so that the next time the page vector is
> + * populated, the pages will be cleared.
> + */
> +void ttm_tt_mark_for_clear(struct ttm_tt *ttm);
> +
> void ttm_tt_mgr_init(unsigned long num_pages, unsigned long num_dma32_pages);
>
> struct ttm_kmap_iter *ttm_kmap_iter_tt_init(struct ttm_kmap_iter_tt *iter_tt,
> --
> 2.31.1
>
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2021-05-25 11:00 ` Matthew Auld
@ 2021-05-25 13:37 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 13:37 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 2021-05-25 at 12:00 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > If the bo is idle when calling ttm_bo_pipeline_gutting(), we
> > unnecessarily
> > create a ghost object and push it out to delayed destroy.
> > Fix this by adding a path for idle, and document the function.
> >
> > Also avoid having the bo end up in a bad state vulnerable to user-
> > space
> > triggered kernel BUGs if the call to ttm_tt_create() fails.
> >
> > Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
> >
> > Cc: Christian König <christian.koenig@amd.com>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
> > drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
> > drivers/gpu/drm/ttm/ttm_bo_util.c | 52
> > ++++++++++++++++++++++++++++---
> > drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> > include/drm/ttm/ttm_tt.h | 10 ++++++
> > 4 files changed, 73 insertions(+), 14 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > b/drivers/gpu/drm/ttm/ttm_bo.c
> > index ca1b098b6a56..a8fa3375b8aa 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct
> > ttm_buffer_object *bo,
> > bdev->funcs->evict_flags(bo, &placement);
> >
> > if (!placement.num_placement &&
> > !placement.num_busy_placement) {
> > - ttm_bo_wait(bo, false, false);
> > + ret = ttm_bo_wait(bo, true, false);
> > + if (ret)
> > + return ret;
> >
> > - ttm_bo_cleanup_memtype_use(bo);
> > - return ttm_tt_create(bo, false);
> > + /*
> > + * Since we've already synced, this frees backing
> > store
> > + * immediately.
> > + */
> > + return ttm_bo_pipeline_gutting(bo);
> > }
> >
> > ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> > @@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object
> > *bo,
> > /*
> > * Remove the backing store if no placement is given.
> > */
> > - if (!placement->num_placement && !placement-
> > >num_busy_placement) {
> > - ret = ttm_bo_pipeline_gutting(bo);
> > - if (ret)
> > - return ret;
> > -
> > - return ttm_tt_create(bo, false);
> > - }
> > + if (!placement->num_placement && !placement-
> > >num_busy_placement)
> > + return ttm_bo_pipeline_gutting(bo);
> >
> > /*
> > * Check whether we need to move buffer.
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 4a7d3d672f9a..7fa9b3a852eb 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct
> > ttm_buffer_object *bo,
> > }
> > EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
> >
> > +/**
> > + * ttm_bo_pipeline_gutting - purge the contents of a bo
> > + * @bo: The buffer object
> > + *
> > + * Purge the contents of a bo, async if the bo is not idle.
> > + * After a successful call, the bo is left unpopulated in
> > + * system placement. The function may wait uninterruptible
> > + * for idle on OOM.
> > + *
> > + * Return: 0 if successful, negative error code on failure.
> > + */
> > int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> > {
> > static const struct ttm_place sys_mem = { .mem_type =
> > TTM_PL_SYSTEM };
> > struct ttm_buffer_object *ghost;
> > + struct ttm_tt *ttm;
> > int ret;
> >
> > - ret = ttm_buffer_object_transfer(bo, &ghost);
> > + /* If already idle, no need for ghost object dance. */
> > + ret = ttm_bo_wait(bo, false, true);
> > + if (ret != -EBUSY) {
> > + if (!bo->ttm) {
> > + ret = ttm_tt_create(bo, true);
>
> Why do we now unconditionally add clearing? Below also.
Here we've dropped the bo content and we add but do not populate a page
vector. Now if someone resurrects this object we obtain new pages and
those must be cleared, at least that's the intention.
/Thomas
^ permalink raw reply [flat|nested] 60+ messages in thread* Re: [Intel-gfx] [PATCH v3 09/12] drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
@ 2021-05-25 13:37 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-25 13:37 UTC (permalink / raw)
To: Matthew Auld
Cc: Intel Graphics Development, Christian König, ML dri-devel
On Tue, 2021-05-25 at 12:00 +0100, Matthew Auld wrote:
> On Fri, 21 May 2021 at 16:33, Thomas Hellström
> <thomas.hellstrom@linux.intel.com> wrote:
> >
> > If the bo is idle when calling ttm_bo_pipeline_gutting(), we
> > unnecessarily
> > create a ghost object and push it out to delayed destroy.
> > Fix this by adding a path for idle, and document the function.
> >
> > Also avoid having the bo end up in a bad state vulnerable to user-
> > space
> > triggered kernel BUGs if the call to ttm_tt_create() fails.
> >
> > Finally reuse ttm_bo_pipeline_gutting() in ttm_bo_evict().
> >
> > Cc: Christian König <christian.koenig@amd.com>
> > Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > ---
> > drivers/gpu/drm/ttm/ttm_bo.c | 20 ++++++------
> > drivers/gpu/drm/ttm/ttm_bo_util.c | 52
> > ++++++++++++++++++++++++++++---
> > drivers/gpu/drm/ttm/ttm_tt.c | 5 +++
> > include/drm/ttm/ttm_tt.h | 10 ++++++
> > 4 files changed, 73 insertions(+), 14 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo.c
> > b/drivers/gpu/drm/ttm/ttm_bo.c
> > index ca1b098b6a56..a8fa3375b8aa 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo.c
> > @@ -501,10 +501,15 @@ static int ttm_bo_evict(struct
> > ttm_buffer_object *bo,
> > bdev->funcs->evict_flags(bo, &placement);
> >
> > if (!placement.num_placement &&
> > !placement.num_busy_placement) {
> > - ttm_bo_wait(bo, false, false);
> > + ret = ttm_bo_wait(bo, true, false);
> > + if (ret)
> > + return ret;
> >
> > - ttm_bo_cleanup_memtype_use(bo);
> > - return ttm_tt_create(bo, false);
> > + /*
> > + * Since we've already synced, this frees backing
> > store
> > + * immediately.
> > + */
> > + return ttm_bo_pipeline_gutting(bo);
> > }
> >
> > ret = ttm_bo_mem_space(bo, &placement, &evict_mem, ctx);
> > @@ -974,13 +979,8 @@ int ttm_bo_validate(struct ttm_buffer_object
> > *bo,
> > /*
> > * Remove the backing store if no placement is given.
> > */
> > - if (!placement->num_placement && !placement-
> > >num_busy_placement) {
> > - ret = ttm_bo_pipeline_gutting(bo);
> > - if (ret)
> > - return ret;
> > -
> > - return ttm_tt_create(bo, false);
> > - }
> > + if (!placement->num_placement && !placement-
> > >num_busy_placement)
> > + return ttm_bo_pipeline_gutting(bo);
> >
> > /*
> > * Check whether we need to move buffer.
> > diff --git a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > index 4a7d3d672f9a..7fa9b3a852eb 100644
> > --- a/drivers/gpu/drm/ttm/ttm_bo_util.c
> > +++ b/drivers/gpu/drm/ttm/ttm_bo_util.c
> > @@ -585,26 +585,70 @@ int ttm_bo_move_accel_cleanup(struct
> > ttm_buffer_object *bo,
> > }
> > EXPORT_SYMBOL(ttm_bo_move_accel_cleanup);
> >
> > +/**
> > + * ttm_bo_pipeline_gutting - purge the contents of a bo
> > + * @bo: The buffer object
> > + *
> > + * Purge the contents of a bo, async if the bo is not idle.
> > + * After a successful call, the bo is left unpopulated in
> > + * system placement. The function may wait uninterruptible
> > + * for idle on OOM.
> > + *
> > + * Return: 0 if successful, negative error code on failure.
> > + */
> > int ttm_bo_pipeline_gutting(struct ttm_buffer_object *bo)
> > {
> > static const struct ttm_place sys_mem = { .mem_type =
> > TTM_PL_SYSTEM };
> > struct ttm_buffer_object *ghost;
> > + struct ttm_tt *ttm;
> > int ret;
> >
> > - ret = ttm_buffer_object_transfer(bo, &ghost);
> > + /* If already idle, no need for ghost object dance. */
> > + ret = ttm_bo_wait(bo, false, true);
> > + if (ret != -EBUSY) {
> > + if (!bo->ttm) {
> > + ret = ttm_tt_create(bo, true);
>
> Why do we now unconditionally add clearing? Below also.
Here we've dropped the bo content and we add but do not populate a page
vector. Now if someone resurrects this object we obtain new pages and
those must be cleared, at least that's the intention.
/Thomas
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH v3 10/12] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
We are calling the eviction_valuable driver callback at eviction time to
determine whether we actually can evict a buffer object.
The upcoming i915 TTM backend needs the same functionality for swapout,
and that might actually be beneficial to other drivers as well.
Add an eviction_valuable call also in the swapout path. Try to keep the
current behaviour for all drivers by returning true if the buffer object
is already in the TTM_PL_SYSTEM placement. We change behaviour for the
case where a buffer object is in a TT backed placement when swapped out,
in which case the drivers normal eviction_valuable path is run.
Finally make sure we don't try to swapout a bo that was recently purged
and therefore unpopulated.
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Don't export ttm_tt_unpopulate
- Fix confusion reading the locked pointer instead of the value
pointed to in ttm_bo_evict_swapout_allowable (Reported by
Maarten Lankhorst)
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
3 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 8c7ec09eb1a4..d5a9d7a88315 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1399,6 +1399,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
struct dma_fence *f;
int i;
+ /* Swapout? */
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
if (bo->type == ttm_bo_type_kernel &&
!amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
return false;
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index a8fa3375b8aa..f45486837b44 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
const struct ttm_place *place)
{
+ dma_resv_assert_held(bo->base.resv);
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
/* Don't evict this BO if it's outside of the
* requested placement range
*/
@@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
* b. Otherwise, trylock it.
*/
static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
- struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
+ struct ttm_operation_ctx *ctx,
+ const struct ttm_place *place,
+ bool *locked, bool *busy)
{
bool ret = false;
@@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
*busy = !ret;
}
+ if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
+ ret = false;
+ if (*locked) {
+ dma_resv_unlock(bo->base.resv);
+ *locked = false;
+ }
+ }
+
return ret;
}
@@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
list_for_each_entry(bo, &man->lru[i], lru) {
bool busy;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
- &busy)) {
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
+ &locked, &busy)) {
if (busy && !busy_bo && ticket !=
dma_resv_locking_ctx(bo->base.resv))
busy_bo = bo;
continue;
}
- if (place && !bdev->funcs->eviction_valuable(bo,
- place)) {
- if (locked)
- dma_resv_unlock(bo->base.resv);
- continue;
- }
if (!ttm_bo_get_unless_zero(bo)) {
if (locked)
dma_resv_unlock(bo->base.resv);
@@ -1138,10 +1146,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
gfp_t gfp_flags)
{
+ struct ttm_place place = {};
bool locked;
int ret;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
+ /*
+ * While the bo may already reside in SYSTEM placement, set
+ * SYSTEM as new placement to cover also the move further below.
+ * The driver may use the fact that we're moving from SYSTEM
+ * as an indication that we're about to swap out.
+ */
+ place.mem_type = TTM_PL_SYSTEM;
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
return -EBUSY;
if (!ttm_bo_get_unless_zero(bo)) {
@@ -1166,12 +1182,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
if (bo->mem.mem_type != TTM_PL_SYSTEM) {
struct ttm_operation_ctx ctx = { false, false };
struct ttm_resource evict_mem;
- struct ttm_place place, hop;
-
- memset(&place, 0, sizeof(place));
- memset(&hop, 0, sizeof(hop));
-
- place.mem_type = TTM_PL_SYSTEM;
+ struct ttm_place hop = {};
ret = ttm_resource_alloc(bo, &place, &evict_mem);
if (unlikely(ret))
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 913b330a234b..d9793cbb6d13 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
struct page *to_page;
int i, ret;
+ if (!ttm_tt_is_populated(ttm))
+ return 0;
+
swap_storage = shmem_file_setup("ttm swap", size, 0);
if (IS_ERR(swap_storage)) {
pr_err("Failed allocating swap storage\n");
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 10/12] drm/ttm, drm/amdgpu: Allow the driver some control over swapping
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Christian König
We are calling the eviction_valuable driver callback at eviction time to
determine whether we actually can evict a buffer object.
The upcoming i915 TTM backend needs the same functionality for swapout,
and that might actually be beneficial to other drivers as well.
Add an eviction_valuable call also in the swapout path. Try to keep the
current behaviour for all drivers by returning true if the buffer object
is already in the TTM_PL_SYSTEM placement. We change behaviour for the
case where a buffer object is in a TT backed placement when swapped out,
in which case the drivers normal eviction_valuable path is run.
Finally make sure we don't try to swapout a bo that was recently purged
and therefore unpopulated.
Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Christian König <christian.koenig@amd.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
v3:
- Don't export ttm_tt_unpopulate
- Fix confusion reading the locked pointer instead of the value
pointed to in ttm_bo_evict_swapout_allowable (Reported by
Maarten Lankhorst)
---
drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 +++
drivers/gpu/drm/ttm/ttm_bo.c | 43 ++++++++++++++++---------
drivers/gpu/drm/ttm/ttm_tt.c | 3 ++
3 files changed, 34 insertions(+), 16 deletions(-)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 8c7ec09eb1a4..d5a9d7a88315 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1399,6 +1399,10 @@ static bool amdgpu_ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
struct dma_fence *f;
int i;
+ /* Swapout? */
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
if (bo->type == ttm_bo_type_kernel &&
!amdgpu_vm_evictable(ttm_to_amdgpu_bo(bo)))
return false;
diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index a8fa3375b8aa..f45486837b44 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -536,6 +536,10 @@ static int ttm_bo_evict(struct ttm_buffer_object *bo,
bool ttm_bo_eviction_valuable(struct ttm_buffer_object *bo,
const struct ttm_place *place)
{
+ dma_resv_assert_held(bo->base.resv);
+ if (bo->mem.mem_type == TTM_PL_SYSTEM)
+ return true;
+
/* Don't evict this BO if it's outside of the
* requested placement range
*/
@@ -558,7 +562,9 @@ EXPORT_SYMBOL(ttm_bo_eviction_valuable);
* b. Otherwise, trylock it.
*/
static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
- struct ttm_operation_ctx *ctx, bool *locked, bool *busy)
+ struct ttm_operation_ctx *ctx,
+ const struct ttm_place *place,
+ bool *locked, bool *busy)
{
bool ret = false;
@@ -576,6 +582,14 @@ static bool ttm_bo_evict_swapout_allowable(struct ttm_buffer_object *bo,
*busy = !ret;
}
+ if (ret && place && !bo->bdev->funcs->eviction_valuable(bo, place)) {
+ ret = false;
+ if (*locked) {
+ dma_resv_unlock(bo->base.resv);
+ *locked = false;
+ }
+ }
+
return ret;
}
@@ -630,20 +644,14 @@ int ttm_mem_evict_first(struct ttm_device *bdev,
list_for_each_entry(bo, &man->lru[i], lru) {
bool busy;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked,
- &busy)) {
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, place,
+ &locked, &busy)) {
if (busy && !busy_bo && ticket !=
dma_resv_locking_ctx(bo->base.resv))
busy_bo = bo;
continue;
}
- if (place && !bdev->funcs->eviction_valuable(bo,
- place)) {
- if (locked)
- dma_resv_unlock(bo->base.resv);
- continue;
- }
if (!ttm_bo_get_unless_zero(bo)) {
if (locked)
dma_resv_unlock(bo->base.resv);
@@ -1138,10 +1146,18 @@ EXPORT_SYMBOL(ttm_bo_wait);
int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
gfp_t gfp_flags)
{
+ struct ttm_place place = {};
bool locked;
int ret;
- if (!ttm_bo_evict_swapout_allowable(bo, ctx, &locked, NULL))
+ /*
+ * While the bo may already reside in SYSTEM placement, set
+ * SYSTEM as new placement to cover also the move further below.
+ * The driver may use the fact that we're moving from SYSTEM
+ * as an indication that we're about to swap out.
+ */
+ place.mem_type = TTM_PL_SYSTEM;
+ if (!ttm_bo_evict_swapout_allowable(bo, ctx, &place, &locked, NULL))
return -EBUSY;
if (!ttm_bo_get_unless_zero(bo)) {
@@ -1166,12 +1182,7 @@ int ttm_bo_swapout(struct ttm_buffer_object *bo, struct ttm_operation_ctx *ctx,
if (bo->mem.mem_type != TTM_PL_SYSTEM) {
struct ttm_operation_ctx ctx = { false, false };
struct ttm_resource evict_mem;
- struct ttm_place place, hop;
-
- memset(&place, 0, sizeof(place));
- memset(&hop, 0, sizeof(hop));
-
- place.mem_type = TTM_PL_SYSTEM;
+ struct ttm_place hop = {};
ret = ttm_resource_alloc(bo, &place, &evict_mem);
if (unlikely(ret))
diff --git a/drivers/gpu/drm/ttm/ttm_tt.c b/drivers/gpu/drm/ttm/ttm_tt.c
index 913b330a234b..d9793cbb6d13 100644
--- a/drivers/gpu/drm/ttm/ttm_tt.c
+++ b/drivers/gpu/drm/ttm/ttm_tt.c
@@ -263,6 +263,9 @@ int ttm_tt_swapout(struct ttm_device *bdev, struct ttm_tt *ttm,
struct page *to_page;
int i, ret;
+ if (!ttm_tt_is_populated(ttm))
+ return 0;
+
swap_storage = shmem_file_setup("ttm swap", size, 0);
if (IS_ERR(swap_storage)) {
pr_err("Failed allocating swap storage\n");
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 11/12] drm/i915/ttm: Introduce a TTM i915 gem object backend
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Most logical place to introduce TTM buffer objects is as an i915
gem object backend. We need to add some ops to account for added
functionality like delayed delete and LRU list manipulation.
Initially we support only LMEM and SYSTEM memory, but SYSTEM
(which in this case means evicted LMEM objects) is not
visible to i915 GEM yet. The plan is to move the i915 gem system region
over to the TTM system memory type in upcoming patches.
We set up GPU bindings directly both from LMEM and from the system region,
as there is no need to use the legacy TTM_TT memory type. We reserve
that for future porting of GGTT bindings to TTM.
Remove the old lmem backend.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2:
- Break out needed TTM functionality to a separate patch (Reported by
Christian König).
- Fix an unhandled error (Reported by Matthew Auld and Maarten Lankhorst)
- Remove a stray leftover sg_table allocation (Reported by Matthew Auld)
- Use ttm_tt_unpopulate() rather than ttm_tt_destroy() in the purge path
as some TTM functionality relies on having a ttm_tt present for !is_iomem.
v3:
- Use ttm_bo_type_device for userspace visible objects so that TTM can
allocate an address space offset for mmap'ing.
- Fix up the destruction path (Reported by Matthew Auld)
- Use ttm_bo_validate() for purging (Reported by Christian König)
- Create ttm_tts write-combined as they are currently for eviction only and
we want to maintain consistent write-combined caching for bos that are
not in system only. (Suggested by Daniel Vetter)
- Make struct ttm_placements static.
- Add the ttm device funcs/ops to i915_gem_ttm.h for the region code.
- Rename new->dst and old->src. Check for swapin in the move callback.
---
drivers/gpu/drm/i915/Makefile | 1 +
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 84 ---
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_object.c | 125 +++--
drivers/gpu/drm/i915/gem/i915_gem_object.h | 9 +
.../gpu/drm/i915/gem/i915_gem_object_types.h | 27 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 531 ++++++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 50 ++
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 3 +-
drivers/gpu/drm/i915/i915_gem.c | 5 +-
drivers/gpu/drm/i915/intel_memory_region.c | 1 -
drivers/gpu/drm/i915/intel_memory_region.h | 1 -
drivers/gpu/drm/i915/intel_region_ttm.c | 6 +-
drivers/gpu/drm/i915/intel_region_ttm.h | 8 +-
16 files changed, 717 insertions(+), 154 deletions(-)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 998606b7f49f..2022e763f8f7 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -154,6 +154,7 @@ gem-y += \
gem/i915_gem_stolen.o \
gem/i915_gem_throttle.o \
gem/i915_gem_tiling.o \
+ gem/i915_gem_ttm.o \
gem/i915_gem_userptr.o \
gem/i915_gem_wait.o \
gem/i915_gemfs.o
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_create.c b/drivers/gpu/drm/i915/gem/i915_gem_create.c
index 548ddf39d853..93bf63bbaff1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_create.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_create.c
@@ -85,13 +85,10 @@ i915_gem_setup(struct drm_i915_gem_object *obj, u64 size)
return -E2BIG;
/*
- * For now resort to CPU based clearing for device local-memory, in the
- * near future this will use the blitter engine for accelerated, GPU
- * based clearing.
+ * I915_BO_ALLOC_USER will make sure the object is cleared before
+ * any user access.
*/
- flags = 0;
- if (mr->type == INTEL_MEMORY_LOCAL)
- flags = I915_BO_ALLOC_CPU_CLEAR;
+ flags = I915_BO_ALLOC_USER;
ret = mr->ops->init_object(mr, obj, size, flags);
if (ret)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 3b4aa28a076d..2b8cd15de1d9 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,74 +4,10 @@
*/
#include "intel_memory_region.h"
-#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
-static void lmem_put_pages(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-static int lmem_get_pages(struct drm_i915_gem_object *obj)
-{
- unsigned int flags;
- struct sg_table *pages;
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
- obj->base.size,
- flags);
- if (IS_ERR(obj->mm.st_mm_node))
- return PTR_ERR(obj->mm.st_mm_node);
-
- /* Range manager is always contigous */
- if (obj->mm.region->is_range_manager)
- obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
- pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
- if (IS_ERR(pages)) {
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- return PTR_ERR(pages);
- }
-
- __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
-
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- void __iomem *vaddr =
- i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
-
- if (!vaddr) {
- struct sg_table *pages =
- __i915_gem_object_unset_pages(obj);
-
- if (!IS_ERR_OR_NULL(pages))
- lmem_put_pages(obj, pages);
- }
-
- memset_io(vaddr, 0, obj->base.size);
- io_mapping_unmap(vaddr);
- }
-
- return 0;
-}
-
-const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
- .name = "i915_gem_object_lmem",
- .flags = I915_GEM_OBJECT_HAS_IOMEM,
-
- .get_pages = lmem_get_pages,
- .put_pages = lmem_put_pages,
- .release = i915_gem_object_release_memory_region,
-};
-
void __iomem *
i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
unsigned long n,
@@ -103,23 +39,3 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
return i915_gem_object_create_region(i915->mm.regions[INTEL_REGION_LMEM],
size, flags);
}
-
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags)
-{
- static struct lock_class_key lock_class;
- struct drm_i915_private *i915 = mem->i915;
-
- drm_gem_private_object_init(&i915->drm, &obj->base, size);
- i915_gem_object_init(obj, &i915_gem_lmem_obj_ops, &lock_class, flags);
-
- obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
-
- i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
-
- i915_gem_object_init_memory_region(obj, mem);
-
- return 0;
-}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
index fac6bc5a5ebb..ea76fd11ccb0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
@@ -26,9 +26,4 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
resource_size_t size,
unsigned int flags);
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags);
-
#endif /* !__I915_GEM_LMEM_H */
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index e9247afb0320..df2b4e6b9bcc 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -173,7 +173,7 @@ static void i915_gem_close_object(struct drm_gem_object *gem, struct drm_file *f
}
}
-static void __i915_gem_free_object_rcu(struct rcu_head *head)
+void __i915_gem_free_object_rcu(struct rcu_head *head)
{
struct drm_i915_gem_object *obj =
container_of(head, typeof(*obj), rcu);
@@ -209,59 +209,69 @@ static void __i915_gem_object_free_mmaps(struct drm_i915_gem_object *obj)
}
}
-static void __i915_gem_free_objects(struct drm_i915_private *i915,
- struct llist_node *freed)
+void __i915_gem_free_object(struct drm_i915_gem_object *obj)
{
- struct drm_i915_gem_object *obj, *on;
+ trace_i915_gem_object_destroy(obj);
- llist_for_each_entry_safe(obj, on, freed, freed) {
- trace_i915_gem_object_destroy(obj);
+ if (!list_empty(&obj->vma.list)) {
+ struct i915_vma *vma;
+
+ /*
+ * Note that the vma keeps an object reference while
+ * it is active, so it *should* not sleep while we
+ * destroy it. Our debug code errs insits it *might*.
+ * For the moment, play along.
+ */
+ spin_lock(&obj->vma.lock);
+ while ((vma = list_first_entry_or_null(&obj->vma.list,
+ struct i915_vma,
+ obj_link))) {
+ GEM_BUG_ON(vma->obj != obj);
+ spin_unlock(&obj->vma.lock);
- if (!list_empty(&obj->vma.list)) {
- struct i915_vma *vma;
+ __i915_vma_put(vma);
- /*
- * Note that the vma keeps an object reference while
- * it is active, so it *should* not sleep while we
- * destroy it. Our debug code errs insits it *might*.
- * For the moment, play along.
- */
spin_lock(&obj->vma.lock);
- while ((vma = list_first_entry_or_null(&obj->vma.list,
- struct i915_vma,
- obj_link))) {
- GEM_BUG_ON(vma->obj != obj);
- spin_unlock(&obj->vma.lock);
+ }
+ spin_unlock(&obj->vma.lock);
+ }
- __i915_vma_put(vma);
+ __i915_gem_object_free_mmaps(obj);
- spin_lock(&obj->vma.lock);
- }
- spin_unlock(&obj->vma.lock);
- }
+ GEM_BUG_ON(!list_empty(&obj->lut_list));
- __i915_gem_object_free_mmaps(obj);
+ atomic_set(&obj->mm.pages_pin_count, 0);
+ __i915_gem_object_put_pages(obj);
+ GEM_BUG_ON(i915_gem_object_has_pages(obj));
+ bitmap_free(obj->bit_17);
- GEM_BUG_ON(!list_empty(&obj->lut_list));
+ if (obj->base.import_attach)
+ drm_prime_gem_destroy(&obj->base, NULL);
- atomic_set(&obj->mm.pages_pin_count, 0);
- __i915_gem_object_put_pages(obj);
- GEM_BUG_ON(i915_gem_object_has_pages(obj));
- bitmap_free(obj->bit_17);
+ drm_gem_free_mmap_offset(&obj->base);
- if (obj->base.import_attach)
- drm_prime_gem_destroy(&obj->base, NULL);
+ if (obj->ops->release)
+ obj->ops->release(obj);
- drm_gem_free_mmap_offset(&obj->base);
+ if (obj->mm.n_placements > 1)
+ kfree(obj->mm.placements);
- if (obj->ops->release)
- obj->ops->release(obj);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+}
- if (obj->mm.n_placements > 1)
- kfree(obj->mm.placements);
+static void __i915_gem_free_objects(struct drm_i915_private *i915,
+ struct llist_node *freed)
+{
+ struct drm_i915_gem_object *obj, *on;
- if (obj->shares_resv_from)
- i915_vm_resv_put(obj->shares_resv_from);
+ llist_for_each_entry_safe(obj, on, freed, freed) {
+ might_sleep();
+ if (obj->ops->delayed_free) {
+ obj->ops->delayed_free(obj);
+ continue;
+ }
+ __i915_gem_free_object(obj);
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
@@ -319,6 +329,7 @@ static void i915_gem_free_object(struct drm_gem_object *gem_obj)
* worker and performing frees directly from subsequent allocations for
* crude but effective memory throttling.
*/
+
if (llist_add(&obj->freed, &i915->mm.free_list))
queue_work(i915->wq, &i915->mm.free_work);
}
@@ -411,6 +422,42 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
return 0;
}
+/**
+ * i915_gem_object_evictable - Whether object is likely evictable after unbind.
+ * @obj: The object to check
+ *
+ * This function checks whether the object is likely unvictable after unbind.
+ * If the object is not locked when checking, the result is only advisory.
+ * If the object is locked when checking, and the function returns true,
+ * then an eviction should indeed be possible. But since unlocked vma
+ * unpinning and unbinding is currently possible, the object can actually
+ * become evictable even if this function returns false.
+ *
+ * Return: true if the object may be evictable. False otherwise.
+ */
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
+{
+ struct i915_vma *vma;
+ int pin_count = atomic_read(&obj->mm.pages_pin_count);
+
+ if (!pin_count)
+ return true;
+
+ spin_lock(&obj->vma.lock);
+ list_for_each_entry(vma, &obj->vma.list, obj_link) {
+ if (i915_vma_is_pinned(vma)) {
+ spin_unlock(&obj->vma.lock);
+ return false;
+ }
+ if (atomic_read(&vma->pages_count))
+ pin_count--;
+ }
+ spin_unlock(&obj->vma.lock);
+ GEM_WARN_ON(pin_count < 0);
+
+ return pin_count == 0;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index 2ebd79537aea..ae5930e307d5 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -200,6 +200,9 @@ static inline bool i915_gem_object_trylock(struct drm_i915_gem_object *obj)
static inline void i915_gem_object_unlock(struct drm_i915_gem_object *obj)
{
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+
dma_resv_unlock(obj->base.resv);
}
@@ -587,6 +590,12 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
bool i915_gem_object_is_shmem(const struct drm_i915_gem_object *obj);
+void __i915_gem_free_object_rcu(struct rcu_head *head);
+
+void __i915_gem_free_object(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index d047ea126029..68313474e6a6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -63,6 +63,20 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pwrite *arg);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
+
+ /**
+ * adjust_lru - notify that the madvise value was updated
+ * @obj: The gem object
+ *
+ * The madvise value may have been updated, or object was recently
+ * referenced so act accordingly (Perhaps changing an LRU list etc).
+ */
+ void (*adjust_lru)(struct drm_i915_gem_object *obj);
+
+ /**
+ * delayed_free - Override the default delayed free implementation
+ */
+ void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
const char *name; /* friendly name for debug, e.g. lockdep classes */
@@ -187,12 +201,14 @@ struct drm_i915_gem_object {
#define I915_BO_ALLOC_VOLATILE BIT(1)
#define I915_BO_ALLOC_STRUCT_PAGE BIT(2)
#define I915_BO_ALLOC_CPU_CLEAR BIT(3)
+#define I915_BO_ALLOC_USER BIT(4)
#define I915_BO_ALLOC_FLAGS (I915_BO_ALLOC_CONTIGUOUS | \
I915_BO_ALLOC_VOLATILE | \
I915_BO_ALLOC_STRUCT_PAGE | \
- I915_BO_ALLOC_CPU_CLEAR)
-#define I915_BO_READONLY BIT(4)
-#define I915_TILING_QUIRK_BIT 5 /* unknown swizzling; do not release! */
+ I915_BO_ALLOC_CPU_CLEAR | \
+ I915_BO_ALLOC_USER)
+#define I915_BO_READONLY BIT(5)
+#define I915_TILING_QUIRK_BIT 6 /* unknown swizzling; do not release! */
/*
* Is the object to be mapped as read-only to the GPU
@@ -310,6 +326,11 @@ struct drm_i915_gem_object {
bool dirty:1;
} mm;
+ struct {
+ struct sg_table *cached_io_st;
+ bool created:1;
+ } ttm;
+
/** Record of address bit 17 of each page at last unbind. */
unsigned long *bit_17;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index f25e6646c5b7..d1f1840540dd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -18,11 +18,7 @@ void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
mutex_lock(&mem->objects.lock);
- if (obj->flags & I915_BO_ALLOC_VOLATILE)
- list_add(&obj->mm.region_link, &mem->objects.purgeable);
- else
- list_add(&obj->mm.region_link, &mem->objects.list);
-
+ list_add(&obj->mm.region_link, &mem->objects.list);
mutex_unlock(&mem->objects.lock);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
new file mode 100644
index 000000000000..3cec777be346
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -0,0 +1,531 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_placement.h>
+
+#include "i915_drv.h"
+#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
+
+#include "gem/i915_gem_object.h"
+#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
+
+#define I915_PL_LMEM0 TTM_PL_PRIV
+#define I915_PL_SYSTEM TTM_PL_SYSTEM
+#define I915_PL_STOLEN TTM_PL_VRAM
+#define I915_PL_GGTT TTM_PL_TT
+
+#define I915_TTM_PRIO_PURGE 0
+#define I915_TTM_PRIO_NO_PAGES 1
+#define I915_TTM_PRIO_HAS_PAGES 2
+
+/**
+ * struct i915_ttm_tt - TTM page vector with additional private information
+ * @ttm: The base TTM page vector.
+ * @dev: The struct device used for dma mapping and unmapping.
+ * @cached_st: The cached scatter-gather table.
+ *
+ * Note that DMA may be going on right up to the point where the page-
+ * vector is unpopulated in delayed destroy. Hence keep the
+ * scatter-gather table mapped and cached up to that point. This is
+ * different from the cached gem object io scatter-gather table which
+ * doesn't have an associated dma mapping.
+ */
+struct i915_ttm_tt {
+ struct ttm_tt ttm;
+ struct device *dev;
+ struct sg_table *cached_st;
+};
+
+static const struct ttm_place lmem0_sys_placement_flags[] = {
+ {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_LMEM0,
+ .flags = 0,
+ }, {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_SYSTEM,
+ .flags = 0,
+ }
+};
+
+static struct ttm_placement i915_lmem0_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[0],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[0],
+};
+
+static struct ttm_placement i915_sys_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[1],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[1],
+};
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj);
+
+static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,
+ uint32_t page_flags)
+{
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct i915_ttm_tt *i915_tt;
+ int ret;
+
+ i915_tt = kzalloc(sizeof(*i915_tt), GFP_KERNEL);
+ if (!i915_tt)
+ return NULL;
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR &&
+ man->use_tt)
+ page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+
+ ret = ttm_tt_init(&i915_tt->ttm, bo, page_flags, ttm_write_combined);
+ if (ret) {
+ kfree(i915_tt);
+ return NULL;
+ }
+
+ i915_tt->dev = obj->base.dev->dev;
+
+ return &i915_tt->ttm;
+}
+
+static void i915_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ if (i915_tt->cached_st) {
+ dma_unmap_sgtable(i915_tt->dev, i915_tt->cached_st,
+ DMA_BIDIRECTIONAL, 0);
+ sg_free_table(i915_tt->cached_st);
+ kfree(i915_tt->cached_st);
+ i915_tt->cached_st = NULL;
+ }
+ ttm_pool_free(&bdev->pool, ttm);
+}
+
+static void i915_ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ ttm_tt_destroy_common(bdev, ttm);
+ kfree(i915_tt);
+}
+
+static bool i915_ttm_eviction_valuable(struct ttm_buffer_object *bo,
+ const struct ttm_place *place)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ /* Will do for now. Our pinned objects are still on TTM's LRU lists */
+ if (!i915_gem_object_evictable(obj))
+ return false;
+
+ /* This isn't valid with a buddy allocator */
+ return ttm_bo_eviction_valuable(bo, place);
+}
+
+static void i915_ttm_evict_flags(struct ttm_buffer_object *bo,
+ struct ttm_placement *placement)
+{
+ *placement = i915_sys_placement;
+}
+
+static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret;
+
+ ret = i915_gem_object_unbind(obj, I915_GEM_OBJECT_UNBIND_ACTIVE);
+ if (ret)
+ return ret;
+
+ ret = __i915_gem_object_put_pages(obj);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.cached_io_st) {
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
+ }
+}
+
+static void i915_ttm_purge(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct ttm_placement place = {};
+ int ret;
+
+ if (obj->mm.madv == __I915_MADV_PURGED)
+ return;
+
+ /* TTM's purge interface. Note that we might be reentering. */
+ ret = ttm_bo_validate(bo, &place, &ctx);
+
+ if (!ret) {
+ i915_ttm_free_cached_io_st(obj);
+ obj->mm.madv = __I915_MADV_PURGED;
+ }
+}
+
+static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret = i915_ttm_move_notify(bo);
+
+ GEM_WARN_ON(ret);
+ GEM_WARN_ON(obj->ttm.cached_io_st);
+ if (!ret && obj->mm.madv != I915_MADV_WILLNEED)
+ i915_ttm_purge(obj);
+}
+
+static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ if (likely(obj)) {
+ /* This releases all gem object bindings to the backend. */
+ __i915_gem_free_object(obj);
+ }
+}
+
+static struct intel_memory_region *
+i915_ttm_region(struct ttm_device *bdev, int ttm_mem_type)
+{
+ struct drm_i915_private *i915 = container_of(bdev, typeof(*i915), bdev);
+
+ /* There's some room for optimization here... */
+ GEM_BUG_ON(ttm_mem_type != I915_PL_SYSTEM &&
+ ttm_mem_type < I915_PL_LMEM0);
+ if (ttm_mem_type == I915_PL_SYSTEM)
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_SYSTEM,
+ 0);
+
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_LOCAL,
+ ttm_mem_type - I915_PL_LMEM0);
+}
+
+static struct sg_table *i915_ttm_tt_get_st(struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+ struct scatterlist *sg;
+ struct sg_table *st;
+ int ret;
+
+ if (i915_tt->cached_st)
+ return i915_tt->cached_st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ sg = __sg_alloc_table_from_pages
+ (st, ttm->pages, ttm->num_pages, 0,
+ (unsigned long)ttm->num_pages << PAGE_SHIFT,
+ i915_sg_segment_size(), NULL, 0, GFP_KERNEL);
+ if (IS_ERR(sg)) {
+ kfree(st);
+ return ERR_CAST(sg);
+ }
+
+ ret = dma_map_sgtable(i915_tt->dev, st, DMA_BIDIRECTIONAL, 0);
+ if (ret) {
+ sg_free_table(st);
+ kfree(st);
+ return ERR_PTR(ret);
+ }
+
+ i915_tt->cached_st = st;
+ return st;
+}
+
+static struct sg_table *
+i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
+ struct ttm_resource *res)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, res->mem_type);
+
+ if (man->use_tt)
+ return i915_ttm_tt_get_st(bo->ttm);
+
+ return intel_region_ttm_node_to_st(obj->mm.region, res->mm_node);
+}
+
+static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
+ struct ttm_operation_ctx *ctx,
+ struct ttm_resource *dst_mem,
+ struct ttm_place *hop)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct intel_memory_region *dst_reg, *src_reg;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_iomap io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ struct sg_table *dst_st;
+ int ret;
+
+ dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
+ src_reg = i915_ttm_region(bo->bdev, bo->mem.mem_type);
+ GEM_BUG_ON(!dst_reg || !src_reg);
+
+ /* Sync for now. We could do the actual copy async. */
+ ret = ttm_bo_wait_ctx(bo, ctx);
+ if (ret)
+ return ret;
+
+ ret = i915_ttm_move_notify(bo);
+ if (ret)
+ return ret;
+
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ i915_ttm_purge(obj);
+ ttm_resource_free(bo, dst_mem);
+ return 0;
+ }
+
+ /* Populate ttm with pages if needed. Typically system memory. */
+ if (bo->ttm && (dst_man->use_tt ||
+ (bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED))) {
+ ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx);
+ if (ret)
+ return ret;
+ }
+
+ dst_st = i915_ttm_resource_get_st(obj, dst_mem);
+ if (IS_ERR(dst_st))
+ return PTR_ERR(dst_st);
+
+ /* If we start mapping GGTT, we can no longer use man::use_tt here. */
+ dst_iter = dst_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+ dst_st, dst_reg->region.start);
+
+ src_iter = src_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+ obj->ttm.cached_io_st,
+ src_reg->region.start);
+
+ ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
+ i915_ttm_free_cached_io_st(obj);
+
+ if (!dst_man->use_tt)
+ obj->ttm.cached_io_st = dst_st;
+
+ return 0;
+}
+
+struct ttm_device_funcs i915_ttm_bo_driver = {
+ .ttm_tt_create = i915_ttm_tt_create,
+ .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
+ .ttm_tt_destroy = i915_ttm_tt_destroy,
+ .eviction_valuable = i915_ttm_eviction_valuable,
+ .evict_flags = i915_ttm_evict_flags,
+ .move = i915_ttm_move,
+ .verify_access = NULL,
+ .swap_notify = i915_ttm_swap_notify,
+ .delete_mem_notify = i915_ttm_delete_mem_notify,
+};
+
+static int i915_ttm_get_pages(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct sg_table *st;
+ int ret;
+
+ /* Move to the requested placement. */
+ ret = ttm_bo_validate(bo, &i915_lmem0_placement, &ctx);
+ if (ret)
+ return ret == -ENOSPC ? -ENXIO : ret;
+
+ /* Object either has a page vector or is an iomem object */
+ st = bo->ttm ? i915_ttm_tt_get_st(bo->ttm) : obj->ttm.cached_io_st;
+ if (IS_ERR(st))
+ return PTR_ERR(st);
+
+ __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl));
+
+ i915_ttm_adjust_lru(obj);
+
+ return ret;
+}
+
+static void i915_ttm_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *st)
+{
+ /*
+ * We're currently not called from a shrinker, so put_pages()
+ * typically means the object is about to destroyed, or called
+ * from move_notify(). So just avoid doing much for now.
+ * If the object is not destroyed next, The TTM eviction logic
+ * and shrinkers will move it out if needed.
+ */
+
+ i915_ttm_adjust_lru(obj);
+}
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+
+ /*
+ * Don't manipulate the TTM LRUs while in TTM bo destruction.
+ * We're called through i915_ttm_delete_mem_notify().
+ */
+ if (!kref_read(&bo->kref))
+ return;
+
+ /*
+ * Put on the correct LRU list depending on the MADV status
+ */
+ spin_lock(&bo->bdev->lru_lock);
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ bo->priority = I915_TTM_PRIO_PURGE;
+ } else if (!i915_gem_object_has_pages(obj)) {
+ if (bo->priority < I915_TTM_PRIO_HAS_PAGES)
+ bo->priority = I915_TTM_PRIO_HAS_PAGES;
+ } else {
+ if (bo->priority > I915_TTM_PRIO_NO_PAGES)
+ bo->priority = I915_TTM_PRIO_NO_PAGES;
+ }
+
+ ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL);
+ spin_unlock(&bo->bdev->lru_lock);
+}
+
+/*
+ * TTM-backed gem object destruction requires some clarification.
+ * Basically we have two possibilities here. We can either rely on the
+ * i915 delayed destruction and put the TTM object when the object
+ * is idle. This would be detected by TTM which would bypass the
+ * TTM delayed destroy handling. The other approach is to put the TTM
+ * object early and rely on the TTM destroyed handling, and then free
+ * the leftover parts of the GEM object once TTM's destroyed list handling is
+ * complete. For now, we rely on the latter for two reasons:
+ * a) TTM can evict an object even when it's on the delayed destroy list,
+ * which in theory allows for complete eviction.
+ * b) There is work going on in TTM to allow freeing an object even when
+ * it's not idle, and using the TTM destroyed list handling could help us
+ * benefit from that.
+ */
+static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.created) {
+ ttm_bo_put(i915_gem_to_ttm(obj));
+ } else {
+ __i915_gem_free_object(obj);
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+ }
+}
+
+static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+ .name = "i915_gem_object_ttm",
+ .flags = I915_GEM_OBJECT_HAS_IOMEM,
+
+ .get_pages = i915_ttm_get_pages,
+ .put_pages = i915_ttm_put_pages,
+ .truncate = i915_ttm_purge,
+ .adjust_lru = i915_ttm_adjust_lru,
+ .delayed_free = i915_ttm_delayed_free,
+};
+
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ i915_gem_object_release_memory_region(obj);
+ if (obj->ttm.created)
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+}
+
+/**
+ * __i915_gem_ttm_object_init - Initialize a ttm-backed i915 gem object
+ * @mem: The initial memory region for the object.
+ * @obj: The gem object.
+ * @size: Object size in bytes.
+ * @flags: gem object flags.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags)
+{
+ static struct lock_class_key lock_class;
+ struct drm_i915_private *i915 = mem->i915;
+ enum ttm_bo_type bo_type;
+ size_t alignment = 0;
+ int ret;
+
+ /* Adjust alignment to GPU- and CPU huge page sizes. */
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ drm_gem_private_object_init(&i915->drm, &obj->base, size);
+ i915_gem_object_init(obj, &i915_gem_ttm_obj_ops, &lock_class, flags);
+ i915_gem_object_init_memory_region(obj, mem);
+ i915_gem_object_make_unshrinkable(obj);
+ obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+
+ bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
+ ttm_bo_type_kernel;
+
+ /*
+ * If this function fails, it will call the destructor, but
+ * our caller still owns the object. So no freeing in the
+ * destructor until obj->ttm.created is true.
+ * Similarly, in delayed_destroy, we can't call ttm_bo_put()
+ * until successful initialization.
+ */
+ ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
+ bo_type, &i915_sys_placement, alignment,
+ true, NULL, NULL, i915_ttm_bo_destroy);
+
+ if (!ret)
+ obj->ttm.created = true;
+
+ /* i915 wants -ENXIO when out of memory region space. */
+ return (ret == -ENOSPC) ? -ENXIO : ret;
+}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
new file mode 100644
index 000000000000..05c114e35b4d
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _I915_GEM_TTM_H_
+#define _I915_GEM_TTM_H_
+
+#include "gem/i915_gem_object_types.h"
+
+/**
+ * i915_gem_to_ttm - Convert a struct drm_i915_gem_object to a
+ * struct ttm_buffer_object.
+ * @obj: Pointer to the gem object.
+ *
+ * Return: Pointer to the embedded struct ttm_buffer_object.
+ */
+static inline struct ttm_buffer_object *
+i915_gem_to_ttm(struct drm_i915_gem_object *obj)
+{
+ return &obj->__do_not_access;
+}
+
+/*
+ * i915 ttm gem object destructor. Internal use only.
+ */
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo);
+
+/**
+ * i915_ttm_to_gem - Convert a struct ttm_buffer_object to an embedding
+ * struct drm_i915_gem_object.
+ *
+ * Return: Pointer to the embedding struct ttm_buffer_object, or NULL
+ * if the object was not an i915 ttm object.
+ */
+static inline struct drm_i915_gem_object *
+i915_ttm_to_gem(struct ttm_buffer_object *bo)
+{
+ if (GEM_WARN_ON(bo->destroy != i915_ttm_bo_destroy))
+ return NULL;
+
+ return container_of(bo, struct drm_i915_gem_object, __do_not_access);
+}
+
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags);
+
+extern struct ttm_device_funcs i915_ttm_bo_driver;
+#endif
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index f7366b054f8e..4ae1f717a94c 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -9,6 +9,7 @@
#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
#include "intel_region_lmem.h"
static int init_fake_lmem_bar(struct intel_memory_region *mem)
@@ -107,7 +108,7 @@ region_lmem_init(struct intel_memory_region *mem)
static const struct intel_memory_region_ops intel_region_lmem_ops = {
.init = region_lmem_init,
.release = region_lmem_release,
- .init_object = __i915_gem_lmem_object_init,
+ .init_object = __i915_gem_ttm_object_init,
};
struct intel_memory_region *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 180f6e9107d4..350283ab9a83 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1005,8 +1005,11 @@ i915_gem_madvise_ioctl(struct drm_device *dev, void *data,
}
}
- if (obj->mm.madv != __I915_MADV_PURGED)
+ if (obj->mm.madv != __I915_MADV_PURGED) {
obj->mm.madv = args->madv;
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+ }
if (i915_gem_object_has_pages(obj)) {
struct list_head *list;
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index 4092cc987679..bd27e897d4d0 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -149,7 +149,6 @@ intel_memory_region_create(struct drm_i915_private *i915,
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
- INIT_LIST_HEAD(&mem->objects.purgeable);
INIT_LIST_HEAD(&mem->reserved);
mutex_init(&mem->mm_lock);
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index e69cde13daf2..7b5fa97c0b59 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -100,7 +100,6 @@ struct intel_memory_region {
struct {
struct mutex lock; /* Protects access to objects */
struct list_head list;
- struct list_head purgeable;
} objects;
size_t chunk_size;
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index 69c98298326b..52352dbe047d 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -10,6 +10,7 @@
#include "intel_region_ttm.h"
+#include "gem/i915_gem_ttm.h" /* For the funcs/ops export only */
/**
* DOC: TTM support structure
*
@@ -31,9 +32,6 @@ struct intel_region_ttm_private {
I915_SELFTEST_DECLARE(struct drm_vma_offset_manager mock_vma);
};
-/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
-static struct ttm_device_funcs i915_ttm_bo_driver;
-
static struct intel_region_ttm_private *to_ttm_private(void *data)
{
return (struct intel_region_ttm_private *)data;
@@ -219,6 +217,7 @@ struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
return i915_sg_from_mm_node(node, mem->region.start);
}
+#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_node_alloc - Allocate memory resources from a region
* @mem: The memory region,
@@ -267,7 +266,6 @@ void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
return ret ? ERR_PTR(ret) : res.mm_node;
}
-#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_init_mock - Initialize a mock memory region for TTM.
* @mem: The region to initialize.
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
index 23c6c2d4b1cd..a08998c3a3d8 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.h
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -23,14 +23,14 @@ void intel_region_ttm_fini(struct intel_memory_region *mem);
struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
void *node);
-void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-
void intel_region_ttm_node_free(struct intel_memory_region *mem,
void *node);
#ifdef CONFIG_DRM_I915_SELFTEST
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
int intel_region_ttm_init_mock(struct intel_memory_region *mem);
void intel_region_ttm_fini_mock(struct intel_memory_region *mem);
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 11/12] drm/i915/ttm: Introduce a TTM i915 gem object backend
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Most logical place to introduce TTM buffer objects is as an i915
gem object backend. We need to add some ops to account for added
functionality like delayed delete and LRU list manipulation.
Initially we support only LMEM and SYSTEM memory, but SYSTEM
(which in this case means evicted LMEM objects) is not
visible to i915 GEM yet. The plan is to move the i915 gem system region
over to the TTM system memory type in upcoming patches.
We set up GPU bindings directly both from LMEM and from the system region,
as there is no need to use the legacy TTM_TT memory type. We reserve
that for future porting of GGTT bindings to TTM.
Remove the old lmem backend.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2:
- Break out needed TTM functionality to a separate patch (Reported by
Christian König).
- Fix an unhandled error (Reported by Matthew Auld and Maarten Lankhorst)
- Remove a stray leftover sg_table allocation (Reported by Matthew Auld)
- Use ttm_tt_unpopulate() rather than ttm_tt_destroy() in the purge path
as some TTM functionality relies on having a ttm_tt present for !is_iomem.
v3:
- Use ttm_bo_type_device for userspace visible objects so that TTM can
allocate an address space offset for mmap'ing.
- Fix up the destruction path (Reported by Matthew Auld)
- Use ttm_bo_validate() for purging (Reported by Christian König)
- Create ttm_tts write-combined as they are currently for eviction only and
we want to maintain consistent write-combined caching for bos that are
not in system only. (Suggested by Daniel Vetter)
- Make struct ttm_placements static.
- Add the ttm device funcs/ops to i915_gem_ttm.h for the region code.
- Rename new->dst and old->src. Check for swapin in the move callback.
---
drivers/gpu/drm/i915/Makefile | 1 +
drivers/gpu/drm/i915/gem/i915_gem_create.c | 9 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 84 ---
drivers/gpu/drm/i915/gem/i915_gem_lmem.h | 5 -
drivers/gpu/drm/i915/gem/i915_gem_object.c | 125 +++--
drivers/gpu/drm/i915/gem/i915_gem_object.h | 9 +
.../gpu/drm/i915/gem/i915_gem_object_types.h | 27 +-
drivers/gpu/drm/i915/gem/i915_gem_region.c | 6 +-
drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 531 ++++++++++++++++++
drivers/gpu/drm/i915/gem/i915_gem_ttm.h | 50 ++
drivers/gpu/drm/i915/gt/intel_region_lmem.c | 3 +-
drivers/gpu/drm/i915/i915_gem.c | 5 +-
drivers/gpu/drm/i915/intel_memory_region.c | 1 -
drivers/gpu/drm/i915/intel_memory_region.h | 1 -
drivers/gpu/drm/i915/intel_region_ttm.c | 6 +-
drivers/gpu/drm/i915/intel_region_ttm.h | 8 +-
16 files changed, 717 insertions(+), 154 deletions(-)
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.c
create mode 100644 drivers/gpu/drm/i915/gem/i915_gem_ttm.h
diff --git a/drivers/gpu/drm/i915/Makefile b/drivers/gpu/drm/i915/Makefile
index 998606b7f49f..2022e763f8f7 100644
--- a/drivers/gpu/drm/i915/Makefile
+++ b/drivers/gpu/drm/i915/Makefile
@@ -154,6 +154,7 @@ gem-y += \
gem/i915_gem_stolen.o \
gem/i915_gem_throttle.o \
gem/i915_gem_tiling.o \
+ gem/i915_gem_ttm.o \
gem/i915_gem_userptr.o \
gem/i915_gem_wait.o \
gem/i915_gemfs.o
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_create.c b/drivers/gpu/drm/i915/gem/i915_gem_create.c
index 548ddf39d853..93bf63bbaff1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_create.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_create.c
@@ -85,13 +85,10 @@ i915_gem_setup(struct drm_i915_gem_object *obj, u64 size)
return -E2BIG;
/*
- * For now resort to CPU based clearing for device local-memory, in the
- * near future this will use the blitter engine for accelerated, GPU
- * based clearing.
+ * I915_BO_ALLOC_USER will make sure the object is cleared before
+ * any user access.
*/
- flags = 0;
- if (mr->type == INTEL_MEMORY_LOCAL)
- flags = I915_BO_ALLOC_CPU_CLEAR;
+ flags = I915_BO_ALLOC_USER;
ret = mr->ops->init_object(mr, obj, size, flags);
if (ret)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 3b4aa28a076d..2b8cd15de1d9 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -4,74 +4,10 @@
*/
#include "intel_memory_region.h"
-#include "intel_region_ttm.h"
#include "gem/i915_gem_region.h"
#include "gem/i915_gem_lmem.h"
#include "i915_drv.h"
-static void lmem_put_pages(struct drm_i915_gem_object *obj,
- struct sg_table *pages)
-{
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- obj->mm.dirty = false;
- sg_free_table(pages);
- kfree(pages);
-}
-
-static int lmem_get_pages(struct drm_i915_gem_object *obj)
-{
- unsigned int flags;
- struct sg_table *pages;
-
- flags = I915_ALLOC_MIN_PAGE_SIZE;
- if (obj->flags & I915_BO_ALLOC_CONTIGUOUS)
- flags |= I915_ALLOC_CONTIGUOUS;
-
- obj->mm.st_mm_node = intel_region_ttm_node_alloc(obj->mm.region,
- obj->base.size,
- flags);
- if (IS_ERR(obj->mm.st_mm_node))
- return PTR_ERR(obj->mm.st_mm_node);
-
- /* Range manager is always contigous */
- if (obj->mm.region->is_range_manager)
- obj->flags |= I915_BO_ALLOC_CONTIGUOUS;
- pages = intel_region_ttm_node_to_st(obj->mm.region, obj->mm.st_mm_node);
- if (IS_ERR(pages)) {
- intel_region_ttm_node_free(obj->mm.region, obj->mm.st_mm_node);
- return PTR_ERR(pages);
- }
-
- __i915_gem_object_set_pages(obj, pages, i915_sg_dma_sizes(pages->sgl));
-
- if (obj->flags & I915_BO_ALLOC_CPU_CLEAR) {
- void __iomem *vaddr =
- i915_gem_object_lmem_io_map(obj, 0, obj->base.size);
-
- if (!vaddr) {
- struct sg_table *pages =
- __i915_gem_object_unset_pages(obj);
-
- if (!IS_ERR_OR_NULL(pages))
- lmem_put_pages(obj, pages);
- }
-
- memset_io(vaddr, 0, obj->base.size);
- io_mapping_unmap(vaddr);
- }
-
- return 0;
-}
-
-const struct drm_i915_gem_object_ops i915_gem_lmem_obj_ops = {
- .name = "i915_gem_object_lmem",
- .flags = I915_GEM_OBJECT_HAS_IOMEM,
-
- .get_pages = lmem_get_pages,
- .put_pages = lmem_put_pages,
- .release = i915_gem_object_release_memory_region,
-};
-
void __iomem *
i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
unsigned long n,
@@ -103,23 +39,3 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
return i915_gem_object_create_region(i915->mm.regions[INTEL_REGION_LMEM],
size, flags);
}
-
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags)
-{
- static struct lock_class_key lock_class;
- struct drm_i915_private *i915 = mem->i915;
-
- drm_gem_private_object_init(&i915->drm, &obj->base, size);
- i915_gem_object_init(obj, &i915_gem_lmem_obj_ops, &lock_class, flags);
-
- obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
-
- i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
-
- i915_gem_object_init_memory_region(obj, mem);
-
- return 0;
-}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
index fac6bc5a5ebb..ea76fd11ccb0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.h
@@ -26,9 +26,4 @@ i915_gem_object_create_lmem(struct drm_i915_private *i915,
resource_size_t size,
unsigned int flags);
-int __i915_gem_lmem_object_init(struct intel_memory_region *mem,
- struct drm_i915_gem_object *obj,
- resource_size_t size,
- unsigned int flags);
-
#endif /* !__I915_GEM_LMEM_H */
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index e9247afb0320..df2b4e6b9bcc 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -173,7 +173,7 @@ static void i915_gem_close_object(struct drm_gem_object *gem, struct drm_file *f
}
}
-static void __i915_gem_free_object_rcu(struct rcu_head *head)
+void __i915_gem_free_object_rcu(struct rcu_head *head)
{
struct drm_i915_gem_object *obj =
container_of(head, typeof(*obj), rcu);
@@ -209,59 +209,69 @@ static void __i915_gem_object_free_mmaps(struct drm_i915_gem_object *obj)
}
}
-static void __i915_gem_free_objects(struct drm_i915_private *i915,
- struct llist_node *freed)
+void __i915_gem_free_object(struct drm_i915_gem_object *obj)
{
- struct drm_i915_gem_object *obj, *on;
+ trace_i915_gem_object_destroy(obj);
- llist_for_each_entry_safe(obj, on, freed, freed) {
- trace_i915_gem_object_destroy(obj);
+ if (!list_empty(&obj->vma.list)) {
+ struct i915_vma *vma;
+
+ /*
+ * Note that the vma keeps an object reference while
+ * it is active, so it *should* not sleep while we
+ * destroy it. Our debug code errs insits it *might*.
+ * For the moment, play along.
+ */
+ spin_lock(&obj->vma.lock);
+ while ((vma = list_first_entry_or_null(&obj->vma.list,
+ struct i915_vma,
+ obj_link))) {
+ GEM_BUG_ON(vma->obj != obj);
+ spin_unlock(&obj->vma.lock);
- if (!list_empty(&obj->vma.list)) {
- struct i915_vma *vma;
+ __i915_vma_put(vma);
- /*
- * Note that the vma keeps an object reference while
- * it is active, so it *should* not sleep while we
- * destroy it. Our debug code errs insits it *might*.
- * For the moment, play along.
- */
spin_lock(&obj->vma.lock);
- while ((vma = list_first_entry_or_null(&obj->vma.list,
- struct i915_vma,
- obj_link))) {
- GEM_BUG_ON(vma->obj != obj);
- spin_unlock(&obj->vma.lock);
+ }
+ spin_unlock(&obj->vma.lock);
+ }
- __i915_vma_put(vma);
+ __i915_gem_object_free_mmaps(obj);
- spin_lock(&obj->vma.lock);
- }
- spin_unlock(&obj->vma.lock);
- }
+ GEM_BUG_ON(!list_empty(&obj->lut_list));
- __i915_gem_object_free_mmaps(obj);
+ atomic_set(&obj->mm.pages_pin_count, 0);
+ __i915_gem_object_put_pages(obj);
+ GEM_BUG_ON(i915_gem_object_has_pages(obj));
+ bitmap_free(obj->bit_17);
- GEM_BUG_ON(!list_empty(&obj->lut_list));
+ if (obj->base.import_attach)
+ drm_prime_gem_destroy(&obj->base, NULL);
- atomic_set(&obj->mm.pages_pin_count, 0);
- __i915_gem_object_put_pages(obj);
- GEM_BUG_ON(i915_gem_object_has_pages(obj));
- bitmap_free(obj->bit_17);
+ drm_gem_free_mmap_offset(&obj->base);
- if (obj->base.import_attach)
- drm_prime_gem_destroy(&obj->base, NULL);
+ if (obj->ops->release)
+ obj->ops->release(obj);
- drm_gem_free_mmap_offset(&obj->base);
+ if (obj->mm.n_placements > 1)
+ kfree(obj->mm.placements);
- if (obj->ops->release)
- obj->ops->release(obj);
+ if (obj->shares_resv_from)
+ i915_vm_resv_put(obj->shares_resv_from);
+}
- if (obj->mm.n_placements > 1)
- kfree(obj->mm.placements);
+static void __i915_gem_free_objects(struct drm_i915_private *i915,
+ struct llist_node *freed)
+{
+ struct drm_i915_gem_object *obj, *on;
- if (obj->shares_resv_from)
- i915_vm_resv_put(obj->shares_resv_from);
+ llist_for_each_entry_safe(obj, on, freed, freed) {
+ might_sleep();
+ if (obj->ops->delayed_free) {
+ obj->ops->delayed_free(obj);
+ continue;
+ }
+ __i915_gem_free_object(obj);
/* But keep the pointer alive for RCU-protected lookups */
call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
@@ -319,6 +329,7 @@ static void i915_gem_free_object(struct drm_gem_object *gem_obj)
* worker and performing frees directly from subsequent allocations for
* crude but effective memory throttling.
*/
+
if (llist_add(&obj->freed, &i915->mm.free_list))
queue_work(i915->wq, &i915->mm.free_work);
}
@@ -411,6 +422,42 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
return 0;
}
+/**
+ * i915_gem_object_evictable - Whether object is likely evictable after unbind.
+ * @obj: The object to check
+ *
+ * This function checks whether the object is likely unvictable after unbind.
+ * If the object is not locked when checking, the result is only advisory.
+ * If the object is locked when checking, and the function returns true,
+ * then an eviction should indeed be possible. But since unlocked vma
+ * unpinning and unbinding is currently possible, the object can actually
+ * become evictable even if this function returns false.
+ *
+ * Return: true if the object may be evictable. False otherwise.
+ */
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
+{
+ struct i915_vma *vma;
+ int pin_count = atomic_read(&obj->mm.pages_pin_count);
+
+ if (!pin_count)
+ return true;
+
+ spin_lock(&obj->vma.lock);
+ list_for_each_entry(vma, &obj->vma.list, obj_link) {
+ if (i915_vma_is_pinned(vma)) {
+ spin_unlock(&obj->vma.lock);
+ return false;
+ }
+ if (atomic_read(&vma->pages_count))
+ pin_count--;
+ }
+ spin_unlock(&obj->vma.lock);
+ GEM_WARN_ON(pin_count < 0);
+
+ return pin_count == 0;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index 2ebd79537aea..ae5930e307d5 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -200,6 +200,9 @@ static inline bool i915_gem_object_trylock(struct drm_i915_gem_object *obj)
static inline void i915_gem_object_unlock(struct drm_i915_gem_object *obj)
{
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+
dma_resv_unlock(obj->base.resv);
}
@@ -587,6 +590,12 @@ int i915_gem_object_read_from_page(struct drm_i915_gem_object *obj, u64 offset,
bool i915_gem_object_is_shmem(const struct drm_i915_gem_object *obj);
+void __i915_gem_free_object_rcu(struct rcu_head *head);
+
+void __i915_gem_free_object(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
index d047ea126029..68313474e6a6 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object_types.h
@@ -63,6 +63,20 @@ struct drm_i915_gem_object_ops {
const struct drm_i915_gem_pwrite *arg);
int (*dmabuf_export)(struct drm_i915_gem_object *obj);
+
+ /**
+ * adjust_lru - notify that the madvise value was updated
+ * @obj: The gem object
+ *
+ * The madvise value may have been updated, or object was recently
+ * referenced so act accordingly (Perhaps changing an LRU list etc).
+ */
+ void (*adjust_lru)(struct drm_i915_gem_object *obj);
+
+ /**
+ * delayed_free - Override the default delayed free implementation
+ */
+ void (*delayed_free)(struct drm_i915_gem_object *obj);
void (*release)(struct drm_i915_gem_object *obj);
const char *name; /* friendly name for debug, e.g. lockdep classes */
@@ -187,12 +201,14 @@ struct drm_i915_gem_object {
#define I915_BO_ALLOC_VOLATILE BIT(1)
#define I915_BO_ALLOC_STRUCT_PAGE BIT(2)
#define I915_BO_ALLOC_CPU_CLEAR BIT(3)
+#define I915_BO_ALLOC_USER BIT(4)
#define I915_BO_ALLOC_FLAGS (I915_BO_ALLOC_CONTIGUOUS | \
I915_BO_ALLOC_VOLATILE | \
I915_BO_ALLOC_STRUCT_PAGE | \
- I915_BO_ALLOC_CPU_CLEAR)
-#define I915_BO_READONLY BIT(4)
-#define I915_TILING_QUIRK_BIT 5 /* unknown swizzling; do not release! */
+ I915_BO_ALLOC_CPU_CLEAR | \
+ I915_BO_ALLOC_USER)
+#define I915_BO_READONLY BIT(5)
+#define I915_TILING_QUIRK_BIT 6 /* unknown swizzling; do not release! */
/*
* Is the object to be mapped as read-only to the GPU
@@ -310,6 +326,11 @@ struct drm_i915_gem_object {
bool dirty:1;
} mm;
+ struct {
+ struct sg_table *cached_io_st;
+ bool created:1;
+ } ttm;
+
/** Record of address bit 17 of each page at last unbind. */
unsigned long *bit_17;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_region.c b/drivers/gpu/drm/i915/gem/i915_gem_region.c
index f25e6646c5b7..d1f1840540dd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_region.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_region.c
@@ -18,11 +18,7 @@ void i915_gem_object_init_memory_region(struct drm_i915_gem_object *obj,
mutex_lock(&mem->objects.lock);
- if (obj->flags & I915_BO_ALLOC_VOLATILE)
- list_add(&obj->mm.region_link, &mem->objects.purgeable);
- else
- list_add(&obj->mm.region_link, &mem->objects.list);
-
+ list_add(&obj->mm.region_link, &mem->objects.list);
mutex_unlock(&mem->objects.lock);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
new file mode 100644
index 000000000000..3cec777be346
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -0,0 +1,531 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+
+#include <drm/ttm/ttm_bo_driver.h>
+#include <drm/ttm/ttm_placement.h>
+
+#include "i915_drv.h"
+#include "intel_memory_region.h"
+#include "intel_region_ttm.h"
+
+#include "gem/i915_gem_object.h"
+#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
+
+#define I915_PL_LMEM0 TTM_PL_PRIV
+#define I915_PL_SYSTEM TTM_PL_SYSTEM
+#define I915_PL_STOLEN TTM_PL_VRAM
+#define I915_PL_GGTT TTM_PL_TT
+
+#define I915_TTM_PRIO_PURGE 0
+#define I915_TTM_PRIO_NO_PAGES 1
+#define I915_TTM_PRIO_HAS_PAGES 2
+
+/**
+ * struct i915_ttm_tt - TTM page vector with additional private information
+ * @ttm: The base TTM page vector.
+ * @dev: The struct device used for dma mapping and unmapping.
+ * @cached_st: The cached scatter-gather table.
+ *
+ * Note that DMA may be going on right up to the point where the page-
+ * vector is unpopulated in delayed destroy. Hence keep the
+ * scatter-gather table mapped and cached up to that point. This is
+ * different from the cached gem object io scatter-gather table which
+ * doesn't have an associated dma mapping.
+ */
+struct i915_ttm_tt {
+ struct ttm_tt ttm;
+ struct device *dev;
+ struct sg_table *cached_st;
+};
+
+static const struct ttm_place lmem0_sys_placement_flags[] = {
+ {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_LMEM0,
+ .flags = 0,
+ }, {
+ .fpfn = 0,
+ .lpfn = 0,
+ .mem_type = I915_PL_SYSTEM,
+ .flags = 0,
+ }
+};
+
+static struct ttm_placement i915_lmem0_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[0],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[0],
+};
+
+static struct ttm_placement i915_sys_placement = {
+ .num_placement = 1,
+ .placement = &lmem0_sys_placement_flags[1],
+ .num_busy_placement = 1,
+ .busy_placement = &lmem0_sys_placement_flags[1],
+};
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj);
+
+static struct ttm_tt *i915_ttm_tt_create(struct ttm_buffer_object *bo,
+ uint32_t page_flags)
+{
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct i915_ttm_tt *i915_tt;
+ int ret;
+
+ i915_tt = kzalloc(sizeof(*i915_tt), GFP_KERNEL);
+ if (!i915_tt)
+ return NULL;
+
+ if (obj->flags & I915_BO_ALLOC_CPU_CLEAR &&
+ man->use_tt)
+ page_flags |= TTM_PAGE_FLAG_ZERO_ALLOC;
+
+ ret = ttm_tt_init(&i915_tt->ttm, bo, page_flags, ttm_write_combined);
+ if (ret) {
+ kfree(i915_tt);
+ return NULL;
+ }
+
+ i915_tt->dev = obj->base.dev->dev;
+
+ return &i915_tt->ttm;
+}
+
+static void i915_ttm_tt_unpopulate(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ if (i915_tt->cached_st) {
+ dma_unmap_sgtable(i915_tt->dev, i915_tt->cached_st,
+ DMA_BIDIRECTIONAL, 0);
+ sg_free_table(i915_tt->cached_st);
+ kfree(i915_tt->cached_st);
+ i915_tt->cached_st = NULL;
+ }
+ ttm_pool_free(&bdev->pool, ttm);
+}
+
+static void i915_ttm_tt_destroy(struct ttm_device *bdev, struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+
+ ttm_tt_destroy_common(bdev, ttm);
+ kfree(i915_tt);
+}
+
+static bool i915_ttm_eviction_valuable(struct ttm_buffer_object *bo,
+ const struct ttm_place *place)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ /* Will do for now. Our pinned objects are still on TTM's LRU lists */
+ if (!i915_gem_object_evictable(obj))
+ return false;
+
+ /* This isn't valid with a buddy allocator */
+ return ttm_bo_eviction_valuable(bo, place);
+}
+
+static void i915_ttm_evict_flags(struct ttm_buffer_object *bo,
+ struct ttm_placement *placement)
+{
+ *placement = i915_sys_placement;
+}
+
+static int i915_ttm_move_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret;
+
+ ret = i915_gem_object_unbind(obj, I915_GEM_OBJECT_UNBIND_ACTIVE);
+ if (ret)
+ return ret;
+
+ ret = __i915_gem_object_put_pages(obj);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static void i915_ttm_free_cached_io_st(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.cached_io_st) {
+ sg_free_table(obj->ttm.cached_io_st);
+ kfree(obj->ttm.cached_io_st);
+ obj->ttm.cached_io_st = NULL;
+ }
+}
+
+static void i915_ttm_purge(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct ttm_placement place = {};
+ int ret;
+
+ if (obj->mm.madv == __I915_MADV_PURGED)
+ return;
+
+ /* TTM's purge interface. Note that we might be reentering. */
+ ret = ttm_bo_validate(bo, &place, &ctx);
+
+ if (!ret) {
+ i915_ttm_free_cached_io_st(obj);
+ obj->mm.madv = __I915_MADV_PURGED;
+ }
+}
+
+static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ int ret = i915_ttm_move_notify(bo);
+
+ GEM_WARN_ON(ret);
+ GEM_WARN_ON(obj->ttm.cached_io_st);
+ if (!ret && obj->mm.madv != I915_MADV_WILLNEED)
+ i915_ttm_purge(obj);
+}
+
+static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ if (likely(obj)) {
+ /* This releases all gem object bindings to the backend. */
+ __i915_gem_free_object(obj);
+ }
+}
+
+static struct intel_memory_region *
+i915_ttm_region(struct ttm_device *bdev, int ttm_mem_type)
+{
+ struct drm_i915_private *i915 = container_of(bdev, typeof(*i915), bdev);
+
+ /* There's some room for optimization here... */
+ GEM_BUG_ON(ttm_mem_type != I915_PL_SYSTEM &&
+ ttm_mem_type < I915_PL_LMEM0);
+ if (ttm_mem_type == I915_PL_SYSTEM)
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_SYSTEM,
+ 0);
+
+ return intel_memory_region_lookup(i915, INTEL_MEMORY_LOCAL,
+ ttm_mem_type - I915_PL_LMEM0);
+}
+
+static struct sg_table *i915_ttm_tt_get_st(struct ttm_tt *ttm)
+{
+ struct i915_ttm_tt *i915_tt = container_of(ttm, typeof(*i915_tt), ttm);
+ struct scatterlist *sg;
+ struct sg_table *st;
+ int ret;
+
+ if (i915_tt->cached_st)
+ return i915_tt->cached_st;
+
+ st = kzalloc(sizeof(*st), GFP_KERNEL);
+ if (!st)
+ return ERR_PTR(-ENOMEM);
+
+ sg = __sg_alloc_table_from_pages
+ (st, ttm->pages, ttm->num_pages, 0,
+ (unsigned long)ttm->num_pages << PAGE_SHIFT,
+ i915_sg_segment_size(), NULL, 0, GFP_KERNEL);
+ if (IS_ERR(sg)) {
+ kfree(st);
+ return ERR_CAST(sg);
+ }
+
+ ret = dma_map_sgtable(i915_tt->dev, st, DMA_BIDIRECTIONAL, 0);
+ if (ret) {
+ sg_free_table(st);
+ kfree(st);
+ return ERR_PTR(ret);
+ }
+
+ i915_tt->cached_st = st;
+ return st;
+}
+
+static struct sg_table *
+i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
+ struct ttm_resource *res)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_resource_manager *man =
+ ttm_manager_type(bo->bdev, res->mem_type);
+
+ if (man->use_tt)
+ return i915_ttm_tt_get_st(bo->ttm);
+
+ return intel_region_ttm_node_to_st(obj->mm.region, res->mm_node);
+}
+
+static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
+ struct ttm_operation_ctx *ctx,
+ struct ttm_resource *dst_mem,
+ struct ttm_place *hop)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+ struct ttm_resource_manager *dst_man =
+ ttm_manager_type(bo->bdev, dst_mem->mem_type);
+ struct ttm_resource_manager *src_man =
+ ttm_manager_type(bo->bdev, bo->mem.mem_type);
+ struct intel_memory_region *dst_reg, *src_reg;
+ union {
+ struct ttm_kmap_iter_tt tt;
+ struct ttm_kmap_iter_iomap io;
+ } _dst_iter, _src_iter;
+ struct ttm_kmap_iter *dst_iter, *src_iter;
+ struct sg_table *dst_st;
+ int ret;
+
+ dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
+ src_reg = i915_ttm_region(bo->bdev, bo->mem.mem_type);
+ GEM_BUG_ON(!dst_reg || !src_reg);
+
+ /* Sync for now. We could do the actual copy async. */
+ ret = ttm_bo_wait_ctx(bo, ctx);
+ if (ret)
+ return ret;
+
+ ret = i915_ttm_move_notify(bo);
+ if (ret)
+ return ret;
+
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ i915_ttm_purge(obj);
+ ttm_resource_free(bo, dst_mem);
+ return 0;
+ }
+
+ /* Populate ttm with pages if needed. Typically system memory. */
+ if (bo->ttm && (dst_man->use_tt ||
+ (bo->ttm->page_flags & TTM_PAGE_FLAG_SWAPPED))) {
+ ret = ttm_tt_populate(bo->bdev, bo->ttm, ctx);
+ if (ret)
+ return ret;
+ }
+
+ dst_st = i915_ttm_resource_get_st(obj, dst_mem);
+ if (IS_ERR(dst_st))
+ return PTR_ERR(dst_st);
+
+ /* If we start mapping GGTT, we can no longer use man::use_tt here. */
+ dst_iter = dst_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+ dst_st, dst_reg->region.start);
+
+ src_iter = src_man->use_tt ?
+ ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
+ ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+ obj->ttm.cached_io_st,
+ src_reg->region.start);
+
+ ttm_move_memcpy(bo, dst_mem, dst_iter, src_iter);
+ ttm_bo_move_sync_cleanup(bo, dst_mem);
+ i915_ttm_free_cached_io_st(obj);
+
+ if (!dst_man->use_tt)
+ obj->ttm.cached_io_st = dst_st;
+
+ return 0;
+}
+
+struct ttm_device_funcs i915_ttm_bo_driver = {
+ .ttm_tt_create = i915_ttm_tt_create,
+ .ttm_tt_unpopulate = i915_ttm_tt_unpopulate,
+ .ttm_tt_destroy = i915_ttm_tt_destroy,
+ .eviction_valuable = i915_ttm_eviction_valuable,
+ .evict_flags = i915_ttm_evict_flags,
+ .move = i915_ttm_move,
+ .verify_access = NULL,
+ .swap_notify = i915_ttm_swap_notify,
+ .delete_mem_notify = i915_ttm_delete_mem_notify,
+};
+
+static int i915_ttm_get_pages(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+ struct ttm_operation_ctx ctx = {
+ .interruptible = true,
+ .no_wait_gpu = false,
+ };
+ struct sg_table *st;
+ int ret;
+
+ /* Move to the requested placement. */
+ ret = ttm_bo_validate(bo, &i915_lmem0_placement, &ctx);
+ if (ret)
+ return ret == -ENOSPC ? -ENXIO : ret;
+
+ /* Object either has a page vector or is an iomem object */
+ st = bo->ttm ? i915_ttm_tt_get_st(bo->ttm) : obj->ttm.cached_io_st;
+ if (IS_ERR(st))
+ return PTR_ERR(st);
+
+ __i915_gem_object_set_pages(obj, st, i915_sg_dma_sizes(st->sgl));
+
+ i915_ttm_adjust_lru(obj);
+
+ return ret;
+}
+
+static void i915_ttm_put_pages(struct drm_i915_gem_object *obj,
+ struct sg_table *st)
+{
+ /*
+ * We're currently not called from a shrinker, so put_pages()
+ * typically means the object is about to destroyed, or called
+ * from move_notify(). So just avoid doing much for now.
+ * If the object is not destroyed next, The TTM eviction logic
+ * and shrinkers will move it out if needed.
+ */
+
+ i915_ttm_adjust_lru(obj);
+}
+
+static void i915_ttm_adjust_lru(struct drm_i915_gem_object *obj)
+{
+ struct ttm_buffer_object *bo = i915_gem_to_ttm(obj);
+
+ /*
+ * Don't manipulate the TTM LRUs while in TTM bo destruction.
+ * We're called through i915_ttm_delete_mem_notify().
+ */
+ if (!kref_read(&bo->kref))
+ return;
+
+ /*
+ * Put on the correct LRU list depending on the MADV status
+ */
+ spin_lock(&bo->bdev->lru_lock);
+ if (obj->mm.madv != I915_MADV_WILLNEED) {
+ bo->priority = I915_TTM_PRIO_PURGE;
+ } else if (!i915_gem_object_has_pages(obj)) {
+ if (bo->priority < I915_TTM_PRIO_HAS_PAGES)
+ bo->priority = I915_TTM_PRIO_HAS_PAGES;
+ } else {
+ if (bo->priority > I915_TTM_PRIO_NO_PAGES)
+ bo->priority = I915_TTM_PRIO_NO_PAGES;
+ }
+
+ ttm_bo_move_to_lru_tail(bo, &bo->mem, NULL);
+ spin_unlock(&bo->bdev->lru_lock);
+}
+
+/*
+ * TTM-backed gem object destruction requires some clarification.
+ * Basically we have two possibilities here. We can either rely on the
+ * i915 delayed destruction and put the TTM object when the object
+ * is idle. This would be detected by TTM which would bypass the
+ * TTM delayed destroy handling. The other approach is to put the TTM
+ * object early and rely on the TTM destroyed handling, and then free
+ * the leftover parts of the GEM object once TTM's destroyed list handling is
+ * complete. For now, we rely on the latter for two reasons:
+ * a) TTM can evict an object even when it's on the delayed destroy list,
+ * which in theory allows for complete eviction.
+ * b) There is work going on in TTM to allow freeing an object even when
+ * it's not idle, and using the TTM destroyed list handling could help us
+ * benefit from that.
+ */
+static void i915_ttm_delayed_free(struct drm_i915_gem_object *obj)
+{
+ if (obj->ttm.created) {
+ ttm_bo_put(i915_gem_to_ttm(obj));
+ } else {
+ __i915_gem_free_object(obj);
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+ }
+}
+
+static const struct drm_i915_gem_object_ops i915_gem_ttm_obj_ops = {
+ .name = "i915_gem_object_ttm",
+ .flags = I915_GEM_OBJECT_HAS_IOMEM,
+
+ .get_pages = i915_ttm_get_pages,
+ .put_pages = i915_ttm_put_pages,
+ .truncate = i915_ttm_purge,
+ .adjust_lru = i915_ttm_adjust_lru,
+ .delayed_free = i915_ttm_delayed_free,
+};
+
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
+{
+ struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+
+ i915_gem_object_release_memory_region(obj);
+ if (obj->ttm.created)
+ call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
+}
+
+/**
+ * __i915_gem_ttm_object_init - Initialize a ttm-backed i915 gem object
+ * @mem: The initial memory region for the object.
+ * @obj: The gem object.
+ * @size: Object size in bytes.
+ * @flags: gem object flags.
+ *
+ * Return: 0 on success, negative error code on failure.
+ */
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags)
+{
+ static struct lock_class_key lock_class;
+ struct drm_i915_private *i915 = mem->i915;
+ enum ttm_bo_type bo_type;
+ size_t alignment = 0;
+ int ret;
+
+ /* Adjust alignment to GPU- and CPU huge page sizes. */
+
+ if (mem->is_range_manager) {
+ if (size >= SZ_1G)
+ alignment = SZ_1G >> PAGE_SHIFT;
+ else if (size >= SZ_2M)
+ alignment = SZ_2M >> PAGE_SHIFT;
+ else if (size >= SZ_64K)
+ alignment = SZ_64K >> PAGE_SHIFT;
+ }
+
+ drm_gem_private_object_init(&i915->drm, &obj->base, size);
+ i915_gem_object_init(obj, &i915_gem_ttm_obj_ops, &lock_class, flags);
+ i915_gem_object_init_memory_region(obj, mem);
+ i915_gem_object_make_unshrinkable(obj);
+ obj->read_domains = I915_GEM_DOMAIN_WC | I915_GEM_DOMAIN_GTT;
+ i915_gem_object_set_cache_coherency(obj, I915_CACHE_NONE);
+
+ bo_type = (obj->flags & I915_BO_ALLOC_USER) ? ttm_bo_type_device :
+ ttm_bo_type_kernel;
+
+ /*
+ * If this function fails, it will call the destructor, but
+ * our caller still owns the object. So no freeing in the
+ * destructor until obj->ttm.created is true.
+ * Similarly, in delayed_destroy, we can't call ttm_bo_put()
+ * until successful initialization.
+ */
+ ret = ttm_bo_init(&i915->bdev, i915_gem_to_ttm(obj), size,
+ bo_type, &i915_sys_placement, alignment,
+ true, NULL, NULL, i915_ttm_bo_destroy);
+
+ if (!ret)
+ obj->ttm.created = true;
+
+ /* i915 wants -ENXIO when out of memory region space. */
+ return (ret == -ENOSPC) ? -ENXIO : ret;
+}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.h b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
new file mode 100644
index 000000000000..05c114e35b4d
--- /dev/null
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.h
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: MIT */
+/*
+ * Copyright © 2021 Intel Corporation
+ */
+#ifndef _I915_GEM_TTM_H_
+#define _I915_GEM_TTM_H_
+
+#include "gem/i915_gem_object_types.h"
+
+/**
+ * i915_gem_to_ttm - Convert a struct drm_i915_gem_object to a
+ * struct ttm_buffer_object.
+ * @obj: Pointer to the gem object.
+ *
+ * Return: Pointer to the embedded struct ttm_buffer_object.
+ */
+static inline struct ttm_buffer_object *
+i915_gem_to_ttm(struct drm_i915_gem_object *obj)
+{
+ return &obj->__do_not_access;
+}
+
+/*
+ * i915 ttm gem object destructor. Internal use only.
+ */
+void i915_ttm_bo_destroy(struct ttm_buffer_object *bo);
+
+/**
+ * i915_ttm_to_gem - Convert a struct ttm_buffer_object to an embedding
+ * struct drm_i915_gem_object.
+ *
+ * Return: Pointer to the embedding struct ttm_buffer_object, or NULL
+ * if the object was not an i915 ttm object.
+ */
+static inline struct drm_i915_gem_object *
+i915_ttm_to_gem(struct ttm_buffer_object *bo)
+{
+ if (GEM_WARN_ON(bo->destroy != i915_ttm_bo_destroy))
+ return NULL;
+
+ return container_of(bo, struct drm_i915_gem_object, __do_not_access);
+}
+
+int __i915_gem_ttm_object_init(struct intel_memory_region *mem,
+ struct drm_i915_gem_object *obj,
+ resource_size_t size,
+ unsigned int flags);
+
+extern struct ttm_device_funcs i915_ttm_bo_driver;
+#endif
diff --git a/drivers/gpu/drm/i915/gt/intel_region_lmem.c b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
index f7366b054f8e..4ae1f717a94c 100644
--- a/drivers/gpu/drm/i915/gt/intel_region_lmem.c
+++ b/drivers/gpu/drm/i915/gt/intel_region_lmem.c
@@ -9,6 +9,7 @@
#include "intel_region_ttm.h"
#include "gem/i915_gem_lmem.h"
#include "gem/i915_gem_region.h"
+#include "gem/i915_gem_ttm.h"
#include "intel_region_lmem.h"
static int init_fake_lmem_bar(struct intel_memory_region *mem)
@@ -107,7 +108,7 @@ region_lmem_init(struct intel_memory_region *mem)
static const struct intel_memory_region_ops intel_region_lmem_ops = {
.init = region_lmem_init,
.release = region_lmem_release,
- .init_object = __i915_gem_lmem_object_init,
+ .init_object = __i915_gem_ttm_object_init,
};
struct intel_memory_region *
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 180f6e9107d4..350283ab9a83 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1005,8 +1005,11 @@ i915_gem_madvise_ioctl(struct drm_device *dev, void *data,
}
}
- if (obj->mm.madv != __I915_MADV_PURGED)
+ if (obj->mm.madv != __I915_MADV_PURGED) {
obj->mm.madv = args->madv;
+ if (obj->ops->adjust_lru)
+ obj->ops->adjust_lru(obj);
+ }
if (i915_gem_object_has_pages(obj)) {
struct list_head *list;
diff --git a/drivers/gpu/drm/i915/intel_memory_region.c b/drivers/gpu/drm/i915/intel_memory_region.c
index 4092cc987679..bd27e897d4d0 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.c
+++ b/drivers/gpu/drm/i915/intel_memory_region.c
@@ -149,7 +149,6 @@ intel_memory_region_create(struct drm_i915_private *i915,
mutex_init(&mem->objects.lock);
INIT_LIST_HEAD(&mem->objects.list);
- INIT_LIST_HEAD(&mem->objects.purgeable);
INIT_LIST_HEAD(&mem->reserved);
mutex_init(&mem->mm_lock);
diff --git a/drivers/gpu/drm/i915/intel_memory_region.h b/drivers/gpu/drm/i915/intel_memory_region.h
index e69cde13daf2..7b5fa97c0b59 100644
--- a/drivers/gpu/drm/i915/intel_memory_region.h
+++ b/drivers/gpu/drm/i915/intel_memory_region.h
@@ -100,7 +100,6 @@ struct intel_memory_region {
struct {
struct mutex lock; /* Protects access to objects */
struct list_head list;
- struct list_head purgeable;
} objects;
size_t chunk_size;
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.c b/drivers/gpu/drm/i915/intel_region_ttm.c
index 69c98298326b..52352dbe047d 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.c
+++ b/drivers/gpu/drm/i915/intel_region_ttm.c
@@ -10,6 +10,7 @@
#include "intel_region_ttm.h"
+#include "gem/i915_gem_ttm.h" /* For the funcs/ops export only */
/**
* DOC: TTM support structure
*
@@ -31,9 +32,6 @@ struct intel_region_ttm_private {
I915_SELFTEST_DECLARE(struct drm_vma_offset_manager mock_vma);
};
-/* A Zero-initialized driver for now. We don't have a TTM backend yet. */
-static struct ttm_device_funcs i915_ttm_bo_driver;
-
static struct intel_region_ttm_private *to_ttm_private(void *data)
{
return (struct intel_region_ttm_private *)data;
@@ -219,6 +217,7 @@ struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
return i915_sg_from_mm_node(node, mem->region.start);
}
+#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_node_alloc - Allocate memory resources from a region
* @mem: The memory region,
@@ -267,7 +266,6 @@ void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
return ret ? ERR_PTR(ret) : res.mm_node;
}
-#ifdef CONFIG_DRM_I915_SELFTEST
/**
* intel_region_ttm_init_mock - Initialize a mock memory region for TTM.
* @mem: The region to initialize.
diff --git a/drivers/gpu/drm/i915/intel_region_ttm.h b/drivers/gpu/drm/i915/intel_region_ttm.h
index 23c6c2d4b1cd..a08998c3a3d8 100644
--- a/drivers/gpu/drm/i915/intel_region_ttm.h
+++ b/drivers/gpu/drm/i915/intel_region_ttm.h
@@ -23,14 +23,14 @@ void intel_region_ttm_fini(struct intel_memory_region *mem);
struct sg_table *intel_region_ttm_node_to_st(struct intel_memory_region *mem,
void *node);
-void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
- resource_size_t size,
- unsigned int flags);
-
void intel_region_ttm_node_free(struct intel_memory_region *mem,
void *node);
#ifdef CONFIG_DRM_I915_SELFTEST
+void *intel_region_ttm_node_alloc(struct intel_memory_region *mem,
+ resource_size_t size,
+ unsigned int flags);
+
int intel_region_ttm_init_mock(struct intel_memory_region *mem);
void intel_region_ttm_fini_mock(struct intel_memory_region *mem);
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH v3 12/12] drm/i915/lmem: Verify checks for lmem residency
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
@ 2021-05-21 15:32 ` Thomas Hellström
-1 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Since objects can be migrated or evicted when not pinned or locked,
update the checks for lmem residency or future residency so that
the value returned is not immediately stale.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2: Simplify i915_gem_object_migratable() (Reported by Mattew Auld)
---
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 42 +++++++++++++++++++-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 18 +++++++++
drivers/gpu/drm/i915/gem/i915_gem_object.h | 4 ++
4 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index de1f13d203b5..b95def2d5af3 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -11615,7 +11615,7 @@ intel_user_framebuffer_create(struct drm_device *dev,
/* object is backed with LMEM for discrete */
i915 = to_i915(obj->base.dev);
- if (HAS_LMEM(i915) && !i915_gem_object_is_lmem(obj)) {
+ if (HAS_LMEM(i915) && !i915_gem_object_validates_to_lmem(obj)) {
/* object is "remote", not in local memory */
i915_gem_object_put(obj);
return ERR_PTR(-EREMOTE);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 2b8cd15de1d9..d539dffa1554 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -23,10 +23,50 @@ i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
return io_mapping_map_wc(&obj->mm.region->iomap, offset, size);
}
+/**
+ * i915_gem_object_validates_to_lmem - Whether the object is resident in
+ * lmem when pages are present.
+ * @obj: The object to check.
+ *
+ * Migratable objects residency may change from under us if the object is
+ * not pinned or locked. This function is intended to be used to check whether
+ * the object can only reside in lmem when pages are present.
+ *
+ * Return: Whether the object is always resident in lmem when pages are
+ * present.
+ */
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ return !i915_gem_object_migratable(obj) &&
+ mr && (mr->type == INTEL_MEMORY_LOCAL ||
+ mr->type == INTEL_MEMORY_STOLEN_LOCAL);
+}
+
+/**
+ * i915_gem_object_is_lmem - Whether the object is resident in
+ * lmem
+ * @obj: The object to check.
+ *
+ * Even if an object is allowed to migrate and change memory region,
+ * this function checks whether it will always be present in lmem when
+ * valid *or* if that's not the case, whether it's currently resident in lmem.
+ * For migratable and evictable objects, the latter only makes sense when
+ * the object is locked.
+ *
+ * Return: Whether the object migratable but resident in lmem, or not
+ * migratable and will be present in lmem when valid.
+ */
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj)
{
- struct intel_memory_region *mr = obj->mm.region;
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+#ifdef CONFIG_LOCKDEP
+ if (i915_gem_object_migratable(obj) &&
+ i915_gem_object_evictable(obj))
+ assert_object_held(obj);
+#endif
return mr && (mr->type == INTEL_MEMORY_LOCAL ||
mr->type == INTEL_MEMORY_STOLEN_LOCAL);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index df2b4e6b9bcc..c8bb6fb1dba3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -458,6 +458,24 @@ bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
return pin_count == 0;
}
+/**
+ * i915_gem_object_migratable - Whether the object is migratable out of the
+ * current region.
+ * @obj: Pointer to the object.
+ *
+ * Return: Whether the object is allowed to be resident in other
+ * regions than the current while pages are present.
+ */
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ if (!mr)
+ return false;
+
+ return obj->mm.n_placements > 1;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index ae5930e307d5..a3ad8cf4eefd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -596,6 +596,10 @@ void __i915_gem_free_object(struct drm_i915_gem_object *obj);
bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
--
2.31.1
^ permalink raw reply related [flat|nested] 60+ messages in thread* [Intel-gfx] [PATCH v3 12/12] drm/i915/lmem: Verify checks for lmem residency
@ 2021-05-21 15:32 ` Thomas Hellström
0 siblings, 0 replies; 60+ messages in thread
From: Thomas Hellström @ 2021-05-21 15:32 UTC (permalink / raw)
To: intel-gfx, dri-devel; +Cc: Thomas Hellström, Matthew Auld
Since objects can be migrated or evicted when not pinned or locked,
update the checks for lmem residency or future residency so that
the value returned is not immediately stale.
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>
---
v2: Simplify i915_gem_object_migratable() (Reported by Mattew Auld)
---
drivers/gpu/drm/i915/display/intel_display.c | 2 +-
drivers/gpu/drm/i915/gem/i915_gem_lmem.c | 42 +++++++++++++++++++-
drivers/gpu/drm/i915/gem/i915_gem_object.c | 18 +++++++++
drivers/gpu/drm/i915/gem/i915_gem_object.h | 4 ++
4 files changed, 64 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index de1f13d203b5..b95def2d5af3 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -11615,7 +11615,7 @@ intel_user_framebuffer_create(struct drm_device *dev,
/* object is backed with LMEM for discrete */
i915 = to_i915(obj->base.dev);
- if (HAS_LMEM(i915) && !i915_gem_object_is_lmem(obj)) {
+ if (HAS_LMEM(i915) && !i915_gem_object_validates_to_lmem(obj)) {
/* object is "remote", not in local memory */
i915_gem_object_put(obj);
return ERR_PTR(-EREMOTE);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
index 2b8cd15de1d9..d539dffa1554 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_lmem.c
@@ -23,10 +23,50 @@ i915_gem_object_lmem_io_map(struct drm_i915_gem_object *obj,
return io_mapping_map_wc(&obj->mm.region->iomap, offset, size);
}
+/**
+ * i915_gem_object_validates_to_lmem - Whether the object is resident in
+ * lmem when pages are present.
+ * @obj: The object to check.
+ *
+ * Migratable objects residency may change from under us if the object is
+ * not pinned or locked. This function is intended to be used to check whether
+ * the object can only reside in lmem when pages are present.
+ *
+ * Return: Whether the object is always resident in lmem when pages are
+ * present.
+ */
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ return !i915_gem_object_migratable(obj) &&
+ mr && (mr->type == INTEL_MEMORY_LOCAL ||
+ mr->type == INTEL_MEMORY_STOLEN_LOCAL);
+}
+
+/**
+ * i915_gem_object_is_lmem - Whether the object is resident in
+ * lmem
+ * @obj: The object to check.
+ *
+ * Even if an object is allowed to migrate and change memory region,
+ * this function checks whether it will always be present in lmem when
+ * valid *or* if that's not the case, whether it's currently resident in lmem.
+ * For migratable and evictable objects, the latter only makes sense when
+ * the object is locked.
+ *
+ * Return: Whether the object migratable but resident in lmem, or not
+ * migratable and will be present in lmem when valid.
+ */
bool i915_gem_object_is_lmem(struct drm_i915_gem_object *obj)
{
- struct intel_memory_region *mr = obj->mm.region;
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+#ifdef CONFIG_LOCKDEP
+ if (i915_gem_object_migratable(obj) &&
+ i915_gem_object_evictable(obj))
+ assert_object_held(obj);
+#endif
return mr && (mr->type == INTEL_MEMORY_LOCAL ||
mr->type == INTEL_MEMORY_STOLEN_LOCAL);
}
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index df2b4e6b9bcc..c8bb6fb1dba3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -458,6 +458,24 @@ bool i915_gem_object_evictable(struct drm_i915_gem_object *obj)
return pin_count == 0;
}
+/**
+ * i915_gem_object_migratable - Whether the object is migratable out of the
+ * current region.
+ * @obj: Pointer to the object.
+ *
+ * Return: Whether the object is allowed to be resident in other
+ * regions than the current while pages are present.
+ */
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj)
+{
+ struct intel_memory_region *mr = READ_ONCE(obj->mm.region);
+
+ if (!mr)
+ return false;
+
+ return obj->mm.n_placements > 1;
+}
+
void i915_gem_init__objects(struct drm_i915_private *i915)
{
INIT_WORK(&i915->mm.free_work, __i915_gem_free_work);
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index ae5930e307d5..a3ad8cf4eefd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -596,6 +596,10 @@ void __i915_gem_free_object(struct drm_i915_gem_object *obj);
bool i915_gem_object_evictable(struct drm_i915_gem_object *obj);
+bool i915_gem_object_migratable(struct drm_i915_gem_object *obj);
+
+bool i915_gem_object_validates_to_lmem(struct drm_i915_gem_object *obj);
+
#ifdef CONFIG_MMU_NOTIFIER
static inline bool
i915_gem_object_is_userptr(struct drm_i915_gem_object *obj)
--
2.31.1
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply related [flat|nested] 60+ messages in thread
* [Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
` (12 preceding siblings ...)
(?)
@ 2021-05-21 16:06 ` Patchwork
-1 siblings, 0 replies; 60+ messages in thread
From: Patchwork @ 2021-05-21 16:06 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
URL : https://patchwork.freedesktop.org/series/90022/
State : warning
== Summary ==
$ dim checkpatch origin/drm-tip
1bf39c8cd594 drm/i915: Untangle the vma pages_mutex
650181499469 drm/i915: Don't free shared locks while shared
e0cd60e6a0d3 drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
4f895c63cc41 drm/i915/ttm Initialize the ttm device and memory managers
-:480: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#480:
deleted file mode 100644
total: 0 errors, 1 warnings, 0 checks, 1578 lines checked
fbd2191fd254 drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
67e58a6747b0 drm/ttm: Add a generic TTM memcpy move for page-based iomem
-:383: CHECK:ARCH_DEFINES: architecture specific defines should be avoided
#383: FILE: drivers/gpu/drm/ttm/ttm_module.c:56:
+#if defined(__i386__) || defined(__x86_64__)
-:699: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#699:
new file mode 100644
total: 0 errors, 1 warnings, 1 checks, 812 lines checked
fa7a4164f770 drm, drm/i915: Move the memcpy_from_wc functionality to core drm
-:54: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#54:
rename from drivers/gpu/drm/i915/i915_memcpy.c
total: 0 errors, 1 warnings, 0 checks, 410 lines checked
533ef0911275 drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
655c519cecf1 drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
2f38e8009d75 drm/ttm, drm/amdgpu: Allow the driver some control over swapping
356c346230f1 drm/i915/ttm: Introduce a TTM i915 gem object backend
-:449: WARNING:FILE_PATH_CHANGES: added, moved or deleted file(s), does MAINTAINERS need updating?
#449:
new file mode 100644
total: 0 errors, 1 warnings, 0 checks, 1024 lines checked
7ced8a9675bb drm/i915/lmem: Verify checks for lmem residency
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* [Intel-gfx] ✗ Fi.CI.SPARSE: warning for drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
` (13 preceding siblings ...)
(?)
@ 2021-05-21 16:09 ` Patchwork
-1 siblings, 0 replies; 60+ messages in thread
From: Patchwork @ 2021-05-21 16:09 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
URL : https://patchwork.freedesktop.org/series/90022/
State : warning
== Summary ==
$ dim sparse --fast origin/drm-tip
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+./drivers/gpu/drm/amd/amdgpu/../amdgpu/amdgv_sriovmsg.h:316:49: error: static assertion failed: "amd_sriov_msg_pf2vf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1345:25: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1346:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c:1405:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: error: incompatible types in comparison expression (different type sizes):
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: unsigned long *
+drivers/gpu/drm/amd/amdgpu/amdgpu_device.c:293:16: unsigned long long *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:275:25: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:276:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: error: incompatible types in comparison expression (different address spaces):
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: struct dma_fence *
+drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c:330:17: struct dma_fence [noderef] __rcu *
+drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h:90:56: error: marked inline, but without a definition
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_info must be 1 KB"
+drivers/gpu/drm/amd/amdgpu/amdgv_sriovmsg.h:312:49: error: static assertion failed: "amd_sriov_msg_vf2pf_in
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* [Intel-gfx] ✓ Fi.CI.BAT: success for drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
` (14 preceding siblings ...)
(?)
@ 2021-05-21 16:36 ` Patchwork
-1 siblings, 0 replies; 60+ messages in thread
From: Patchwork @ 2021-05-21 16:36 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
[-- Attachment #1.1: Type: text/plain, Size: 6330 bytes --]
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
URL : https://patchwork.freedesktop.org/series/90022/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_10120 -> Patchwork_20170
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/index.html
Known issues
------------
Here are the changes found in Patchwork_20170 that come from known issues:
### IGT changes ###
#### Possible fixes ####
* igt@kms_chamelium@dp-crc-fast:
- fi-kbl-7500u: [FAIL][1] ([i915#1372]) -> [PASS][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-kbl-7500u/igt@kms_chamelium@dp-crc-fast.html
* igt@kms_frontbuffer_tracking@basic:
- fi-icl-u2: [FAIL][3] ([i915#49]) -> [PASS][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-icl-u2/igt@kms_frontbuffer_tracking@basic.html
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-icl-u2/igt@kms_frontbuffer_tracking@basic.html
#### Warnings ####
* igt@i915_selftest@live@execlists:
- fi-bsw-kefka: [INCOMPLETE][5] ([i915#2782] / [i915#2940] / [i915#3462]) -> [DMESG-FAIL][6] ([i915#3462])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-bsw-kefka/igt@i915_selftest@live@execlists.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-bsw-kefka/igt@i915_selftest@live@execlists.html
* igt@runner@aborted:
- fi-skl-6600u: [FAIL][7] ([i915#1436] / [i915#3363]) -> [FAIL][8] ([i915#1436] / [i915#2426] / [i915#3363])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-skl-6600u/igt@runner@aborted.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-skl-6600u/igt@runner@aborted.html
- fi-bdw-5557u: [FAIL][9] ([i915#3462]) -> [FAIL][10] ([i915#1602] / [i915#2029])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-bdw-5557u/igt@runner@aborted.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-bdw-5557u/igt@runner@aborted.html
- fi-kbl-soraka: [FAIL][11] ([i915#1436] / [i915#2426] / [i915#3363]) -> [FAIL][12] ([i915#1436] / [i915#3363])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-kbl-soraka/igt@runner@aborted.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-kbl-soraka/igt@runner@aborted.html
- fi-cml-u2: [FAIL][13] ([i915#2082] / [i915#2426] / [i915#3363] / [i915#3462]) -> [FAIL][14] ([i915#3363] / [i915#3462])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/fi-cml-u2/igt@runner@aborted.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/fi-cml-u2/igt@runner@aborted.html
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
[i915#1372]: https://gitlab.freedesktop.org/drm/intel/issues/1372
[i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
[i915#1602]: https://gitlab.freedesktop.org/drm/intel/issues/1602
[i915#2029]: https://gitlab.freedesktop.org/drm/intel/issues/2029
[i915#2082]: https://gitlab.freedesktop.org/drm/intel/issues/2082
[i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
[i915#2426]: https://gitlab.freedesktop.org/drm/intel/issues/2426
[i915#2782]: https://gitlab.freedesktop.org/drm/intel/issues/2782
[i915#2932]: https://gitlab.freedesktop.org/drm/intel/issues/2932
[i915#2940]: https://gitlab.freedesktop.org/drm/intel/issues/2940
[i915#2966]: https://gitlab.freedesktop.org/drm/intel/issues/2966
[i915#3012]: https://gitlab.freedesktop.org/drm/intel/issues/3012
[i915#3276]: https://gitlab.freedesktop.org/drm/intel/issues/3276
[i915#3277]: https://gitlab.freedesktop.org/drm/intel/issues/3277
[i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
[i915#3283]: https://gitlab.freedesktop.org/drm/intel/issues/3283
[i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
[i915#3301]: https://gitlab.freedesktop.org/drm/intel/issues/3301
[i915#3363]: https://gitlab.freedesktop.org/drm/intel/issues/3363
[i915#3462]: https://gitlab.freedesktop.org/drm/intel/issues/3462
[i915#49]: https://gitlab.freedesktop.org/drm/intel/issues/49
[i915#533]: https://gitlab.freedesktop.org/drm/intel/issues/533
Participating hosts (41 -> 39)
------------------------------
Additional (1): fi-rkl-11500t
Missing (3): fi-bsw-cyan fi-bdw-samus fi-hsw-4200u
Build changes
-------------
* Linux: CI_DRM_10120 -> Patchwork_20170
CI-20190529: 20190529
CI_DRM_10120: 9221d50d353487d2e10226318d89027037255621 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_6091: a7016bde81f6e6ee9f2ded3c091c56766a6adc46 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
Patchwork_20170: 7ced8a9675bbb379bd5287d6bc210960c31f34a6 @ git://anongit.freedesktop.org/gfx-ci/linux
== Linux commits ==
7ced8a9675bb drm/i915/lmem: Verify checks for lmem residency
356c346230f1 drm/i915/ttm: Introduce a TTM i915 gem object backend
2f38e8009d75 drm/ttm, drm/amdgpu: Allow the driver some control over swapping
655c519cecf1 drm/ttm: Document and optimize ttm_bo_pipeline_gutting()
533ef0911275 drm/ttm: Use drm_memcpy_from_wc_dbm for TTM bo moves
fa7a4164f770 drm, drm/i915: Move the memcpy_from_wc functionality to core drm
67e58a6747b0 drm/ttm: Add a generic TTM memcpy move for page-based iomem
fbd2191fd254 drm/i915/ttm: Embed a ttm buffer object in the i915 gem object
4f895c63cc41 drm/i915/ttm Initialize the ttm device and memory managers
e0cd60e6a0d3 drm/i915: Fix i915_sg_page_sizes to record dma segments rather than physical pages
650181499469 drm/i915: Don't free shared locks while shared
1bf39c8cd594 drm/i915: Untangle the vma pages_mutex
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/index.html
[-- Attachment #1.2: Type: text/html, Size: 7097 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread* [Intel-gfx] ✗ Fi.CI.IGT: failure for drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
2021-05-21 15:32 ` [Intel-gfx] " Thomas Hellström
` (15 preceding siblings ...)
(?)
@ 2021-05-24 0:10 ` Patchwork
-1 siblings, 0 replies; 60+ messages in thread
From: Patchwork @ 2021-05-24 0:10 UTC (permalink / raw)
To: Thomas Hellström; +Cc: intel-gfx
[-- Attachment #1.1: Type: text/plain, Size: 30279 bytes --]
== Series Details ==
Series: drm/i915: Move LMEM (VRAM) management over to TTM (rev3)
URL : https://patchwork.freedesktop.org/series/90022/
State : failure
== Summary ==
CI Bug Log - changes from CI_DRM_10120_full -> Patchwork_20170_full
====================================================
Summary
-------
**FAILURE**
Serious unknown changes coming with Patchwork_20170_full absolutely need to be
verified manually.
If you think the reported changes have nothing to do with the changes
introduced in Patchwork_20170_full, please notify your bug team to allow them
to document this new failure mode, which will reduce false positives in CI.
Possible new issues
-------------------
Here are the unknown changes that may have been introduced in Patchwork_20170_full:
### Piglit changes ###
#### Possible regressions ####
* spec@arb_texture_barrier@arb_texture_barrier-blending-in-shader 512 42 1 8 3:
- pig-glk-j5005: NOTRUN -> [INCOMPLETE][1] +1 similar issue
[1]: None
Known issues
------------
Here are the changes found in Patchwork_20170_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_create@create-clear:
- shard-glk: [PASS][2] -> [DMESG-WARN][3] ([i915#118] / [i915#95])
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk1/igt@gem_create@create-clear.html
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk9/igt@gem_create@create-clear.html
* igt@gem_create@create-massive:
- shard-snb: NOTRUN -> [DMESG-WARN][4] ([i915#3002])
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb7/igt@gem_create@create-massive.html
- shard-skl: NOTRUN -> [DMESG-WARN][5] ([i915#3002])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@gem_create@create-massive.html
* igt@gem_ctx_isolation@preservation-s3@vecs0:
- shard-kbl: [PASS][6] -> [DMESG-WARN][7] ([i915#180]) +5 similar issues
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@gem_ctx_isolation@preservation-s3@vecs0.html
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@gem_ctx_isolation@preservation-s3@vecs0.html
* igt@gem_ctx_persistence@legacy-engines-mixed-process:
- shard-snb: NOTRUN -> [SKIP][8] ([fdo#109271] / [i915#1099]) +4 similar issues
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb2/igt@gem_ctx_persistence@legacy-engines-mixed-process.html
* igt@gem_ctx_ringsize@active@bcs0:
- shard-skl: [PASS][9] -> [INCOMPLETE][10] ([i915#3316])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl9/igt@gem_ctx_ringsize@active@bcs0.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl9/igt@gem_ctx_ringsize@active@bcs0.html
* igt@gem_exec_fair@basic-pace-solo@rcs0:
- shard-glk: [PASS][11] -> [FAIL][12] ([i915#2842])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk2/igt@gem_exec_fair@basic-pace-solo@rcs0.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk6/igt@gem_exec_fair@basic-pace-solo@rcs0.html
* igt@gem_exec_fair@basic-pace@bcs0:
- shard-tglb: [PASS][13] -> [FAIL][14] ([i915#2842]) +1 similar issue
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-tglb5/igt@gem_exec_fair@basic-pace@bcs0.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-tglb5/igt@gem_exec_fair@basic-pace@bcs0.html
* igt@gem_exec_reloc@basic-wide-active@bcs0:
- shard-apl: NOTRUN -> [FAIL][15] ([i915#2389]) +3 similar issues
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@gem_exec_reloc@basic-wide-active@bcs0.html
* igt@gem_mmap_gtt@big-copy-odd:
- shard-glk: [PASS][16] -> [FAIL][17] ([i915#307])
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk2/igt@gem_mmap_gtt@big-copy-odd.html
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk2/igt@gem_mmap_gtt@big-copy-odd.html
* igt@gem_mmap_gtt@fault-concurrent-x:
- shard-apl: NOTRUN -> [INCOMPLETE][18] ([i915#3468])
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl1/igt@gem_mmap_gtt@fault-concurrent-x.html
* igt@gem_mmap_gtt@medium-copy-xy:
- shard-snb: [PASS][19] -> [INCOMPLETE][20] ([i915#2055] / [i915#2502] / [i915#3468])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-snb2/igt@gem_mmap_gtt@medium-copy-xy.html
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb2/igt@gem_mmap_gtt@medium-copy-xy.html
* igt@gem_userptr_blits@set-cache-level:
- shard-apl: NOTRUN -> [FAIL][21] ([i915#3324])
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@gem_userptr_blits@set-cache-level.html
* igt@gen9_exec_parse@bb-large:
- shard-apl: NOTRUN -> [FAIL][22] ([i915#3296])
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl1/igt@gen9_exec_parse@bb-large.html
* igt@i915_pm_lpsp@kms-lpsp@kms-lpsp-dp:
- shard-apl: NOTRUN -> [SKIP][23] ([fdo#109271] / [i915#1937])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl1/igt@i915_pm_lpsp@kms-lpsp@kms-lpsp-dp.html
* igt@i915_selftest@live@execlists:
- shard-apl: NOTRUN -> [DMESG-FAIL][24] ([i915#3462])
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@i915_selftest@live@execlists.html
* igt@kms_big_joiner@invalid-modeset:
- shard-kbl: NOTRUN -> [SKIP][25] ([fdo#109271] / [i915#2705])
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@kms_big_joiner@invalid-modeset.html
- shard-apl: NOTRUN -> [SKIP][26] ([fdo#109271] / [i915#2705])
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_big_joiner@invalid-modeset.html
* igt@kms_chamelium@dp-hpd:
- shard-skl: NOTRUN -> [SKIP][27] ([fdo#109271] / [fdo#111827]) +1 similar issue
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_chamelium@dp-hpd.html
* igt@kms_chamelium@dp-mode-timings:
- shard-apl: NOTRUN -> [SKIP][28] ([fdo#109271] / [fdo#111827]) +26 similar issues
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@kms_chamelium@dp-mode-timings.html
* igt@kms_color@pipe-c-ctm-0-5:
- shard-skl: [PASS][29] -> [DMESG-WARN][30] ([i915#1982])
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl4/igt@kms_color@pipe-c-ctm-0-5.html
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl4/igt@kms_color@pipe-c-ctm-0-5.html
* igt@kms_color_chamelium@pipe-a-ctm-0-25:
- shard-snb: NOTRUN -> [SKIP][31] ([fdo#109271] / [fdo#111827]) +19 similar issues
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb5/igt@kms_color_chamelium@pipe-a-ctm-0-25.html
* igt@kms_color_chamelium@pipe-c-ctm-0-5:
- shard-kbl: NOTRUN -> [SKIP][32] ([fdo#109271] / [fdo#111827]) +4 similar issues
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@kms_color_chamelium@pipe-c-ctm-0-5.html
* igt@kms_color_chamelium@pipe-invalid-ctm-matrix-sizes:
- shard-glk: NOTRUN -> [SKIP][33] ([fdo#109271] / [fdo#111827]) +2 similar issues
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_color_chamelium@pipe-invalid-ctm-matrix-sizes.html
* igt@kms_content_protection@legacy:
- shard-apl: NOTRUN -> [TIMEOUT][34] ([i915#1319])
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_content_protection@legacy.html
* igt@kms_cursor_crc@pipe-b-cursor-32x32-onscreen:
- shard-skl: NOTRUN -> [SKIP][35] ([fdo#109271]) +35 similar issues
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_cursor_crc@pipe-b-cursor-32x32-onscreen.html
* igt@kms_cursor_crc@pipe-d-cursor-256x256-onscreen:
- shard-kbl: NOTRUN -> [SKIP][36] ([fdo#109271]) +65 similar issues
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@kms_cursor_crc@pipe-d-cursor-256x256-onscreen.html
* igt@kms_cursor_crc@pipe-d-cursor-64x21-random:
- shard-glk: NOTRUN -> [SKIP][37] ([fdo#109271]) +32 similar issues
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_cursor_crc@pipe-d-cursor-64x21-random.html
* igt@kms_cursor_edge_walk@pipe-d-128x128-right-edge:
- shard-snb: NOTRUN -> [SKIP][38] ([fdo#109271]) +355 similar issues
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb2/igt@kms_cursor_edge_walk@pipe-d-128x128-right-edge.html
* igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size:
- shard-skl: [PASS][39] -> [FAIL][40] ([i915#2346] / [i915#533])
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl9/igt@kms_cursor_legacy@flip-vs-cursor-atomic-transitions-varying-size.html
* igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible@b-edp1:
- shard-skl: [PASS][41] -> [FAIL][42] ([i915#2122]) +1 similar issue
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl2/igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible@b-edp1.html
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl2/igt@kms_flip@flip-vs-absolute-wf_vblank-interruptible@b-edp1.html
* igt@kms_flip@flip-vs-expired-vblank@a-edp1:
- shard-skl: [PASS][43] -> [FAIL][44] ([i915#79])
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl7/igt@kms_flip@flip-vs-expired-vblank@a-edp1.html
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl4/igt@kms_flip@flip-vs-expired-vblank@a-edp1.html
* igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs:
- shard-skl: NOTRUN -> [SKIP][45] ([fdo#109271] / [i915#2672])
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs.html
* igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile:
- shard-apl: NOTRUN -> [SKIP][46] ([fdo#109271] / [i915#2642])
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-64bpp-ytile.html
* igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs:
- shard-kbl: NOTRUN -> [SKIP][47] ([fdo#109271] / [i915#2672])
[47]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs.html
- shard-glk: NOTRUN -> [SKIP][48] ([fdo#109271] / [i915#2672])
[48]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytilercccs.html
* igt@kms_hdr@bpc-switch-dpms:
- shard-skl: NOTRUN -> [FAIL][49] ([i915#1188])
[49]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_hdr@bpc-switch-dpms.html
* igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d:
- shard-apl: NOTRUN -> [SKIP][50] ([fdo#109271] / [i915#533]) +4 similar issues
[50]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d.html
- shard-kbl: NOTRUN -> [SKIP][51] ([fdo#109271] / [i915#533])
[51]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d.html
* igt@kms_plane_alpha_blend@pipe-a-alpha-transparent-fb:
- shard-apl: NOTRUN -> [FAIL][52] ([i915#265])
[52]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@kms_plane_alpha_blend@pipe-a-alpha-transparent-fb.html
* igt@kms_plane_alpha_blend@pipe-b-alpha-transparent-fb:
- shard-skl: NOTRUN -> [FAIL][53] ([i915#265])
[53]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_plane_alpha_blend@pipe-b-alpha-transparent-fb.html
* igt@kms_plane_alpha_blend@pipe-b-constant-alpha-max:
- shard-apl: NOTRUN -> [FAIL][54] ([fdo#108145] / [i915#265]) +2 similar issues
[54]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_plane_alpha_blend@pipe-b-constant-alpha-max.html
* igt@kms_plane_alpha_blend@pipe-b-coverage-7efc:
- shard-skl: [PASS][55] -> [FAIL][56] ([fdo#108145] / [i915#265])
[55]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@kms_plane_alpha_blend@pipe-b-coverage-7efc.html
[56]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@kms_plane_alpha_blend@pipe-b-coverage-7efc.html
* igt@kms_plane_alpha_blend@pipe-c-alpha-7efc:
- shard-kbl: NOTRUN -> [FAIL][57] ([fdo#108145] / [i915#265])
[57]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@kms_plane_alpha_blend@pipe-c-alpha-7efc.html
* igt@kms_plane_scaling@scaler-with-clipping-clamping@pipe-c-scaler-with-clipping-clamping:
- shard-apl: NOTRUN -> [SKIP][58] ([fdo#109271] / [i915#2733])
[58]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_plane_scaling@scaler-with-clipping-clamping@pipe-c-scaler-with-clipping-clamping.html
* igt@kms_psr2_sf@overlay-plane-update-sf-dmg-area-3:
- shard-apl: NOTRUN -> [SKIP][59] ([fdo#109271] / [i915#658]) +6 similar issues
[59]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@kms_psr2_sf@overlay-plane-update-sf-dmg-area-3.html
* igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-2:
- shard-skl: NOTRUN -> [SKIP][60] ([fdo#109271] / [i915#658])
[60]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-2.html
* igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3:
- shard-kbl: NOTRUN -> [SKIP][61] ([fdo#109271] / [i915#658]) +1 similar issue
[61]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3.html
- shard-glk: NOTRUN -> [SKIP][62] ([fdo#109271] / [i915#658])
[62]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_psr2_sf@overlay-primary-update-sf-dmg-area-3.html
* igt@kms_writeback@writeback-check-output:
- shard-apl: NOTRUN -> [SKIP][63] ([fdo#109271] / [i915#2437])
[63]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@kms_writeback@writeback-check-output.html
- shard-skl: NOTRUN -> [SKIP][64] ([fdo#109271] / [i915#2437])
[64]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@kms_writeback@writeback-check-output.html
* igt@kms_writeback@writeback-pixel-formats:
- shard-glk: NOTRUN -> [SKIP][65] ([fdo#109271] / [i915#2437])
[65]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_writeback@writeback-pixel-formats.html
- shard-kbl: NOTRUN -> [SKIP][66] ([fdo#109271] / [i915#2437])
[66]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@kms_writeback@writeback-pixel-formats.html
* igt@perf@polling-parameterized:
- shard-skl: [PASS][67] -> [FAIL][68] ([i915#1542])
[67]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@perf@polling-parameterized.html
[68]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl9/igt@perf@polling-parameterized.html
* igt@prime_nv_api@i915_nv_reimport_twice_check_flink_name:
- shard-apl: NOTRUN -> [SKIP][69] ([fdo#109271]) +254 similar issues
[69]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl1/igt@prime_nv_api@i915_nv_reimport_twice_check_flink_name.html
* igt@sysfs_clients@fair-0:
- shard-kbl: NOTRUN -> [SKIP][70] ([fdo#109271] / [i915#2994])
[70]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@sysfs_clients@fair-0.html
* igt@sysfs_clients@fair-7:
- shard-apl: NOTRUN -> [SKIP][71] ([fdo#109271] / [i915#2994]) +6 similar issues
[71]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl7/igt@sysfs_clients@fair-7.html
* igt@sysfs_heartbeat_interval@mixed@vcs0:
- shard-skl: [PASS][72] -> [FAIL][73] ([i915#1731])
[72]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@sysfs_heartbeat_interval@mixed@vcs0.html
[73]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl9/igt@sysfs_heartbeat_interval@mixed@vcs0.html
* igt@sysfs_timeslice_duration@timeout@vecs0:
- shard-tglb: [PASS][74] -> [FAIL][75] ([i915#1755])
[74]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-tglb5/igt@sysfs_timeslice_duration@timeout@vecs0.html
[75]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-tglb8/igt@sysfs_timeslice_duration@timeout@vecs0.html
#### Possible fixes ####
* igt@gem_exec_fair@basic-deadline:
- shard-glk: [FAIL][76] ([i915#2846]) -> [PASS][77]
[76]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk1/igt@gem_exec_fair@basic-deadline.html
[77]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk9/igt@gem_exec_fair@basic-deadline.html
* igt@gem_exec_fair@basic-none-share@rcs0:
- shard-tglb: [FAIL][78] ([i915#2842]) -> [PASS][79]
[78]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-tglb1/igt@gem_exec_fair@basic-none-share@rcs0.html
[79]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-tglb8/igt@gem_exec_fair@basic-none-share@rcs0.html
* igt@gem_exec_fair@basic-pace-share@rcs0:
- shard-glk: [FAIL][80] ([i915#2842]) -> [PASS][81] +1 similar issue
[80]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk4/igt@gem_exec_fair@basic-pace-share@rcs0.html
[81]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk2/igt@gem_exec_fair@basic-pace-share@rcs0.html
* igt@gen9_exec_parse@allowed-single:
- shard-skl: [DMESG-WARN][82] ([i915#1436] / [i915#716]) -> [PASS][83]
[82]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl4/igt@gen9_exec_parse@allowed-single.html
[83]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@gen9_exec_parse@allowed-single.html
* igt@i915_suspend@sysfs-reader:
- shard-kbl: [INCOMPLETE][84] ([i915#155]) -> [PASS][85]
[84]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl2/igt@i915_suspend@sysfs-reader.html
[85]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@i915_suspend@sysfs-reader.html
* igt@kms_cursor_legacy@2x-long-flip-vs-cursor-legacy:
- shard-glk: [FAIL][86] ([i915#72]) -> [PASS][87]
[86]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk7/igt@kms_cursor_legacy@2x-long-flip-vs-cursor-legacy.html
[87]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk5/igt@kms_cursor_legacy@2x-long-flip-vs-cursor-legacy.html
* igt@kms_draw_crc@draw-method-xrgb2101010-pwrite-ytiled:
- shard-skl: [DMESG-WARN][88] ([i915#1982]) -> [PASS][89]
[88]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@kms_draw_crc@draw-method-xrgb2101010-pwrite-ytiled.html
[89]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@kms_draw_crc@draw-method-xrgb2101010-pwrite-ytiled.html
* igt@kms_fbcon_fbt@fbc-suspend:
- shard-kbl: [INCOMPLETE][90] ([i915#155] / [i915#180] / [i915#636]) -> [PASS][91]
[90]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl1/igt@kms_fbcon_fbt@fbc-suspend.html
[91]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@kms_fbcon_fbt@fbc-suspend.html
* igt@kms_flip@flip-vs-expired-vblank-interruptible@c-hdmi-a2:
- shard-glk: [FAIL][92] ([i915#79]) -> [PASS][93]
[92]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-glk9/igt@kms_flip@flip-vs-expired-vblank-interruptible@c-hdmi-a2.html
[93]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-glk6/igt@kms_flip@flip-vs-expired-vblank-interruptible@c-hdmi-a2.html
* igt@kms_flip@flip-vs-suspend-interruptible@a-dp1:
- shard-kbl: [DMESG-WARN][94] ([i915#180]) -> [PASS][95]
[94]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@kms_flip@flip-vs-suspend-interruptible@a-dp1.html
[95]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl2/igt@kms_flip@flip-vs-suspend-interruptible@a-dp1.html
- shard-apl: [DMESG-WARN][96] ([i915#180]) -> [PASS][97]
[96]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-apl1/igt@kms_flip@flip-vs-suspend-interruptible@a-dp1.html
[97]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@kms_flip@flip-vs-suspend-interruptible@a-dp1.html
* igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b:
- shard-skl: [INCOMPLETE][98] ([i915#198]) -> [PASS][99]
[98]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl2/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html
[99]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b.html
#### Warnings ####
* igt@gem_mmap_gtt@cpuset-basic-small-copy-xy:
- shard-snb: [INCOMPLETE][100] ([i915#2055]) -> [INCOMPLETE][101] ([i915#2055] / [i915#3468])
[100]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-snb7/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
[101]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-snb7/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
- shard-skl: [INCOMPLETE][102] ([i915#198] / [i915#3468]) -> [INCOMPLETE][103] ([i915#198])
[102]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl2/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
[103]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl2/igt@gem_mmap_gtt@cpuset-basic-small-copy-xy.html
* igt@gem_mmap_gtt@fault-concurrent-x:
- shard-skl: [INCOMPLETE][104] ([i915#198]) -> [INCOMPLETE][105] ([i915#198] / [i915#3468])
[104]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl7/igt@gem_mmap_gtt@fault-concurrent-x.html
[105]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl4/igt@gem_mmap_gtt@fault-concurrent-x.html
* igt@runner@aborted:
- shard-kbl: ([FAIL][106], [FAIL][107], [FAIL][108], [FAIL][109], [FAIL][110], [FAIL][111], [FAIL][112], [FAIL][113], [FAIL][114], [FAIL][115], [FAIL][116], [FAIL][117], [FAIL][118], [FAIL][119], [FAIL][120], [FAIL][121]) ([fdo#109271] / [i915#1436] / [i915#180] / [i915#1814] / [i915#2292] / [i915#2722] / [i915#3002] / [i915#3363] / [i915#92]) -> ([FAIL][122], [FAIL][123], [FAIL][124], [FAIL][125], [FAIL][126], [FAIL][127], [FAIL][128], [FAIL][129], [FAIL][130], [FAIL][131], [FAIL][132], [FAIL][133], [FAIL][134], [FAIL][135], [FAIL][136], [FAIL][137]) ([fdo#109271] / [i915#1436] / [i915#180] / [i915#1814] / [i915#2292] / [i915#2722] / [i915#3002] / [i915#3363])
[106]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[107]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[108]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl3/igt@runner@aborted.html
[109]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl7/igt@runner@aborted.html
[110]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[111]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl1/igt@runner@aborted.html
[112]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl1/igt@runner@aborted.html
[113]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl1/igt@runner@aborted.html
[114]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl3/igt@runner@aborted.html
[115]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl3/igt@runner@aborted.html
[116]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl7/igt@runner@aborted.html
[117]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[118]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl2/igt@runner@aborted.html
[119]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl2/igt@runner@aborted.html
[120]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[121]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-kbl4/igt@runner@aborted.html
[122]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@runner@aborted.html
[123]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl7/igt@runner@aborted.html
[124]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[125]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[126]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[127]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl3/igt@runner@aborted.html
[128]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl2/igt@runner@aborted.html
[129]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[130]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl2/igt@runner@aborted.html
[131]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[132]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[133]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@runner@aborted.html
[134]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl4/igt@runner@aborted.html
[135]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl3/igt@runner@aborted.html
[136]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
[137]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-kbl1/igt@runner@aborted.html
- shard-apl: ([FAIL][138], [FAIL][139], [FAIL][140], [FAIL][141]) ([i915#180] / [i915#2722] / [i915#3002] / [i915#3363]) -> ([FAIL][142], [FAIL][143], [FAIL][144]) ([fdo#109271] / [i915#2722] / [i915#3363])
[138]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-apl1/igt@runner@aborted.html
[139]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-apl1/igt@runner@aborted.html
[140]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-apl7/igt@runner@aborted.html
[141]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-apl6/igt@runner@aborted.html
[142]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl3/igt@runner@aborted.html
[143]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl6/igt@runner@aborted.html
[144]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-apl1/igt@runner@aborted.html
- shard-skl: ([FAIL][145], [FAIL][146], [FAIL][147], [FAIL][148], [FAIL][149], [FAIL][150]) ([i915#1436] / [i915#2426] / [i915#2722] / [i915#3002] / [i915#3363]) -> ([FAIL][151], [FAIL][152], [FAIL][153], [FAIL][154], [FAIL][155], [FAIL][156], [FAIL][157]) ([i915#1436] / [i915#1814] / [i915#2029] / [i915#2722] / [i915#3002] / [i915#3363])
[145]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl4/igt@runner@aborted.html
[146]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl9/igt@runner@aborted.html
[147]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl4/igt@runner@aborted.html
[148]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl7/igt@runner@aborted.html
[149]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl2/igt@runner@aborted.html
[150]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10120/shard-skl3/igt@runner@aborted.html
[151]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl4/igt@runner@aborted.html
[152]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@runner@aborted.html
[153]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl4/igt@runner@aborted.html
[154]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl7/igt@runner@aborted.html
[155]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl2/igt@runner@aborted.html
[156]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@runner@aborted.html
[157]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/shard-skl3/igt@runner@aborted.html
[fdo#108145]: https://bugs.freedesktop.org/show_bug.cgi?id=108145
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[i915#1099]: https://gitlab.freedesktop.org/drm/intel/issues/1099
[i915#118]: https://gitlab.freedesktop.org/drm/intel/issues/118
[i915#1188]: https://gitlab.freedesktop.org/drm/intel/issues/1188
[i915#1319]: https://gitlab.freedesktop.org/drm/intel/issues/1319
[i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
[i915#1542]: https://gitlab.freedeskt
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20170/index.html
[-- Attachment #1.2: Type: text/html, Size: 36080 bytes --]
[-- Attachment #2: Type: text/plain, Size: 160 bytes --]
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
^ permalink raw reply [flat|nested] 60+ messages in thread