[Intel-gfx] [PATCH 00/19] drm/i915: Short-term pinning and async eviction.

[Intel-gfx] [PATCH 00/19] drm/i915: Short-term pinning and async eviction.

[Maarten Lankhorst]

Remove some parts of the i915_vma api, ensure obj->vma always exists,
and finally force the object lock to be taken when calling i915_vma_unbind
is called.

With this, locking is a lot cleaner, and we no longer need all the if (!obj->vma) checks.
We kill off the locking around i915_vma->set/get pages, and kill off the short-term
pinning/unpinning in execbuf.

After this, we go 1 step further, populate obj->moving where required, and add support
for async eviction/clearing.

Maarten Lankhorst (19):
  drm/i915: Move __i915_gem_free_object to ttm_bo_destroy
  drm/i915: Remove unused bits of i915_vma/active api
  drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling
  drm/i915: Remove gen6_ppgtt_unpin_all
  drm/i915: Create a dummy object for gen6 ppgtt
  drm/i915: Create a full object for mock_ring
  drm/i915: vma is always backed by an object.
  drm/i915: Fix runtime pm handling in i915_gem_shrink
  drm/i915: Change shrink ordering to use locking around unbinding.
  Move CONTEXT_VALID_BIT check
  drm/i915: Remove resv from i915_vma
  drm/i915: Remove pages_mutex and intel_gtt->vma_ops.set/clear_pages
    members
  drm/i915: Take object lock in i915_ggtt_pin if ww is not set
  drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for
    i915_vma_unbind
  drm/i915: Remove support for unlocked i915_vma unbind
  drm/i915: Remove short-term pins from execbuf
  drm/i915: Add functions to set/get moving fence
  drm/i915: Add support for asynchronous moving fence waiting
  drm/i915: Add accelerated migration to ttm

 drivers/gpu/drm/i915/display/intel_display.c  |   2 +-
 drivers/gpu/drm/i915/display/intel_dpt.c      |   2 -
 .../gpu/drm/i915/gem/i915_gem_execbuffer.c    |  94 ++--
 drivers/gpu/drm/i915/gem/i915_gem_internal.c  |  44 +-
 drivers/gpu/drm/i915/gem/i915_gem_object.c    |  15 +
 drivers/gpu/drm/i915/gem/i915_gem_object.h    |   6 +
 drivers/gpu/drm/i915/gem/i915_gem_pages.c     |  10 +
 drivers/gpu/drm/i915/gem/i915_gem_shrinker.c  |  63 ++-
 drivers/gpu/drm/i915/gem/i915_gem_ttm.c       | 111 ++--
 .../gpu/drm/i915/gem/selftests/huge_pages.c   |   4 +-
 .../i915/gem/selftests/i915_gem_client_blt.c  |   2 +-
 .../drm/i915/gem/selftests/i915_gem_mman.c    |   6 +
 drivers/gpu/drm/i915/gt/gen6_ppgtt.c          | 148 +++--
 drivers/gpu/drm/i915/gt/gen6_ppgtt.h          |   2 -
 drivers/gpu/drm/i915/gt/intel_context.c       |   2 +-
 drivers/gpu/drm/i915/gt/intel_engine_pm.c     |   5 +-
 drivers/gpu/drm/i915/gt/intel_ggtt.c          | 366 ++----------
 drivers/gpu/drm/i915/gt/intel_gtt.c           |  13 -
 drivers/gpu/drm/i915/gt/intel_gtt.h           |   7 -
 drivers/gpu/drm/i915/gt/intel_ppgtt.c         |  12 -
 .../gpu/drm/i915/gt/intel_ring_submission.c   |   2 +-
 drivers/gpu/drm/i915/gt/mock_engine.c         |  35 +-
 drivers/gpu/drm/i915/i915_active.c            |  28 +-
 drivers/gpu/drm/i915/i915_active.h            |  17 +-
 drivers/gpu/drm/i915/i915_drv.h               |  11 +-
 drivers/gpu/drm/i915/i915_gem.c               |  29 +-
 drivers/gpu/drm/i915/i915_gem_evict.c         |  74 ++-
 drivers/gpu/drm/i915/i915_gem_gtt.h           |   1 +
 drivers/gpu/drm/i915/i915_gpu_error.c         |   9 +-
 drivers/gpu/drm/i915/i915_request.c           |   9 +-
 drivers/gpu/drm/i915/i915_request.h           |   7 +-
 drivers/gpu/drm/i915/i915_vma.c               | 522 ++++++++++++++----
 drivers/gpu/drm/i915/i915_vma.h               |  15 +-
 drivers/gpu/drm/i915/i915_vma_types.h         |   2 -
 drivers/gpu/drm/i915/selftests/i915_gem_gtt.c |  34 +-
 drivers/gpu/drm/i915/selftests/i915_vma.c     |   2 +-
 drivers/gpu/drm/i915/selftests/mock_gtt.c     |   4 -
 37 files changed, 899 insertions(+), 816 deletions(-)

-- 
2.32.0

[Intel-gfx] [PATCH 01/19] drm/i915: Move __i915_gem_free_object to ttm_bo_destroy

[Maarten Lankhorst]

When we implement delayed destroy, we may have a second
call to the delete_mem_notify() handler, while free_object()
only should be called once.

Move it to bo->destroy(), to ensure it's only called once.
This fixes some weird memory corruption issues with delayed
destroy when async eviction is used.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index 59ca53a3ef6a..eaf2ff29dd4a 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -355,11 +355,8 @@ static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
 {
 	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
 
-	if (likely(obj)) {
-		/* This releases all gem object bindings to the backend. */
+	if (likely(obj))
 		i915_ttm_free_cached_io_st(obj);
-		__i915_gem_free_object(obj);
-	}
 }
 
 static struct intel_memory_region *
@@ -886,8 +883,12 @@ void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
 {
 	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
 
+	/* This releases all gem object bindings to the backend. */
+	__i915_gem_free_object(obj);
+
 	i915_gem_object_release_memory_region(obj);
 	mutex_destroy(&obj->ttm.get_io_page.lock);
+
 	if (obj->ttm.created)
 		call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
 }
-- 
2.32.0

[Intel-gfx] [PATCH 02/19] drm/i915: Remove unused bits of i915_vma/active api

[Maarten Lankhorst]

When reworking the code to move the eviction fence to the object,
the best code is removed code.

Remove some functions that are unused, and change the function definition
if it's only used in 1 place.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/i915_active.c | 28 +++-------------------------
 drivers/gpu/drm/i915/i915_active.h | 17 +----------------
 drivers/gpu/drm/i915/i915_vma.c    |  2 +-
 drivers/gpu/drm/i915/i915_vma.h    |  2 --
 4 files changed, 5 insertions(+), 44 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c
index 3103c1e1fd14..ee2b3a375362 100644
--- a/drivers/gpu/drm/i915/i915_active.c
+++ b/drivers/gpu/drm/i915/i915_active.c
@@ -426,8 +426,9 @@ replace_barrier(struct i915_active *ref, struct i915_active_fence *active)
 	return true;
 }
 
-int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
+int i915_active_add_request(struct i915_active *ref, struct i915_request *rq)
 {
+	struct dma_fence *fence = &rq->fence;
 	struct i915_active_fence *active;
 	int err;
 
@@ -436,7 +437,7 @@ int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
 	if (err)
 		return err;
 
-	active = active_instance(ref, idx);
+	active = active_instance(ref, i915_request_timeline(rq)->fence_context);
 	if (!active) {
 		err = -ENOMEM;
 		goto out;
@@ -477,29 +478,6 @@ __i915_active_set_fence(struct i915_active *ref,
 	return prev;
 }
 
-static struct i915_active_fence *
-__active_fence(struct i915_active *ref, u64 idx)
-{
-	struct active_node *it;
-
-	it = __active_lookup(ref, idx);
-	if (unlikely(!it)) { /* Contention with parallel tree builders! */
-		spin_lock_irq(&ref->tree_lock);
-		it = __active_lookup(ref, idx);
-		spin_unlock_irq(&ref->tree_lock);
-	}
-	GEM_BUG_ON(!it); /* slot must be preallocated */
-
-	return &it->base;
-}
-
-struct dma_fence *
-__i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
-{
-	/* Only valid while active, see i915_active_acquire_for_context() */
-	return __i915_active_set_fence(ref, __active_fence(ref, idx), fence);
-}
-
 struct dma_fence *
 i915_active_set_exclusive(struct i915_active *ref, struct dma_fence *f)
 {
diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h
index 5fcdb0e2bc9e..7eb44132183a 100644
--- a/drivers/gpu/drm/i915/i915_active.h
+++ b/drivers/gpu/drm/i915/i915_active.h
@@ -164,26 +164,11 @@ void __i915_active_init(struct i915_active *ref,
 	__i915_active_init(ref, active, retire, flags, &__mkey, &__wkey);	\
 } while (0)
 
-struct dma_fence *
-__i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence);
-int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence);
-
-static inline int
-i915_active_add_request(struct i915_active *ref, struct i915_request *rq)
-{
-	return i915_active_ref(ref,
-			       i915_request_timeline(rq)->fence_context,
-			       &rq->fence);
-}
+int i915_active_add_request(struct i915_active *ref, struct i915_request *rq);
 
 struct dma_fence *
 i915_active_set_exclusive(struct i915_active *ref, struct dma_fence *f);
 
-static inline bool i915_active_has_exclusive(struct i915_active *ref)
-{
-	return rcu_access_pointer(ref->excl.fence);
-}
-
 int __i915_active_wait(struct i915_active *ref, int state);
 static inline int i915_active_wait(struct i915_active *ref)
 {
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index 4b7fc4647e46..f9ac33e0bac9 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -1220,7 +1220,7 @@ __i915_request_await_bind(struct i915_request *rq, struct i915_vma *vma)
 	return __i915_request_await_exclusive(rq, &vma->active);
 }
 
-int __i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *rq)
+static int __i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *rq)
 {
 	int err;
 
diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
index ed69f66c7ab0..1c930515ec3d 100644
--- a/drivers/gpu/drm/i915/i915_vma.h
+++ b/drivers/gpu/drm/i915/i915_vma.h
@@ -55,8 +55,6 @@ static inline bool i915_vma_is_active(const struct i915_vma *vma)
 /* do not reserve memory to prevent deadlocks */
 #define __EXEC_OBJECT_NO_RESERVE BIT(31)
 
-int __must_check __i915_vma_move_to_active(struct i915_vma *vma,
-					   struct i915_request *rq);
 int __must_check i915_vma_move_to_active(struct i915_vma *vma,
 					 struct i915_request *rq,
 					 unsigned int flags);
-- 
2.32.0

[Intel-gfx] [PATCH 03/19] drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling

[Maarten Lankhorst]

Use a single null-terminated array for simplicity instead of a linked
list. This might slightly speed up execbuf when many vma's may be marked
as capture, but definitely removes an allocation from a signaling path.

We are not allowed to allocate memory in eb_move_to_gpu, but we can't
enforce it yet through annotations.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 .../gpu/drm/i915/gem/i915_gem_execbuffer.c    | 26 ++++++++++++-------
 drivers/gpu/drm/i915/i915_gpu_error.c         |  9 ++++---
 drivers/gpu/drm/i915/i915_request.c           |  9 ++-----
 drivers/gpu/drm/i915/i915_request.h           |  7 +----
 4 files changed, 24 insertions(+), 27 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 8290bdadd167..4f10a81befd1 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -255,6 +255,9 @@ struct i915_execbuffer {
 	/** actual size of execobj[] as we may extend it for the cmdparser */
 	unsigned int buffer_count;
 
+	/* Number of objects with EXEC_OBJECT_CAPTURE set */
+	unsigned int capture_count;
+
 	/** list of vma not yet bound during reservation phase */
 	struct list_head unbound;
 
@@ -859,6 +862,9 @@ static int eb_lookup_vmas(struct i915_execbuffer *eb)
 			goto err;
 		}
 
+		if (eb->exec[i].flags & EXEC_OBJECT_CAPTURE)
+			eb->capture_count++;
+
 		err = eb_validate_vma(eb, &eb->exec[i], vma);
 		if (unlikely(err)) {
 			i915_vma_put(vma);
@@ -1890,16 +1896,8 @@ static int eb_move_to_gpu(struct i915_execbuffer *eb)
 
 		assert_vma_held(vma);
 
-		if (flags & EXEC_OBJECT_CAPTURE) {
-			struct i915_capture_list *capture;
-
-			capture = kmalloc(sizeof(*capture), GFP_KERNEL);
-			if (capture) {
-				capture->next = eb->request->capture_list;
-				capture->vma = vma;
-				eb->request->capture_list = capture;
-			}
-		}
+		if (flags & EXEC_OBJECT_CAPTURE && eb->request->capture_list)
+			eb->request->capture_list[--eb->capture_count] = vma;
 
 		/*
 		 * If the GPU is not _reading_ through the CPU cache, we need
@@ -2828,6 +2826,7 @@ i915_gem_do_execbuffer(struct drm_device *dev,
 
 	eb.fences = NULL;
 	eb.num_fences = 0;
+	eb.capture_count = 0;
 
 	eb.batch_flags = 0;
 	if (args->flags & I915_EXEC_SECURE) {
@@ -2956,6 +2955,13 @@ i915_gem_do_execbuffer(struct drm_device *dev,
 		}
 	}
 
+	if (eb.capture_count) {
+		eb.request->capture_list =
+			kvcalloc(eb.capture_count + 1,
+				 sizeof(*eb.request->capture_list),
+				 GFP_KERNEL | __GFP_NOWARN);
+	}
+
 	/*
 	 * Whilst this request exists, batch_obj will be on the
 	 * active_list, and so will hold the active reference. Only when this
diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
index b9f66dbd46bb..44d07923c8d6 100644
--- a/drivers/gpu/drm/i915/i915_gpu_error.c
+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
@@ -1355,10 +1355,10 @@ capture_user(struct intel_engine_capture_vma *capture,
 	     const struct i915_request *rq,
 	     gfp_t gfp)
 {
-	struct i915_capture_list *c;
+	int i;
 
-	for (c = rq->capture_list; c; c = c->next)
-		capture = capture_vma(capture, c->vma, "user", gfp);
+	for (i = 0; rq->capture_list[i]; i++)
+		capture = capture_vma(capture, rq->capture_list[i], "user", gfp);
 
 	return capture;
 }
@@ -1406,7 +1406,8 @@ intel_engine_coredump_add_request(struct intel_engine_coredump *ee,
 	 * by userspace.
 	 */
 	vma = capture_vma(vma, rq->batch, "batch", gfp);
-	vma = capture_user(vma, rq, gfp);
+	if (rq->capture_list)
+		vma = capture_user(vma, rq, gfp);
 	vma = capture_vma(vma, rq->ring->vma, "ring", gfp);
 	vma = capture_vma(vma, rq->context->state, "HW context", gfp);
 
diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index ce446716d092..4fca2722891c 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -188,15 +188,10 @@ void i915_request_notify_execute_cb_imm(struct i915_request *rq)
 
 static void free_capture_list(struct i915_request *request)
 {
-	struct i915_capture_list *capture;
+	struct i915_vma **capture;
 
 	capture = fetch_and_zero(&request->capture_list);
-	while (capture) {
-		struct i915_capture_list *next = capture->next;
-
-		kfree(capture);
-		capture = next;
-	}
+	kvfree(capture);
 }
 
 static void __i915_request_fill(struct i915_request *rq, u8 val)
diff --git a/drivers/gpu/drm/i915/i915_request.h b/drivers/gpu/drm/i915/i915_request.h
index 1bc1349ba3c2..e7c8cacefcca 100644
--- a/drivers/gpu/drm/i915/i915_request.h
+++ b/drivers/gpu/drm/i915/i915_request.h
@@ -48,11 +48,6 @@ struct drm_i915_gem_object;
 struct drm_printer;
 struct i915_request;
 
-struct i915_capture_list {
-	struct i915_capture_list *next;
-	struct i915_vma *vma;
-};
-
 #define RQ_TRACE(rq, fmt, ...) do {					\
 	const struct i915_request *rq__ = (rq);				\
 	ENGINE_TRACE(rq__->engine, "fence %llx:%lld, current %d " fmt,	\
@@ -271,7 +266,7 @@ struct i915_request {
 	 * active reference - all objects on this list must also be
 	 * on the active_list (of their final request).
 	 */
-	struct i915_capture_list *capture_list;
+	struct i915_vma **capture_list;
 
 	/** Time at which this request was emitted, in jiffies. */
 	unsigned long emitted_jiffies;
-- 
2.32.0

[Intel-gfx] [PATCH 04/19] drm/i915: Remove gen6_ppgtt_unpin_all

[Maarten Lankhorst]

gen6_ppgtt_unpin_all is unused, kill it.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gt/gen6_ppgtt.c | 11 -----------
 drivers/gpu/drm/i915/gt/gen6_ppgtt.h |  1 -
 2 files changed, 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
index 1aee5e6b1b23..efc243f40d0f 100644
--- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
@@ -405,17 +405,6 @@ void gen6_ppgtt_unpin(struct i915_ppgtt *base)
 		i915_vma_unpin(ppgtt->vma);
 }
 
-void gen6_ppgtt_unpin_all(struct i915_ppgtt *base)
-{
-	struct gen6_ppgtt *ppgtt = to_gen6_ppgtt(base);
-
-	if (!atomic_read(&ppgtt->pin_count))
-		return;
-
-	i915_vma_unpin(ppgtt->vma);
-	atomic_set(&ppgtt->pin_count, 0);
-}
-
 struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt)
 {
 	struct i915_ggtt * const ggtt = gt->ggtt;
diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
index 6a61a5c3a85a..ab0eecb086dd 100644
--- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
+++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
@@ -71,7 +71,6 @@ static inline struct gen6_ppgtt *to_gen6_ppgtt(struct i915_ppgtt *base)
 
 int gen6_ppgtt_pin(struct i915_ppgtt *base, struct i915_gem_ww_ctx *ww);
 void gen6_ppgtt_unpin(struct i915_ppgtt *base);
-void gen6_ppgtt_unpin_all(struct i915_ppgtt *base);
 void gen6_ppgtt_enable(struct intel_gt *gt);
 void gen7_ppgtt_enable(struct intel_gt *gt);
 struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt);
-- 
2.32.0

[Intel-gfx] [PATCH 05/19] drm/i915: Create a dummy object for gen6 ppgtt

[Maarten Lankhorst]

We currently have to special case vma->obj being NULL because
of gen6 ppgtt and mock_engine. Fix gen6 ppgtt, so we may soon
be able to remove a few checks. As the object only exists as
a fake object pointing to ggtt, we have no backing storage,
so no real object is created. It just has to look real enough.

Also kill pin_mutex, it's not compatible with ww locking,
and we can use the vm lock instead.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_internal.c |  44 ++++---
 drivers/gpu/drm/i915/gt/gen6_ppgtt.c         | 122 +++++++++++--------
 drivers/gpu/drm/i915/gt/gen6_ppgtt.h         |   1 -
 drivers/gpu/drm/i915/i915_drv.h              |   4 +
 4 files changed, 99 insertions(+), 72 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_internal.c b/drivers/gpu/drm/i915/gem/i915_gem_internal.c
index 13b217f75055..a78a18d5d1a9 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_internal.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_internal.c
@@ -143,24 +143,10 @@ static const struct drm_i915_gem_object_ops i915_gem_object_internal_ops = {
 	.put_pages = i915_gem_object_put_pages_internal,
 };
 
-/**
- * i915_gem_object_create_internal: create an object with volatile pages
- * @i915: the i915 device
- * @size: the size in bytes of backing storage to allocate for the object
- *
- * Creates a new object that wraps some internal memory for private use.
- * This object is not backed by swappable storage, and as such its contents
- * are volatile and only valid whilst pinned. If the object is reaped by the
- * shrinker, its pages and data will be discarded. Equally, it is not a full
- * GEM object and so not valid for access from userspace. This makes it useful
- * for hardware interfaces like ringbuffers (which are pinned from the time
- * the request is written to the time the hardware stops accessing it), but
- * not for contexts (which need to be preserved when not active for later
- * reuse). Note that it is not cleared upon allocation.
- */
 struct drm_i915_gem_object *
-i915_gem_object_create_internal(struct drm_i915_private *i915,
-				phys_addr_t size)
+__i915_gem_object_create_internal(struct drm_i915_private *i915,
+				  const struct drm_i915_gem_object_ops *ops,
+				  phys_addr_t size)
 {
 	static struct lock_class_key lock_class;
 	struct drm_i915_gem_object *obj;
@@ -177,7 +163,7 @@ i915_gem_object_create_internal(struct drm_i915_private *i915,
 		return ERR_PTR(-ENOMEM);
 
 	drm_gem_private_object_init(&i915->drm, &obj->base, size);
-	i915_gem_object_init(obj, &i915_gem_object_internal_ops, &lock_class, 0);
+	i915_gem_object_init(obj, ops, &lock_class, 0);
 	obj->mem_flags |= I915_BO_FLAG_STRUCT_PAGE;
 
 	/*
@@ -197,3 +183,25 @@ i915_gem_object_create_internal(struct drm_i915_private *i915,
 
 	return obj;
 }
+
+/**
+ * i915_gem_object_create_internal: create an object with volatile pages
+ * @i915: the i915 device
+ * @size: the size in bytes of backing storage to allocate for the object
+ *
+ * Creates a new object that wraps some internal memory for private use.
+ * This object is not backed by swappable storage, and as such its contents
+ * are volatile and only valid whilst pinned. If the object is reaped by the
+ * shrinker, its pages and data will be discarded. Equally, it is not a full
+ * GEM object and so not valid for access from userspace. This makes it useful
+ * for hardware interfaces like ringbuffers (which are pinned from the time
+ * the request is written to the time the hardware stops accessing it), but
+ * not for contexts (which need to be preserved when not active for later
+ * reuse). Note that it is not cleared upon allocation.
+ */
+struct drm_i915_gem_object *
+i915_gem_object_create_internal(struct drm_i915_private *i915,
+				phys_addr_t size)
+{
+	return __i915_gem_object_create_internal(i915, &i915_gem_object_internal_ops, size);
+}
diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
index efc243f40d0f..744be15f3e0d 100644
--- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
@@ -262,13 +262,10 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 {
 	struct gen6_ppgtt *ppgtt = to_gen6_ppgtt(i915_vm_to_ppgtt(vm));
 
-	__i915_vma_put(ppgtt->vma);
-
 	gen6_ppgtt_free_pd(ppgtt);
 	free_scratch(vm);
 
 	mutex_destroy(&ppgtt->flush);
-	mutex_destroy(&ppgtt->pin_mutex);
 
 	free_pd(&ppgtt->base.vm, ppgtt->base.pd);
 }
@@ -331,37 +328,6 @@ static const struct i915_vma_ops pd_vma_ops = {
 	.unbind_vma = pd_vma_unbind,
 };
 
-static struct i915_vma *pd_vma_create(struct gen6_ppgtt *ppgtt, int size)
-{
-	struct i915_ggtt *ggtt = ppgtt->base.vm.gt->ggtt;
-	struct i915_vma *vma;
-
-	GEM_BUG_ON(!IS_ALIGNED(size, I915_GTT_PAGE_SIZE));
-	GEM_BUG_ON(size > ggtt->vm.total);
-
-	vma = i915_vma_alloc();
-	if (!vma)
-		return ERR_PTR(-ENOMEM);
-
-	i915_active_init(&vma->active, NULL, NULL, 0);
-
-	kref_init(&vma->ref);
-	mutex_init(&vma->pages_mutex);
-	vma->vm = i915_vm_get(&ggtt->vm);
-	vma->ops = &pd_vma_ops;
-	vma->private = ppgtt;
-
-	vma->size = size;
-	vma->fence_size = size;
-	atomic_set(&vma->flags, I915_VMA_GGTT);
-	vma->ggtt_view.type = I915_GGTT_VIEW_ROTATED; /* prevent fencing */
-
-	INIT_LIST_HEAD(&vma->obj_link);
-	INIT_LIST_HEAD(&vma->closed_link);
-
-	return vma;
-}
-
 int gen6_ppgtt_pin(struct i915_ppgtt *base, struct i915_gem_ww_ctx *ww)
 {
 	struct gen6_ppgtt *ppgtt = to_gen6_ppgtt(base);
@@ -378,24 +344,84 @@ int gen6_ppgtt_pin(struct i915_ppgtt *base, struct i915_gem_ww_ctx *ww)
 	if (atomic_add_unless(&ppgtt->pin_count, 1, 0))
 		return 0;
 
-	if (mutex_lock_interruptible(&ppgtt->pin_mutex))
-		return -EINTR;
+	/* grab the ppgtt resv to pin the object */
+	err = i915_vm_lock_objects(&ppgtt->base.vm, ww);
+	if (err)
+		return err;
 
 	/*
 	 * PPGTT PDEs reside in the GGTT and consists of 512 entries. The
 	 * allocator works in address space sizes, so it's multiplied by page
 	 * size. We allocate at the top of the GTT to avoid fragmentation.
 	 */
-	err = 0;
-	if (!atomic_read(&ppgtt->pin_count))
+	if (!atomic_read(&ppgtt->pin_count)) {
 		err = i915_ggtt_pin(ppgtt->vma, ww, GEN6_PD_ALIGN, PIN_HIGH);
+
+		GEM_BUG_ON(ppgtt->vma->fence);
+		clear_bit(I915_VMA_CAN_FENCE_BIT, __i915_vma_flags(ppgtt->vma));
+	}
 	if (!err)
 		atomic_inc(&ppgtt->pin_count);
-	mutex_unlock(&ppgtt->pin_mutex);
 
 	return err;
 }
 
+static int pd_dummy_obj_get_pages(struct drm_i915_gem_object *obj)
+{
+	obj->mm.pages = ZERO_SIZE_PTR;
+	return 0;
+}
+
+static void pd_dummy_obj_put_pages(struct drm_i915_gem_object *obj,
+				     struct sg_table *pages)
+{
+}
+
+static const struct drm_i915_gem_object_ops pd_dummy_obj_ops = {
+	.name = "pd_dummy_obj",
+	.flags = I915_GEM_OBJECT_IS_SHRINKABLE,
+	.get_pages = pd_dummy_obj_get_pages,
+	.put_pages = pd_dummy_obj_put_pages,
+};
+
+static struct i915_page_directory *
+gen6_alloc_top_pd(struct gen6_ppgtt *ppgtt)
+{
+	struct i915_ggtt * const ggtt = ppgtt->base.vm.gt->ggtt;
+	struct i915_page_directory *pd;
+	int err;
+
+	pd = __alloc_pd(I915_PDES);
+	if (unlikely(!pd))
+		return ERR_PTR(-ENOMEM);
+
+	pd->pt.base = __i915_gem_object_create_internal(ppgtt->base.vm.gt->i915, &pd_dummy_obj_ops, I915_PDES * SZ_4K);
+	if (IS_ERR(pd->pt.base)) {
+		err = PTR_ERR(pd->pt.base);
+		pd->pt.base = NULL;
+		goto err_pd;
+	}
+
+	pd->pt.base->base.resv = i915_vm_resv_get(&ppgtt->base.vm);
+	pd->pt.base->shares_resv_from = &ppgtt->base.vm;
+
+	ppgtt->vma = i915_vma_instance(pd->pt.base, &ggtt->vm, NULL);
+	if (IS_ERR(ppgtt->vma)) {
+		err = PTR_ERR(ppgtt->vma);
+		ppgtt->vma = NULL;
+		goto err_pd;
+	}
+
+	/* The dummy object we create is special, override ops.. */
+	ppgtt->vma->ops = &pd_vma_ops;
+	ppgtt->vma->private = ppgtt;
+	return pd;
+
+err_pd:
+	free_pd(&ppgtt->base.vm, pd);
+	return ERR_PTR(err);
+}
+
 void gen6_ppgtt_unpin(struct i915_ppgtt *base)
 {
 	struct gen6_ppgtt *ppgtt = to_gen6_ppgtt(base);
@@ -416,7 +442,6 @@ struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt)
 		return ERR_PTR(-ENOMEM);
 
 	mutex_init(&ppgtt->flush);
-	mutex_init(&ppgtt->pin_mutex);
 
 	ppgtt_init(&ppgtt->base, gt);
 	ppgtt->base.vm.pd_shift = ilog2(SZ_4K * SZ_4K / sizeof(gen6_pte_t));
@@ -431,19 +456,13 @@ struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt)
 	ppgtt->base.vm.alloc_pt_dma = alloc_pt_dma;
 	ppgtt->base.vm.pte_encode = ggtt->vm.pte_encode;
 
-	ppgtt->base.pd = __alloc_pd(I915_PDES);
-	if (!ppgtt->base.pd) {
-		err = -ENOMEM;
-		goto err_free;
-	}
-
 	err = gen6_ppgtt_init_scratch(ppgtt);
 	if (err)
-		goto err_pd;
+		goto err_free;
 
-	ppgtt->vma = pd_vma_create(ppgtt, GEN6_PD_SIZE);
-	if (IS_ERR(ppgtt->vma)) {
-		err = PTR_ERR(ppgtt->vma);
+	ppgtt->base.pd = gen6_alloc_top_pd(ppgtt);
+	if (IS_ERR(ppgtt->base.pd)) {
+		err = PTR_ERR(ppgtt->base.pd);
 		goto err_scratch;
 	}
 
@@ -451,10 +470,7 @@ struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt)
 
 err_scratch:
 	free_scratch(&ppgtt->base.vm);
-err_pd:
-	free_pd(&ppgtt->base.vm, ppgtt->base.pd);
 err_free:
-	mutex_destroy(&ppgtt->pin_mutex);
 	kfree(ppgtt);
 	return ERR_PTR(err);
 }
diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
index ab0eecb086dd..5e5cf2ec3309 100644
--- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
+++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
@@ -19,7 +19,6 @@ struct gen6_ppgtt {
 	u32 pp_dir;
 
 	atomic_t pin_count;
-	struct mutex pin_mutex;
 
 	bool scan_for_unused_pt;
 };
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 033031169d74..ac4cea32601f 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1900,6 +1900,10 @@ int i915_gem_evict_vm(struct i915_address_space *vm);
 struct drm_i915_gem_object *
 i915_gem_object_create_internal(struct drm_i915_private *dev_priv,
 				phys_addr_t size);
+struct drm_i915_gem_object *
+__i915_gem_object_create_internal(struct drm_i915_private *dev_priv,
+				  const struct drm_i915_gem_object_ops *ops,
+				  phys_addr_t size);
 
 /* i915_gem_tiling.c */
 static inline bool i915_gem_object_needs_bit17_swizzle(struct drm_i915_gem_object *obj)
-- 
2.32.0

[Intel-gfx] [PATCH 06/19] drm/i915: Create a full object for mock_ring

[Maarten Lankhorst]

This allows us to finally get rid of all the assumptions that vma->obj is NULL.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gt/mock_engine.c | 35 ++++++++++++++++++++-------
 1 file changed, 26 insertions(+), 9 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/mock_engine.c b/drivers/gpu/drm/i915/gt/mock_engine.c
index 2c1af030310c..67c17188e17a 100644
--- a/drivers/gpu/drm/i915/gt/mock_engine.c
+++ b/drivers/gpu/drm/i915/gt/mock_engine.c
@@ -35,9 +35,31 @@ static void mock_timeline_unpin(struct intel_timeline *tl)
 	atomic_dec(&tl->pin_count);
 }
 
+static struct i915_vma *create_ring_vma(struct i915_ggtt *ggtt, int size)
+{
+	struct i915_address_space *vm = &ggtt->vm;
+	struct drm_i915_private *i915 = vm->i915;
+	struct drm_i915_gem_object *obj;
+	struct i915_vma *vma;
+
+	obj = i915_gem_object_create_internal(i915, size);
+	if (IS_ERR(obj))
+		return ERR_CAST(obj);
+
+	vma = i915_vma_instance(obj, vm, NULL);
+	if (IS_ERR(vma))
+		goto err;
+
+	return vma;
+
+err:
+	i915_gem_object_put(obj);
+	return vma;
+}
+
 static struct intel_ring *mock_ring(struct intel_engine_cs *engine)
 {
-	const unsigned long sz = PAGE_SIZE / 2;
+	const unsigned long sz = PAGE_SIZE;
 	struct intel_ring *ring;
 
 	ring = kzalloc(sizeof(*ring) + sz, GFP_KERNEL);
@@ -50,15 +72,11 @@ static struct intel_ring *mock_ring(struct intel_engine_cs *engine)
 	ring->vaddr = (void *)(ring + 1);
 	atomic_set(&ring->pin_count, 1);
 
-	ring->vma = i915_vma_alloc();
-	if (!ring->vma) {
+	ring->vma = create_ring_vma(engine->gt->ggtt, PAGE_SIZE);
+	if (IS_ERR(ring->vma)) {
 		kfree(ring);
 		return NULL;
 	}
-	i915_active_init(&ring->vma->active, NULL, NULL, 0);
-	__set_bit(I915_VMA_GGTT_BIT, __i915_vma_flags(ring->vma));
-	__set_bit(DRM_MM_NODE_ALLOCATED_BIT, &ring->vma->node.flags);
-	ring->vma->node.size = sz;
 
 	intel_ring_update_space(ring);
 
@@ -67,8 +85,7 @@ static struct intel_ring *mock_ring(struct intel_engine_cs *engine)
 
 static void mock_ring_free(struct intel_ring *ring)
 {
-	i915_active_fini(&ring->vma->active);
-	i915_vma_free(ring->vma);
+	i915_vma_put(ring->vma);
 
 	kfree(ring);
 }
-- 
2.32.0

[Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Maarten Lankhorst]

vma->obj and vma->resv are now never NULL, and some checks can be removed.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gt/intel_context.c       |  2 +-
 .../gpu/drm/i915/gt/intel_ring_submission.c   |  2 +-
 drivers/gpu/drm/i915/i915_vma.c               | 48 ++++++++-----------
 drivers/gpu/drm/i915/i915_vma.h               |  3 --
 4 files changed, 22 insertions(+), 33 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_context.c b/drivers/gpu/drm/i915/gt/intel_context.c
index 745e84c72c90..d3ad16df3ca2 100644
--- a/drivers/gpu/drm/i915/gt/intel_context.c
+++ b/drivers/gpu/drm/i915/gt/intel_context.c
@@ -219,7 +219,7 @@ int __intel_context_do_pin_ww(struct intel_context *ce,
 	 */
 
 	err = i915_gem_object_lock(ce->timeline->hwsp_ggtt->obj, ww);
-	if (!err && ce->ring->vma->obj)
+	if (!err)
 		err = i915_gem_object_lock(ce->ring->vma->obj, ww);
 	if (!err && ce->state)
 		err = i915_gem_object_lock(ce->state->obj, ww);
diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
index 3c65efcb7bed..cc31ccc13bfb 100644
--- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c
+++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
@@ -1354,7 +1354,7 @@ int intel_ring_submission_setup(struct intel_engine_cs *engine)
 	err = i915_gem_object_lock(timeline->hwsp_ggtt->obj, &ww);
 	if (!err && gen7_wa_vma)
 		err = i915_gem_object_lock(gen7_wa_vma->obj, &ww);
-	if (!err && engine->legacy.ring->vma->obj)
+	if (!err)
 		err = i915_gem_object_lock(engine->legacy.ring->vma->obj, &ww);
 	if (!err)
 		err = intel_timeline_pin(timeline, &ww);
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index f9ac33e0bac9..ad5d52b33eb6 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -40,12 +40,12 @@
 
 static struct kmem_cache *slab_vmas;
 
-struct i915_vma *i915_vma_alloc(void)
+static struct i915_vma *i915_vma_alloc(void)
 {
 	return kmem_cache_zalloc(slab_vmas, GFP_KERNEL);
 }
 
-void i915_vma_free(struct i915_vma *vma)
+static void i915_vma_free(struct i915_vma *vma)
 {
 	return kmem_cache_free(slab_vmas, vma);
 }
@@ -426,10 +426,8 @@ int i915_vma_bind(struct i915_vma *vma,
 
 		work->base.dma.error = 0; /* enable the queue_work() */
 
-		if (vma->obj) {
-			__i915_gem_object_pin_pages(vma->obj);
-			work->pinned = i915_gem_object_get(vma->obj);
-		}
+		__i915_gem_object_pin_pages(vma->obj);
+		work->pinned = i915_gem_object_get(vma->obj);
 	} else {
 		vma->ops->bind_vma(vma->vm, NULL, vma, cache_level, bind_flags);
 	}
@@ -670,7 +668,7 @@ i915_vma_insert(struct i915_vma *vma, u64 size, u64 alignment, u64 flags)
 	}
 
 	color = 0;
-	if (vma->obj && i915_vm_has_cache_coloring(vma->vm))
+	if (i915_vm_has_cache_coloring(vma->vm))
 		color = vma->obj->cache_level;
 
 	if (flags & PIN_OFFSET_FIXED) {
@@ -795,17 +793,14 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
 static int vma_get_pages(struct i915_vma *vma)
 {
 	int err = 0;
-	bool pinned_pages = false;
+	bool pinned_pages = true;
 
 	if (atomic_add_unless(&vma->pages_count, 1, 0))
 		return 0;
 
-	if (vma->obj) {
-		err = i915_gem_object_pin_pages(vma->obj);
-		if (err)
-			return err;
-		pinned_pages = true;
-	}
+	err = i915_gem_object_pin_pages(vma->obj);
+	if (err)
+		return err;
 
 	/* Allocations ahoy! */
 	if (mutex_lock_interruptible(&vma->pages_mutex)) {
@@ -838,8 +833,8 @@ static void __vma_put_pages(struct i915_vma *vma, unsigned int count)
 	if (atomic_sub_return(count, &vma->pages_count) == 0) {
 		vma->ops->clear_pages(vma);
 		GEM_BUG_ON(vma->pages);
-		if (vma->obj)
-			i915_gem_object_unpin_pages(vma->obj);
+
+		i915_gem_object_unpin_pages(vma->obj);
 	}
 	mutex_unlock(&vma->pages_mutex);
 }
@@ -875,7 +870,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	int err;
 
 #ifdef CONFIG_PROVE_LOCKING
-	if (debug_locks && !WARN_ON(!ww) && vma->resv)
+	if (debug_locks && !WARN_ON(!ww))
 		assert_vma_held(vma);
 #endif
 
@@ -983,7 +978,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 
 	GEM_BUG_ON(!vma->pages);
 	err = i915_vma_bind(vma,
-			    vma->obj ? vma->obj->cache_level : 0,
+			    vma->obj->cache_level,
 			    flags, work);
 	if (err)
 		goto err_remove;
@@ -1037,7 +1032,7 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
 
 #ifdef CONFIG_LOCKDEP
-	WARN_ON(!ww && vma->resv && dma_resv_held(vma->resv));
+	WARN_ON(!ww && dma_resv_held(vma->resv));
 #endif
 
 	do {
@@ -1116,6 +1111,7 @@ void i915_vma_reopen(struct i915_vma *vma)
 void i915_vma_release(struct kref *ref)
 {
 	struct i915_vma *vma = container_of(ref, typeof(*vma), ref);
+	struct drm_i915_gem_object *obj = vma->obj;
 
 	if (drm_mm_node_allocated(&vma->node)) {
 		mutex_lock(&vma->vm->mutex);
@@ -1126,15 +1122,11 @@ void i915_vma_release(struct kref *ref)
 	}
 	GEM_BUG_ON(i915_vma_is_active(vma));
 
-	if (vma->obj) {
-		struct drm_i915_gem_object *obj = vma->obj;
-
-		spin_lock(&obj->vma.lock);
-		list_del(&vma->obj_link);
-		if (!RB_EMPTY_NODE(&vma->obj_node))
-			rb_erase(&vma->obj_node, &obj->vma.tree);
-		spin_unlock(&obj->vma.lock);
-	}
+	spin_lock(&obj->vma.lock);
+	list_del(&vma->obj_link);
+	if (!RB_EMPTY_NODE(&vma->obj_node))
+		rb_erase(&vma->obj_node, &obj->vma.tree);
+	spin_unlock(&obj->vma.lock);
 
 	__i915_vma_remove_closed(vma);
 	i915_vm_put(vma->vm);
diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
index 1c930515ec3d..1ba82bf863a5 100644
--- a/drivers/gpu/drm/i915/i915_vma.h
+++ b/drivers/gpu/drm/i915/i915_vma.h
@@ -409,9 +409,6 @@ static inline void i915_vma_clear_scanout(struct i915_vma *vma)
 	list_for_each_entry(V, &(OBJ)->vma.list, obj_link)		\
 		for_each_until(!i915_vma_is_ggtt(V))
 
-struct i915_vma *i915_vma_alloc(void);
-void i915_vma_free(struct i915_vma *vma);
-
 struct i915_vma *i915_vma_make_unshrinkable(struct i915_vma *vma);
 void i915_vma_make_shrinkable(struct i915_vma *vma);
 void i915_vma_make_purgeable(struct i915_vma *vma);
-- 
2.32.0

[Intel-gfx] [PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Maarten Lankhorst]

We forgot to call intel_runtime_pm_put on error, fix it!

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: <stable@vger.kernel.org> # v5.13+
---
 drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
index e382b7f2353b..5ab136ffdeb2 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
@@ -118,7 +118,7 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
 	intel_wakeref_t wakeref = 0;
 	unsigned long count = 0;
 	unsigned long scanned = 0;
-	int err;
+	int err = 0;
 
 	/* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
 	bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
@@ -242,12 +242,15 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
 		list_splice_tail(&still_in_list, phase->list);
 		spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
 		if (err)
-			return err;
+			break;
 	}
 
 	if (shrink & I915_SHRINK_BOUND)
 		intel_runtime_pm_put(&i915->runtime_pm, wakeref);
 
+	if (err)
+		return err;
+
 	if (nr_scanned)
 		*nr_scanned += scanned;
 	return count;
-- 
2.32.0

[PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Maarten Lankhorst]

We forgot to call intel_runtime_pm_put on error, fix it!

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
Cc: <stable@vger.kernel.org> # v5.13+
---
 drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
index e382b7f2353b..5ab136ffdeb2 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
@@ -118,7 +118,7 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
 	intel_wakeref_t wakeref = 0;
 	unsigned long count = 0;
 	unsigned long scanned = 0;
-	int err;
+	int err = 0;
 
 	/* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
 	bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
@@ -242,12 +242,15 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
 		list_splice_tail(&still_in_list, phase->list);
 		spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
 		if (err)
-			return err;
+			break;
 	}
 
 	if (shrink & I915_SHRINK_BOUND)
 		intel_runtime_pm_put(&i915->runtime_pm, wakeref);
 
+	if (err)
+		return err;
+
 	if (nr_scanned)
 		*nr_scanned += scanned;
 	return count;
-- 
2.32.0

[Intel-gfx] [PATCH 09/19] drm/i915: Change shrink ordering to use locking around unbinding.

[Maarten Lankhorst]

Call drop_pages with the gem object lock held, instead of the other
way around. This will allow us to drop the vma bindings with the
gem object lock held.

We plan to require the object lock for unpinning in the future,
and this is an easy target.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 42 ++++++++++----------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
index 5ab136ffdeb2..7f7849b6296d 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
@@ -37,8 +37,8 @@ static bool can_release_pages(struct drm_i915_gem_object *obj)
 	return swap_available() || obj->mm.madv == I915_MADV_DONTNEED;
 }
 
-static bool unsafe_drop_pages(struct drm_i915_gem_object *obj,
-			      unsigned long shrink, bool trylock_vm)
+static int drop_pages(struct drm_i915_gem_object *obj,
+		       unsigned long shrink, bool trylock_vm)
 {
 	unsigned long flags;
 
@@ -209,27 +209,27 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
 
 			spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
 
-			err = 0;
-			if (unsafe_drop_pages(obj, shrink, trylock_vm)) {
-				/* May arrive from get_pages on another bo */
-				if (!ww) {
-					if (!i915_gem_object_trylock(obj))
-						goto skip;
-				} else {
-					err = i915_gem_object_lock(obj, ww);
-					if (err)
-						goto skip;
-				}
-
-				if (!__i915_gem_object_put_pages(obj)) {
-					try_to_writeback(obj, shrink);
-					count += obj->base.size >> PAGE_SHIFT;
-				}
-				if (!ww)
-					i915_gem_object_unlock(obj);
+			/* May arrive from get_pages on another bo */
+			if (!ww) {
+				if (!i915_gem_object_trylock(obj))
+					goto skip;
+			} else {
+				err = i915_gem_object_lock(obj, ww);
+				if (err)
+					goto skip;
 			}
 
-			dma_resv_prune(obj->base.resv);
+			if (drop_pages(obj, shrink, trylock_vm) &&
+			    !__i915_gem_object_put_pages(obj)) {
+				try_to_writeback(obj, shrink);
+				count += obj->base.size >> PAGE_SHIFT;
+			}
+
+			if (dma_resv_test_signaled(obj->base.resv, true))
+				dma_resv_add_excl_fence(obj->base.resv, NULL);
+
+			if (!ww)
+				i915_gem_object_unlock(obj);
 
 			scanned += obj->base.size >> PAGE_SHIFT;
 skip:
-- 
2.32.0

[Intel-gfx] [PATCH 10/19] Move CONTEXT_VALID_BIT check

[Maarten Lankhorst]

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_pm.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_pm.c b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
index 1f07ac4e0672..df81a0dc481e 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_pm.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
@@ -52,8 +52,6 @@ static int __engine_unpark(struct intel_wakeref *wf)
 	/* Discard stale context state from across idling */
 	ce = engine->kernel_context;
 	if (ce) {
-		GEM_BUG_ON(test_bit(CONTEXT_VALID_BIT, &ce->flags));
-
 		/* Flush all pending HW writes before we touch the context */
 		while (unlikely(intel_context_inflight(ce)))
 			intel_engine_flush_submission(engine);
@@ -68,6 +66,9 @@ static int __engine_unpark(struct intel_wakeref *wf)
 			 ce->timeline->seqno,
 			 READ_ONCE(*ce->timeline->hwsp_seqno),
 			 ce->ring->emit);
+
+		GEM_BUG_ON(test_bit(CONTEXT_VALID_BIT, &ce->flags));
+
 		GEM_BUG_ON(ce->timeline->seqno !=
 			   READ_ONCE(*ce->timeline->hwsp_seqno));
 	}
-- 
2.32.0

[Intel-gfx] [PATCH 11/19] drm/i915: Remove resv from i915_vma

[Maarten Lankhorst]

It's just an alias to vma->obj->base.resv, no need to duplicate it.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 ++--
 drivers/gpu/drm/i915/i915_vma.c                | 9 ++++-----
 drivers/gpu/drm/i915/i915_vma.h                | 6 +++---
 drivers/gpu/drm/i915/i915_vma_types.h          | 1 -
 4 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 4f10a81befd1..6832b255294e 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -975,7 +975,7 @@ static int eb_validate_vmas(struct i915_execbuffer *eb)
 		}
 
 		if (!(ev->flags & EXEC_OBJECT_WRITE)) {
-			err = dma_resv_reserve_shared(vma->resv, 1);
+			err = dma_resv_reserve_shared(vma->obj->base.resv, 1);
 			if (err)
 				return err;
 		}
@@ -2121,7 +2121,7 @@ static int eb_parse(struct i915_execbuffer *eb)
 		goto err_trampoline;
 	}
 
-	err = dma_resv_reserve_shared(shadow->resv, 1);
+	err = dma_resv_reserve_shared(shadow->obj->base.resv, 1);
 	if (err)
 		goto err_trampoline;
 
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index ad5d52b33eb6..36db641e863e 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -116,7 +116,6 @@ vma_create(struct drm_i915_gem_object *obj,
 	vma->vm = i915_vm_get(vm);
 	vma->ops = &vm->vma_ops;
 	vma->obj = obj;
-	vma->resv = obj->base.resv;
 	vma->size = obj->base.size;
 	vma->display_alignment = I915_GTT_MIN_ALIGNMENT;
 
@@ -1032,7 +1031,7 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
 
 #ifdef CONFIG_LOCKDEP
-	WARN_ON(!ww && dma_resv_held(vma->resv));
+	WARN_ON(!ww && dma_resv_held(vma->obj->base.resv));
 #endif
 
 	do {
@@ -1249,17 +1248,17 @@ int i915_vma_move_to_active(struct i915_vma *vma,
 			intel_frontbuffer_put(front);
 		}
 
-		dma_resv_add_excl_fence(vma->resv, &rq->fence);
+		dma_resv_add_excl_fence(vma->obj->base.resv, &rq->fence);
 		obj->write_domain = I915_GEM_DOMAIN_RENDER;
 		obj->read_domains = 0;
 	} else {
 		if (!(flags & __EXEC_OBJECT_NO_RESERVE)) {
-			err = dma_resv_reserve_shared(vma->resv, 1);
+			err = dma_resv_reserve_shared(vma->obj->base.resv, 1);
 			if (unlikely(err))
 				return err;
 		}
 
-		dma_resv_add_shared_fence(vma->resv, &rq->fence);
+		dma_resv_add_shared_fence(vma->obj->base.resv, &rq->fence);
 		obj->write_domain = 0;
 	}
 
diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
index 1ba82bf863a5..0b11c188f749 100644
--- a/drivers/gpu/drm/i915/i915_vma.h
+++ b/drivers/gpu/drm/i915/i915_vma.h
@@ -225,16 +225,16 @@ static inline void __i915_vma_put(struct i915_vma *vma)
 	kref_put(&vma->ref, i915_vma_release);
 }
 
-#define assert_vma_held(vma) dma_resv_assert_held((vma)->resv)
+#define assert_vma_held(vma) dma_resv_assert_held((vma)->obj->base.resv)
 
 static inline void i915_vma_lock(struct i915_vma *vma)
 {
-	dma_resv_lock(vma->resv, NULL);
+	dma_resv_lock(vma->obj->base.resv, NULL);
 }
 
 static inline void i915_vma_unlock(struct i915_vma *vma)
 {
-	dma_resv_unlock(vma->resv);
+	dma_resv_unlock(vma->obj->base.resv);
 }
 
 int __must_check
diff --git a/drivers/gpu/drm/i915/i915_vma_types.h b/drivers/gpu/drm/i915/i915_vma_types.h
index 995b502d7e5d..47573ed2d11f 100644
--- a/drivers/gpu/drm/i915/i915_vma_types.h
+++ b/drivers/gpu/drm/i915/i915_vma_types.h
@@ -177,7 +177,6 @@ struct i915_vma {
 	const struct i915_vma_ops *ops;
 
 	struct drm_i915_gem_object *obj;
-	struct dma_resv *resv; /** Alias of obj->resv */
 
 	struct sg_table *pages;
 	void __iomem *iomap;
-- 
2.32.0

[Intel-gfx] [PATCH 12/19] drm/i915: Remove pages_mutex and intel_gtt->vma_ops.set/clear_pages members

[Maarten Lankhorst]

Big delta, but boils down to moving set_pages to i915_vma.c, and removing
the special handling, all callers use the defaults anyway. We only remap
in ggtt, so default case will fall through.

Because we still don't require locking in i915_vma_unpin(), handle this by
using xchg in get_pages(), as it's locked with obj->mutex, and cmpxchg in
unpin, which only fails if we race a against a new pin.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/display/intel_dpt.c      |   2 -
 drivers/gpu/drm/i915/gt/gen6_ppgtt.c          |  15 -
 drivers/gpu/drm/i915/gt/intel_ggtt.c          | 320 ----------------
 drivers/gpu/drm/i915/gt/intel_gtt.c           |  13 -
 drivers/gpu/drm/i915/gt/intel_gtt.h           |   7 -
 drivers/gpu/drm/i915/gt/intel_ppgtt.c         |  12 -
 drivers/gpu/drm/i915/i915_vma.c               | 362 ++++++++++++++++--
 drivers/gpu/drm/i915/i915_vma.h               |   3 +
 drivers/gpu/drm/i915/i915_vma_types.h         |   1 -
 drivers/gpu/drm/i915/selftests/i915_gem_gtt.c |  12 +-
 drivers/gpu/drm/i915/selftests/mock_gtt.c     |   4 -
 11 files changed, 342 insertions(+), 409 deletions(-)

diff --git a/drivers/gpu/drm/i915/display/intel_dpt.c b/drivers/gpu/drm/i915/display/intel_dpt.c
index de62bd77b15e..a66726f8908f 100644
--- a/drivers/gpu/drm/i915/display/intel_dpt.c
+++ b/drivers/gpu/drm/i915/display/intel_dpt.c
@@ -211,8 +211,6 @@ intel_dpt_create(struct intel_framebuffer *fb)
 
 	vm->vma_ops.bind_vma    = dpt_bind_vma;
 	vm->vma_ops.unbind_vma  = dpt_unbind_vma;
-	vm->vma_ops.set_pages   = ggtt_set_pages;
-	vm->vma_ops.clear_pages = clear_pages;
 
 	vm->pte_encode = gen8_ggtt_pte_encode;
 
diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
index 744be15f3e0d..9cc5d2379c3e 100644
--- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
@@ -270,19 +270,6 @@ static void gen6_ppgtt_cleanup(struct i915_address_space *vm)
 	free_pd(&ppgtt->base.vm, ppgtt->base.pd);
 }
 
-static int pd_vma_set_pages(struct i915_vma *vma)
-{
-	vma->pages = ERR_PTR(-ENODEV);
-	return 0;
-}
-
-static void pd_vma_clear_pages(struct i915_vma *vma)
-{
-	GEM_BUG_ON(!vma->pages);
-
-	vma->pages = NULL;
-}
-
 static void pd_vma_bind(struct i915_address_space *vm,
 			struct i915_vm_pt_stash *stash,
 			struct i915_vma *vma,
@@ -322,8 +309,6 @@ static void pd_vma_unbind(struct i915_address_space *vm, struct i915_vma *vma)
 }
 
 static const struct i915_vma_ops pd_vma_ops = {
-	.set_pages = pd_vma_set_pages,
-	.clear_pages = pd_vma_clear_pages,
 	.bind_vma = pd_vma_bind,
 	.unbind_vma = pd_vma_unbind,
 };
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index de3ac58fceec..8c46a44e5a40 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -20,9 +20,6 @@
 #include "intel_gtt.h"
 #include "gen8_ppgtt.h"
 
-static int
-i915_get_ggtt_vma_pages(struct i915_vma *vma);
-
 static void i915_ggtt_color_adjust(const struct drm_mm_node *node,
 				   unsigned long color,
 				   u64 *start,
@@ -861,21 +858,6 @@ static int ggtt_probe_common(struct i915_ggtt *ggtt, u64 size)
 	return 0;
 }
 
-int ggtt_set_pages(struct i915_vma *vma)
-{
-	int ret;
-
-	GEM_BUG_ON(vma->pages);
-
-	ret = i915_get_ggtt_vma_pages(vma);
-	if (ret)
-		return ret;
-
-	vma->page_sizes = vma->obj->mm.page_sizes;
-
-	return 0;
-}
-
 static void gen6_gmch_remove(struct i915_address_space *vm)
 {
 	struct i915_ggtt *ggtt = i915_vm_to_ggtt(vm);
@@ -935,8 +917,6 @@ static int gen8_gmch_probe(struct i915_ggtt *ggtt)
 
 	ggtt->vm.vma_ops.bind_vma    = ggtt_bind_vma;
 	ggtt->vm.vma_ops.unbind_vma  = ggtt_unbind_vma;
-	ggtt->vm.vma_ops.set_pages   = ggtt_set_pages;
-	ggtt->vm.vma_ops.clear_pages = clear_pages;
 
 	ggtt->vm.pte_encode = gen8_ggtt_pte_encode;
 
@@ -1085,8 +1065,6 @@ static int gen6_gmch_probe(struct i915_ggtt *ggtt)
 
 	ggtt->vm.vma_ops.bind_vma    = ggtt_bind_vma;
 	ggtt->vm.vma_ops.unbind_vma  = ggtt_unbind_vma;
-	ggtt->vm.vma_ops.set_pages   = ggtt_set_pages;
-	ggtt->vm.vma_ops.clear_pages = clear_pages;
 
 	return ggtt_probe_common(ggtt, size);
 }
@@ -1130,8 +1108,6 @@ static int i915_gmch_probe(struct i915_ggtt *ggtt)
 
 	ggtt->vm.vma_ops.bind_vma    = ggtt_bind_vma;
 	ggtt->vm.vma_ops.unbind_vma  = ggtt_unbind_vma;
-	ggtt->vm.vma_ops.set_pages   = ggtt_set_pages;
-	ggtt->vm.vma_ops.clear_pages = clear_pages;
 
 	if (unlikely(ggtt->do_idle_maps))
 		drm_notice(&i915->drm,
@@ -1279,299 +1255,3 @@ void i915_ggtt_resume(struct i915_ggtt *ggtt)
 
 	intel_ggtt_restore_fences(ggtt);
 }
-
-static struct scatterlist *
-rotate_pages(struct drm_i915_gem_object *obj, unsigned int offset,
-	     unsigned int width, unsigned int height,
-	     unsigned int src_stride, unsigned int dst_stride,
-	     struct sg_table *st, struct scatterlist *sg)
-{
-	unsigned int column, row;
-	unsigned int src_idx;
-
-	for (column = 0; column < width; column++) {
-		unsigned int left;
-
-		src_idx = src_stride * (height - 1) + column + offset;
-		for (row = 0; row < height; row++) {
-			st->nents++;
-			/*
-			 * We don't need the pages, but need to initialize
-			 * the entries so the sg list can be happily traversed.
-			 * The only thing we need are DMA addresses.
-			 */
-			sg_set_page(sg, NULL, I915_GTT_PAGE_SIZE, 0);
-			sg_dma_address(sg) =
-				i915_gem_object_get_dma_address(obj, src_idx);
-			sg_dma_len(sg) = I915_GTT_PAGE_SIZE;
-			sg = sg_next(sg);
-			src_idx -= src_stride;
-		}
-
-		left = (dst_stride - height) * I915_GTT_PAGE_SIZE;
-
-		if (!left)
-			continue;
-
-		st->nents++;
-
-		/*
-		 * The DE ignores the PTEs for the padding tiles, the sg entry
-		 * here is just a conenience to indicate how many padding PTEs
-		 * to insert at this spot.
-		 */
-		sg_set_page(sg, NULL, left, 0);
-		sg_dma_address(sg) = 0;
-		sg_dma_len(sg) = left;
-		sg = sg_next(sg);
-	}
-
-	return sg;
-}
-
-static noinline struct sg_table *
-intel_rotate_pages(struct intel_rotation_info *rot_info,
-		   struct drm_i915_gem_object *obj)
-{
-	unsigned int size = intel_rotation_info_size(rot_info);
-	struct drm_i915_private *i915 = to_i915(obj->base.dev);
-	struct sg_table *st;
-	struct scatterlist *sg;
-	int ret = -ENOMEM;
-	int i;
-
-	/* Allocate target SG list. */
-	st = kmalloc(sizeof(*st), GFP_KERNEL);
-	if (!st)
-		goto err_st_alloc;
-
-	ret = sg_alloc_table(st, size, GFP_KERNEL);
-	if (ret)
-		goto err_sg_alloc;
-
-	st->nents = 0;
-	sg = st->sgl;
-
-	for (i = 0 ; i < ARRAY_SIZE(rot_info->plane); i++)
-		sg = rotate_pages(obj, rot_info->plane[i].offset,
-				  rot_info->plane[i].width, rot_info->plane[i].height,
-				  rot_info->plane[i].src_stride,
-				  rot_info->plane[i].dst_stride,
-				  st, sg);
-
-	return st;
-
-err_sg_alloc:
-	kfree(st);
-err_st_alloc:
-
-	drm_dbg(&i915->drm, "Failed to create rotated mapping for object size %zu! (%ux%u tiles, %u pages)\n",
-		obj->base.size, rot_info->plane[0].width,
-		rot_info->plane[0].height, size);
-
-	return ERR_PTR(ret);
-}
-
-static struct scatterlist *
-remap_pages(struct drm_i915_gem_object *obj, unsigned int offset,
-	    unsigned int width, unsigned int height,
-	    unsigned int src_stride, unsigned int dst_stride,
-	    struct sg_table *st, struct scatterlist *sg)
-{
-	unsigned int row;
-
-	for (row = 0; row < height; row++) {
-		unsigned int left = width * I915_GTT_PAGE_SIZE;
-
-		while (left) {
-			dma_addr_t addr;
-			unsigned int length;
-
-			/*
-			 * We don't need the pages, but need to initialize
-			 * the entries so the sg list can be happily traversed.
-			 * The only thing we need are DMA addresses.
-			 */
-
-			addr = i915_gem_object_get_dma_address_len(obj, offset, &length);
-
-			length = min(left, length);
-
-			st->nents++;
-
-			sg_set_page(sg, NULL, length, 0);
-			sg_dma_address(sg) = addr;
-			sg_dma_len(sg) = length;
-			sg = sg_next(sg);
-
-			offset += length / I915_GTT_PAGE_SIZE;
-			left -= length;
-		}
-
-		offset += src_stride - width;
-
-		left = (dst_stride - width) * I915_GTT_PAGE_SIZE;
-
-		if (!left)
-			continue;
-
-		st->nents++;
-
-		/*
-		 * The DE ignores the PTEs for the padding tiles, the sg entry
-		 * here is just a conenience to indicate how many padding PTEs
-		 * to insert at this spot.
-		 */
-		sg_set_page(sg, NULL, left, 0);
-		sg_dma_address(sg) = 0;
-		sg_dma_len(sg) = left;
-		sg = sg_next(sg);
-	}
-
-	return sg;
-}
-
-static noinline struct sg_table *
-intel_remap_pages(struct intel_remapped_info *rem_info,
-		  struct drm_i915_gem_object *obj)
-{
-	unsigned int size = intel_remapped_info_size(rem_info);
-	struct drm_i915_private *i915 = to_i915(obj->base.dev);
-	struct sg_table *st;
-	struct scatterlist *sg;
-	int ret = -ENOMEM;
-	int i;
-
-	/* Allocate target SG list. */
-	st = kmalloc(sizeof(*st), GFP_KERNEL);
-	if (!st)
-		goto err_st_alloc;
-
-	ret = sg_alloc_table(st, size, GFP_KERNEL);
-	if (ret)
-		goto err_sg_alloc;
-
-	st->nents = 0;
-	sg = st->sgl;
-
-	for (i = 0 ; i < ARRAY_SIZE(rem_info->plane); i++) {
-		sg = remap_pages(obj, rem_info->plane[i].offset,
-				 rem_info->plane[i].width, rem_info->plane[i].height,
-				 rem_info->plane[i].src_stride, rem_info->plane[i].dst_stride,
-				 st, sg);
-	}
-
-	i915_sg_trim(st);
-
-	return st;
-
-err_sg_alloc:
-	kfree(st);
-err_st_alloc:
-
-	drm_dbg(&i915->drm, "Failed to create remapped mapping for object size %zu! (%ux%u tiles, %u pages)\n",
-		obj->base.size, rem_info->plane[0].width,
-		rem_info->plane[0].height, size);
-
-	return ERR_PTR(ret);
-}
-
-static noinline struct sg_table *
-intel_partial_pages(const struct i915_ggtt_view *view,
-		    struct drm_i915_gem_object *obj)
-{
-	struct sg_table *st;
-	struct scatterlist *sg, *iter;
-	unsigned int count = view->partial.size;
-	unsigned int offset;
-	int ret = -ENOMEM;
-
-	st = kmalloc(sizeof(*st), GFP_KERNEL);
-	if (!st)
-		goto err_st_alloc;
-
-	ret = sg_alloc_table(st, count, GFP_KERNEL);
-	if (ret)
-		goto err_sg_alloc;
-
-	iter = i915_gem_object_get_sg_dma(obj, view->partial.offset, &offset);
-	GEM_BUG_ON(!iter);
-
-	sg = st->sgl;
-	st->nents = 0;
-	do {
-		unsigned int len;
-
-		len = min(sg_dma_len(iter) - (offset << PAGE_SHIFT),
-			  count << PAGE_SHIFT);
-		sg_set_page(sg, NULL, len, 0);
-		sg_dma_address(sg) =
-			sg_dma_address(iter) + (offset << PAGE_SHIFT);
-		sg_dma_len(sg) = len;
-
-		st->nents++;
-		count -= len >> PAGE_SHIFT;
-		if (count == 0) {
-			sg_mark_end(sg);
-			i915_sg_trim(st); /* Drop any unused tail entries. */
-
-			return st;
-		}
-
-		sg = __sg_next(sg);
-		iter = __sg_next(iter);
-		offset = 0;
-	} while (1);
-
-err_sg_alloc:
-	kfree(st);
-err_st_alloc:
-	return ERR_PTR(ret);
-}
-
-static int
-i915_get_ggtt_vma_pages(struct i915_vma *vma)
-{
-	int ret;
-
-	/*
-	 * The vma->pages are only valid within the lifespan of the borrowed
-	 * obj->mm.pages. When the obj->mm.pages sg_table is regenerated, so
-	 * must be the vma->pages. A simple rule is that vma->pages must only
-	 * be accessed when the obj->mm.pages are pinned.
-	 */
-	GEM_BUG_ON(!i915_gem_object_has_pinned_pages(vma->obj));
-
-	switch (vma->ggtt_view.type) {
-	default:
-		GEM_BUG_ON(vma->ggtt_view.type);
-		fallthrough;
-	case I915_GGTT_VIEW_NORMAL:
-		vma->pages = vma->obj->mm.pages;
-		return 0;
-
-	case I915_GGTT_VIEW_ROTATED:
-		vma->pages =
-			intel_rotate_pages(&vma->ggtt_view.rotated, vma->obj);
-		break;
-
-	case I915_GGTT_VIEW_REMAPPED:
-		vma->pages =
-			intel_remap_pages(&vma->ggtt_view.remapped, vma->obj);
-		break;
-
-	case I915_GGTT_VIEW_PARTIAL:
-		vma->pages = intel_partial_pages(&vma->ggtt_view, vma->obj);
-		break;
-	}
-
-	ret = 0;
-	if (IS_ERR(vma->pages)) {
-		ret = PTR_ERR(vma->pages);
-		vma->pages = NULL;
-		drm_err(&vma->vm->i915->drm,
-			"Failed to get pages for VMA view type %u (%d)!\n",
-			vma->ggtt_view.type, ret);
-	}
-	return ret;
-}
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.c b/drivers/gpu/drm/i915/gt/intel_gtt.c
index e137dd32b5b8..6367278c220a 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.c
@@ -220,19 +220,6 @@ void i915_address_space_init(struct i915_address_space *vm, int subclass)
 	INIT_LIST_HEAD(&vm->bound_list);
 }
 
-void clear_pages(struct i915_vma *vma)
-{
-	GEM_BUG_ON(!vma->pages);
-
-	if (vma->pages != vma->obj->mm.pages) {
-		sg_free_table(vma->pages);
-		kfree(vma->pages);
-	}
-	vma->pages = NULL;
-
-	memset(&vma->page_sizes, 0, sizeof(vma->page_sizes));
-}
-
 void *__px_vaddr(struct drm_i915_gem_object *p)
 {
 	enum i915_map_type type;
diff --git a/drivers/gpu/drm/i915/gt/intel_gtt.h b/drivers/gpu/drm/i915/gt/intel_gtt.h
index bc7153018ebd..1a97129561dd 100644
--- a/drivers/gpu/drm/i915/gt/intel_gtt.h
+++ b/drivers/gpu/drm/i915/gt/intel_gtt.h
@@ -206,9 +206,6 @@ struct i915_vma_ops {
 	 */
 	void (*unbind_vma)(struct i915_address_space *vm,
 			   struct i915_vma *vma);
-
-	int (*set_pages)(struct i915_vma *vma);
-	void (*clear_pages)(struct i915_vma *vma);
 };
 
 struct i915_address_space {
@@ -589,10 +586,6 @@ release_pd_entry(struct i915_page_directory * const pd,
 		 const struct drm_i915_gem_object * const scratch);
 void gen6_ggtt_invalidate(struct i915_ggtt *ggtt);
 
-int ggtt_set_pages(struct i915_vma *vma);
-int ppgtt_set_pages(struct i915_vma *vma);
-void clear_pages(struct i915_vma *vma);
-
 void ppgtt_bind_vma(struct i915_address_space *vm,
 		    struct i915_vm_pt_stash *stash,
 		    struct i915_vma *vma,
diff --git a/drivers/gpu/drm/i915/gt/intel_ppgtt.c b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
index 886060f7e6fc..4a10a1371d41 100644
--- a/drivers/gpu/drm/i915/gt/intel_ppgtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ppgtt.c
@@ -288,16 +288,6 @@ void i915_vm_free_pt_stash(struct i915_address_space *vm,
 	}
 }
 
-int ppgtt_set_pages(struct i915_vma *vma)
-{
-	GEM_BUG_ON(vma->pages);
-
-	vma->pages = vma->obj->mm.pages;
-	vma->page_sizes = vma->obj->mm.page_sizes;
-
-	return 0;
-}
-
 void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
 {
 	struct drm_i915_private *i915 = gt->i915;
@@ -312,6 +302,4 @@ void ppgtt_init(struct i915_ppgtt *ppgtt, struct intel_gt *gt)
 
 	ppgtt->vm.vma_ops.bind_vma    = ppgtt_bind_vma;
 	ppgtt->vm.vma_ops.unbind_vma  = ppgtt_unbind_vma;
-	ppgtt->vm.vma_ops.set_pages   = ppgtt_set_pages;
-	ppgtt->vm.vma_ops.clear_pages = clear_pages;
 }
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index 36db641e863e..5e0eaa9ea890 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -112,7 +112,6 @@ vma_create(struct drm_i915_gem_object *obj,
 		return ERR_PTR(-ENOMEM);
 
 	kref_init(&vma->ref);
-	mutex_init(&vma->pages_mutex);
 	vma->vm = i915_vm_get(vm);
 	vma->ops = &vm->vma_ops;
 	vma->obj = obj;
@@ -789,10 +788,317 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
 	return pinned;
 }
 
-static int vma_get_pages(struct i915_vma *vma)
+
+static struct scatterlist *
+rotate_pages(struct drm_i915_gem_object *obj, unsigned int offset,
+	     unsigned int width, unsigned int height,
+	     unsigned int src_stride, unsigned int dst_stride,
+	     struct sg_table *st, struct scatterlist *sg)
 {
-	int err = 0;
-	bool pinned_pages = true;
+	unsigned int column, row;
+	unsigned int src_idx;
+
+	for (column = 0; column < width; column++) {
+		unsigned int left;
+
+		src_idx = src_stride * (height - 1) + column + offset;
+		for (row = 0; row < height; row++) {
+			st->nents++;
+			/*
+			 * We don't need the pages, but need to initialize
+			 * the entries so the sg list can be happily traversed.
+			 * The only thing we need are DMA addresses.
+			 */
+			sg_set_page(sg, NULL, I915_GTT_PAGE_SIZE, 0);
+			sg_dma_address(sg) =
+				i915_gem_object_get_dma_address(obj, src_idx);
+			sg_dma_len(sg) = I915_GTT_PAGE_SIZE;
+			sg = sg_next(sg);
+			src_idx -= src_stride;
+		}
+
+		left = (dst_stride - height) * I915_GTT_PAGE_SIZE;
+
+		if (!left)
+			continue;
+
+		st->nents++;
+
+		/*
+		 * The DE ignores the PTEs for the padding tiles, the sg entry
+		 * here is just a conenience to indicate how many padding PTEs
+		 * to insert at this spot.
+		 */
+		sg_set_page(sg, NULL, left, 0);
+		sg_dma_address(sg) = 0;
+		sg_dma_len(sg) = left;
+		sg = sg_next(sg);
+	}
+
+	return sg;
+}
+
+static noinline struct sg_table *
+intel_rotate_pages(struct intel_rotation_info *rot_info,
+		   struct drm_i915_gem_object *obj)
+{
+	unsigned int size = intel_rotation_info_size(rot_info);
+	struct drm_i915_private *i915 = to_i915(obj->base.dev);
+	struct sg_table *st;
+	struct scatterlist *sg;
+	int ret = -ENOMEM;
+	int i;
+
+	/* Allocate target SG list. */
+	st = kmalloc(sizeof(*st), GFP_KERNEL);
+	if (!st)
+		goto err_st_alloc;
+
+	ret = sg_alloc_table(st, size, GFP_KERNEL);
+	if (ret)
+		goto err_sg_alloc;
+
+	st->nents = 0;
+	sg = st->sgl;
+
+	for (i = 0 ; i < ARRAY_SIZE(rot_info->plane); i++)
+		sg = rotate_pages(obj, rot_info->plane[i].offset,
+				  rot_info->plane[i].width, rot_info->plane[i].height,
+				  rot_info->plane[i].src_stride,
+				  rot_info->plane[i].dst_stride,
+				  st, sg);
+
+	return st;
+
+err_sg_alloc:
+	kfree(st);
+err_st_alloc:
+
+	drm_dbg(&i915->drm, "Failed to create rotated mapping for object size %zu! (%ux%u tiles, %u pages)\n",
+		obj->base.size, rot_info->plane[0].width,
+		rot_info->plane[0].height, size);
+
+	return ERR_PTR(ret);
+}
+
+static struct scatterlist *
+remap_pages(struct drm_i915_gem_object *obj, unsigned int offset,
+	    unsigned int width, unsigned int height,
+	    unsigned int src_stride, unsigned int dst_stride,
+	    struct sg_table *st, struct scatterlist *sg)
+{
+	unsigned int row;
+
+	for (row = 0; row < height; row++) {
+		unsigned int left = width * I915_GTT_PAGE_SIZE;
+
+		while (left) {
+			dma_addr_t addr;
+			unsigned int length;
+
+			/*
+			 * We don't need the pages, but need to initialize
+			 * the entries so the sg list can be happily traversed.
+			 * The only thing we need are DMA addresses.
+			 */
+
+			addr = i915_gem_object_get_dma_address_len(obj, offset, &length);
+
+			length = min(left, length);
+
+			st->nents++;
+
+			sg_set_page(sg, NULL, length, 0);
+			sg_dma_address(sg) = addr;
+			sg_dma_len(sg) = length;
+			sg = sg_next(sg);
+
+			offset += length / I915_GTT_PAGE_SIZE;
+			left -= length;
+		}
+
+		offset += src_stride - width;
+
+		left = (dst_stride - width) * I915_GTT_PAGE_SIZE;
+
+		if (!left)
+			continue;
+
+		st->nents++;
+
+		/*
+		 * The DE ignores the PTEs for the padding tiles, the sg entry
+		 * here is just a conenience to indicate how many padding PTEs
+		 * to insert at this spot.
+		 */
+		sg_set_page(sg, NULL, left, 0);
+		sg_dma_address(sg) = 0;
+		sg_dma_len(sg) = left;
+		sg = sg_next(sg);
+	}
+
+	return sg;
+}
+
+static noinline struct sg_table *
+intel_remap_pages(struct intel_remapped_info *rem_info,
+		  struct drm_i915_gem_object *obj)
+{
+	unsigned int size = intel_remapped_info_size(rem_info);
+	struct drm_i915_private *i915 = to_i915(obj->base.dev);
+	struct sg_table *st;
+	struct scatterlist *sg;
+	int ret = -ENOMEM;
+	int i;
+
+	/* Allocate target SG list. */
+	st = kmalloc(sizeof(*st), GFP_KERNEL);
+	if (!st)
+		goto err_st_alloc;
+
+	ret = sg_alloc_table(st, size, GFP_KERNEL);
+	if (ret)
+		goto err_sg_alloc;
+
+	st->nents = 0;
+	sg = st->sgl;
+
+	for (i = 0 ; i < ARRAY_SIZE(rem_info->plane); i++) {
+		sg = remap_pages(obj, rem_info->plane[i].offset,
+				 rem_info->plane[i].width, rem_info->plane[i].height,
+				 rem_info->plane[i].src_stride, rem_info->plane[i].dst_stride,
+				 st, sg);
+	}
+
+	i915_sg_trim(st);
+
+	return st;
+
+err_sg_alloc:
+	kfree(st);
+err_st_alloc:
+
+	drm_dbg(&i915->drm, "Failed to create remapped mapping for object size %zu! (%ux%u tiles, %u pages)\n",
+		obj->base.size, rem_info->plane[0].width,
+		rem_info->plane[0].height, size);
+
+	return ERR_PTR(ret);
+}
+
+static noinline struct sg_table *
+intel_partial_pages(const struct i915_ggtt_view *view,
+		    struct drm_i915_gem_object *obj)
+{
+	struct sg_table *st;
+	struct scatterlist *sg, *iter;
+	unsigned int count = view->partial.size;
+	unsigned int offset;
+	int ret = -ENOMEM;
+
+	st = kmalloc(sizeof(*st), GFP_KERNEL);
+	if (!st)
+		goto err_st_alloc;
+
+	ret = sg_alloc_table(st, count, GFP_KERNEL);
+	if (ret)
+		goto err_sg_alloc;
+
+	iter = i915_gem_object_get_sg_dma(obj, view->partial.offset, &offset);
+	GEM_BUG_ON(!iter);
+
+	sg = st->sgl;
+	st->nents = 0;
+	do {
+		unsigned int len;
+
+		len = min(sg_dma_len(iter) - (offset << PAGE_SHIFT),
+			  count << PAGE_SHIFT);
+		sg_set_page(sg, NULL, len, 0);
+		sg_dma_address(sg) =
+			sg_dma_address(iter) + (offset << PAGE_SHIFT);
+		sg_dma_len(sg) = len;
+
+		st->nents++;
+		count -= len >> PAGE_SHIFT;
+		if (count == 0) {
+			sg_mark_end(sg);
+			i915_sg_trim(st); /* Drop any unused tail entries. */
+
+			return st;
+		}
+
+		sg = __sg_next(sg);
+		iter = __sg_next(iter);
+		offset = 0;
+	} while (1);
+
+err_sg_alloc:
+	kfree(st);
+err_st_alloc:
+	return ERR_PTR(ret);
+}
+
+static int
+__i915_vma_get_pages(struct i915_vma *vma)
+{
+	struct sg_table *pages;
+	int ret;
+
+	/*
+	 * The vma->pages are only valid within the lifespan of the borrowed
+	 * obj->mm.pages. When the obj->mm.pages sg_table is regenerated, so
+	 * must be the vma->pages. A simple rule is that vma->pages must only
+	 * be accessed when the obj->mm.pages are pinned.
+	 */
+	GEM_BUG_ON(!i915_gem_object_has_pinned_pages(vma->obj));
+
+	switch (vma->ggtt_view.type) {
+	default:
+		GEM_BUG_ON(vma->ggtt_view.type);
+		fallthrough;
+	case I915_GGTT_VIEW_NORMAL:
+		pages = vma->obj->mm.pages;
+		break;
+
+	case I915_GGTT_VIEW_ROTATED:
+		pages =
+			intel_rotate_pages(&vma->ggtt_view.rotated, vma->obj);
+		break;
+
+	case I915_GGTT_VIEW_REMAPPED:
+		pages =
+			intel_remap_pages(&vma->ggtt_view.remapped, vma->obj);
+		break;
+
+	case I915_GGTT_VIEW_PARTIAL:
+		pages = intel_partial_pages(&vma->ggtt_view, vma->obj);
+		break;
+	}
+
+	ret = 0;
+	/* gen6 ppgtt doesn't have backing pages, special-case it */
+	if (IS_ERR(pages) && pages != ERR_PTR(-ENODEV)) {
+		ret = PTR_ERR(pages);
+		pages = NULL;
+		drm_err(&vma->vm->i915->drm,
+			"Failed to get pages for VMA view type %u (%d)!\n",
+			vma->ggtt_view.type, ret);
+	}
+
+	pages = xchg(&vma->pages, pages);
+
+	/* did we race against a put_pages? */
+	if (pages && pages != vma->obj->mm.pages) {
+		sg_free_table(vma->pages);
+		kfree(vma->pages);
+	}
+
+	return ret;
+}
+
+I915_SELFTEST_EXPORT int i915_vma_get_pages(struct i915_vma *vma)
+{
+	int err;
 
 	if (atomic_add_unless(&vma->pages_count, 1, 0))
 		return 0;
@@ -801,25 +1107,17 @@ static int vma_get_pages(struct i915_vma *vma)
 	if (err)
 		return err;
 
-	/* Allocations ahoy! */
-	if (mutex_lock_interruptible(&vma->pages_mutex)) {
-		err = -EINTR;
-		goto unpin;
-	}
+	err = __i915_vma_get_pages(vma);
+	if (err)
+		goto err_unpin;
 
-	if (!atomic_read(&vma->pages_count)) {
-		err = vma->ops->set_pages(vma);
-		if (err)
-			goto unlock;
-		pinned_pages = false;
-	}
+	vma->page_sizes = vma->obj->mm.page_sizes;
 	atomic_inc(&vma->pages_count);
 
-unlock:
-	mutex_unlock(&vma->pages_mutex);
-unpin:
-	if (pinned_pages)
-		__i915_gem_object_unpin_pages(vma->obj);
+	return 0;
+
+err_unpin:
+	__i915_gem_object_unpin_pages(vma->obj);
 
 	return err;
 }
@@ -827,18 +1125,22 @@ static int vma_get_pages(struct i915_vma *vma)
 static void __vma_put_pages(struct i915_vma *vma, unsigned int count)
 {
 	/* We allocate under vma_get_pages, so beware the shrinker */
-	mutex_lock_nested(&vma->pages_mutex, SINGLE_DEPTH_NESTING);
+	struct sg_table *pages = READ_ONCE(vma->pages);
+
 	GEM_BUG_ON(atomic_read(&vma->pages_count) < count);
+
 	if (atomic_sub_return(count, &vma->pages_count) == 0) {
-		vma->ops->clear_pages(vma);
-		GEM_BUG_ON(vma->pages);
+		if (pages == cmpxchg(&vma->pages, pages, NULL) &&
+		    pages != vma->obj->mm.pages) {
+			sg_free_table(pages);
+			kfree(pages);
+		}
 
 		i915_gem_object_unpin_pages(vma->obj);
 	}
-	mutex_unlock(&vma->pages_mutex);
 }
 
-static void vma_put_pages(struct i915_vma *vma)
+I915_SELFTEST_EXPORT void i915_vma_put_pages(struct i915_vma *vma)
 {
 	if (atomic_add_unless(&vma->pages_count, -1, 1))
 		return;
@@ -868,10 +1170,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	unsigned int bound;
 	int err;
 
-#ifdef CONFIG_PROVE_LOCKING
-	if (debug_locks && !WARN_ON(!ww))
-		assert_vma_held(vma);
-#endif
+	assert_vma_held(vma);
+	GEM_BUG_ON(!ww);
 
 	BUILD_BUG_ON(PIN_GLOBAL != I915_VMA_GLOBAL_BIND);
 	BUILD_BUG_ON(PIN_USER != I915_VMA_LOCAL_BIND);
@@ -882,7 +1182,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	if (try_qad_pin(vma, flags & I915_VMA_BIND_MASK))
 		return 0;
 
-	err = vma_get_pages(vma);
+	err = i915_vma_get_pages(vma);
 	if (err)
 		return err;
 
@@ -1007,7 +1307,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 err_rpm:
 	if (wakeref)
 		intel_runtime_pm_put(&vma->vm->i915->runtime_pm, wakeref);
-	vma_put_pages(vma);
+	i915_vma_put_pages(vma);
 	return err;
 }
 
diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
index 0b11c188f749..e4dc5187c8ad 100644
--- a/drivers/gpu/drm/i915/i915_vma.h
+++ b/drivers/gpu/drm/i915/i915_vma.h
@@ -424,4 +424,7 @@ static inline int i915_vma_sync(struct i915_vma *vma)
 void i915_vma_module_exit(void);
 int i915_vma_module_init(void);
 
+I915_SELFTEST_DECLARE(int i915_vma_get_pages(struct i915_vma *vma));
+I915_SELFTEST_DECLARE(void i915_vma_put_pages(struct i915_vma *vma));
+
 #endif
diff --git a/drivers/gpu/drm/i915/i915_vma_types.h b/drivers/gpu/drm/i915/i915_vma_types.h
index 47573ed2d11f..a129ab9b6405 100644
--- a/drivers/gpu/drm/i915/i915_vma_types.h
+++ b/drivers/gpu/drm/i915/i915_vma_types.h
@@ -260,7 +260,6 @@ struct i915_vma {
 #define I915_VMA_PAGES_BIAS 24
 #define I915_VMA_PAGES_ACTIVE (BIT(24) | 1)
 	atomic_t pages_count; /* number of active binds to the pages */
-	struct mutex pages_mutex; /* protect acquire/release of backing pages */
 
 	/**
 	 * Support different GGTT views into the same object.
diff --git a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
index f843a5040706..a3e3d97728e7 100644
--- a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
@@ -1953,7 +1953,9 @@ static int igt_cs_tlb(void *arg)
 				goto end;
 			}
 
-			err = vma->ops->set_pages(vma);
+			i915_gem_object_lock(bbe, NULL);
+			err = i915_vma_get_pages(vma);
+			i915_gem_object_unlock(bbe);
 			if (err)
 				goto end;
 
@@ -1994,7 +1996,7 @@ static int igt_cs_tlb(void *arg)
 				i915_request_put(rq);
 			}
 
-			vma->ops->clear_pages(vma);
+			i915_vma_put_pages(vma);
 
 			err = context_sync(ce);
 			if (err) {
@@ -2009,7 +2011,9 @@ static int igt_cs_tlb(void *arg)
 				goto end;
 			}
 
-			err = vma->ops->set_pages(vma);
+			i915_gem_object_lock(act, NULL);
+			err = i915_vma_get_pages(vma);
+			i915_gem_object_unlock(act);
 			if (err)
 				goto end;
 
@@ -2047,7 +2051,7 @@ static int igt_cs_tlb(void *arg)
 			}
 			end_spin(batch, count - 1);
 
-			vma->ops->clear_pages(vma);
+			i915_vma_put_pages(vma);
 
 			err = context_sync(ce);
 			if (err) {
diff --git a/drivers/gpu/drm/i915/selftests/mock_gtt.c b/drivers/gpu/drm/i915/selftests/mock_gtt.c
index cc047ec594f9..096679014d99 100644
--- a/drivers/gpu/drm/i915/selftests/mock_gtt.c
+++ b/drivers/gpu/drm/i915/selftests/mock_gtt.c
@@ -86,8 +86,6 @@ struct i915_ppgtt *mock_ppgtt(struct drm_i915_private *i915, const char *name)
 
 	ppgtt->vm.vma_ops.bind_vma    = mock_bind_ppgtt;
 	ppgtt->vm.vma_ops.unbind_vma  = mock_unbind_ppgtt;
-	ppgtt->vm.vma_ops.set_pages   = ppgtt_set_pages;
-	ppgtt->vm.vma_ops.clear_pages = clear_pages;
 
 	return ppgtt;
 }
@@ -126,8 +124,6 @@ void mock_init_ggtt(struct drm_i915_private *i915, struct i915_ggtt *ggtt)
 
 	ggtt->vm.vma_ops.bind_vma    = mock_bind_ggtt;
 	ggtt->vm.vma_ops.unbind_vma  = mock_unbind_ggtt;
-	ggtt->vm.vma_ops.set_pages   = ggtt_set_pages;
-	ggtt->vm.vma_ops.clear_pages = clear_pages;
 
 	i915_address_space_init(&ggtt->vm, VM_CLASS_GGTT);
 	i915->gt.ggtt = ggtt;
-- 
2.32.0

[Intel-gfx] [PATCH 13/19] drm/i915: Take object lock in i915_ggtt_pin if ww is not set

[Maarten Lankhorst]

i915_vma_wait_for_bind needs the vma lock held, fix the caller.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/i915_vma.c | 40 +++++++++++++++++++++++----------
 1 file changed, 28 insertions(+), 12 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index 5e0eaa9ea890..77319619dba0 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -1322,23 +1322,15 @@ static void flush_idle_contexts(struct intel_gt *gt)
 	intel_gt_wait_for_idle(gt, MAX_SCHEDULE_TIMEOUT);
 }
 
-int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
-		  u32 align, unsigned int flags)
+static int __i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
+			   u32 align, unsigned int flags)
 {
 	struct i915_address_space *vm = vma->vm;
 	int err;
 
-	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
-
-#ifdef CONFIG_LOCKDEP
-	WARN_ON(!ww && dma_resv_held(vma->obj->base.resv));
-#endif
-
 	do {
-		if (ww)
-			err = i915_vma_pin_ww(vma, ww, 0, align, flags | PIN_GLOBAL);
-		else
-			err = i915_vma_pin(vma, 0, align, flags | PIN_GLOBAL);
+		err = i915_vma_pin_ww(vma, ww, 0, align, flags | PIN_GLOBAL);
+
 		if (err != -ENOSPC) {
 			if (!err) {
 				err = i915_vma_wait_for_bind(vma);
@@ -1357,6 +1349,30 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	} while (1);
 }
 
+int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
+		  u32 align, unsigned int flags)
+{
+	struct i915_gem_ww_ctx _ww;
+	int err;
+
+	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
+
+	if (ww)
+		return __i915_ggtt_pin(vma, ww, align, flags);
+
+#ifdef CONFIG_LOCKDEP
+	WARN_ON(dma_resv_held(vma->obj->base.resv));
+#endif
+
+	for_i915_gem_ww(&_ww, err, true) {
+		err = i915_gem_object_lock(vma->obj, &_ww);
+		if (!err)
+			err = __i915_ggtt_pin(vma, &_ww, align, flags);
+	}
+
+	return err;
+}
+
 static void __vma_close(struct i915_vma *vma, struct intel_gt *gt)
 {
 	/*
-- 
2.32.0

[Intel-gfx] [PATCH 14/19] drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for i915_vma_unbind

[Maarten Lankhorst]

We want to remove more members of i915_vma, which requires the locking to be
held more often.

Start requiring gem object lock for i915_vma_unbind, as it's one of the
callers that may unpin pages.

Some special care is needed when evicting, because the last reference to the
object may be held by the VMA, so after __i915_vma_unbind, vma may be garbage,
and we need to cache vma->obj before unlocking.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/display/intel_display.c  |  2 +-
 drivers/gpu/drm/i915/gem/i915_gem_shrinker.c  | 14 +++-
 .../gpu/drm/i915/gem/selftests/huge_pages.c   |  4 +-
 .../i915/gem/selftests/i915_gem_client_blt.c  |  2 +-
 .../drm/i915/gem/selftests/i915_gem_mman.c    |  6 ++
 drivers/gpu/drm/i915/gt/intel_ggtt.c          | 46 ++++++++++--
 drivers/gpu/drm/i915/i915_drv.h               |  7 +-
 drivers/gpu/drm/i915/i915_gem.c               | 29 +++++++-
 drivers/gpu/drm/i915/i915_gem_evict.c         | 74 +++++++++++++++++--
 drivers/gpu/drm/i915/i915_vma.c               | 27 ++++++-
 drivers/gpu/drm/i915/i915_vma.h               |  1 +
 drivers/gpu/drm/i915/selftests/i915_gem_gtt.c | 22 +++---
 drivers/gpu/drm/i915/selftests/i915_vma.c     |  2 +-
 13 files changed, 195 insertions(+), 41 deletions(-)

diff --git a/drivers/gpu/drm/i915/display/intel_display.c b/drivers/gpu/drm/i915/display/intel_display.c
index 5560d2f4c352..b46ea14ac10f 100644
--- a/drivers/gpu/drm/i915/display/intel_display.c
+++ b/drivers/gpu/drm/i915/display/intel_display.c
@@ -940,7 +940,7 @@ intel_pin_fb_obj_dpt(struct drm_framebuffer *fb,
 		goto err;
 
 	if (i915_vma_misplaced(vma, 0, alignment, 0)) {
-		ret = i915_vma_unbind(vma);
+		ret = i915_vma_unbind_unlocked(vma);
 		if (ret) {
 			vma = ERR_PTR(ret);
 			goto err;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
index 7f7849b6296d..71e855fcbd8c 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
@@ -404,12 +404,22 @@ i915_gem_shrinker_vmap(struct notifier_block *nb, unsigned long event, void *ptr
 	list_for_each_entry_safe(vma, next,
 				 &i915->ggtt.vm.bound_list, vm_link) {
 		unsigned long count = vma->node.size >> PAGE_SHIFT;
+		struct drm_i915_gem_object *obj;
 
 		if (!vma->iomap || i915_vma_is_active(vma))
 			continue;
 
-		if (__i915_vma_unbind(vma) == 0)
-			freed_pages += count;
+		obj = vma->obj;
+		if (!kref_get_unless_zero(&obj->base.refcount))
+			continue;
+
+		if (i915_gem_object_trylock(obj)) {
+			if (__i915_vma_unbind(vma) == 0)
+				freed_pages += count;
+
+			i915_gem_object_unlock(obj);
+		}
+		i915_gem_object_put(obj);
 	}
 	mutex_unlock(&i915->ggtt.vm.mutex);
 
diff --git a/drivers/gpu/drm/i915/gem/selftests/huge_pages.c b/drivers/gpu/drm/i915/gem/selftests/huge_pages.c
index a094f3ce1a90..a43f55c52252 100644
--- a/drivers/gpu/drm/i915/gem/selftests/huge_pages.c
+++ b/drivers/gpu/drm/i915/gem/selftests/huge_pages.c
@@ -626,7 +626,7 @@ static int igt_mock_ppgtt_misaligned_dma(void *arg)
 		 * pages.
 		 */
 		for (offset = 4096; offset < page_size; offset += 4096) {
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			if (err)
 				goto out_unpin;
 
@@ -1075,7 +1075,7 @@ static int __igt_write_huge(struct intel_context *ce,
 	if (IS_ERR(vma))
 		return PTR_ERR(vma);
 
-	err = i915_vma_unbind(vma);
+	err = i915_vma_unbind_unlocked(vma);
 	if (err)
 		return err;
 
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c
index ecbcbb86ae1e..d132933c5054 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_client_blt.c
@@ -311,7 +311,7 @@ static int pin_buffer(struct i915_vma *vma, u64 addr)
 	int err;
 
 	if (drm_mm_node_allocated(&vma->node) && vma->node.start != addr) {
-		err = i915_vma_unbind(vma);
+		err = i915_vma_unbind_unlocked(vma);
 		if (err)
 			return err;
 	}
diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
index b20f5621f62b..4c51c30b12e6 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
@@ -165,7 +165,9 @@ static int check_partial_mapping(struct drm_i915_gem_object *obj,
 	kunmap(p);
 
 out:
+	i915_gem_object_lock(obj, NULL);
 	__i915_vma_put(vma);
+	i915_gem_object_unlock(obj);
 	return err;
 }
 
@@ -259,7 +261,9 @@ static int check_partial_mappings(struct drm_i915_gem_object *obj,
 		if (err)
 			return err;
 
+		i915_gem_object_lock(obj, NULL);
 		__i915_vma_put(vma);
+		i915_gem_object_unlock(obj);
 
 		if (igt_timeout(end_time,
 				"%s: timed out after tiling=%d stride=%d\n",
@@ -1333,7 +1337,9 @@ static int __igt_mmap_revoke(struct drm_i915_private *i915,
 	 * for other objects. Ergo we have to revoke the previous mmap PTE
 	 * access as it no longer points to the same object.
 	 */
+	i915_gem_object_lock(obj, NULL);
 	err = i915_gem_object_unbind(obj, I915_GEM_OBJECT_UNBIND_ACTIVE);
+	i915_gem_object_unlock(obj);
 	if (err) {
 		pr_err("Failed to unbind object!\n");
 		goto out_unmap;
diff --git a/drivers/gpu/drm/i915/gt/intel_ggtt.c b/drivers/gpu/drm/i915/gt/intel_ggtt.c
index 8c46a44e5a40..7bf15293a471 100644
--- a/drivers/gpu/drm/i915/gt/intel_ggtt.c
+++ b/drivers/gpu/drm/i915/gt/intel_ggtt.c
@@ -118,22 +118,44 @@ void i915_ggtt_suspend(struct i915_ggtt *ggtt)
 	struct i915_vma *vma, *vn;
 	int open;
 
+retry:
 	mutex_lock(&ggtt->vm.mutex);
 
 	/* Skip rewriting PTE on VMA unbind. */
 	open = atomic_xchg(&ggtt->vm.open, 0);
 
 	list_for_each_entry_safe(vma, vn, &ggtt->vm.bound_list, vm_link) {
+		struct drm_i915_gem_object *obj = vma->obj;
+
 		GEM_BUG_ON(!drm_mm_node_allocated(&vma->node));
-		i915_vma_wait_for_bind(vma);
 
-		if (i915_vma_is_pinned(vma))
+		if (i915_vma_is_pinned(vma) || !i915_vma_is_bound(vma, I915_VMA_GLOBAL_BIND))
 			continue;
 
-		if (!i915_vma_is_bound(vma, I915_VMA_GLOBAL_BIND)) {
-			__i915_vma_evict(vma);
-			drm_mm_remove_node(&vma->node);
+		i915_gem_object_get(obj);
+
+		/* unlikely to race when GPU is idle, so no worry about slowpath.. */
+		if (!i915_gem_object_trylock(obj)) {
+			atomic_set(&ggtt->vm.open, open);
+			mutex_unlock(&ggtt->vm.mutex);
+
+			i915_gem_object_lock(obj, NULL);
+			open = i915_vma_unbind(vma);
+			i915_gem_object_unlock(obj);
+
+			GEM_WARN_ON(open);
+
+			i915_gem_object_put(obj);
+			goto retry;
 		}
+
+		i915_vma_wait_for_bind(vma);
+
+		__i915_vma_evict(vma);
+		drm_mm_remove_node(&vma->node);
+
+		i915_gem_object_unlock(obj);
+		i915_gem_object_put(obj);
 	}
 
 	ggtt->vm.clear_range(&ggtt->vm, 0, ggtt->vm.total);
@@ -729,9 +751,21 @@ static void ggtt_cleanup_hw(struct i915_ggtt *ggtt)
 
 	mutex_lock(&ggtt->vm.mutex);
 
-	list_for_each_entry_safe(vma, vn, &ggtt->vm.bound_list, vm_link)
+	list_for_each_entry_safe(vma, vn, &ggtt->vm.bound_list, vm_link) {
+		struct drm_i915_gem_object *obj = vma->obj;
+		bool trylock;
+
+		i915_gem_object_get(obj);
+		trylock = i915_gem_object_trylock(obj);
+
+		WARN_ON(!trylock);
 		WARN_ON(__i915_vma_unbind(vma));
 
+		if (trylock)
+			i915_gem_object_unlock(obj);
+		i915_gem_object_put(obj);
+	}
+
 	if (drm_mm_node_allocated(&ggtt->error_capture))
 		drm_mm_remove_node(&ggtt->error_capture);
 	mutex_destroy(&ggtt->error_mutex);
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index ac4cea32601f..39ffa765eb61 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -1818,13 +1818,10 @@ i915_gem_object_ggtt_pin_ww(struct drm_i915_gem_object *obj,
 			    const struct i915_ggtt_view *view,
 			    u64 size, u64 alignment, u64 flags);
 
-static inline struct i915_vma * __must_check
+struct i915_vma * __must_check
 i915_gem_object_ggtt_pin(struct drm_i915_gem_object *obj,
 			 const struct i915_ggtt_view *view,
-			 u64 size, u64 alignment, u64 flags)
-{
-	return i915_gem_object_ggtt_pin_ww(obj, NULL, view, size, alignment, flags);
-}
+			 u64 size, u64 alignment, u64 flags);
 
 int i915_gem_object_unbind(struct drm_i915_gem_object *obj,
 			   unsigned long flags);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 590efc8b0265..3727e77f651f 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -118,6 +118,8 @@ int i915_gem_object_unbind(struct drm_i915_gem_object *obj,
 	struct i915_vma *vma;
 	int ret;
 
+	assert_object_held(obj);
+
 	if (list_empty(&obj->vma.list))
 		return 0;
 
@@ -936,10 +938,7 @@ i915_gem_object_ggtt_pin_ww(struct drm_i915_gem_object *obj,
 			return ERR_PTR(ret);
 	}
 
-	if (ww)
-		ret = i915_vma_pin_ww(vma, ww, size, alignment, flags | PIN_GLOBAL);
-	else
-		ret = i915_vma_pin(vma, size, alignment, flags | PIN_GLOBAL);
+	ret = i915_vma_pin_ww(vma, ww, size, alignment, flags | PIN_GLOBAL);
 
 	if (ret)
 		return ERR_PTR(ret);
@@ -959,6 +958,28 @@ i915_gem_object_ggtt_pin_ww(struct drm_i915_gem_object *obj,
 	return vma;
 }
 
+struct i915_vma *
+i915_gem_object_ggtt_pin(struct drm_i915_gem_object *obj,
+			 const struct i915_ggtt_view *view,
+			 u64 size, u64 alignment, u64 flags)
+{
+	struct i915_vma *vma = ERR_PTR(-ENODEV);
+	struct i915_gem_ww_ctx ww;
+	int err;
+
+	for_i915_gem_ww(&ww, err, true) {
+		err = i915_gem_object_lock(obj, &ww);
+		if (!err) {
+			vma = i915_gem_object_ggtt_pin_ww(obj, &ww, view, size,
+							  alignment, flags);
+			if (IS_ERR(vma))
+				err = PTR_ERR(vma);
+		}
+	}
+
+	return err ? ERR_PTR(err) : vma;
+}
+
 int
 i915_gem_madvise_ioctl(struct drm_device *dev, void *data,
 		       struct drm_file *file_priv)
diff --git a/drivers/gpu/drm/i915/i915_gem_evict.c b/drivers/gpu/drm/i915/i915_gem_evict.c
index 2b73ddb11c66..872c631e59b9 100644
--- a/drivers/gpu/drm/i915/i915_gem_evict.c
+++ b/drivers/gpu/drm/i915/i915_gem_evict.c
@@ -58,6 +58,14 @@ mark_free(struct drm_mm_scan *scan,
 	if (i915_vma_is_pinned(vma))
 		return false;
 
+	if (!kref_get_unless_zero(&vma->obj->base.refcount))
+		return false;
+
+	if (!i915_gem_object_trylock(vma->obj)) {
+		i915_vma_put(vma);
+		return false;
+	}
+
 	list_add(&vma->evict_link, unwind);
 	return drm_mm_scan_add_block(scan, &vma->node);
 }
@@ -178,6 +186,8 @@ i915_gem_evict_something(struct i915_address_space *vm,
 	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
 		ret = drm_mm_scan_remove_block(&scan, &vma->node);
 		BUG_ON(ret);
+		i915_gem_object_unlock(vma->obj);
+		i915_gem_object_put(vma->obj);
 	}
 
 	/*
@@ -222,10 +232,13 @@ i915_gem_evict_something(struct i915_address_space *vm,
 	 * of any of our objects, thus corrupting the list).
 	 */
 	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
-		if (drm_mm_scan_remove_block(&scan, &vma->node))
+		if (drm_mm_scan_remove_block(&scan, &vma->node)) {
 			__i915_vma_pin(vma);
-		else
+		} else {
 			list_del(&vma->evict_link);
+			i915_gem_object_unlock(vma->obj);
+			i915_gem_object_put(vma->obj);
+		}
 	}
 
 	/* Unbinding will emit any required flushes */
@@ -234,16 +247,28 @@ i915_gem_evict_something(struct i915_address_space *vm,
 		__i915_vma_unpin(vma);
 		if (ret == 0)
 			ret = __i915_vma_unbind(vma);
+
+		i915_gem_object_unlock(vma->obj);
+		i915_gem_object_put(vma->obj);
 	}
 
 	while (ret == 0 && (node = drm_mm_scan_color_evict(&scan))) {
 		vma = container_of(node, struct i915_vma, node);
 
+
 		/* If we find any non-objects (!vma), we cannot evict them */
-		if (vma->node.color != I915_COLOR_UNEVICTABLE)
-			ret = __i915_vma_unbind(vma);
-		else
-			ret = -ENOSPC; /* XXX search failed, try again? */
+		ret = -ENOSPC;
+
+		if (vma->node.color != I915_COLOR_UNEVICTABLE &&
+		    !kref_get_unless_zero(&vma->obj->base.refcount)) {
+			struct drm_i915_gem_object *obj = vma->obj;
+
+			if (i915_gem_object_trylock(obj)) {
+				ret = __i915_vma_unbind(vma);
+				i915_gem_object_unlock(obj);
+			}
+			i915_gem_object_put(obj);
+		}
 	}
 
 	return ret;
@@ -333,6 +358,17 @@ int i915_gem_evict_for_node(struct i915_address_space *vm,
 			break;
 		}
 
+		if (!kref_get_unless_zero(&vma->obj->base.refcount)) {
+			ret = -ENOSPC;
+			break;
+		}
+
+		if (!i915_gem_object_trylock(vma->obj)) {
+			ret = -ENOSPC;
+			i915_gem_object_put(vma->obj);
+			break;
+		}
+
 		/*
 		 * Never show fear in the face of dragons!
 		 *
@@ -347,9 +383,14 @@ int i915_gem_evict_for_node(struct i915_address_space *vm,
 	}
 
 	list_for_each_entry_safe(vma, next, &eviction_list, evict_link) {
+		struct drm_i915_gem_object *obj = vma->obj;
+
 		__i915_vma_unpin(vma);
 		if (ret == 0)
 			ret = __i915_vma_unbind(vma);
+
+		i915_gem_object_unlock(obj);
+		i915_gem_object_put(obj);
 	}
 
 	return ret;
@@ -388,6 +429,7 @@ int i915_gem_evict_vm(struct i915_address_space *vm)
 	do {
 		struct i915_vma *vma, *vn;
 		LIST_HEAD(eviction_list);
+		bool evicted = false;
 
 		list_for_each_entry(vma, &vm->bound_list, vm_link) {
 			if (i915_vma_is_pinned(vma))
@@ -401,12 +443,32 @@ int i915_gem_evict_vm(struct i915_address_space *vm)
 
 		ret = 0;
 		list_for_each_entry_safe(vma, vn, &eviction_list, evict_link) {
+			struct drm_i915_gem_object *obj = vma->obj;
+
 			__i915_vma_unpin(vma);
+
+			if (!kref_get_unless_zero(&vma->obj->base.refcount))
+				continue;
+
+			if (!i915_gem_object_trylock(vma->obj)) {
+				i915_gem_object_put(vma->obj);
+				continue;
+			}
+
 			if (ret == 0)
 				ret = __i915_vma_unbind(vma);
 			if (ret != -EINTR) /* "Get me out of here!" */
 				ret = 0;
+
+			i915_gem_object_unlock(obj);
+			i915_gem_object_put(obj);
+
+			evicted = true;
 		}
+
+		/* Nothing evicted because objects were dead or locked? */
+		if (!evicted)
+			break;
 	} while (ret == 0);
 
 	return ret;
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index 77319619dba0..da54e6882650 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -1479,8 +1479,16 @@ void i915_vma_parked(struct intel_gt *gt)
 		struct drm_i915_gem_object *obj = vma->obj;
 		struct i915_address_space *vm = vma->vm;
 
-		INIT_LIST_HEAD(&vma->closed_link);
-		__i915_vma_put(vma);
+		if (i915_gem_object_trylock(obj)) {
+			INIT_LIST_HEAD(&vma->closed_link);
+			__i915_vma_put(vma);
+			i915_gem_object_unlock(obj);
+		} else {
+			/* back you go.. */
+			spin_lock_irq(&gt->closed_lock);
+			list_move(&vma->closed_link, &gt->closed_vma);
+			spin_unlock_irq(&gt->closed_lock);
+		}
 
 		i915_gem_object_put(obj);
 		i915_vm_close(vm);
@@ -1591,6 +1599,7 @@ int i915_vma_move_to_active(struct i915_vma *vma,
 void __i915_vma_evict(struct i915_vma *vma)
 {
 	GEM_BUG_ON(i915_vma_is_pinned(vma));
+	assert_object_held_shared(vma->obj);
 
 	if (i915_vma_is_map_and_fenceable(vma)) {
 		/* Force a pagefault for domain tracking on next user access */
@@ -1636,6 +1645,7 @@ int __i915_vma_unbind(struct i915_vma *vma)
 	int ret;
 
 	lockdep_assert_held(&vma->vm->mutex);
+	assert_object_held_shared(vma->obj);
 
 	if (!drm_mm_node_allocated(&vma->node))
 		return 0;
@@ -1667,6 +1677,8 @@ int i915_vma_unbind(struct i915_vma *vma)
 	intel_wakeref_t wakeref = 0;
 	int err;
 
+	assert_object_held_shared(vma->obj);
+
 	/* Optimistic wait before taking the mutex */
 	err = i915_vma_sync(vma);
 	if (err)
@@ -1697,6 +1709,17 @@ int i915_vma_unbind(struct i915_vma *vma)
 	return err;
 }
 
+int i915_vma_unbind_unlocked(struct i915_vma *vma)
+{
+	int err;
+
+	i915_gem_object_lock(vma->obj, NULL);
+	err = i915_vma_unbind(vma);
+	i915_gem_object_unlock(vma->obj);
+
+	return err;
+}
+
 struct i915_vma *i915_vma_make_unshrinkable(struct i915_vma *vma)
 {
 	i915_gem_object_make_unshrinkable(vma->obj);
diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
index e4dc5187c8ad..74357ea71a9f 100644
--- a/drivers/gpu/drm/i915/i915_vma.h
+++ b/drivers/gpu/drm/i915/i915_vma.h
@@ -207,6 +207,7 @@ void i915_vma_revoke_mmap(struct i915_vma *vma);
 void __i915_vma_evict(struct i915_vma *vma);
 int __i915_vma_unbind(struct i915_vma *vma);
 int __must_check i915_vma_unbind(struct i915_vma *vma);
+int __must_check i915_vma_unbind_unlocked(struct i915_vma *vma);
 void i915_vma_unlink_ctx(struct i915_vma *vma);
 void i915_vma_close(struct i915_vma *vma);
 void i915_vma_reopen(struct i915_vma *vma);
diff --git a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
index a3e3d97728e7..4b936a2e724d 100644
--- a/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
+++ b/drivers/gpu/drm/i915/selftests/i915_gem_gtt.c
@@ -385,7 +385,7 @@ static void close_object_list(struct list_head *objects,
 
 		vma = i915_vma_instance(obj, vm, NULL);
 		if (!IS_ERR(vma))
-			ignored = i915_vma_unbind(vma);
+			ignored = i915_vma_unbind_unlocked(vma);
 
 		list_del(&obj->st_link);
 		i915_gem_object_put(obj);
@@ -496,7 +496,7 @@ static int fill_hole(struct i915_address_space *vm,
 						goto err;
 					}
 
-					err = i915_vma_unbind(vma);
+					err = i915_vma_unbind_unlocked(vma);
 					if (err) {
 						pr_err("%s(%s) (forward) unbind of vma.node=%llx + %llx failed with err=%d\n",
 						       __func__, p->name, vma->node.start, vma->node.size,
@@ -569,7 +569,7 @@ static int fill_hole(struct i915_address_space *vm,
 						goto err;
 					}
 
-					err = i915_vma_unbind(vma);
+					err = i915_vma_unbind_unlocked(vma);
 					if (err) {
 						pr_err("%s(%s) (backward) unbind of vma.node=%llx + %llx failed with err=%d\n",
 						       __func__, p->name, vma->node.start, vma->node.size,
@@ -655,7 +655,7 @@ static int walk_hole(struct i915_address_space *vm,
 				goto err_put;
 			}
 
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			if (err) {
 				pr_err("%s unbind failed at %llx + %llx  with err=%d\n",
 				       __func__, addr, vma->size, err);
@@ -732,13 +732,13 @@ static int pot_hole(struct i915_address_space *vm,
 				pr_err("%s incorrect at %llx + %llx\n",
 				       __func__, addr, vma->size);
 				i915_vma_unpin(vma);
-				err = i915_vma_unbind(vma);
+				err = i915_vma_unbind_unlocked(vma);
 				err = -EINVAL;
 				goto err_obj;
 			}
 
 			i915_vma_unpin(vma);
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			GEM_BUG_ON(err);
 		}
 
@@ -832,13 +832,13 @@ static int drunk_hole(struct i915_address_space *vm,
 				pr_err("%s incorrect at %llx + %llx\n",
 				       __func__, addr, BIT_ULL(size));
 				i915_vma_unpin(vma);
-				err = i915_vma_unbind(vma);
+				err = i915_vma_unbind_unlocked(vma);
 				err = -EINVAL;
 				goto err_obj;
 			}
 
 			i915_vma_unpin(vma);
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			GEM_BUG_ON(err);
 
 			if (igt_timeout(end_time,
@@ -906,7 +906,7 @@ static int __shrink_hole(struct i915_address_space *vm,
 			pr_err("%s incorrect at %llx + %llx\n",
 			       __func__, addr, size);
 			i915_vma_unpin(vma);
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			err = -EINVAL;
 			break;
 		}
@@ -1465,7 +1465,7 @@ static int igt_gtt_reserve(void *arg)
 			goto out;
 		}
 
-		err = i915_vma_unbind(vma);
+		err = i915_vma_unbind_unlocked(vma);
 		if (err) {
 			pr_err("i915_vma_unbind failed with err=%d!\n", err);
 			goto out;
@@ -1647,7 +1647,7 @@ static int igt_gtt_insert(void *arg)
 		GEM_BUG_ON(!drm_mm_node_allocated(&vma->node));
 		offset = vma->node.start;
 
-		err = i915_vma_unbind(vma);
+		err = i915_vma_unbind_unlocked(vma);
 		if (err) {
 			pr_err("i915_vma_unbind failed with err=%d!\n", err);
 			goto out;
diff --git a/drivers/gpu/drm/i915/selftests/i915_vma.c b/drivers/gpu/drm/i915/selftests/i915_vma.c
index dd0607254a95..3a1eb487826a 100644
--- a/drivers/gpu/drm/i915/selftests/i915_vma.c
+++ b/drivers/gpu/drm/i915/selftests/i915_vma.c
@@ -340,7 +340,7 @@ static int igt_vma_pin1(void *arg)
 
 		if (!err) {
 			i915_vma_unpin(vma);
-			err = i915_vma_unbind(vma);
+			err = i915_vma_unbind_unlocked(vma);
 			if (err) {
 				pr_err("Failed to unbind single page from GGTT, err=%d\n", err);
 				goto out;
-- 
2.32.0

[Intel-gfx] [PATCH 15/19] drm/i915: Remove support for unlocked i915_vma unbind

[Maarten Lankhorst]

Now that we require the object lock for all ops, some code handling
race conditions can be removed.

This is required to not take short-term pins inside execbuf.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/i915_vma.c | 40 +++++----------------------------
 1 file changed, 5 insertions(+), 35 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index da54e6882650..f6dacfc3e840 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -748,7 +748,6 @@ i915_vma_detach(struct i915_vma *vma)
 static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
 {
 	unsigned int bound;
-	bool pinned = true;
 
 	bound = atomic_read(&vma->flags);
 	do {
@@ -758,34 +757,10 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
 		if (unlikely(bound & (I915_VMA_OVERFLOW | I915_VMA_ERROR)))
 			return false;
 
-		if (!(bound & I915_VMA_PIN_MASK))
-			goto unpinned;
-
 		GEM_BUG_ON(((bound + 1) & I915_VMA_PIN_MASK) == 0);
 	} while (!atomic_try_cmpxchg(&vma->flags, &bound, bound + 1));
 
 	return true;
-
-unpinned:
-	/*
-	 * If pin_count==0, but we are bound, check under the lock to avoid
-	 * racing with a concurrent i915_vma_unbind().
-	 */
-	mutex_lock(&vma->vm->mutex);
-	do {
-		if (unlikely(bound & (I915_VMA_OVERFLOW | I915_VMA_ERROR))) {
-			pinned = false;
-			break;
-		}
-
-		if (unlikely(flags & ~bound)) {
-			pinned = false;
-			break;
-		}
-	} while (!atomic_try_cmpxchg(&vma->flags, &bound, bound + 1));
-	mutex_unlock(&vma->vm->mutex);
-
-	return pinned;
 }
 
 
@@ -1085,13 +1060,7 @@ __i915_vma_get_pages(struct i915_vma *vma)
 			vma->ggtt_view.type, ret);
 	}
 
-	pages = xchg(&vma->pages, pages);
-
-	/* did we race against a put_pages? */
-	if (pages && pages != vma->obj->mm.pages) {
-		sg_free_table(vma->pages);
-		kfree(vma->pages);
-	}
+	vma->pages = pages;
 
 	return ret;
 }
@@ -1125,13 +1094,14 @@ I915_SELFTEST_EXPORT int i915_vma_get_pages(struct i915_vma *vma)
 static void __vma_put_pages(struct i915_vma *vma, unsigned int count)
 {
 	/* We allocate under vma_get_pages, so beware the shrinker */
-	struct sg_table *pages = READ_ONCE(vma->pages);
+	struct sg_table *pages = vma->pages;
 
 	GEM_BUG_ON(atomic_read(&vma->pages_count) < count);
 
 	if (atomic_sub_return(count, &vma->pages_count) == 0) {
-		if (pages == cmpxchg(&vma->pages, pages, NULL) &&
-		    pages != vma->obj->mm.pages) {
+		vma->pages = NULL;
+
+		if (pages != vma->obj->mm.pages) {
 			sg_free_table(pages);
 			kfree(pages);
 		}
-- 
2.32.0

[Intel-gfx] [PATCH 16/19] drm/i915: Remove short-term pins from execbuf

[Maarten Lankhorst]

Add a flag PIN_VALIDATE, to indicate we don't need to pin and only
protected by the object lock.

This removes the need to unpin, which is done by just releasing the
lock.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 .../gpu/drm/i915/gem/i915_gem_execbuffer.c    | 64 ++++++-------------
 drivers/gpu/drm/i915/i915_gem_gtt.h           |  1 +
 drivers/gpu/drm/i915/i915_vma.c               | 24 +++++--
 3 files changed, 38 insertions(+), 51 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 6832b255294e..5a72f98d607c 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -429,7 +429,7 @@ eb_pin_vma(struct i915_execbuffer *eb,
 	else
 		pin_flags = entry->offset & PIN_OFFSET_MASK;
 
-	pin_flags |= PIN_USER | PIN_NOEVICT | PIN_OFFSET_FIXED;
+	pin_flags |= PIN_USER | PIN_NOEVICT | PIN_OFFSET_FIXED | PIN_VALIDATE;
 	if (unlikely(ev->flags & EXEC_OBJECT_NEEDS_GTT))
 		pin_flags |= PIN_GLOBAL;
 
@@ -447,17 +447,15 @@ eb_pin_vma(struct i915_execbuffer *eb,
 					     entry->pad_to_size,
 					     entry->alignment,
 					     eb_pin_flags(entry, ev->flags) |
-					     PIN_USER | PIN_NOEVICT);
+					     PIN_USER | PIN_NOEVICT | PIN_VALIDATE);
 		if (unlikely(err))
 			return err;
 	}
 
 	if (unlikely(ev->flags & EXEC_OBJECT_NEEDS_FENCE)) {
 		err = i915_vma_pin_fence(vma);
-		if (unlikely(err)) {
-			i915_vma_unpin(vma);
+		if (unlikely(err))
 			return err;
-		}
 
 		if (vma->fence)
 			ev->flags |= __EXEC_OBJECT_HAS_FENCE;
@@ -473,13 +471,9 @@ eb_pin_vma(struct i915_execbuffer *eb,
 static inline void
 eb_unreserve_vma(struct eb_vma *ev)
 {
-	if (!(ev->flags & __EXEC_OBJECT_HAS_PIN))
-		return;
-
 	if (unlikely(ev->flags & __EXEC_OBJECT_HAS_FENCE))
 		__i915_vma_unpin_fence(ev->vma);
 
-	__i915_vma_unpin(ev->vma);
 	ev->flags &= ~__EXEC_OBJECT_RESERVED;
 }
 
@@ -634,10 +628,8 @@ static int eb_reserve_vma(struct i915_execbuffer *eb,
 
 	if (unlikely(ev->flags & EXEC_OBJECT_NEEDS_FENCE)) {
 		err = i915_vma_pin_fence(vma);
-		if (unlikely(err)) {
-			i915_vma_unpin(vma);
+		if (unlikely(err))
 			return err;
-		}
 
 		if (vma->fence)
 			ev->flags |= __EXEC_OBJECT_HAS_FENCE;
@@ -652,7 +644,7 @@ static int eb_reserve_vma(struct i915_execbuffer *eb,
 static int eb_reserve(struct i915_execbuffer *eb)
 {
 	const unsigned int count = eb->buffer_count;
-	unsigned int pin_flags = PIN_USER | PIN_NONBLOCK;
+	unsigned int pin_flags = PIN_USER | PIN_NONBLOCK | PIN_VALIDATE;
 	struct list_head last;
 	struct eb_vma *ev;
 	unsigned int i, pass;
@@ -726,7 +718,7 @@ static int eb_reserve(struct i915_execbuffer *eb)
 			return -ENOSPC;
 		}
 
-		pin_flags = PIN_USER;
+		pin_flags = PIN_USER | PIN_VALIDATE;
 	} while (1);
 }
 
@@ -2025,7 +2017,7 @@ shadow_batch_pin(struct i915_execbuffer *eb,
 	if (IS_ERR(vma))
 		return vma;
 
-	err = i915_vma_pin_ww(vma, &eb->ww, 0, 0, flags);
+	err = i915_vma_pin_ww(vma, &eb->ww, 0, 0, flags | PIN_VALIDATE);
 	if (err)
 		return ERR_PTR(err);
 
@@ -2039,7 +2031,7 @@ static struct i915_vma *eb_dispatch_secure(struct i915_execbuffer *eb, struct i9
 	 * batch" bit. Hence we need to pin secure batches into the global gtt.
 	 * hsw should have this fixed, but bdw mucks it up again. */
 	if (eb->batch_flags & I915_DISPATCH_SECURE)
-		return i915_gem_object_ggtt_pin_ww(vma->obj, &eb->ww, NULL, 0, 0, 0);
+		return i915_gem_object_ggtt_pin_ww(vma->obj, &eb->ww, NULL, 0, 0, PIN_VALIDATE);
 
 	return NULL;
 }
@@ -2087,13 +2079,12 @@ static int eb_parse(struct i915_execbuffer *eb)
 
 	err = i915_gem_object_lock(pool->obj, &eb->ww);
 	if (err)
-		goto err;
+		return err;
 
 	shadow = shadow_batch_pin(eb, pool->obj, eb->context->vm, PIN_USER);
-	if (IS_ERR(shadow)) {
-		err = PTR_ERR(shadow);
-		goto err;
-	}
+	if (IS_ERR(shadow))
+		return PTR_ERR(shadow);
+
 	intel_gt_buffer_pool_mark_used(pool);
 	i915_gem_object_set_readonly(shadow->obj);
 	shadow->private = pool;
@@ -2105,25 +2096,21 @@ static int eb_parse(struct i915_execbuffer *eb)
 		shadow = shadow_batch_pin(eb, pool->obj,
 					  &eb->engine->gt->ggtt->vm,
 					  PIN_GLOBAL);
-		if (IS_ERR(shadow)) {
-			err = PTR_ERR(shadow);
-			shadow = trampoline;
-			goto err_shadow;
-		}
+		if (IS_ERR(shadow))
+			return PTR_ERR(shadow);
+
 		shadow->private = pool;
 
 		eb->batch_flags |= I915_DISPATCH_SECURE;
 	}
 
 	batch = eb_dispatch_secure(eb, shadow);
-	if (IS_ERR(batch)) {
-		err = PTR_ERR(batch);
-		goto err_trampoline;
-	}
+	if (IS_ERR(batch))
+		return PTR_ERR(batch);
 
 	err = dma_resv_reserve_shared(shadow->obj->base.resv, 1);
 	if (err)
-		goto err_trampoline;
+		return err;
 
 	err = intel_engine_cmd_parser(eb->engine,
 				      eb->batch->vma,
@@ -2131,7 +2118,7 @@ static int eb_parse(struct i915_execbuffer *eb)
 				      eb->batch_len,
 				      shadow, trampoline);
 	if (err)
-		goto err_unpin_batch;
+		return err;
 
 	eb->batch = &eb->vma[eb->buffer_count++];
 	eb->batch->vma = i915_vma_get(shadow);
@@ -2147,17 +2134,6 @@ static int eb_parse(struct i915_execbuffer *eb)
 		eb->batch->vma = i915_vma_get(batch);
 	}
 	return 0;
-
-err_unpin_batch:
-	if (batch)
-		i915_vma_unpin(batch);
-err_trampoline:
-	if (trampoline)
-		i915_vma_unpin(trampoline);
-err_shadow:
-	i915_vma_unpin(shadow);
-err:
-	return err;
 }
 
 static int eb_submit(struct i915_execbuffer *eb, struct i915_vma *batch)
@@ -3003,8 +2979,6 @@ i915_gem_do_execbuffer(struct drm_device *dev,
 
 err_vma:
 	eb_release_vmas(&eb, true);
-	if (eb.trampoline)
-		i915_vma_unpin(eb.trampoline);
 	WARN_ON(err == -EDEADLK);
 	i915_gem_ww_ctx_fini(&eb.ww);
 
diff --git a/drivers/gpu/drm/i915/i915_gem_gtt.h b/drivers/gpu/drm/i915/i915_gem_gtt.h
index c9b0ee5e1d23..078e73004d01 100644
--- a/drivers/gpu/drm/i915/i915_gem_gtt.h
+++ b/drivers/gpu/drm/i915/i915_gem_gtt.h
@@ -41,6 +41,7 @@ int i915_gem_gtt_insert(struct i915_address_space *vm,
 #define PIN_HIGH		BIT_ULL(5)
 #define PIN_OFFSET_BIAS		BIT_ULL(6)
 #define PIN_OFFSET_FIXED	BIT_ULL(7)
+#define PIN_VALIDATE		BIT_ULL(8) /* validate placement only, no need to call unpin() */
 
 #define PIN_GLOBAL		BIT_ULL(10) /* I915_VMA_GLOBAL_BIND */
 #define PIN_USER		BIT_ULL(11) /* I915_VMA_LOCAL_BIND */
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index f6dacfc3e840..bfba299d905c 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -750,6 +750,15 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
 	unsigned int bound;
 
 	bound = atomic_read(&vma->flags);
+
+	if (flags & PIN_VALIDATE) {
+		flags &= I915_VMA_BIND_MASK;
+
+		return (flags & bound) == flags;
+	}
+
+	/* with the lock mandatory for unbind, we don't race here */
+	flags &= I915_VMA_BIND_MASK;
 	do {
 		if (unlikely(flags & ~bound))
 			return false;
@@ -1149,7 +1158,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	GEM_BUG_ON(!(flags & (PIN_USER | PIN_GLOBAL)));
 
 	/* First try and grab the pin without rebinding the vma */
-	if (try_qad_pin(vma, flags & I915_VMA_BIND_MASK))
+	if (try_qad_pin(vma, flags))
 		return 0;
 
 	err = i915_vma_get_pages(vma);
@@ -1228,7 +1237,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	}
 
 	if (unlikely(!(flags & ~bound & I915_VMA_BIND_MASK))) {
-		__i915_vma_pin(vma);
+		if (!(flags & PIN_VALIDATE))
+			__i915_vma_pin(vma);
 		goto err_unlock;
 	}
 
@@ -1257,8 +1267,10 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	atomic_add(I915_VMA_PAGES_ACTIVE, &vma->pages_count);
 	list_move_tail(&vma->vm_link, &vma->vm->bound_list);
 
-	__i915_vma_pin(vma);
-	GEM_BUG_ON(!i915_vma_is_pinned(vma));
+	if (!(flags & PIN_VALIDATE)) {
+		__i915_vma_pin(vma);
+		GEM_BUG_ON(!i915_vma_is_pinned(vma));
+	}
 	GEM_BUG_ON(!i915_vma_is_bound(vma, flags));
 	GEM_BUG_ON(i915_vma_misplaced(vma, size, alignment, flags));
 
@@ -1509,8 +1521,6 @@ static int __i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *
 {
 	int err;
 
-	GEM_BUG_ON(!i915_vma_is_pinned(vma));
-
 	/* Wait for the vma to be bound before we start! */
 	err = __i915_request_await_bind(rq, vma);
 	if (err)
@@ -1528,6 +1538,8 @@ int i915_vma_move_to_active(struct i915_vma *vma,
 
 	assert_object_held(obj);
 
+	GEM_BUG_ON(!vma->pages);
+
 	err = __i915_vma_move_to_active(vma, rq);
 	if (unlikely(err))
 		return err;
-- 
2.32.0

[Intel-gfx] [PATCH 17/19] drm/i915: Add functions to set/get moving fence

[Maarten Lankhorst]

We want to get rid of i915_vma tracking to simplify the code and
lifetimes. Add a way to set/put the moving fence, in preparation for
removing the tracking.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_object.c | 15 +++++++++++++++
 drivers/gpu/drm/i915/gem/i915_gem_object.h |  6 ++++++
 2 files changed, 21 insertions(+)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
index 6fb9afb65034..dc0d2da297a0 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
@@ -32,6 +32,7 @@
 #include "i915_gem_object.h"
 #include "i915_memcpy.h"
 #include "i915_trace.h"
+#include "i915_gem_ttm.h"
 
 static struct kmem_cache *slab_objects;
 
@@ -674,6 +675,20 @@ static const struct drm_gem_object_funcs i915_gem_object_funcs = {
 	.export = i915_gem_prime_export,
 };
 
+struct dma_fence *
+i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj)
+{
+	return dma_fence_get(i915_gem_to_ttm(obj)->moving);
+}
+
+void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
+				       struct dma_fence *fence)
+{
+	dma_fence_put(i915_gem_to_ttm(obj)->moving);
+
+	i915_gem_to_ttm(obj)->moving = dma_fence_get(fence);
+}
+
 #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
 #include "selftests/huge_gem_object.c"
 #include "selftests/huge_pages.c"
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
index 48112b9d76df..a23acfa98e21 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
@@ -520,6 +520,12 @@ i915_gem_object_last_write_engine(struct drm_i915_gem_object *obj)
 	return engine;
 }
 
+struct dma_fence *
+i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj);
+
+void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
+				       struct dma_fence *fence);
+
 void i915_gem_object_set_cache_coherency(struct drm_i915_gem_object *obj,
 					 unsigned int cache_level);
 void i915_gem_object_flush_if_display(struct drm_i915_gem_object *obj);
-- 
2.32.0

[Intel-gfx] [PATCH 18/19] drm/i915: Add support for asynchronous moving fence waiting

[Maarten Lankhorst]

For now, we will only allow async migration when TTM is used,
so the paths we care about are related to TTM.

The mmap path is handled by having the fence in ttm_bo->moving,
when pinning, the binding only becomes available after the moving
fence is signaled, and pinning a cpu map will only work after
the moving fence signals.

This should close all holes where userspace can read a buffer
before it's fully migrated.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_pages.c | 10 ++++++++++
 drivers/gpu/drm/i915/i915_vma.c           | 10 ++++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
index 8eb1c3a6fc9c..a044d1829729 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
@@ -345,6 +345,7 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj,
 			      enum i915_map_type type)
 {
 	enum i915_map_type has_type;
+	struct dma_fence *moving;
 	bool pinned;
 	void *ptr;
 	int err;
@@ -355,6 +356,15 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj,
 
 	assert_object_held(obj);
 
+	moving = i915_gem_object_get_moving_fence(obj);
+	if (moving) {
+		err = dma_fence_wait(moving, true);
+		dma_fence_put(moving);
+
+		if (err)
+			return ERR_PTR(err);
+	}
+
 	pinned = !(type & I915_MAP_OVERRIDE);
 	type &= ~I915_MAP_OVERRIDE;
 
diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
index bfba299d905c..320678e9b4e1 100644
--- a/drivers/gpu/drm/i915/i915_vma.c
+++ b/drivers/gpu/drm/i915/i915_vma.c
@@ -398,7 +398,7 @@ int i915_vma_bind(struct i915_vma *vma,
 	GEM_BUG_ON(!vma->pages);
 
 	trace_i915_vma_bind(vma, bind_flags);
-	if (work && bind_flags & vma->vm->bind_async_flags) {
+	if (work) {
 		struct dma_fence *prev;
 
 		work->vma = vma;
@@ -1145,6 +1145,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 		    u64 size, u64 alignment, u64 flags)
 {
 	struct i915_vma_work *work = NULL;
+	struct dma_fence *moving = NULL;
 	intel_wakeref_t wakeref = 0;
 	unsigned int bound;
 	int err;
@@ -1168,7 +1169,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 	if (flags & PIN_GLOBAL)
 		wakeref = intel_runtime_pm_get(&vma->vm->i915->runtime_pm);
 
-	if (flags & vma->vm->bind_async_flags) {
+	moving = i915_gem_object_get_moving_fence(vma->obj);
+	if (flags & vma->vm->bind_async_flags || moving) {
 		/* lock VM */
 		err = i915_vm_lock_objects(vma->vm, ww);
 		if (err)
@@ -1182,6 +1184,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 
 		work->vm = i915_vm_get(vma->vm);
 
+		dma_fence_work_chain(&work->base, moving);
+
 		/* Allocate enough page directories to used PTE */
 		if (vma->vm->allocate_va_range) {
 			err = i915_vm_alloc_pt_stash(vma->vm,
@@ -1289,6 +1293,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
 err_rpm:
 	if (wakeref)
 		intel_runtime_pm_put(&vma->vm->i915->runtime_pm, wakeref);
+	if (moving)
+		dma_fence_put(moving);
 	i915_vma_put_pages(vma);
 	return err;
 }
-- 
2.32.0

[Intel-gfx] [PATCH 19/19] drm/i915: Add accelerated migration to ttm

[Maarten Lankhorst]

Expose the fence to ttm_bo->moving, which will get picked up by i915
through the i915_gem_object_get_moving_fence call. Should be sufficient
for the needs we have.

Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
---
 drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 104 ++++++++++++------------
 1 file changed, 54 insertions(+), 50 deletions(-)

diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
index eaf2ff29dd4a..751abbc3ccd3 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
@@ -353,10 +353,14 @@ static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
 
 static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
 {
-	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+	struct drm_i915_gem_object *obj;
 
-	if (likely(obj))
-		i915_ttm_free_cached_io_st(obj);
+	if (bo->destroy != i915_ttm_bo_destroy)
+		return;
+
+	obj = i915_ttm_to_gem(bo);
+
+	i915_ttm_free_cached_io_st(obj);
 }
 
 static struct intel_memory_region *
@@ -426,10 +430,10 @@ i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
 	return intel_region_ttm_resource_to_st(obj->mm.region, res);
 }
 
-static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
-			       bool clear,
-			       struct ttm_resource *dst_mem,
-			       struct sg_table *dst_st)
+static struct i915_request *i915_ttm_accel_move(struct ttm_buffer_object *bo,
+						bool clear,
+						struct ttm_resource *dst_mem,
+						struct sg_table *dst_st)
 {
 	struct drm_i915_private *i915 = container_of(bo->bdev, typeof(*i915),
 						     bdev);
@@ -437,18 +441,18 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
 		ttm_manager_type(bo->bdev, bo->resource->mem_type);
 	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
 	struct sg_table *src_st;
-	struct i915_request *rq;
+	struct i915_request *rq = NULL;
 	struct ttm_tt *ttm = bo->ttm;
 	enum i915_cache_level src_level, dst_level;
 	int ret;
 
 	if (!i915->gt.migrate.context)
-		return -EINVAL;
+		return ERR_PTR(-EINVAL);
 
 	dst_level = i915_ttm_cache_level(i915, dst_mem, ttm);
 	if (clear) {
 		if (bo->type == ttm_bo_type_kernel)
-			return -EINVAL;
+			return ERR_PTR(-EINVAL);
 
 		intel_engine_pm_get(i915->gt.migrate.context->engine);
 		ret = intel_context_migrate_clear(i915->gt.migrate.context, NULL,
@@ -456,10 +460,6 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
 						  gpu_binds_iomem(dst_mem),
 						  0, &rq);
 
-		if (!ret && rq) {
-			i915_request_wait(rq, 0, MAX_SCHEDULE_TIMEOUT);
-			i915_request_put(rq);
-		}
 		intel_engine_pm_put(i915->gt.migrate.context->engine);
 	} else {
 		src_st = src_man->use_tt ? i915_ttm_tt_get_st(ttm) :
@@ -473,49 +473,40 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
 						 dst_st->sgl, dst_level,
 						 gpu_binds_iomem(dst_mem),
 						 &rq);
-		if (!ret && rq) {
-			i915_request_wait(rq, 0, MAX_SCHEDULE_TIMEOUT);
-			i915_request_put(rq);
-		}
 		intel_engine_pm_put(i915->gt.migrate.context->engine);
 	}
 
-	return ret;
+	return ret ? ERR_PTR(ret) : rq;
 }
 
-static void __i915_ttm_move(struct ttm_buffer_object *bo, bool clear,
+static void i915_ttm_memcpy(struct ttm_buffer_object *bo, bool clear,
 			    struct ttm_resource *dst_mem,
 			    struct sg_table *dst_st)
 {
-	int ret;
+	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
+	struct intel_memory_region *dst_reg, *src_reg;
+	union {
+		struct ttm_kmap_iter_tt tt;
+		struct ttm_kmap_iter_iomap io;
+	} _dst_iter, _src_iter;
+	struct ttm_kmap_iter *dst_iter, *src_iter;
 
-	ret = i915_ttm_accel_move(bo, clear, dst_mem, dst_st);
-	if (ret) {
-		struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
-		struct intel_memory_region *dst_reg, *src_reg;
-		union {
-			struct ttm_kmap_iter_tt tt;
-			struct ttm_kmap_iter_iomap io;
-		} _dst_iter, _src_iter;
-		struct ttm_kmap_iter *dst_iter, *src_iter;
-
-		dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
-		src_reg = i915_ttm_region(bo->bdev, bo->resource->mem_type);
-		GEM_BUG_ON(!dst_reg || !src_reg);
-
-		dst_iter = !cpu_maps_iomem(dst_mem) ?
-			ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
-			ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
-						 dst_st, dst_reg->region.start);
-
-		src_iter = !cpu_maps_iomem(bo->resource) ?
-			ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
-			ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
-						 obj->ttm.cached_io_st,
-						 src_reg->region.start);
-
-		ttm_move_memcpy(clear, dst_mem->num_pages, dst_iter, src_iter);
-	}
+	dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
+	src_reg = i915_ttm_region(bo->bdev, bo->resource->mem_type);
+	GEM_BUG_ON(!dst_reg || !src_reg);
+
+	dst_iter = !cpu_maps_iomem(dst_mem) ?
+		ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
+		ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+					  dst_st, dst_reg->region.start);
+
+	src_iter = !cpu_maps_iomem(bo->resource) ?
+		ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
+		ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+					  obj->ttm.cached_io_st,
+					  src_reg->region.start);
+
+	ttm_move_memcpy(clear, dst_mem->num_pages, dst_iter, src_iter);
 }
 
 static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
@@ -530,6 +521,7 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
 	struct sg_table *dst_st;
 	bool clear;
 	int ret;
+	struct i915_request *rq;
 
 	/* Sync for now. We could do the actual copy async. */
 	ret = ttm_bo_wait_ctx(bo, ctx);
@@ -558,10 +550,22 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
 		return PTR_ERR(dst_st);
 
 	clear = !cpu_maps_iomem(bo->resource) && (!ttm || !ttm_tt_is_populated(ttm));
+
+	rq = NULL;
 	if (!(clear && ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)))
-		__i915_ttm_move(bo, clear, dst_mem, dst_st);
+		rq = i915_ttm_accel_move(bo, clear, dst_mem, dst_st);
+
+	if (IS_ERR(rq))
+		i915_ttm_memcpy(bo, clear, dst_mem, dst_st);
+
+	/* Below dst_mem becomes bo->resource. */
+	if (!IS_ERR_OR_NULL(rq)) {
+		ttm_bo_move_accel_cleanup(bo, &rq->fence, evict, true, dst_mem);
+		i915_request_put(rq);
+	} else {
+		ttm_bo_move_sync_cleanup(bo, dst_mem);
+	}
 
-	ttm_bo_move_sync_cleanup(bo, dst_mem);
 	i915_ttm_adjust_domains_after_move(obj);
 	i915_ttm_free_cached_io_st(obj);
 
-- 
2.32.0

[Intel-gfx] ✗ Fi.CI.CHECKPATCH: warning for drm/i915: Short-term pinning and async eviction.

[Patchwork]

== Series Details ==

Series: drm/i915: Short-term pinning and async eviction.
URL   : https://patchwork.freedesktop.org/series/94162/
State : warning

== Summary ==

$ dim checkpatch origin/drm-tip
fa0ce1cd930e drm/i915: Move __i915_gem_free_object to ttm_bo_destroy
667b39786fe3 drm/i915: Remove unused bits of i915_vma/active api
98c59acb1911 drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling
ce526eaaf6dc drm/i915: Remove gen6_ppgtt_unpin_all
7f62692b5277 drm/i915: Create a dummy object for gen6 ppgtt
-:178: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis
#178: FILE: drivers/gpu/drm/i915/gt/gen6_ppgtt.c:376:
+static void pd_dummy_obj_put_pages(struct drm_i915_gem_object *obj,
+				     struct sg_table *pages)

-:200: WARNING:LONG_LINE: line length of 119 exceeds 100 columns
#200: FILE: drivers/gpu/drm/i915/gt/gen6_ppgtt.c:398:
+	pd->pt.base = __i915_gem_object_create_internal(ppgtt->base.vm.gt->i915, &pd_dummy_obj_ops, I915_PDES * SZ_4K);

total: 0 errors, 1 warnings, 1 checks, 256 lines checked
faee95e0ebdf drm/i915: Create a full object for mock_ring
-:6: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line)
#6: 
This allows us to finally get rid of all the assumptions that vma->obj is NULL.

total: 0 errors, 1 warnings, 0 checks, 58 lines checked
c13839878262 drm/i915: vma is always backed by an object.
2885005de36d drm/i915: Fix runtime pm handling in i915_gem_shrink
92874150b4e7 drm/i915: Change shrink ordering to use locking around unbinding.
-:27: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis
#27: FILE: drivers/gpu/drm/i915/gem/i915_gem_shrinker.c:41:
+static int drop_pages(struct drm_i915_gem_object *obj,
+		       unsigned long shrink, bool trylock_vm)

total: 0 errors, 0 warnings, 1 checks, 56 lines checked
c7b50af89f7a Move CONTEXT_VALID_BIT check
-:7: WARNING:COMMIT_MESSAGE: Missing commit description - Add an appropriate one

total: 0 errors, 1 warnings, 0 checks, 17 lines checked
9a8f87820ed4 drm/i915: Remove resv from i915_vma
81200f4aea0f drm/i915: Remove pages_mutex and intel_gtt->vma_ops.set/clear_pages members
-:520: CHECK:LINE_SPACING: Please don't use multiple blank lines
#520: FILE: drivers/gpu/drm/i915/i915_vma.c:791:
 
+

total: 0 errors, 0 warnings, 1 checks, 633 lines checked
d0e5dee58521 drm/i915: Take object lock in i915_ggtt_pin if ww is not set
eea081368486 drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for i915_vma_unbind
-:7: WARNING:COMMIT_LOG_LONG_LINE: Possible unwrapped commit description (prefer a maximum 75 chars per line)
#7: 
We want to remove more members of i915_vma, which requires the locking to be

-:337: CHECK:LINE_SPACING: Please don't use multiple blank lines
#337: FILE: drivers/gpu/drm/i915/i915_gem_evict.c:258:
 
+

total: 0 errors, 1 warnings, 1 checks, 513 lines checked
741722532bf6 drm/i915: Remove support for unlocked i915_vma unbind
66f02650e597 drm/i915: Remove short-term pins from execbuf
49d5bf5ffe41 drm/i915: Add functions to set/get moving fence
c29934165e29 drm/i915: Add support for asynchronous moving fence waiting
992d547f7e4d drm/i915: Add accelerated migration to ttm
-:145: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis
#145: FILE: drivers/gpu/drm/i915/gem/i915_gem_ttm.c:501:
+		ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
+					  dst_st, dst_reg->region.start);

-:150: CHECK:PARENTHESIS_ALIGNMENT: Alignment should match open parenthesis
#150: FILE: drivers/gpu/drm/i915/gem/i915_gem_ttm.c:506:
+		ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
+					  obj->ttm.cached_io_st,

total: 0 errors, 0 warnings, 2 checks, 167 lines checked

[Intel-gfx] ✗ Fi.CI.SPARSE: warning for drm/i915: Short-term pinning and async eviction.

[Patchwork]

== Series Details ==

Series: drm/i915: Short-term pinning and async eviction.
URL   : https://patchwork.freedesktop.org/series/94162/
State : warning

== Summary ==

$ dim sparse --fast origin/drm-tip
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.
-
+drivers/gpu/drm/i915/gem/i915_gem_context.c:1374:34:    expected struct i915_address_space *vm
+drivers/gpu/drm/i915/gem/i915_gem_context.c:1374:34:    got struct i915_address_space [noderef] __rcu *vm
+drivers/gpu/drm/i915/gem/i915_gem_context.c:1374:34: warning: incorrect type in argument 1 (different address spaces)
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:43:25:    expected struct i915_address_space [noderef] __rcu *vm
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:43:25:    got struct i915_address_space *
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:43:25: warning: incorrect type in assignment (different address spaces)
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:60:34:    expected struct i915_address_space *vm
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:60:34:    got struct i915_address_space [noderef] __rcu *vm
+drivers/gpu/drm/i915/gem/selftests/mock_context.c:60:34: warning: incorrect type in argument 1 (different address spaces)
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:27:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:27:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:27:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:32:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:32:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:49:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:49:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:49:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:56:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_engine_stats.h:56:9: warning: trying to copy expression type 31
+drivers/gpu/drm/i915/gt/intel_reset.c:1392:5: warning: context imbalance in 'intel_gt_reset_trylock' - different lock contexts for basic block
+drivers/gpu/drm/i915/i915_perf.c:1442:15: warning: memset with byte count of 16777216
+drivers/gpu/drm/i915/i915_perf.c:1496:15: warning: memset with byte count of 16777216
+./include/asm-generic/bitops/find.h:112:45: warning: shift count is negative (-262080)
+./include/asm-generic/bitops/find.h:32:31: warning: shift count is negative (-262080)
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_read16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_read32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_read64' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_read8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_write16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_write32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'fwtable_write8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_read16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_read32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_read64' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_read8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_write16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_write32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen11_fwtable_write8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen12_fwtable_write16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen12_fwtable_write32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen12_fwtable_write8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_read16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_read32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_read64' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_read8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_write16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_write32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen6_write8' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen8_write16' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen8_write32' - different lock contexts for basic block
+./include/linux/spinlock.h:409:9: warning: context imbalance in 'gen8_write8' - different lock contexts for basic block
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+./include/linux/stddef.h:17:9: this was the original definition
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined
+/usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h:417:9: warning: preprocessor token offsetof redefined

[Intel-gfx] ✗ Fi.CI.BAT: failure for drm/i915: Short-term pinning and async eviction.

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 16565 bytes --]

== Series Details ==

Series: drm/i915: Short-term pinning and async eviction.
URL   : https://patchwork.freedesktop.org/series/94162/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_10536 -> Patchwork_20917
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with Patchwork_20917 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_20917, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/index.html

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in Patchwork_20917:

### IGT changes ###

#### Possible regressions ####

  * igt@gem_busy@busy@all:
    - fi-ilk-650:         [PASS][1] -> [FAIL][2] +1 similar issue
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-ilk-650/igt@gem_busy@busy@all.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-ilk-650/igt@gem_busy@busy@all.html

  * igt@gem_exec_gttfill@basic:
    - fi-pnv-d510:        [PASS][3] -> [FAIL][4]
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-pnv-d510/igt@gem_exec_gttfill@basic.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-pnv-d510/igt@gem_exec_gttfill@basic.html
    - fi-elk-e7500:       [PASS][5] -> [FAIL][6] +1 similar issue
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-elk-e7500/igt@gem_exec_gttfill@basic.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-elk-e7500/igt@gem_exec_gttfill@basic.html

  * igt@i915_selftest@live@client:
    - fi-ivb-3770:        [PASS][7] -> [DMESG-FAIL][8]
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-ivb-3770/igt@i915_selftest@live@client.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-ivb-3770/igt@i915_selftest@live@client.html
    - fi-hsw-4770:        [PASS][9] -> [DMESG-FAIL][10]
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-hsw-4770/igt@i915_selftest@live@client.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-hsw-4770/igt@i915_selftest@live@client.html
    - fi-snb-2600:        [PASS][11] -> [DMESG-FAIL][12]
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-snb-2600/igt@i915_selftest@live@client.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2600/igt@i915_selftest@live@client.html
    - fi-snb-2520m:       [PASS][13] -> [DMESG-FAIL][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-snb-2520m/igt@i915_selftest@live@client.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2520m/igt@i915_selftest@live@client.html

  * igt@i915_selftest@live@evict:
    - fi-rkl-guc:         NOTRUN -> [DMESG-WARN][15]
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-rkl-guc/igt@i915_selftest@live@evict.html

  * igt@i915_selftest@live@gem_contexts:
    - fi-skl-6600u:       [PASS][16] -> [DMESG-FAIL][17] +1 similar issue
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-skl-6600u/igt@i915_selftest@live@gem_contexts.html
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-skl-6600u/igt@i915_selftest@live@gem_contexts.html
    - fi-bsw-kefka:       [PASS][18] -> [DMESG-FAIL][19]
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-bsw-kefka/igt@i915_selftest@live@gem_contexts.html
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-bsw-kefka/igt@i915_selftest@live@gem_contexts.html
    - fi-snb-2520m:       [PASS][20] -> [DMESG-WARN][21]
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-snb-2520m/igt@i915_selftest@live@gem_contexts.html
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2520m/igt@i915_selftest@live@gem_contexts.html
    - fi-glk-dsi:         [PASS][22] -> [DMESG-FAIL][23] +1 similar issue
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-glk-dsi/igt@i915_selftest@live@gem_contexts.html
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-glk-dsi/igt@i915_selftest@live@gem_contexts.html
    - fi-icl-u2:          [PASS][24] -> [DMESG-FAIL][25]
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-icl-u2/igt@i915_selftest@live@gem_contexts.html
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-icl-u2/igt@i915_selftest@live@gem_contexts.html
    - fi-skl-6700k2:      [PASS][26] -> [DMESG-FAIL][27]
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-skl-6700k2/igt@i915_selftest@live@gem_contexts.html
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-skl-6700k2/igt@i915_selftest@live@gem_contexts.html
    - fi-cfl-8700k:       [PASS][28] -> [DMESG-FAIL][29]
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-cfl-8700k/igt@i915_selftest@live@gem_contexts.html
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-cfl-8700k/igt@i915_selftest@live@gem_contexts.html
    - fi-ivb-3770:        [PASS][30] -> [DMESG-WARN][31]
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-ivb-3770/igt@i915_selftest@live@gem_contexts.html
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-ivb-3770/igt@i915_selftest@live@gem_contexts.html
    - fi-kbl-7567u:       [PASS][32] -> [DMESG-FAIL][33]
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-7567u/igt@i915_selftest@live@gem_contexts.html
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-7567u/igt@i915_selftest@live@gem_contexts.html
    - fi-skl-guc:         [PASS][34] -> [DMESG-FAIL][35]
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-skl-guc/igt@i915_selftest@live@gem_contexts.html
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-skl-guc/igt@i915_selftest@live@gem_contexts.html
    - fi-cfl-guc:         [PASS][36] -> [DMESG-FAIL][37]
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-cfl-guc/igt@i915_selftest@live@gem_contexts.html
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-cfl-guc/igt@i915_selftest@live@gem_contexts.html
    - fi-snb-2600:        [PASS][38] -> [DMESG-WARN][39]
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-snb-2600/igt@i915_selftest@live@gem_contexts.html
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2600/igt@i915_selftest@live@gem_contexts.html
    - fi-hsw-4770:        [PASS][40] -> [DMESG-WARN][41]
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-hsw-4770/igt@i915_selftest@live@gem_contexts.html
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-hsw-4770/igt@i915_selftest@live@gem_contexts.html
    - fi-bxt-dsi:         [PASS][42] -> [DMESG-FAIL][43] +1 similar issue
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-bxt-dsi/igt@i915_selftest@live@gem_contexts.html
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-bxt-dsi/igt@i915_selftest@live@gem_contexts.html
    - fi-tgl-1115g4:      [PASS][44] -> [DMESG-FAIL][45] +1 similar issue
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-tgl-1115g4/igt@i915_selftest@live@gem_contexts.html
   [45]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-tgl-1115g4/igt@i915_selftest@live@gem_contexts.html
    - fi-cml-u2:          [PASS][46] -> [DMESG-FAIL][47]
   [46]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-cml-u2/igt@i915_selftest@live@gem_contexts.html
   [47]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-cml-u2/igt@i915_selftest@live@gem_contexts.html
    - fi-kbl-8809g:       [PASS][48] -> [DMESG-FAIL][49]
   [48]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-8809g/igt@i915_selftest@live@gem_contexts.html
   [49]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-8809g/igt@i915_selftest@live@gem_contexts.html
    - fi-cfl-8109u:       [PASS][50] -> [DMESG-FAIL][51]
   [50]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-cfl-8109u/igt@i915_selftest@live@gem_contexts.html
   [51]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-cfl-8109u/igt@i915_selftest@live@gem_contexts.html
    - fi-bsw-nick:        [PASS][52] -> [DMESG-FAIL][53]
   [52]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-bsw-nick/igt@i915_selftest@live@gem_contexts.html
   [53]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-bsw-nick/igt@i915_selftest@live@gem_contexts.html

  * igt@i915_selftest@live@hugepages:
    - fi-rkl-11600:       [PASS][54] -> [DMESG-FAIL][55] +1 similar issue
   [54]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-rkl-11600/igt@i915_selftest@live@hugepages.html
   [55]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-rkl-11600/igt@i915_selftest@live@hugepages.html
    - fi-icl-y:           [PASS][56] -> [DMESG-FAIL][57] +1 similar issue
   [56]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-icl-y/igt@i915_selftest@live@hugepages.html
   [57]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-icl-y/igt@i915_selftest@live@hugepages.html
    - fi-kbl-guc:         [PASS][58] -> [DMESG-FAIL][59] +1 similar issue
   [58]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-guc/igt@i915_selftest@live@hugepages.html
   [59]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-guc/igt@i915_selftest@live@hugepages.html
    - fi-kbl-7500u:       [PASS][60] -> [DMESG-FAIL][61] +1 similar issue
   [60]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-7500u/igt@i915_selftest@live@hugepages.html
   [61]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-7500u/igt@i915_selftest@live@hugepages.html
    - fi-kbl-r:           [PASS][62] -> [DMESG-FAIL][63] +1 similar issue
   [62]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-r/igt@i915_selftest@live@hugepages.html
   [63]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-r/igt@i915_selftest@live@hugepages.html
    - fi-kbl-soraka:      [PASS][64] -> [DMESG-FAIL][65] +1 similar issue
   [64]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-kbl-soraka/igt@i915_selftest@live@hugepages.html
   [65]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-soraka/igt@i915_selftest@live@hugepages.html
    - fi-rkl-guc:         NOTRUN -> [DMESG-FAIL][66]
   [66]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-rkl-guc/igt@i915_selftest@live@hugepages.html

  
#### Suppressed ####

  The following results come from untrusted machines, tests, or statuses.
  They do not affect the overall result.

  * igt@i915_selftest@live@gem_contexts:
    - {fi-jsl-1}:         [PASS][67] -> [DMESG-FAIL][68] +1 similar issue
   [67]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-jsl-1/igt@i915_selftest@live@gem_contexts.html
   [68]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-jsl-1/igt@i915_selftest@live@gem_contexts.html

  * igt@i915_selftest@live@hugepages:
    - {fi-ehl-2}:         [PASS][69] -> [DMESG-FAIL][70] +1 similar issue
   [69]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-ehl-2/igt@i915_selftest@live@hugepages.html
   [70]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-ehl-2/igt@i915_selftest@live@hugepages.html
    - {fi-tgl-dsi}:       [PASS][71] -> [DMESG-FAIL][72]
   [71]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-tgl-dsi/igt@i915_selftest@live@hugepages.html
   [72]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-tgl-dsi/igt@i915_selftest@live@hugepages.html

  
Known issues
------------

  Here are the changes found in Patchwork_20917 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@amdgpu/amd_cs_nop@sync-fork-compute0:
    - fi-kbl-soraka:      NOTRUN -> [SKIP][73] ([fdo#109271]) +9 similar issues
   [73]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-kbl-soraka/igt@amdgpu/amd_cs_nop@sync-fork-compute0.html

  * igt@gem_tiled_blits@basic:
    - fi-pnv-d510:        [PASS][74] -> [INCOMPLETE][75] ([i915#299])
   [74]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-pnv-d510/igt@gem_tiled_blits@basic.html
   [75]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-pnv-d510/igt@gem_tiled_blits@basic.html

  * igt@runner@aborted:
    - fi-snb-2520m:       NOTRUN -> [FAIL][76] ([i915#2426])
   [76]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2520m/igt@runner@aborted.html
    - fi-hsw-4770:        NOTRUN -> [FAIL][77] ([fdo#109271] / [i915#1436])
   [77]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-hsw-4770/igt@runner@aborted.html
    - fi-snb-2600:        NOTRUN -> [FAIL][78] ([i915#2426])
   [78]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-snb-2600/igt@runner@aborted.html
    - fi-ivb-3770:        NOTRUN -> [FAIL][79] ([fdo#109271] / [i915#2426])
   [79]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-ivb-3770/igt@runner@aborted.html

  
#### Possible fixes ####

  * igt@i915_selftest@live@workarounds:
    - fi-rkl-guc:         [DMESG-FAIL][80] ([i915#3928]) -> [PASS][81]
   [80]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-rkl-guc/igt@i915_selftest@live@workarounds.html
   [81]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-rkl-guc/igt@i915_selftest@live@workarounds.html

  
#### Warnings ####

  * igt@runner@aborted:
    - fi-rkl-guc:         [FAIL][82] ([i915#3928]) -> [FAIL][83] ([i915#2426] / [i915#3928])
   [82]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10536/fi-rkl-guc/igt@runner@aborted.html
   [83]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/fi-rkl-guc/igt@runner@aborted.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
  [i915#2426]: https://gitlab.freedesktop.org/drm/intel/issues/2426
  [i915#299]: https://gitlab.freedesktop.org/drm/intel/issues/299
  [i915#3717]: https://gitlab.freedesktop.org/drm/intel/issues/3717
  [i915#3928]: https://gitlab.freedesktop.org/drm/intel/issues/3928


Participating hosts (44 -> 36)
------------------------------

  Missing    (8): fi-ilk-m540 bat-adls-5 bat-dg1-6 bat-dg1-5 fi-bsw-cyan fi-ctg-p8600 fi-bdw-samus bat-jsl-1 


Build changes
-------------

  * Linux: CI_DRM_10536 -> Patchwork_20917

  CI-20190529: 20190529
  CI_DRM_10536: b6d834a72c14ca22b2df32a607c1ee36e8629f1a @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_6191: e9292b533691784f46eeb9bae522ca7a8710c920 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_20917: 992d547f7e4dfe47ce09a9603c9da84697be9c30 @ git://anongit.freedesktop.org/gfx-ci/linux


== Linux commits ==

992d547f7e4d drm/i915: Add accelerated migration to ttm
c29934165e29 drm/i915: Add support for asynchronous moving fence waiting
49d5bf5ffe41 drm/i915: Add functions to set/get moving fence
66f02650e597 drm/i915: Remove short-term pins from execbuf
741722532bf6 drm/i915: Remove support for unlocked i915_vma unbind
eea081368486 drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for i915_vma_unbind
d0e5dee58521 drm/i915: Take object lock in i915_ggtt_pin if ww is not set
81200f4aea0f drm/i915: Remove pages_mutex and intel_gtt->vma_ops.set/clear_pages members
9a8f87820ed4 drm/i915: Remove resv from i915_vma
c7b50af89f7a Move CONTEXT_VALID_BIT check
92874150b4e7 drm/i915: Change shrink ordering to use locking around unbinding.
2885005de36d drm/i915: Fix runtime pm handling in i915_gem_shrink
c13839878262 drm/i915: vma is always backed by an object.
faee95e0ebdf drm/i915: Create a full object for mock_ring
7f62692b5277 drm/i915: Create a dummy object for gen6 ppgtt
ce526eaaf6dc drm/i915: Remove gen6_ppgtt_unpin_all
98c59acb1911 drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling
667b39786fe3 drm/i915: Remove unused bits of i915_vma/active api
fa0ce1cd930e drm/i915: Move __i915_gem_free_object to ttm_bo_destroy

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20917/index.html

[-- Attachment #2: Type: text/html, Size: 18515 bytes --]

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Tvrtko Ursulin]

On 30/08/2021 13:09, Maarten Lankhorst wrote:
> vma->obj and vma->resv are now never NULL, and some checks can be removed.

Is the direction here compatible with SVM / VM_BIND?

Regards,

Tvrtko

> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
>   drivers/gpu/drm/i915/gt/intel_context.c       |  2 +-
>   .../gpu/drm/i915/gt/intel_ring_submission.c   |  2 +-
>   drivers/gpu/drm/i915/i915_vma.c               | 48 ++++++++-----------
>   drivers/gpu/drm/i915/i915_vma.h               |  3 --
>   4 files changed, 22 insertions(+), 33 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_context.c b/drivers/gpu/drm/i915/gt/intel_context.c
> index 745e84c72c90..d3ad16df3ca2 100644
> --- a/drivers/gpu/drm/i915/gt/intel_context.c
> +++ b/drivers/gpu/drm/i915/gt/intel_context.c
> @@ -219,7 +219,7 @@ int __intel_context_do_pin_ww(struct intel_context *ce,
>   	 */
>   
>   	err = i915_gem_object_lock(ce->timeline->hwsp_ggtt->obj, ww);
> -	if (!err && ce->ring->vma->obj)
> +	if (!err)
>   		err = i915_gem_object_lock(ce->ring->vma->obj, ww);
>   	if (!err && ce->state)
>   		err = i915_gem_object_lock(ce->state->obj, ww);
> diff --git a/drivers/gpu/drm/i915/gt/intel_ring_submission.c b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> index 3c65efcb7bed..cc31ccc13bfb 100644
> --- a/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> +++ b/drivers/gpu/drm/i915/gt/intel_ring_submission.c
> @@ -1354,7 +1354,7 @@ int intel_ring_submission_setup(struct intel_engine_cs *engine)
>   	err = i915_gem_object_lock(timeline->hwsp_ggtt->obj, &ww);
>   	if (!err && gen7_wa_vma)
>   		err = i915_gem_object_lock(gen7_wa_vma->obj, &ww);
> -	if (!err && engine->legacy.ring->vma->obj)
> +	if (!err)
>   		err = i915_gem_object_lock(engine->legacy.ring->vma->obj, &ww);
>   	if (!err)
>   		err = intel_timeline_pin(timeline, &ww);
> diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
> index f9ac33e0bac9..ad5d52b33eb6 100644
> --- a/drivers/gpu/drm/i915/i915_vma.c
> +++ b/drivers/gpu/drm/i915/i915_vma.c
> @@ -40,12 +40,12 @@
>   
>   static struct kmem_cache *slab_vmas;
>   
> -struct i915_vma *i915_vma_alloc(void)
> +static struct i915_vma *i915_vma_alloc(void)
>   {
>   	return kmem_cache_zalloc(slab_vmas, GFP_KERNEL);
>   }
>   
> -void i915_vma_free(struct i915_vma *vma)
> +static void i915_vma_free(struct i915_vma *vma)
>   {
>   	return kmem_cache_free(slab_vmas, vma);
>   }
> @@ -426,10 +426,8 @@ int i915_vma_bind(struct i915_vma *vma,
>   
>   		work->base.dma.error = 0; /* enable the queue_work() */
>   
> -		if (vma->obj) {
> -			__i915_gem_object_pin_pages(vma->obj);
> -			work->pinned = i915_gem_object_get(vma->obj);
> -		}
> +		__i915_gem_object_pin_pages(vma->obj);
> +		work->pinned = i915_gem_object_get(vma->obj);
>   	} else {
>   		vma->ops->bind_vma(vma->vm, NULL, vma, cache_level, bind_flags);
>   	}
> @@ -670,7 +668,7 @@ i915_vma_insert(struct i915_vma *vma, u64 size, u64 alignment, u64 flags)
>   	}
>   
>   	color = 0;
> -	if (vma->obj && i915_vm_has_cache_coloring(vma->vm))
> +	if (i915_vm_has_cache_coloring(vma->vm))
>   		color = vma->obj->cache_level;
>   
>   	if (flags & PIN_OFFSET_FIXED) {
> @@ -795,17 +793,14 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
>   static int vma_get_pages(struct i915_vma *vma)
>   {
>   	int err = 0;
> -	bool pinned_pages = false;
> +	bool pinned_pages = true;
>   
>   	if (atomic_add_unless(&vma->pages_count, 1, 0))
>   		return 0;
>   
> -	if (vma->obj) {
> -		err = i915_gem_object_pin_pages(vma->obj);
> -		if (err)
> -			return err;
> -		pinned_pages = true;
> -	}
> +	err = i915_gem_object_pin_pages(vma->obj);
> +	if (err)
> +		return err;
>   
>   	/* Allocations ahoy! */
>   	if (mutex_lock_interruptible(&vma->pages_mutex)) {
> @@ -838,8 +833,8 @@ static void __vma_put_pages(struct i915_vma *vma, unsigned int count)
>   	if (atomic_sub_return(count, &vma->pages_count) == 0) {
>   		vma->ops->clear_pages(vma);
>   		GEM_BUG_ON(vma->pages);
> -		if (vma->obj)
> -			i915_gem_object_unpin_pages(vma->obj);
> +
> +		i915_gem_object_unpin_pages(vma->obj);
>   	}
>   	mutex_unlock(&vma->pages_mutex);
>   }
> @@ -875,7 +870,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   	int err;
>   
>   #ifdef CONFIG_PROVE_LOCKING
> -	if (debug_locks && !WARN_ON(!ww) && vma->resv)
> +	if (debug_locks && !WARN_ON(!ww))
>   		assert_vma_held(vma);
>   #endif
>   
> @@ -983,7 +978,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   
>   	GEM_BUG_ON(!vma->pages);
>   	err = i915_vma_bind(vma,
> -			    vma->obj ? vma->obj->cache_level : 0,
> +			    vma->obj->cache_level,
>   			    flags, work);
>   	if (err)
>   		goto err_remove;
> @@ -1037,7 +1032,7 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
>   
>   #ifdef CONFIG_LOCKDEP
> -	WARN_ON(!ww && vma->resv && dma_resv_held(vma->resv));
> +	WARN_ON(!ww && dma_resv_held(vma->resv));
>   #endif
>   
>   	do {
> @@ -1116,6 +1111,7 @@ void i915_vma_reopen(struct i915_vma *vma)
>   void i915_vma_release(struct kref *ref)
>   {
>   	struct i915_vma *vma = container_of(ref, typeof(*vma), ref);
> +	struct drm_i915_gem_object *obj = vma->obj;
>   
>   	if (drm_mm_node_allocated(&vma->node)) {
>   		mutex_lock(&vma->vm->mutex);
> @@ -1126,15 +1122,11 @@ void i915_vma_release(struct kref *ref)
>   	}
>   	GEM_BUG_ON(i915_vma_is_active(vma));
>   
> -	if (vma->obj) {
> -		struct drm_i915_gem_object *obj = vma->obj;
> -
> -		spin_lock(&obj->vma.lock);
> -		list_del(&vma->obj_link);
> -		if (!RB_EMPTY_NODE(&vma->obj_node))
> -			rb_erase(&vma->obj_node, &obj->vma.tree);
> -		spin_unlock(&obj->vma.lock);
> -	}
> +	spin_lock(&obj->vma.lock);
> +	list_del(&vma->obj_link);
> +	if (!RB_EMPTY_NODE(&vma->obj_node))
> +		rb_erase(&vma->obj_node, &obj->vma.tree);
> +	spin_unlock(&obj->vma.lock);
>   
>   	__i915_vma_remove_closed(vma);
>   	i915_vm_put(vma->vm);
> diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
> index 1c930515ec3d..1ba82bf863a5 100644
> --- a/drivers/gpu/drm/i915/i915_vma.h
> +++ b/drivers/gpu/drm/i915/i915_vma.h
> @@ -409,9 +409,6 @@ static inline void i915_vma_clear_scanout(struct i915_vma *vma)
>   	list_for_each_entry(V, &(OBJ)->vma.list, obj_link)		\
>   		for_each_until(!i915_vma_is_ggtt(V))
>   
> -struct i915_vma *i915_vma_alloc(void);
> -void i915_vma_free(struct i915_vma *vma);
> -
>   struct i915_vma *i915_vma_make_unshrinkable(struct i915_vma *vma);
>   void i915_vma_make_shrinkable(struct i915_vma *vma);
>   void i915_vma_make_purgeable(struct i915_vma *vma);
> 

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Maarten Lankhorst]

Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>
> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>
> Is the direction here compatible with SVM / VM_BIND? 


Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.

I also have some patches on top to remove i915_vma->active, which was 1 of the requirements for VM_BIND iirc.

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Tvrtko Ursulin]

On 31/08/2021 10:34, Maarten Lankhorst wrote:
> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>
>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>>
>> Is the direction here compatible with SVM / VM_BIND?
> 
> 
> Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.

Hm but what will vma->obj point to in case of SVM, when there is no GEM BO?

Regards,

Tvrtko

> 
> I also have some patches on top to remove i915_vma->active, which was 1 of the requirements for VM_BIND iirc.
> 

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Maarten Lankhorst]

Op 31-08-2021 om 12:29 schreef Tvrtko Ursulin:
>
> On 31/08/2021 10:34, Maarten Lankhorst wrote:
>> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>>
>>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>>>
>>> Is the direction here compatible with SVM / VM_BIND?
>>
>>
>> Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.
>
> Hm but what will vma->obj point to in case of SVM, when there is no GEM BO? 

Probably to one of the bo's in i915_vm, or a dummy bo that least shares the vm_resv object, similar to the aliasing gtt handling.

Re: [Intel-gfx] [PATCH 14/19] drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for i915_vma_unbind

[Maarten Lankhorst]

Op 30-08-2021 om 14:10 schreef Maarten Lankhorst:
> We want to remove more members of i915_vma, which requires the locking to be
> held more often.
>
> Start requiring gem object lock for i915_vma_unbind, as it's one of the
> callers that may unpin pages.
>
> Some special care is needed when evicting, because the last reference to the
> object may be held by the VMA, so after __i915_vma_unbind, vma may be garbage,
> and we need to cache vma->obj before unlocking.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
>  drivers/gpu/drm/i915/display/intel_display.c  |  2 +-
>  drivers/gpu/drm/i915/gem/i915_gem_shrinker.c  | 14 +++-
>  .../gpu/drm/i915/gem/selftests/huge_pages.c   |  4 +-
>  .../i915/gem/selftests/i915_gem_client_blt.c  |  2 +-
>  .../drm/i915/gem/selftests/i915_gem_mman.c    |  6 ++
>  drivers/gpu/drm/i915/gt/intel_ggtt.c          | 46 ++++++++++--
>  drivers/gpu/drm/i915/i915_drv.h               |  7 +-
>  drivers/gpu/drm/i915/i915_gem.c               | 29 +++++++-
>  drivers/gpu/drm/i915/i915_gem_evict.c         | 74 +++++++++++++++++--
>  drivers/gpu/drm/i915/i915_vma.c               | 27 ++++++-
>  drivers/gpu/drm/i915/i915_vma.h               |  1 +
>  drivers/gpu/drm/i915/selftests/i915_gem_gtt.c | 22 +++---
>  drivers/gpu/drm/i915/selftests/i915_vma.c     |  2 +-
>  13 files changed, 195 insertions(+), 41 deletions(-)

This patch breaks the selftests because lockdep doesn't like nested trylocks, and eviction needed some love.

I've split it out, and will resubmit this part, rest can still be reviewed as it doesn't affect that.

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Tvrtko Ursulin]

On 03/09/2021 10:31, Maarten Lankhorst wrote:
> Op 31-08-2021 om 12:29 schreef Tvrtko Ursulin:
>>
>> On 31/08/2021 10:34, Maarten Lankhorst wrote:
>>> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>>>
>>>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>>>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>>>>
>>>> Is the direction here compatible with SVM / VM_BIND?
>>>
>>>
>>> Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.
>>
>> Hm but what will vma->obj point to in case of SVM, when there is no GEM BO?
> 
> Probably to one of the bo's in i915_vm, or a dummy bo that least shares the vm_resv object, similar to the aliasing gtt handling.

As a long term or short term solution? Worried that would waste a lot of 
SLAB space just for convenience (whole struct drm_i915_gem_object just 
to store a single pointer to a dma_resv object, if I got that right), 
while it should be possible to come up with a cleaner design.

Even for the upcoming page granularity work we will need multiple VMAs 
point to single GEM bo in ppgtt and that, when SVM is considered, could 
for instance mean that VMAs should instead be backed by some new backing 
store objects, which in turn may (or may not) point to GEM BOs.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 09/19] drm/i915: Change shrink ordering to use locking around unbinding.

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:56PM +0200, Maarten Lankhorst wrote:
>Call drop_pages with the gem object lock held, instead of the other
>way around. This will allow us to drop the vma bindings with the
>gem object lock held.
>
>We plan to require the object lock for unpinning in the future,
>and this is an easy target.
>
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

Looks good to me, we need this for upcoming vm_bind as well.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>---
> drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 42 ++++++++++----------
> 1 file changed, 21 insertions(+), 21 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>index 5ab136ffdeb2..7f7849b6296d 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>@@ -37,8 +37,8 @@ static bool can_release_pages(struct drm_i915_gem_object *obj)
> 	return swap_available() || obj->mm.madv == I915_MADV_DONTNEED;
> }
>
>-static bool unsafe_drop_pages(struct drm_i915_gem_object *obj,
>-			      unsigned long shrink, bool trylock_vm)
>+static int drop_pages(struct drm_i915_gem_object *obj,
>+		       unsigned long shrink, bool trylock_vm)
> {
> 	unsigned long flags;
>
>@@ -209,27 +209,27 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
>
> 			spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
>
>-			err = 0;
>-			if (unsafe_drop_pages(obj, shrink, trylock_vm)) {
>-				/* May arrive from get_pages on another bo */
>-				if (!ww) {
>-					if (!i915_gem_object_trylock(obj))
>-						goto skip;
>-				} else {
>-					err = i915_gem_object_lock(obj, ww);
>-					if (err)
>-						goto skip;
>-				}
>-
>-				if (!__i915_gem_object_put_pages(obj)) {
>-					try_to_writeback(obj, shrink);
>-					count += obj->base.size >> PAGE_SHIFT;
>-				}
>-				if (!ww)
>-					i915_gem_object_unlock(obj);
>+			/* May arrive from get_pages on another bo */
>+			if (!ww) {
>+				if (!i915_gem_object_trylock(obj))
>+					goto skip;
>+			} else {
>+				err = i915_gem_object_lock(obj, ww);
>+				if (err)
>+					goto skip;
> 			}
>
>-			dma_resv_prune(obj->base.resv);
>+			if (drop_pages(obj, shrink, trylock_vm) &&
>+			    !__i915_gem_object_put_pages(obj)) {
>+				try_to_writeback(obj, shrink);
>+				count += obj->base.size >> PAGE_SHIFT;
>+			}
>+
>+			if (dma_resv_test_signaled(obj->base.resv, true))
>+				dma_resv_add_excl_fence(obj->base.resv, NULL);
>+
>+			if (!ww)
>+				i915_gem_object_unlock(obj);
>
> 			scanned += obj->base.size >> PAGE_SHIFT;
> skip:
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 11/19] drm/i915: Remove resv from i915_vma

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:58PM +0200, Maarten Lankhorst wrote:
>It's just an alias to vma->obj->base.resv, no need to duplicate it.

Looks good to me.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>---
> drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c | 4 ++--
> drivers/gpu/drm/i915/i915_vma.c                | 9 ++++-----
> drivers/gpu/drm/i915/i915_vma.h                | 6 +++---
> drivers/gpu/drm/i915/i915_vma_types.h          | 1 -
> 4 files changed, 9 insertions(+), 11 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>index 4f10a81befd1..6832b255294e 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>@@ -975,7 +975,7 @@ static int eb_validate_vmas(struct i915_execbuffer *eb)
> 		}
>
> 		if (!(ev->flags & EXEC_OBJECT_WRITE)) {
>-			err = dma_resv_reserve_shared(vma->resv, 1);
>+			err = dma_resv_reserve_shared(vma->obj->base.resv, 1);
> 			if (err)
> 				return err;
> 		}
>@@ -2121,7 +2121,7 @@ static int eb_parse(struct i915_execbuffer *eb)
> 		goto err_trampoline;
> 	}
>
>-	err = dma_resv_reserve_shared(shadow->resv, 1);
>+	err = dma_resv_reserve_shared(shadow->obj->base.resv, 1);
> 	if (err)
> 		goto err_trampoline;
>
>diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
>index ad5d52b33eb6..36db641e863e 100644
>--- a/drivers/gpu/drm/i915/i915_vma.c
>+++ b/drivers/gpu/drm/i915/i915_vma.c
>@@ -116,7 +116,6 @@ vma_create(struct drm_i915_gem_object *obj,
> 	vma->vm = i915_vm_get(vm);
> 	vma->ops = &vm->vma_ops;
> 	vma->obj = obj;
>-	vma->resv = obj->base.resv;
> 	vma->size = obj->base.size;
> 	vma->display_alignment = I915_GTT_MIN_ALIGNMENT;
>
>@@ -1032,7 +1031,7 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
> 	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
>
> #ifdef CONFIG_LOCKDEP
>-	WARN_ON(!ww && dma_resv_held(vma->resv));
>+	WARN_ON(!ww && dma_resv_held(vma->obj->base.resv));
> #endif
>
> 	do {
>@@ -1249,17 +1248,17 @@ int i915_vma_move_to_active(struct i915_vma *vma,
> 			intel_frontbuffer_put(front);
> 		}
>
>-		dma_resv_add_excl_fence(vma->resv, &rq->fence);
>+		dma_resv_add_excl_fence(vma->obj->base.resv, &rq->fence);
> 		obj->write_domain = I915_GEM_DOMAIN_RENDER;
> 		obj->read_domains = 0;
> 	} else {
> 		if (!(flags & __EXEC_OBJECT_NO_RESERVE)) {
>-			err = dma_resv_reserve_shared(vma->resv, 1);
>+			err = dma_resv_reserve_shared(vma->obj->base.resv, 1);
> 			if (unlikely(err))
> 				return err;
> 		}
>
>-		dma_resv_add_shared_fence(vma->resv, &rq->fence);
>+		dma_resv_add_shared_fence(vma->obj->base.resv, &rq->fence);
> 		obj->write_domain = 0;
> 	}
>
>diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
>index 1ba82bf863a5..0b11c188f749 100644
>--- a/drivers/gpu/drm/i915/i915_vma.h
>+++ b/drivers/gpu/drm/i915/i915_vma.h
>@@ -225,16 +225,16 @@ static inline void __i915_vma_put(struct i915_vma *vma)
> 	kref_put(&vma->ref, i915_vma_release);
> }
>
>-#define assert_vma_held(vma) dma_resv_assert_held((vma)->resv)
>+#define assert_vma_held(vma) dma_resv_assert_held((vma)->obj->base.resv)
>
> static inline void i915_vma_lock(struct i915_vma *vma)
> {
>-	dma_resv_lock(vma->resv, NULL);
>+	dma_resv_lock(vma->obj->base.resv, NULL);
> }
>
> static inline void i915_vma_unlock(struct i915_vma *vma)
> {
>-	dma_resv_unlock(vma->resv);
>+	dma_resv_unlock(vma->obj->base.resv);
> }
>
> int __must_check
>diff --git a/drivers/gpu/drm/i915/i915_vma_types.h b/drivers/gpu/drm/i915/i915_vma_types.h
>index 995b502d7e5d..47573ed2d11f 100644
>--- a/drivers/gpu/drm/i915/i915_vma_types.h
>+++ b/drivers/gpu/drm/i915/i915_vma_types.h
>@@ -177,7 +177,6 @@ struct i915_vma {
> 	const struct i915_vma_ops *ops;
>
> 	struct drm_i915_gem_object *obj;
>-	struct dma_resv *resv; /** Alias of obj->resv */
>
> 	struct sg_table *pages;
> 	void __iomem *iomap;
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:55PM +0200, Maarten Lankhorst wrote:
>We forgot to call intel_runtime_pm_put on error, fix it!
>

Looks good to me.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
>Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
>Cc: <stable@vger.kernel.org> # v5.13+
>---
> drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 7 +++++--
> 1 file changed, 5 insertions(+), 2 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>index e382b7f2353b..5ab136ffdeb2 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
>@@ -118,7 +118,7 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
> 	intel_wakeref_t wakeref = 0;
> 	unsigned long count = 0;
> 	unsigned long scanned = 0;
>-	int err;
>+	int err = 0;
>
> 	/* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
> 	bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
>@@ -242,12 +242,15 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
> 		list_splice_tail(&still_in_list, phase->list);
> 		spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
> 		if (err)
>-			return err;
>+			break;
> 	}
>
> 	if (shrink & I915_SHRINK_BOUND)
> 		intel_runtime_pm_put(&i915->runtime_pm, wakeref);
>
>+	if (err)
>+		return err;
>+
> 	if (nr_scanned)
> 		*nr_scanned += scanned;
> 	return count;
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 02/19] drm/i915: Remove unused bits of i915_vma/active api

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:49PM +0200, Maarten Lankhorst wrote:
>When reworking the code to move the eviction fence to the object,
>the best code is removed code.
>
>Remove some functions that are unused, and change the function definition
>if it's only used in 1 place.
>
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

Looks good to me.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>---
> drivers/gpu/drm/i915/i915_active.c | 28 +++-------------------------
> drivers/gpu/drm/i915/i915_active.h | 17 +----------------
> drivers/gpu/drm/i915/i915_vma.c    |  2 +-
> drivers/gpu/drm/i915/i915_vma.h    |  2 --
> 4 files changed, 5 insertions(+), 44 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c
>index 3103c1e1fd14..ee2b3a375362 100644
>--- a/drivers/gpu/drm/i915/i915_active.c
>+++ b/drivers/gpu/drm/i915/i915_active.c
>@@ -426,8 +426,9 @@ replace_barrier(struct i915_active *ref, struct i915_active_fence *active)
> 	return true;
> }
>
>-int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
>+int i915_active_add_request(struct i915_active *ref, struct i915_request *rq)
> {
>+	struct dma_fence *fence = &rq->fence;
> 	struct i915_active_fence *active;
> 	int err;
>
>@@ -436,7 +437,7 @@ int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
> 	if (err)
> 		return err;
>
>-	active = active_instance(ref, idx);
>+	active = active_instance(ref, i915_request_timeline(rq)->fence_context);
> 	if (!active) {
> 		err = -ENOMEM;
> 		goto out;
>@@ -477,29 +478,6 @@ __i915_active_set_fence(struct i915_active *ref,
> 	return prev;
> }
>
>-static struct i915_active_fence *
>-__active_fence(struct i915_active *ref, u64 idx)
>-{
>-	struct active_node *it;
>-
>-	it = __active_lookup(ref, idx);
>-	if (unlikely(!it)) { /* Contention with parallel tree builders! */
>-		spin_lock_irq(&ref->tree_lock);
>-		it = __active_lookup(ref, idx);
>-		spin_unlock_irq(&ref->tree_lock);
>-	}
>-	GEM_BUG_ON(!it); /* slot must be preallocated */
>-
>-	return &it->base;
>-}
>-
>-struct dma_fence *
>-__i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence)
>-{
>-	/* Only valid while active, see i915_active_acquire_for_context() */
>-	return __i915_active_set_fence(ref, __active_fence(ref, idx), fence);
>-}
>-
> struct dma_fence *
> i915_active_set_exclusive(struct i915_active *ref, struct dma_fence *f)
> {
>diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h
>index 5fcdb0e2bc9e..7eb44132183a 100644
>--- a/drivers/gpu/drm/i915/i915_active.h
>+++ b/drivers/gpu/drm/i915/i915_active.h
>@@ -164,26 +164,11 @@ void __i915_active_init(struct i915_active *ref,
> 	__i915_active_init(ref, active, retire, flags, &__mkey, &__wkey);	\
> } while (0)
>
>-struct dma_fence *
>-__i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence);
>-int i915_active_ref(struct i915_active *ref, u64 idx, struct dma_fence *fence);
>-
>-static inline int
>-i915_active_add_request(struct i915_active *ref, struct i915_request *rq)
>-{
>-	return i915_active_ref(ref,
>-			       i915_request_timeline(rq)->fence_context,
>-			       &rq->fence);
>-}
>+int i915_active_add_request(struct i915_active *ref, struct i915_request *rq);
>
> struct dma_fence *
> i915_active_set_exclusive(struct i915_active *ref, struct dma_fence *f);
>
>-static inline bool i915_active_has_exclusive(struct i915_active *ref)
>-{
>-	return rcu_access_pointer(ref->excl.fence);
>-}
>-
> int __i915_active_wait(struct i915_active *ref, int state);
> static inline int i915_active_wait(struct i915_active *ref)
> {
>diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
>index 4b7fc4647e46..f9ac33e0bac9 100644
>--- a/drivers/gpu/drm/i915/i915_vma.c
>+++ b/drivers/gpu/drm/i915/i915_vma.c
>@@ -1220,7 +1220,7 @@ __i915_request_await_bind(struct i915_request *rq, struct i915_vma *vma)
> 	return __i915_request_await_exclusive(rq, &vma->active);
> }
>
>-int __i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *rq)
>+static int __i915_vma_move_to_active(struct i915_vma *vma, struct i915_request *rq)
> {
> 	int err;
>
>diff --git a/drivers/gpu/drm/i915/i915_vma.h b/drivers/gpu/drm/i915/i915_vma.h
>index ed69f66c7ab0..1c930515ec3d 100644
>--- a/drivers/gpu/drm/i915/i915_vma.h
>+++ b/drivers/gpu/drm/i915/i915_vma.h
>@@ -55,8 +55,6 @@ static inline bool i915_vma_is_active(const struct i915_vma *vma)
> /* do not reserve memory to prevent deadlocks */
> #define __EXEC_OBJECT_NO_RESERVE BIT(31)
>
>-int __must_check __i915_vma_move_to_active(struct i915_vma *vma,
>-					   struct i915_request *rq);
> int __must_check i915_vma_move_to_active(struct i915_vma *vma,
> 					 struct i915_request *rq,
> 					 unsigned int flags);
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 03/19] drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:50PM +0200, Maarten Lankhorst wrote:
>Use a single null-terminated array for simplicity instead of a linked
>list. This might slightly speed up execbuf when many vma's may be marked
>as capture, but definitely removes an allocation from a signaling path.
>
>We are not allowed to allocate memory in eb_move_to_gpu, but we can't
>enforce it yet through annotations.
>
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

Looks good to me.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>---
> .../gpu/drm/i915/gem/i915_gem_execbuffer.c    | 26 ++++++++++++-------
> drivers/gpu/drm/i915/i915_gpu_error.c         |  9 ++++---
> drivers/gpu/drm/i915/i915_request.c           |  9 ++-----
> drivers/gpu/drm/i915/i915_request.h           |  7 +----
> 4 files changed, 24 insertions(+), 27 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>index 8290bdadd167..4f10a81befd1 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
>@@ -255,6 +255,9 @@ struct i915_execbuffer {
> 	/** actual size of execobj[] as we may extend it for the cmdparser */
> 	unsigned int buffer_count;
>
>+	/* Number of objects with EXEC_OBJECT_CAPTURE set */
>+	unsigned int capture_count;
>+
> 	/** list of vma not yet bound during reservation phase */
> 	struct list_head unbound;
>
>@@ -859,6 +862,9 @@ static int eb_lookup_vmas(struct i915_execbuffer *eb)
> 			goto err;
> 		}
>
>+		if (eb->exec[i].flags & EXEC_OBJECT_CAPTURE)
>+			eb->capture_count++;
>+
> 		err = eb_validate_vma(eb, &eb->exec[i], vma);
> 		if (unlikely(err)) {
> 			i915_vma_put(vma);
>@@ -1890,16 +1896,8 @@ static int eb_move_to_gpu(struct i915_execbuffer *eb)
>
> 		assert_vma_held(vma);
>
>-		if (flags & EXEC_OBJECT_CAPTURE) {
>-			struct i915_capture_list *capture;
>-
>-			capture = kmalloc(sizeof(*capture), GFP_KERNEL);
>-			if (capture) {
>-				capture->next = eb->request->capture_list;
>-				capture->vma = vma;
>-				eb->request->capture_list = capture;
>-			}
>-		}
>+		if (flags & EXEC_OBJECT_CAPTURE && eb->request->capture_list)
>+			eb->request->capture_list[--eb->capture_count] = vma;
>
> 		/*
> 		 * If the GPU is not _reading_ through the CPU cache, we need
>@@ -2828,6 +2826,7 @@ i915_gem_do_execbuffer(struct drm_device *dev,
>
> 	eb.fences = NULL;
> 	eb.num_fences = 0;
>+	eb.capture_count = 0;
>
> 	eb.batch_flags = 0;
> 	if (args->flags & I915_EXEC_SECURE) {
>@@ -2956,6 +2955,13 @@ i915_gem_do_execbuffer(struct drm_device *dev,
> 		}
> 	}
>
>+	if (eb.capture_count) {
>+		eb.request->capture_list =
>+			kvcalloc(eb.capture_count + 1,
>+				 sizeof(*eb.request->capture_list),
>+				 GFP_KERNEL | __GFP_NOWARN);
>+	}
>+
> 	/*
> 	 * Whilst this request exists, batch_obj will be on the
> 	 * active_list, and so will hold the active reference. Only when this
>diff --git a/drivers/gpu/drm/i915/i915_gpu_error.c b/drivers/gpu/drm/i915/i915_gpu_error.c
>index b9f66dbd46bb..44d07923c8d6 100644
>--- a/drivers/gpu/drm/i915/i915_gpu_error.c
>+++ b/drivers/gpu/drm/i915/i915_gpu_error.c
>@@ -1355,10 +1355,10 @@ capture_user(struct intel_engine_capture_vma *capture,
> 	     const struct i915_request *rq,
> 	     gfp_t gfp)
> {
>-	struct i915_capture_list *c;
>+	int i;
>
>-	for (c = rq->capture_list; c; c = c->next)
>-		capture = capture_vma(capture, c->vma, "user", gfp);
>+	for (i = 0; rq->capture_list[i]; i++)
>+		capture = capture_vma(capture, rq->capture_list[i], "user", gfp);
>
> 	return capture;
> }
>@@ -1406,7 +1406,8 @@ intel_engine_coredump_add_request(struct intel_engine_coredump *ee,
> 	 * by userspace.
> 	 */
> 	vma = capture_vma(vma, rq->batch, "batch", gfp);
>-	vma = capture_user(vma, rq, gfp);
>+	if (rq->capture_list)
>+		vma = capture_user(vma, rq, gfp);
> 	vma = capture_vma(vma, rq->ring->vma, "ring", gfp);
> 	vma = capture_vma(vma, rq->context->state, "HW context", gfp);
>
>diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
>index ce446716d092..4fca2722891c 100644
>--- a/drivers/gpu/drm/i915/i915_request.c
>+++ b/drivers/gpu/drm/i915/i915_request.c
>@@ -188,15 +188,10 @@ void i915_request_notify_execute_cb_imm(struct i915_request *rq)
>
> static void free_capture_list(struct i915_request *request)
> {
>-	struct i915_capture_list *capture;
>+	struct i915_vma **capture;
>
> 	capture = fetch_and_zero(&request->capture_list);
>-	while (capture) {
>-		struct i915_capture_list *next = capture->next;
>-
>-		kfree(capture);
>-		capture = next;
>-	}
>+	kvfree(capture);
> }
>
> static void __i915_request_fill(struct i915_request *rq, u8 val)
>diff --git a/drivers/gpu/drm/i915/i915_request.h b/drivers/gpu/drm/i915/i915_request.h
>index 1bc1349ba3c2..e7c8cacefcca 100644
>--- a/drivers/gpu/drm/i915/i915_request.h
>+++ b/drivers/gpu/drm/i915/i915_request.h
>@@ -48,11 +48,6 @@ struct drm_i915_gem_object;
> struct drm_printer;
> struct i915_request;
>
>-struct i915_capture_list {
>-	struct i915_capture_list *next;
>-	struct i915_vma *vma;
>-};
>-
> #define RQ_TRACE(rq, fmt, ...) do {					\
> 	const struct i915_request *rq__ = (rq);				\
> 	ENGINE_TRACE(rq__->engine, "fence %llx:%lld, current %d " fmt,	\
>@@ -271,7 +266,7 @@ struct i915_request {
> 	 * active reference - all objects on this list must also be
> 	 * on the active_list (of their final request).
> 	 */
>-	struct i915_capture_list *capture_list;
>+	struct i915_vma **capture_list;
>
> 	/** Time at which this request was emitted, in jiffies. */
> 	unsigned long emitted_jiffies;
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 13/19] drm/i915: Take object lock in i915_ggtt_pin if ww is not set

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:10:00PM +0200, Maarten Lankhorst wrote:
>i915_vma_wait_for_bind needs the vma lock held, fix the caller.
>

Need an lock assert in i915_vma_wait_for_bind()?

Niranjana

>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>---
> drivers/gpu/drm/i915/i915_vma.c | 40 +++++++++++++++++++++++----------
> 1 file changed, 28 insertions(+), 12 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
>index 5e0eaa9ea890..77319619dba0 100644
>--- a/drivers/gpu/drm/i915/i915_vma.c
>+++ b/drivers/gpu/drm/i915/i915_vma.c
>@@ -1322,23 +1322,15 @@ static void flush_idle_contexts(struct intel_gt *gt)
> 	intel_gt_wait_for_idle(gt, MAX_SCHEDULE_TIMEOUT);
> }
>
>-int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>-		  u32 align, unsigned int flags)
>+static int __i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>+			   u32 align, unsigned int flags)
> {
> 	struct i915_address_space *vm = vma->vm;
> 	int err;
>
>-	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
>-
>-#ifdef CONFIG_LOCKDEP
>-	WARN_ON(!ww && dma_resv_held(vma->obj->base.resv));
>-#endif
>-
> 	do {
>-		if (ww)
>-			err = i915_vma_pin_ww(vma, ww, 0, align, flags | PIN_GLOBAL);
>-		else
>-			err = i915_vma_pin(vma, 0, align, flags | PIN_GLOBAL);
>+		err = i915_vma_pin_ww(vma, ww, 0, align, flags | PIN_GLOBAL);
>+
> 		if (err != -ENOSPC) {
> 			if (!err) {
> 				err = i915_vma_wait_for_bind(vma);
>@@ -1357,6 +1349,30 @@ int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
> 	} while (1);
> }
>
>+int i915_ggtt_pin(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>+		  u32 align, unsigned int flags)
>+{
>+	struct i915_gem_ww_ctx _ww;
>+	int err;
>+
>+	GEM_BUG_ON(!i915_vma_is_ggtt(vma));
>+
>+	if (ww)
>+		return __i915_ggtt_pin(vma, ww, align, flags);
>+
>+#ifdef CONFIG_LOCKDEP
>+	WARN_ON(dma_resv_held(vma->obj->base.resv));
>+#endif
>+
>+	for_i915_gem_ww(&_ww, err, true) {
>+		err = i915_gem_object_lock(vma->obj, &_ww);
>+		if (!err)
>+			err = __i915_ggtt_pin(vma, &_ww, align, flags);
>+	}
>+
>+	return err;
>+}
>+
> static void __vma_close(struct i915_vma *vma, struct intel_gt *gt)
> {
> 	/*
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 15/19] drm/i915: Remove support for unlocked i915_vma unbind

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:10:02PM +0200, Maarten Lankhorst wrote:
>Now that we require the object lock for all ops, some code handling
>race conditions can be removed.
>
>This is required to not take short-term pins inside execbuf.
>

Ok I get it, as i915_vma_unbind() is now called uner obj lock
we don't need these race handling. But would like someone else
also take a look.

Acked-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>---
> drivers/gpu/drm/i915/i915_vma.c | 40 +++++----------------------------
> 1 file changed, 5 insertions(+), 35 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
>index da54e6882650..f6dacfc3e840 100644
>--- a/drivers/gpu/drm/i915/i915_vma.c
>+++ b/drivers/gpu/drm/i915/i915_vma.c
>@@ -748,7 +748,6 @@ i915_vma_detach(struct i915_vma *vma)
> static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
> {
> 	unsigned int bound;
>-	bool pinned = true;
>
> 	bound = atomic_read(&vma->flags);
> 	do {
>@@ -758,34 +757,10 @@ static bool try_qad_pin(struct i915_vma *vma, unsigned int flags)
> 		if (unlikely(bound & (I915_VMA_OVERFLOW | I915_VMA_ERROR)))
> 			return false;
>
>-		if (!(bound & I915_VMA_PIN_MASK))
>-			goto unpinned;
>-
> 		GEM_BUG_ON(((bound + 1) & I915_VMA_PIN_MASK) == 0);
> 	} while (!atomic_try_cmpxchg(&vma->flags, &bound, bound + 1));
>
> 	return true;
>-
>-unpinned:
>-	/*
>-	 * If pin_count==0, but we are bound, check under the lock to avoid
>-	 * racing with a concurrent i915_vma_unbind().
>-	 */
>-	mutex_lock(&vma->vm->mutex);
>-	do {
>-		if (unlikely(bound & (I915_VMA_OVERFLOW | I915_VMA_ERROR))) {
>-			pinned = false;
>-			break;
>-		}
>-
>-		if (unlikely(flags & ~bound)) {
>-			pinned = false;
>-			break;
>-		}
>-	} while (!atomic_try_cmpxchg(&vma->flags, &bound, bound + 1));
>-	mutex_unlock(&vma->vm->mutex);
>-
>-	return pinned;
> }
>
>
>@@ -1085,13 +1060,7 @@ __i915_vma_get_pages(struct i915_vma *vma)
> 			vma->ggtt_view.type, ret);
> 	}
>
>-	pages = xchg(&vma->pages, pages);
>-
>-	/* did we race against a put_pages? */
>-	if (pages && pages != vma->obj->mm.pages) {
>-		sg_free_table(vma->pages);
>-		kfree(vma->pages);
>-	}
>+	vma->pages = pages;
>
> 	return ret;
> }
>@@ -1125,13 +1094,14 @@ I915_SELFTEST_EXPORT int i915_vma_get_pages(struct i915_vma *vma)
> static void __vma_put_pages(struct i915_vma *vma, unsigned int count)
> {
> 	/* We allocate under vma_get_pages, so beware the shrinker */
>-	struct sg_table *pages = READ_ONCE(vma->pages);
>+	struct sg_table *pages = vma->pages;
>
> 	GEM_BUG_ON(atomic_read(&vma->pages_count) < count);
>
> 	if (atomic_sub_return(count, &vma->pages_count) == 0) {
>-		if (pages == cmpxchg(&vma->pages, pages, NULL) &&
>-		    pages != vma->obj->mm.pages) {
>+		vma->pages = NULL;
>+
>+		if (pages != vma->obj->mm.pages) {
> 			sg_free_table(pages);
> 			kfree(pages);
> 		}
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 17/19] drm/i915: Add functions to set/get moving fence

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:10:04PM +0200, Maarten Lankhorst wrote:
>We want to get rid of i915_vma tracking to simplify the code and
>lifetimes. Add a way to set/put the moving fence, in preparation for
>removing the tracking.
>
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>---
> drivers/gpu/drm/i915/gem/i915_gem_object.c | 15 +++++++++++++++
> drivers/gpu/drm/i915/gem/i915_gem_object.h |  6 ++++++
> 2 files changed, 21 insertions(+)
>
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
>index 6fb9afb65034..dc0d2da297a0 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
>@@ -32,6 +32,7 @@
> #include "i915_gem_object.h"
> #include "i915_memcpy.h"
> #include "i915_trace.h"
>+#include "i915_gem_ttm.h"
>
> static struct kmem_cache *slab_objects;
>
>@@ -674,6 +675,20 @@ static const struct drm_gem_object_funcs i915_gem_object_funcs = {
> 	.export = i915_gem_prime_export,
> };
>
>+struct dma_fence *
>+i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj)
>+{
>+	return dma_fence_get(i915_gem_to_ttm(obj)->moving);
>+}
>+
>+void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
>+				       struct dma_fence *fence)
>+{
>+	dma_fence_put(i915_gem_to_ttm(obj)->moving);
>+
>+	i915_gem_to_ttm(obj)->moving = dma_fence_get(fence);
>+}

When is i915_gem_object_set_moving_fence() called?
I am not seeing it being called in this patch series.

Niranjana

>+
> #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
> #include "selftests/huge_gem_object.c"
> #include "selftests/huge_pages.c"
>diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>index 48112b9d76df..a23acfa98e21 100644
>--- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
>+++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
>@@ -520,6 +520,12 @@ i915_gem_object_last_write_engine(struct drm_i915_gem_object *obj)
> 	return engine;
> }
>
>+struct dma_fence *
>+i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj);
>+
>+void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
>+				       struct dma_fence *fence);
>+
> void i915_gem_object_set_cache_coherency(struct drm_i915_gem_object *obj,
> 					 unsigned int cache_level);
> void i915_gem_object_flush_if_display(struct drm_i915_gem_object *obj);
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 10/19] Move CONTEXT_VALID_BIT check

[Niranjana Vishwanathapura]

On Mon, Aug 30, 2021 at 02:09:57PM +0200, Maarten Lankhorst wrote:
>Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>---
> drivers/gpu/drm/i915/gt/intel_engine_pm.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
>diff --git a/drivers/gpu/drm/i915/gt/intel_engine_pm.c b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
>index 1f07ac4e0672..df81a0dc481e 100644
>--- a/drivers/gpu/drm/i915/gt/intel_engine_pm.c
>+++ b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
>@@ -52,8 +52,6 @@ static int __engine_unpark(struct intel_wakeref *wf)
> 	/* Discard stale context state from across idling */
> 	ce = engine->kernel_context;
> 	if (ce) {
>-		GEM_BUG_ON(test_bit(CONTEXT_VALID_BIT, &ce->flags));
>-
> 		/* Flush all pending HW writes before we touch the context */
> 		while (unlikely(intel_context_inflight(ce)))
> 			intel_engine_flush_submission(engine);
>@@ -68,6 +66,9 @@ static int __engine_unpark(struct intel_wakeref *wf)
> 			 ce->timeline->seqno,
> 			 READ_ONCE(*ce->timeline->hwsp_seqno),
> 			 ce->ring->emit);
>+
>+		GEM_BUG_ON(test_bit(CONTEXT_VALID_BIT, &ce->flags));
>+

Looks good to me.
Reviewed-by: Niranjana Vishwanathapura <niranjana.vishwanathapura@intel.com>

> 		GEM_BUG_ON(ce->timeline->seqno !=
> 			   READ_ONCE(*ce->timeline->hwsp_seqno));
> 	}
>-- 
>2.32.0
>

Re: [Intel-gfx] [PATCH 00/19] drm/i915: Short-term pinning and async eviction.

[Thomas Hellström (Intel)]

On 8/30/21 2:09 PM, Maarten Lankhorst wrote:
> Remove some parts of the i915_vma api, ensure obj->vma always exists,
> and finally force the object lock to be taken when calling i915_vma_unbind
> is called.

Should this be vma->obj?



>
> With this, locking is a lot cleaner, and we no longer need all the if (!obj->vma) checks.
> We kill off the locking around i915_vma->set/get pages, and kill off the short-term
> pinning/unpinning in execbuf.
>
> After this, we go 1 step further, populate obj->moving where required, and add support
> for async eviction/clearing.

Does the async eviction / clearing depend on the vma cleanup? If not can 
we move that into a separate series?

Thanks,

Thomas



>
> Maarten Lankhorst (19):
>    drm/i915: Move __i915_gem_free_object to ttm_bo_destroy
>    drm/i915: Remove unused bits of i915_vma/active api
>    drm/i915: Slightly rework EXEC_OBJECT_CAPTURE handling
>    drm/i915: Remove gen6_ppgtt_unpin_all
>    drm/i915: Create a dummy object for gen6 ppgtt
>    drm/i915: Create a full object for mock_ring
>    drm/i915: vma is always backed by an object.
>    drm/i915: Fix runtime pm handling in i915_gem_shrink
>    drm/i915: Change shrink ordering to use locking around unbinding.
>    Move CONTEXT_VALID_BIT check
>    drm/i915: Remove resv from i915_vma
>    drm/i915: Remove pages_mutex and intel_gtt->vma_ops.set/clear_pages
>      members
>    drm/i915: Take object lock in i915_ggtt_pin if ww is not set
>    drm/i915: Add i915_vma_unbind_unlocked, and take obj lock for
>      i915_vma_unbind
>    drm/i915: Remove support for unlocked i915_vma unbind
>    drm/i915: Remove short-term pins from execbuf
>    drm/i915: Add functions to set/get moving fence
>    drm/i915: Add support for asynchronous moving fence waiting
>    drm/i915: Add accelerated migration to ttm
>
>   drivers/gpu/drm/i915/display/intel_display.c  |   2 +-
>   drivers/gpu/drm/i915/display/intel_dpt.c      |   2 -
>   .../gpu/drm/i915/gem/i915_gem_execbuffer.c    |  94 ++--
>   drivers/gpu/drm/i915/gem/i915_gem_internal.c  |  44 +-
>   drivers/gpu/drm/i915/gem/i915_gem_object.c    |  15 +
>   drivers/gpu/drm/i915/gem/i915_gem_object.h    |   6 +
>   drivers/gpu/drm/i915/gem/i915_gem_pages.c     |  10 +
>   drivers/gpu/drm/i915/gem/i915_gem_shrinker.c  |  63 ++-
>   drivers/gpu/drm/i915/gem/i915_gem_ttm.c       | 111 ++--
>   .../gpu/drm/i915/gem/selftests/huge_pages.c   |   4 +-
>   .../i915/gem/selftests/i915_gem_client_blt.c  |   2 +-
>   .../drm/i915/gem/selftests/i915_gem_mman.c    |   6 +
>   drivers/gpu/drm/i915/gt/gen6_ppgtt.c          | 148 +++--
>   drivers/gpu/drm/i915/gt/gen6_ppgtt.h          |   2 -
>   drivers/gpu/drm/i915/gt/intel_context.c       |   2 +-
>   drivers/gpu/drm/i915/gt/intel_engine_pm.c     |   5 +-
>   drivers/gpu/drm/i915/gt/intel_ggtt.c          | 366 ++----------
>   drivers/gpu/drm/i915/gt/intel_gtt.c           |  13 -
>   drivers/gpu/drm/i915/gt/intel_gtt.h           |   7 -
>   drivers/gpu/drm/i915/gt/intel_ppgtt.c         |  12 -
>   .../gpu/drm/i915/gt/intel_ring_submission.c   |   2 +-
>   drivers/gpu/drm/i915/gt/mock_engine.c         |  35 +-
>   drivers/gpu/drm/i915/i915_active.c            |  28 +-
>   drivers/gpu/drm/i915/i915_active.h            |  17 +-
>   drivers/gpu/drm/i915/i915_drv.h               |  11 +-
>   drivers/gpu/drm/i915/i915_gem.c               |  29 +-
>   drivers/gpu/drm/i915/i915_gem_evict.c         |  74 ++-
>   drivers/gpu/drm/i915/i915_gem_gtt.h           |   1 +
>   drivers/gpu/drm/i915/i915_gpu_error.c         |   9 +-
>   drivers/gpu/drm/i915/i915_request.c           |   9 +-
>   drivers/gpu/drm/i915/i915_request.h           |   7 +-
>   drivers/gpu/drm/i915/i915_vma.c               | 522 ++++++++++++++----
>   drivers/gpu/drm/i915/i915_vma.h               |  15 +-
>   drivers/gpu/drm/i915/i915_vma_types.h         |   2 -
>   drivers/gpu/drm/i915/selftests/i915_gem_gtt.c |  34 +-
>   drivers/gpu/drm/i915/selftests/i915_vma.c     |   2 +-
>   drivers/gpu/drm/i915/selftests/mock_gtt.c     |   4 -
>   37 files changed, 899 insertions(+), 816 deletions(-)
>

Re: [Intel-gfx] [PATCH 01/19] drm/i915: Move __i915_gem_free_object to ttm_bo_destroy

[Thomas Hellström (Intel)]

On 8/30/21 2:09 PM, Maarten Lankhorst wrote:
> When we implement delayed destroy, we may have a second
> call to the delete_mem_notify() handler, while free_object()
> only should be called once.
>
> Move it to bo->destroy(), to ensure it's only called once.
> This fixes some weird memory corruption issues with delayed
> destroy when async eviction is used.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>

Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

I wonder whether you could push this early with a Fixes: tag, perhaps. I 
actually managed to hit this once without any vma- or async 
modifications, but a bit curious how.

Thanks,

Thomas





> ---
>   drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 9 +++++----
>   1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> index 59ca53a3ef6a..eaf2ff29dd4a 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> @@ -355,11 +355,8 @@ static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
>   {
>   	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
>   
> -	if (likely(obj)) {
> -		/* This releases all gem object bindings to the backend. */
> +	if (likely(obj))
>   		i915_ttm_free_cached_io_st(obj);
> -		__i915_gem_free_object(obj);
> -	}
>   }
>   
>   static struct intel_memory_region *
> @@ -886,8 +883,12 @@ void i915_ttm_bo_destroy(struct ttm_buffer_object *bo)
>   {
>   	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
>   
> +	/* This releases all gem object bindings to the backend. */
> +	__i915_gem_free_object(obj);
> +
>   	i915_gem_object_release_memory_region(obj);
>   	mutex_destroy(&obj->ttm.get_io_page.lock);
> +
>   	if (obj->ttm.created)
>   		call_rcu(&obj->rcu, __i915_gem_free_object_rcu);
>   }

Re: [Intel-gfx] [PATCH 17/19] drm/i915: Add functions to set/get moving fence

[Thomas Hellström]

On 8/30/21 2:10 PM, Maarten Lankhorst wrote:
> We want to get rid of i915_vma tracking to simplify the code and
> lifetimes. Add a way to set/put the moving fence, in preparation for
> removing the tracking.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
>   drivers/gpu/drm/i915/gem/i915_gem_object.c | 15 +++++++++++++++
>   drivers/gpu/drm/i915/gem/i915_gem_object.h |  6 ++++++
>   2 files changed, 21 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> index 6fb9afb65034..dc0d2da297a0 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
> @@ -32,6 +32,7 @@
>   #include "i915_gem_object.h"
>   #include "i915_memcpy.h"
>   #include "i915_trace.h"
> +#include "i915_gem_ttm.h"
>   
>   static struct kmem_cache *slab_objects;
>   
> @@ -674,6 +675,20 @@ static const struct drm_gem_object_funcs i915_gem_object_funcs = {
>   	.export = i915_gem_prime_export,
>   };
>   
> +struct dma_fence *
> +i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj)
> +{
> +	return dma_fence_get(i915_gem_to_ttm(obj)->moving);
> +}
> +
> +void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,

Two spaces after void?

With that fixed:

Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>



> +				       struct dma_fence *fence)
> +{
> +	dma_fence_put(i915_gem_to_ttm(obj)->moving);
> +
> +	i915_gem_to_ttm(obj)->moving = dma_fence_get(fence);
> +}
> +
>   #if IS_ENABLED(CONFIG_DRM_I915_SELFTEST)
>   #include "selftests/huge_gem_object.c"
>   #include "selftests/huge_pages.c"
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.h b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> index 48112b9d76df..a23acfa98e21 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.h
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.h
> @@ -520,6 +520,12 @@ i915_gem_object_last_write_engine(struct drm_i915_gem_object *obj)
>   	return engine;
>   }
>   
> +struct dma_fence *
> +i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj);
> +
> +void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
> +				       struct dma_fence *fence);
> +
>   void i915_gem_object_set_cache_coherency(struct drm_i915_gem_object *obj,
>   					 unsigned int cache_level);
>   void i915_gem_object_flush_if_display(struct drm_i915_gem_object *obj);

Re: [Intel-gfx] [PATCH 18/19] drm/i915: Add support for asynchronous moving fence waiting

[Thomas Hellström (Intel)]

Hi,

We're moving towards a failsafe migration on DG1, but on DG2+ we'd have 
to wedge the GPU when an error occurs during initial clearing or swapin. 
But CPU maps don't care whether the GPU is wedged so we need to check 
the error status both below and in the TTM fault handler, I guess.

On 8/30/21 2:10 PM, Maarten Lankhorst wrote:
> For now, we will only allow async migration when TTM is used,
> so the paths we care about are related to TTM.
>
> The mmap path is handled by having the fence in ttm_bo->moving,
> when pinning, the binding only becomes available after the moving
> fence is signaled, and pinning a cpu map will only work after
> the moving fence signals.
>
> This should close all holes where userspace can read a buffer
> before it's fully migrated.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
>   drivers/gpu/drm/i915/gem/i915_gem_pages.c | 10 ++++++++++
>   drivers/gpu/drm/i915/i915_vma.c           | 10 ++++++++--
>   2 files changed, 18 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_pages.c b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> index 8eb1c3a6fc9c..a044d1829729 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_pages.c
> @@ -345,6 +345,7 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj,
>   			      enum i915_map_type type)
>   {
>   	enum i915_map_type has_type;
> +	struct dma_fence *moving;
>   	bool pinned;
>   	void *ptr;
>   	int err;
> @@ -355,6 +356,15 @@ void *i915_gem_object_pin_map(struct drm_i915_gem_object *obj,
>   
>   	assert_object_held(obj);
>   
> +	moving = i915_gem_object_get_moving_fence(obj);
> +	if (moving) {
> +		err = dma_fence_wait(moving, true);
> +		dma_fence_put(moving);
> +
> +		if (err)
> +			return ERR_PTR(err);
> +	}
> +
Error check on the moving fence?
>   	pinned = !(type & I915_MAP_OVERRIDE);
>   	type &= ~I915_MAP_OVERRIDE;
>   
> diff --git a/drivers/gpu/drm/i915/i915_vma.c b/drivers/gpu/drm/i915/i915_vma.c
> index bfba299d905c..320678e9b4e1 100644
> --- a/drivers/gpu/drm/i915/i915_vma.c
> +++ b/drivers/gpu/drm/i915/i915_vma.c
> @@ -398,7 +398,7 @@ int i915_vma_bind(struct i915_vma *vma,
>   	GEM_BUG_ON(!vma->pages);
>   
>   	trace_i915_vma_bind(vma, bind_flags);
> -	if (work && bind_flags & vma->vm->bind_async_flags) {
> +	if (work) {
>   		struct dma_fence *prev;
>   
>   		work->vma = vma;
> @@ -1145,6 +1145,7 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   		    u64 size, u64 alignment, u64 flags)
>   {
>   	struct i915_vma_work *work = NULL;
> +	struct dma_fence *moving = NULL;
>   	intel_wakeref_t wakeref = 0;
>   	unsigned int bound;
>   	int err;
> @@ -1168,7 +1169,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   	if (flags & PIN_GLOBAL)
>   		wakeref = intel_runtime_pm_get(&vma->vm->i915->runtime_pm);
>   
> -	if (flags & vma->vm->bind_async_flags) {
> +	moving = i915_gem_object_get_moving_fence(vma->obj);
> +	if (flags & vma->vm->bind_async_flags || moving) {
>   		/* lock VM */
>   		err = i915_vm_lock_objects(vma->vm, ww);
>   		if (err)
> @@ -1182,6 +1184,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   
>   		work->vm = i915_vm_get(vma->vm);
>   
> +		dma_fence_work_chain(&work->base, moving);
> +

We need to error check the async binding in the dma-fence-work release 
callback, and set the VMA_ERROR flag if necessary, at least until we 
have the error-safe / gpu wedge stuff. VMA_ERROR is racy, though.

>   		/* Allocate enough page directories to used PTE */
>   		if (vma->vm->allocate_va_range) {
>   			err = i915_vm_alloc_pt_stash(vma->vm,
> @@ -1289,6 +1293,8 @@ int i915_vma_pin_ww(struct i915_vma *vma, struct i915_gem_ww_ctx *ww,
>   err_rpm:
>   	if (wakeref)
>   		intel_runtime_pm_put(&vma->vm->i915->runtime_pm, wakeref);
> +	if (moving)
> +		dma_fence_put(moving);
>   	i915_vma_put_pages(vma);
>   	return err;
>   }

Re: [Intel-gfx] [PATCH 19/19] drm/i915: Add accelerated migration to ttm

[Thomas Hellström (Intel)]

drm/i915/ttm: Add async migration to the TTM backend?

On 8/30/21 2:10 PM, Maarten Lankhorst wrote:
> Expose the fence to ttm_bo->moving, which will get picked up by i915
> through the i915_gem_object_get_moving_fence call. Should be sufficient
> for the needs we have.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> ---
>   drivers/gpu/drm/i915/gem/i915_gem_ttm.c | 104 ++++++++++++------------
>   1 file changed, 54 insertions(+), 50 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> index eaf2ff29dd4a..751abbc3ccd3 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_ttm.c
> @@ -353,10 +353,14 @@ static void i915_ttm_swap_notify(struct ttm_buffer_object *bo)
>   
>   static void i915_ttm_delete_mem_notify(struct ttm_buffer_object *bo)
>   {
> -	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
> +	struct drm_i915_gem_object *obj;
>   
> -	if (likely(obj))
> -		i915_ttm_free_cached_io_st(obj);
> +	if (bo->destroy != i915_ttm_bo_destroy)
> +		return;
> +
> +	obj = i915_ttm_to_gem(bo);
> +
> +	i915_ttm_free_cached_io_st(obj);
>   }
>   
>   static struct intel_memory_region *
> @@ -426,10 +430,10 @@ i915_ttm_resource_get_st(struct drm_i915_gem_object *obj,
>   	return intel_region_ttm_resource_to_st(obj->mm.region, res);
>   }
>   
> -static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
> -			       bool clear,
> -			       struct ttm_resource *dst_mem,
> -			       struct sg_table *dst_st)
> +static struct i915_request *i915_ttm_accel_move(struct ttm_buffer_object *bo,
> +						bool clear,
> +						struct ttm_resource *dst_mem,
> +						struct sg_table *dst_st)
>   {
>   	struct drm_i915_private *i915 = container_of(bo->bdev, typeof(*i915),
>   						     bdev);
> @@ -437,18 +441,18 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
>   		ttm_manager_type(bo->bdev, bo->resource->mem_type);
>   	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
>   	struct sg_table *src_st;
> -	struct i915_request *rq;
> +	struct i915_request *rq = NULL;
>   	struct ttm_tt *ttm = bo->ttm;
>   	enum i915_cache_level src_level, dst_level;
>   	int ret;
>   
>   	if (!i915->gt.migrate.context)
> -		return -EINVAL;
> +		return ERR_PTR(-EINVAL);
>   
>   	dst_level = i915_ttm_cache_level(i915, dst_mem, ttm);
>   	if (clear) {
>   		if (bo->type == ttm_bo_type_kernel)
> -			return -EINVAL;
> +			return ERR_PTR(-EINVAL);
>   
>   		intel_engine_pm_get(i915->gt.migrate.context->engine);
>   		ret = intel_context_migrate_clear(i915->gt.migrate.context, NULL,


In the eviction or migration case, do we rely on unbind to sync on 
previous rendering?


> @@ -456,10 +460,6 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
>   						  gpu_binds_iomem(dst_mem),
>   						  0, &rq);
>   
> -		if (!ret && rq) {
> -			i915_request_wait(rq, 0, MAX_SCHEDULE_TIMEOUT);
> -			i915_request_put(rq);
> -		}
>   		intel_engine_pm_put(i915->gt.migrate.context->engine);
>   	} else {
>   		src_st = src_man->use_tt ? i915_ttm_tt_get_st(ttm) :
> @@ -473,49 +473,40 @@ static int i915_ttm_accel_move(struct ttm_buffer_object *bo,
>   						 dst_st->sgl, dst_level,
>   						 gpu_binds_iomem(dst_mem),
>   						 &rq);
> -		if (!ret && rq) {
> -			i915_request_wait(rq, 0, MAX_SCHEDULE_TIMEOUT);
> -			i915_request_put(rq);
> -		}
>   		intel_engine_pm_put(i915->gt.migrate.context->engine);
>   	}
>   
> -	return ret;
> +	return ret ? ERR_PTR(ret) : rq;

Actually, IIRC, rq can be Non-NULL even if we hit an error, in which 
case the request blit payload is ditched. Not sure what the best action 
is in that case, but perhaps just sync and unref. Perhaps we don't even 
need to sync.


>   }
>   
> -static void __i915_ttm_move(struct ttm_buffer_object *bo, bool clear,
> +static void i915_ttm_memcpy(struct ttm_buffer_object *bo, bool clear,
>   			    struct ttm_resource *dst_mem,
>   			    struct sg_table *dst_st)
>   {
> -	int ret;
> +	struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
> +	struct intel_memory_region *dst_reg, *src_reg;
> +	union {
> +		struct ttm_kmap_iter_tt tt;
> +		struct ttm_kmap_iter_iomap io;
> +	} _dst_iter, _src_iter;
> +	struct ttm_kmap_iter *dst_iter, *src_iter;
>   
> -	ret = i915_ttm_accel_move(bo, clear, dst_mem, dst_st);
> -	if (ret) {
> -		struct drm_i915_gem_object *obj = i915_ttm_to_gem(bo);
> -		struct intel_memory_region *dst_reg, *src_reg;
> -		union {
> -			struct ttm_kmap_iter_tt tt;
> -			struct ttm_kmap_iter_iomap io;
> -		} _dst_iter, _src_iter;
> -		struct ttm_kmap_iter *dst_iter, *src_iter;
> -
> -		dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
> -		src_reg = i915_ttm_region(bo->bdev, bo->resource->mem_type);
> -		GEM_BUG_ON(!dst_reg || !src_reg);
> -
> -		dst_iter = !cpu_maps_iomem(dst_mem) ?
> -			ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
> -			ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
> -						 dst_st, dst_reg->region.start);
> -
> -		src_iter = !cpu_maps_iomem(bo->resource) ?
> -			ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
> -			ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
> -						 obj->ttm.cached_io_st,
> -						 src_reg->region.start);
> -
> -		ttm_move_memcpy(clear, dst_mem->num_pages, dst_iter, src_iter);
> -	}
> +	dst_reg = i915_ttm_region(bo->bdev, dst_mem->mem_type);
> +	src_reg = i915_ttm_region(bo->bdev, bo->resource->mem_type);
> +	GEM_BUG_ON(!dst_reg || !src_reg);
> +
> +	dst_iter = !cpu_maps_iomem(dst_mem) ?
> +		ttm_kmap_iter_tt_init(&_dst_iter.tt, bo->ttm) :
> +		ttm_kmap_iter_iomap_init(&_dst_iter.io, &dst_reg->iomap,
> +					  dst_st, dst_reg->region.start);
> +
> +	src_iter = !cpu_maps_iomem(bo->resource) ?
> +		ttm_kmap_iter_tt_init(&_src_iter.tt, bo->ttm) :
> +		ttm_kmap_iter_iomap_init(&_src_iter.io, &src_reg->iomap,
> +					  obj->ttm.cached_io_st,
> +					  src_reg->region.start);
> +
> +	ttm_move_memcpy(clear, dst_mem->num_pages, dst_iter, src_iter);
>   }
>   
>   static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
> @@ -530,6 +521,7 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
>   	struct sg_table *dst_st;
>   	bool clear;
>   	int ret;
> +	struct i915_request *rq;
>   
>   	/* Sync for now. We could do the actual copy async. */
>   	ret = ttm_bo_wait_ctx(bo, ctx);
> @@ -558,10 +550,22 @@ static int i915_ttm_move(struct ttm_buffer_object *bo, bool evict,
>   		return PTR_ERR(dst_st);
>   
>   	clear = !cpu_maps_iomem(bo->resource) && (!ttm || !ttm_tt_is_populated(ttm));
> +
> +	rq = NULL;
>   	if (!(clear && ttm && !(ttm->page_flags & TTM_PAGE_FLAG_ZERO_ALLOC)))
> -		__i915_ttm_move(bo, clear, dst_mem, dst_st);
> +		rq = i915_ttm_accel_move(bo, clear, dst_mem, dst_st);
> +
> +	if (IS_ERR(rq))
> +		i915_ttm_memcpy(bo, clear, dst_mem, dst_st);
> +
> +	/* Below dst_mem becomes bo->resource. */
> +	if (!IS_ERR_OR_NULL(rq)) {
> +		ttm_bo_move_accel_cleanup(bo, &rq->fence, evict, true, dst_mem);
> +		i915_request_put(rq);
> +	} else {
> +		ttm_bo_move_sync_cleanup(bo, dst_mem);
> +	}
>   
> -	ttm_bo_move_sync_cleanup(bo, dst_mem);
>   	i915_ttm_adjust_domains_after_move(obj);
>   	i915_ttm_free_cached_io_st(obj);
>   

Re: [Intel-gfx] [PATCH 00/19] drm/i915: Short-term pinning and async eviction.

[Maarten Lankhorst]

Op 16-09-2021 om 11:40 schreef Thomas Hellström (Intel):
>
> On 8/30/21 2:09 PM, Maarten Lankhorst wrote:
>> Remove some parts of the i915_vma api, ensure obj->vma always exists,
>> and finally force the object lock to be taken when calling i915_vma_unbind
>> is called.
>
> Should this be vma->obj?
>
Correct, missed that.
>
>>
>> With this, locking is a lot cleaner, and we no longer need all the if (!obj->vma) checks.
>> We kill off the locking around i915_vma->set/get pages, and kill off the short-term
>> pinning/unpinning in execbuf.
>>
>> After this, we go 1 step further, populate obj->moving where required, and add support
>> for async eviction/clearing.
>
> Does the async eviction / clearing depend on the vma cleanup? If not can we move that into a separate series? 

It kind of does. We cannot always require vma->obj to be locked and remove vma locks if we don't have vma->obj all the time.

Re: [Intel-gfx] [PATCH 01/19] drm/i915: Move __i915_gem_free_object to ttm_bo_destroy

[Maarten Lankhorst]

Op 16-09-2021 om 11:43 schreef Thomas Hellström (Intel):
>
> On 8/30/21 2:09 PM, Maarten Lankhorst wrote:
>> When we implement delayed destroy, we may have a second
>> call to the delete_mem_notify() handler, while free_object()
>> only should be called once.
>>
>> Move it to bo->destroy(), to ensure it's only called once.
>> This fixes some weird memory corruption issues with delayed
>> destroy when async eviction is used.
>>
>> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>
> Reviewed-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>
> I wonder whether you could push this early with a Fixes: tag, perhaps. I actually managed to hit this once without any vma- or async modifications, but a bit curious how.
>
> Thanks,
>
> Thomas
>
I think I triggered that as well once when testing the series with no patch applied yet as base, but I wasn't sure a the time. Since that's 2 suspicious cases, I will push this now with a fixes tag.

Thanks & pushed,

Maarten

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Maarten Lankhorst]

Op 03-09-2021 om 12:48 schreef Tvrtko Ursulin:
>
> On 03/09/2021 10:31, Maarten Lankhorst wrote:
>> Op 31-08-2021 om 12:29 schreef Tvrtko Ursulin:
>>>
>>> On 31/08/2021 10:34, Maarten Lankhorst wrote:
>>>> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>>>>
>>>>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>>>>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>>>>>
>>>>> Is the direction here compatible with SVM / VM_BIND?
>>>>
>>>>
>>>> Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.
>>>
>>> Hm but what will vma->obj point to in case of SVM, when there is no GEM BO?
>>
>> Probably to one of the bo's in i915_vm, or a dummy bo that least shares the vm_resv object, similar to the aliasing gtt handling.
>
> As a long term or short term solution? Worried that would waste a lot of SLAB space just for convenience (whole struct drm_i915_gem_object just to store a single pointer to a dma_resv object, if I got that right), while it should be possible to come up with a cleaner design.
>
> Even for the upcoming page granularity work we will need multiple VMAs point to single GEM bo in ppgtt and that, when SVM is considered, could for instance mean that VMAs should instead be backed by some new backing store objects, which in turn may (or may not) point to GEM BOs.
>
> Regards,
>
> Tvrtko

We could revisit this if it's required for SVM, since we can always optimize that code later, I don't think it's a problem to remove it for now, especially since it's a lot easier if VMA becomes a pointer to a memory slab for an object only, without its own lifetime rules. :)

Re: [Intel-gfx] [PATCH 17/19] drm/i915: Add functions to set/get moving fence

[Maarten Lankhorst]

Op 08-09-2021 om 06:08 schreef Niranjana Vishwanathapura:
> On Mon, Aug 30, 2021 at 02:10:04PM +0200, Maarten Lankhorst wrote:
>> We want to get rid of i915_vma tracking to simplify the code and
>> lifetimes. Add a way to set/put the moving fence, in preparation for
>> removing the tracking.
>>
>> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> ---
>> drivers/gpu/drm/i915/gem/i915_gem_object.c | 15 +++++++++++++++
>> drivers/gpu/drm/i915/gem/i915_gem_object.h |  6 ++++++
>> 2 files changed, 21 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_object.c b/drivers/gpu/drm/i915/gem/i915_gem_object.c
>> index 6fb9afb65034..dc0d2da297a0 100644
>> --- a/drivers/gpu/drm/i915/gem/i915_gem_object.c
>> +++ b/drivers/gpu/drm/i915/gem/i915_gem_object.c
>> @@ -32,6 +32,7 @@
>> #include "i915_gem_object.h"
>> #include "i915_memcpy.h"
>> #include "i915_trace.h"
>> +#include "i915_gem_ttm.h"
>>
>> static struct kmem_cache *slab_objects;
>>
>> @@ -674,6 +675,20 @@ static const struct drm_gem_object_funcs i915_gem_object_funcs = {
>>     .export = i915_gem_prime_export,
>> };
>>
>> +struct dma_fence *
>> +i915_gem_object_get_moving_fence(struct drm_i915_gem_object *obj)
>> +{
>> +    return dma_fence_get(i915_gem_to_ttm(obj)->moving);
>> +}
>> +
>> +void  i915_gem_object_set_moving_fence(struct drm_i915_gem_object *obj,
>> +                       struct dma_fence *fence)
>> +{
>> +    dma_fence_put(i915_gem_to_ttm(obj)->moving);
>> +
>> +    i915_gem_to_ttm(obj)->moving = dma_fence_get(fence);
>> +}
>
> When is i915_gem_object_set_moving_fence() called?
> I am not seeing it being called in this patch series. 

It's not called directly any more but it was in an earlier version; it could be removed but I kept it for now.

TTM will set ttm_bo->moving directly in the callbacks, could drop this function if you want.

Re: [Intel-gfx] [PATCH 13/19] drm/i915: Take object lock in i915_ggtt_pin if ww is not set

[Maarten Lankhorst]

Op 08-09-2021 om 05:11 schreef Niranjana Vishwanathapura:
> On Mon, Aug 30, 2021 at 02:10:00PM +0200, Maarten Lankhorst wrote:
>> i915_vma_wait_for_bind needs the vma lock held, fix the caller.
>>
>
> Need an lock assert in i915_vma_wait_for_bind()?
>
> Niranjana 
Yeah, makes sense. I suppose changing dma_resv_get_excl_unlocked() to dma_resv_excl_fence() will work too. Will do that instead..

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Tvrtko Ursulin]

On 16/09/2021 14:41, Maarten Lankhorst wrote:
> Op 03-09-2021 om 12:48 schreef Tvrtko Ursulin:
>>
>> On 03/09/2021 10:31, Maarten Lankhorst wrote:
>>> Op 31-08-2021 om 12:29 schreef Tvrtko Ursulin:
>>>>
>>>> On 31/08/2021 10:34, Maarten Lankhorst wrote:
>>>>> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>>>>>
>>>>>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>>>>>> vma->obj and vma->resv are now never NULL, and some checks can be removed.
>>>>>>
>>>>>> Is the direction here compatible with SVM / VM_BIND?
>>>>>
>>>>>
>>>>> Yeah, it should be. The changes here make the obj->resv->lock the main lock, so it should at least simplify locking for VM_BIND.
>>>>
>>>> Hm but what will vma->obj point to in case of SVM, when there is no GEM BO?
>>>
>>> Probably to one of the bo's in i915_vm, or a dummy bo that least shares the vm_resv object, similar to the aliasing gtt handling.
>>
>> As a long term or short term solution? Worried that would waste a lot of SLAB space just for convenience (whole struct drm_i915_gem_object just to store a single pointer to a dma_resv object, if I got that right), while it should be possible to come up with a cleaner design.
>>
>> Even for the upcoming page granularity work we will need multiple VMAs point to single GEM bo in ppgtt and that, when SVM is considered, could for instance mean that VMAs should instead be backed by some new backing store objects, which in turn may (or may not) point to GEM BOs.
>>
>> Regards,
>>
>> Tvrtko
> 
> We could revisit this if it's required for SVM, since we can always optimize that code later, I don't think it's a problem to remove it for now, especially since it's a lot easier if VMA becomes a pointer to a memory slab for an object only, without its own lifetime rules. :)

Not sure what you meant with "pointer to memory slab for an object"?

But on the high level, what worries me is whether the direction is not 
wrong. Sure you can change it all again later, but if we are moving 
towards the world where VMAs are fundamentally and predominantly *not* 
backed by GEM buffer objects, then I have to ask the question whether 
this plan of rewriting everything twice is the most efficient one.

Maybe its not that scary, not sure, but again, all I see is a large-ish 
series which implements some very important functionality, and _seems_ 
to rely on what I think is a design direction incompatible with where I 
thought we needed to go.

I suppose all I can do is ask you to verify this direction with Daniel. 
Maybe you already have but I did not see it in public at least. So 
adding him to CC.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 07/19] drm/i915: vma is always backed by an object.

[Tvrtko Ursulin]

On 16/09/2021 15:30, Tvrtko Ursulin wrote:
> 
> On 16/09/2021 14:41, Maarten Lankhorst wrote:
>> Op 03-09-2021 om 12:48 schreef Tvrtko Ursulin:
>>>
>>> On 03/09/2021 10:31, Maarten Lankhorst wrote:
>>>> Op 31-08-2021 om 12:29 schreef Tvrtko Ursulin:
>>>>>
>>>>> On 31/08/2021 10:34, Maarten Lankhorst wrote:
>>>>>> Op 31-08-2021 om 11:18 schreef Tvrtko Ursulin:
>>>>>>>
>>>>>>> On 30/08/2021 13:09, Maarten Lankhorst wrote:
>>>>>>>> vma->obj and vma->resv are now never NULL, and some checks can 
>>>>>>>> be removed.
>>>>>>>
>>>>>>> Is the direction here compatible with SVM / VM_BIND?
>>>>>>
>>>>>>
>>>>>> Yeah, it should be. The changes here make the obj->resv->lock the 
>>>>>> main lock, so it should at least simplify locking for VM_BIND.
>>>>>
>>>>> Hm but what will vma->obj point to in case of SVM, when there is no 
>>>>> GEM BO?
>>>>
>>>> Probably to one of the bo's in i915_vm, or a dummy bo that least 
>>>> shares the vm_resv object, similar to the aliasing gtt handling.
>>>
>>> As a long term or short term solution? Worried that would waste a lot 
>>> of SLAB space just for convenience (whole struct drm_i915_gem_object 
>>> just to store a single pointer to a dma_resv object, if I got that 
>>> right), while it should be possible to come up with a cleaner design.
>>>
>>> Even for the upcoming page granularity work we will need multiple 
>>> VMAs point to single GEM bo in ppgtt and that, when SVM is 
>>> considered, could for instance mean that VMAs should instead be 
>>> backed by some new backing store objects, which in turn may (or may 
>>> not) point to GEM BOs.
>>>
>>> Regards,
>>>
>>> Tvrtko
>>
>> We could revisit this if it's required for SVM, since we can always 
>> optimize that code later, I don't think it's a problem to remove it 
>> for now, especially since it's a lot easier if VMA becomes a pointer 
>> to a memory slab for an object only, without its own lifetime rules. :)
> 
> Not sure what you meant with "pointer to memory slab for an object"?
> 
> But on the high level, what worries me is whether the direction is not 
> wrong. Sure you can change it all again later, but if we are moving 
> towards the world where VMAs are fundamentally and predominantly *not* 
> backed by GEM buffer objects, then I have to ask the question whether 
> this plan of rewriting everything twice is the most efficient one.
> 
> Maybe its not that scary, not sure, but again, all I see is a large-ish 
> series which implements some very important functionality, and _seems_ 
> to rely on what I think is a design direction incompatible with where I 
> thought we needed to go.
> 
> I suppose all I can do is ask you to verify this direction with Daniel. 
> Maybe you already have but I did not see it in public at least. So 
> adding him to CC.

Okay I reminded myself of how the SVM is implemented and perhaps it is 
not a concern. It seems that it doesn't use the VMA for more than a 
temporary vehicle during PTE setup stage.

And for page granularity paths over legacy binding I think it should 
also be fine since, as you say, all VMAs can and should point to the 
same obj.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 04/19] drm/i915: Remove gen6_ppgtt_unpin_all

[Matthew Auld]

On Mon, 30 Aug 2021 at 13:09, Maarten Lankhorst
<maarten.lankhorst@linux.intel.com> wrote:
>
> gen6_ppgtt_unpin_all is unused, kill it.
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Reviewed-by: Matthew Auld <matthew.auld@intel.com>

> ---
>  drivers/gpu/drm/i915/gt/gen6_ppgtt.c | 11 -----------
>  drivers/gpu/drm/i915/gt/gen6_ppgtt.h |  1 -
>  2 files changed, 12 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
> index 1aee5e6b1b23..efc243f40d0f 100644
> --- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
> +++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.c
> @@ -405,17 +405,6 @@ void gen6_ppgtt_unpin(struct i915_ppgtt *base)
>                 i915_vma_unpin(ppgtt->vma);
>  }
>
> -void gen6_ppgtt_unpin_all(struct i915_ppgtt *base)
> -{
> -       struct gen6_ppgtt *ppgtt = to_gen6_ppgtt(base);
> -
> -       if (!atomic_read(&ppgtt->pin_count))
> -               return;
> -
> -       i915_vma_unpin(ppgtt->vma);
> -       atomic_set(&ppgtt->pin_count, 0);
> -}
> -
>  struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt)
>  {
>         struct i915_ggtt * const ggtt = gt->ggtt;
> diff --git a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
> index 6a61a5c3a85a..ab0eecb086dd 100644
> --- a/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
> +++ b/drivers/gpu/drm/i915/gt/gen6_ppgtt.h
> @@ -71,7 +71,6 @@ static inline struct gen6_ppgtt *to_gen6_ppgtt(struct i915_ppgtt *base)
>
>  int gen6_ppgtt_pin(struct i915_ppgtt *base, struct i915_gem_ww_ctx *ww);
>  void gen6_ppgtt_unpin(struct i915_ppgtt *base);
> -void gen6_ppgtt_unpin_all(struct i915_ppgtt *base);
>  void gen6_ppgtt_enable(struct intel_gt *gt);
>  void gen7_ppgtt_enable(struct intel_gt *gt);
>  struct i915_ppgtt *gen6_ppgtt_create(struct intel_gt *gt);
> --
> 2.32.0
>

Re: [Intel-gfx] [PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Matthew Auld]

On Mon, 30 Aug 2021 at 13:10, Maarten Lankhorst
<maarten.lankhorst@linux.intel.com> wrote:
>
> We forgot to call intel_runtime_pm_put on error, fix it!
>
> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> Cc: <stable@vger.kernel.org> # v5.13+

How does the err handling work? gem_shrink is meant to return the
number of pages reclaimed which is an unsigned long, and yet we are
also just returning the err here? Fortunately it looks like nothing is
calling gem_shrinker with an actual ww context, so nothing will hit
this yet? I think the interface needs to be reworked or something.

> ---
>  drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> index e382b7f2353b..5ab136ffdeb2 100644
> --- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> +++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> @@ -118,7 +118,7 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
>         intel_wakeref_t wakeref = 0;
>         unsigned long count = 0;
>         unsigned long scanned = 0;
> -       int err;
> +       int err = 0;
>
>         /* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
>         bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
> @@ -242,12 +242,15 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
>                 list_splice_tail(&still_in_list, phase->list);
>                 spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
>                 if (err)
> -                       return err;
> +                       break;
>         }
>
>         if (shrink & I915_SHRINK_BOUND)
>                 intel_runtime_pm_put(&i915->runtime_pm, wakeref);
>
> +       if (err)
> +               return err;
> +
>         if (nr_scanned)
>                 *nr_scanned += scanned;
>         return count;
> --
> 2.32.0
>

Re: [Intel-gfx] [PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Matthew Auld]

On Wed, 29 Sept 2021 at 09:37, Matthew Auld
<matthew.william.auld@gmail.com> wrote:
>
> On Mon, 30 Aug 2021 at 13:10, Maarten Lankhorst
> <maarten.lankhorst@linux.intel.com> wrote:
> >
> > We forgot to call intel_runtime_pm_put on error, fix it!
> >
> > Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
> > Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
> > Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> > Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
> > Cc: <stable@vger.kernel.org> # v5.13+
>
> How does the err handling work? gem_shrink is meant to return the
> number of pages reclaimed which is an unsigned long, and yet we are
> also just returning the err here? Fortunately it looks like nothing is
> calling gem_shrinker with an actual ww context, so nothing will hit
> this yet? I think the interface needs to be reworked or something.

Can we just remove the ww context argument, or is that needed for
something in the future?

>
> > ---
> >  drivers/gpu/drm/i915/gem/i915_gem_shrinker.c | 7 +++++--
> >  1 file changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> > index e382b7f2353b..5ab136ffdeb2 100644
> > --- a/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> > +++ b/drivers/gpu/drm/i915/gem/i915_gem_shrinker.c
> > @@ -118,7 +118,7 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
> >         intel_wakeref_t wakeref = 0;
> >         unsigned long count = 0;
> >         unsigned long scanned = 0;
> > -       int err;
> > +       int err = 0;
> >
> >         /* CHV + VTD workaround use stop_machine(); need to trylock vm->mutex */
> >         bool trylock_vm = !ww && intel_vm_no_concurrent_access_wa(i915);
> > @@ -242,12 +242,15 @@ i915_gem_shrink(struct i915_gem_ww_ctx *ww,
> >                 list_splice_tail(&still_in_list, phase->list);
> >                 spin_unlock_irqrestore(&i915->mm.obj_lock, flags);
> >                 if (err)
> > -                       return err;
> > +                       break;
> >         }
> >
> >         if (shrink & I915_SHRINK_BOUND)
> >                 intel_runtime_pm_put(&i915->runtime_pm, wakeref);
> >
> > +       if (err)
> > +               return err;
> > +
> >         if (nr_scanned)
> >                 *nr_scanned += scanned;
> >         return count;
> > --
> > 2.32.0
> >

Re: [Intel-gfx] [PATCH 08/19] drm/i915: Fix runtime pm handling in i915_gem_shrink

[Maarten Lankhorst]

Op 29-09-2021 om 10:37 schreef Matthew Auld:
> On Mon, 30 Aug 2021 at 13:10, Maarten Lankhorst
> <maarten.lankhorst@linux.intel.com> wrote:
>> We forgot to call intel_runtime_pm_put on error, fix it!
>>
>> Signed-off-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
>> Fixes: cf41a8f1dc1e ("drm/i915: Finally remove obj->mm.lock.")
>> Cc: Thomas Hellström <thomas.hellstrom@linux.intel.com>
>> Cc: Daniel Vetter <daniel.vetter@ffwll.ch>
>> Cc: <stable@vger.kernel.org> # v5.13+
> How does the err handling work? gem_shrink is meant to return the
> number of pages reclaimed which is an unsigned long, and yet we are
> also just returning the err here? Fortunately it looks like nothing is
> calling gem_shrinker with an actual ww context, so nothing will hit
> this yet? I think the interface needs to be reworked or something.

We should probably make it signed in the future when used.
It should never hit the LONG_MAX limit, since max value returned would be ULONG_MAX >> PAGE_SHIFT,
assuming the entire address space is filled with evictable buffers.

I've kept the ww lock, in case we want to evict in the future. Without ww context the buffers
are trylocked, with ww we can evict the entire address space as much as possible.
In most cases we only want to evict idle objects, in that case no ww is needed.

Pushed just this patch, thanks for feedback. :)

~Maarten