[RFC v4 00/12] Waitboost drm syncobj waits

[RFC v4 00/12] Waitboost drm syncobj waits

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

In i915 we have this concept of "wait boosting" where we give a priority boost
for instance to fences which are actively waited upon from userspace. This has
it's pros and cons and can certainly be discussed at lenght. However fact is
some workloads really like it.

Problem is that with the arrival of drm syncobj and a new userspace waiting
entry point it added, the waitboost mechanism was bypassed. AFAIU this mostly
happens with all Vulkan based userspaces. Hence I cooked up this mini series to
see if discussion about restoring the waitboost can be had.

The series adds a concept of "wait count" to dma fence which is intended to
represent explicit userspace waits. It is therefore incremented for every
explicit dma_fence_enable_sw_signaling and dma_fence_add_wait_callback (like
dma_fence_add_callback but from explicit/userspace wait paths). Individual
drivers can then inspect this via dma_fence_wait_count() and decide to wait
boost the waits on such fences.

Patch has been slightly tested for performance impact by Google using some clvk
workloads and shows a good improvement (frame time improved from 16ms to 13ms).

It is also important to mention that although benefits of waitboosting may not
only be with workloads related to frame presentation time, but also with
serialized computations which constantly move between the CPU and GPU, there are
also workloads which do not improve on the performance but degrade on
efficiency. Therefore extending this approach needs to be carefully evaluated.

*)
https://gitlab.freedesktop.org/drm/intel/-/issues/8014

v2:
 * Small fixups based on CI feedback:
    * Handle decrement correctly for already signalled case while adding callback.
    * Remove i915 assert which was making sure struct i915_request does not grow.
 * Split out the i915 patch into three separate functional changes.

v3:
 * Handle drivers which open-code callback additions.

v4:
 * Handle dma-fence-array and dma-fence-chain.

Tvrtko Ursulin (12):
  dma-fence: Move i915 helpers into common
  dma-fence: Add callback initialization helper
  drm/i915: Use fence callback initialization helper
  drm/vmwgfx: Use fence callback initialization helper
  dma-fence: Track explicit waiters
  dma-fence: Export __dma_fence_add_callback
  dma-fence-array: Propagate wait status to contained fences
  dma-fence-chain: Propagate wait status to contained fences
  drm/syncobj: Mark syncobj waits as external waiters
  drm/i915: Waitboost external waits
  drm/i915: Mark waits as explicit
  drm/i915: Wait boost requests waited upon by others

 drivers/dma-buf/dma-fence-array.c           |   5 +-
 drivers/dma-buf/dma-fence-chain.c           |  22 +++-
 drivers/dma-buf/dma-fence.c                 | 138 ++++++++++++++------
 drivers/gpu/drm/drm_syncobj.c               |   6 +-
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c |  22 ----
 drivers/gpu/drm/i915/gt/intel_engine_pm.c   |   1 -
 drivers/gpu/drm/i915/i915_active.c          |   2 +-
 drivers/gpu/drm/i915/i915_active.h          |   2 +-
 drivers/gpu/drm/i915/i915_request.c         |  13 +-
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c       |   2 +-
 include/linux/dma-fence-chain.h             |   1 +
 include/linux/dma-fence.h                   |  29 ++++
 12 files changed, 165 insertions(+), 78 deletions(-)

-- 
2.37.2

[Intel-gfx] [RFC v4 00/12] Waitboost drm syncobj waits

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

In i915 we have this concept of "wait boosting" where we give a priority boost
for instance to fences which are actively waited upon from userspace. This has
it's pros and cons and can certainly be discussed at lenght. However fact is
some workloads really like it.

Problem is that with the arrival of drm syncobj and a new userspace waiting
entry point it added, the waitboost mechanism was bypassed. AFAIU this mostly
happens with all Vulkan based userspaces. Hence I cooked up this mini series to
see if discussion about restoring the waitboost can be had.

The series adds a concept of "wait count" to dma fence which is intended to
represent explicit userspace waits. It is therefore incremented for every
explicit dma_fence_enable_sw_signaling and dma_fence_add_wait_callback (like
dma_fence_add_callback but from explicit/userspace wait paths). Individual
drivers can then inspect this via dma_fence_wait_count() and decide to wait
boost the waits on such fences.

Patch has been slightly tested for performance impact by Google using some clvk
workloads and shows a good improvement (frame time improved from 16ms to 13ms).

It is also important to mention that although benefits of waitboosting may not
only be with workloads related to frame presentation time, but also with
serialized computations which constantly move between the CPU and GPU, there are
also workloads which do not improve on the performance but degrade on
efficiency. Therefore extending this approach needs to be carefully evaluated.

*)
https://gitlab.freedesktop.org/drm/intel/-/issues/8014

v2:
 * Small fixups based on CI feedback:
    * Handle decrement correctly for already signalled case while adding callback.
    * Remove i915 assert which was making sure struct i915_request does not grow.
 * Split out the i915 patch into three separate functional changes.

v3:
 * Handle drivers which open-code callback additions.

v4:
 * Handle dma-fence-array and dma-fence-chain.

Tvrtko Ursulin (12):
  dma-fence: Move i915 helpers into common
  dma-fence: Add callback initialization helper
  drm/i915: Use fence callback initialization helper
  drm/vmwgfx: Use fence callback initialization helper
  dma-fence: Track explicit waiters
  dma-fence: Export __dma_fence_add_callback
  dma-fence-array: Propagate wait status to contained fences
  dma-fence-chain: Propagate wait status to contained fences
  drm/syncobj: Mark syncobj waits as external waiters
  drm/i915: Waitboost external waits
  drm/i915: Mark waits as explicit
  drm/i915: Wait boost requests waited upon by others

 drivers/dma-buf/dma-fence-array.c           |   5 +-
 drivers/dma-buf/dma-fence-chain.c           |  22 +++-
 drivers/dma-buf/dma-fence.c                 | 138 ++++++++++++++------
 drivers/gpu/drm/drm_syncobj.c               |   6 +-
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c |  22 ----
 drivers/gpu/drm/i915/gt/intel_engine_pm.c   |   1 -
 drivers/gpu/drm/i915/i915_active.c          |   2 +-
 drivers/gpu/drm/i915/i915_active.h          |   2 +-
 drivers/gpu/drm/i915/i915_request.c         |  13 +-
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c       |   2 +-
 include/linux/dma-fence-chain.h             |   1 +
 include/linux/dma-fence.h                   |  29 ++++
 12 files changed, 165 insertions(+), 78 deletions(-)

-- 
2.37.2

[RFC 01/12] dma-fence: Move i915 helpers into common

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Unhide some i915 helpers which are used for splitting the signalled
check vs notification stages during en masse fence processing.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c                 | 35 +++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c | 22 -------------
 include/linux/dma-fence.h                   |  4 +++
 3 files changed, 30 insertions(+), 31 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index 0de0482cd36e..41da0da17eba 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -344,6 +344,30 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
+{
+	lockdep_assert_held(fence->lock);
+
+	fence->timestamp = timestamp;
+	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
+	trace_dma_fence_signaled(fence);
+}
+EXPORT_SYMBOL(__dma_fence_signal__timestamp);
+
+void
+__dma_fence_signal__notify(struct dma_fence *fence,
+			   const struct list_head *list)
+{
+	struct dma_fence_cb *cur, *tmp;
+
+	lockdep_assert_held(fence->lock);
+
+	list_for_each_entry_safe(cur, tmp, list, node) {
+		INIT_LIST_HEAD(&cur->node);
+		cur->func(fence, cur);
+	}
+}
+EXPORT_SYMBOL(__dma_fence_signal__notify);
 
 /**
  * dma_fence_signal_timestamp_locked - signal completion of a fence
@@ -366,7 +390,6 @@ void __dma_fence_might_wait(void)
 int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp)
 {
-	struct dma_fence_cb *cur, *tmp;
 	struct list_head cb_list;
 
 	lockdep_assert_held(fence->lock);
@@ -378,14 +401,8 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 	/* Stash the cb_list before replacing it with the timestamp */
 	list_replace(&fence->cb_list, &cb_list);
 
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-
-	list_for_each_entry_safe(cur, tmp, &cb_list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
+	__dma_fence_signal__timestamp(fence, timestamp);
+	__dma_fence_signal__notify(fence, &cb_list);
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
index ecc990ec1b95..26b6b777c479 100644
--- a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
+++ b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
@@ -127,28 +127,6 @@ __dma_fence_signal(struct dma_fence *fence)
 	return !test_and_set_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags);
 }
 
-static void
-__dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
-{
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-}
-
-static void
-__dma_fence_signal__notify(struct dma_fence *fence,
-			   const struct list_head *list)
-{
-	struct dma_fence_cb *cur, *tmp;
-
-	lockdep_assert_held(fence->lock);
-
-	list_for_each_entry_safe(cur, tmp, list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
-}
-
 static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl)
 {
 	if (b->irq_engine)
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 775cdc0b4f24..2af328a89468 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -367,6 +367,10 @@ static inline void dma_fence_end_signalling(bool cookie) {}
 static inline void __dma_fence_might_wait(void) {}
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp);
+void __dma_fence_signal__notify(struct dma_fence *fence,
+				const struct list_head *list);
+
 int dma_fence_signal(struct dma_fence *fence);
 int dma_fence_signal_locked(struct dma_fence *fence);
 int dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp);
-- 
2.37.2

[Intel-gfx] [RFC 01/12] dma-fence: Move i915 helpers into common

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Unhide some i915 helpers which are used for splitting the signalled
check vs notification stages during en masse fence processing.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c                 | 35 +++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c | 22 -------------
 include/linux/dma-fence.h                   |  4 +++
 3 files changed, 30 insertions(+), 31 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index 0de0482cd36e..41da0da17eba 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -344,6 +344,30 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
+{
+	lockdep_assert_held(fence->lock);
+
+	fence->timestamp = timestamp;
+	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
+	trace_dma_fence_signaled(fence);
+}
+EXPORT_SYMBOL(__dma_fence_signal__timestamp);
+
+void
+__dma_fence_signal__notify(struct dma_fence *fence,
+			   const struct list_head *list)
+{
+	struct dma_fence_cb *cur, *tmp;
+
+	lockdep_assert_held(fence->lock);
+
+	list_for_each_entry_safe(cur, tmp, list, node) {
+		INIT_LIST_HEAD(&cur->node);
+		cur->func(fence, cur);
+	}
+}
+EXPORT_SYMBOL(__dma_fence_signal__notify);
 
 /**
  * dma_fence_signal_timestamp_locked - signal completion of a fence
@@ -366,7 +390,6 @@ void __dma_fence_might_wait(void)
 int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp)
 {
-	struct dma_fence_cb *cur, *tmp;
 	struct list_head cb_list;
 
 	lockdep_assert_held(fence->lock);
@@ -378,14 +401,8 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 	/* Stash the cb_list before replacing it with the timestamp */
 	list_replace(&fence->cb_list, &cb_list);
 
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-
-	list_for_each_entry_safe(cur, tmp, &cb_list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
+	__dma_fence_signal__timestamp(fence, timestamp);
+	__dma_fence_signal__notify(fence, &cb_list);
 
 	return 0;
 }
diff --git a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
index ecc990ec1b95..26b6b777c479 100644
--- a/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
+++ b/drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
@@ -127,28 +127,6 @@ __dma_fence_signal(struct dma_fence *fence)
 	return !test_and_set_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags);
 }
 
-static void
-__dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
-{
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-}
-
-static void
-__dma_fence_signal__notify(struct dma_fence *fence,
-			   const struct list_head *list)
-{
-	struct dma_fence_cb *cur, *tmp;
-
-	lockdep_assert_held(fence->lock);
-
-	list_for_each_entry_safe(cur, tmp, list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
-}
-
 static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl)
 {
 	if (b->irq_engine)
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 775cdc0b4f24..2af328a89468 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -367,6 +367,10 @@ static inline void dma_fence_end_signalling(bool cookie) {}
 static inline void __dma_fence_might_wait(void) {}
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp);
+void __dma_fence_signal__notify(struct dma_fence *fence,
+				const struct list_head *list);
+
 int dma_fence_signal(struct dma_fence *fence);
 int dma_fence_signal_locked(struct dma_fence *fence);
 int dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp);
-- 
2.37.2

[RFC 02/12] dma-fence: Add callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

In preparation of adding a new field to struct dma_fence_cb we will need
an initialization helper for those callers who add callbacks by open-
coding. That will ensure they initialize all the fields so common code
does not get confused by potential garbage in some fields.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c | 10 ++++------
 include/linux/dma-fence.h   |  7 +++++++
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index 41da0da17eba..ea4a1f82c9bf 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -662,21 +662,19 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 	unsigned long flags;
 	int ret = 0;
 
+	__dma_fence_cb_init(cb, func);
+
 	if (WARN_ON(!fence || !func))
 		return -EINVAL;
 
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
-		INIT_LIST_HEAD(&cb->node);
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
 		return -ENOENT;
-	}
 
 	spin_lock_irqsave(fence->lock, flags);
 
 	if (__dma_fence_enable_signaling(fence)) {
-		cb->func = func;
 		list_add_tail(&cb->node, &fence->cb_list);
 	} else {
-		INIT_LIST_HEAD(&cb->node);
 		ret = -ENOENT;
 	}
 
@@ -795,7 +793,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = dma_fence_default_wait_cb;
+	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &fence->cb_list);
 
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 2af328a89468..35933e0ae62c 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -629,4 +629,11 @@ static inline bool dma_fence_is_container(struct dma_fence *fence)
 	return dma_fence_is_array(fence) || dma_fence_is_chain(fence);
 }
 
+static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
+				       dma_fence_func_t func)
+{
+	INIT_LIST_HEAD(&cb->node);
+	cb->func = func;
+}
+
 #endif /* __LINUX_DMA_FENCE_H */
-- 
2.37.2

[Intel-gfx] [RFC 02/12] dma-fence: Add callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

In preparation of adding a new field to struct dma_fence_cb we will need
an initialization helper for those callers who add callbacks by open-
coding. That will ensure they initialize all the fields so common code
does not get confused by potential garbage in some fields.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c | 10 ++++------
 include/linux/dma-fence.h   |  7 +++++++
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index 41da0da17eba..ea4a1f82c9bf 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -662,21 +662,19 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 	unsigned long flags;
 	int ret = 0;
 
+	__dma_fence_cb_init(cb, func);
+
 	if (WARN_ON(!fence || !func))
 		return -EINVAL;
 
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
-		INIT_LIST_HEAD(&cb->node);
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
 		return -ENOENT;
-	}
 
 	spin_lock_irqsave(fence->lock, flags);
 
 	if (__dma_fence_enable_signaling(fence)) {
-		cb->func = func;
 		list_add_tail(&cb->node, &fence->cb_list);
 	} else {
-		INIT_LIST_HEAD(&cb->node);
 		ret = -ENOENT;
 	}
 
@@ -795,7 +793,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = dma_fence_default_wait_cb;
+	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &fence->cb_list);
 
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 2af328a89468..35933e0ae62c 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -629,4 +629,11 @@ static inline bool dma_fence_is_container(struct dma_fence *fence)
 	return dma_fence_is_array(fence) || dma_fence_is_chain(fence);
 }
 
+static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
+				       dma_fence_func_t func)
+{
+	INIT_LIST_HEAD(&cb->node);
+	cb->func = func;
+}
+
 #endif /* __LINUX_DMA_FENCE_H */
-- 
2.37.2

[RFC 03/12] drm/i915: Use fence callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added initialization helper to ensure correct operation
of the common code.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_active.c | 2 +-
 drivers/gpu/drm/i915/i915_active.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c
index a9fea115f2d2..a3ad64583520 100644
--- a/drivers/gpu/drm/i915/i915_active.c
+++ b/drivers/gpu/drm/i915/i915_active.c
@@ -890,7 +890,7 @@ int i915_active_acquire_preallocate_barrier(struct i915_active *ref,
 				goto unwind;
 
 			RCU_INIT_POINTER(node->base.fence, NULL);
-			node->base.cb.func = node_retire;
+			__dma_fence_cb_init(&node->base.cb, node_retire);
 			node->timeline = idx;
 			node->ref = ref;
 		}
diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h
index 7eb44132183a..da0c5b917cb1 100644
--- a/drivers/gpu/drm/i915/i915_active.h
+++ b/drivers/gpu/drm/i915/i915_active.h
@@ -65,7 +65,7 @@ __i915_active_fence_init(struct i915_active_fence *active,
 			 dma_fence_func_t fn)
 {
 	RCU_INIT_POINTER(active->fence, fence);
-	active->cb.func = fn ?: i915_active_noop;
+	__dma_fence_cb_init(&active->cb, fn ?: i915_active_noop);
 }
 
 #define INIT_ACTIVE_FENCE(A) \
-- 
2.37.2

[Intel-gfx] [RFC 03/12] drm/i915: Use fence callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added initialization helper to ensure correct operation
of the common code.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_active.c | 2 +-
 drivers/gpu/drm/i915/i915_active.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/i915/i915_active.c b/drivers/gpu/drm/i915/i915_active.c
index a9fea115f2d2..a3ad64583520 100644
--- a/drivers/gpu/drm/i915/i915_active.c
+++ b/drivers/gpu/drm/i915/i915_active.c
@@ -890,7 +890,7 @@ int i915_active_acquire_preallocate_barrier(struct i915_active *ref,
 				goto unwind;
 
 			RCU_INIT_POINTER(node->base.fence, NULL);
-			node->base.cb.func = node_retire;
+			__dma_fence_cb_init(&node->base.cb, node_retire);
 			node->timeline = idx;
 			node->ref = ref;
 		}
diff --git a/drivers/gpu/drm/i915/i915_active.h b/drivers/gpu/drm/i915/i915_active.h
index 7eb44132183a..da0c5b917cb1 100644
--- a/drivers/gpu/drm/i915/i915_active.h
+++ b/drivers/gpu/drm/i915/i915_active.h
@@ -65,7 +65,7 @@ __i915_active_fence_init(struct i915_active_fence *active,
 			 dma_fence_func_t fn)
 {
 	RCU_INIT_POINTER(active->fence, fence);
-	active->cb.func = fn ?: i915_active_noop;
+	__dma_fence_cb_init(&active->cb, fn ?: i915_active_noop);
 }
 
 #define INIT_ACTIVE_FENCE(A) \
-- 
2.37.2

[RFC 04/12] drm/vmwgfx: Use fence callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added initialization helper to ensure correct operation
of the common code.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Zack Rusin <zackr@vmware.com>
---
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
index 2a0cda324703..0306596765e5 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
@@ -200,7 +200,7 @@ static long vmw_fence_wait(struct dma_fence *f, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = vmwgfx_wait_cb;
+	__dma_fence_cb_init(&cb.base, vmwgfx_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &f->cb_list);
 
-- 
2.37.2

[Intel-gfx] [RFC 04/12] drm/vmwgfx: Use fence callback initialization helper

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added initialization helper to ensure correct operation
of the common code.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Zack Rusin <zackr@vmware.com>
---
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
index 2a0cda324703..0306596765e5 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_fence.c
@@ -200,7 +200,7 @@ static long vmw_fence_wait(struct dma_fence *f, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = vmwgfx_wait_cb;
+	__dma_fence_cb_init(&cb.base, vmwgfx_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &f->cb_list);
 
-- 
2.37.2

[RFC 05/12] dma-fence: Track explicit waiters

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Track how many callers are explicity waiting on a fence to signal and
allow querying that via new dma_fence_wait_count() API.

This provides infrastructure on top of which generic "waitboost" concepts
can be implemented by individual drivers. Wait-boosting is any reactive
activity, such as raising the GPU clocks, which happens while there are
active external waiters.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c               | 98 +++++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_engine_pm.c |  1 -
 include/linux/dma-fence.h                 | 15 ++++
 3 files changed, 87 insertions(+), 27 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index ea4a1f82c9bf..bdba5a8e21b1 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -344,6 +344,25 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+static void incr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	__set_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
+}
+
+static void decr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	if (__test_and_clear_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags)) {
+		WARN_ON_ONCE(!fence->waitcount);
+		fence->waitcount--;
+	}
+}
+
 void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
 {
 	lockdep_assert_held(fence->lock);
@@ -363,6 +382,7 @@ __dma_fence_signal__notify(struct dma_fence *fence,
 	lockdep_assert_held(fence->lock);
 
 	list_for_each_entry_safe(cur, tmp, list, node) {
+		decr_wait_count(fence, cur);
 		INIT_LIST_HEAD(&cur->node);
 		cur->func(fence, cur);
 	}
@@ -629,11 +649,44 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 	unsigned long flags;
 
 	spin_lock_irqsave(fence->lock, flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
 	__dma_fence_enable_signaling(fence);
 	spin_unlock_irqrestore(fence->lock, flags);
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
+static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			dma_fence_func_t func, bool wait)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	__dma_fence_cb_init(cb, func);
+
+	if (WARN_ON(!fence || !func))
+		return -EINVAL;
+
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
+		return -ENOENT;
+
+	spin_lock_irqsave(fence->lock, flags);
+
+	if (wait)
+		incr_wait_count(fence, cb);
+
+	if (__dma_fence_enable_signaling(fence)) {
+		list_add_tail(&cb->node, &fence->cb_list);
+	} else {
+		decr_wait_count(fence, cb);
+		ret = -ENOENT;
+	}
+
+	spin_unlock_irqrestore(fence->lock, flags);
+
+	return ret;
+}
+
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
  * is signaled
@@ -659,31 +712,18 @@ EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	unsigned long flags;
-	int ret = 0;
-
-	__dma_fence_cb_init(cb, func);
-
-	if (WARN_ON(!fence || !func))
-		return -EINVAL;
-
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
-		return -ENOENT;
-
-	spin_lock_irqsave(fence->lock, flags);
-
-	if (__dma_fence_enable_signaling(fence)) {
-		list_add_tail(&cb->node, &fence->cb_list);
-	} else {
-		ret = -ENOENT;
-	}
-
-	spin_unlock_irqrestore(fence->lock, flags);
-
-	return ret;
+	return add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func)
+{
+	return add_callback(fence, cb, func, true);
+}
+EXPORT_SYMBOL(dma_fence_add_wait_callback);
+
 /**
  * dma_fence_get_status - returns the status upon completion
  * @fence: the dma_fence to query
@@ -736,8 +776,10 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb)
 	spin_lock_irqsave(fence->lock, flags);
 
 	ret = !list_empty(&cb->node);
-	if (ret)
+	if (ret) {
+		decr_wait_count(fence, cb);
 		list_del_init(&cb->node);
+	}
 
 	spin_unlock_irqrestore(fence->lock, flags);
 
@@ -795,6 +837,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 
 	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
+	incr_wait_count(fence, &cb.base);
 	list_add(&cb.base.node, &fence->cb_list);
 
 	while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) {
@@ -811,8 +854,10 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 			ret = -ERESTARTSYS;
 	}
 
-	if (!list_empty(&cb.base.node))
+	if (!list_empty(&cb.base.node)) {
+		decr_wait_count(fence, &cb.base);
 		list_del(&cb.base.node);
+	}
 	__set_current_state(TASK_RUNNING);
 
 out:
@@ -890,8 +935,8 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count,
 		struct dma_fence *fence = fences[i];
 
 		cb[i].task = current;
-		if (dma_fence_add_callback(fence, &cb[i].base,
-					   dma_fence_default_wait_cb)) {
+		if (dma_fence_add_wait_callback(fence, &cb[i].base,
+						dma_fence_default_wait_cb)) {
 			/* This fence is already signaled */
 			if (idx)
 				*idx = i;
@@ -972,6 +1017,7 @@ dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
 	fence->context = context;
 	fence->seqno = seqno;
 	fence->flags = 0UL;
+	fence->waitcount = 0;
 	fence->error = 0;
 
 	trace_dma_fence_init(fence);
diff --git a/drivers/gpu/drm/i915/gt/intel_engine_pm.c b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
index e971b153fda9..2693a0151a6b 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_pm.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
@@ -218,7 +218,6 @@ static bool switch_to_kernel_context(struct intel_engine_cs *engine)
 		 * until the background request retirement running every
 		 * second or two).
 		 */
-		BUILD_BUG_ON(sizeof(rq->duration) > sizeof(rq->submitq));
 		dma_fence_add_callback(&rq->fence, &rq->duration.cb, duration);
 		rq->duration.emitted = ktime_get();
 	}
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 35933e0ae62c..2b696f9de276 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -92,6 +92,7 @@ struct dma_fence {
 	u64 seqno;
 	unsigned long flags;
 	struct kref refcount;
+	unsigned int waitcount;
 	int error;
 };
 
@@ -116,6 +117,11 @@ typedef void (*dma_fence_func_t)(struct dma_fence *fence,
 struct dma_fence_cb {
 	struct list_head node;
 	dma_fence_func_t func;
+	unsigned long flags;
+};
+
+enum dma_fence_cb_flag_bits {
+	DMA_FENCE_CB_FLAG_WAITCOUNT_BIT,
 };
 
 /**
@@ -381,6 +387,9 @@ signed long dma_fence_default_wait(struct dma_fence *fence,
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func);
 bool dma_fence_remove_callback(struct dma_fence *fence,
 			       struct dma_fence_cb *cb);
 void dma_fence_enable_sw_signaling(struct dma_fence *fence);
@@ -532,6 +541,11 @@ static inline int dma_fence_get_status_locked(struct dma_fence *fence)
 
 int dma_fence_get_status(struct dma_fence *fence);
 
+static inline unsigned int dma_fence_wait_count(struct dma_fence *fence)
+{
+	return fence->waitcount;
+}
+
 /**
  * dma_fence_set_error - flag an error condition on the fence
  * @fence: the dma_fence
@@ -634,6 +648,7 @@ static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
 {
 	INIT_LIST_HEAD(&cb->node);
 	cb->func = func;
+	cb->flags = 0;
 }
 
 #endif /* __LINUX_DMA_FENCE_H */
-- 
2.37.2

[Intel-gfx] [RFC 05/12] dma-fence: Track explicit waiters

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Track how many callers are explicity waiting on a fence to signal and
allow querying that via new dma_fence_wait_count() API.

This provides infrastructure on top of which generic "waitboost" concepts
can be implemented by individual drivers. Wait-boosting is any reactive
activity, such as raising the GPU clocks, which happens while there are
active external waiters.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c               | 98 +++++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_engine_pm.c |  1 -
 include/linux/dma-fence.h                 | 15 ++++
 3 files changed, 87 insertions(+), 27 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index ea4a1f82c9bf..bdba5a8e21b1 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -344,6 +344,25 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+static void incr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	__set_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
+}
+
+static void decr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	if (__test_and_clear_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags)) {
+		WARN_ON_ONCE(!fence->waitcount);
+		fence->waitcount--;
+	}
+}
+
 void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
 {
 	lockdep_assert_held(fence->lock);
@@ -363,6 +382,7 @@ __dma_fence_signal__notify(struct dma_fence *fence,
 	lockdep_assert_held(fence->lock);
 
 	list_for_each_entry_safe(cur, tmp, list, node) {
+		decr_wait_count(fence, cur);
 		INIT_LIST_HEAD(&cur->node);
 		cur->func(fence, cur);
 	}
@@ -629,11 +649,44 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 	unsigned long flags;
 
 	spin_lock_irqsave(fence->lock, flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
 	__dma_fence_enable_signaling(fence);
 	spin_unlock_irqrestore(fence->lock, flags);
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
+static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			dma_fence_func_t func, bool wait)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	__dma_fence_cb_init(cb, func);
+
+	if (WARN_ON(!fence || !func))
+		return -EINVAL;
+
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
+		return -ENOENT;
+
+	spin_lock_irqsave(fence->lock, flags);
+
+	if (wait)
+		incr_wait_count(fence, cb);
+
+	if (__dma_fence_enable_signaling(fence)) {
+		list_add_tail(&cb->node, &fence->cb_list);
+	} else {
+		decr_wait_count(fence, cb);
+		ret = -ENOENT;
+	}
+
+	spin_unlock_irqrestore(fence->lock, flags);
+
+	return ret;
+}
+
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
  * is signaled
@@ -659,31 +712,18 @@ EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	unsigned long flags;
-	int ret = 0;
-
-	__dma_fence_cb_init(cb, func);
-
-	if (WARN_ON(!fence || !func))
-		return -EINVAL;
-
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
-		return -ENOENT;
-
-	spin_lock_irqsave(fence->lock, flags);
-
-	if (__dma_fence_enable_signaling(fence)) {
-		list_add_tail(&cb->node, &fence->cb_list);
-	} else {
-		ret = -ENOENT;
-	}
-
-	spin_unlock_irqrestore(fence->lock, flags);
-
-	return ret;
+	return add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func)
+{
+	return add_callback(fence, cb, func, true);
+}
+EXPORT_SYMBOL(dma_fence_add_wait_callback);
+
 /**
  * dma_fence_get_status - returns the status upon completion
  * @fence: the dma_fence to query
@@ -736,8 +776,10 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb)
 	spin_lock_irqsave(fence->lock, flags);
 
 	ret = !list_empty(&cb->node);
-	if (ret)
+	if (ret) {
+		decr_wait_count(fence, cb);
 		list_del_init(&cb->node);
+	}
 
 	spin_unlock_irqrestore(fence->lock, flags);
 
@@ -795,6 +837,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 
 	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
+	incr_wait_count(fence, &cb.base);
 	list_add(&cb.base.node, &fence->cb_list);
 
 	while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) {
@@ -811,8 +854,10 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 			ret = -ERESTARTSYS;
 	}
 
-	if (!list_empty(&cb.base.node))
+	if (!list_empty(&cb.base.node)) {
+		decr_wait_count(fence, &cb.base);
 		list_del(&cb.base.node);
+	}
 	__set_current_state(TASK_RUNNING);
 
 out:
@@ -890,8 +935,8 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count,
 		struct dma_fence *fence = fences[i];
 
 		cb[i].task = current;
-		if (dma_fence_add_callback(fence, &cb[i].base,
-					   dma_fence_default_wait_cb)) {
+		if (dma_fence_add_wait_callback(fence, &cb[i].base,
+						dma_fence_default_wait_cb)) {
 			/* This fence is already signaled */
 			if (idx)
 				*idx = i;
@@ -972,6 +1017,7 @@ dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
 	fence->context = context;
 	fence->seqno = seqno;
 	fence->flags = 0UL;
+	fence->waitcount = 0;
 	fence->error = 0;
 
 	trace_dma_fence_init(fence);
diff --git a/drivers/gpu/drm/i915/gt/intel_engine_pm.c b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
index e971b153fda9..2693a0151a6b 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_pm.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_pm.c
@@ -218,7 +218,6 @@ static bool switch_to_kernel_context(struct intel_engine_cs *engine)
 		 * until the background request retirement running every
 		 * second or two).
 		 */
-		BUILD_BUG_ON(sizeof(rq->duration) > sizeof(rq->submitq));
 		dma_fence_add_callback(&rq->fence, &rq->duration.cb, duration);
 		rq->duration.emitted = ktime_get();
 	}
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 35933e0ae62c..2b696f9de276 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -92,6 +92,7 @@ struct dma_fence {
 	u64 seqno;
 	unsigned long flags;
 	struct kref refcount;
+	unsigned int waitcount;
 	int error;
 };
 
@@ -116,6 +117,11 @@ typedef void (*dma_fence_func_t)(struct dma_fence *fence,
 struct dma_fence_cb {
 	struct list_head node;
 	dma_fence_func_t func;
+	unsigned long flags;
+};
+
+enum dma_fence_cb_flag_bits {
+	DMA_FENCE_CB_FLAG_WAITCOUNT_BIT,
 };
 
 /**
@@ -381,6 +387,9 @@ signed long dma_fence_default_wait(struct dma_fence *fence,
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func);
 bool dma_fence_remove_callback(struct dma_fence *fence,
 			       struct dma_fence_cb *cb);
 void dma_fence_enable_sw_signaling(struct dma_fence *fence);
@@ -532,6 +541,11 @@ static inline int dma_fence_get_status_locked(struct dma_fence *fence)
 
 int dma_fence_get_status(struct dma_fence *fence);
 
+static inline unsigned int dma_fence_wait_count(struct dma_fence *fence)
+{
+	return fence->waitcount;
+}
+
 /**
  * dma_fence_set_error - flag an error condition on the fence
  * @fence: the dma_fence
@@ -634,6 +648,7 @@ static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
 {
 	INIT_LIST_HEAD(&cb->node);
 	cb->func = func;
+	cb->flags = 0;
 }
 
 #endif /* __LINUX_DMA_FENCE_H */
-- 
2.37.2

[RFC 06/12] dma-fence: Export __dma_fence_add_callback

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

...

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c | 9 +++++----
 include/linux/dma-fence.h   | 3 +++
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index bdba5a8e21b1..5607acdb6ccf 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -656,8 +656,8 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
-static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
-			dma_fence_func_t func, bool wait)
+int __dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait)
 {
 	unsigned long flags;
 	int ret = 0;
@@ -686,6 +686,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 
 	return ret;
 }
+EXPORT_SYMBOL(__dma_fence_add_callback);
 
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
@@ -712,7 +713,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, false);
+	return __dma_fence_add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
@@ -720,7 +721,7 @@ int dma_fence_add_wait_callback(struct dma_fence *fence,
 				struct dma_fence_cb *cb,
 				dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, true);
+	return __dma_fence_add_callback(fence, cb, func, true);
 }
 EXPORT_SYMBOL(dma_fence_add_wait_callback);
 
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 2b696f9de276..1f41d60ce6f7 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -384,6 +384,9 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp);
 signed long dma_fence_default_wait(struct dma_fence *fence,
 				   bool intr, signed long timeout);
+int __dma_fence_add_callback(struct dma_fence *fence,
+			     struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait);
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
-- 
2.37.2

[Intel-gfx] [RFC 06/12] dma-fence: Export __dma_fence_add_callback

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

...

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence.c | 9 +++++----
 include/linux/dma-fence.h   | 3 +++
 2 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c
index bdba5a8e21b1..5607acdb6ccf 100644
--- a/drivers/dma-buf/dma-fence.c
+++ b/drivers/dma-buf/dma-fence.c
@@ -656,8 +656,8 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
-static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
-			dma_fence_func_t func, bool wait)
+int __dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait)
 {
 	unsigned long flags;
 	int ret = 0;
@@ -686,6 +686,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 
 	return ret;
 }
+EXPORT_SYMBOL(__dma_fence_add_callback);
 
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
@@ -712,7 +713,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, false);
+	return __dma_fence_add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
@@ -720,7 +721,7 @@ int dma_fence_add_wait_callback(struct dma_fence *fence,
 				struct dma_fence_cb *cb,
 				dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, true);
+	return __dma_fence_add_callback(fence, cb, func, true);
 }
 EXPORT_SYMBOL(dma_fence_add_wait_callback);
 
diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h
index 2b696f9de276..1f41d60ce6f7 100644
--- a/include/linux/dma-fence.h
+++ b/include/linux/dma-fence.h
@@ -384,6 +384,9 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp);
 signed long dma_fence_default_wait(struct dma_fence *fence,
 				   bool intr, signed long timeout);
+int __dma_fence_add_callback(struct dma_fence *fence,
+			     struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait);
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
-- 
2.37.2

[RFC 07/12] dma-fence-array: Propagate wait status to contained fences

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As signaling is enabled on the container fence we need to propagate any
external waiting status to individual fences in order to enable owning
drivers see it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence-array.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/dma-buf/dma-fence-array.c b/drivers/dma-buf/dma-fence-array.c
index 5c8a7084577b..e732adc230d1 100644
--- a/drivers/dma-buf/dma-fence-array.c
+++ b/drivers/dma-buf/dma-fence-array.c
@@ -84,8 +84,9 @@ static bool dma_fence_array_enable_signaling(struct dma_fence *fence)
 		 * insufficient).
 		 */
 		dma_fence_get(&array->base);
-		if (dma_fence_add_callback(array->fences[i], &cb[i].cb,
-					   dma_fence_array_cb_func)) {
+		if (__dma_fence_add_callback(array->fences[i], &cb[i].cb,
+					     dma_fence_array_cb_func,
+					     fence->waitcount > 0)) {
 			int error = array->fences[i]->error;
 
 			dma_fence_array_set_pending_error(array, error);
-- 
2.37.2

[Intel-gfx] [RFC 07/12] dma-fence-array: Propagate wait status to contained fences

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As signaling is enabled on the container fence we need to propagate any
external waiting status to individual fences in order to enable owning
drivers see it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence-array.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/dma-buf/dma-fence-array.c b/drivers/dma-buf/dma-fence-array.c
index 5c8a7084577b..e732adc230d1 100644
--- a/drivers/dma-buf/dma-fence-array.c
+++ b/drivers/dma-buf/dma-fence-array.c
@@ -84,8 +84,9 @@ static bool dma_fence_array_enable_signaling(struct dma_fence *fence)
 		 * insufficient).
 		 */
 		dma_fence_get(&array->base);
-		if (dma_fence_add_callback(array->fences[i], &cb[i].cb,
-					   dma_fence_array_cb_func)) {
+		if (__dma_fence_add_callback(array->fences[i], &cb[i].cb,
+					     dma_fence_array_cb_func,
+					     fence->waitcount > 0)) {
 			int error = array->fences[i]->error;
 
 			dma_fence_array_set_pending_error(array, error);
-- 
2.37.2

[RFC 08/12] dma-fence-chain: Propagate wait status to contained fences

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As signaling is enabled on the container fence we need to propagate any
external waiting status to individual fences in order to enable owning
drivers see it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence-chain.c | 22 ++++++++++++++++------
 include/linux/dma-fence-chain.h   |  1 +
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/dma-buf/dma-fence-chain.c b/drivers/dma-buf/dma-fence-chain.c
index a0d920576ba6..ef6a5988c8aa 100644
--- a/drivers/dma-buf/dma-fence-chain.c
+++ b/drivers/dma-buf/dma-fence-chain.c
@@ -145,20 +145,30 @@ static void dma_fence_chain_cb(struct dma_fence *f, struct dma_fence_cb *cb)
 static bool dma_fence_chain_enable_signaling(struct dma_fence *fence)
 {
 	struct dma_fence_chain *head = to_dma_fence_chain(fence);
+	struct dma_fence *callback = NULL;
 
 	dma_fence_get(&head->base);
 	dma_fence_chain_for_each(fence, &head->base) {
 		struct dma_fence *f = dma_fence_chain_contained(fence);
 
-		dma_fence_get(f);
-		if (!dma_fence_add_callback(f, &head->cb, dma_fence_chain_cb)) {
+		if (!callback) {
+			dma_fence_get(f);
+			if (!dma_fence_add_callback(f, &head->cb,
+						    dma_fence_chain_cb))
+				callback = f;
+			else
+				dma_fence_put(f);
+		} else if (head->base.waitcount && !head->waitcount) {
+			dma_fence_enable_sw_signaling(f);
+		} else {
 			dma_fence_put(fence);
-			return true;
+			break;
 		}
-		dma_fence_put(f);
 	}
-	dma_fence_put(&head->base);
-	return false;
+	head->waitcount = head->base.waitcount;
+	if (!callback)
+		dma_fence_put(&head->base);
+	return callback;
 }
 
 static bool dma_fence_chain_signaled(struct dma_fence *fence)
diff --git a/include/linux/dma-fence-chain.h b/include/linux/dma-fence-chain.h
index 4bdf0b96da28..349b882d31ea 100644
--- a/include/linux/dma-fence-chain.h
+++ b/include/linux/dma-fence-chain.h
@@ -25,6 +25,7 @@
 struct dma_fence_chain {
 	struct dma_fence base;
 	struct dma_fence __rcu *prev;
+	bool waitcount;
 	u64 prev_seqno;
 	struct dma_fence *fence;
 	union {
-- 
2.37.2

[Intel-gfx] [RFC 08/12] dma-fence-chain: Propagate wait status to contained fences

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As signaling is enabled on the container fence we need to propagate any
external waiting status to individual fences in order to enable owning
drivers see it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/dma-buf/dma-fence-chain.c | 22 ++++++++++++++++------
 include/linux/dma-fence-chain.h   |  1 +
 2 files changed, 17 insertions(+), 6 deletions(-)

diff --git a/drivers/dma-buf/dma-fence-chain.c b/drivers/dma-buf/dma-fence-chain.c
index a0d920576ba6..ef6a5988c8aa 100644
--- a/drivers/dma-buf/dma-fence-chain.c
+++ b/drivers/dma-buf/dma-fence-chain.c
@@ -145,20 +145,30 @@ static void dma_fence_chain_cb(struct dma_fence *f, struct dma_fence_cb *cb)
 static bool dma_fence_chain_enable_signaling(struct dma_fence *fence)
 {
 	struct dma_fence_chain *head = to_dma_fence_chain(fence);
+	struct dma_fence *callback = NULL;
 
 	dma_fence_get(&head->base);
 	dma_fence_chain_for_each(fence, &head->base) {
 		struct dma_fence *f = dma_fence_chain_contained(fence);
 
-		dma_fence_get(f);
-		if (!dma_fence_add_callback(f, &head->cb, dma_fence_chain_cb)) {
+		if (!callback) {
+			dma_fence_get(f);
+			if (!dma_fence_add_callback(f, &head->cb,
+						    dma_fence_chain_cb))
+				callback = f;
+			else
+				dma_fence_put(f);
+		} else if (head->base.waitcount && !head->waitcount) {
+			dma_fence_enable_sw_signaling(f);
+		} else {
 			dma_fence_put(fence);
-			return true;
+			break;
 		}
-		dma_fence_put(f);
 	}
-	dma_fence_put(&head->base);
-	return false;
+	head->waitcount = head->base.waitcount;
+	if (!callback)
+		dma_fence_put(&head->base);
+	return callback;
 }
 
 static bool dma_fence_chain_signaled(struct dma_fence *fence)
diff --git a/include/linux/dma-fence-chain.h b/include/linux/dma-fence-chain.h
index 4bdf0b96da28..349b882d31ea 100644
--- a/include/linux/dma-fence-chain.h
+++ b/include/linux/dma-fence-chain.h
@@ -25,6 +25,7 @@
 struct dma_fence_chain {
 	struct dma_fence base;
 	struct dma_fence __rcu *prev;
+	bool waitcount;
 	u64 prev_seqno;
 	struct dma_fence *fence;
 	union {
-- 
2.37.2

[RFC 09/12] drm/syncobj: Mark syncobj waits as external waiters

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added dma-fence tracking of explicit waiters.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/drm_syncobj.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/drm_syncobj.c b/drivers/gpu/drm/drm_syncobj.c
index 0c2be8360525..776b90774a64 100644
--- a/drivers/gpu/drm/drm_syncobj.c
+++ b/drivers/gpu/drm/drm_syncobj.c
@@ -1065,9 +1065,9 @@ static signed long drm_syncobj_array_wait_timeout(struct drm_syncobj **syncobjs,
 			if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) ||
 			    dma_fence_is_signaled(fence) ||
 			    (!entries[i].fence_cb.func &&
-			     dma_fence_add_callback(fence,
-						    &entries[i].fence_cb,
-						    syncobj_wait_fence_func))) {
+			     dma_fence_add_wait_callback(fence,
+							 &entries[i].fence_cb,
+							 syncobj_wait_fence_func))) {
 				/* The fence has been signaled */
 				if (flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL) {
 					signaled_count++;
-- 
2.37.2

[Intel-gfx] [RFC 09/12] drm/syncobj: Mark syncobj waits as external waiters

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added dma-fence tracking of explicit waiters.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/drm_syncobj.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/drm_syncobj.c b/drivers/gpu/drm/drm_syncobj.c
index 0c2be8360525..776b90774a64 100644
--- a/drivers/gpu/drm/drm_syncobj.c
+++ b/drivers/gpu/drm/drm_syncobj.c
@@ -1065,9 +1065,9 @@ static signed long drm_syncobj_array_wait_timeout(struct drm_syncobj **syncobjs,
 			if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) ||
 			    dma_fence_is_signaled(fence) ||
 			    (!entries[i].fence_cb.func &&
-			     dma_fence_add_callback(fence,
-						    &entries[i].fence_cb,
-						    syncobj_wait_fence_func))) {
+			     dma_fence_add_wait_callback(fence,
+							 &entries[i].fence_cb,
+							 syncobj_wait_fence_func))) {
 				/* The fence has been signaled */
 				if (flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL) {
 					signaled_count++;
-- 
2.37.2

[RFC 10/12] drm/i915: Waitboost external waits

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Userspace waits coming via the drm_syncobj route have so far been
bypassing the waitboost mechanism.

Use the previously added dma-fence wait tracking API and apply the
same waitboosting logic which applies to other entry points.

This should fix the perfomance regressions experience by clvk and
similar userspace which relies on drm_syncobj.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index 630a732aaecc..42b04cced6f0 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -93,7 +93,12 @@ static bool i915_fence_signaled(struct dma_fence *fence)
 
 static bool i915_fence_enable_signaling(struct dma_fence *fence)
 {
-	return i915_request_enable_breadcrumb(to_request(fence));
+	struct i915_request *rq = to_request(fence);
+
+	if (dma_fence_wait_count(&rq->fence) && !i915_request_started(rq))
+		intel_rps_boost(rq);
+
+	return i915_request_enable_breadcrumb(rq);
 }
 
 static signed long i915_fence_wait(struct dma_fence *fence,
-- 
2.37.2

[Intel-gfx] [RFC 10/12] drm/i915: Waitboost external waits

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Userspace waits coming via the drm_syncobj route have so far been
bypassing the waitboost mechanism.

Use the previously added dma-fence wait tracking API and apply the
same waitboosting logic which applies to other entry points.

This should fix the perfomance regressions experience by clvk and
similar userspace which relies on drm_syncobj.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index 630a732aaecc..42b04cced6f0 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -93,7 +93,12 @@ static bool i915_fence_signaled(struct dma_fence *fence)
 
 static bool i915_fence_enable_signaling(struct dma_fence *fence)
 {
-	return i915_request_enable_breadcrumb(to_request(fence));
+	struct i915_request *rq = to_request(fence);
+
+	if (dma_fence_wait_count(&rq->fence) && !i915_request_started(rq))
+		intel_rps_boost(rq);
+
+	return i915_request_enable_breadcrumb(rq);
 }
 
 static signed long i915_fence_wait(struct dma_fence *fence,
-- 
2.37.2

[RFC 11/12] drm/i915: Mark waits as explicit

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added dma-fence API to mark the direct i915 waits as
explicit. This has no significant effect apart from following the new
pattern.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index 42b04cced6f0..dd8a9045086a 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -2045,7 +2045,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
-	if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake))
+	if (dma_fence_add_wait_callback(&rq->fence, &wait.cb,
+					request_wait_wake))
 		goto out;
 
 	/*
-- 
2.37.2

[Intel-gfx] [RFC 11/12] drm/i915: Mark waits as explicit

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the previously added dma-fence API to mark the direct i915 waits as
explicit. This has no significant effect apart from following the new
pattern.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index 42b04cced6f0..dd8a9045086a 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -2045,7 +2045,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
-	if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake))
+	if (dma_fence_add_wait_callback(&rq->fence, &wait.cb,
+					request_wait_wake))
 		goto out;
 
 	/*
-- 
2.37.2

[RFC 12/12] drm/i915: Wait boost requests waited upon by others

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the newly added dma-fence API to apply waitboost not only requests
which have been marked with I915_WAIT_PRIORITY by i915, but which may be
waited upon by others (such as for instance buffer sharing in multi-GPU
scenarios).

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index dd8a9045086a..c4adf34016d0 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -2041,7 +2041,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 	 * but at a cost of spending more power processing the workload
 	 * (bad for battery).
 	 */
-	if (flags & I915_WAIT_PRIORITY && !i915_request_started(rq))
+	if (((flags & I915_WAIT_PRIORITY) || dma_fence_wait_count(&rq->fence))
+	    && !i915_request_started(rq))
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
-- 
2.37.2

[Intel-gfx] [RFC 12/12] drm/i915: Wait boost requests waited upon by others

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Use the newly added dma-fence API to apply waitboost not only requests
which have been marked with I915_WAIT_PRIORITY by i915, but which may be
waited upon by others (such as for instance buffer sharing in multi-GPU
scenarios).

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/i915_request.c b/drivers/gpu/drm/i915/i915_request.c
index dd8a9045086a..c4adf34016d0 100644
--- a/drivers/gpu/drm/i915/i915_request.c
+++ b/drivers/gpu/drm/i915/i915_request.c
@@ -2041,7 +2041,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 	 * but at a cost of spending more power processing the workload
 	 * (bad for battery).
 	 */
-	if (flags & I915_WAIT_PRIORITY && !i915_request_started(rq))
+	if (((flags & I915_WAIT_PRIORITY) || dma_fence_wait_count(&rq->fence))
+	    && !i915_request_started(rq))
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
-- 
2.37.2

[Intel-gfx] ✗ Fi.CI.BAT: failure for Waitboost drm syncobj waits (rev4)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 7653 bytes --]

== Series Details ==

Series: Waitboost drm syncobj waits (rev4)
URL   : https://patchwork.freedesktop.org/series/113846/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_12821 -> Patchwork_113846v4
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with Patchwork_113846v4 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_113846v4, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/index.html

Participating hosts (36 -> 36)
------------------------------

  Additional (1): fi-kbl-soraka 
  Missing    (1): fi-snb-2520m 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in Patchwork_113846v4:

### IGT changes ###

#### Possible regressions ####

  * igt@i915_suspend@basic-s2idle-without-i915:
    - fi-hsw-4770:        [PASS][1] -> [ABORT][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-hsw-4770/igt@i915_suspend@basic-s2idle-without-i915.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-hsw-4770/igt@i915_suspend@basic-s2idle-without-i915.html

  
Known issues
------------

  Here are the changes found in Patchwork_113846v4 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_huc_copy@huc-copy:
    - fi-kbl-soraka:      NOTRUN -> [SKIP][3] ([fdo#109271] / [i915#2190])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@gem_huc_copy@huc-copy.html

  * igt@gem_lmem_swapping@basic:
    - fi-kbl-soraka:      NOTRUN -> [SKIP][4] ([fdo#109271] / [i915#4613]) +3 similar issues
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@gem_lmem_swapping@basic.html

  * igt@i915_selftest@live@execlists:
    - fi-bsw-n3050:       [PASS][5] -> [ABORT][6] ([i915#7911])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-bsw-n3050/igt@i915_selftest@live@execlists.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-bsw-n3050/igt@i915_selftest@live@execlists.html
    - fi-bsw-nick:        [PASS][7] -> [ABORT][8] ([i915#7911] / [i915#7913])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-bsw-nick/igt@i915_selftest@live@execlists.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-bsw-nick/igt@i915_selftest@live@execlists.html

  * igt@i915_selftest@live@gem_contexts:
    - fi-kbl-soraka:      NOTRUN -> [INCOMPLETE][9] ([i915#7913])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@i915_selftest@live@gem_contexts.html

  * igt@i915_selftest@live@gt_pm:
    - fi-kbl-soraka:      NOTRUN -> [DMESG-FAIL][10] ([i915#1886])
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@i915_selftest@live@gt_pm.html

  * igt@i915_selftest@live@hangcheck:
    - fi-skl-guc:         [PASS][11] -> [DMESG-WARN][12] ([i915#8073])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-skl-guc/igt@i915_selftest@live@hangcheck.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-skl-guc/igt@i915_selftest@live@hangcheck.html

  * igt@i915_selftest@live@reset:
    - bat-rpls-2:         [PASS][13] -> [ABORT][14] ([i915#4983] / [i915#7913])
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-2/igt@i915_selftest@live@reset.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-2/igt@i915_selftest@live@reset.html

  * igt@i915_selftest@live@slpc:
    - bat-rpls-1:         [PASS][15] -> [DMESG-FAIL][16] ([i915#6367])
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-1/igt@i915_selftest@live@slpc.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@i915_selftest@live@slpc.html

  * igt@kms_chamelium_frames@hdmi-crc-fast:
    - fi-kbl-soraka:      NOTRUN -> [SKIP][17] ([fdo#109271]) +16 similar issues
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@kms_chamelium_frames@hdmi-crc-fast.html

  * igt@kms_chamelium_hpd@common-hpd-after-suspend:
    - bat-rpls-1:         NOTRUN -> [SKIP][18] ([i915#7828])
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@kms_chamelium_hpd@common-hpd-after-suspend.html

  * igt@kms_pipe_crc_basic@read-crc:
    - bat-adlp-9:         NOTRUN -> [SKIP][19] ([i915#3546]) +1 similar issue
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-adlp-9/igt@kms_pipe_crc_basic@read-crc.html

  * igt@kms_pipe_crc_basic@suspend-read-crc:
    - bat-rpls-1:         NOTRUN -> [SKIP][20] ([i915#1845])
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@kms_pipe_crc_basic@suspend-read-crc.html

  
#### Possible fixes ####

  * igt@gem_exec_suspend@basic-s3@smem:
    - bat-rpls-1:         [ABORT][21] ([i915#6687] / [i915#7978]) -> [PASS][22]
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-1/igt@gem_exec_suspend@basic-s3@smem.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@gem_exec_suspend@basic-s3@smem.html

  
  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#1886]: https://gitlab.freedesktop.org/drm/intel/issues/1886
  [i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
  [i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
  [i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
  [i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
  [i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
  [i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
  [i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
  [i915#7911]: https://gitlab.freedesktop.org/drm/intel/issues/7911
  [i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
  [i915#7978]: https://gitlab.freedesktop.org/drm/intel/issues/7978
  [i915#8073]: https://gitlab.freedesktop.org/drm/intel/issues/8073


Build changes
-------------

  * Linux: CI_DRM_12821 -> Patchwork_113846v4

  CI-20190529: 20190529
  CI_DRM_12821: 24f94240c4bca70cadfd00528ffd56c3049e5f58 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_7182: b25ca07c7a75bfda3358c3450810bc023dd7cee9 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_113846v4: 24f94240c4bca70cadfd00528ffd56c3049e5f58 @ git://anongit.freedesktop.org/gfx-ci/linux


### Linux commits

23685a131d74 drm/i915: Wait boost requests waited upon by others
5b86272fb7f4 drm/i915: Mark waits as explicit
11b69ba30cf5 drm/i915: Waitboost external waits
e0a5f922af61 drm/syncobj: Mark syncobj waits as external waiters
be104075d13d dma-fence-chain: Propagate wait status to contained fences
0a24463f803c dma-fence-array: Propagate wait status to contained fences
f64808b8ce78 dma-fence: Export __dma_fence_add_callback
c27b7561c45a dma-fence: Track explicit waiters
212098e25a8d drm/vmwgfx: Use fence callback initialization helper
5183a52e9ee2 drm/i915: Use fence callback initialization helper
3c92d9f13b31 dma-fence: Add callback initialization helper
9ba9823a5aae dma-fence: Move i915 helpers into common

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/index.html

[-- Attachment #2: Type: text/html, Size: 9026 bytes --]