[PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Compute workloads are inherently not pre-emptible on current hardware.
Thus the pre-emption timeout was disabled as a workaround to prevent
unwanted resets. Instead, the hang detection was left to the heartbeat
and its (longer) timeout. This is undesirable with GuC submission as
the heartbeat is a full GT reset rather than a per engine reset and so
is much more destructive. Instead, just bump the pre-emption timeout
to a big value. Also, update the heartbeat to allow such a long
pre-emption delay in the final heartbeat period.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>


John Harrison (3):
  drm/i915/guc: Limit scheduling properties to avoid overflow
  drm/i915/gt: Make the heartbeat play nice with long pre-emption
    timeouts
  drm/i915: Improve long running OCL w/a for GuC submission

 drivers/gpu/drm/i915/gt/intel_engine_cs.c     | 37 +++++++++++++++++--
 .../gpu/drm/i915/gt/intel_engine_heartbeat.c  | 16 ++++++++
 drivers/gpu/drm/i915/gt/sysfs_engines.c       | 14 +++++++
 drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h   |  9 +++++
 4 files changed, 73 insertions(+), 3 deletions(-)

-- 
2.25.1

[Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Compute workloads are inherently not pre-emptible on current hardware.
Thus the pre-emption timeout was disabled as a workaround to prevent
unwanted resets. Instead, the hang detection was left to the heartbeat
and its (longer) timeout. This is undesirable with GuC submission as
the heartbeat is a full GT reset rather than a per engine reset and so
is much more destructive. Instead, just bump the pre-emption timeout
to a big value. Also, update the heartbeat to allow such a long
pre-emption delay in the final heartbeat period.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>


John Harrison (3):
  drm/i915/guc: Limit scheduling properties to avoid overflow
  drm/i915/gt: Make the heartbeat play nice with long pre-emption
    timeouts
  drm/i915: Improve long running OCL w/a for GuC submission

 drivers/gpu/drm/i915/gt/intel_engine_cs.c     | 37 +++++++++++++++++--
 .../gpu/drm/i915/gt/intel_engine_heartbeat.c  | 16 ++++++++
 drivers/gpu/drm/i915/gt/sysfs_engines.c       | 14 +++++++
 drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h   |  9 +++++
 4 files changed, 73 insertions(+), 3 deletions(-)

-- 
2.25.1

[PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

GuC converts the pre-emption timeout and timeslice quantum values into
clock ticks internally. That significantly reduces the point of 32bit
overflow. On current platforms, worst case scenario is approximately
110 seconds. Rather than allowing the user to set higher values and
then get confused by early timeouts, add limits when setting these
values.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
 drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
 drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
 3 files changed, 38 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index e53008b4dd05..2a1e9f36e6f5 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
 	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
 		engine->props.preempt_timeout_ms = 0;
 
+	/* Cap timeouts to prevent overflow inside GuC */
+	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
+		if (engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
+			drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %d to prevent possibly overflow\n",
+				 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
+			engine->props.timeslice_duration_ms = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
+		}
+
+		if (engine->props.preempt_timeout_ms > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
+			drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %d to prevent possibly overflow\n",
+				 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
+			engine->props.preempt_timeout_ms = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
+		}
+	}
+
 	engine->defaults = engine->props; /* never to change again */
 
 	engine->context_size = intel_engine_context_size(gt, engine->class);
diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c b/drivers/gpu/drm/i915/gt/sysfs_engines.c
index 967031056202..f57efe026474 100644
--- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
+++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
@@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr,
 	if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
 		return -EINVAL;
 
+	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
+	    duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
+		duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
+		drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %lld to prevent possibly overflow\n",
+			 duration);
+	}
+
 	WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
 
 	if (execlists_active(&engine->execlists))
@@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr,
 	if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
 		return -EINVAL;
 
+	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
+	    timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
+		timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
+		drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %lld to prevent possibly overflow\n",
+			 timeout);
+	}
+
 	WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
 
 	if (READ_ONCE(engine->execlists.pending[0]))
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
index 6a4612a852e2..ad131092f8df 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
@@ -248,6 +248,15 @@ struct guc_lrc_desc {
 
 #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
 
+/*
+ * GuC converts the timeout to clock ticks internally. Different platforms have
+ * different GuC clocks. Thus, the maximum value before overflow is platform
+ * dependent. Current worst case scenario is about 110s. So, limit to 100s to be
+ * safe.
+ */
+#define GUC_POLICY_MAX_EXEC_QUANTUM_MS		(100 * 1000)
+#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS	(100 * 1000)
+
 struct guc_policies {
 	u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
 	/* In micro seconds. How much time to allow before DPC processing is
-- 
2.25.1

[Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

GuC converts the pre-emption timeout and timeslice quantum values into
clock ticks internally. That significantly reduces the point of 32bit
overflow. On current platforms, worst case scenario is approximately
110 seconds. Rather than allowing the user to set higher values and
then get confused by early timeouts, add limits when setting these
values.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
 drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
 drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
 3 files changed, 38 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index e53008b4dd05..2a1e9f36e6f5 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
 	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
 		engine->props.preempt_timeout_ms = 0;
 
+	/* Cap timeouts to prevent overflow inside GuC */
+	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
+		if (engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
+			drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %d to prevent possibly overflow\n",
+				 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
+			engine->props.timeslice_duration_ms = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
+		}
+
+		if (engine->props.preempt_timeout_ms > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
+			drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %d to prevent possibly overflow\n",
+				 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
+			engine->props.preempt_timeout_ms = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
+		}
+	}
+
 	engine->defaults = engine->props; /* never to change again */
 
 	engine->context_size = intel_engine_context_size(gt, engine->class);
diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c b/drivers/gpu/drm/i915/gt/sysfs_engines.c
index 967031056202..f57efe026474 100644
--- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
+++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
@@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr,
 	if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
 		return -EINVAL;
 
+	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
+	    duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
+		duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
+		drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %lld to prevent possibly overflow\n",
+			 duration);
+	}
+
 	WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
 
 	if (execlists_active(&engine->execlists))
@@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr,
 	if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
 		return -EINVAL;
 
+	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
+	    timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
+		timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
+		drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %lld to prevent possibly overflow\n",
+			 timeout);
+	}
+
 	WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
 
 	if (READ_ONCE(engine->execlists.pending[0]))
diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
index 6a4612a852e2..ad131092f8df 100644
--- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
+++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
@@ -248,6 +248,15 @@ struct guc_lrc_desc {
 
 #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
 
+/*
+ * GuC converts the timeout to clock ticks internally. Different platforms have
+ * different GuC clocks. Thus, the maximum value before overflow is platform
+ * dependent. Current worst case scenario is about 110s. So, limit to 100s to be
+ * safe.
+ */
+#define GUC_POLICY_MAX_EXEC_QUANTUM_MS		(100 * 1000)
+#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS	(100 * 1000)
+
 struct guc_policies {
 	u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
 	/* In micro seconds. How much time to allow before DPC processing is
-- 
2.25.1

[PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Compute workloads are inherantly not pre-emptible for long periods on
current hardware. As a workaround for this, the pre-emption timeout
for compute capable engines was disabled. This is undesirable with GuC
submission as it prevents per engine reset of hung contexts. Hence the
next patch will re-enable the timeout but bumped up by an order of
magnititude.

However, the heartbeat might not respect that. Depending upon current
activity, a pre-emption to the heartbeat pulse might not even be
attempted until the last heartbeat period. Which means that only one
period is granted for the pre-emption to occur. With the aforesaid
bump, the pre-emption timeout could be significantly larger than this
heartbeat period.

So adjust the heartbeat code to take the pre-emption timeout into
account. When it reaches the final (high priority) period, it now
ensures the delay before hitting reset is bigger than the pre-emption
timeout.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
index a3698f611f45..72a82a6085e0 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
@@ -22,9 +22,25 @@
 
 static bool next_heartbeat(struct intel_engine_cs *engine)
 {
+	struct i915_request *rq;
 	long delay;
 
 	delay = READ_ONCE(engine->props.heartbeat_interval_ms);
+
+	rq = engine->heartbeat.systole;
+	if (rq && rq->sched.attr.priority >= I915_PRIORITY_BARRIER) {
+		long longer;
+
+		/*
+		 * The final try is at the highest priority possible. Up until now
+		 * a pre-emption might not even have been attempted. So make sure
+		 * this last attempt allows enough time for a pre-emption to occur.
+		 */
+		longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
+		if (longer > delay)
+			delay = longer;
+	}
+
 	if (!delay)
 		return false;
 
-- 
2.25.1

[Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

Compute workloads are inherantly not pre-emptible for long periods on
current hardware. As a workaround for this, the pre-emption timeout
for compute capable engines was disabled. This is undesirable with GuC
submission as it prevents per engine reset of hung contexts. Hence the
next patch will re-enable the timeout but bumped up by an order of
magnititude.

However, the heartbeat might not respect that. Depending upon current
activity, a pre-emption to the heartbeat pulse might not even be
attempted until the last heartbeat period. Which means that only one
period is granted for the pre-emption to occur. With the aforesaid
bump, the pre-emption timeout could be significantly larger than this
heartbeat period.

So adjust the heartbeat code to take the pre-emption timeout into
account. When it reaches the final (high priority) period, it now
ensures the delay before hitting reset is bigger than the pre-emption
timeout.

Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
index a3698f611f45..72a82a6085e0 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
@@ -22,9 +22,25 @@
 
 static bool next_heartbeat(struct intel_engine_cs *engine)
 {
+	struct i915_request *rq;
 	long delay;
 
 	delay = READ_ONCE(engine->props.heartbeat_interval_ms);
+
+	rq = engine->heartbeat.systole;
+	if (rq && rq->sched.attr.priority >= I915_PRIORITY_BARRIER) {
+		long longer;
+
+		/*
+		 * The final try is at the highest priority possible. Up until now
+		 * a pre-emption might not even have been attempted. So make sure
+		 * this last attempt allows enough time for a pre-emption to occur.
+		 */
+		longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
+		if (longer > delay)
+			delay = longer;
+	}
+
 	if (!delay)
 		return false;
 
-- 
2.25.1

[PATCH 3/3] drm/i915: Improve long running OCL w/a for GuC submission

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A workaround was added to the driver to allow OpenCL workloads to run
'forever' by disabling pre-emption on the RCS engine for Gen12.
It is not totally unbound as the heartbeat will kick in eventually
and cause a reset of the hung engine.

However, this does not work well in GuC submission mode. In GuC mode,
the pre-emption timeout is how GuC detects hung contexts and triggers
a per engine reset. Thus, disabling the timeout means also losing all
per engine reset ability. A full GT reset will still occur when the
heartbeat finally expires, but that is a much more destructive and
undesirable mechanism.

The purpose of the workaround is actually to give OpenCL tasks longer
to reach a pre-emption point after a pre-emption request has been
issued. This is necessary because Gen12 does not support mid-thread
pre-emption and OpenCL can have long running threads.

So, rather than disabling the timeout completely, just set it to a
'long' value.

CC: Michal Mrozek <michal.mrozek@intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index 2a1e9f36e6f5..64249301a227 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -385,9 +385,25 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
 	engine->props.timeslice_duration_ms =
 		CONFIG_DRM_I915_TIMESLICE_DURATION;
 
-	/* Override to uninterruptible for OpenCL workloads. */
-	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
-		engine->props.preempt_timeout_ms = 0;
+	/*
+	 * Mid-thread pre-emption is not available in Gen12. Unfortunately,
+	 * some OpenCL workloads run quite long threads. That means they get
+	 * reset due to not pre-empting in a timely manner. So, bump the
+	 * pre-emption timeout value to be much higher for compute engines.
+	 * Using three times the heartbeat period seems long enough for a
+	 * reasonable task to reach a pre-emption point but not so long as to
+	 * allow genuine hangs to go unresolved.
+	 */
+	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS) {
+		unsigned long triple_beat = engine->props.heartbeat_interval_ms * 3;
+
+		if (triple_beat > engine->props.preempt_timeout_ms) {
+			drm_info(&gt->i915->drm, "Bumping pre-emption timeout from %ld to %ld on %s to allow slow compute pre-emption\n",
+				 engine->props.preempt_timeout_ms, triple_beat, engine->name);
+
+			engine->props.preempt_timeout_ms = triple_beat;
+		}
+	}
 
 	/* Cap timeouts to prevent overflow inside GuC */
 	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
-- 
2.25.1

[Intel-gfx] [PATCH 3/3] drm/i915: Improve long running OCL w/a for GuC submission

[John.C.Harrison]

From: John Harrison <John.C.Harrison@Intel.com>

A workaround was added to the driver to allow OpenCL workloads to run
'forever' by disabling pre-emption on the RCS engine for Gen12.
It is not totally unbound as the heartbeat will kick in eventually
and cause a reset of the hung engine.

However, this does not work well in GuC submission mode. In GuC mode,
the pre-emption timeout is how GuC detects hung contexts and triggers
a per engine reset. Thus, disabling the timeout means also losing all
per engine reset ability. A full GT reset will still occur when the
heartbeat finally expires, but that is a much more destructive and
undesirable mechanism.

The purpose of the workaround is actually to give OpenCL tasks longer
to reach a pre-emption point after a pre-emption request has been
issued. This is necessary because Gen12 does not support mid-thread
pre-emption and OpenCL can have long running threads.

So, rather than disabling the timeout completely, just set it to a
'long' value.

CC: Michal Mrozek <michal.mrozek@intel.com>
Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index 2a1e9f36e6f5..64249301a227 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -385,9 +385,25 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
 	engine->props.timeslice_duration_ms =
 		CONFIG_DRM_I915_TIMESLICE_DURATION;
 
-	/* Override to uninterruptible for OpenCL workloads. */
-	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
-		engine->props.preempt_timeout_ms = 0;
+	/*
+	 * Mid-thread pre-emption is not available in Gen12. Unfortunately,
+	 * some OpenCL workloads run quite long threads. That means they get
+	 * reset due to not pre-empting in a timely manner. So, bump the
+	 * pre-emption timeout value to be much higher for compute engines.
+	 * Using three times the heartbeat period seems long enough for a
+	 * reasonable task to reach a pre-emption point but not so long as to
+	 * allow genuine hangs to go unresolved.
+	 */
+	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS) {
+		unsigned long triple_beat = engine->props.heartbeat_interval_ms * 3;
+
+		if (triple_beat > engine->props.preempt_timeout_ms) {
+			drm_info(&gt->i915->drm, "Bumping pre-emption timeout from %ld to %ld on %s to allow slow compute pre-emption\n",
+				 engine->props.preempt_timeout_ms, triple_beat, engine->name);
+
+			engine->props.preempt_timeout_ms = triple_beat;
+		}
+	}
 
 	/* Cap timeouts to prevent overflow inside GuC */
 	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
-- 
2.25.1

[Intel-gfx] ✗ Fi.CI.SPARSE: warning for Improve anti-pre-emption w/a for compute workloads

[Patchwork]

== Series Details ==

Series: Improve anti-pre-emption w/a for compute workloads
URL   : https://patchwork.freedesktop.org/series/100428/
State : warning

== Summary ==

$ dim sparse --fast origin/drm-tip
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.

[Intel-gfx] ✗ Fi.CI.BAT: failure for Improve anti-pre-emption w/a for compute workloads

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 13532 bytes --]

== Series Details ==

Series: Improve anti-pre-emption w/a for compute workloads
URL   : https://patchwork.freedesktop.org/series/100428/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_11254 -> Patchwork_22337
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with Patchwork_22337 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in Patchwork_22337, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/index.html

Participating hosts (44 -> 42)
------------------------------

  Missing    (2): fi-bsw-cyan shard-tglu 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in Patchwork_22337:

### IGT changes ###

#### Possible regressions ####

  * igt@i915_selftest@live@execlists:
    - fi-cfl-8109u:       [PASS][1] -> [INCOMPLETE][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-cfl-8109u/igt@i915_selftest@live@execlists.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-8109u/igt@i915_selftest@live@execlists.html
    - fi-glk-dsi:         [PASS][3] -> [INCOMPLETE][4]
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-glk-dsi/igt@i915_selftest@live@execlists.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-glk-dsi/igt@i915_selftest@live@execlists.html
    - fi-cfl-8700k:       [PASS][5] -> [INCOMPLETE][6]
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-cfl-8700k/igt@i915_selftest@live@execlists.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-8700k/igt@i915_selftest@live@execlists.html
    - fi-glk-j4005:       [PASS][7] -> [INCOMPLETE][8]
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-glk-j4005/igt@i915_selftest@live@execlists.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-glk-j4005/igt@i915_selftest@live@execlists.html
    - fi-skl-guc:         [PASS][9] -> [INCOMPLETE][10]
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-skl-guc/igt@i915_selftest@live@execlists.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-guc/igt@i915_selftest@live@execlists.html
    - fi-skl-6700k2:      [PASS][11] -> [INCOMPLETE][12]
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-skl-6700k2/igt@i915_selftest@live@execlists.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-6700k2/igt@i915_selftest@live@execlists.html
    - fi-cfl-guc:         [PASS][13] -> [INCOMPLETE][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-cfl-guc/igt@i915_selftest@live@execlists.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-guc/igt@i915_selftest@live@execlists.html
    - fi-tgl-1115g4:      [PASS][15] -> [INCOMPLETE][16]
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-tgl-1115g4/igt@i915_selftest@live@execlists.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-tgl-1115g4/igt@i915_selftest@live@execlists.html
    - fi-cml-u2:          [PASS][17] -> [INCOMPLETE][18]
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-cml-u2/igt@i915_selftest@live@execlists.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cml-u2/igt@i915_selftest@live@execlists.html

  
#### Suppressed ####

  The following results come from untrusted machines, tests, or statuses.
  They do not affect the overall result.

  * igt@i915_selftest@live@execlists:
    - {fi-ehl-2}:         [PASS][19] -> [INCOMPLETE][20]
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-ehl-2/igt@i915_selftest@live@execlists.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-ehl-2/igt@i915_selftest@live@execlists.html
    - {fi-jsl-1}:         [PASS][21] -> [INCOMPLETE][22]
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-jsl-1/igt@i915_selftest@live@execlists.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-jsl-1/igt@i915_selftest@live@execlists.html
    - {bat-jsl-1}:        [PASS][23] -> [INCOMPLETE][24]
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/bat-jsl-1/igt@i915_selftest@live@execlists.html
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/bat-jsl-1/igt@i915_selftest@live@execlists.html
    - {fi-tgl-dsi}:       [PASS][25] -> [INCOMPLETE][26]
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-tgl-dsi/igt@i915_selftest@live@execlists.html
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-tgl-dsi/igt@i915_selftest@live@execlists.html

  
Known issues
------------

  Here are the changes found in Patchwork_22337 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_flink_basic@bad-flink:
    - fi-skl-6600u:       [PASS][27] -> [FAIL][28] ([i915#4547])
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-skl-6600u/igt@gem_flink_basic@bad-flink.html
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-6600u/igt@gem_flink_basic@bad-flink.html

  * igt@i915_selftest@live@execlists:
    - fi-bsw-nick:        [PASS][29] -> [INCOMPLETE][30] ([i915#2940])
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-bsw-nick/igt@i915_selftest@live@execlists.html
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-bsw-nick/igt@i915_selftest@live@execlists.html
    - fi-kbl-x1275:       [PASS][31] -> [INCOMPLETE][32] ([i915#794])
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-kbl-x1275/igt@i915_selftest@live@execlists.html
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-x1275/igt@i915_selftest@live@execlists.html
    - fi-bsw-n3050:       [PASS][33] -> [INCOMPLETE][34] ([i915#2940])
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-bsw-n3050/igt@i915_selftest@live@execlists.html
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-bsw-n3050/igt@i915_selftest@live@execlists.html
    - fi-kbl-7567u:       [PASS][35] -> [INCOMPLETE][36] ([i915#794])
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-kbl-7567u/igt@i915_selftest@live@execlists.html
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-7567u/igt@i915_selftest@live@execlists.html
    - fi-kbl-soraka:      [PASS][37] -> [INCOMPLETE][38] ([i915#794])
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-kbl-soraka/igt@i915_selftest@live@execlists.html
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-soraka/igt@i915_selftest@live@execlists.html
    - fi-kbl-7500u:       [PASS][39] -> [INCOMPLETE][40] ([i915#794])
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-kbl-7500u/igt@i915_selftest@live@execlists.html
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-7500u/igt@i915_selftest@live@execlists.html
    - fi-kbl-8809g:       [PASS][41] -> [INCOMPLETE][42] ([i915#794])
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-kbl-8809g/igt@i915_selftest@live@execlists.html
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-8809g/igt@i915_selftest@live@execlists.html

  * igt@runner@aborted:
    - fi-kbl-x1275:       NOTRUN -> [FAIL][43] ([i915#1436] / [i915#4312])
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-x1275/igt@runner@aborted.html
    - fi-cfl-8700k:       NOTRUN -> [FAIL][44] ([i915#4312])
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-8700k/igt@runner@aborted.html
    - fi-skl-6600u:       NOTRUN -> [FAIL][45] ([i915#4312])
   [45]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-6600u/igt@runner@aborted.html
    - fi-cfl-8109u:       NOTRUN -> [FAIL][46] ([i915#4312])
   [46]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-8109u/igt@runner@aborted.html
    - fi-glk-dsi:         NOTRUN -> [FAIL][47] ([i915#4312] / [k.org#202321])
   [47]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-glk-dsi/igt@runner@aborted.html
    - fi-bsw-nick:        NOTRUN -> [FAIL][48] ([fdo#109271] / [i915#1436] / [i915#3428] / [i915#4312])
   [48]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-bsw-nick/igt@runner@aborted.html
    - fi-kbl-8809g:       NOTRUN -> [FAIL][49] ([i915#1436] / [i915#4312])
   [49]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-8809g/igt@runner@aborted.html
    - fi-kbl-soraka:      NOTRUN -> [FAIL][50] ([i915#1436] / [i915#4312])
   [50]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-soraka/igt@runner@aborted.html
    - fi-kbl-7500u:       NOTRUN -> [FAIL][51] ([i915#1436] / [i915#4312])
   [51]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-7500u/igt@runner@aborted.html
    - fi-cml-u2:          NOTRUN -> [FAIL][52] ([i915#4312])
   [52]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cml-u2/igt@runner@aborted.html
    - fi-tgl-1115g4:      NOTRUN -> [FAIL][53] ([i915#1436] / [i915#4312])
   [53]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-tgl-1115g4/igt@runner@aborted.html
    - fi-cfl-guc:         NOTRUN -> [FAIL][54] ([i915#4312])
   [54]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-cfl-guc/igt@runner@aborted.html
    - fi-glk-j4005:       NOTRUN -> [FAIL][55] ([i915#4312] / [k.org#202321])
   [55]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-glk-j4005/igt@runner@aborted.html
    - fi-kbl-7567u:       NOTRUN -> [FAIL][56] ([i915#1436] / [i915#4312])
   [56]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-kbl-7567u/igt@runner@aborted.html
    - fi-skl-guc:         NOTRUN -> [FAIL][57] ([i915#1436] / [i915#4312])
   [57]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-guc/igt@runner@aborted.html
    - fi-skl-6700k2:      NOTRUN -> [FAIL][58] ([i915#1436] / [i915#4312])
   [58]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-skl-6700k2/igt@runner@aborted.html
    - fi-bsw-n3050:       NOTRUN -> [FAIL][59] ([fdo#109271] / [i915#1436] / [i915#3428] / [i915#4312])
   [59]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-bsw-n3050/igt@runner@aborted.html

  
#### Possible fixes ####

  * igt@i915_selftest@live@gt_heartbeat:
    - {fi-tgl-dsi}:       [DMESG-FAIL][60] ([i915#541]) -> [PASS][61]
   [60]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-tgl-dsi/igt@i915_selftest@live@gt_heartbeat.html
   [61]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-tgl-dsi/igt@i915_selftest@live@gt_heartbeat.html

  * igt@i915_selftest@live@hangcheck:
    - bat-dg1-6:          [DMESG-FAIL][62] ([i915#4494] / [i915#4957]) -> [PASS][63]
   [62]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/bat-dg1-6/igt@i915_selftest@live@hangcheck.html
   [63]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/bat-dg1-6/igt@i915_selftest@live@hangcheck.html

  
#### Warnings ####

  * igt@runner@aborted:
    - fi-bsw-kefka:       [FAIL][64] ([fdo#109271] / [i915#1436] / [i915#2722] / [i915#3428] / [i915#4312]) -> [FAIL][65] ([fdo#109271] / [i915#1436] / [i915#3428] / [i915#4312])
   [64]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11254/fi-bsw-kefka/igt@runner@aborted.html
   [65]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/fi-bsw-kefka/igt@runner@aborted.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
  [i915#2722]: https://gitlab.freedesktop.org/drm/intel/issues/2722
  [i915#2940]: https://gitlab.freedesktop.org/drm/intel/issues/2940
  [i915#3428]: https://gitlab.freedesktop.org/drm/intel/issues/3428
  [i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
  [i915#4494]: https://gitlab.freedesktop.org/drm/intel/issues/4494
  [i915#4547]: https://gitlab.freedesktop.org/drm/intel/issues/4547
  [i915#4897]: https://gitlab.freedesktop.org/drm/intel/issues/4897
  [i915#4957]: https://gitlab.freedesktop.org/drm/intel/issues/4957
  [i915#541]: https://gitlab.freedesktop.org/drm/intel/issues/541
  [i915#794]: https://gitlab.freedesktop.org/drm/intel/issues/794
  [k.org#202321]: https://bugzilla.kernel.org/show_bug.cgi?id=202321


Build changes
-------------

  * Linux: CI_DRM_11254 -> Patchwork_22337

  CI-20190529: 20190529
  CI_DRM_11254: 16159219b34bb0c431b00c019892a561292c81e0 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_6348: 9cb64a757d2ff1e180b1648e611439d94afd697d @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_22337: b6686e9596b7a75b735671ab3f1ec60e20feb19d @ git://anongit.freedesktop.org/gfx-ci/linux


== Linux commits ==

b6686e9596b7 drm/i915: Improve long running OCL w/a for GuC submission
444e8dd3a202 drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts
5abcc364e049 drm/i915/guc: Limit scheduling properties to avoid overflow

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_22337/index.html

[-- Attachment #2: Type: text/html, Size: 17439 bytes --]

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> GuC converts the pre-emption timeout and timeslice quantum values into
> clock ticks internally. That significantly reduces the point of 32bit
> overflow. On current platforms, worst case scenario is approximately

Where does 32-bit come from, the GuC side? We already use 64-bits so that something to fix to start with. Yep...

./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;

./gt/uc/intel_guc_submission.c: desc->execution_quantum = engine->props.timeslice_duration_ms * 1000;

./gt/intel_engine_types.h:              unsigned long timeslice_duration_ms;

timeslice_store/preempt_timeout_store:
err = kstrtoull(buf, 0, &duration);

So both kconfig and sysfs can already overflow GuC, not only because of tick conversion internally but because at backend level nothing was done for assigning 64-bit into 32-bit. Or I failed to find where it is handled.

> 110 seconds. Rather than allowing the user to set higher values and
> then get confused by early timeouts, add limits when setting these
> values.

Btw who is reviewing GuC patches these days - things have somehow gotten pretty quiet in activity and I don't think that's due absence of stuff to improve or fix? Asking since I think I noticed a few already which you posted and then crickets on the mailing list.

> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>   3 files changed, 38 insertions(+)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> index e53008b4dd05..2a1e9f36e6f5 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
>   	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>   		engine->props.preempt_timeout_ms = 0;
>   
> +	/* Cap timeouts to prevent overflow inside GuC */
> +	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
> +		if (engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {

Hm "wanted".. There's been too much back and forth on the GuC load options over the years to keep track.. intel_engine_uses_guc work sounds like would work and read nicer.

And limit to class instead of applying to all engines looks like a miss.

> +			drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %d to prevent possibly overflow\n",
> +				 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
> +			engine->props.timeslice_duration_ms = GUC_POLICY_MAX_EXEC_QUANTUM_MS;

I am not sure logging such message during driver load is useful. Sounds more like a confused driver which starts with one value and then overrides itself. I'd just silently set the value appropriate for the active backend. Preemption timeout kconfig text already documents the fact timeouts can get overriden at runtime depending on platform+engine. So maybe just add same text to timeslice kconfig.

> +		}
> +
> +		if (engine->props.preempt_timeout_ms > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
> +			drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %d to prevent possibly overflow\n",
> +				 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
> +			engine->props.preempt_timeout_ms = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
> +		}
> +	}
> +
>   	engine->defaults = engine->props; /* never to change again */
>   
>   	engine->context_size = intel_engine_context_size(gt, engine->class);
> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c b/drivers/gpu/drm/i915/gt/sysfs_engines.c
> index 967031056202..f57efe026474 100644
> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr,
>   	if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>   		return -EINVAL;
>   
> +	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
> +	    duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
> +		duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
> +		drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %lld to prevent possibly overflow\n",
> +			 duration);
> +	}

I would suggest to avoid duplicated clamping logic. Maybe hide the all backend logic into the helpers then, like maybe:

   d = intel_engine_validate_timeslice/preempt_timeout(engine, duration);
   if (d != duration)
	return -EINVAL:

Returning -EINVAL would be equivalent to existing behaviour:

	if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
		return -EINVAL;

That way userspace has explicit notification and read-back is identical to written in value. From engine setup you can just call the helper silently.

> +
>   	WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>   
>   	if (execlists_active(&engine->execlists))
> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr,
>   	if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>   		return -EINVAL;
>   
> +	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
> +	    timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
> +		timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
> +		drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %lld to prevent possibly overflow\n",
> +			 timeout);
> +	}
> +
>   	WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>   
>   	if (READ_ONCE(engine->execlists.pending[0]))
> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> index 6a4612a852e2..ad131092f8df 100644
> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>   
>   #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>   
> +/*
> + * GuC converts the timeout to clock ticks internally. Different platforms have
> + * different GuC clocks. Thus, the maximum value before overflow is platform
> + * dependent. Current worst case scenario is about 110s. So, limit to 100s to be
> + * safe.
> + */
> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS		(100 * 1000)
> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS	(100 * 1000)

Most important question -
how will we know/notice if/when new GuC arrives where these timeouts would still overflow? Can this be queried somehow at runtime or where does the limit comes from? How is GuC told about it? Set in some field and it just allows too large values silently break things?

Regards,

Tvrtko

> +
>   struct guc_policies {
>   	u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
>   	/* In micro seconds. How much time to allow before DPC processing is

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Compute workloads are inherently not pre-emptible on current hardware.
> Thus the pre-emption timeout was disabled as a workaround to prevent
> unwanted resets. Instead, the hang detection was left to the heartbeat
> and its (longer) timeout. This is undesirable with GuC submission as
> the heartbeat is a full GT reset rather than a per engine reset and so
> is much more destructive. Instead, just bump the pre-emption timeout

Can we have a feature request to allow asking GuC for an engine reset?

Regards,

Tvrtko

> to a big value. Also, update the heartbeat to allow such a long
> pre-emption delay in the final heartbeat period.
> 
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> 
> 
> John Harrison (3):
>    drm/i915/guc: Limit scheduling properties to avoid overflow
>    drm/i915/gt: Make the heartbeat play nice with long pre-emption
>      timeouts
>    drm/i915: Improve long running OCL w/a for GuC submission
> 
>   drivers/gpu/drm/i915/gt/intel_engine_cs.c     | 37 +++++++++++++++++--
>   .../gpu/drm/i915/gt/intel_engine_heartbeat.c  | 16 ++++++++
>   drivers/gpu/drm/i915/gt/sysfs_engines.c       | 14 +++++++
>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h   |  9 +++++
>   4 files changed, 73 insertions(+), 3 deletions(-)
> 

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 22/02/2022 09:52, Tvrtko Ursulin wrote:
> 
> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> GuC converts the pre-emption timeout and timeslice quantum values into
>> clock ticks internally. That significantly reduces the point of 32bit
>> overflow. On current platforms, worst case scenario is approximately
> 
> Where does 32-bit come from, the GuC side? We already use 64-bits so 
> that something to fix to start with. Yep...
> 
> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
> 
> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
> engine->props.timeslice_duration_ms * 1000;
> 
> ./gt/intel_engine_types.h:              unsigned long 
> timeslice_duration_ms;
> 
> timeslice_store/preempt_timeout_store:
> err = kstrtoull(buf, 0, &duration);
> 
> So both kconfig and sysfs can already overflow GuC, not only because of 
> tick conversion internally but because at backend level nothing was done 
> for assigning 64-bit into 32-bit. Or I failed to find where it is handled.
> 
>> 110 seconds. Rather than allowing the user to set higher values and
>> then get confused by early timeouts, add limits when setting these
>> values.
> 
> Btw who is reviewing GuC patches these days - things have somehow gotten 
> pretty quiet in activity and I don't think that's due absence of stuff 
> to improve or fix? Asking since I think I noticed a few already which 
> you posted and then crickets on the mailing list.
> 
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>   3 files changed, 38 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> index e53008b4dd05..2a1e9f36e6f5 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>> *gt, enum intel_engine_id id,
>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>           engine->props.preempt_timeout_ms = 0;
>> +    /* Cap timeouts to prevent overflow inside GuC */
>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>> +        if (engine->props.timeslice_duration_ms > 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
> 
> Hm "wanted".. There's been too much back and forth on the GuC load 
> options over the years to keep track.. intel_engine_uses_guc work sounds 
> like would work and read nicer.
> 
> And limit to class instead of applying to all engines looks like a miss.

Sorry limit to class does not apply here, I confused this with the last 
patch.

Regards,

Tvrtko

> 
>> +            drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>> duration to %d to prevent possibly overflow\n",
>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>> +            engine->props.timeslice_duration_ms = 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
> 
> I am not sure logging such message during driver load is useful. Sounds 
> more like a confused driver which starts with one value and then 
> overrides itself. I'd just silently set the value appropriate for the 
> active backend. Preemption timeout kconfig text already documents the 
> fact timeouts can get overriden at runtime depending on platform+engine. 
> So maybe just add same text to timeslice kconfig.
> 
>> +        }
>> +
>> +        if (engine->props.preempt_timeout_ms > 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>> pre-emption timeout to %d to prevent possibly overflow\n",
>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>> +            engine->props.preempt_timeout_ms = 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        }
>> +    }
>> +
>>       engine->defaults = engine->props; /* never to change again */
>>       engine->context_size = intel_engine_context_size(gt, 
>> engine->class);
>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> index 967031056202..f57efe026474 100644
>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>> kobj_attribute *attr,
>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>> +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>> duration to %lld to prevent possibly overflow\n",
>> +             duration);
>> +    }
> 
> I would suggest to avoid duplicated clamping logic. Maybe hide the all 
> backend logic into the helpers then, like maybe:
> 
>    d = intel_engine_validate_timeslice/preempt_timeout(engine, duration);
>    if (d != duration)
>      return -EINVAL:
> 
> Returning -EINVAL would be equivalent to existing behaviour:
> 
>      if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>          return -EINVAL;
> 
> That way userspace has explicit notification and read-back is identical 
> to written in value. From engine setup you can just call the helper 
> silently.
> 
>> +
>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>       if (execlists_active(&engine->execlists))
>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>> struct kobj_attribute *attr,
>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>> +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping pre-emption 
>> timeout to %lld to prevent possibly overflow\n",
>> +             timeout);
>> +    }
>> +
>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>       if (READ_ONCE(engine->execlists.pending[0]))
>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> index 6a4612a852e2..ad131092f8df 100644
>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>   #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>> +/*
>> + * GuC converts the timeout to clock ticks internally. Different 
>> platforms have
>> + * different GuC clocks. Thus, the maximum value before overflow is 
>> platform
>> + * dependent. Current worst case scenario is about 110s. So, limit to 
>> 100s to be
>> + * safe.
>> + */
>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
> 
> Most important question -
> how will we know/notice if/when new GuC arrives where these timeouts 
> would still overflow? Can this be queried somehow at runtime or where 
> does the limit comes from? How is GuC told about it? Set in some field 
> and it just allows too large values silently break things?
> 
> Regards,
> 
> Tvrtko
> 
>> +
>>   struct guc_policies {
>>       u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
>>       /* In micro seconds. How much time to allow before DPC 
>> processing is

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
> 
> Compute workloads are inherantly not pre-emptible for long periods on
> current hardware. As a workaround for this, the pre-emption timeout
> for compute capable engines was disabled. This is undesirable with GuC
> submission as it prevents per engine reset of hung contexts. Hence the
> next patch will re-enable the timeout but bumped up by an order of
> magnititude.

(Some typos above.)

> However, the heartbeat might not respect that. Depending upon current
> activity, a pre-emption to the heartbeat pulse might not even be
> attempted until the last heartbeat period. Which means that only one

Might not be attempted, but could be if something is running with lower 
priority. In which case I think special casing the last heartbeat does 
not feel right because it can end up resetting the engine before it was 
intended.

Like if first heartbeat decides to preempt (the decision is backend 
specific, could be same prio + timeslicing), and preempt timeout has 
been set to heartbeat interval * 3, then 2nd heartbeat gets queued up, 
then 3rd, and so reset is triggered even before the first preempt 
timeout legitimately expires (or just as it is about to react).

Instead, how about preempt timeout is always considered when calculating 
when to emit the next heartbeat? End result would be similar to your 
patch, in terms of avoiding the direct problem, although hang detection 
would be overall longer (but more correct I think).

And it also means in the next patch you don't have to add coupling 
between preempt timeout and heartbeat to intel_engine_setup. Instead 
just some long preempt timeout would be needed. Granted, the decoupling 
argument is not super strong since then the heartbeat code has the 
coupling instead, but that still feels better to me. (Since we can say 
heartbeats only make sense on loaded engines, and so things like preempt 
timeout can legitimately be considered from there.)

Incidentally, that would be similar to a patch which Chris had a year 
ago (https://patchwork.freedesktop.org/patch/419783/?series=86841&rev=1) 
to fix some CI issue.

On a related topic, if GuC engine resets stop working when preempt 
timeout is set to zero - I think we need to somehow let the user know if 
they try to tweak it via sysfs. Perhaps go as far as -EINVAL in GuC 
mode, if i915.reset has not explicitly disabled engine resets.

Regards,

Tvrtko

> period is granted for the pre-emption to occur. With the aforesaid
> bump, the pre-emption timeout could be significantly larger than this
> heartbeat period.
> 
> So adjust the heartbeat code to take the pre-emption timeout into
> account. When it reaches the final (high priority) period, it now
> ensures the delay before hitting reset is bigger than the pre-emption
> timeout.
> 
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>   drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c | 16 ++++++++++++++++
>   1 file changed, 16 insertions(+)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
> index a3698f611f45..72a82a6085e0 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
> @@ -22,9 +22,25 @@
>   
>   static bool next_heartbeat(struct intel_engine_cs *engine)
>   {
> +	struct i915_request *rq;
>   	long delay;
>   
>   	delay = READ_ONCE(engine->props.heartbeat_interval_ms);
> +
> +	rq = engine->heartbeat.systole;
> +	if (rq && rq->sched.attr.priority >= I915_PRIORITY_BARRIER) {
> +		long longer;
> +
> +		/*
> +		 * The final try is at the highest priority possible. Up until now
> +		 * a pre-emption might not even have been attempted. So make sure
> +		 * this last attempt allows enough time for a pre-emption to occur.
> +		 */
> +		longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
> +		if (longer > delay)
> +			delay = longer;
> +	}
> +
>   	if (!delay)
>   		return false;
>   

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Ceraolo Spurio, Daniele]

On 2/18/2022 1:33 PM, John.C.Harrison@Intel.com wrote:
> From: John Harrison <John.C.Harrison@Intel.com>
>
> GuC converts the pre-emption timeout and timeslice quantum values into
> clock ticks internally. That significantly reduces the point of 32bit
> overflow. On current platforms, worst case scenario is approximately
> 110 seconds. Rather than allowing the user to set higher values and
> then get confused by early timeouts, add limits when setting these
> values.
>
> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
> ---
>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>   3 files changed, 38 insertions(+)
>
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> index e53008b4dd05..2a1e9f36e6f5 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt *gt, enum intel_engine_id id,
>   	if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>   		engine->props.preempt_timeout_ms = 0;
>   
> +	/* Cap timeouts to prevent overflow inside GuC */
> +	if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
> +		if (engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
> +			drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %d to prevent possibly overflow\n",

I'd drop the word "possibly"

> +				 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
> +			engine->props.timeslice_duration_ms = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
> +		}
> +
> +		if (engine->props.preempt_timeout_ms > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
> +			drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %d to prevent possibly overflow\n",
> +				 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
> +			engine->props.preempt_timeout_ms = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
> +		}
> +	}
> +
>   	engine->defaults = engine->props; /* never to change again */
>   
>   	engine->context_size = intel_engine_context_size(gt, engine->class);
> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c b/drivers/gpu/drm/i915/gt/sysfs_engines.c
> index 967031056202..f57efe026474 100644
> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct kobj_attribute *attr,
>   	if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>   		return -EINVAL;
>   
> +	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
> +	    duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
> +		duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
> +		drm_info(&engine->i915->drm, "Warning, clamping timeslice duration to %lld to prevent possibly overflow\n",
> +			 duration);
> +	}
> +
>   	WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>   
>   	if (execlists_active(&engine->execlists))
> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, struct kobj_attribute *attr,
>   	if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>   		return -EINVAL;
>   
> +	if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
> +	    timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
> +		timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
> +		drm_info(&engine->i915->drm, "Warning, clamping pre-emption timeout to %lld to prevent possibly overflow\n",
> +			 timeout);
> +	}
> +
>   	WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>   
>   	if (READ_ONCE(engine->execlists.pending[0]))
> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> index 6a4612a852e2..ad131092f8df 100644
> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>   
>   #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>   
> +/*
> + * GuC converts the timeout to clock ticks internally. Different platforms have
> + * different GuC clocks. Thus, the maximum value before overflow is platform
> + * dependent. Current worst case scenario is about 110s. So, limit to 100s to be
> + * safe.
> + */
> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS		(100 * 1000)
> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS	(100 * 1000)

Those values don't seem to be defined in the GuC interface. If I'm 
correct, IMO we need to ask the GuC team to add them in, because it 
shouldn't be our responsibility to convert from ms to GuC clocks, 
considering that the interface is in ms. Not a blocker for this patch.

Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>

Daniele

> +
>   struct guc_policies {
>   	u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
>   	/* In micro seconds. How much time to allow before DPC processing is

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/22/2022 01:52, Tvrtko Ursulin wrote:
> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> GuC converts the pre-emption timeout and timeslice quantum values into
>> clock ticks internally. That significantly reduces the point of 32bit
>> overflow. On current platforms, worst case scenario is approximately
>
> Where does 32-bit come from, the GuC side? We already use 64-bits so 
> that something to fix to start with. Yep...
Yes, the GuC API is defined as 32bits only and then does a straight 
multiply by the clock speed with no range checking. We have requested 
64bit support but there was push back on the grounds that it is not 
something the GuC timer hardware supports and such long timeouts are not 
real world usable anyway.


>
> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>
> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
> engine->props.timeslice_duration_ms * 1000;
>
> ./gt/intel_engine_types.h:              unsigned long 
> timeslice_duration_ms;
>
> timeslice_store/preempt_timeout_store:
> err = kstrtoull(buf, 0, &duration);
>
> So both kconfig and sysfs can already overflow GuC, not only because 
> of tick conversion internally but because at backend level nothing was 
> done for assigning 64-bit into 32-bit. Or I failed to find where it is 
> handled.
That's why I'm adding this range check to make sure we don't allow 
overflows.

>
>> 110 seconds. Rather than allowing the user to set higher values and
>> then get confused by early timeouts, add limits when setting these
>> values.
>
> Btw who is reviewing GuC patches these days - things have somehow 
> gotten pretty quiet in activity and I don't think that's due absence 
> of stuff to improve or fix? Asking since I think I noticed a few 
> already which you posted and then crickets on the mailing list.
Too much work to do and not enough engineers to do it all :(.


>
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>   3 files changed, 38 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> index e53008b4dd05..2a1e9f36e6f5 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>> *gt, enum intel_engine_id id,
>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>           engine->props.preempt_timeout_ms = 0;
>>   +    /* Cap timeouts to prevent overflow inside GuC */
>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>> +        if (engine->props.timeslice_duration_ms > 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>
> Hm "wanted".. There's been too much back and forth on the GuC load 
> options over the years to keep track.. intel_engine_uses_guc work 
> sounds like would work and read nicer.
I'm not adding a new feature check here. I'm just using the existing 
one. If we want to rename it yet again then that would be a different 
patch set.

>
> And limit to class instead of applying to all engines looks like a miss.
As per follow up email, the class limit is not applied here.

>
>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice duration 
>> to %d to prevent possibly overflow\n",
>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>> +            engine->props.timeslice_duration_ms = 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>
> I am not sure logging such message during driver load is useful. 
> Sounds more like a confused driver which starts with one value and 
> then overrides itself. I'd just silently set the value appropriate for 
> the active backend. Preemption timeout kconfig text already documents 
> the fact timeouts can get overriden at runtime depending on 
> platform+engine. So maybe just add same text to timeslice kconfig.
The point is to make people aware if they compile with unsupported 
config options. As far as I know, there is no way to apply range 
checking or other limits to config defines. Which means that a user 
would silently get unwanted behaviour. That seems like a bad thing to 
me. If the driver is confused because the user built it in a confused 
manner then we should let them know.


>
>> +        }
>> +
>> +        if (engine->props.preempt_timeout_ms > 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>> pre-emption timeout to %d to prevent possibly overflow\n",
>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>> +            engine->props.preempt_timeout_ms = 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        }
>> +    }
>> +
>>       engine->defaults = engine->props; /* never to change again */
>>         engine->context_size = intel_engine_context_size(gt, 
>> engine->class);
>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> index 967031056202..f57efe026474 100644
>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>> kobj_attribute *attr,
>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>> duration to %lld to prevent possibly overflow\n",
>> +             duration);
>> +    }
>
> I would suggest to avoid duplicated clamping logic. Maybe hide the all 
> backend logic into the helpers then, like maybe:
>
>   d = intel_engine_validate_timeslice/preempt_timeout(engine, duration);
>   if (d != duration)
>     return -EINVAL:
>
> Returning -EINVAL would be equivalent to existing behaviour:
>
>     if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>         return -EINVAL;
>
> That way userspace has explicit notification and read-back is 
> identical to written in value. From engine setup you can just call the 
> helper silently.
Sure, EINVAL rather than clamping works as well. And can certainly add 
helper wrappers. But as above, I don't like the idea of silently 
disregarding a user specified config option.

>
>> +
>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>         if (execlists_active(&engine->execlists))
>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>> struct kobj_attribute *attr,
>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping pre-emption 
>> timeout to %lld to prevent possibly overflow\n",
>> +             timeout);
>> +    }
>> +
>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>         if (READ_ONCE(engine->execlists.pending[0]))
>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> index 6a4612a852e2..ad131092f8df 100644
>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>   +/*
>> + * GuC converts the timeout to clock ticks internally. Different 
>> platforms have
>> + * different GuC clocks. Thus, the maximum value before overflow is 
>> platform
>> + * dependent. Current worst case scenario is about 110s. So, limit 
>> to 100s to be
>> + * safe.
>> + */
>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>
> Most important question -
> how will we know/notice if/when new GuC arrives where these timeouts 
> would still overflow? Can this be queried somehow at runtime or where 
> does the limit comes from? How is GuC told about it? Set in some field 
> and it just allows too large values silently break things?
Currently, we don't notice except by debugging peculiar test failures :(.

These limits are not in any GuC spec. Indeed, it took a while to 
actually work out why increasing the value actually caused shorter 
timeouts to occur! As above, there is no range checking inside GuC 
itself. It does a truncated multiply which results in an effectively 
random number and just happily uses it.

John.



>
> Regards,
>
> Tvrtko
>
>> +
>>   struct guc_policies {
>>       u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
>>       /* In micro seconds. How much time to allow before DPC 
>> processing is

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/22/2022 16:52, Ceraolo Spurio, Daniele wrote:
> On 2/18/2022 1:33 PM, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> GuC converts the pre-emption timeout and timeslice quantum values into
>> clock ticks internally. That significantly reduces the point of 32bit
>> overflow. On current platforms, worst case scenario is approximately
>> 110 seconds. Rather than allowing the user to set higher values and
>> then get confused by early timeouts, add limits when setting these
>> values.
>>
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>   3 files changed, 38 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> index e53008b4dd05..2a1e9f36e6f5 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>> *gt, enum intel_engine_id id,
>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>           engine->props.preempt_timeout_ms = 0;
>>   +    /* Cap timeouts to prevent overflow inside GuC */
>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>> +        if (engine->props.timeslice_duration_ms > 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>> timeslice duration to %d to prevent possibly overflow\n",
>
> I'd drop the word "possibly"
>
>> + GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>> +            engine->props.timeslice_duration_ms = 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>> +        }
>> +
>> +        if (engine->props.preempt_timeout_ms > 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>> pre-emption timeout to %d to prevent possibly overflow\n",
>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>> +            engine->props.preempt_timeout_ms = 
>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        }
>> +    }
>> +
>>       engine->defaults = engine->props; /* never to change again */
>>         engine->context_size = intel_engine_context_size(gt, 
>> engine->class);
>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> index 967031056202..f57efe026474 100644
>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>> kobj_attribute *attr,
>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>> duration to %lld to prevent possibly overflow\n",
>> +             duration);
>> +    }
>> +
>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>         if (execlists_active(&engine->execlists))
>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>> struct kobj_attribute *attr,
>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>           return -EINVAL;
>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>> +        drm_info(&engine->i915->drm, "Warning, clamping pre-emption 
>> timeout to %lld to prevent possibly overflow\n",
>> +             timeout);
>> +    }
>> +
>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>         if (READ_ONCE(engine->execlists.pending[0]))
>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> index 6a4612a852e2..ad131092f8df 100644
>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>   +/*
>> + * GuC converts the timeout to clock ticks internally. Different 
>> platforms have
>> + * different GuC clocks. Thus, the maximum value before overflow is 
>> platform
>> + * dependent. Current worst case scenario is about 110s. So, limit 
>> to 100s to be
>> + * safe.
>> + */
>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>
> Those values don't seem to be defined in the GuC interface. If I'm 
> correct, IMO we need to ask the GuC team to add them in, because it 
> shouldn't be our responsibility to convert from ms to GuC clocks, 
> considering that the interface is in ms. Not a blocker for this patch.
>
As per other reply, no. GuC doesn't give us any hints or clues on any 
limits of these values. But yes, we can push them to at least document 
the limits.

John.


> Reviewed-by: Daniele Ceraolo Spurio <daniele.ceraolospurio@intel.com>
>
> Daniele
>
>> +
>>   struct guc_policies {
>>       u32 submission_queue_depth[GUC_MAX_ENGINE_CLASSES];
>>       /* In micro seconds. How much time to allow before DPC 
>> processing is
>

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/22/2022 01:53, Tvrtko Ursulin wrote:
> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> Compute workloads are inherently not pre-emptible on current hardware.
>> Thus the pre-emption timeout was disabled as a workaround to prevent
>> unwanted resets. Instead, the hang detection was left to the heartbeat
>> and its (longer) timeout. This is undesirable with GuC submission as
>> the heartbeat is a full GT reset rather than a per engine reset and so
>> is much more destructive. Instead, just bump the pre-emption timeout
>
> Can we have a feature request to allow asking GuC for an engine reset?
For what purpose?

GuC manages the scheduling of contexts across engines. With virtual 
engines, the KMD has no knowledge of which engine a context might be 
executing on. Even without virtual engines, the KMD still has no 
knowledge of which context is currently executing on any given engine at 
any given time.

There is a reason why hang detection should be left to the entity that 
is doing the scheduling. Any other entity is second guessing at best.

The reason for keeping the heartbeat around even when GuC submission is 
enabled is for the case where the KMD/GuC have got out of sync with 
either other somehow or GuC itself has just crashed. I.e. when no 
submission at all is working and we need to reset the GuC itself and 
start over.

John.


>
> Regards,
>
> Tvrtko
>
>> to a big value. Also, update the heartbeat to allow such a long
>> pre-emption delay in the final heartbeat period.
>>
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>
>>
>> John Harrison (3):
>>    drm/i915/guc: Limit scheduling properties to avoid overflow
>>    drm/i915/gt: Make the heartbeat play nice with long pre-emption
>>      timeouts
>>    drm/i915: Improve long running OCL w/a for GuC submission
>>
>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c     | 37 +++++++++++++++++--
>>   .../gpu/drm/i915/gt/intel_engine_heartbeat.c  | 16 ++++++++
>>   drivers/gpu/drm/i915/gt/sysfs_engines.c       | 14 +++++++
>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h   |  9 +++++
>>   4 files changed, 73 insertions(+), 3 deletions(-)
>>

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 2/22/2022 03:19, Tvrtko Ursulin wrote:
> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>> From: John Harrison <John.C.Harrison@Intel.com>
>>
>> Compute workloads are inherantly not pre-emptible for long periods on
>> current hardware. As a workaround for this, the pre-emption timeout
>> for compute capable engines was disabled. This is undesirable with GuC
>> submission as it prevents per engine reset of hung contexts. Hence the
>> next patch will re-enable the timeout but bumped up by an order of
>> magnititude.
>
> (Some typos above.)
I'm spotting 'inherently' but not anything else.

>
>> However, the heartbeat might not respect that. Depending upon current
>> activity, a pre-emption to the heartbeat pulse might not even be
>> attempted until the last heartbeat period. Which means that only one
>
> Might not be attempted, but could be if something is running with 
> lower priority. In which case I think special casing the last 
> heartbeat does not feel right because it can end up resetting the 
> engine before it was intended.
>
> Like if first heartbeat decides to preempt (the decision is backend 
> specific, could be same prio + timeslicing), and preempt timeout has 
> been set to heartbeat interval * 3, then 2nd heartbeat gets queued up, 
> then 3rd, and so reset is triggered even before the first preempt 
> timeout legitimately expires (or just as it is about to react).
>
> Instead, how about preempt timeout is always considered when 
> calculating when to emit the next heartbeat? End result would be 
> similar to your patch, in terms of avoiding the direct problem, 
> although hang detection would be overall longer (but more correct I 
> think).
>
> And it also means in the next patch you don't have to add coupling 
> between preempt timeout and heartbeat to intel_engine_setup. Instead 
> just some long preempt timeout would be needed. Granted, the 
> decoupling argument is not super strong since then the heartbeat code 
> has the coupling instead, but that still feels better to me. (Since we 
> can say heartbeats only make sense on loaded engines, and so things 
> like preempt timeout can legitimately be considered from there.)
>
> Incidentally, that would be similar to a patch which Chris had a year 
> ago 
> (https://patchwork.freedesktop.org/patch/419783/?series=86841&rev=1) 
> to fix some CI issue.
>
I'm not following your arguments.

Chris' patch is about not having two i915 based resets triggered 
concurrently - i915 based engine reset and i915 based GT reset. The 
purpose of this patch is to allow the GuC based engine reset to have a 
chance to occur before the i915 based GT reset kicks in.

It sounds like your argument above is about making the engine reset 
slower so that it doesn't happen before the appropriate heartbeat period 
for that potential reset scenario has expired. I don't see why that is 
at all necessary or useful.

If an early heartbeat period triggers an engine reset then the heartbeat 
pulse will go through. The heartbeat will thus see a happy system and 
not do anything further. If the given period does not trigger an engine 
reset but still does not get the pulse through (because the pulse is of 
too low a priority) then we move on to the next period and bump the 
priority. If the pre-emption has actually already been triggered anyway 
(and we are just waiting a while for it to timeout) then that's fine. 
The priority bump will have no effect because the context is already 
attempting to run. The heartbeat code doesn't care which priority level 
actually triggers the reset. It just cares whether or not the pulse 
finally makes it through. And the GuC doesn't care which heartbeat 
period the i915 is in. All it knows is that it has a request to schedule 
and whether the current context is pre-empting or not. So if period #1 
triggers the pre-emption but the timeout doesn't happen until period #3, 
who cares? The result is the same as if period #3 triggered the 
pre-emption and the timeout was shorter. The result being that the hung 
context is reset, the pulse makes it through and the heartbeat goes to 
sleep again.

The only period that really matters is the final one. At that point the 
pulse request is at highest priority and so must trigger a pre-emption 
request. We then need at least one full pre-emption period (plus some 
wiggle room for random delays in reset time, context switching, 
processing messages, etc.) to allow the GuC based timeout and reset to 
occur. Hence ensuring that the final heartbeat period is at least twice 
the pre-emption timeout (because 1.25 times is just messy when working 
with ints!).

That guarantees that GuC will get at least one complete opportunity to 
detect and recover the hang before i915 nukes the universe.

Whereas, bumping all heartbeat periods to be greater than the 
pre-emption timeout is wasteful and unnecessary. That leads to a total 
heartbeat time of about a minute. Which is a very long time to wait for 
a hang to be detected and recovered. Especially when the official limit 
on a context responding to an 'are you dead' query is only 7.5 seconds.



> On a related topic, if GuC engine resets stop working when preempt 
> timeout is set to zero - I think we need to somehow let the user know 
> if they try to tweak it via sysfs. Perhaps go as far as -EINVAL in GuC 
> mode, if i915.reset has not explicitly disabled engine resets.
Define 'stops working'. The definition of the sysfs interface is that a 
value of zero disables pre-emption. If you don't have pre-emption and 
your hang detection mechanism relies on pre-emption then you don't have 
a hang detection mechanism either. If the user really wants to allow 
their context to run forever and never be pre-empted then that means 
they also don't want it to be reset arbitrarily. Which means they would 
also be disabling the heartbeat timer as well. Indeed, this is what we 
advise compute customers to do. It is then up to the user themselves to 
spot a hang and to manually kill (Ctrl+C, kill ###, etc.) their task. 
Killing the CPU task will automatically clear up any GPU resources 
allocated to that task (excepting context persistence, which is a) 
broken and b) something we also tell compute customers to disable).

John.

>
> Regards,
>
> Tvrtko
>
>> period is granted for the pre-emption to occur. With the aforesaid
>> bump, the pre-emption timeout could be significantly larger than this
>> heartbeat period.
>>
>> So adjust the heartbeat code to take the pre-emption timeout into
>> account. When it reaches the final (high priority) period, it now
>> ensures the delay before hitting reset is bigger than the pre-emption
>> timeout.
>>
>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>> ---
>>   drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c | 16 ++++++++++++++++
>>   1 file changed, 16 insertions(+)
>>
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c 
>> b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
>> index a3698f611f45..72a82a6085e0 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_heartbeat.c
>> @@ -22,9 +22,25 @@
>>     static bool next_heartbeat(struct intel_engine_cs *engine)
>>   {
>> +    struct i915_request *rq;
>>       long delay;
>>         delay = READ_ONCE(engine->props.heartbeat_interval_ms);
>> +
>> +    rq = engine->heartbeat.systole;
>> +    if (rq && rq->sched.attr.priority >= I915_PRIORITY_BARRIER) {
>> +        long longer;
>> +
>> +        /*
>> +         * The final try is at the highest priority possible. Up 
>> until now
>> +         * a pre-emption might not even have been attempted. So make 
>> sure
>> +         * this last attempt allows enough time for a pre-emption to 
>> occur.
>> +         */
>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>> +        if (longer > delay)
>> +            delay = longer;
>> +    }
>> +
>>       if (!delay)
>>           return false;

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 23/02/2022 02:22, John Harrison wrote:
> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>
>>> Compute workloads are inherently not pre-emptible on current hardware.
>>> Thus the pre-emption timeout was disabled as a workaround to prevent
>>> unwanted resets. Instead, the hang detection was left to the heartbeat
>>> and its (longer) timeout. This is undesirable with GuC submission as
>>> the heartbeat is a full GT reset rather than a per engine reset and so
>>> is much more destructive. Instead, just bump the pre-emption timeout
>>
>> Can we have a feature request to allow asking GuC for an engine reset?
> For what purpose?

To allow "stopped heartbeat" to reset the engine, however..

> GuC manages the scheduling of contexts across engines. With virtual 
> engines, the KMD has no knowledge of which engine a context might be 
> executing on. Even without virtual engines, the KMD still has no 
> knowledge of which context is currently executing on any given engine at 
> any given time.
> 
> There is a reason why hang detection should be left to the entity that 
> is doing the scheduling. Any other entity is second guessing at best.
> 
> The reason for keeping the heartbeat around even when GuC submission is 
> enabled is for the case where the KMD/GuC have got out of sync with 
> either other somehow or GuC itself has just crashed. I.e. when no 
> submission at all is working and we need to reset the GuC itself and 
> start over.

.. I wasn't really up to speed to know/remember heartbeats are nerfed 
already in GuC mode.

I am not sure it was the best way since full reset penalizes everyone 
for one hanging engine. Better question would be why leave heartbeats 
around to start with with GuC? If you want to use it to health check 
GuC, as you say, maybe just send a CT message and expect replies? Then 
full reset would make sense. It also achieves the goal of not seconding 
guessing the submission backend you raise.

Like it is now, and the need for this series demonstrates it, the whole 
thing has a pretty poor "impedance" match. Not even sure what 
intel_guc_find_hung_context is doing in intel_engine_hearbeat.c - why is 
that not in intel_gt_handle_error at least? Why is hearbeat code special 
and other callers of intel_gt_handle_error don't need it?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 23/02/2022 02:11, John Harrison wrote:
> On 2/22/2022 01:52, Tvrtko Ursulin wrote:
>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>
>>> GuC converts the pre-emption timeout and timeslice quantum values into
>>> clock ticks internally. That significantly reduces the point of 32bit
>>> overflow. On current platforms, worst case scenario is approximately
>>
>> Where does 32-bit come from, the GuC side? We already use 64-bits so 
>> that something to fix to start with. Yep...
> Yes, the GuC API is defined as 32bits only and then does a straight 
> multiply by the clock speed with no range checking. We have requested 
> 64bit support but there was push back on the grounds that it is not 
> something the GuC timer hardware supports and such long timeouts are not 
> real world usable anyway.

As long as compute are happy with 100 seconds, then it "should be enough 
for everbody". :D

>>
>> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>>
>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>> engine->props.timeslice_duration_ms * 1000;
>>
>> ./gt/intel_engine_types.h:              unsigned long 
>> timeslice_duration_ms;
>>
>> timeslice_store/preempt_timeout_store:
>> err = kstrtoull(buf, 0, &duration);
>>
>> So both kconfig and sysfs can already overflow GuC, not only because 
>> of tick conversion internally but because at backend level nothing was 
>> done for assigning 64-bit into 32-bit. Or I failed to find where it is 
>> handled.
> That's why I'm adding this range check to make sure we don't allow 
> overflows.

Yes and no, this fixes it, but the first bug was not only due GuC 
internal tick conversion. It was present ever since the u64 from i915 
was shoved into u32 sent to GuC. So even if GuC used the value without 
additional multiplication, bug was be there. My point being when GuC 
backend was added timeout_ms values should have been limited/clamped to 
U32_MAX. The tick discovery is additional limit on top.

>>> 110 seconds. Rather than allowing the user to set higher values and
>>> then get confused by early timeouts, add limits when setting these
>>> values.
>>
>> Btw who is reviewing GuC patches these days - things have somehow 
>> gotten pretty quiet in activity and I don't think that's due absence 
>> of stuff to improve or fix? Asking since I think I noticed a few 
>> already which you posted and then crickets on the mailing list.
> Too much work to do and not enough engineers to do it all :(.
> 
> 
>>
>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>> ---
>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>   3 files changed, 38 insertions(+)
>>>
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>>> *gt, enum intel_engine_id id,
>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>>           engine->props.preempt_timeout_ms = 0;
>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>> +        if (engine->props.timeslice_duration_ms > 
>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>
>> Hm "wanted".. There's been too much back and forth on the GuC load 
>> options over the years to keep track.. intel_engine_uses_guc work 
>> sounds like would work and read nicer.
> I'm not adding a new feature check here. I'm just using the existing 
> one. If we want to rename it yet again then that would be a different 
> patch set.

$ grep intel_engine_uses_guc . -rl
./i915_perf.c
./i915_request.c
./selftests/intel_scheduler_helpers.c
./gem/i915_gem_context.c
./gt/intel_context.c
./gt/intel_engine.h
./gt/intel_engine_cs.c
./gt/intel_engine_heartbeat.c
./gt/intel_engine_pm.c
./gt/intel_reset.c
./gt/intel_lrc.c
./gt/selftest_context.c
./gt/selftest_engine_pm.c
./gt/selftest_hangcheck.c
./gt/selftest_mocs.c
./gt/selftest_workarounds.c

Sounds better to me than intel_guc_submission_is_wanted. What does the 
reader know whether "is wanted" translates to "is actually used". Shrug 
on "is wanted".

>> And limit to class instead of applying to all engines looks like a miss.
> As per follow up email, the class limit is not applied here.
> 
>>
>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice duration 
>>> to %d to prevent possibly overflow\n",
>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>> +            engine->props.timeslice_duration_ms = 
>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>
>> I am not sure logging such message during driver load is useful. 
>> Sounds more like a confused driver which starts with one value and 
>> then overrides itself. I'd just silently set the value appropriate for 
>> the active backend. Preemption timeout kconfig text already documents 
>> the fact timeouts can get overriden at runtime depending on 
>> platform+engine. So maybe just add same text to timeslice kconfig.
> The point is to make people aware if they compile with unsupported 
> config options. As far as I know, there is no way to apply range 
> checking or other limits to config defines. Which means that a user 
> would silently get unwanted behaviour. That seems like a bad thing to 
> me. If the driver is confused because the user built it in a confused 
> manner then we should let them know.

Okay, but I think make it notice low level.

Also consider in patch 3/3 when you triple it, and then clamp back down 
here. That's even more confused state since tripling gets nerfed. I 
think that's also an argument to always account preempt timeout in 
heartbeat interval calculation. Haven't got to your reply on 2/3 yet 
though..

>>
>>> +        }
>>> +
>>> +        if (engine->props.preempt_timeout_ms > 
>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>>> pre-emption timeout to %d to prevent possibly overflow\n",
>>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>>> +            engine->props.preempt_timeout_ms = 
>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>> +        }
>>> +    }
>>> +
>>>       engine->defaults = engine->props; /* never to change again */
>>>         engine->context_size = intel_engine_context_size(gt, 
>>> engine->class);
>>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>> index 967031056202..f57efe026474 100644
>>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>>> kobj_attribute *attr,
>>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>           return -EINVAL;
>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>> duration to %lld to prevent possibly overflow\n",
>>> +             duration);
>>> +    }
>>
>> I would suggest to avoid duplicated clamping logic. Maybe hide the all 
>> backend logic into the helpers then, like maybe:
>>
>>   d = intel_engine_validate_timeslice/preempt_timeout(engine, duration);
>>   if (d != duration)
>>     return -EINVAL:
>>
>> Returning -EINVAL would be equivalent to existing behaviour:
>>
>>     if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>         return -EINVAL;
>>
>> That way userspace has explicit notification and read-back is 
>> identical to written in value. From engine setup you can just call the 
>> helper silently.
> Sure, EINVAL rather than clamping works as well. And can certainly add 
> helper wrappers. But as above, I don't like the idea of silently 
> disregarding a user specified config option.

Deal - with the open of heartbeat interval TBD.

> 
>>
>>> +
>>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>>         if (execlists_active(&engine->execlists))
>>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>>> struct kobj_attribute *attr,
>>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>           return -EINVAL;
>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>> +        drm_info(&engine->i915->drm, "Warning, clamping pre-emption 
>>> timeout to %lld to prevent possibly overflow\n",
>>> +             timeout);
>>> +    }
>>> +
>>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>>         if (READ_ONCE(engine->execlists.pending[0]))
>>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>> index 6a4612a852e2..ad131092f8df 100644
>>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>>   +/*
>>> + * GuC converts the timeout to clock ticks internally. Different 
>>> platforms have
>>> + * different GuC clocks. Thus, the maximum value before overflow is 
>>> platform
>>> + * dependent. Current worst case scenario is about 110s. So, limit 
>>> to 100s to be
>>> + * safe.
>>> + */
>>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>>
>> Most important question -
>> how will we know/notice if/when new GuC arrives where these timeouts 
>> would still overflow? Can this be queried somehow at runtime or where 
>> does the limit comes from? How is GuC told about it? Set in some field 
>> and it just allows too large values silently break things?
> Currently, we don't notice except by debugging peculiar test failures :(.
> 
> These limits are not in any GuC spec. Indeed, it took a while to 
> actually work out why increasing the value actually caused shorter 
> timeouts to occur! As above, there is no range checking inside GuC 
> itself. It does a truncated multiply which results in an effectively 
> random number and just happily uses it.

I will agree with what Daniele said - push on GuC fw folks to document 
the max values they guarantee to support in the interface spec. 
Otherwise it is too fragile.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 23/02/2022 02:45, John Harrison wrote:
> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>
>>> Compute workloads are inherantly not pre-emptible for long periods on
>>> current hardware. As a workaround for this, the pre-emption timeout
>>> for compute capable engines was disabled. This is undesirable with GuC
>>> submission as it prevents per engine reset of hung contexts. Hence the
>>> next patch will re-enable the timeout but bumped up by an order of
>>> magnititude.
>>
>> (Some typos above.)
> I'm spotting 'inherently' but not anything else.

Magnititude! O;)

>>> However, the heartbeat might not respect that. Depending upon current
>>> activity, a pre-emption to the heartbeat pulse might not even be
>>> attempted until the last heartbeat period. Which means that only one
>>
>> Might not be attempted, but could be if something is running with 
>> lower priority. In which case I think special casing the last 
>> heartbeat does not feel right because it can end up resetting the 
>> engine before it was intended.
>>
>> Like if first heartbeat decides to preempt (the decision is backend 
>> specific, could be same prio + timeslicing), and preempt timeout has 
>> been set to heartbeat interval * 3, then 2nd heartbeat gets queued up, 
>> then 3rd, and so reset is triggered even before the first preempt 
>> timeout legitimately expires (or just as it is about to react).
>>
>> Instead, how about preempt timeout is always considered when 
>> calculating when to emit the next heartbeat? End result would be 
>> similar to your patch, in terms of avoiding the direct problem, 
>> although hang detection would be overall longer (but more correct I 
>> think).
>>
>> And it also means in the next patch you don't have to add coupling 
>> between preempt timeout and heartbeat to intel_engine_setup. Instead 
>> just some long preempt timeout would be needed. Granted, the 
>> decoupling argument is not super strong since then the heartbeat code 
>> has the coupling instead, but that still feels better to me. (Since we 
>> can say heartbeats only make sense on loaded engines, and so things 
>> like preempt timeout can legitimately be considered from there.)
>>
>> Incidentally, that would be similar to a patch which Chris had a year 
>> ago 
>> (https://patchwork.freedesktop.org/patch/419783/?series=86841&rev=1) 
>> to fix some CI issue.
>>
> I'm not following your arguments.
> 
> Chris' patch is about not having two i915 based resets triggered 
> concurrently - i915 based engine reset and i915 based GT reset. The 
> purpose of this patch is to allow the GuC based engine reset to have a 
> chance to occur before the i915 based GT reset kicks in.
> 
> It sounds like your argument above is about making the engine reset 
> slower so that it doesn't happen before the appropriate heartbeat period 
> for that potential reset scenario has expired. I don't see why that is 
> at all necessary or useful.
> 
> If an early heartbeat period triggers an engine reset then the heartbeat 
> pulse will go through. The heartbeat will thus see a happy system and 
> not do anything further. If the given period does not trigger an engine 
> reset but still does not get the pulse through (because the pulse is of 
> too low a priority) then we move on to the next period and bump the 
> priority. If the pre-emption has actually already been triggered anyway 
> (and we are just waiting a while for it to timeout) then that's fine. 
> The priority bump will have no effect because the context is already 
> attempting to run. The heartbeat code doesn't care which priority level 
> actually triggers the reset. It just cares whether or not the pulse 
> finally makes it through. And the GuC doesn't care which heartbeat 
> period the i915 is in. All it knows is that it has a request to schedule 
> and whether the current context is pre-empting or not. So if period #1 
> triggers the pre-emption but the timeout doesn't happen until period #3, 
> who cares? The result is the same as if period #3 triggered the 
> pre-emption and the timeout was shorter. The result being that the hung 
> context is reset, the pulse makes it through and the heartbeat goes to 
> sleep again.
> 
> The only period that really matters is the final one. At that point the 
> pulse request is at highest priority and so must trigger a pre-emption 
> request. We then need at least one full pre-emption period (plus some 
> wiggle room for random delays in reset time, context switching, 
> processing messages, etc.) to allow the GuC based timeout and reset to 
> occur. Hence ensuring that the final heartbeat period is at least twice 
> the pre-emption timeout (because 1.25 times is just messy when working 
> with ints!).
> 
> That guarantees that GuC will get at least one complete opportunity to 
> detect and recover the hang before i915 nukes the universe.
> 
> Whereas, bumping all heartbeat periods to be greater than the 
> pre-emption timeout is wasteful and unnecessary. That leads to a total 
> heartbeat time of about a minute. Which is a very long time to wait for 
> a hang to be detected and recovered. Especially when the official limit 
> on a context responding to an 'are you dead' query is only 7.5 seconds.

Not sure how did you get one minute?

Regardless, crux of argument was to avoid GuC engine reset and heartbeat reset racing with each other, and to do that by considering the preempt timeout with the heartbeat interval. I was thinking about this scenario in this series:

[Please use fixed width font and no line wrap to view.]

A)

tP = preempt timeout
tH = hearbeat interval

tP = 3 * tH

1) Background load = I915_PRIORITY_DISPLAY

<-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * tH] ----> FULL RESET
                                                        |
                                                        \- preemption triggered, tP = 3 * tH ------\
                                                                                                    \-> preempt timeout would hit here

Here we have collateral damage due full reset, since we can't tell GuC to reset just one engine and we fudged tP just to "account" for heartbeats.

2) Background load = I915_CONTEXT_MAX_USER_PRIORITY

<-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * tH] ----> FULL RESET
                                     |
                                     \- preemption triggered, tP = 3 * tH ------> Preempt timeout races with "heartbeat stopped"

Here possible collateral damage due non-deterministic race (GuC and i915 run different clocks, or even if they did not). Can we do better?
                                                                                                   
3) Background load = I915_CONTEXT_MIN_USER_PRIORITY

<-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * tH] ----> full reset would be here
                |
                \- preemption triggered, tP = 3 * tH ----------------\
                                                                      \-> Preempt timeout reset

Here is is kind of least worse, but question is why we fudged tP when it gives us nothing good in this case.

B)

Instead, my idea to account for preempt timeout when calculating when to schedule next hearbeat would look like this:

First of all tP can be left at a large value unrelated to tH. Lets say tP = 640ms. tH stays 2.5s.

1) Background load = I915_PRIORITY_DISPLAY

<-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> Pulse3 <-- [tH + tP] --> full reset would be here
                                                                       |
                                                                       \-[tP]-> Engine reset here
                                                                                                    
Collateral damage avoided here.

2) Background load = I915_CONTEXT_MAX_USER_PRIORITY

<-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> Pulse3 <-- [tH + tP] --> full reset would be here
                                              |
                                              \-[tP]-> Engine reset here

No race, collateral damage avoided. Not sure race can be hit due built-in interdependency between tH and tP, unless maybe for really short timeouts which are not realistic to start with.
                                                                                                   
3) Background load = I915_CONTEXT_MIN_USER_PRIORITY

<-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> Pulse3 <-- [tH + tP] --> full reset would be here
                     |
                     \-[tP]-> Engine reset here
                                                                      
So as long as "640ms" preempt timeout is good enough for compute this option looks better to me. Or even if it is not, the timelines for option A above show engine reset effectively doesn't work for compute anyway. So if they set it to zero manually it's pretty much the same full reset for them.

Am I missing some requirement or you see another problem with this idea?

>> On a related topic, if GuC engine resets stop working when preempt 
>> timeout is set to zero - I think we need to somehow let the user know 
>> if they try to tweak it via sysfs. Perhaps go as far as -EINVAL in GuC 
>> mode, if i915.reset has not explicitly disabled engine resets.
> Define 'stops working'. The definition of the sysfs interface is that a 
> value of zero disables pre-emption. If you don't have pre-emption and 
> your hang detection mechanism relies on pre-emption then you don't have 
> a hang detection mechanism either. If the user really wants to allow 

By stops working I meant that it stops working. :)

With execlist one can disable preempt timeout and "stopped heartbeat" can still reset the stuck engine and so avoid collateral damage. With GuC it appears this is not possible. So I was thinking this is something worthy a log notice.

> their context to run forever and never be pre-empted then that means 
> they also don't want it to be reset arbitrarily. Which means they would 
> also be disabling the heartbeat timer as well. Indeed, this is what we 

I don't think so. Preempt timeout is disabled already on TGL/RCS upstream but hearbeat is not and so hangcheck still works.

> advise compute customers to do. It is then up to the user themselves to 
> spot a hang and to manually kill (Ctrl+C, kill ###, etc.) their task. 
> Killing the CPU task will automatically clear up any GPU resources 
> allocated to that task (excepting context persistence, which is a) 
> broken and b) something we also tell compute customers to disable).

What is broken with context persistence? I noticed one patch claiming to be fixing something in that area which looked suspect. Has it been established no userspace relies on it?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/23/2022 04:13, Tvrtko Ursulin wrote:
> On 23/02/2022 02:11, John Harrison wrote:
>> On 2/22/2022 01:52, Tvrtko Ursulin wrote:
>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>
>>>> GuC converts the pre-emption timeout and timeslice quantum values into
>>>> clock ticks internally. That significantly reduces the point of 32bit
>>>> overflow. On current platforms, worst case scenario is approximately
>>>
>>> Where does 32-bit come from, the GuC side? We already use 64-bits so 
>>> that something to fix to start with. Yep...
>> Yes, the GuC API is defined as 32bits only and then does a straight 
>> multiply by the clock speed with no range checking. We have requested 
>> 64bit support but there was push back on the grounds that it is not 
>> something the GuC timer hardware supports and such long timeouts are 
>> not real world usable anyway.
>
> As long as compute are happy with 100 seconds, then it "should be 
> enough for everbody". :D
Compute disable all forms of reset and rely on manual kill. So yes.

But even if they aren't. That's all we can do at the moment. If there is 
a genuine customer requirement for more then we can push for full 64bit 
software implemented timers in the GuC but until that happens, we don't 
have much choice.

>
>>>
>>> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>>>
>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>> engine->props.timeslice_duration_ms * 1000;
>>>
>>> ./gt/intel_engine_types.h:              unsigned long 
>>> timeslice_duration_ms;
>>>
>>> timeslice_store/preempt_timeout_store:
>>> err = kstrtoull(buf, 0, &duration);
>>>
>>> So both kconfig and sysfs can already overflow GuC, not only because 
>>> of tick conversion internally but because at backend level nothing 
>>> was done for assigning 64-bit into 32-bit. Or I failed to find where 
>>> it is handled.
>> That's why I'm adding this range check to make sure we don't allow 
>> overflows.
>
> Yes and no, this fixes it, but the first bug was not only due GuC 
> internal tick conversion. It was present ever since the u64 from i915 
> was shoved into u32 sent to GuC. So even if GuC used the value without 
> additional multiplication, bug was be there. My point being when GuC 
> backend was added timeout_ms values should have been limited/clamped 
> to U32_MAX. The tick discovery is additional limit on top.
I'm not disagreeing. I'm just saying that the truncation wasn't noticed 
until I actually tried using very long timeouts to debug a particular 
problem. Now that it is noticed, we need some method of range checking 
and this simple clamp solves all the truncation problems.


>
>>>> 110 seconds. Rather than allowing the user to set higher values and
>>>> then get confused by early timeouts, add limits when setting these
>>>> values.
>>>
>>> Btw who is reviewing GuC patches these days - things have somehow 
>>> gotten pretty quiet in activity and I don't think that's due absence 
>>> of stuff to improve or fix? Asking since I think I noticed a few 
>>> already which you posted and then crickets on the mailing list.
>> Too much work to do and not enough engineers to do it all :(.
>>
>>
>>>
>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>> ---
>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>   3 files changed, 38 insertions(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>>>> *gt, enum intel_engine_id id,
>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>>>           engine->props.preempt_timeout_ms = 0;
>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>> +        if (engine->props.timeslice_duration_ms > 
>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>
>>> Hm "wanted".. There's been too much back and forth on the GuC load 
>>> options over the years to keep track.. intel_engine_uses_guc work 
>>> sounds like would work and read nicer.
>> I'm not adding a new feature check here. I'm just using the existing 
>> one. If we want to rename it yet again then that would be a different 
>> patch set.
>
> $ grep intel_engine_uses_guc . -rl
> ./i915_perf.c
> ./i915_request.c
> ./selftests/intel_scheduler_helpers.c
> ./gem/i915_gem_context.c
> ./gt/intel_context.c
> ./gt/intel_engine.h
> ./gt/intel_engine_cs.c
> ./gt/intel_engine_heartbeat.c
> ./gt/intel_engine_pm.c
> ./gt/intel_reset.c
> ./gt/intel_lrc.c
> ./gt/selftest_context.c
> ./gt/selftest_engine_pm.c
> ./gt/selftest_hangcheck.c
> ./gt/selftest_mocs.c
> ./gt/selftest_workarounds.c
>
> Sounds better to me than intel_guc_submission_is_wanted. What does the 
> reader know whether "is wanted" translates to "is actually used". 
> Shrug on "is wanted".
Yes, but isn't '_uses' the one that hits a BUG_ON if you call it too 
early in the boot up sequence? I never understood why that was necessary 
or why we need so many different ways to ask the same question. But this 
version already exists and definitely works without hitting any explosions.

>
>>> And limit to class instead of applying to all engines looks like a 
>>> miss.
>> As per follow up email, the class limit is not applied here.
>>
>>>
>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>> duration to %d to prevent possibly overflow\n",
>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>> +            engine->props.timeslice_duration_ms = 
>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>
>>> I am not sure logging such message during driver load is useful. 
>>> Sounds more like a confused driver which starts with one value and 
>>> then overrides itself. I'd just silently set the value appropriate 
>>> for the active backend. Preemption timeout kconfig text already 
>>> documents the fact timeouts can get overriden at runtime depending 
>>> on platform+engine. So maybe just add same text to timeslice kconfig.
>> The point is to make people aware if they compile with unsupported 
>> config options. As far as I know, there is no way to apply range 
>> checking or other limits to config defines. Which means that a user 
>> would silently get unwanted behaviour. That seems like a bad thing to 
>> me. If the driver is confused because the user built it in a confused 
>> manner then we should let them know.
>
> Okay, but I think make it notice low level.
>
> Also consider in patch 3/3 when you triple it, and then clamp back 
> down here. That's even more confused state since tripling gets nerfed. 
> I think that's also an argument to always account preempt timeout in 
> heartbeat interval calculation. Haven't got to your reply on 2/3 yet 
> though..
That sounds like even more reason to make sure the warning gets seen. 
The more complex the system and the more chances there are to get it 
wrong, the more important it is to have a nice easy to see and 
understand notification that it did go wrong.


>
>>>
>>>> +        }
>>>> +
>>>> +        if (engine->props.preempt_timeout_ms > 
>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>>>> pre-emption timeout to %d to prevent possibly overflow\n",
>>>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>>>> +            engine->props.preempt_timeout_ms = 
>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>> +        }
>>>> +    }
>>>> +
>>>>       engine->defaults = engine->props; /* never to change again */
>>>>         engine->context_size = intel_engine_context_size(gt, 
>>>> engine->class);
>>>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>>>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>> index 967031056202..f57efe026474 100644
>>>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>>>> kobj_attribute *attr,
>>>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>           return -EINVAL;
>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>> duration to %lld to prevent possibly overflow\n",
>>>> +             duration);
>>>> +    }
>>>
>>> I would suggest to avoid duplicated clamping logic. Maybe hide the 
>>> all backend logic into the helpers then, like maybe:
>>>
>>>   d = intel_engine_validate_timeslice/preempt_timeout(engine, 
>>> duration);
>>>   if (d != duration)
>>>     return -EINVAL:
>>>
>>> Returning -EINVAL would be equivalent to existing behaviour:
>>>
>>>     if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>         return -EINVAL;
>>>
>>> That way userspace has explicit notification and read-back is 
>>> identical to written in value. From engine setup you can just call 
>>> the helper silently.
>> Sure, EINVAL rather than clamping works as well. And can certainly 
>> add helper wrappers. But as above, I don't like the idea of silently 
>> disregarding a user specified config option.
>
> Deal - with the open of heartbeat interval TBD.
>
>>
>>>
>>>> +
>>>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>>>         if (execlists_active(&engine->execlists))
>>>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>>>> struct kobj_attribute *attr,
>>>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>           return -EINVAL;
>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>> +        drm_info(&engine->i915->drm, "Warning, clamping 
>>>> pre-emption timeout to %lld to prevent possibly overflow\n",
>>>> +             timeout);
>>>> +    }
>>>> +
>>>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>>>         if (READ_ONCE(engine->execlists.pending[0]))
>>>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>>>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>> index 6a4612a852e2..ad131092f8df 100644
>>>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>>>   +/*
>>>> + * GuC converts the timeout to clock ticks internally. Different 
>>>> platforms have
>>>> + * different GuC clocks. Thus, the maximum value before overflow 
>>>> is platform
>>>> + * dependent. Current worst case scenario is about 110s. So, limit 
>>>> to 100s to be
>>>> + * safe.
>>>> + */
>>>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>>>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>>>
>>> Most important question -
>>> how will we know/notice if/when new GuC arrives where these timeouts 
>>> would still overflow? Can this be queried somehow at runtime or 
>>> where does the limit comes from? How is GuC told about it? Set in 
>>> some field and it just allows too large values silently break things?
>> Currently, we don't notice except by debugging peculiar test failures 
>> :(.
>>
>> These limits are not in any GuC spec. Indeed, it took a while to 
>> actually work out why increasing the value actually caused shorter 
>> timeouts to occur! As above, there is no range checking inside GuC 
>> itself. It does a truncated multiply which results in an effectively 
>> random number and just happily uses it.
>
> I will agree with what Daniele said - push on GuC fw folks to document 
> the max values they guarantee to support in the interface spec. 
> Otherwise it is too fragile.
I do agree. But that is going to take time. I would like to get 
something merged now while we fight over spec updates.

John.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 2/23/2022 05:58, Tvrtko Ursulin wrote:
> On 23/02/2022 02:45, John Harrison wrote:
>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>
>>>> Compute workloads are inherantly not pre-emptible for long periods on
>>>> current hardware. As a workaround for this, the pre-emption timeout
>>>> for compute capable engines was disabled. This is undesirable with GuC
>>>> submission as it prevents per engine reset of hung contexts. Hence the
>>>> next patch will re-enable the timeout but bumped up by an order of
>>>> magnititude.
>>>
>>> (Some typos above.)
>> I'm spotting 'inherently' but not anything else.
>
> Magnititude! O;)
Doh!

[snip]

>> Whereas, bumping all heartbeat periods to be greater than the 
>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>> total heartbeat time of about a minute. Which is a very long time to 
>> wait for a hang to be detected and recovered. Especially when the 
>> official limit on a context responding to an 'are you dead' query is 
>> only 7.5 seconds.
>
> Not sure how did you get one minute?
7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 75 => 
1 minute 15 seconds

Even ignoring any safety factor and just going with 7.5 * 5 still gets 
you to 37.5 seconds which is over a half a minute and likely to race.

>
> Regardless, crux of argument was to avoid GuC engine reset and 
> heartbeat reset racing with each other, and to do that by considering 
> the preempt timeout with the heartbeat interval. I was thinking about 
> this scenario in this series:
>
> [Please use fixed width font and no line wrap to view.]
>
> A)
>
> tP = preempt timeout
> tH = hearbeat interval
>
> tP = 3 * tH
>
> 1) Background load = I915_PRIORITY_DISPLAY
>
> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * 
> tH] ----> FULL RESET
>                                                        |
>                                                        \- preemption 
> triggered, tP = 3 * tH ------\
> \-> preempt timeout would hit here
>
> Here we have collateral damage due full reset, since we can't tell GuC 
> to reset just one engine and we fudged tP just to "account" for 
> heartbeats.
You are missing the whole point of the patch series which is that the 
last heartbeat period is '2 * tP' not '2 * tH'.
+        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;

By making the last period double the pre-emption timeout, it is 
guaranteed that the FULL RESET stage cannot be hit before the hardware 
has attempted and timed-out on at least one pre-emption.

[snip]


> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * 
> tH] ----> full reset would be here
>                |
>                \- preemption triggered, tP = 3 * tH ----------------\
> \-> Preempt timeout reset
>
> Here is is kind of least worse, but question is why we fudged tP when 
> it gives us nothing good in this case.
>
The point of fudging tP(RCS) is to give compute workloads longer to 
reach a pre-emptible point (given that EU walkers are basically not 
pre-emptible). The reason for doing the fudge is not connected to the 
heartbeat at all. The fact that it causes problems for the heartbeat is 
an undesired side effect.

Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
calculation that gave us something that all interested parties were 
vaguely happy with. It could just as easily be a fixed, hard coded value 
of 7.5s but having it based on something configurable seemed more 
sensible. The other option was 'tP(RCS) = tP * 12' but that felt more 
arbitrary than basing it on the average heartbeat timeout. As in, three 
heartbeat periods is about what a normal prio task gets before it gets 
pre-empted by the heartbeat. So using that for general purpose 
pre-emptions (e.g. time slicing between multiple user apps) seems 
reasonable.


> B)
>
> Instead, my idea to account for preempt timeout when calculating when 
> to schedule next hearbeat would look like this:
>
> First of all tP can be left at a large value unrelated to tH. Lets say 
> tP = 640ms. tH stays 2.5s.
640ms is not 'large'. The requirement is either zero (disabled) or 
region of 7.5s. The 640ms figure is the default for non-compute engines. 
Anything that can run EUs needs to be 'huge'.


>
> 1) Background load = I915_PRIORITY_DISPLAY
>
> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> 
> Pulse3 <-- [tH + tP] --> full reset would be here
Sure, this works but each period is now 2.5 + 7.5 = 10s. The full five 
periods is therefore 50s, which is practically a minute.

[snip]

> Am I missing some requirement or you see another problem with this idea?
>
>>> On a related topic, if GuC engine resets stop working when preempt 
>>> timeout is set to zero - I think we need to somehow let the user 
>>> know if they try to tweak it via sysfs. Perhaps go as far as -EINVAL 
>>> in GuC mode, if i915.reset has not explicitly disabled engine resets.
>> Define 'stops working'. The definition of the sysfs interface is that 
>> a value of zero disables pre-emption. If you don't have pre-emption 
>> and your hang detection mechanism relies on pre-emption then you 
>> don't have a hang detection mechanism either. If the user really 
>> wants to allow 
>
> By stops working I meant that it stops working. :)
>
> With execlist one can disable preempt timeout and "stopped heartbeat" 
> can still reset the stuck engine and so avoid collateral damage. With 
> GuC it appears this is not possible. So I was thinking this is 
> something worthy a log notice.
>
>> their context to run forever and never be pre-empted then that means 
>> they also don't want it to be reset arbitrarily. Which means they 
>> would also be disabling the heartbeat timer as well. Indeed, this is 
>> what we 
>
> I don't think so. Preempt timeout is disabled already on TGL/RCS 
> upstream but hearbeat is not and so hangcheck still works.
The pre-emption disable in upstream is not a valid solution for compute 
customers. It is a worst-of-all-worlds hack for general usage. As noted 
already, any actual compute specific customer is advised to disable all 
forms of reset and do their hang detection manually. A slightly less 
worse hack for customers that are not actually running long compute 
workloads (i.e. the vast majority of end users) is to just use a long 
pre-emption timeout.

>
>> advise compute customers to do. It is then up to the user themselves 
>> to spot a hang and to manually kill (Ctrl+C, kill ###, etc.) their 
>> task. Killing the CPU task will automatically clear up any GPU 
>> resources allocated to that task (excepting context persistence, 
>> which is a) broken and b) something we also tell compute customers to 
>> disable).
>
> What is broken with context persistence? I noticed one patch claiming 
> to be fixing something in that area which looked suspect. Has it been 
> established no userspace relies on it?
One major issue is that it has hooks into the execlist scheduler 
backend. I forget the exact details right now. The implementation as a 
whole is incredibly complex and convoluted :(. But there's stuff about 
what happens when you disable the heartbeat after having closed a 
persistence context's handle (and thus made it persisting). There's also 
things like it sends a super high priority heartbeat pulse at the point 
of becoming persisting. That plays havoc for platforms with dependent 
engines and/or compute workloads. A context becomes persisting on RCS 
and results in your unrealted CCS work being reset. It's a mess.

The comment from Daniel Vetter is that persistence should have no 
connection to the heartbeat at all. All of that dynamic behaviour and 
complexity should just be removed.

Persistence itself can stay. There are valid UMD use cases. It is just 
massively over complicated and doesn't work in all corner cases when not 
using execlist submission or on newer platforms. The simplification that 
is planned is to allow contexts to persist until the associated DRM 
master handle is closed. At that point, all contexts associated with 
that DRM handle are killed. That is what AMD and others apparently 
implement.

John.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 23/02/2022 19:03, John Harrison wrote:
> On 2/23/2022 04:13, Tvrtko Ursulin wrote:
>> On 23/02/2022 02:11, John Harrison wrote:
>>> On 2/22/2022 01:52, Tvrtko Ursulin wrote:
>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>
>>>>> GuC converts the pre-emption timeout and timeslice quantum values into
>>>>> clock ticks internally. That significantly reduces the point of 32bit
>>>>> overflow. On current platforms, worst case scenario is approximately
>>>>
>>>> Where does 32-bit come from, the GuC side? We already use 64-bits so 
>>>> that something to fix to start with. Yep...
>>> Yes, the GuC API is defined as 32bits only and then does a straight 
>>> multiply by the clock speed with no range checking. We have requested 
>>> 64bit support but there was push back on the grounds that it is not 
>>> something the GuC timer hardware supports and such long timeouts are 
>>> not real world usable anyway.
>>
>> As long as compute are happy with 100 seconds, then it "should be 
>> enough for everbody". :D
> Compute disable all forms of reset and rely on manual kill. So yes.
> 
> But even if they aren't. That's all we can do at the moment. If there is 
> a genuine customer requirement for more then we can push for full 64bit 
> software implemented timers in the GuC but until that happens, we don't 
> have much choice.

Yeah.

> 
>>
>>>>
>>>> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>>>>
>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>> engine->props.timeslice_duration_ms * 1000;
>>>>
>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>> timeslice_duration_ms;
>>>>
>>>> timeslice_store/preempt_timeout_store:
>>>> err = kstrtoull(buf, 0, &duration);
>>>>
>>>> So both kconfig and sysfs can already overflow GuC, not only because 
>>>> of tick conversion internally but because at backend level nothing 
>>>> was done for assigning 64-bit into 32-bit. Or I failed to find where 
>>>> it is handled.
>>> That's why I'm adding this range check to make sure we don't allow 
>>> overflows.
>>
>> Yes and no, this fixes it, but the first bug was not only due GuC 
>> internal tick conversion. It was present ever since the u64 from i915 
>> was shoved into u32 sent to GuC. So even if GuC used the value without 
>> additional multiplication, bug was be there. My point being when GuC 
>> backend was added timeout_ms values should have been limited/clamped 
>> to U32_MAX. The tick discovery is additional limit on top.
> I'm not disagreeing. I'm just saying that the truncation wasn't noticed 
> until I actually tried using very long timeouts to debug a particular 
> problem. Now that it is noticed, we need some method of range checking 
> and this simple clamp solves all the truncation problems.

Agreed in principle, just please mention in the commit message all aspects of the problem.

I think we can get away without a Fixes: tag since it requires user fiddling to break things in unexpected ways.

I would though put in a code a clamping which expresses both, something like min(u32, ..GUC LIMIT..). So the full story is documented forever. Or "if > u32 || > ..GUC LIMIT..) return -EINVAL". Just in case GuC limit one day changes but u32 stays. Perhaps internal ticks go away or anything and we are left with plain 1:1 millisecond relationship.

>>>>> 110 seconds. Rather than allowing the user to set higher values and
>>>>> then get confused by early timeouts, add limits when setting these
>>>>> values.
>>>>
>>>> Btw who is reviewing GuC patches these days - things have somehow 
>>>> gotten pretty quiet in activity and I don't think that's due absence 
>>>> of stuff to improve or fix? Asking since I think I noticed a few 
>>>> already which you posted and then crickets on the mailing list.
>>> Too much work to do and not enough engineers to do it all :(.
>>>
>>>
>>>>
>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>> ---
>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>>   3 files changed, 38 insertions(+)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct intel_gt 
>>>>> *gt, enum intel_engine_id id,
>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>
>>>> Hm "wanted".. There's been too much back and forth on the GuC load 
>>>> options over the years to keep track.. intel_engine_uses_guc work 
>>>> sounds like would work and read nicer.
>>> I'm not adding a new feature check here. I'm just using the existing 
>>> one. If we want to rename it yet again then that would be a different 
>>> patch set.
>>
>> $ grep intel_engine_uses_guc . -rl
>> ./i915_perf.c
>> ./i915_request.c
>> ./selftests/intel_scheduler_helpers.c
>> ./gem/i915_gem_context.c
>> ./gt/intel_context.c
>> ./gt/intel_engine.h
>> ./gt/intel_engine_cs.c
>> ./gt/intel_engine_heartbeat.c
>> ./gt/intel_engine_pm.c
>> ./gt/intel_reset.c
>> ./gt/intel_lrc.c
>> ./gt/selftest_context.c
>> ./gt/selftest_engine_pm.c
>> ./gt/selftest_hangcheck.c
>> ./gt/selftest_mocs.c
>> ./gt/selftest_workarounds.c
>>
>> Sounds better to me than intel_guc_submission_is_wanted. What does the 
>> reader know whether "is wanted" translates to "is actually used". 
>> Shrug on "is wanted".
> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it too 
> early in the boot up sequence? I never understood why that was necessary 
> or why we need so many different ways to ask the same question. But this 
> version already exists and definitely works without hitting any explosions.

No idea if it causes a bug on, doesn't in the helper itself so maybe you are saying it is called too early? Might be.. I think over time the nice idea we had that "setup" and "init" phases of engine setup clearly separated got destroyed a bit. There would always be an option to move this clamping in a later phase, once the submission method is known. One could argue that if the submission method is not yet known at this point, it is even wrong to clamp based on something which will only be decided later. Because:

int intel_engines_init(struct intel_gt *gt)
{
	int (*setup)(struct intel_engine_cs *engine);
	struct intel_engine_cs *engine;
	enum intel_engine_id id;
	int err;

	if (intel_uc_uses_guc_submission(&gt->uc)) {
		gt->submission_method = INTEL_SUBMISSION_GUC;

So this uses "uses", not "wanted". Presumably the point for having "wanted" and "uses" is that they can disagree, in which case if you clamp early based on "wanted" that suggests it could be wrong.

>>>> And limit to class instead of applying to all engines looks like a 
>>>> miss.
>>> As per follow up email, the class limit is not applied here.
>>>
>>>>
>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>> duration to %d to prevent possibly overflow\n",
>>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>> +            engine->props.timeslice_duration_ms = 
>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>
>>>> I am not sure logging such message during driver load is useful. 
>>>> Sounds more like a confused driver which starts with one value and 
>>>> then overrides itself. I'd just silently set the value appropriate 
>>>> for the active backend. Preemption timeout kconfig text already 
>>>> documents the fact timeouts can get overriden at runtime depending 
>>>> on platform+engine. So maybe just add same text to timeslice kconfig.
>>> The point is to make people aware if they compile with unsupported 
>>> config options. As far as I know, there is no way to apply range 
>>> checking or other limits to config defines. Which means that a user 
>>> would silently get unwanted behaviour. That seems like a bad thing to 
>>> me. If the driver is confused because the user built it in a confused 
>>> manner then we should let them know.
>>
>> Okay, but I think make it notice low level.
>>
>> Also consider in patch 3/3 when you triple it, and then clamp back 
>> down here. That's even more confused state since tripling gets nerfed. 
>> I think that's also an argument to always account preempt timeout in 
>> heartbeat interval calculation. Haven't got to your reply on 2/3 yet 
>> though..
> That sounds like even more reason to make sure the warning gets seen. 
> The more complex the system and the more chances there are to get it 
> wrong, the more important it is to have a nice easy to see and 
> understand notification that it did go wrong.

I did not disagree, just said make it notice, one level higher than info! :)

But also think how, if we agree to go with tripling, that you'd have to consider that in the sysfs store when hearbeat timeout is written, to consider whether or not to triple and error out if preemption timeout is over limit.

>>>>> +        }
>>>>> +
>>>>> +        if (engine->props.preempt_timeout_ms > 
>>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>>>>> pre-emption timeout to %d to prevent possibly overflow\n",
>>>>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>>>>> +            engine->props.preempt_timeout_ms = 
>>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>>> +        }
>>>>> +    }
>>>>> +
>>>>>       engine->defaults = engine->props; /* never to change again */
>>>>>         engine->context_size = intel_engine_context_size(gt, 
>>>>> engine->class);
>>>>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>>>>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>> index 967031056202..f57efe026474 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>>>>> kobj_attribute *attr,
>>>>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>>           return -EINVAL;
>>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>> +        drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>> duration to %lld to prevent possibly overflow\n",
>>>>> +             duration);
>>>>> +    }
>>>>
>>>> I would suggest to avoid duplicated clamping logic. Maybe hide the 
>>>> all backend logic into the helpers then, like maybe:
>>>>
>>>>   d = intel_engine_validate_timeslice/preempt_timeout(engine, 
>>>> duration);
>>>>   if (d != duration)
>>>>     return -EINVAL:
>>>>
>>>> Returning -EINVAL would be equivalent to existing behaviour:
>>>>
>>>>     if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>         return -EINVAL;
>>>>
>>>> That way userspace has explicit notification and read-back is 
>>>> identical to written in value. From engine setup you can just call 
>>>> the helper silently.
>>> Sure, EINVAL rather than clamping works as well. And can certainly 
>>> add helper wrappers. But as above, I don't like the idea of silently 
>>> disregarding a user specified config option.
>>
>> Deal - with the open of heartbeat interval TBD.
>>
>>>
>>>>
>>>>> +
>>>>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>>>>         if (execlists_active(&engine->execlists))
>>>>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>>>>> struct kobj_attribute *attr,
>>>>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>>           return -EINVAL;
>>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>>> +        drm_info(&engine->i915->drm, "Warning, clamping 
>>>>> pre-emption timeout to %lld to prevent possibly overflow\n",
>>>>> +             timeout);
>>>>> +    }
>>>>> +
>>>>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>>>>         if (READ_ONCE(engine->execlists.pending[0]))
>>>>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>>>>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>> index 6a4612a852e2..ad131092f8df 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>>>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>>>>   +/*
>>>>> + * GuC converts the timeout to clock ticks internally. Different 
>>>>> platforms have
>>>>> + * different GuC clocks. Thus, the maximum value before overflow 
>>>>> is platform
>>>>> + * dependent. Current worst case scenario is about 110s. So, limit 
>>>>> to 100s to be
>>>>> + * safe.
>>>>> + */
>>>>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>>>>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>>>>
>>>> Most important question -
>>>> how will we know/notice if/when new GuC arrives where these timeouts 
>>>> would still overflow? Can this be queried somehow at runtime or 
>>>> where does the limit comes from? How is GuC told about it? Set in 
>>>> some field and it just allows too large values silently break things?
>>> Currently, we don't notice except by debugging peculiar test failures 
>>> :(.
>>>
>>> These limits are not in any GuC spec. Indeed, it took a while to 
>>> actually work out why increasing the value actually caused shorter 
>>> timeouts to occur! As above, there is no range checking inside GuC 
>>> itself. It does a truncated multiply which results in an effectively 
>>> random number and just happily uses it.
>>
>> I will agree with what Daniele said - push on GuC fw folks to document 
>> the max values they guarantee to support in the interface spec. 
>> Otherwise it is too fragile.
> I do agree. But that is going to take time. I would like to get 
> something merged now while we fight over spec updates.

Yeah that's okay, did not mean to imply I am against a quick fix. "Otherwise it is too fragile, *in the long run*" should have written or something like that.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 23/02/2022 20:00, John Harrison wrote:
> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>> On 23/02/2022 02:45, John Harrison wrote:
>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>
>>>>> Compute workloads are inherantly not pre-emptible for long periods on
>>>>> current hardware. As a workaround for this, the pre-emption timeout
>>>>> for compute capable engines was disabled. This is undesirable with GuC
>>>>> submission as it prevents per engine reset of hung contexts. Hence the
>>>>> next patch will re-enable the timeout but bumped up by an order of
>>>>> magnititude.
>>>>
>>>> (Some typos above.)
>>> I'm spotting 'inherently' but not anything else.
>>
>> Magnititude! O;)
> Doh!
> 
> [snip]
> 
>>> Whereas, bumping all heartbeat periods to be greater than the 
>>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>>> total heartbeat time of about a minute. Which is a very long time to 
>>> wait for a hang to be detected and recovered. Especially when the 
>>> official limit on a context responding to an 'are you dead' query is 
>>> only 7.5 seconds.
>>
>> Not sure how did you get one minute?
> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 75 => 
> 1 minute 15 seconds
> 
> Even ignoring any safety factor and just going with 7.5 * 5 still gets 
> you to 37.5 seconds which is over a half a minute and likely to race.

Ah because my starting point is there should be no preempt timeout = heartbeat * 3, I just think that's too ugly.

>> Regardless, crux of argument was to avoid GuC engine reset and 
>> heartbeat reset racing with each other, and to do that by considering 
>> the preempt timeout with the heartbeat interval. I was thinking about 
>> this scenario in this series:
>>
>> [Please use fixed width font and no line wrap to view.]
>>
>> A)
>>
>> tP = preempt timeout
>> tH = hearbeat interval
>>
>> tP = 3 * tH
>>
>> 1) Background load = I915_PRIORITY_DISPLAY
>>
>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * 
>> tH] ----> FULL RESET
>>                                                        |
>>                                                        \- preemption 
>> triggered, tP = 3 * tH ------\
>> \-> preempt timeout would hit here
>>
>> Here we have collateral damage due full reset, since we can't tell GuC 
>> to reset just one engine and we fudged tP just to "account" for 
>> heartbeats.
> You are missing the whole point of the patch series which is that the 
> last heartbeat period is '2 * tP' not '2 * tH'.
> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
> 
> By making the last period double the pre-emption timeout, it is 
> guaranteed that the FULL RESET stage cannot be hit before the hardware 
> has attempted and timed-out on at least one pre-emption.

Oh well :) that probably means the overall scheme is too odd for me. tp = 3tH and last pulse after 2tP I mean.

> [snip]
> 
> 
>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 * 
>> tH] ----> full reset would be here
>>                |
>>                \- preemption triggered, tP = 3 * tH ----------------\
>> \-> Preempt timeout reset
>>
>> Here is is kind of least worse, but question is why we fudged tP when 
>> it gives us nothing good in this case.
>>
> The point of fudging tP(RCS) is to give compute workloads longer to 
> reach a pre-emptible point (given that EU walkers are basically not 
> pre-emptible). The reason for doing the fudge is not connected to the 
> heartbeat at all. The fact that it causes problems for the heartbeat is 
> an undesired side effect.
> 
> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
> calculation that gave us something that all interested parties were 
> vaguely happy with. It could just as easily be a fixed, hard coded value 
> of 7.5s but having it based on something configurable seemed more 
> sensible. The other option was 'tP(RCS) = tP * 12' but that felt more 
> arbitrary than basing it on the average heartbeat timeout. As in, three 
> heartbeat periods is about what a normal prio task gets before it gets 
> pre-empted by the heartbeat. So using that for general purpose 
> pre-emptions (e.g. time slicing between multiple user apps) seems 
> reasonable.

I think the fact you say tP fudge is not related to heartbeats and then go to mention heartbeat even in the "formula" which uses no tH is saying something (at least that's how I read the 7.5s option). :)

>> B)
>>
>> Instead, my idea to account for preempt timeout when calculating when 
>> to schedule next hearbeat would look like this:
>>
>> First of all tP can be left at a large value unrelated to tH. Lets say 
>> tP = 640ms. tH stays 2.5s.
> 640ms is not 'large'. The requirement is either zero (disabled) or 
> region of 7.5s. The 640ms figure is the default for non-compute engines. 
> Anything that can run EUs needs to be 'huge'.
> 
> 
>>
>> 1) Background load = I915_PRIORITY_DISPLAY
>>
>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> 
>> Pulse3 <-- [tH + tP] --> full reset would be here
> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full five 
> periods is therefore 50s, which is practically a minute.

No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.

> [snip]
> 
>> Am I missing some requirement or you see another problem with this idea?
>>
>>>> On a related topic, if GuC engine resets stop working when preempt 
>>>> timeout is set to zero - I think we need to somehow let the user 
>>>> know if they try to tweak it via sysfs. Perhaps go as far as -EINVAL 
>>>> in GuC mode, if i915.reset has not explicitly disabled engine resets.
>>> Define 'stops working'. The definition of the sysfs interface is that 
>>> a value of zero disables pre-emption. If you don't have pre-emption 
>>> and your hang detection mechanism relies on pre-emption then you 
>>> don't have a hang detection mechanism either. If the user really 
>>> wants to allow 
>>
>> By stops working I meant that it stops working. :)
>>
>> With execlist one can disable preempt timeout and "stopped heartbeat" 
>> can still reset the stuck engine and so avoid collateral damage. With 
>> GuC it appears this is not possible. So I was thinking this is 
>> something worthy a log notice.
>>
>>> their context to run forever and never be pre-empted then that means 
>>> they also don't want it to be reset arbitrarily. Which means they 
>>> would also be disabling the heartbeat timer as well. Indeed, this is 
>>> what we 
>>
>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>> upstream but hearbeat is not and so hangcheck still works.
> The pre-emption disable in upstream is not a valid solution for compute 
> customers. It is a worst-of-all-worlds hack for general usage. As noted 
> already, any actual compute specific customer is advised to disable all 
> forms of reset and do their hang detection manually. A slightly less 
> worse hack for customers that are not actually running long compute 
> workloads (i.e. the vast majority of end users) is to just use a long 
> pre-emption timeout.

If disabled preemption timeout is worst of all words and compute needs to disable heartbeat as well then why did we put it in? Perhaps it was not know at the time it would not be good enough. But anyway, do I read you correct that current thinking is it would be better to leave it at default 640ms?

If so, if we went with my proposal, would everyone be happy? If yes, isn't it a simpler scheme? No special casing when setting the preempt timeout, no special casing of the last heartbeat pulse. Works predictably regardless of the priority of the hypothetical non-preemptible workload.

>>> advise compute customers to do. It is then up to the user themselves 
>>> to spot a hang and to manually kill (Ctrl+C, kill ###, etc.) their 
>>> task. Killing the CPU task will automatically clear up any GPU 
>>> resources allocated to that task (excepting context persistence, 
>>> which is a) broken and b) something we also tell compute customers to 
>>> disable).
>>
>> What is broken with context persistence? I noticed one patch claiming 
>> to be fixing something in that area which looked suspect. Has it been 
>> established no userspace relies on it?
> One major issue is that it has hooks into the execlist scheduler 
> backend. I forget the exact details right now. The implementation as a 
> whole is incredibly complex and convoluted :(. But there's stuff about 
> what happens when you disable the heartbeat after having closed a 
> persistence context's handle (and thus made it persisting). There's also 
> things like it sends a super high priority heartbeat pulse at the point 
> of becoming persisting. That plays havoc for platforms with dependent 
> engines and/or compute workloads. A context becomes persisting on RCS 
> and results in your unrealted CCS work being reset. It's a mess.
> 
> The comment from Daniel Vetter is that persistence should have no 
> connection to the heartbeat at all. All of that dynamic behaviour and 
> complexity should just be removed.

Dependent engines is definitely a topic on it's own, outside hearbeats, persistence and all.

Otherwise there is definitely complexity in the execlists backend but I am not sure if logic persistence and heartbeats are so very connected. It does send a pulse when heartbeat interval is changed, which if going to zero, it will kick of closed contexts if it can:

static struct intel_engine_cs *
__execlists_schedule_in(struct i915_request *rq)
{
	struct intel_engine_cs * const engine = rq->engine;
	struct intel_context * const ce = rq->context;

	intel_context_get(ce);

	if (unlikely(intel_context_is_closed(ce) &&
		     !intel_engine_has_heartbeat(engine)))
		intel_context_set_banned(ce);

	if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
		reset_active(rq, engine);

Is this what you mean? The point of this is to make sure persistent context does not hog the engine forever if hangcheck has been disabled.

Reminds me of my improvement to customer experience which never got in (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). Point of that one was to avoid engine reset (or worse) after user presses "Ctrl-C" if something takes just over 1ms to cleanly complete.

> Persistence itself can stay. There are valid UMD use cases. It is just 
> massively over complicated and doesn't work in all corner cases when not 
> using execlist submission or on newer platforms. The simplification that 
> is planned is to allow contexts to persist until the associated DRM 
> master handle is closed. At that point, all contexts associated with 
> that DRM handle are killed. That is what AMD and others apparently 
> implement.

Okay, that goes against one recent IGT patch which appeared to work around something by moving the position of _context_ close.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/24/2022 01:59, Tvrtko Ursulin wrote:
> On 23/02/2022 19:03, John Harrison wrote:
>> On 2/23/2022 04:13, Tvrtko Ursulin wrote:
>>> On 23/02/2022 02:11, John Harrison wrote:
>>>> On 2/22/2022 01:52, Tvrtko Ursulin wrote:
>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>
>>>>>> GuC converts the pre-emption timeout and timeslice quantum values 
>>>>>> into
>>>>>> clock ticks internally. That significantly reduces the point of 
>>>>>> 32bit
>>>>>> overflow. On current platforms, worst case scenario is approximately
>>>>>
>>>>> Where does 32-bit come from, the GuC side? We already use 64-bits 
>>>>> so that something to fix to start with. Yep...
>>>> Yes, the GuC API is defined as 32bits only and then does a straight 
>>>> multiply by the clock speed with no range checking. We have 
>>>> requested 64bit support but there was push back on the grounds that 
>>>> it is not something the GuC timer hardware supports and such long 
>>>> timeouts are not real world usable anyway.
>>>
>>> As long as compute are happy with 100 seconds, then it "should be 
>>> enough for everbody". :D
>> Compute disable all forms of reset and rely on manual kill. So yes.
>>
>> But even if they aren't. That's all we can do at the moment. If there 
>> is a genuine customer requirement for more then we can push for full 
>> 64bit software implemented timers in the GuC but until that happens, 
>> we don't have much choice.
>
> Yeah.
>
>>
>>>
>>>>>
>>>>> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>>>>>
>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>
>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>> timeslice_duration_ms;
>>>>>
>>>>> timeslice_store/preempt_timeout_store:
>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>
>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>> because of tick conversion internally but because at backend level 
>>>>> nothing was done for assigning 64-bit into 32-bit. Or I failed to 
>>>>> find where it is handled.
>>>> That's why I'm adding this range check to make sure we don't allow 
>>>> overflows.
>>>
>>> Yes and no, this fixes it, but the first bug was not only due GuC 
>>> internal tick conversion. It was present ever since the u64 from 
>>> i915 was shoved into u32 sent to GuC. So even if GuC used the value 
>>> without additional multiplication, bug was be there. My point being 
>>> when GuC backend was added timeout_ms values should have been 
>>> limited/clamped to U32_MAX. The tick discovery is additional limit 
>>> on top.
>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>> noticed until I actually tried using very long timeouts to debug a 
>> particular problem. Now that it is noticed, we need some method of 
>> range checking and this simple clamp solves all the truncation problems.
>
> Agreed in principle, just please mention in the commit message all 
> aspects of the problem.
>
> I think we can get away without a Fixes: tag since it requires user 
> fiddling to break things in unexpected ways.
>
> I would though put in a code a clamping which expresses both, 
> something like min(u32, ..GUC LIMIT..). So the full story is 
> documented forever. Or "if > u32 || > ..GUC LIMIT..) return -EINVAL". 
> Just in case GuC limit one day changes but u32 stays. Perhaps internal 
> ticks go away or anything and we are left with plain 1:1 millisecond 
> relationship.
Can certainly add a comment along the lines of "GuC API only takes a 
32bit field but that is further reduced to GUC_LIMIT due to internal 
calculations which would otherwise overflow".

But if the GuC limit is > u32 then, by definition, that means the GuC 
API has changed to take a u64 instead of a u32. So there will no u32 
truncation any more. So I'm not seeing a need to explicitly test the 
integer size when the value check covers that.

>
>>>>>> 110 seconds. Rather than allowing the user to set higher values and
>>>>>> then get confused by early timeouts, add limits when setting these
>>>>>> values.
>>>>>
>>>>> Btw who is reviewing GuC patches these days - things have somehow 
>>>>> gotten pretty quiet in activity and I don't think that's due 
>>>>> absence of stuff to improve or fix? Asking since I think I noticed 
>>>>> a few already which you posted and then crickets on the mailing list.
>>>> Too much work to do and not enough engineers to do it all :(.
>>>>
>>>>
>>>>>
>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>> ---
>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>>>   3 files changed, 38 insertions(+)
>>>>>>
>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>
>>>>> Hm "wanted".. There's been too much back and forth on the GuC load 
>>>>> options over the years to keep track.. intel_engine_uses_guc work 
>>>>> sounds like would work and read nicer.
>>>> I'm not adding a new feature check here. I'm just using the 
>>>> existing one. If we want to rename it yet again then that would be 
>>>> a different patch set.
>>>
>>> $ grep intel_engine_uses_guc . -rl
>>> ./i915_perf.c
>>> ./i915_request.c
>>> ./selftests/intel_scheduler_helpers.c
>>> ./gem/i915_gem_context.c
>>> ./gt/intel_context.c
>>> ./gt/intel_engine.h
>>> ./gt/intel_engine_cs.c
>>> ./gt/intel_engine_heartbeat.c
>>> ./gt/intel_engine_pm.c
>>> ./gt/intel_reset.c
>>> ./gt/intel_lrc.c
>>> ./gt/selftest_context.c
>>> ./gt/selftest_engine_pm.c
>>> ./gt/selftest_hangcheck.c
>>> ./gt/selftest_mocs.c
>>> ./gt/selftest_workarounds.c
>>>
>>> Sounds better to me than intel_guc_submission_is_wanted. What does 
>>> the reader know whether "is wanted" translates to "is actually 
>>> used". Shrug on "is wanted".
>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it too 
>> early in the boot up sequence? I never understood why that was 
>> necessary or why we need so many different ways to ask the same 
>> question. But this version already exists and definitely works 
>> without hitting any explosions.
>
> No idea if it causes a bug on, doesn't in the helper itself so maybe 
> you are saying it is called too early? Might be.. I think over time 
> the nice idea we had that "setup" and "init" phases of engine setup 
> clearly separated got destroyed a bit. There would always be an option 
> to move this clamping in a later phase, once the submission method is 
> known. One could argue that if the submission method is not yet known 
> at this point, it is even wrong to clamp based on something which will 
> only be decided later. Because:
>
> int intel_engines_init(struct intel_gt *gt)
> {
>     int (*setup)(struct intel_engine_cs *engine);
>     struct intel_engine_cs *engine;
>     enum intel_engine_id id;
>     int err;
>
>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>         gt->submission_method = INTEL_SUBMISSION_GUC;
>
> So this uses "uses", not "wanted". Presumably the point for having 
> "wanted" and "uses" is that they can disagree, in which case if you 
> clamp early based on "wanted" that suggests it could be wrong.

Okay, looks like I was getting confused with intel_guc_is_used(). That 
one blows up if called too early.

I'll change it to _uses_ and repost, then.

>
>>>>> And limit to class instead of applying to all engines looks like a 
>>>>> miss.
>>>> As per follow up email, the class limit is not applied here.
>>>>
>>>>>
>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>> +            engine->props.timeslice_duration_ms = 
>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>
>>>>> I am not sure logging such message during driver load is useful. 
>>>>> Sounds more like a confused driver which starts with one value and 
>>>>> then overrides itself. I'd just silently set the value appropriate 
>>>>> for the active backend. Preemption timeout kconfig text already 
>>>>> documents the fact timeouts can get overriden at runtime depending 
>>>>> on platform+engine. So maybe just add same text to timeslice kconfig.
>>>> The point is to make people aware if they compile with unsupported 
>>>> config options. As far as I know, there is no way to apply range 
>>>> checking or other limits to config defines. Which means that a user 
>>>> would silently get unwanted behaviour. That seems like a bad thing 
>>>> to me. If the driver is confused because the user built it in a 
>>>> confused manner then we should let them know.
>>>
>>> Okay, but I think make it notice low level.
>>>
>>> Also consider in patch 3/3 when you triple it, and then clamp back 
>>> down here. That's even more confused state since tripling gets 
>>> nerfed. I think that's also an argument to always account preempt 
>>> timeout in heartbeat interval calculation. Haven't got to your reply 
>>> on 2/3 yet though..
>> That sounds like even more reason to make sure the warning gets seen. 
>> The more complex the system and the more chances there are to get it 
>> wrong, the more important it is to have a nice easy to see and 
>> understand notification that it did go wrong.
>
> I did not disagree, just said make it notice, one level higher than 
> info! :)
But then it won't appear unless you have explicitly said an elevated 
debug level. Whereas info appears in dmesg by default (but is still not 
classed as an error by CI and such).

>
> But also think how, if we agree to go with tripling, that you'd have 
> to consider that in the sysfs store when hearbeat timeout is written, 
> to consider whether or not to triple and error out if preemption 
> timeout is over limit.
I see this as just setting the default values. If an end user is 
explicitly overriding the defaults then we should obey what they have 
requested. If they are changing the heartbeat interval then they can 
also change the pre-emption timeout appropriately.

John.


>
>>>>>> +        }
>>>>>> +
>>>>>> +        if (engine->props.preempt_timeout_ms > 
>>>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>>>> +            drm_info(&engine->i915->drm, "Warning, clamping 
>>>>>> pre-emption timeout to %d to prevent possibly overflow\n",
>>>>>> +                 GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS);
>>>>>> +            engine->props.preempt_timeout_ms = 
>>>>>> GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>>>> +        }
>>>>>> +    }
>>>>>> +
>>>>>>       engine->defaults = engine->props; /* never to change again */
>>>>>>         engine->context_size = intel_engine_context_size(gt, 
>>>>>> engine->class);
>>>>>> diff --git a/drivers/gpu/drm/i915/gt/sysfs_engines.c 
>>>>>> b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>>> index 967031056202..f57efe026474 100644
>>>>>> --- a/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>>> +++ b/drivers/gpu/drm/i915/gt/sysfs_engines.c
>>>>>> @@ -221,6 +221,13 @@ timeslice_store(struct kobject *kobj, struct 
>>>>>> kobj_attribute *attr,
>>>>>>       if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>>>           return -EINVAL;
>>>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>>>> +        duration > GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>> +        duration = GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>> +        drm_info(&engine->i915->drm, "Warning, clamping 
>>>>>> timeslice duration to %lld to prevent possibly overflow\n",
>>>>>> +             duration);
>>>>>> +    }
>>>>>
>>>>> I would suggest to avoid duplicated clamping logic. Maybe hide the 
>>>>> all backend logic into the helpers then, like maybe:
>>>>>
>>>>>   d = intel_engine_validate_timeslice/preempt_timeout(engine, 
>>>>> duration);
>>>>>   if (d != duration)
>>>>>     return -EINVAL:
>>>>>
>>>>> Returning -EINVAL would be equivalent to existing behaviour:
>>>>>
>>>>>     if (duration > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>>         return -EINVAL;
>>>>>
>>>>> That way userspace has explicit notification and read-back is 
>>>>> identical to written in value. From engine setup you can just call 
>>>>> the helper silently.
>>>> Sure, EINVAL rather than clamping works as well. And can certainly 
>>>> add helper wrappers. But as above, I don't like the idea of 
>>>> silently disregarding a user specified config option.
>>>
>>> Deal - with the open of heartbeat interval TBD.
>>>
>>>>
>>>>>
>>>>>> +
>>>>>>       WRITE_ONCE(engine->props.timeslice_duration_ms, duration);
>>>>>>         if (execlists_active(&engine->execlists))
>>>>>> @@ -325,6 +332,13 @@ preempt_timeout_store(struct kobject *kobj, 
>>>>>> struct kobj_attribute *attr,
>>>>>>       if (timeout > jiffies_to_msecs(MAX_SCHEDULE_TIMEOUT))
>>>>>>           return -EINVAL;
>>>>>>   +    if (intel_uc_uses_guc_submission(&engine->gt->uc) &&
>>>>>> +        timeout > GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS) {
>>>>>> +        timeout = GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS;
>>>>>> +        drm_info(&engine->i915->drm, "Warning, clamping 
>>>>>> pre-emption timeout to %lld to prevent possibly overflow\n",
>>>>>> +             timeout);
>>>>>> +    }
>>>>>> +
>>>>>>       WRITE_ONCE(engine->props.preempt_timeout_ms, timeout);
>>>>>>         if (READ_ONCE(engine->execlists.pending[0]))
>>>>>> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h 
>>>>>> b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>>> index 6a4612a852e2..ad131092f8df 100644
>>>>>> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>>> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h
>>>>>> @@ -248,6 +248,15 @@ struct guc_lrc_desc {
>>>>>>     #define GLOBAL_POLICY_DEFAULT_DPC_PROMOTE_TIME_US 500000
>>>>>>   +/*
>>>>>> + * GuC converts the timeout to clock ticks internally. Different 
>>>>>> platforms have
>>>>>> + * different GuC clocks. Thus, the maximum value before overflow 
>>>>>> is platform
>>>>>> + * dependent. Current worst case scenario is about 110s. So, 
>>>>>> limit to 100s to be
>>>>>> + * safe.
>>>>>> + */
>>>>>> +#define GUC_POLICY_MAX_EXEC_QUANTUM_MS        (100 * 1000)
>>>>>> +#define GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS    (100 * 1000)
>>>>>
>>>>> Most important question -
>>>>> how will we know/notice if/when new GuC arrives where these 
>>>>> timeouts would still overflow? Can this be queried somehow at 
>>>>> runtime or where does the limit comes from? How is GuC told about 
>>>>> it? Set in some field and it just allows too large values silently 
>>>>> break things?
>>>> Currently, we don't notice except by debugging peculiar test 
>>>> failures :(.
>>>>
>>>> These limits are not in any GuC spec. Indeed, it took a while to 
>>>> actually work out why increasing the value actually caused shorter 
>>>> timeouts to occur! As above, there is no range checking inside GuC 
>>>> itself. It does a truncated multiply which results in an 
>>>> effectively random number and just happily uses it.
>>>
>>> I will agree with what Daniele said - push on GuC fw folks to 
>>> document the max values they guarantee to support in the interface 
>>> spec. Otherwise it is too fragile.
>> I do agree. But that is going to take time. I would like to get 
>> something merged now while we fight over spec updates.
>
> Yeah that's okay, did not mean to imply I am against a quick fix. 
> "Otherwise it is too fragile, *in the long run*" should have written 
> or something like that.
>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 2/24/2022 03:41, Tvrtko Ursulin wrote:
> On 23/02/2022 20:00, John Harrison wrote:
>> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>>> On 23/02/2022 02:45, John Harrison wrote:
>>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>
>>>>>> Compute workloads are inherantly not pre-emptible for long 
>>>>>> periods on
>>>>>> current hardware. As a workaround for this, the pre-emption timeout
>>>>>> for compute capable engines was disabled. This is undesirable 
>>>>>> with GuC
>>>>>> submission as it prevents per engine reset of hung contexts. 
>>>>>> Hence the
>>>>>> next patch will re-enable the timeout but bumped up by an order of
>>>>>> magnititude.
>>>>>
>>>>> (Some typos above.)
>>>> I'm spotting 'inherently' but not anything else.
>>>
>>> Magnititude! O;)
>> Doh!
>>
>> [snip]
>>
>>>> Whereas, bumping all heartbeat periods to be greater than the 
>>>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>>>> total heartbeat time of about a minute. Which is a very long time 
>>>> to wait for a hang to be detected and recovered. Especially when 
>>>> the official limit on a context responding to an 'are you dead' 
>>>> query is only 7.5 seconds.
>>>
>>> Not sure how did you get one minute?
>> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 75 
>> => 1 minute 15 seconds
>>
>> Even ignoring any safety factor and just going with 7.5 * 5 still 
>> gets you to 37.5 seconds which is over a half a minute and likely to 
>> race.
>
> Ah because my starting point is there should be no preempt timeout = 
> heartbeat * 3, I just think that's too ugly.
Then complain at the hardware designers to give us mid-thread 
pre-emption back. The heartbeat is only one source of pre-emption 
events. For example, a user can be running multiple contexts in parallel 
and expecting them to time slice on a single engine. Or maybe a user is 
just running one compute task in the background but is doing render work 
in the foreground. Etc.

There was a reason the original hack was to disable pre-emption rather 
than increase the heartbeat. This is simply a slightly less ugly version 
of the same hack. And unfortunately, the basic idea of the hack is 
non-negotiable.

As per other comments, 'tP(RCS) = tH *3' or 'tP(RCS) = tP(default) * 12' 
or 'tP(RCS) = 7500' are the available options. Given that the heartbeat 
is the ever present hard limit, it seems most plausible to base the hack 
on that. Any of the others works, though. Although I think a explicit 
hardcoded value is the most ugly. I guess the other option is to add 
CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE and default that to 7500.

Take your pick. But 640ms is not allowed.

>
>>> Regardless, crux of argument was to avoid GuC engine reset and 
>>> heartbeat reset racing with each other, and to do that by 
>>> considering the preempt timeout with the heartbeat interval. I was 
>>> thinking about this scenario in this series:
>>>
>>> [Please use fixed width font and no line wrap to view.]
>>>
>>> A)
>>>
>>> tP = preempt timeout
>>> tH = hearbeat interval
>>>
>>> tP = 3 * tH
>>>
>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>
>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 
>>> * tH] ----> FULL RESET
>>>                                                        |
>>>                                                        \- preemption 
>>> triggered, tP = 3 * tH ------\
>>> \-> preempt timeout would hit here
>>>
>>> Here we have collateral damage due full reset, since we can't tell 
>>> GuC to reset just one engine and we fudged tP just to "account" for 
>>> heartbeats.
>> You are missing the whole point of the patch series which is that the 
>> last heartbeat period is '2 * tP' not '2 * tH'.
>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>>
>> By making the last period double the pre-emption timeout, it is 
>> guaranteed that the FULL RESET stage cannot be hit before the 
>> hardware has attempted and timed-out on at least one pre-emption.
>
> Oh well :) that probably means the overall scheme is too odd for me. 
> tp = 3tH and last pulse after 2tP I mean.
To be accurate, it is 'tP(RCS) = 3 * tH(default); tH(final) = 
tP(current) * 2;'. Seems fairly straight forward to me. It's not a 
recursive definition or anything like that. It gives us a total 
heartbeat timeout that is close to the original version but still allows 
at least one pre-emption event.

>
>> [snip]
>>
>>
>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 
>>> * tH] ----> full reset would be here
>>>                |
>>>                \- preemption triggered, tP = 3 * tH ----------------\
>>> \-> Preempt timeout reset
>>>
>>> Here is is kind of least worse, but question is why we fudged tP 
>>> when it gives us nothing good in this case.
>>>
>> The point of fudging tP(RCS) is to give compute workloads longer to 
>> reach a pre-emptible point (given that EU walkers are basically not 
>> pre-emptible). The reason for doing the fudge is not connected to the 
>> heartbeat at all. The fact that it causes problems for the heartbeat 
>> is an undesired side effect.
>>
>> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
>> calculation that gave us something that all interested parties were 
>> vaguely happy with. It could just as easily be a fixed, hard coded 
>> value of 7.5s but having it based on something configurable seemed 
>> more sensible. The other option was 'tP(RCS) = tP * 12' but that felt 
>> more arbitrary than basing it on the average heartbeat timeout. As 
>> in, three heartbeat periods is about what a normal prio task gets 
>> before it gets pre-empted by the heartbeat. So using that for general 
>> purpose pre-emptions (e.g. time slicing between multiple user apps) 
>> seems reasonable.
>
> I think the fact you say tP fudge is not related to heartbeats and 
> then go to mention heartbeat even in the "formula" which uses no tH is 
> saying something (at least that's how I read the 7.5s option). :)
I said the tP fudge is not because of the heartbeat. It is obviously 
related.

As per comment above, the fudge factor is based on the heartbeat because 
the heartbeat is the ultimate limit. But the *reason* for the fudge fact 
has nothing to do with the heartbeat. It is required even if heartbeats 
are disabled.

>
>>> B)
>>>
>>> Instead, my idea to account for preempt timeout when calculating 
>>> when to schedule next hearbeat would look like this:
>>>
>>> First of all tP can be left at a large value unrelated to tH. Lets 
>>> say tP = 640ms. tH stays 2.5s.
>> 640ms is not 'large'. The requirement is either zero (disabled) or 
>> region of 7.5s. The 640ms figure is the default for non-compute 
>> engines. Anything that can run EUs needs to be 'huge'.
>>
>>
>>>
>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>
>>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> 
>>> Pulse3 <-- [tH + tP] --> full reset would be here
>> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full 
>> five periods is therefore 50s, which is practically a minute.
>
> No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.
Not good enough. After 2.5s, we send a pulse. After a further 640ms we 
perform an engine reset. That means your compute workload had only 640ms 
after being told to pre-empt to reach a pre-emption point. That won't 
work. It needs to be multiple seconds.

>
>> [snip]
>>
>>> Am I missing some requirement or you see another problem with this 
>>> idea?
>>>
>>>>> On a related topic, if GuC engine resets stop working when preempt 
>>>>> timeout is set to zero - I think we need to somehow let the user 
>>>>> know if they try to tweak it via sysfs. Perhaps go as far as 
>>>>> -EINVAL in GuC mode, if i915.reset has not explicitly disabled 
>>>>> engine resets.
>>>> Define 'stops working'. The definition of the sysfs interface is 
>>>> that a value of zero disables pre-emption. If you don't have 
>>>> pre-emption and your hang detection mechanism relies on pre-emption 
>>>> then you don't have a hang detection mechanism either. If the user 
>>>> really wants to allow 
>>>
>>> By stops working I meant that it stops working. :)
>>>
>>> With execlist one can disable preempt timeout and "stopped 
>>> heartbeat" can still reset the stuck engine and so avoid collateral 
>>> damage. With GuC it appears this is not possible. So I was thinking 
>>> this is something worthy a log notice.
>>>
>>>> their context to run forever and never be pre-empted then that 
>>>> means they also don't want it to be reset arbitrarily. Which means 
>>>> they would also be disabling the heartbeat timer as well. Indeed, 
>>>> this is what we 
>>>
>>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>>> upstream but hearbeat is not and so hangcheck still works.
>> The pre-emption disable in upstream is not a valid solution for 
>> compute customers. It is a worst-of-all-worlds hack for general 
>> usage. As noted already, any actual compute specific customer is 
>> advised to disable all forms of reset and do their hang detection 
>> manually. A slightly less worse hack for customers that are not 
>> actually running long compute workloads (i.e. the vast majority of 
>> end users) is to just use a long pre-emption timeout.
>
> If disabled preemption timeout is worst of all words and compute needs 
> to disable heartbeat as well then why did we put it in? Perhaps it was 
> not know at the time it would not be good enough. But anyway, do I 
> read you correct that current thinking is it would be better to leave 
> it at default 640ms?
No. We cannot have the RCS default to 640ms.

Note that there is a difference between 'general end user who might run 
some compute' and 'compute focused customer'. The driver defaults 
(disabled or 7500ms) are for the general user who gets the 
out-of-the-box experience and expects to be able to run 'normal' 
workloads without hitting problems. I.e. they expect hung tasks to get 
reset in a timely manner and while they might run some AI or other 
compute workloads, they are not a HPC datacenter. Whereas the compute 
datacenter customer expects their workloads to run for arbitrarily long 
times (minutes, hours, maybe even days) without being arbitrarily 
killed. Those customers will be explicitly configuring their datacenter 
server for that scenario and thus don't care what the defaults are.


>
> If so, if we went with my proposal, would everyone be happy? If yes, 
> isn't it a simpler scheme? No special casing when setting the preempt 
> timeout, no special casing of the last heartbeat pulse. Works 
> predictably regardless of the priority of the hypothetical 
> non-preemptible workload.
No, we have to have the increased pre-emption timeout. And that has 
ripple effects of making very long heartbeats or risking races with the 
heartbeat beating the per engine reset.


>
>>>> advise compute customers to do. It is then up to the user 
>>>> themselves to spot a hang and to manually kill (Ctrl+C, kill ###, 
>>>> etc.) their task. Killing the CPU task will automatically clear up 
>>>> any GPU resources allocated to that task (excepting context 
>>>> persistence, which is a) broken and b) something we also tell 
>>>> compute customers to disable).
>>>
>>> What is broken with context persistence? I noticed one patch 
>>> claiming to be fixing something in that area which looked suspect. 
>>> Has it been established no userspace relies on it?
>> One major issue is that it has hooks into the execlist scheduler 
>> backend. I forget the exact details right now. The implementation as 
>> a whole is incredibly complex and convoluted :(. But there's stuff 
>> about what happens when you disable the heartbeat after having closed 
>> a persistence context's handle (and thus made it persisting). There's 
>> also things like it sends a super high priority heartbeat pulse at 
>> the point of becoming persisting. That plays havoc for platforms with 
>> dependent engines and/or compute workloads. A context becomes 
>> persisting on RCS and results in your unrealted CCS work being reset. 
>> It's a mess.
>>
>> The comment from Daniel Vetter is that persistence should have no 
>> connection to the heartbeat at all. All of that dynamic behaviour and 
>> complexity should just be removed.
>
> Dependent engines is definitely a topic on it's own, outside 
> hearbeats, persistence and all.
Except that it has implications for persistence which the current driver 
does not take into account.

>
> Otherwise there is definitely complexity in the execlists backend but 
> I am not sure if logic persistence and heartbeats are so very 
> connected. It does send a pulse when heartbeat interval is changed, 
> which if going to zero, it will kick of closed contexts if it can:
>
> static struct intel_engine_cs *
> __execlists_schedule_in(struct i915_request *rq)
> {
>     struct intel_engine_cs * const engine = rq->engine;
>     struct intel_context * const ce = rq->context;
>
>     intel_context_get(ce);
>
>     if (unlikely(intel_context_is_closed(ce) &&
>              !intel_engine_has_heartbeat(engine)))
>         intel_context_set_banned(ce);
>
>     if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
>         reset_active(rq, engine);
>
> Is this what you mean? The point of this is to make sure persistent 
> context does not hog the engine forever if hangcheck has been disabled.
>
> Reminds me of my improvement to customer experience which never got in 
> (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). 
> Point of that one was to avoid engine reset (or worse) after user 
> presses "Ctrl-C" if something takes just over 1ms to cleanly complete.
The plan is that the persistent contexts would still get the default 
grace period (pre-emption timeout at least) to finish but Ctrl+C will 
kill it within a timely manner if it does not finish.

>
>> Persistence itself can stay. There are valid UMD use cases. It is 
>> just massively over complicated and doesn't work in all corner cases 
>> when not using execlist submission or on newer platforms. The 
>> simplification that is planned is to allow contexts to persist until 
>> the associated DRM master handle is closed. At that point, all 
>> contexts associated with that DRM handle are killed. That is what AMD 
>> and others apparently implement.
>
> Okay, that goes against one recent IGT patch which appeared to work 
> around something by moving the position of _context_ close.
No it does not. The context close is not the trigger. The trigger is 
closing the top level DRM handle. If your context has persistence 
enabled (the default) then closing the context handle will have no 
effect. No pulse, no pre-emption, no kill, nothing. But when the top 
level handle is closed (application exit through whatever mechanism) 
then all GPU resources will be cleaned up. As above, with at least a 
pre-emption timeout grace period, but after that it is termination time.

The media use cases for persistence are all happy with this scheme. I 
don't actually recall if we got a reply back from the OGL people. They 
were definitely on the email thread/Jira task and did not complain. OCL 
obviously don't care as their first action is to explicitly disable 
persistence.

John.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/24/2022 11:19, John Harrison wrote:
> [snip]
>
> I'll change it to _uses_ and repost, then.
>
[    7.683149] kernel BUG at drivers/gpu/drm/i915/gt/uc/intel_guc.h:367!

Told you that one went bang.

John.

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/23/2022 04:00, Tvrtko Ursulin wrote:
> On 23/02/2022 02:22, John Harrison wrote:
>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>
>>>> Compute workloads are inherently not pre-emptible on current hardware.
>>>> Thus the pre-emption timeout was disabled as a workaround to prevent
>>>> unwanted resets. Instead, the hang detection was left to the heartbeat
>>>> and its (longer) timeout. This is undesirable with GuC submission as
>>>> the heartbeat is a full GT reset rather than a per engine reset and so
>>>> is much more destructive. Instead, just bump the pre-emption timeout
>>>
>>> Can we have a feature request to allow asking GuC for an engine reset?
>> For what purpose?
>
> To allow "stopped heartbeat" to reset the engine, however..
>
>> GuC manages the scheduling of contexts across engines. With virtual 
>> engines, the KMD has no knowledge of which engine a context might be 
>> executing on. Even without virtual engines, the KMD still has no 
>> knowledge of which context is currently executing on any given engine 
>> at any given time.
>>
>> There is a reason why hang detection should be left to the entity 
>> that is doing the scheduling. Any other entity is second guessing at 
>> best.
>>
>> The reason for keeping the heartbeat around even when GuC submission 
>> is enabled is for the case where the KMD/GuC have got out of sync 
>> with either other somehow or GuC itself has just crashed. I.e. when 
>> no submission at all is working and we need to reset the GuC itself 
>> and start over.
>
> .. I wasn't really up to speed to know/remember heartbeats are nerfed 
> already in GuC mode.
Not sure what you mean by that claim. Engine resets are handled by GuC 
because GuC handles the scheduling. You can't do the former if you 
aren't doing the latter. However, the heartbeat is still present and is 
still the watchdog by which engine resets are triggered. As per the rest 
of the submission process, the hang detection and recovery is split 
between i915 and GuC.


>
> I am not sure it was the best way since full reset penalizes everyone 
> for one hanging engine. Better question would be why leave heartbeats 
> around to start with with GuC? If you want to use it to health check 
> GuC, as you say, maybe just send a CT message and expect replies? Then 
> full reset would make sense. It also achieves the goal of not 
> seconding guessing the submission backend you raise.
Adding yet another hang detection mechanism is yet more complication and 
is unnecessary when we already have one that works well enough. As 
above, the heartbeat is still required for sending the pulses that cause 
pre-emptions and so let GuC detect hangs. It also provides a fallback 
against a dead GuC by default. So why invent yet another wheel?

>
> Like it is now, and the need for this series demonstrates it, the 
> whole thing has a pretty poor "impedance" match. Not even sure what 
> intel_guc_find_hung_context is doing in intel_engine_hearbeat.c - why 
> is that not in intel_gt_handle_error at least? Why is hearbeat code 
> special and other callers of intel_gt_handle_error don't need it?
There is no guilty context if the reset was triggered via debugfs or 
similar. And as stated ad nauseam, i915 is no longer handling the 
scheduling and so cannot make assumptions about what is or is not 
running on the hardware at any given time. And obviously, if the reset 
initiated by GuC itself then i915 should not be second guessing the 
guilty context as the GuC notification has already told us who was 
responsible.

And to be clear, the 'poor impedance match' is purely because we don't 
have mid-thread pre-emption and so need a stupidly huge timeout on 
compute capable engines. Whereas, we don't want a total heatbeat timeout 
of a minute or more. That is the impedance mis-match. If the 640ms was 
acceptable for RCS then none of this hacky timeout algorithm mush would 
be needed.

John.


>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 24/02/2022 20:02, John Harrison wrote:
> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>> On 23/02/2022 02:22, John Harrison wrote:
>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>
>>>>> Compute workloads are inherently not pre-emptible on current hardware.
>>>>> Thus the pre-emption timeout was disabled as a workaround to prevent
>>>>> unwanted resets. Instead, the hang detection was left to the heartbeat
>>>>> and its (longer) timeout. This is undesirable with GuC submission as
>>>>> the heartbeat is a full GT reset rather than a per engine reset and so
>>>>> is much more destructive. Instead, just bump the pre-emption timeout
>>>>
>>>> Can we have a feature request to allow asking GuC for an engine reset?
>>> For what purpose?
>>
>> To allow "stopped heartbeat" to reset the engine, however..
>>
>>> GuC manages the scheduling of contexts across engines. With virtual 
>>> engines, the KMD has no knowledge of which engine a context might be 
>>> executing on. Even without virtual engines, the KMD still has no 
>>> knowledge of which context is currently executing on any given engine 
>>> at any given time.
>>>
>>> There is a reason why hang detection should be left to the entity 
>>> that is doing the scheduling. Any other entity is second guessing at 
>>> best.
>>>
>>> The reason for keeping the heartbeat around even when GuC submission 
>>> is enabled is for the case where the KMD/GuC have got out of sync 
>>> with either other somehow or GuC itself has just crashed. I.e. when 
>>> no submission at all is working and we need to reset the GuC itself 
>>> and start over.
>>
>> .. I wasn't really up to speed to know/remember heartbeats are nerfed 
>> already in GuC mode.
> Not sure what you mean by that claim. Engine resets are handled by GuC 
> because GuC handles the scheduling. You can't do the former if you 
> aren't doing the latter. However, the heartbeat is still present and is 
> still the watchdog by which engine resets are triggered. As per the rest 
> of the submission process, the hang detection and recovery is split 
> between i915 and GuC.

I meant that "stopped heartbeat on engine XXX" can only do a full GPU reset on GuC.

	intel_gt_handle_error(engine->gt, engine->mask,
			      I915_ERROR_CAPTURE,
			      "stopped heartbeat on %s",
			      engine->name);

intel_gt_handle_error:

	/*
	 * Try engine reset when available. We fall back to full reset if
	 * single reset fails.
	 */
	if (!intel_uc_uses_guc_submission(&gt->uc) &&
	    intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
		local_bh_disable();
		for_each_engine_masked(engine, gt, engine_mask, tmp) {

You said "However, the heartbeat is still present and is still the watchdog by which engine resets are triggered", now I don't know what you meant by this. It actually triggers a single engine reset in GuC mode? Where in code does that happen if this block above shows it not taking the engine reset path?

>> I am not sure it was the best way since full reset penalizes everyone 
>> for one hanging engine. Better question would be why leave heartbeats 
>> around to start with with GuC? If you want to use it to health check 
>> GuC, as you say, maybe just send a CT message and expect replies? Then 
>> full reset would make sense. It also achieves the goal of not 
>> seconding guessing the submission backend you raise.
> Adding yet another hang detection mechanism is yet more complication and 
> is unnecessary when we already have one that works well enough. As 
> above, the heartbeat is still required for sending the pulses that cause 
> pre-emptions and so let GuC detect hangs. It also provides a fallback 
> against a dead GuC by default. So why invent yet another wheel?

Lets first clarify the previous block to make sure there aren't any misunderstandings there.

Regards,

Tvrtko

>> Like it is now, and the need for this series demonstrates it, the 
>> whole thing has a pretty poor "impedance" match. Not even sure what 
>> intel_guc_find_hung_context is doing in intel_engine_hearbeat.c - why 
>> is that not in intel_gt_handle_error at least? Why is hearbeat code 
>> special and other callers of intel_gt_handle_error don't need it?
> There is no guilty context if the reset was triggered via debugfs or 
> similar. And as stated ad nauseam, i915 is no longer handling the 
> scheduling and so cannot make assumptions about what is or is not 
> running on the hardware at any given time. And obviously, if the reset 
> initiated by GuC itself then i915 should not be second guessing the 
> guilty context as the GuC notification has already told us who was 
> responsible.
> 
> And to be clear, the 'poor impedance match' is purely because we don't 
> have mid-thread pre-emption and so need a stupidly huge timeout on 
> compute capable engines. Whereas, we don't want a total heatbeat timeout 
> of a minute or more. That is the impedance mis-match. If the 640ms was 
> acceptable for RCS then none of this hacky timeout algorithm mush would 
> be needed.
> 
> John.
> 
> 
>>
>> Regards,
>>
>> Tvrtko
> 

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 24/02/2022 19:19, John Harrison wrote:

[snip]

>>>>>> ./gt/uc/intel_guc_fwif.h:       u32 execution_quantum;
>>>>>>
>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>
>>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>>> timeslice_duration_ms;
>>>>>>
>>>>>> timeslice_store/preempt_timeout_store:
>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>
>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>> because of tick conversion internally but because at backend level 
>>>>>> nothing was done for assigning 64-bit into 32-bit. Or I failed to 
>>>>>> find where it is handled.
>>>>> That's why I'm adding this range check to make sure we don't allow 
>>>>> overflows.
>>>>
>>>> Yes and no, this fixes it, but the first bug was not only due GuC 
>>>> internal tick conversion. It was present ever since the u64 from 
>>>> i915 was shoved into u32 sent to GuC. So even if GuC used the value 
>>>> without additional multiplication, bug was be there. My point being 
>>>> when GuC backend was added timeout_ms values should have been 
>>>> limited/clamped to U32_MAX. The tick discovery is additional limit 
>>>> on top.
>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>> noticed until I actually tried using very long timeouts to debug a 
>>> particular problem. Now that it is noticed, we need some method of 
>>> range checking and this simple clamp solves all the truncation problems.
>>
>> Agreed in principle, just please mention in the commit message all 
>> aspects of the problem.
>>
>> I think we can get away without a Fixes: tag since it requires user 
>> fiddling to break things in unexpected ways.
>>
>> I would though put in a code a clamping which expresses both, 
>> something like min(u32, ..GUC LIMIT..). So the full story is 
>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return -EINVAL". 
>> Just in case GuC limit one day changes but u32 stays. Perhaps internal 
>> ticks go away or anything and we are left with plain 1:1 millisecond 
>> relationship.
> Can certainly add a comment along the lines of "GuC API only takes a 
> 32bit field but that is further reduced to GUC_LIMIT due to internal 
> calculations which would otherwise overflow".
> 
> But if the GuC limit is > u32 then, by definition, that means the GuC 
> API has changed to take a u64 instead of a u32. So there will no u32 
> truncation any more. So I'm not seeing a need to explicitly test the 
> integer size when the value check covers that.

Hmm I was thinking if the internal conversion in the GuC fw changes so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, then to be extra safe by documenting in code there is the additional limit of the data structure field. Say the field was changed to take some unit larger than a millisecond. Then the check against the GuC MAX limit define would not be enough, unless that would account both for internal implementation and u32 in the protocol. Maybe that is overdefensive but I don't see that it harms. 50-50, but it's do it once and forget so I'd do it.

>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>> ---
>>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>
>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == RENDER_CLASS)
>>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>
>>>>>> Hm "wanted".. There's been too much back and forth on the GuC load 
>>>>>> options over the years to keep track.. intel_engine_uses_guc work 
>>>>>> sounds like would work and read nicer.
>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>> existing one. If we want to rename it yet again then that would be 
>>>>> a different patch set.
>>>>
>>>> $ grep intel_engine_uses_guc . -rl
>>>> ./i915_perf.c
>>>> ./i915_request.c
>>>> ./selftests/intel_scheduler_helpers.c
>>>> ./gem/i915_gem_context.c
>>>> ./gt/intel_context.c
>>>> ./gt/intel_engine.h
>>>> ./gt/intel_engine_cs.c
>>>> ./gt/intel_engine_heartbeat.c
>>>> ./gt/intel_engine_pm.c
>>>> ./gt/intel_reset.c
>>>> ./gt/intel_lrc.c
>>>> ./gt/selftest_context.c
>>>> ./gt/selftest_engine_pm.c
>>>> ./gt/selftest_hangcheck.c
>>>> ./gt/selftest_mocs.c
>>>> ./gt/selftest_workarounds.c
>>>>
>>>> Sounds better to me than intel_guc_submission_is_wanted. What does 
>>>> the reader know whether "is wanted" translates to "is actually 
>>>> used". Shrug on "is wanted".
>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it too 
>>> early in the boot up sequence? I never understood why that was 
>>> necessary or why we need so many different ways to ask the same 
>>> question. But this version already exists and definitely works 
>>> without hitting any explosions.
>>
>> No idea if it causes a bug on, doesn't in the helper itself so maybe 
>> you are saying it is called too early? Might be.. I think over time 
>> the nice idea we had that "setup" and "init" phases of engine setup 
>> clearly separated got destroyed a bit. There would always be an option 
>> to move this clamping in a later phase, once the submission method is 
>> known. One could argue that if the submission method is not yet known 
>> at this point, it is even wrong to clamp based on something which will 
>> only be decided later. Because:
>>
>> int intel_engines_init(struct intel_gt *gt)
>> {
>>     int (*setup)(struct intel_engine_cs *engine);
>>     struct intel_engine_cs *engine;
>>     enum intel_engine_id id;
>>     int err;
>>
>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>
>> So this uses "uses", not "wanted". Presumably the point for having 
>> "wanted" and "uses" is that they can disagree, in which case if you 
>> clamp early based on "wanted" that suggests it could be wrong.
> 
> Okay, looks like I was getting confused with intel_guc_is_used(). That 
> one blows up if called too early.
> 
> I'll change it to _uses_ and repost, then.

Check that it isn't called too early, before gt->submission_setup is set.

> 
>>
>>>>>> And limit to class instead of applying to all engines looks like a 
>>>>>> miss.
>>>>> As per follow up email, the class limit is not applied here.
>>>>>
>>>>>>
>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>> +            engine->props.timeslice_duration_ms = 
>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>
>>>>>> I am not sure logging such message during driver load is useful. 
>>>>>> Sounds more like a confused driver which starts with one value and 
>>>>>> then overrides itself. I'd just silently set the value appropriate 
>>>>>> for the active backend. Preemption timeout kconfig text already 
>>>>>> documents the fact timeouts can get overriden at runtime depending 
>>>>>> on platform+engine. So maybe just add same text to timeslice kconfig.
>>>>> The point is to make people aware if they compile with unsupported 
>>>>> config options. As far as I know, there is no way to apply range 
>>>>> checking or other limits to config defines. Which means that a user 
>>>>> would silently get unwanted behaviour. That seems like a bad thing 
>>>>> to me. If the driver is confused because the user built it in a 
>>>>> confused manner then we should let them know.
>>>>
>>>> Okay, but I think make it notice low level.
>>>>
>>>> Also consider in patch 3/3 when you triple it, and then clamp back 
>>>> down here. That's even more confused state since tripling gets 
>>>> nerfed. I think that's also an argument to always account preempt 
>>>> timeout in heartbeat interval calculation. Haven't got to your reply 
>>>> on 2/3 yet though..
>>> That sounds like even more reason to make sure the warning gets seen. 
>>> The more complex the system and the more chances there are to get it 
>>> wrong, the more important it is to have a nice easy to see and 
>>> understand notification that it did go wrong.
>>
>> I did not disagree, just said make it notice, one level higher than 
>> info! :)
> But then it won't appear unless you have explicitly said an elevated 
> debug level. Whereas info appears in dmesg by default (but is still not 
> classed as an error by CI and such).

Notice is higher than info! :) If info appears by default so does notice, warning, err, etc...

#define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
#define KERN_ALERT      KERN_SOH "1"    /* action must be taken immediately */
#define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
#define KERN_ERR        KERN_SOH "3"    /* error conditions */
#define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
#define KERN_NOTICE     KERN_SOH "5"    /* normal but significant condition */
#define KERN_INFO       KERN_SOH "6"    /* informational */
#define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */

>> But also think how, if we agree to go with tripling, that you'd have 
>> to consider that in the sysfs store when hearbeat timeout is written, 
>> to consider whether or not to triple and error out if preemption 
>> timeout is over limit.
> I see this as just setting the default values. If an end user is 
> explicitly overriding the defaults then we should obey what they have 
> requested. If they are changing the heartbeat interval then they can 
> also change the pre-emption timeout appropriately.

Question is can they unknowingly and without any feedback configure a much worse state than they expect? Like when they set heartbeats up to some value, everything is configured as you intended - but if you go over a certain hidden limit the overall scheme degrades in some way. What is the failure mode here if you silently let them do that?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/25/2022 08:36, Tvrtko Ursulin wrote:
> On 24/02/2022 20:02, John Harrison wrote:
>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>> On 23/02/2022 02:22, John Harrison wrote:
>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>
>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>> hardware.
>>>>>> Thus the pre-emption timeout was disabled as a workaround to prevent
>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>> heartbeat
>>>>>> and its (longer) timeout. This is undesirable with GuC submission as
>>>>>> the heartbeat is a full GT reset rather than a per engine reset 
>>>>>> and so
>>>>>> is much more destructive. Instead, just bump the pre-emption timeout
>>>>>
>>>>> Can we have a feature request to allow asking GuC for an engine 
>>>>> reset?
>>>> For what purpose?
>>>
>>> To allow "stopped heartbeat" to reset the engine, however..
>>>
>>>> GuC manages the scheduling of contexts across engines. With virtual 
>>>> engines, the KMD has no knowledge of which engine a context might 
>>>> be executing on. Even without virtual engines, the KMD still has no 
>>>> knowledge of which context is currently executing on any given 
>>>> engine at any given time.
>>>>
>>>> There is a reason why hang detection should be left to the entity 
>>>> that is doing the scheduling. Any other entity is second guessing 
>>>> at best.
>>>>
>>>> The reason for keeping the heartbeat around even when GuC 
>>>> submission is enabled is for the case where the KMD/GuC have got 
>>>> out of sync with either other somehow or GuC itself has just 
>>>> crashed. I.e. when no submission at all is working and we need to 
>>>> reset the GuC itself and start over.
>>>
>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>> nerfed already in GuC mode.
>> Not sure what you mean by that claim. Engine resets are handled by 
>> GuC because GuC handles the scheduling. You can't do the former if 
>> you aren't doing the latter. However, the heartbeat is still present 
>> and is still the watchdog by which engine resets are triggered. As 
>> per the rest of the submission process, the hang detection and 
>> recovery is split between i915 and GuC.
>
> I meant that "stopped heartbeat on engine XXX" can only do a full GPU 
> reset on GuC.
I mean that there is no 'stopped heartbeat on engine XXX' when i915 is 
not handling the recovery part of the process.


>
>     intel_gt_handle_error(engine->gt, engine->mask,
>                   I915_ERROR_CAPTURE,
>                   "stopped heartbeat on %s",
>                   engine->name);
>
> intel_gt_handle_error:
>
>     /*
>      * Try engine reset when available. We fall back to full reset if
>      * single reset fails.
>      */
>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>         local_bh_disable();
>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>
> You said "However, the heartbeat is still present and is still the 
> watchdog by which engine resets are triggered", now I don't know what 
> you meant by this. It actually triggers a single engine reset in GuC 
> mode? Where in code does that happen if this block above shows it not 
> taking the engine reset path?
i915 sends down the per engine pulse.
GuC schedules the pulse
GuC attempts to pre-empt the currently active context
GuC detects the pre-emption timeout
GuC resets the engine

The fundamental process is exactly the same as in execlist mode. It's 
just that the above blocks of code (calls to intel_gt_handle_error and 
such) are now inside the GuC not i915.

Without the heartbeat going ping, there would be no context switching 
and thus no pre-emption, no pre-emption timeout and so no hang and reset 
recovery. And GuC cannot sent pulses by itself - it has no ability to 
generate context workloads. So we need i915 to send the pings and to 
gradually raise their priority. But the back half of the heartbeat code 
is now inside the GuC. It will simply never reach the i915 side timeout 
if GuC is doing the recovery (unless the heartbeat's final period is too 
short). We should only reach the i915 side timeout if GuC itself is 
toast. At which point we need the full GT reset to recover the GuC.

John.


>
>>> I am not sure it was the best way since full reset penalizes 
>>> everyone for one hanging engine. Better question would be why leave 
>>> heartbeats around to start with with GuC? If you want to use it to 
>>> health check GuC, as you say, maybe just send a CT message and 
>>> expect replies? Then full reset would make sense. It also achieves 
>>> the goal of not seconding guessing the submission backend you raise.
>> Adding yet another hang detection mechanism is yet more complication 
>> and is unnecessary when we already have one that works well enough. 
>> As above, the heartbeat is still required for sending the pulses that 
>> cause pre-emptions and so let GuC detect hangs. It also provides a 
>> fallback against a dead GuC by default. So why invent yet another wheel?
>
> Lets first clarify the previous block to make sure there aren't any 
> misunderstandings there.
>
> Regards,
>
> Tvrtko
>
>>> Like it is now, and the need for this series demonstrates it, the 
>>> whole thing has a pretty poor "impedance" match. Not even sure what 
>>> intel_guc_find_hung_context is doing in intel_engine_hearbeat.c - 
>>> why is that not in intel_gt_handle_error at least? Why is hearbeat 
>>> code special and other callers of intel_gt_handle_error don't need it?
>> There is no guilty context if the reset was triggered via debugfs or 
>> similar. And as stated ad nauseam, i915 is no longer handling the 
>> scheduling and so cannot make assumptions about what is or is not 
>> running on the hardware at any given time. And obviously, if the 
>> reset initiated by GuC itself then i915 should not be second guessing 
>> the guilty context as the GuC notification has already told us who 
>> was responsible.
>>
>> And to be clear, the 'poor impedance match' is purely because we 
>> don't have mid-thread pre-emption and so need a stupidly huge timeout 
>> on compute capable engines. Whereas, we don't want a total heatbeat 
>> timeout of a minute or more. That is the impedance mis-match. If the 
>> 640ms was acceptable for RCS then none of this hacky timeout 
>> algorithm mush would be needed.
>>
>> John.
>>
>>
>>>
>>> Regards,
>>>
>>> Tvrtko
>>

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>
> On 24/02/2022 19:19, John Harrison wrote:
>
> [snip]
>
>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>
>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>
>>>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>>>> timeslice_duration_ms;
>>>>>>>
>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>
>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>> because of tick conversion internally but because at backend 
>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or I 
>>>>>>> failed to find where it is handled.
>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>> allow overflows.
>>>>>
>>>>> Yes and no, this fixes it, but the first bug was not only due GuC 
>>>>> internal tick conversion. It was present ever since the u64 from 
>>>>> i915 was shoved into u32 sent to GuC. So even if GuC used the 
>>>>> value without additional multiplication, bug was be there. My 
>>>>> point being when GuC backend was added timeout_ms values should 
>>>>> have been limited/clamped to U32_MAX. The tick discovery is 
>>>>> additional limit on top.
>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>> noticed until I actually tried using very long timeouts to debug a 
>>>> particular problem. Now that it is noticed, we need some method of 
>>>> range checking and this simple clamp solves all the truncation 
>>>> problems.
>>>
>>> Agreed in principle, just please mention in the commit message all 
>>> aspects of the problem.
>>>
>>> I think we can get away without a Fixes: tag since it requires user 
>>> fiddling to break things in unexpected ways.
>>>
>>> I would though put in a code a clamping which expresses both, 
>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>> Perhaps internal ticks go away or anything and we are left with 
>>> plain 1:1 millisecond relationship.
>> Can certainly add a comment along the lines of "GuC API only takes a 
>> 32bit field but that is further reduced to GUC_LIMIT due to internal 
>> calculations which would otherwise overflow".
>>
>> But if the GuC limit is > u32 then, by definition, that means the GuC 
>> API has changed to take a u64 instead of a u32. So there will no u32 
>> truncation any more. So I'm not seeing a need to explicitly test the 
>> integer size when the value check covers that.
>
> Hmm I was thinking if the internal conversion in the GuC fw changes so 
> that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, then to be 
> extra safe by documenting in code there is the additional limit of the 
> data structure field. Say the field was changed to take some unit 
> larger than a millisecond. Then the check against the GuC MAX limit 
> define would not be enough, unless that would account both for 
> internal implementation and u32 in the protocol. Maybe that is 
> overdefensive but I don't see that it harms. 50-50, but it's do it 
> once and forget so I'd do it.
Huh?

How can the limit be greater than a u32 if the interface only takes a 
u32? By definition the limit would be clamped to u32 size.

If you mean that the GuC policy is in different units and those units 
might not overflow but ms units do, then actually that is already the 
case. The GuC works in us not ms. That's part of why the wrap around is 
so low, we have to multiply by 1000 before sending to GuC. However, that 
is actually irrelevant because the comparison is being done on the i915 
side in i915's units. We have to scale the GuC limit to match what i915 
is using. And the i915 side is u64 so if the scaling to i915 numbers 
overflows a u32 then who cares because that comparison can be done at 64 
bits wide.

If the units change then that is a backwards breaking API change that 
will require a manual driver code update. You can't just recompile with 
a new header and magically get an ms to us or ms to s conversion in your 
a = b assignment. The code will need to be changed to do the new unit 
conversion (note we already convert from ms to us, the GuC API is all 
expressed in us). And that code change will mean having to revisit any 
and all scaling, type conversions, etc. I.e. any pre-existing checks 
will not necessarily be valid and will need to be re-visted anyway. But 
as above, any scaling to GuC units has to be incorporated into the limit 
already because otherwise the limit would not fit in the GuC's own API.

John.

>
>>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>> ---
>>>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == 
>>>>>>>> RENDER_CLASS)
>>>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>>
>>>>>>> Hm "wanted".. There's been too much back and forth on the GuC 
>>>>>>> load options over the years to keep track.. 
>>>>>>> intel_engine_uses_guc work sounds like would work and read nicer.
>>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>>> existing one. If we want to rename it yet again then that would 
>>>>>> be a different patch set.
>>>>>
>>>>> $ grep intel_engine_uses_guc . -rl
>>>>> ./i915_perf.c
>>>>> ./i915_request.c
>>>>> ./selftests/intel_scheduler_helpers.c
>>>>> ./gem/i915_gem_context.c
>>>>> ./gt/intel_context.c
>>>>> ./gt/intel_engine.h
>>>>> ./gt/intel_engine_cs.c
>>>>> ./gt/intel_engine_heartbeat.c
>>>>> ./gt/intel_engine_pm.c
>>>>> ./gt/intel_reset.c
>>>>> ./gt/intel_lrc.c
>>>>> ./gt/selftest_context.c
>>>>> ./gt/selftest_engine_pm.c
>>>>> ./gt/selftest_hangcheck.c
>>>>> ./gt/selftest_mocs.c
>>>>> ./gt/selftest_workarounds.c
>>>>>
>>>>> Sounds better to me than intel_guc_submission_is_wanted. What does 
>>>>> the reader know whether "is wanted" translates to "is actually 
>>>>> used". Shrug on "is wanted".
>>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it 
>>>> too early in the boot up sequence? I never understood why that was 
>>>> necessary or why we need so many different ways to ask the same 
>>>> question. But this version already exists and definitely works 
>>>> without hitting any explosions.
>>>
>>> No idea if it causes a bug on, doesn't in the helper itself so maybe 
>>> you are saying it is called too early? Might be.. I think over time 
>>> the nice idea we had that "setup" and "init" phases of engine setup 
>>> clearly separated got destroyed a bit. There would always be an 
>>> option to move this clamping in a later phase, once the submission 
>>> method is known. One could argue that if the submission method is 
>>> not yet known at this point, it is even wrong to clamp based on 
>>> something which will only be decided later. Because:
>>>
>>> int intel_engines_init(struct intel_gt *gt)
>>> {
>>>     int (*setup)(struct intel_engine_cs *engine);
>>>     struct intel_engine_cs *engine;
>>>     enum intel_engine_id id;
>>>     int err;
>>>
>>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>>
>>> So this uses "uses", not "wanted". Presumably the point for having 
>>> "wanted" and "uses" is that they can disagree, in which case if you 
>>> clamp early based on "wanted" that suggests it could be wrong.
>>
>> Okay, looks like I was getting confused with intel_guc_is_used(). 
>> That one blows up if called too early.
>>
>> I'll change it to _uses_ and repost, then.
>
> Check that it isn't called too early, before gt->submission_setup is set.
Obviously it is because it blew up. But I am not re-writing the driver 
start up sequence just to use the word 'use' instead of 'want'.

>
>>
>>>
>>>>>>> And limit to class instead of applying to all engines looks like 
>>>>>>> a miss.
>>>>>> As per follow up email, the class limit is not applied here.
>>>>>>
>>>>>>>
>>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>>> +            engine->props.timeslice_duration_ms = 
>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>>
>>>>>>> I am not sure logging such message during driver load is useful. 
>>>>>>> Sounds more like a confused driver which starts with one value 
>>>>>>> and then overrides itself. I'd just silently set the value 
>>>>>>> appropriate for the active backend. Preemption timeout kconfig 
>>>>>>> text already documents the fact timeouts can get overriden at 
>>>>>>> runtime depending on platform+engine. So maybe just add same 
>>>>>>> text to timeslice kconfig.
>>>>>> The point is to make people aware if they compile with 
>>>>>> unsupported config options. As far as I know, there is no way to 
>>>>>> apply range checking or other limits to config defines. Which 
>>>>>> means that a user would silently get unwanted behaviour. That 
>>>>>> seems like a bad thing to me. If the driver is confused because 
>>>>>> the user built it in a confused manner then we should let them know.
>>>>>
>>>>> Okay, but I think make it notice low level.
>>>>>
>>>>> Also consider in patch 3/3 when you triple it, and then clamp back 
>>>>> down here. That's even more confused state since tripling gets 
>>>>> nerfed. I think that's also an argument to always account preempt 
>>>>> timeout in heartbeat interval calculation. Haven't got to your 
>>>>> reply on 2/3 yet though..
>>>> That sounds like even more reason to make sure the warning gets 
>>>> seen. The more complex the system and the more chances there are to 
>>>> get it wrong, the more important it is to have a nice easy to see 
>>>> and understand notification that it did go wrong.
>>>
>>> I did not disagree, just said make it notice, one level higher than 
>>> info! :)
>> But then it won't appear unless you have explicitly said an elevated 
>> debug level. Whereas info appears in dmesg by default (but is still 
>> not classed as an error by CI and such).
>
> Notice is higher than info! :) If info appears by default so does 
> notice, warning, err, etc...
Doh! I could have sworn those were the other way around.

Okay. Will update to use notice :).

>
> #define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
> #define KERN_ALERT      KERN_SOH "1"    /* action must be taken 
> immediately */
> #define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
> #define KERN_ERR        KERN_SOH "3"    /* error conditions */
> #define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
> #define KERN_NOTICE     KERN_SOH "5"    /* normal but significant 
> condition */
> #define KERN_INFO       KERN_SOH "6"    /* informational */
> #define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */
>
>>> But also think how, if we agree to go with tripling, that you'd have 
>>> to consider that in the sysfs store when hearbeat timeout is 
>>> written, to consider whether or not to triple and error out if 
>>> preemption timeout is over limit.
>> I see this as just setting the default values. If an end user is 
>> explicitly overriding the defaults then we should obey what they have 
>> requested. If they are changing the heartbeat interval then they can 
>> also change the pre-emption timeout appropriately.
>
> Question is can they unknowingly and without any feedback configure a 
> much worse state than they expect? Like when they set heartbeats up to 
> some value, everything is configured as you intended - but if you go 
> over a certain hidden limit the overall scheme degrades in some way. 
> What is the failure mode here if you silently let them do that?
You can always configure things to be worse than expected. If you don't 
understand what you are doing then any control can make things worse 
instead of better. The assumption is that if a user is savvy enough to 
be writing to sysfs overrides of kernel parameters then they know what 
those parameters are and what their implications are. If they want to 
set a very short heartbeat with a very long pre-emption timeout then its 
their problem if they hit frequent TDRs. Conversely, if they want to set 
a very long heartbeat with a very short pre-emption timeout then its 
still their problem if they hit frequent TDRs.

But if the user explicitly requests a heartbeat period of 3s and a 
pre-emption timeout of 2s and the i915 arbitrarily splats their 2s and 
makes it 9s then that is wrong.

We should give the driver defaults that work for the majority of users 
and then let the minority specify exactly what they need.

And there is no silent or hidden limit. If the user specifies a value 
too large then they will get -EINVAL. Nothing hidden or silent about 
that. Any other values are legal and the behaviour will be whatever has 
been requested.

John.


>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 25/02/2022 17:11, John Harrison wrote:
> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>> On 24/02/2022 20:02, John Harrison wrote:
>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>
>>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>>> hardware.
>>>>>>> Thus the pre-emption timeout was disabled as a workaround to prevent
>>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>>> heartbeat
>>>>>>> and its (longer) timeout. This is undesirable with GuC submission as
>>>>>>> the heartbeat is a full GT reset rather than a per engine reset 
>>>>>>> and so
>>>>>>> is much more destructive. Instead, just bump the pre-emption timeout
>>>>>>
>>>>>> Can we have a feature request to allow asking GuC for an engine 
>>>>>> reset?
>>>>> For what purpose?
>>>>
>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>
>>>>> GuC manages the scheduling of contexts across engines. With virtual 
>>>>> engines, the KMD has no knowledge of which engine a context might 
>>>>> be executing on. Even without virtual engines, the KMD still has no 
>>>>> knowledge of which context is currently executing on any given 
>>>>> engine at any given time.
>>>>>
>>>>> There is a reason why hang detection should be left to the entity 
>>>>> that is doing the scheduling. Any other entity is second guessing 
>>>>> at best.
>>>>>
>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>> submission is enabled is for the case where the KMD/GuC have got 
>>>>> out of sync with either other somehow or GuC itself has just 
>>>>> crashed. I.e. when no submission at all is working and we need to 
>>>>> reset the GuC itself and start over.
>>>>
>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>> nerfed already in GuC mode.
>>> Not sure what you mean by that claim. Engine resets are handled by 
>>> GuC because GuC handles the scheduling. You can't do the former if 
>>> you aren't doing the latter. However, the heartbeat is still present 
>>> and is still the watchdog by which engine resets are triggered. As 
>>> per the rest of the submission process, the hang detection and 
>>> recovery is split between i915 and GuC.
>>
>> I meant that "stopped heartbeat on engine XXX" can only do a full GPU 
>> reset on GuC.
> I mean that there is no 'stopped heartbeat on engine XXX' when i915 is 
> not handling the recovery part of the process.

Hmmmm?

static void
reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
{
	if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
		show_heartbeat(rq, engine);

	if (intel_engine_uses_guc(engine))
		/*
		 * GuC itself is toast or GuC's hang detection
		 * is disabled. Either way, need to find the
		 * hang culprit manually.
		 */
		intel_guc_find_hung_context(engine);

	intel_gt_handle_error(engine->gt, engine->mask,
			      I915_ERROR_CAPTURE,
			      "stopped heartbeat on %s",
			      engine->name);
}

How there is no "stopped hearbeat" in guc mode? From this code it certainly looks there is.

You say below heartbeats are going in GuC mode. Now I totally don't understand how they are going but there is allegedly no "stopped hearbeat".

>>
>>     intel_gt_handle_error(engine->gt, engine->mask,
>>                   I915_ERROR_CAPTURE,
>>                   "stopped heartbeat on %s",
>>                   engine->name);
>>
>> intel_gt_handle_error:
>>
>>     /*
>>      * Try engine reset when available. We fall back to full reset if
>>      * single reset fails.
>>      */
>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>         local_bh_disable();
>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>
>> You said "However, the heartbeat is still present and is still the 
>> watchdog by which engine resets are triggered", now I don't know what 
>> you meant by this. It actually triggers a single engine reset in GuC 
>> mode? Where in code does that happen if this block above shows it not 
>> taking the engine reset path?
> i915 sends down the per engine pulse.
> GuC schedules the pulse
> GuC attempts to pre-empt the currently active context
> GuC detects the pre-emption timeout
> GuC resets the engine
> 
> The fundamental process is exactly the same as in execlist mode. It's 
> just that the above blocks of code (calls to intel_gt_handle_error and 
> such) are now inside the GuC not i915.
> 
> Without the heartbeat going ping, there would be no context switching 
> and thus no pre-emption, no pre-emption timeout and so no hang and reset 
> recovery. And GuC cannot sent pulses by itself - it has no ability to 
> generate context workloads. So we need i915 to send the pings and to 
> gradually raise their priority. But the back half of the heartbeat code 
> is now inside the GuC. It will simply never reach the i915 side timeout 
> if GuC is doing the recovery (unless the heartbeat's final period is too 
> short). We should only reach the i915 side timeout if GuC itself is 
> toast. At which point we need the full GT reset to recover the GuC.

If workload is not preempting and reset does not work, like engine is truly stuck, does the current code hit "stopped heartbeat" or not in GuC mode?

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 24/02/2022 19:51, John Harrison wrote:
> On 2/24/2022 11:19, John Harrison wrote:
>> [snip]
>>
>> I'll change it to _uses_ and repost, then.
>>
> [    7.683149] kernel BUG at drivers/gpu/drm/i915/gt/uc/intel_guc.h:367!
> 
> Told you that one went bang.

intel_guc_is_used ?

My suggestion was intel_engine_uses_guc. But do note I think it would 
not work either because of setup vs init ordering. Not sure that it 
makes sense at engine granularity anyway.

Still I do think "is wanted" is quite bad.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/25/2022 09:39, Tvrtko Ursulin wrote:
> On 25/02/2022 17:11, John Harrison wrote:
>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>> On 24/02/2022 20:02, John Harrison wrote:
>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>
>>>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>>>> hardware.
>>>>>>>> Thus the pre-emption timeout was disabled as a workaround to 
>>>>>>>> prevent
>>>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>>>> heartbeat
>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>> submission as
>>>>>>>> the heartbeat is a full GT reset rather than a per engine reset 
>>>>>>>> and so
>>>>>>>> is much more destructive. Instead, just bump the pre-emption 
>>>>>>>> timeout
>>>>>>>
>>>>>>> Can we have a feature request to allow asking GuC for an engine 
>>>>>>> reset?
>>>>>> For what purpose?
>>>>>
>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>
>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>> context might be executing on. Even without virtual engines, the 
>>>>>> KMD still has no knowledge of which context is currently 
>>>>>> executing on any given engine at any given time.
>>>>>>
>>>>>> There is a reason why hang detection should be left to the entity 
>>>>>> that is doing the scheduling. Any other entity is second guessing 
>>>>>> at best.
>>>>>>
>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>> submission is enabled is for the case where the KMD/GuC have got 
>>>>>> out of sync with either other somehow or GuC itself has just 
>>>>>> crashed. I.e. when no submission at all is working and we need to 
>>>>>> reset the GuC itself and start over.
>>>>>
>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>> nerfed already in GuC mode.
>>>> Not sure what you mean by that claim. Engine resets are handled by 
>>>> GuC because GuC handles the scheduling. You can't do the former if 
>>>> you aren't doing the latter. However, the heartbeat is still 
>>>> present and is still the watchdog by which engine resets are 
>>>> triggered. As per the rest of the submission process, the hang 
>>>> detection and recovery is split between i915 and GuC.
>>>
>>> I meant that "stopped heartbeat on engine XXX" can only do a full 
>>> GPU reset on GuC.
>> I mean that there is no 'stopped heartbeat on engine XXX' when i915 
>> is not handling the recovery part of the process.
>
> Hmmmm?
>
> static void
> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
> {
>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>         show_heartbeat(rq, engine);
>
>     if (intel_engine_uses_guc(engine))
>         /*
>          * GuC itself is toast or GuC's hang detection
>          * is disabled. Either way, need to find the
>          * hang culprit manually.
>          */
>         intel_guc_find_hung_context(engine);
>
>     intel_gt_handle_error(engine->gt, engine->mask,
>                   I915_ERROR_CAPTURE,
>                   "stopped heartbeat on %s",
>                   engine->name);
> }
>
> How there is no "stopped hearbeat" in guc mode? From this code it 
> certainly looks there is.
Only when the GuC is toast and it is no longer an engine reset but a 
full GT reset that is required. So technically, it is not a 'stopped 
heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.

>
> You say below heartbeats are going in GuC mode. Now I totally don't 
> understand how they are going but there is allegedly no "stopped 
> hearbeat".
Because if GuC is handling the detection and recovery then i915 will not 
reach that point. GuC will do the engine reset and start scheduling the 
next context before the heartbeat period expires. So the notification 
will be a G2H about a specific context being reset rather than the i915 
notification about a stopped heartbeat.

>
>>>
>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>                   I915_ERROR_CAPTURE,
>>>                   "stopped heartbeat on %s",
>>>                   engine->name);
>>>
>>> intel_gt_handle_error:
>>>
>>>     /*
>>>      * Try engine reset when available. We fall back to full reset if
>>>      * single reset fails.
>>>      */
>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>         local_bh_disable();
>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>
>>> You said "However, the heartbeat is still present and is still the 
>>> watchdog by which engine resets are triggered", now I don't know 
>>> what you meant by this. It actually triggers a single engine reset 
>>> in GuC mode? Where in code does that happen if this block above 
>>> shows it not taking the engine reset path?
>> i915 sends down the per engine pulse.
>> GuC schedules the pulse
>> GuC attempts to pre-empt the currently active context
>> GuC detects the pre-emption timeout
>> GuC resets the engine
>>
>> The fundamental process is exactly the same as in execlist mode. It's 
>> just that the above blocks of code (calls to intel_gt_handle_error 
>> and such) are now inside the GuC not i915.
>>
>> Without the heartbeat going ping, there would be no context switching 
>> and thus no pre-emption, no pre-emption timeout and so no hang and 
>> reset recovery. And GuC cannot sent pulses by itself - it has no 
>> ability to generate context workloads. So we need i915 to send the 
>> pings and to gradually raise their priority. But the back half of the 
>> heartbeat code is now inside the GuC. It will simply never reach the 
>> i915 side timeout if GuC is doing the recovery (unless the 
>> heartbeat's final period is too short). We should only reach the i915 
>> side timeout if GuC itself is toast. At which point we need the full 
>> GT reset to recover the GuC.
>
> If workload is not preempting and reset does not work, like engine is 
> truly stuck, does the current code hit "stopped heartbeat" or not in 
> GuC mode?
Hang on, where did 'reset does not work' come into this?

If GuC is alive and the hardware is not broken then no, it won't. That's 
the whole point. GuC does the detection and recovery. The KMD will never 
reach 'stopped heartbeat'.

If the hardware is broken and the reset does not work then GuC will send 
a 'failed reset' notification to the KMD. The KMD treats that as a major 
error and immediately does a full GT reset. So there is still no 
'stopped heartbeat'.

If GuC has died (or a KMD bug has caused sufficient confusion to make it 
think the GuC has died) then yes, you will reach that code. But in that 
case it is not an engine reset that is required, it is a full GT reset 
including a reset of the GuC.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

I'll try to simplify the discussion here:

On 24/02/2022 19:45, John Harrison wrote:
> On 2/24/2022 03:41, Tvrtko Ursulin wrote:
>> On 23/02/2022 20:00, John Harrison wrote:
>>> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>>>> On 23/02/2022 02:45, John Harrison wrote:
>>>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>
>>>>>>> Compute workloads are inherantly not pre-emptible for long 
>>>>>>> periods on
>>>>>>> current hardware. As a workaround for this, the pre-emption timeout
>>>>>>> for compute capable engines was disabled. This is undesirable 
>>>>>>> with GuC
>>>>>>> submission as it prevents per engine reset of hung contexts. 
>>>>>>> Hence the
>>>>>>> next patch will re-enable the timeout but bumped up by an order of
>>>>>>> magnititude.
>>>>>>
>>>>>> (Some typos above.)
>>>>> I'm spotting 'inherently' but not anything else.
>>>>
>>>> Magnititude! O;)
>>> Doh!
>>>
>>> [snip]
>>>
>>>>> Whereas, bumping all heartbeat periods to be greater than the 
>>>>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>>>>> total heartbeat time of about a minute. Which is a very long time 
>>>>> to wait for a hang to be detected and recovered. Especially when 
>>>>> the official limit on a context responding to an 'are you dead' 
>>>>> query is only 7.5 seconds.
>>>>
>>>> Not sure how did you get one minute?
>>> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 75 
>>> => 1 minute 15 seconds
>>>
>>> Even ignoring any safety factor and just going with 7.5 * 5 still 
>>> gets you to 37.5 seconds which is over a half a minute and likely to 
>>> race.
>>
>> Ah because my starting point is there should be no preempt timeout = 
>> heartbeat * 3, I just think that's too ugly.
> Then complain at the hardware designers to give us mid-thread 
> pre-emption back. The heartbeat is only one source of pre-emption 
> events. For example, a user can be running multiple contexts in parallel 
> and expecting them to time slice on a single engine. Or maybe a user is 
> just running one compute task in the background but is doing render work 
> in the foreground. Etc.
> 
> There was a reason the original hack was to disable pre-emption rather 
> than increase the heartbeat. This is simply a slightly less ugly version 
> of the same hack. And unfortunately, the basic idea of the hack is 
> non-negotiable.
> 
> As per other comments, 'tP(RCS) = tH *3' or 'tP(RCS) = tP(default) * 12' 
> or 'tP(RCS) = 7500' are the available options. Given that the heartbeat 
> is the ever present hard limit, it seems most plausible to base the hack 
> on that. Any of the others works, though. Although I think a explicit 
> hardcoded value is the most ugly. I guess the other option is to add 
> CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE and default that to 7500.
> 
> Take your pick. But 640ms is not allowed.
> 
>>
>>>> Regardless, crux of argument was to avoid GuC engine reset and 
>>>> heartbeat reset racing with each other, and to do that by 
>>>> considering the preempt timeout with the heartbeat interval. I was 
>>>> thinking about this scenario in this series:
>>>>
>>>> [Please use fixed width font and no line wrap to view.]
>>>>
>>>> A)
>>>>
>>>> tP = preempt timeout
>>>> tH = hearbeat interval
>>>>
>>>> tP = 3 * tH
>>>>
>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>
>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 
>>>> * tH] ----> FULL RESET
>>>>                                                        |
>>>>                                                        \- preemption 
>>>> triggered, tP = 3 * tH ------\
>>>> \-> preempt timeout would hit here
>>>>
>>>> Here we have collateral damage due full reset, since we can't tell 
>>>> GuC to reset just one engine and we fudged tP just to "account" for 
>>>> heartbeats.
>>> You are missing the whole point of the patch series which is that the 
>>> last heartbeat period is '2 * tP' not '2 * tH'.
>>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>>>
>>> By making the last period double the pre-emption timeout, it is 
>>> guaranteed that the FULL RESET stage cannot be hit before the 
>>> hardware has attempted and timed-out on at least one pre-emption.
>>
>> Oh well :) that probably means the overall scheme is too odd for me. 
>> tp = 3tH and last pulse after 2tP I mean.
> To be accurate, it is 'tP(RCS) = 3 * tH(default); tH(final) = 
> tP(current) * 2;'. Seems fairly straight forward to me. It's not a 
> recursive definition or anything like that. It gives us a total 
> heartbeat timeout that is close to the original version but still allows 
> at least one pre-emption event.
> 
>>
>>> [snip]
>>>
>>>
>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- [2 
>>>> * tH] ----> full reset would be here
>>>>                |
>>>>                \- preemption triggered, tP = 3 * tH ----------------\
>>>> \-> Preempt timeout reset
>>>>
>>>> Here is is kind of least worse, but question is why we fudged tP 
>>>> when it gives us nothing good in this case.
>>>>
>>> The point of fudging tP(RCS) is to give compute workloads longer to 
>>> reach a pre-emptible point (given that EU walkers are basically not 
>>> pre-emptible). The reason for doing the fudge is not connected to the 
>>> heartbeat at all. The fact that it causes problems for the heartbeat 
>>> is an undesired side effect.
>>>
>>> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
>>> calculation that gave us something that all interested parties were 
>>> vaguely happy with. It could just as easily be a fixed, hard coded 
>>> value of 7.5s but having it based on something configurable seemed 
>>> more sensible. The other option was 'tP(RCS) = tP * 12' but that felt 
>>> more arbitrary than basing it on the average heartbeat timeout. As 
>>> in, three heartbeat periods is about what a normal prio task gets 
>>> before it gets pre-empted by the heartbeat. So using that for general 
>>> purpose pre-emptions (e.g. time slicing between multiple user apps) 
>>> seems reasonable.
>>
>> I think the fact you say tP fudge is not related to heartbeats and 
>> then go to mention heartbeat even in the "formula" which uses no tH is 
>> saying something (at least that's how I read the 7.5s option). :)
> I said the tP fudge is not because of the heartbeat. It is obviously 
> related.
> 
> As per comment above, the fudge factor is based on the heartbeat because 
> the heartbeat is the ultimate limit. But the *reason* for the fudge fact 
> has nothing to do with the heartbeat. It is required even if heartbeats 
> are disabled.
> 
>>
>>>> B)
>>>>
>>>> Instead, my idea to account for preempt timeout when calculating 
>>>> when to schedule next hearbeat would look like this:
>>>>
>>>> First of all tP can be left at a large value unrelated to tH. Lets 
>>>> say tP = 640ms. tH stays 2.5s.
>>> 640ms is not 'large'. The requirement is either zero (disabled) or 
>>> region of 7.5s. The 640ms figure is the default for non-compute 
>>> engines. Anything that can run EUs needs to be 'huge'.
>>>
>>>
>>>>
>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>
>>>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] --> 
>>>> Pulse3 <-- [tH + tP] --> full reset would be here
>>> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full 
>>> five periods is therefore 50s, which is practically a minute.
>>
>> No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.
> Not good enough. After 2.5s, we send a pulse. After a further 640ms we 
> perform an engine reset. That means your compute workload had only 640ms 
> after being told to pre-empt to reach a pre-emption point. That won't 
> work. It needs to be multiple seconds.
> 
>>
>>> [snip]
>>>
>>>> Am I missing some requirement or you see another problem with this 
>>>> idea?
>>>>
>>>>>> On a related topic, if GuC engine resets stop working when preempt 
>>>>>> timeout is set to zero - I think we need to somehow let the user 
>>>>>> know if they try to tweak it via sysfs. Perhaps go as far as 
>>>>>> -EINVAL in GuC mode, if i915.reset has not explicitly disabled 
>>>>>> engine resets.
>>>>> Define 'stops working'. The definition of the sysfs interface is 
>>>>> that a value of zero disables pre-emption. If you don't have 
>>>>> pre-emption and your hang detection mechanism relies on pre-emption 
>>>>> then you don't have a hang detection mechanism either. If the user 
>>>>> really wants to allow 
>>>>
>>>> By stops working I meant that it stops working. :)
>>>>
>>>> With execlist one can disable preempt timeout and "stopped 
>>>> heartbeat" can still reset the stuck engine and so avoid collateral 
>>>> damage. With GuC it appears this is not possible. So I was thinking 
>>>> this is something worthy a log notice.
>>>>
>>>>> their context to run forever and never be pre-empted then that 
>>>>> means they also don't want it to be reset arbitrarily. Which means 
>>>>> they would also be disabling the heartbeat timer as well. Indeed, 
>>>>> this is what we 
>>>>
>>>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>>>> upstream but hearbeat is not and so hangcheck still works.
>>> The pre-emption disable in upstream is not a valid solution for 
>>> compute customers. It is a worst-of-all-worlds hack for general 
>>> usage. As noted already, any actual compute specific customer is 
>>> advised to disable all forms of reset and do their hang detection 
>>> manually. A slightly less worse hack for customers that are not 
>>> actually running long compute workloads (i.e. the vast majority of 
>>> end users) is to just use a long pre-emption timeout.
>>
>> If disabled preemption timeout is worst of all words and compute needs 
>> to disable heartbeat as well then why did we put it in? Perhaps it was 
>> not know at the time it would not be good enough. But anyway, do I 
>> read you correct that current thinking is it would be better to leave 
>> it at default 640ms?
> No. We cannot have the RCS default to 640ms.
> 
> Note that there is a difference between 'general end user who might run 
> some compute' and 'compute focused customer'. The driver defaults 
> (disabled or 7500ms) are for the general user who gets the 
> out-of-the-box experience and expects to be able to run 'normal' 
> workloads without hitting problems. I.e. they expect hung tasks to get 
> reset in a timely manner and while they might run some AI or other 
> compute workloads, they are not a HPC datacenter. Whereas the compute 
> datacenter customer expects their workloads to run for arbitrarily long 
> times (minutes, hours, maybe even days) without being arbitrarily 
> killed. Those customers will be explicitly configuring their datacenter 
> server for that scenario and thus don't care what the defaults are.

Okay maybe I misunderstood what you were saying earlier about worst of 
all worlds and all. But tell me this, if preemption timeout on RCS is 
not directly related to hearbeats, but to some pessimistic expected user 
workloads, what is wrong with my scheme of calculating the next 
heartbeat pulse as tH + tP?

We can leave tH as default 2.5s and tP you set for RCS to 12s if that is 
what you say is required. Or whatever long value really.

Your only objection is that ends up with too long total time before 
reset? Or something else as well?

It's long but it is correct in a way. Because we can't expect hearbeat 
to react quicker than the interval + preempt timeout (or timeslice for 
equal priority) + some scheduling latency.

I conceptually disagree with the last hearbeat pulse being special. If 
the user concept is "after N heartbeats you are out" and you want to 
make it "after N-1 heartbeats plus 2 preemption periods you are out", 
where preemption period actually depends on heartbeat period, then that 
sounds really convoluted to me.

And we don't know which of the pulses will trigger preemption since user 
priority we don't control. So low priority compute task gets reset after 
5s, normal priority gets to run for 12s. Effectively making context 
priority a variable in hangcheck.

>>
>> If so, if we went with my proposal, would everyone be happy? If yes, 
>> isn't it a simpler scheme? No special casing when setting the preempt 
>> timeout, no special casing of the last heartbeat pulse. Works 
>> predictably regardless of the priority of the hypothetical 
>> non-preemptible workload.
> No, we have to have the increased pre-emption timeout. And that has 
> ripple effects of making very long heartbeats or risking races with the 
> heartbeat beating the per engine reset.
> 
> 
>>
>>>>> advise compute customers to do. It is then up to the user 
>>>>> themselves to spot a hang and to manually kill (Ctrl+C, kill ###, 
>>>>> etc.) their task. Killing the CPU task will automatically clear up 
>>>>> any GPU resources allocated to that task (excepting context 
>>>>> persistence, which is a) broken and b) something we also tell 
>>>>> compute customers to disable).
>>>>
>>>> What is broken with context persistence? I noticed one patch 
>>>> claiming to be fixing something in that area which looked suspect. 
>>>> Has it been established no userspace relies on it?
>>> One major issue is that it has hooks into the execlist scheduler 
>>> backend. I forget the exact details right now. The implementation as 
>>> a whole is incredibly complex and convoluted :(. But there's stuff 
>>> about what happens when you disable the heartbeat after having closed 
>>> a persistence context's handle (and thus made it persisting). There's 
>>> also things like it sends a super high priority heartbeat pulse at 
>>> the point of becoming persisting. That plays havoc for platforms with 
>>> dependent engines and/or compute workloads. A context becomes 
>>> persisting on RCS and results in your unrealted CCS work being reset. 
>>> It's a mess.
>>>
>>> The comment from Daniel Vetter is that persistence should have no 
>>> connection to the heartbeat at all. All of that dynamic behaviour and 
>>> complexity should just be removed.
>>
>> Dependent engines is definitely a topic on it's own, outside 
>> hearbeats, persistence and all.
> Except that it has implications for persistence which the current driver 
> does not take into account.

Well current driver does not take RCS+CCS dependency into account so 
that should come first, or all in one package at least.

>> Otherwise there is definitely complexity in the execlists backend but 
>> I am not sure if logic persistence and heartbeats are so very 
>> connected. It does send a pulse when heartbeat interval is changed, 
>> which if going to zero, it will kick of closed contexts if it can:
>>
>> static struct intel_engine_cs *
>> __execlists_schedule_in(struct i915_request *rq)
>> {
>>     struct intel_engine_cs * const engine = rq->engine;
>>     struct intel_context * const ce = rq->context;
>>
>>     intel_context_get(ce);
>>
>>     if (unlikely(intel_context_is_closed(ce) &&
>>              !intel_engine_has_heartbeat(engine)))
>>         intel_context_set_banned(ce);
>>
>>     if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
>>         reset_active(rq, engine);
>>
>> Is this what you mean? The point of this is to make sure persistent 
>> context does not hog the engine forever if hangcheck has been disabled.
>>
>> Reminds me of my improvement to customer experience which never got in 
>> (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). 
>> Point of that one was to avoid engine reset (or worse) after user 
>> presses "Ctrl-C" if something takes just over 1ms to cleanly complete.
> The plan is that the persistent contexts would still get the default 
> grace period (pre-emption timeout at least) to finish but Ctrl+C will 
> kill it within a timely manner if it does not finish.

Yes my patch does that. ;) Currently non-persistent is killed to quickly 
triggering pointless and alarming engine resets. Users reported this 
last year and I tried to fix it.

>>> Persistence itself can stay. There are valid UMD use cases. It is 
>>> just massively over complicated and doesn't work in all corner cases 
>>> when not using execlist submission or on newer platforms. The 
>>> simplification that is planned is to allow contexts to persist until 
>>> the associated DRM master handle is closed. At that point, all 
>>> contexts associated with that DRM handle are killed. That is what AMD 
>>> and others apparently implement.
>>
>> Okay, that goes against one recent IGT patch which appeared to work 
>> around something by moving the position of _context_ close.
> No it does not. The context close is not the trigger. The trigger is 

Well patch says:
"""
The spin all test relied on context persistence unecessarily by trying
to destroy contexts while keeping spinners active.
The current implementation of context persistence in i915 can cause
failures with GuC enabled, and persistence is not needed for this test.

Moving intel_ctx_destroy after igt_spin_end.
"""

Implying moving context close to after spin end fixes things for GuC, 
not fd close.

Regards,

Tvrtko

> closing the top level DRM handle. If your context has persistence 
> enabled (the default) then closing the context handle will have no 
> effect. No pulse, no pre-emption, no kill, nothing. But when the top 
> level handle is closed (application exit through whatever mechanism) 
> then all GPU resources will be cleaned up. As above, with at least a 
> pre-emption timeout grace period, but after that it is termination time.
> 
> The media use cases for persistence are all happy with this scheme. I 
> don't actually recall if we got a reply back from the OGL people. They 
> were definitely on the email thread/Jira task and did not complain. OCL 
> obviously don't care as their first action is to explicitly disable 
> persistence.
> 
> John.
> 
>>
>> Regards,
>>
>> Tvrtko
> 

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 25/02/2022 18:01, John Harrison wrote:
> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>> On 25/02/2022 17:11, John Harrison wrote:
>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>
>>>>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>>>>> hardware.
>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround to 
>>>>>>>>> prevent
>>>>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>>>>> heartbeat
>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>> submission as
>>>>>>>>> the heartbeat is a full GT reset rather than a per engine reset 
>>>>>>>>> and so
>>>>>>>>> is much more destructive. Instead, just bump the pre-emption 
>>>>>>>>> timeout
>>>>>>>>
>>>>>>>> Can we have a feature request to allow asking GuC for an engine 
>>>>>>>> reset?
>>>>>>> For what purpose?
>>>>>>
>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>
>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>> context might be executing on. Even without virtual engines, the 
>>>>>>> KMD still has no knowledge of which context is currently 
>>>>>>> executing on any given engine at any given time.
>>>>>>>
>>>>>>> There is a reason why hang detection should be left to the entity 
>>>>>>> that is doing the scheduling. Any other entity is second guessing 
>>>>>>> at best.
>>>>>>>
>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>> submission is enabled is for the case where the KMD/GuC have got 
>>>>>>> out of sync with either other somehow or GuC itself has just 
>>>>>>> crashed. I.e. when no submission at all is working and we need to 
>>>>>>> reset the GuC itself and start over.
>>>>>>
>>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>>> nerfed already in GuC mode.
>>>>> Not sure what you mean by that claim. Engine resets are handled by 
>>>>> GuC because GuC handles the scheduling. You can't do the former if 
>>>>> you aren't doing the latter. However, the heartbeat is still 
>>>>> present and is still the watchdog by which engine resets are 
>>>>> triggered. As per the rest of the submission process, the hang 
>>>>> detection and recovery is split between i915 and GuC.
>>>>
>>>> I meant that "stopped heartbeat on engine XXX" can only do a full 
>>>> GPU reset on GuC.
>>> I mean that there is no 'stopped heartbeat on engine XXX' when i915 
>>> is not handling the recovery part of the process.
>>
>> Hmmmm?
>>
>> static void
>> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
>> {
>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>         show_heartbeat(rq, engine);
>>
>>     if (intel_engine_uses_guc(engine))
>>         /*
>>          * GuC itself is toast or GuC's hang detection
>>          * is disabled. Either way, need to find the
>>          * hang culprit manually.
>>          */
>>         intel_guc_find_hung_context(engine);
>>
>>     intel_gt_handle_error(engine->gt, engine->mask,
>>                   I915_ERROR_CAPTURE,
>>                   "stopped heartbeat on %s",
>>                   engine->name);
>> }
>>
>> How there is no "stopped hearbeat" in guc mode? From this code it 
>> certainly looks there is.
> Only when the GuC is toast and it is no longer an engine reset but a 
> full GT reset that is required. So technically, it is not a 'stopped 
> heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
> 
>>
>> You say below heartbeats are going in GuC mode. Now I totally don't 
>> understand how they are going but there is allegedly no "stopped 
>> hearbeat".
> Because if GuC is handling the detection and recovery then i915 will not 
> reach that point. GuC will do the engine reset and start scheduling the 
> next context before the heartbeat period expires. So the notification 
> will be a G2H about a specific context being reset rather than the i915 
> notification about a stopped heartbeat.
> 
>>
>>>>
>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>                   I915_ERROR_CAPTURE,
>>>>                   "stopped heartbeat on %s",
>>>>                   engine->name);
>>>>
>>>> intel_gt_handle_error:
>>>>
>>>>     /*
>>>>      * Try engine reset when available. We fall back to full reset if
>>>>      * single reset fails.
>>>>      */
>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>         local_bh_disable();
>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>
>>>> You said "However, the heartbeat is still present and is still the 
>>>> watchdog by which engine resets are triggered", now I don't know 
>>>> what you meant by this. It actually triggers a single engine reset 
>>>> in GuC mode? Where in code does that happen if this block above 
>>>> shows it not taking the engine reset path?
>>> i915 sends down the per engine pulse.
>>> GuC schedules the pulse
>>> GuC attempts to pre-empt the currently active context
>>> GuC detects the pre-emption timeout
>>> GuC resets the engine
>>>
>>> The fundamental process is exactly the same as in execlist mode. It's 
>>> just that the above blocks of code (calls to intel_gt_handle_error 
>>> and such) are now inside the GuC not i915.
>>>
>>> Without the heartbeat going ping, there would be no context switching 
>>> and thus no pre-emption, no pre-emption timeout and so no hang and 
>>> reset recovery. And GuC cannot sent pulses by itself - it has no 
>>> ability to generate context workloads. So we need i915 to send the 
>>> pings and to gradually raise their priority. But the back half of the 
>>> heartbeat code is now inside the GuC. It will simply never reach the 
>>> i915 side timeout if GuC is doing the recovery (unless the 
>>> heartbeat's final period is too short). We should only reach the i915 
>>> side timeout if GuC itself is toast. At which point we need the full 
>>> GT reset to recover the GuC.
>>
>> If workload is not preempting and reset does not work, like engine is 
>> truly stuck, does the current code hit "stopped heartbeat" or not in 
>> GuC mode?
> Hang on, where did 'reset does not work' come into this?
> 
> If GuC is alive and the hardware is not broken then no, it won't. That's 
> the whole point. GuC does the detection and recovery. The KMD will never 
> reach 'stopped heartbeat'.
> 
> If the hardware is broken and the reset does not work then GuC will send 
> a 'failed reset' notification to the KMD. The KMD treats that as a major 
> error and immediately does a full GT reset. So there is still no 
> 'stopped heartbeat'.
> 
> If GuC has died (or a KMD bug has caused sufficient confusion to make it 
> think the GuC has died) then yes, you will reach that code. But in that 
> case it is not an engine reset that is required, it is a full GT reset 
> including a reset of the GuC.

Got it, so what is actually wrong is calling intel_gt_handle_error with 
an engine->mask in GuC mode. intel_engine_hearbeat.c/reset_engine should 
fork into two (in some way), depending on backend, so in the case of GuC 
it can call a variant of intel_gt_handle_error which would be explicitly 
about a full GPU reset only, instead of a sprinkle of 
intel_uc_uses_guc_submission in that function. Possibly even off load 
the reset to a single per gt worker, so that if multiple active engines 
trigger the reset roughly simultaneously, there is only one full GPU 
reset. And it gets correctly labeled as "dead GuC" or something.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>
> I'll try to simplify the discussion here:
>
> On 24/02/2022 19:45, John Harrison wrote:
>> On 2/24/2022 03:41, Tvrtko Ursulin wrote:
>>> On 23/02/2022 20:00, John Harrison wrote:
>>>> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>>>>> On 23/02/2022 02:45, John Harrison wrote:
>>>>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>
>>>>>>>> Compute workloads are inherantly not pre-emptible for long 
>>>>>>>> periods on
>>>>>>>> current hardware. As a workaround for this, the pre-emption 
>>>>>>>> timeout
>>>>>>>> for compute capable engines was disabled. This is undesirable 
>>>>>>>> with GuC
>>>>>>>> submission as it prevents per engine reset of hung contexts. 
>>>>>>>> Hence the
>>>>>>>> next patch will re-enable the timeout but bumped up by an order of
>>>>>>>> magnititude.
>>>>>>>
>>>>>>> (Some typos above.)
>>>>>> I'm spotting 'inherently' but not anything else.
>>>>>
>>>>> Magnititude! O;)
>>>> Doh!
>>>>
>>>> [snip]
>>>>
>>>>>> Whereas, bumping all heartbeat periods to be greater than the 
>>>>>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>>>>>> total heartbeat time of about a minute. Which is a very long time 
>>>>>> to wait for a hang to be detected and recovered. Especially when 
>>>>>> the official limit on a context responding to an 'are you dead' 
>>>>>> query is only 7.5 seconds.
>>>>>
>>>>> Not sure how did you get one minute?
>>>> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 
>>>> 75 => 1 minute 15 seconds
>>>>
>>>> Even ignoring any safety factor and just going with 7.5 * 5 still 
>>>> gets you to 37.5 seconds which is over a half a minute and likely 
>>>> to race.
>>>
>>> Ah because my starting point is there should be no preempt timeout = 
>>> heartbeat * 3, I just think that's too ugly.
>> Then complain at the hardware designers to give us mid-thread 
>> pre-emption back. The heartbeat is only one source of pre-emption 
>> events. For example, a user can be running multiple contexts in 
>> parallel and expecting them to time slice on a single engine. Or 
>> maybe a user is just running one compute task in the background but 
>> is doing render work in the foreground. Etc.
>>
>> There was a reason the original hack was to disable pre-emption 
>> rather than increase the heartbeat. This is simply a slightly less 
>> ugly version of the same hack. And unfortunately, the basic idea of 
>> the hack is non-negotiable.
>>
>> As per other comments, 'tP(RCS) = tH *3' or 'tP(RCS) = tP(default) * 
>> 12' or 'tP(RCS) = 7500' are the available options. Given that the 
>> heartbeat is the ever present hard limit, it seems most plausible to 
>> base the hack on that. Any of the others works, though. Although I 
>> think a explicit hardcoded value is the most ugly. I guess the other 
>> option is to add CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE and default 
>> that to 7500.
>>
>> Take your pick. But 640ms is not allowed.
>>
>>>
>>>>> Regardless, crux of argument was to avoid GuC engine reset and 
>>>>> heartbeat reset racing with each other, and to do that by 
>>>>> considering the preempt timeout with the heartbeat interval. I was 
>>>>> thinking about this scenario in this series:
>>>>>
>>>>> [Please use fixed width font and no line wrap to view.]
>>>>>
>>>>> A)
>>>>>
>>>>> tP = preempt timeout
>>>>> tH = hearbeat interval
>>>>>
>>>>> tP = 3 * tH
>>>>>
>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>
>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- 
>>>>> [2 * tH] ----> FULL RESET
>>>>>                                                        |
>>>>>                                                        \- 
>>>>> preemption triggered, tP = 3 * tH ------\
>>>>> \-> preempt timeout would hit here
>>>>>
>>>>> Here we have collateral damage due full reset, since we can't tell 
>>>>> GuC to reset just one engine and we fudged tP just to "account" 
>>>>> for heartbeats.
>>>> You are missing the whole point of the patch series which is that 
>>>> the last heartbeat period is '2 * tP' not '2 * tH'.
>>>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>>>>
>>>> By making the last period double the pre-emption timeout, it is 
>>>> guaranteed that the FULL RESET stage cannot be hit before the 
>>>> hardware has attempted and timed-out on at least one pre-emption.
>>>
>>> Oh well :) that probably means the overall scheme is too odd for me. 
>>> tp = 3tH and last pulse after 2tP I mean.
>> To be accurate, it is 'tP(RCS) = 3 * tH(default); tH(final) = 
>> tP(current) * 2;'. Seems fairly straight forward to me. It's not a 
>> recursive definition or anything like that. It gives us a total 
>> heartbeat timeout that is close to the original version but still 
>> allows at least one pre-emption event.
>>
>>>
>>>> [snip]
>>>>
>>>>
>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- 
>>>>> [2 * tH] ----> full reset would be here
>>>>>                |
>>>>>                \- preemption triggered, tP = 3 * tH ----------------\
>>>>> \-> Preempt timeout reset
>>>>>
>>>>> Here is is kind of least worse, but question is why we fudged tP 
>>>>> when it gives us nothing good in this case.
>>>>>
>>>> The point of fudging tP(RCS) is to give compute workloads longer to 
>>>> reach a pre-emptible point (given that EU walkers are basically not 
>>>> pre-emptible). The reason for doing the fudge is not connected to 
>>>> the heartbeat at all. The fact that it causes problems for the 
>>>> heartbeat is an undesired side effect.
>>>>
>>>> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
>>>> calculation that gave us something that all interested parties were 
>>>> vaguely happy with. It could just as easily be a fixed, hard coded 
>>>> value of 7.5s but having it based on something configurable seemed 
>>>> more sensible. The other option was 'tP(RCS) = tP * 12' but that 
>>>> felt more arbitrary than basing it on the average heartbeat 
>>>> timeout. As in, three heartbeat periods is about what a normal prio 
>>>> task gets before it gets pre-empted by the heartbeat. So using that 
>>>> for general purpose pre-emptions (e.g. time slicing between 
>>>> multiple user apps) seems reasonable.
>>>
>>> I think the fact you say tP fudge is not related to heartbeats and 
>>> then go to mention heartbeat even in the "formula" which uses no tH 
>>> is saying something (at least that's how I read the 7.5s option). :)
>> I said the tP fudge is not because of the heartbeat. It is obviously 
>> related.
>>
>> As per comment above, the fudge factor is based on the heartbeat 
>> because the heartbeat is the ultimate limit. But the *reason* for the 
>> fudge fact has nothing to do with the heartbeat. It is required even 
>> if heartbeats are disabled.
>>
>>>
>>>>> B)
>>>>>
>>>>> Instead, my idea to account for preempt timeout when calculating 
>>>>> when to schedule next hearbeat would look like this:
>>>>>
>>>>> First of all tP can be left at a large value unrelated to tH. Lets 
>>>>> say tP = 640ms. tH stays 2.5s.
>>>> 640ms is not 'large'. The requirement is either zero (disabled) or 
>>>> region of 7.5s. The 640ms figure is the default for non-compute 
>>>> engines. Anything that can run EUs needs to be 'huge'.
>>>>
>>>>
>>>>>
>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>
>>>>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] 
>>>>> --> Pulse3 <-- [tH + tP] --> full reset would be here
>>>> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full 
>>>> five periods is therefore 50s, which is practically a minute.
>>>
>>> No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.
>> Not good enough. After 2.5s, we send a pulse. After a further 640ms 
>> we perform an engine reset. That means your compute workload had only 
>> 640ms after being told to pre-empt to reach a pre-emption point. That 
>> won't work. It needs to be multiple seconds.
>>
>>>
>>>> [snip]
>>>>
>>>>> Am I missing some requirement or you see another problem with this 
>>>>> idea?
>>>>>
>>>>>>> On a related topic, if GuC engine resets stop working when 
>>>>>>> preempt timeout is set to zero - I think we need to somehow let 
>>>>>>> the user know if they try to tweak it via sysfs. Perhaps go as 
>>>>>>> far as -EINVAL in GuC mode, if i915.reset has not explicitly 
>>>>>>> disabled engine resets.
>>>>>> Define 'stops working'. The definition of the sysfs interface is 
>>>>>> that a value of zero disables pre-emption. If you don't have 
>>>>>> pre-emption and your hang detection mechanism relies on 
>>>>>> pre-emption then you don't have a hang detection mechanism 
>>>>>> either. If the user really wants to allow 
>>>>>
>>>>> By stops working I meant that it stops working. :)
>>>>>
>>>>> With execlist one can disable preempt timeout and "stopped 
>>>>> heartbeat" can still reset the stuck engine and so avoid 
>>>>> collateral damage. With GuC it appears this is not possible. So I 
>>>>> was thinking this is something worthy a log notice.
>>>>>
>>>>>> their context to run forever and never be pre-empted then that 
>>>>>> means they also don't want it to be reset arbitrarily. Which 
>>>>>> means they would also be disabling the heartbeat timer as well. 
>>>>>> Indeed, this is what we 
>>>>>
>>>>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>>>>> upstream but hearbeat is not and so hangcheck still works.
>>>> The pre-emption disable in upstream is not a valid solution for 
>>>> compute customers. It is a worst-of-all-worlds hack for general 
>>>> usage. As noted already, any actual compute specific customer is 
>>>> advised to disable all forms of reset and do their hang detection 
>>>> manually. A slightly less worse hack for customers that are not 
>>>> actually running long compute workloads (i.e. the vast majority of 
>>>> end users) is to just use a long pre-emption timeout.
>>>
>>> If disabled preemption timeout is worst of all words and compute 
>>> needs to disable heartbeat as well then why did we put it in? 
>>> Perhaps it was not know at the time it would not be good enough. But 
>>> anyway, do I read you correct that current thinking is it would be 
>>> better to leave it at default 640ms?
>> No. We cannot have the RCS default to 640ms.
>>
>> Note that there is a difference between 'general end user who might 
>> run some compute' and 'compute focused customer'. The driver defaults 
>> (disabled or 7500ms) are for the general user who gets the 
>> out-of-the-box experience and expects to be able to run 'normal' 
>> workloads without hitting problems. I.e. they expect hung tasks to 
>> get reset in a timely manner and while they might run some AI or 
>> other compute workloads, they are not a HPC datacenter. Whereas the 
>> compute datacenter customer expects their workloads to run for 
>> arbitrarily long times (minutes, hours, maybe even days) without 
>> being arbitrarily killed. Those customers will be explicitly 
>> configuring their datacenter server for that scenario and thus don't 
>> care what the defaults are.
>
> Okay maybe I misunderstood what you were saying earlier about worst of 
> all worlds and all. But tell me this, if preemption timeout on RCS is 
> not directly related to hearbeats, but to some pessimistic expected 
> user workloads, what is wrong with my scheme of calculating the next 
> heartbeat pulse as tH + tP?
>
> We can leave tH as default 2.5s and tP you set for RCS to 12s if that 
> is what you say is required. Or whatever long value really.
>
> Your only objection is that ends up with too long total time before 
> reset? Or something else as well?
An unnecessarily long total heartbeat timeout is the main objection. 
(2.5 + 12) * 5 = 72.5 seconds. That is a massive change from the current 
12.5s.

If we are happy with that huge increase then fine. But I'm pretty sure 
you are going to get a lot more bug reports about hung systems not 
recovering. 10-20s is just about long enough for someone to wait before 
leaning on the power button of their machine. Over a minute is not. That 
kind of delay is going to cause support issues.

>
> It's long but it is correct in a way. Because we can't expect hearbeat 
> to react quicker than the interval + preempt timeout (or timeslice for 
> equal priority) + some scheduling latency.
>
> I conceptually disagree with the last hearbeat pulse being special. If 
> the user concept is "after N heartbeats you are out" and you want to 
> make it "after N-1 heartbeats plus 2 preemption periods you are out", 
> where preemption period actually depends on heartbeat period, then 
> that sounds really convoluted to me.
>
> And we don't know which of the pulses will trigger preemption since 
> user priority we don't control. So low priority compute task gets 
> reset after 5s, normal priority gets to run for 12s. Effectively 
> making context priority a variable in hangcheck.
It already is. That is no different. The pre-emption is not triggered 
until the pulse is of equal or higher priority than the busy task. That 
is the case no matter whether you are running GuC or execlist, whether 
you have the original driver or an updated one.

And it doesn't matter which pulse triggers the pre-emption. All that 
matters is that once a pre-emption is attempted, if the busy context 
fails to relinquish the hardware within the pre-emption timeout limit 
then it will be forcibly evicted.


>
>>>
>>> If so, if we went with my proposal, would everyone be happy? If yes, 
>>> isn't it a simpler scheme? No special casing when setting the 
>>> preempt timeout, no special casing of the last heartbeat pulse. 
>>> Works predictably regardless of the priority of the hypothetical 
>>> non-preemptible workload.
>> No, we have to have the increased pre-emption timeout. And that has 
>> ripple effects of making very long heartbeats or risking races with 
>> the heartbeat beating the per engine reset.
>>
>>
>>>
>>>>>> advise compute customers to do. It is then up to the user 
>>>>>> themselves to spot a hang and to manually kill (Ctrl+C, kill ###, 
>>>>>> etc.) their task. Killing the CPU task will automatically clear 
>>>>>> up any GPU resources allocated to that task (excepting context 
>>>>>> persistence, which is a) broken and b) something we also tell 
>>>>>> compute customers to disable).
>>>>>
>>>>> What is broken with context persistence? I noticed one patch 
>>>>> claiming to be fixing something in that area which looked suspect. 
>>>>> Has it been established no userspace relies on it?
>>>> One major issue is that it has hooks into the execlist scheduler 
>>>> backend. I forget the exact details right now. The implementation 
>>>> as a whole is incredibly complex and convoluted :(. But there's 
>>>> stuff about what happens when you disable the heartbeat after 
>>>> having closed a persistence context's handle (and thus made it 
>>>> persisting). There's also things like it sends a super high 
>>>> priority heartbeat pulse at the point of becoming persisting. That 
>>>> plays havoc for platforms with dependent engines and/or compute 
>>>> workloads. A context becomes persisting on RCS and results in your 
>>>> unrealted CCS work being reset. It's a mess.
>>>>
>>>> The comment from Daniel Vetter is that persistence should have no 
>>>> connection to the heartbeat at all. All of that dynamic behaviour 
>>>> and complexity should just be removed.
>>>
>>> Dependent engines is definitely a topic on it's own, outside 
>>> hearbeats, persistence and all.
>> Except that it has implications for persistence which the current 
>> driver does not take into account.
>
> Well current driver does not take RCS+CCS dependency into account so 
> that should come first, or all in one package at least.
Not sure what you are arguing for here? Simplifying persistence as 
described will fix all the problems in one easy go. There is no point in 
adding yet more hideously complex code to make one corner case work when 
the real fix is to rip it all out.

>
>>> Otherwise there is definitely complexity in the execlists backend 
>>> but I am not sure if logic persistence and heartbeats are so very 
>>> connected. It does send a pulse when heartbeat interval is changed, 
>>> which if going to zero, it will kick of closed contexts if it can:
>>>
>>> static struct intel_engine_cs *
>>> __execlists_schedule_in(struct i915_request *rq)
>>> {
>>>     struct intel_engine_cs * const engine = rq->engine;
>>>     struct intel_context * const ce = rq->context;
>>>
>>>     intel_context_get(ce);
>>>
>>>     if (unlikely(intel_context_is_closed(ce) &&
>>>              !intel_engine_has_heartbeat(engine)))
>>>         intel_context_set_banned(ce);
>>>
>>>     if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
>>>         reset_active(rq, engine);
>>>
>>> Is this what you mean? The point of this is to make sure persistent 
>>> context does not hog the engine forever if hangcheck has been disabled.
>>>
>>> Reminds me of my improvement to customer experience which never got 
>>> in 
>>> (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). 
>>> Point of that one was to avoid engine reset (or worse) after user 
>>> presses "Ctrl-C" if something takes just over 1ms to cleanly complete.
>> The plan is that the persistent contexts would still get the default 
>> grace period (pre-emption timeout at least) to finish but Ctrl+C will 
>> kill it within a timely manner if it does not finish.
>
> Yes my patch does that. ;) Currently non-persistent is killed to 
> quickly triggering pointless and alarming engine resets. Users 
> reported this last year and I tried to fix it.
Except that your patch is adding yet more complexity to an already 
complex feature. The desire is to simplify the driver and make it more 
maintainable not complicated it further.

>
>>>> Persistence itself can stay. There are valid UMD use cases. It is 
>>>> just massively over complicated and doesn't work in all corner 
>>>> cases when not using execlist submission or on newer platforms. The 
>>>> simplification that is planned is to allow contexts to persist 
>>>> until the associated DRM master handle is closed. At that point, 
>>>> all contexts associated with that DRM handle are killed. That is 
>>>> what AMD and others apparently implement.
>>>
>>> Okay, that goes against one recent IGT patch which appeared to work 
>>> around something by moving the position of _context_ close.
>> No it does not. The context close is not the trigger. The trigger is 
>
> Well patch says:
> """
> The spin all test relied on context persistence unecessarily by trying
> to destroy contexts while keeping spinners active.
> The current implementation of context persistence in i915 can cause
> failures with GuC enabled, and persistence is not needed for this test.
>
> Moving intel_ctx_destroy after igt_spin_end.
> """
>
> Implying moving context close to after spin end fixes things for GuC, 
> not fd close.
That's because persistence is currently a big pile of poo and does not 
work in all the corner cases. The correct solution is to leave the IGT 
alone and just fix the implementation of persistence. However, the IGT 
update to not use the broken feature is a trivial test change (two 
lines?) whereas fixing the broken feature is a significant KMD re-work. 
It needs to be done but no-one currently has the time to do it. But 
trivially changing the test allows the test to work and test the 
features it is meant to be testing (which is not persistence).

John.

>
> Regards,
>
> Tvrtko
>
>> closing the top level DRM handle. If your context has persistence 
>> enabled (the default) then closing the context handle will have no 
>> effect. No pulse, no pre-emption, no kill, nothing. But when the top 
>> level handle is closed (application exit through whatever mechanism) 
>> then all GPU resources will be cleaned up. As above, with at least a 
>> pre-emption timeout grace period, but after that it is termination time.
>>
>> The media use cases for persistence are all happy with this scheme. I 
>> don't actually recall if we got a reply back from the OGL people. 
>> They were definitely on the email thread/Jira task and did not 
>> complain. OCL obviously don't care as their first action is to 
>> explicitly disable persistence.
>>
>> John.
>>
>>>
>>> Regards,
>>>
>>> Tvrtko
>>

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/25/2022 10:29, Tvrtko Ursulin wrote:
> On 25/02/2022 18:01, John Harrison wrote:
>> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>>> On 25/02/2022 17:11, John Harrison wrote:
>>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>
>>>>>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>>>>>> hardware.
>>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround to 
>>>>>>>>>> prevent
>>>>>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>>>>>> heartbeat
>>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>>> submission as
>>>>>>>>>> the heartbeat is a full GT reset rather than a per engine 
>>>>>>>>>> reset and so
>>>>>>>>>> is much more destructive. Instead, just bump the pre-emption 
>>>>>>>>>> timeout
>>>>>>>>>
>>>>>>>>> Can we have a feature request to allow asking GuC for an 
>>>>>>>>> engine reset?
>>>>>>>> For what purpose?
>>>>>>>
>>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>>
>>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>>> context might be executing on. Even without virtual engines, 
>>>>>>>> the KMD still has no knowledge of which context is currently 
>>>>>>>> executing on any given engine at any given time.
>>>>>>>>
>>>>>>>> There is a reason why hang detection should be left to the 
>>>>>>>> entity that is doing the scheduling. Any other entity is second 
>>>>>>>> guessing at best.
>>>>>>>>
>>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>>> submission is enabled is for the case where the KMD/GuC have 
>>>>>>>> got out of sync with either other somehow or GuC itself has 
>>>>>>>> just crashed. I.e. when no submission at all is working and we 
>>>>>>>> need to reset the GuC itself and start over.
>>>>>>>
>>>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>>>> nerfed already in GuC mode.
>>>>>> Not sure what you mean by that claim. Engine resets are handled 
>>>>>> by GuC because GuC handles the scheduling. You can't do the 
>>>>>> former if you aren't doing the latter. However, the heartbeat is 
>>>>>> still present and is still the watchdog by which engine resets 
>>>>>> are triggered. As per the rest of the submission process, the 
>>>>>> hang detection and recovery is split between i915 and GuC.
>>>>>
>>>>> I meant that "stopped heartbeat on engine XXX" can only do a full 
>>>>> GPU reset on GuC.
>>>> I mean that there is no 'stopped heartbeat on engine XXX' when i915 
>>>> is not handling the recovery part of the process.
>>>
>>> Hmmmm?
>>>
>>> static void
>>> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
>>> {
>>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>>         show_heartbeat(rq, engine);
>>>
>>>     if (intel_engine_uses_guc(engine))
>>>         /*
>>>          * GuC itself is toast or GuC's hang detection
>>>          * is disabled. Either way, need to find the
>>>          * hang culprit manually.
>>>          */
>>>         intel_guc_find_hung_context(engine);
>>>
>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>                   I915_ERROR_CAPTURE,
>>>                   "stopped heartbeat on %s",
>>>                   engine->name);
>>> }
>>>
>>> How there is no "stopped hearbeat" in guc mode? From this code it 
>>> certainly looks there is.
>> Only when the GuC is toast and it is no longer an engine reset but a 
>> full GT reset that is required. So technically, it is not a 'stopped 
>> heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
>>
>>>
>>> You say below heartbeats are going in GuC mode. Now I totally don't 
>>> understand how they are going but there is allegedly no "stopped 
>>> hearbeat".
>> Because if GuC is handling the detection and recovery then i915 will 
>> not reach that point. GuC will do the engine reset and start 
>> scheduling the next context before the heartbeat period expires. So 
>> the notification will be a G2H about a specific context being reset 
>> rather than the i915 notification about a stopped heartbeat.
>>
>>>
>>>>>
>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>                   I915_ERROR_CAPTURE,
>>>>>                   "stopped heartbeat on %s",
>>>>>                   engine->name);
>>>>>
>>>>> intel_gt_handle_error:
>>>>>
>>>>>     /*
>>>>>      * Try engine reset when available. We fall back to full reset if
>>>>>      * single reset fails.
>>>>>      */
>>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>>         local_bh_disable();
>>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>>
>>>>> You said "However, the heartbeat is still present and is still the 
>>>>> watchdog by which engine resets are triggered", now I don't know 
>>>>> what you meant by this. It actually triggers a single engine reset 
>>>>> in GuC mode? Where in code does that happen if this block above 
>>>>> shows it not taking the engine reset path?
>>>> i915 sends down the per engine pulse.
>>>> GuC schedules the pulse
>>>> GuC attempts to pre-empt the currently active context
>>>> GuC detects the pre-emption timeout
>>>> GuC resets the engine
>>>>
>>>> The fundamental process is exactly the same as in execlist mode. 
>>>> It's just that the above blocks of code (calls to 
>>>> intel_gt_handle_error and such) are now inside the GuC not i915.
>>>>
>>>> Without the heartbeat going ping, there would be no context 
>>>> switching and thus no pre-emption, no pre-emption timeout and so no 
>>>> hang and reset recovery. And GuC cannot sent pulses by itself - it 
>>>> has no ability to generate context workloads. So we need i915 to 
>>>> send the pings and to gradually raise their priority. But the back 
>>>> half of the heartbeat code is now inside the GuC. It will simply 
>>>> never reach the i915 side timeout if GuC is doing the recovery 
>>>> (unless the heartbeat's final period is too short). We should only 
>>>> reach the i915 side timeout if GuC itself is toast. At which point 
>>>> we need the full GT reset to recover the GuC.
>>>
>>> If workload is not preempting and reset does not work, like engine 
>>> is truly stuck, does the current code hit "stopped heartbeat" or not 
>>> in GuC mode?
>> Hang on, where did 'reset does not work' come into this?
>>
>> If GuC is alive and the hardware is not broken then no, it won't. 
>> That's the whole point. GuC does the detection and recovery. The KMD 
>> will never reach 'stopped heartbeat'.
>>
>> If the hardware is broken and the reset does not work then GuC will 
>> send a 'failed reset' notification to the KMD. The KMD treats that as 
>> a major error and immediately does a full GT reset. So there is still 
>> no 'stopped heartbeat'.
>>
>> If GuC has died (or a KMD bug has caused sufficient confusion to make 
>> it think the GuC has died) then yes, you will reach that code. But in 
>> that case it is not an engine reset that is required, it is a full GT 
>> reset including a reset of the GuC.
>
> Got it, so what is actually wrong is calling intel_gt_handle_error 
> with an engine->mask in GuC mode. intel_engine_hearbeat.c/reset_engine 
> should fork into two (in some way), depending on backend, so in the 
> case of GuC it can call a variant of intel_gt_handle_error which would 
> be explicitly about a full GPU reset only, instead of a sprinkle of 
> intel_uc_uses_guc_submission in that function. Possibly even off load 
> the reset to a single per gt worker, so that if multiple active 
> engines trigger the reset roughly simultaneously, there is only one 
> full GPU reset. And it gets correctly labeled as "dead GuC" or something.
>
Sure. Feel free to re-write the reset code yet again. That's another 
exceedingly fragile area of the driver.

However, that is unrelated to this patch set.

John.

> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 25/02/2022 19:03, John Harrison wrote:
> On 2/25/2022 10:29, Tvrtko Ursulin wrote:
>> On 25/02/2022 18:01, John Harrison wrote:
>>> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>>>> On 25/02/2022 17:11, John Harrison wrote:
>>>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>>
>>>>>>>>>>> Compute workloads are inherently not pre-emptible on current 
>>>>>>>>>>> hardware.
>>>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround to 
>>>>>>>>>>> prevent
>>>>>>>>>>> unwanted resets. Instead, the hang detection was left to the 
>>>>>>>>>>> heartbeat
>>>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>>>> submission as
>>>>>>>>>>> the heartbeat is a full GT reset rather than a per engine 
>>>>>>>>>>> reset and so
>>>>>>>>>>> is much more destructive. Instead, just bump the pre-emption 
>>>>>>>>>>> timeout
>>>>>>>>>>
>>>>>>>>>> Can we have a feature request to allow asking GuC for an 
>>>>>>>>>> engine reset?
>>>>>>>>> For what purpose?
>>>>>>>>
>>>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>>>
>>>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>>>> context might be executing on. Even without virtual engines, 
>>>>>>>>> the KMD still has no knowledge of which context is currently 
>>>>>>>>> executing on any given engine at any given time.
>>>>>>>>>
>>>>>>>>> There is a reason why hang detection should be left to the 
>>>>>>>>> entity that is doing the scheduling. Any other entity is second 
>>>>>>>>> guessing at best.
>>>>>>>>>
>>>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>>>> submission is enabled is for the case where the KMD/GuC have 
>>>>>>>>> got out of sync with either other somehow or GuC itself has 
>>>>>>>>> just crashed. I.e. when no submission at all is working and we 
>>>>>>>>> need to reset the GuC itself and start over.
>>>>>>>>
>>>>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>>>>> nerfed already in GuC mode.
>>>>>>> Not sure what you mean by that claim. Engine resets are handled 
>>>>>>> by GuC because GuC handles the scheduling. You can't do the 
>>>>>>> former if you aren't doing the latter. However, the heartbeat is 
>>>>>>> still present and is still the watchdog by which engine resets 
>>>>>>> are triggered. As per the rest of the submission process, the 
>>>>>>> hang detection and recovery is split between i915 and GuC.
>>>>>>
>>>>>> I meant that "stopped heartbeat on engine XXX" can only do a full 
>>>>>> GPU reset on GuC.
>>>>> I mean that there is no 'stopped heartbeat on engine XXX' when i915 
>>>>> is not handling the recovery part of the process.
>>>>
>>>> Hmmmm?
>>>>
>>>> static void
>>>> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
>>>> {
>>>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>>>         show_heartbeat(rq, engine);
>>>>
>>>>     if (intel_engine_uses_guc(engine))
>>>>         /*
>>>>          * GuC itself is toast or GuC's hang detection
>>>>          * is disabled. Either way, need to find the
>>>>          * hang culprit manually.
>>>>          */
>>>>         intel_guc_find_hung_context(engine);
>>>>
>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>                   I915_ERROR_CAPTURE,
>>>>                   "stopped heartbeat on %s",
>>>>                   engine->name);
>>>> }
>>>>
>>>> How there is no "stopped hearbeat" in guc mode? From this code it 
>>>> certainly looks there is.
>>> Only when the GuC is toast and it is no longer an engine reset but a 
>>> full GT reset that is required. So technically, it is not a 'stopped 
>>> heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
>>>
>>>>
>>>> You say below heartbeats are going in GuC mode. Now I totally don't 
>>>> understand how they are going but there is allegedly no "stopped 
>>>> hearbeat".
>>> Because if GuC is handling the detection and recovery then i915 will 
>>> not reach that point. GuC will do the engine reset and start 
>>> scheduling the next context before the heartbeat period expires. So 
>>> the notification will be a G2H about a specific context being reset 
>>> rather than the i915 notification about a stopped heartbeat.
>>>
>>>>
>>>>>>
>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>                   "stopped heartbeat on %s",
>>>>>>                   engine->name);
>>>>>>
>>>>>> intel_gt_handle_error:
>>>>>>
>>>>>>     /*
>>>>>>      * Try engine reset when available. We fall back to full reset if
>>>>>>      * single reset fails.
>>>>>>      */
>>>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>>>         local_bh_disable();
>>>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>>>
>>>>>> You said "However, the heartbeat is still present and is still the 
>>>>>> watchdog by which engine resets are triggered", now I don't know 
>>>>>> what you meant by this. It actually triggers a single engine reset 
>>>>>> in GuC mode? Where in code does that happen if this block above 
>>>>>> shows it not taking the engine reset path?
>>>>> i915 sends down the per engine pulse.
>>>>> GuC schedules the pulse
>>>>> GuC attempts to pre-empt the currently active context
>>>>> GuC detects the pre-emption timeout
>>>>> GuC resets the engine
>>>>>
>>>>> The fundamental process is exactly the same as in execlist mode. 
>>>>> It's just that the above blocks of code (calls to 
>>>>> intel_gt_handle_error and such) are now inside the GuC not i915.
>>>>>
>>>>> Without the heartbeat going ping, there would be no context 
>>>>> switching and thus no pre-emption, no pre-emption timeout and so no 
>>>>> hang and reset recovery. And GuC cannot sent pulses by itself - it 
>>>>> has no ability to generate context workloads. So we need i915 to 
>>>>> send the pings and to gradually raise their priority. But the back 
>>>>> half of the heartbeat code is now inside the GuC. It will simply 
>>>>> never reach the i915 side timeout if GuC is doing the recovery 
>>>>> (unless the heartbeat's final period is too short). We should only 
>>>>> reach the i915 side timeout if GuC itself is toast. At which point 
>>>>> we need the full GT reset to recover the GuC.
>>>>
>>>> If workload is not preempting and reset does not work, like engine 
>>>> is truly stuck, does the current code hit "stopped heartbeat" or not 
>>>> in GuC mode?
>>> Hang on, where did 'reset does not work' come into this?
>>>
>>> If GuC is alive and the hardware is not broken then no, it won't. 
>>> That's the whole point. GuC does the detection and recovery. The KMD 
>>> will never reach 'stopped heartbeat'.
>>>
>>> If the hardware is broken and the reset does not work then GuC will 
>>> send a 'failed reset' notification to the KMD. The KMD treats that as 
>>> a major error and immediately does a full GT reset. So there is still 
>>> no 'stopped heartbeat'.
>>>
>>> If GuC has died (or a KMD bug has caused sufficient confusion to make 
>>> it think the GuC has died) then yes, you will reach that code. But in 
>>> that case it is not an engine reset that is required, it is a full GT 
>>> reset including a reset of the GuC.
>>
>> Got it, so what is actually wrong is calling intel_gt_handle_error 
>> with an engine->mask in GuC mode. intel_engine_hearbeat.c/reset_engine 
>> should fork into two (in some way), depending on backend, so in the 
>> case of GuC it can call a variant of intel_gt_handle_error which would 
>> be explicitly about a full GPU reset only, instead of a sprinkle of 
>> intel_uc_uses_guc_submission in that function. Possibly even off load 
>> the reset to a single per gt worker, so that if multiple active 
>> engines trigger the reset roughly simultaneously, there is only one 
>> full GPU reset. And it gets correctly labeled as "dead GuC" or something.
>>
> Sure. Feel free to re-write the reset code yet again. That's another 
> exceedingly fragile area of the driver.

> However, that is unrelated to this patch set.

It's still okay to talk about improving things in my opinion. Especially 
since I think it is the same issue I raised late 2019 during internal 
review.

And I don't think it is fair to say that I should go and rewrite it, 
since I do not own the GuC backend area. Especially given the above.

If there is no motivation to improve it now then please at least track 
this, if it isn't already, in that internal Jira which was tracking all 
the areas of the GuC backend which could be improved.

I am also pretty sure if the design was cleaner it would have been more 
obvious to me, or anyone who happens to stumble there, what the purpose 
of intel_engine_heartbeat.c/reset_engine() is in GuC mode. So we 
wouldn't have to spend this long explaining things.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 25/02/2022 17:39, John Harrison wrote:
> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>
>> On 24/02/2022 19:19, John Harrison wrote:
>>
>> [snip]
>>
>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>
>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>
>>>>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>>>>> timeslice_duration_ms;
>>>>>>>>
>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>
>>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>>> because of tick conversion internally but because at backend 
>>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or I 
>>>>>>>> failed to find where it is handled.
>>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>>> allow overflows.
>>>>>>
>>>>>> Yes and no, this fixes it, but the first bug was not only due GuC 
>>>>>> internal tick conversion. It was present ever since the u64 from 
>>>>>> i915 was shoved into u32 sent to GuC. So even if GuC used the 
>>>>>> value without additional multiplication, bug was be there. My 
>>>>>> point being when GuC backend was added timeout_ms values should 
>>>>>> have been limited/clamped to U32_MAX. The tick discovery is 
>>>>>> additional limit on top.
>>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>>> noticed until I actually tried using very long timeouts to debug a 
>>>>> particular problem. Now that it is noticed, we need some method of 
>>>>> range checking and this simple clamp solves all the truncation 
>>>>> problems.
>>>>
>>>> Agreed in principle, just please mention in the commit message all 
>>>> aspects of the problem.
>>>>
>>>> I think we can get away without a Fixes: tag since it requires user 
>>>> fiddling to break things in unexpected ways.
>>>>
>>>> I would though put in a code a clamping which expresses both, 
>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>>> Perhaps internal ticks go away or anything and we are left with 
>>>> plain 1:1 millisecond relationship.
>>> Can certainly add a comment along the lines of "GuC API only takes a 
>>> 32bit field but that is further reduced to GUC_LIMIT due to internal 
>>> calculations which would otherwise overflow".
>>>
>>> But if the GuC limit is > u32 then, by definition, that means the GuC 
>>> API has changed to take a u64 instead of a u32. So there will no u32 
>>> truncation any more. So I'm not seeing a need to explicitly test the 
>>> integer size when the value check covers that.
>>
>> Hmm I was thinking if the internal conversion in the GuC fw changes so 
>> that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, then to be 
>> extra safe by documenting in code there is the additional limit of the 
>> data structure field. Say the field was changed to take some unit 
>> larger than a millisecond. Then the check against the GuC MAX limit 
>> define would not be enough, unless that would account both for 
>> internal implementation and u32 in the protocol. Maybe that is 
>> overdefensive but I don't see that it harms. 50-50, but it's do it 
>> once and forget so I'd do it.
> Huh?
> 
> How can the limit be greater than a u32 if the interface only takes a 
> u32? By definition the limit would be clamped to u32 size.
> 
> If you mean that the GuC policy is in different units and those units 
> might not overflow but ms units do, then actually that is already the 
> case. The GuC works in us not ms. That's part of why the wrap around is 
> so low, we have to multiply by 1000 before sending to GuC. However, that 
> is actually irrelevant because the comparison is being done on the i915 
> side in i915's units. We have to scale the GuC limit to match what i915 
> is using. And the i915 side is u64 so if the scaling to i915 numbers 
> overflows a u32 then who cares because that comparison can be done at 64 
> bits wide.
> 
> If the units change then that is a backwards breaking API change that 
> will require a manual driver code update. You can't just recompile with 
> a new header and magically get an ms to us or ms to s conversion in your 
> a = b assignment. The code will need to be changed to do the new unit 
> conversion (note we already convert from ms to us, the GuC API is all 
> expressed in us). And that code change will mean having to revisit any 
> and all scaling, type conversions, etc. I.e. any pre-existing checks 
> will not necessarily be valid and will need to be re-visted anyway. But 
> as above, any scaling to GuC units has to be incorporated into the limit 
> already because otherwise the limit would not fit in the GuC's own API.

Yes I get that, I was just worried that u32 field in the protocol and 
GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the source code 
and then how to protect against forgetting to update both in sync.

Like if the protocol was changed to take nanoseconds, and firmware 
implementation changed to support the full range, but define 
left/forgotten at 100s. That would then overflow u32.

Regards,

Tvrtko

> John.
> 
>>
>>>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>> ---
>>>>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 +++++++++++++++
>>>>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 ++++++++++++++
>>>>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h |  9 +++++++++
>>>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == 
>>>>>>>>> RENDER_CLASS)
>>>>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>>>
>>>>>>>> Hm "wanted".. There's been too much back and forth on the GuC 
>>>>>>>> load options over the years to keep track.. 
>>>>>>>> intel_engine_uses_guc work sounds like would work and read nicer.
>>>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>>>> existing one. If we want to rename it yet again then that would 
>>>>>>> be a different patch set.
>>>>>>
>>>>>> $ grep intel_engine_uses_guc . -rl
>>>>>> ./i915_perf.c
>>>>>> ./i915_request.c
>>>>>> ./selftests/intel_scheduler_helpers.c
>>>>>> ./gem/i915_gem_context.c
>>>>>> ./gt/intel_context.c
>>>>>> ./gt/intel_engine.h
>>>>>> ./gt/intel_engine_cs.c
>>>>>> ./gt/intel_engine_heartbeat.c
>>>>>> ./gt/intel_engine_pm.c
>>>>>> ./gt/intel_reset.c
>>>>>> ./gt/intel_lrc.c
>>>>>> ./gt/selftest_context.c
>>>>>> ./gt/selftest_engine_pm.c
>>>>>> ./gt/selftest_hangcheck.c
>>>>>> ./gt/selftest_mocs.c
>>>>>> ./gt/selftest_workarounds.c
>>>>>>
>>>>>> Sounds better to me than intel_guc_submission_is_wanted. What does 
>>>>>> the reader know whether "is wanted" translates to "is actually 
>>>>>> used". Shrug on "is wanted".
>>>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it 
>>>>> too early in the boot up sequence? I never understood why that was 
>>>>> necessary or why we need so many different ways to ask the same 
>>>>> question. But this version already exists and definitely works 
>>>>> without hitting any explosions.
>>>>
>>>> No idea if it causes a bug on, doesn't in the helper itself so maybe 
>>>> you are saying it is called too early? Might be.. I think over time 
>>>> the nice idea we had that "setup" and "init" phases of engine setup 
>>>> clearly separated got destroyed a bit. There would always be an 
>>>> option to move this clamping in a later phase, once the submission 
>>>> method is known. One could argue that if the submission method is 
>>>> not yet known at this point, it is even wrong to clamp based on 
>>>> something which will only be decided later. Because:
>>>>
>>>> int intel_engines_init(struct intel_gt *gt)
>>>> {
>>>>     int (*setup)(struct intel_engine_cs *engine);
>>>>     struct intel_engine_cs *engine;
>>>>     enum intel_engine_id id;
>>>>     int err;
>>>>
>>>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>>>
>>>> So this uses "uses", not "wanted". Presumably the point for having 
>>>> "wanted" and "uses" is that they can disagree, in which case if you 
>>>> clamp early based on "wanted" that suggests it could be wrong.
>>>
>>> Okay, looks like I was getting confused with intel_guc_is_used(). 
>>> That one blows up if called too early.
>>>
>>> I'll change it to _uses_ and repost, then.
>>
>> Check that it isn't called too early, before gt->submission_setup is set.
> Obviously it is because it blew up. But I am not re-writing the driver 
> start up sequence just to use the word 'use' instead of 'want'.
> 
>>
>>>
>>>>
>>>>>>>> And limit to class instead of applying to all engines looks like 
>>>>>>>> a miss.
>>>>>>> As per follow up email, the class limit is not applied here.
>>>>>>>
>>>>>>>>
>>>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>>>> +                 GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>>>> +            engine->props.timeslice_duration_ms = 
>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>>>
>>>>>>>> I am not sure logging such message during driver load is useful. 
>>>>>>>> Sounds more like a confused driver which starts with one value 
>>>>>>>> and then overrides itself. I'd just silently set the value 
>>>>>>>> appropriate for the active backend. Preemption timeout kconfig 
>>>>>>>> text already documents the fact timeouts can get overriden at 
>>>>>>>> runtime depending on platform+engine. So maybe just add same 
>>>>>>>> text to timeslice kconfig.
>>>>>>> The point is to make people aware if they compile with 
>>>>>>> unsupported config options. As far as I know, there is no way to 
>>>>>>> apply range checking or other limits to config defines. Which 
>>>>>>> means that a user would silently get unwanted behaviour. That 
>>>>>>> seems like a bad thing to me. If the driver is confused because 
>>>>>>> the user built it in a confused manner then we should let them know.
>>>>>>
>>>>>> Okay, but I think make it notice low level.
>>>>>>
>>>>>> Also consider in patch 3/3 when you triple it, and then clamp back 
>>>>>> down here. That's even more confused state since tripling gets 
>>>>>> nerfed. I think that's also an argument to always account preempt 
>>>>>> timeout in heartbeat interval calculation. Haven't got to your 
>>>>>> reply on 2/3 yet though..
>>>>> That sounds like even more reason to make sure the warning gets 
>>>>> seen. The more complex the system and the more chances there are to 
>>>>> get it wrong, the more important it is to have a nice easy to see 
>>>>> and understand notification that it did go wrong.
>>>>
>>>> I did not disagree, just said make it notice, one level higher than 
>>>> info! :)
>>> But then it won't appear unless you have explicitly said an elevated 
>>> debug level. Whereas info appears in dmesg by default (but is still 
>>> not classed as an error by CI and such).
>>
>> Notice is higher than info! :) If info appears by default so does 
>> notice, warning, err, etc...
> Doh! I could have sworn those were the other way around.
> 
> Okay. Will update to use notice :).
> 
>>
>> #define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
>> #define KERN_ALERT      KERN_SOH "1"    /* action must be taken 
>> immediately */
>> #define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
>> #define KERN_ERR        KERN_SOH "3"    /* error conditions */
>> #define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
>> #define KERN_NOTICE     KERN_SOH "5"    /* normal but significant 
>> condition */
>> #define KERN_INFO       KERN_SOH "6"    /* informational */
>> #define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */
>>
>>>> But also think how, if we agree to go with tripling, that you'd have 
>>>> to consider that in the sysfs store when hearbeat timeout is 
>>>> written, to consider whether or not to triple and error out if 
>>>> preemption timeout is over limit.
>>> I see this as just setting the default values. If an end user is 
>>> explicitly overriding the defaults then we should obey what they have 
>>> requested. If they are changing the heartbeat interval then they can 
>>> also change the pre-emption timeout appropriately.
>>
>> Question is can they unknowingly and without any feedback configure a 
>> much worse state than they expect? Like when they set heartbeats up to 
>> some value, everything is configured as you intended - but if you go 
>> over a certain hidden limit the overall scheme degrades in some way. 
>> What is the failure mode here if you silently let them do that?
> You can always configure things to be worse than expected. If you don't 
> understand what you are doing then any control can make things worse 
> instead of better. The assumption is that if a user is savvy enough to 
> be writing to sysfs overrides of kernel parameters then they know what 
> those parameters are and what their implications are. If they want to 
> set a very short heartbeat with a very long pre-emption timeout then its 
> their problem if they hit frequent TDRs. Conversely, if they want to set 
> a very long heartbeat with a very short pre-emption timeout then its 
> still their problem if they hit frequent TDRs.
> 
> But if the user explicitly requests a heartbeat period of 3s and a 
> pre-emption timeout of 2s and the i915 arbitrarily splats their 2s and 
> makes it 9s then that is wrong.
> 
> We should give the driver defaults that work for the majority of users 
> and then let the minority specify exactly what they need.
> 
> And there is no silent or hidden limit. If the user specifies a value 
> too large then they will get -EINVAL. Nothing hidden or silent about 
> that. Any other values are legal and the behaviour will be whatever has 
> been requested.
> 
> John.
> 
> 
>>
>> Regards,
>>
>> Tvrtko
> 

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 25/02/2022 18:48, John Harrison wrote:
> On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>>
>> I'll try to simplify the discussion here:
>>
>> On 24/02/2022 19:45, John Harrison wrote:
>>> On 2/24/2022 03:41, Tvrtko Ursulin wrote:
>>>> On 23/02/2022 20:00, John Harrison wrote:
>>>>> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>>>>>> On 23/02/2022 02:45, John Harrison wrote:
>>>>>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>
>>>>>>>>> Compute workloads are inherantly not pre-emptible for long 
>>>>>>>>> periods on
>>>>>>>>> current hardware. As a workaround for this, the pre-emption 
>>>>>>>>> timeout
>>>>>>>>> for compute capable engines was disabled. This is undesirable 
>>>>>>>>> with GuC
>>>>>>>>> submission as it prevents per engine reset of hung contexts. 
>>>>>>>>> Hence the
>>>>>>>>> next patch will re-enable the timeout but bumped up by an order of
>>>>>>>>> magnititude.
>>>>>>>>
>>>>>>>> (Some typos above.)
>>>>>>> I'm spotting 'inherently' but not anything else.
>>>>>>
>>>>>> Magnititude! O;)
>>>>> Doh!
>>>>>
>>>>> [snip]
>>>>>
>>>>>>> Whereas, bumping all heartbeat periods to be greater than the 
>>>>>>> pre-emption timeout is wasteful and unnecessary. That leads to a 
>>>>>>> total heartbeat time of about a minute. Which is a very long time 
>>>>>>> to wait for a hang to be detected and recovered. Especially when 
>>>>>>> the official limit on a context responding to an 'are you dead' 
>>>>>>> query is only 7.5 seconds.
>>>>>>
>>>>>> Not sure how did you get one minute?
>>>>> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 
>>>>> 75 => 1 minute 15 seconds
>>>>>
>>>>> Even ignoring any safety factor and just going with 7.5 * 5 still 
>>>>> gets you to 37.5 seconds which is over a half a minute and likely 
>>>>> to race.
>>>>
>>>> Ah because my starting point is there should be no preempt timeout = 
>>>> heartbeat * 3, I just think that's too ugly.
>>> Then complain at the hardware designers to give us mid-thread 
>>> pre-emption back. The heartbeat is only one source of pre-emption 
>>> events. For example, a user can be running multiple contexts in 
>>> parallel and expecting them to time slice on a single engine. Or 
>>> maybe a user is just running one compute task in the background but 
>>> is doing render work in the foreground. Etc.
>>>
>>> There was a reason the original hack was to disable pre-emption 
>>> rather than increase the heartbeat. This is simply a slightly less 
>>> ugly version of the same hack. And unfortunately, the basic idea of 
>>> the hack is non-negotiable.
>>>
>>> As per other comments, 'tP(RCS) = tH *3' or 'tP(RCS) = tP(default) * 
>>> 12' or 'tP(RCS) = 7500' are the available options. Given that the 
>>> heartbeat is the ever present hard limit, it seems most plausible to 
>>> base the hack on that. Any of the others works, though. Although I 
>>> think a explicit hardcoded value is the most ugly. I guess the other 
>>> option is to add CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE and default 
>>> that to 7500.
>>>
>>> Take your pick. But 640ms is not allowed.
>>>
>>>>
>>>>>> Regardless, crux of argument was to avoid GuC engine reset and 
>>>>>> heartbeat reset racing with each other, and to do that by 
>>>>>> considering the preempt timeout with the heartbeat interval. I was 
>>>>>> thinking about this scenario in this series:
>>>>>>
>>>>>> [Please use fixed width font and no line wrap to view.]
>>>>>>
>>>>>> A)
>>>>>>
>>>>>> tP = preempt timeout
>>>>>> tH = hearbeat interval
>>>>>>
>>>>>> tP = 3 * tH
>>>>>>
>>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>>
>>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- 
>>>>>> [2 * tH] ----> FULL RESET
>>>>>>                                                        |
>>>>>>                                                        \- 
>>>>>> preemption triggered, tP = 3 * tH ------\
>>>>>> \-> preempt timeout would hit here
>>>>>>
>>>>>> Here we have collateral damage due full reset, since we can't tell 
>>>>>> GuC to reset just one engine and we fudged tP just to "account" 
>>>>>> for heartbeats.
>>>>> You are missing the whole point of the patch series which is that 
>>>>> the last heartbeat period is '2 * tP' not '2 * tH'.
>>>>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>>>>>
>>>>> By making the last period double the pre-emption timeout, it is 
>>>>> guaranteed that the FULL RESET stage cannot be hit before the 
>>>>> hardware has attempted and timed-out on at least one pre-emption.
>>>>
>>>> Oh well :) that probably means the overall scheme is too odd for me. 
>>>> tp = 3tH and last pulse after 2tP I mean.
>>> To be accurate, it is 'tP(RCS) = 3 * tH(default); tH(final) = 
>>> tP(current) * 2;'. Seems fairly straight forward to me. It's not a 
>>> recursive definition or anything like that. It gives us a total 
>>> heartbeat timeout that is close to the original version but still 
>>> allows at least one pre-emption event.
>>>
>>>>
>>>>> [snip]
>>>>>
>>>>>
>>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 <---- 
>>>>>> [2 * tH] ----> full reset would be here
>>>>>>                |
>>>>>>                \- preemption triggered, tP = 3 * tH ----------------\
>>>>>> \-> Preempt timeout reset
>>>>>>
>>>>>> Here is is kind of least worse, but question is why we fudged tP 
>>>>>> when it gives us nothing good in this case.
>>>>>>
>>>>> The point of fudging tP(RCS) is to give compute workloads longer to 
>>>>> reach a pre-emptible point (given that EU walkers are basically not 
>>>>> pre-emptible). The reason for doing the fudge is not connected to 
>>>>> the heartbeat at all. The fact that it causes problems for the 
>>>>> heartbeat is an undesired side effect.
>>>>>
>>>>> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
>>>>> calculation that gave us something that all interested parties were 
>>>>> vaguely happy with. It could just as easily be a fixed, hard coded 
>>>>> value of 7.5s but having it based on something configurable seemed 
>>>>> more sensible. The other option was 'tP(RCS) = tP * 12' but that 
>>>>> felt more arbitrary than basing it on the average heartbeat 
>>>>> timeout. As in, three heartbeat periods is about what a normal prio 
>>>>> task gets before it gets pre-empted by the heartbeat. So using that 
>>>>> for general purpose pre-emptions (e.g. time slicing between 
>>>>> multiple user apps) seems reasonable.
>>>>
>>>> I think the fact you say tP fudge is not related to heartbeats and 
>>>> then go to mention heartbeat even in the "formula" which uses no tH 
>>>> is saying something (at least that's how I read the 7.5s option). :)
>>> I said the tP fudge is not because of the heartbeat. It is obviously 
>>> related.
>>>
>>> As per comment above, the fudge factor is based on the heartbeat 
>>> because the heartbeat is the ultimate limit. But the *reason* for the 
>>> fudge fact has nothing to do with the heartbeat. It is required even 
>>> if heartbeats are disabled.
>>>
>>>>
>>>>>> B)
>>>>>>
>>>>>> Instead, my idea to account for preempt timeout when calculating 
>>>>>> when to schedule next hearbeat would look like this:
>>>>>>
>>>>>> First of all tP can be left at a large value unrelated to tH. Lets 
>>>>>> say tP = 640ms. tH stays 2.5s.
>>>>> 640ms is not 'large'. The requirement is either zero (disabled) or 
>>>>> region of 7.5s. The 640ms figure is the default for non-compute 
>>>>> engines. Anything that can run EUs needs to be 'huge'.
>>>>>
>>>>>
>>>>>>
>>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>>
>>>>>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] 
>>>>>> --> Pulse3 <-- [tH + tP] --> full reset would be here
>>>>> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full 
>>>>> five periods is therefore 50s, which is practically a minute.
>>>>
>>>> No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.
>>> Not good enough. After 2.5s, we send a pulse. After a further 640ms 
>>> we perform an engine reset. That means your compute workload had only 
>>> 640ms after being told to pre-empt to reach a pre-emption point. That 
>>> won't work. It needs to be multiple seconds.
>>>
>>>>
>>>>> [snip]
>>>>>
>>>>>> Am I missing some requirement or you see another problem with this 
>>>>>> idea?
>>>>>>
>>>>>>>> On a related topic, if GuC engine resets stop working when 
>>>>>>>> preempt timeout is set to zero - I think we need to somehow let 
>>>>>>>> the user know if they try to tweak it via sysfs. Perhaps go as 
>>>>>>>> far as -EINVAL in GuC mode, if i915.reset has not explicitly 
>>>>>>>> disabled engine resets.
>>>>>>> Define 'stops working'. The definition of the sysfs interface is 
>>>>>>> that a value of zero disables pre-emption. If you don't have 
>>>>>>> pre-emption and your hang detection mechanism relies on 
>>>>>>> pre-emption then you don't have a hang detection mechanism 
>>>>>>> either. If the user really wants to allow 
>>>>>>
>>>>>> By stops working I meant that it stops working. :)
>>>>>>
>>>>>> With execlist one can disable preempt timeout and "stopped 
>>>>>> heartbeat" can still reset the stuck engine and so avoid 
>>>>>> collateral damage. With GuC it appears this is not possible. So I 
>>>>>> was thinking this is something worthy a log notice.
>>>>>>
>>>>>>> their context to run forever and never be pre-empted then that 
>>>>>>> means they also don't want it to be reset arbitrarily. Which 
>>>>>>> means they would also be disabling the heartbeat timer as well. 
>>>>>>> Indeed, this is what we 
>>>>>>
>>>>>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>>>>>> upstream but hearbeat is not and so hangcheck still works.
>>>>> The pre-emption disable in upstream is not a valid solution for 
>>>>> compute customers. It is a worst-of-all-worlds hack for general 
>>>>> usage. As noted already, any actual compute specific customer is 
>>>>> advised to disable all forms of reset and do their hang detection 
>>>>> manually. A slightly less worse hack for customers that are not 
>>>>> actually running long compute workloads (i.e. the vast majority of 
>>>>> end users) is to just use a long pre-emption timeout.
>>>>
>>>> If disabled preemption timeout is worst of all words and compute 
>>>> needs to disable heartbeat as well then why did we put it in? 
>>>> Perhaps it was not know at the time it would not be good enough. But 
>>>> anyway, do I read you correct that current thinking is it would be 
>>>> better to leave it at default 640ms?
>>> No. We cannot have the RCS default to 640ms.
>>>
>>> Note that there is a difference between 'general end user who might 
>>> run some compute' and 'compute focused customer'. The driver defaults 
>>> (disabled or 7500ms) are for the general user who gets the 
>>> out-of-the-box experience and expects to be able to run 'normal' 
>>> workloads without hitting problems. I.e. they expect hung tasks to 
>>> get reset in a timely manner and while they might run some AI or 
>>> other compute workloads, they are not a HPC datacenter. Whereas the 
>>> compute datacenter customer expects their workloads to run for 
>>> arbitrarily long times (minutes, hours, maybe even days) without 
>>> being arbitrarily killed. Those customers will be explicitly 
>>> configuring their datacenter server for that scenario and thus don't 
>>> care what the defaults are.
>>
>> Okay maybe I misunderstood what you were saying earlier about worst of 
>> all worlds and all. But tell me this, if preemption timeout on RCS is 
>> not directly related to hearbeats, but to some pessimistic expected 
>> user workloads, what is wrong with my scheme of calculating the next 
>> heartbeat pulse as tH + tP?
>>
>> We can leave tH as default 2.5s and tP you set for RCS to 12s if that 
>> is what you say is required. Or whatever long value really.
>>
>> Your only objection is that ends up with too long total time before 
>> reset? Or something else as well?
> An unnecessarily long total heartbeat timeout is the main objection. 
> (2.5 + 12) * 5 = 72.5 seconds. That is a massive change from the current 
> 12.5s.
> 
> If we are happy with that huge increase then fine. But I'm pretty sure 
> you are going to get a lot more bug reports about hung systems not 
> recovering. 10-20s is just about long enough for someone to wait before 
> leaning on the power button of their machine. Over a minute is not. That 
> kind of delay is going to cause support issues.

Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, chosen 
just so it is longer than tH * 3?

And how do you keep coming up with factor of five? Isn't it four periods 
before "heartbeat stopped"? (Prio normal, hearbeat, barrier and then reset.)

 From the point of view of user experience I agree reasonable 
responsiveness is needed before user "reaches for the power button".

In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.

Question of workloads.. what is the actual preempt timeout compute is 
happy with? And I don't mean compute setups with disabled hangcheck, 
which you say they want anyway, but if we target defaults for end users. 
Do we have some numbers on what they are likely to run?

What if we gave them a default of 2.5s? That would be 4 * (2.5s + 2.5s) 
= 20s worst case until reset, comparable to your proposal. Are there 
realistic workloads which are non-preemptable for 2.5s? I am having hard 
time imagining someone would run them on a general purpose desktop since 
it would mean frozen and unusable UI anyway.

Advantage still being in my mind that there would be no fudging of 
timeouts during driver load and heartbeat periods depending on priority. 
To me it feels more plausible to account for preempt timeout in 
heartbeat pulses that to calculate preempt timeout to be longer than 
hearbeat pulses, just to avoid races between the two.

>> It's long but it is correct in a way. Because we can't expect hearbeat 
>> to react quicker than the interval + preempt timeout (or timeslice for 
>> equal priority) + some scheduling latency.
>>
>> I conceptually disagree with the last hearbeat pulse being special. If 
>> the user concept is "after N heartbeats you are out" and you want to 
>> make it "after N-1 heartbeats plus 2 preemption periods you are out", 
>> where preemption period actually depends on heartbeat period, then 
>> that sounds really convoluted to me.
>>
>> And we don't know which of the pulses will trigger preemption since 
>> user priority we don't control. So low priority compute task gets 
>> reset after 5s, normal priority gets to run for 12s. Effectively 
>> making context priority a variable in hangcheck.
> It already is. That is no different. The pre-emption is not triggered 
> until the pulse is of equal or higher priority than the busy task. That 
> is the case no matter whether you are running GuC or execlist, whether 
> you have the original driver or an updated one.
> 
> And it doesn't matter which pulse triggers the pre-emption. All that 
> matters is that once a pre-emption is attempted, if the busy context 
> fails to relinquish the hardware within the pre-emption timeout limit 
> then it will be forcibly evicted.
> 
> 
>>
>>>>
>>>> If so, if we went with my proposal, would everyone be happy? If yes, 
>>>> isn't it a simpler scheme? No special casing when setting the 
>>>> preempt timeout, no special casing of the last heartbeat pulse. 
>>>> Works predictably regardless of the priority of the hypothetical 
>>>> non-preemptible workload.
>>> No, we have to have the increased pre-emption timeout. And that has 
>>> ripple effects of making very long heartbeats or risking races with 
>>> the heartbeat beating the per engine reset.
>>>
>>>
>>>>
>>>>>>> advise compute customers to do. It is then up to the user 
>>>>>>> themselves to spot a hang and to manually kill (Ctrl+C, kill ###, 
>>>>>>> etc.) their task. Killing the CPU task will automatically clear 
>>>>>>> up any GPU resources allocated to that task (excepting context 
>>>>>>> persistence, which is a) broken and b) something we also tell 
>>>>>>> compute customers to disable).
>>>>>>
>>>>>> What is broken with context persistence? I noticed one patch 
>>>>>> claiming to be fixing something in that area which looked suspect. 
>>>>>> Has it been established no userspace relies on it?
>>>>> One major issue is that it has hooks into the execlist scheduler 
>>>>> backend. I forget the exact details right now. The implementation 
>>>>> as a whole is incredibly complex and convoluted :(. But there's 
>>>>> stuff about what happens when you disable the heartbeat after 
>>>>> having closed a persistence context's handle (and thus made it 
>>>>> persisting). There's also things like it sends a super high 
>>>>> priority heartbeat pulse at the point of becoming persisting. That 
>>>>> plays havoc for platforms with dependent engines and/or compute 
>>>>> workloads. A context becomes persisting on RCS and results in your 
>>>>> unrealted CCS work being reset. It's a mess.
>>>>>
>>>>> The comment from Daniel Vetter is that persistence should have no 
>>>>> connection to the heartbeat at all. All of that dynamic behaviour 
>>>>> and complexity should just be removed.
>>>>
>>>> Dependent engines is definitely a topic on it's own, outside 
>>>> hearbeats, persistence and all.
>>> Except that it has implications for persistence which the current 
>>> driver does not take into account.
>>
>> Well current driver does not take RCS+CCS dependency into account so 
>> that should come first, or all in one package at least.
> Not sure what you are arguing for here? Simplifying persistence as 
> described will fix all the problems in one easy go. There is no point in 
> adding yet more hideously complex code to make one corner case work when 
> the real fix is to rip it all out.

It sounded like you were arguing there was an issue with context 
persistence and dependent engines so my reply was that if there are no 
dependent engines in upstream there is no issue with persistence. Once 
dependent engines are brought upstream then any reset caused by 
persistence can be looked at and determined if there is any conceptual 
difference between that and any random reset.

>>>> Otherwise there is definitely complexity in the execlists backend 
>>>> but I am not sure if logic persistence and heartbeats are so very 
>>>> connected. It does send a pulse when heartbeat interval is changed, 
>>>> which if going to zero, it will kick of closed contexts if it can:
>>>>
>>>> static struct intel_engine_cs *
>>>> __execlists_schedule_in(struct i915_request *rq)
>>>> {
>>>>     struct intel_engine_cs * const engine = rq->engine;
>>>>     struct intel_context * const ce = rq->context;
>>>>
>>>>     intel_context_get(ce);
>>>>
>>>>     if (unlikely(intel_context_is_closed(ce) &&
>>>>              !intel_engine_has_heartbeat(engine)))
>>>>         intel_context_set_banned(ce);
>>>>
>>>>     if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
>>>>         reset_active(rq, engine);
>>>>
>>>> Is this what you mean? The point of this is to make sure persistent 
>>>> context does not hog the engine forever if hangcheck has been disabled.
>>>>
>>>> Reminds me of my improvement to customer experience which never got 
>>>> in 
>>>> (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). Point 
>>>> of that one was to avoid engine reset (or worse) after user presses 
>>>> "Ctrl-C" if something takes just over 1ms to cleanly complete.
>>> The plan is that the persistent contexts would still get the default 
>>> grace period (pre-emption timeout at least) to finish but Ctrl+C will 
>>> kill it within a timely manner if it does not finish.
>>
>> Yes my patch does that. ;) Currently non-persistent is killed to 
>> quickly triggering pointless and alarming engine resets. Users 
>> reported this last year and I tried to fix it.
> Except that your patch is adding yet more complexity to an already 
> complex feature. The desire is to simplify the driver and make it more 
> maintainable not complicated it further.

It is fixing an user reported issue and is not really adding complexity 
apart from enabling us to distinguish the reason for revoking a request 
(ban vs soft close). I think we should care to fix user reported issues 
in a reasonable time frame. It's a patch from last summer or so..

>>>>> Persistence itself can stay. There are valid UMD use cases. It is 
>>>>> just massively over complicated and doesn't work in all corner 
>>>>> cases when not using execlist submission or on newer platforms. The 
>>>>> simplification that is planned is to allow contexts to persist 
>>>>> until the associated DRM master handle is closed. At that point, 
>>>>> all contexts associated with that DRM handle are killed. That is 
>>>>> what AMD and others apparently implement.
>>>>
>>>> Okay, that goes against one recent IGT patch which appeared to work 
>>>> around something by moving the position of _context_ close.
>>> No it does not. The context close is not the trigger. The trigger is 
>>
>> Well patch says:
>> """
>> The spin all test relied on context persistence unecessarily by trying
>> to destroy contexts while keeping spinners active.
>> The current implementation of context persistence in i915 can cause
>> failures with GuC enabled, and persistence is not needed for this test.
>>
>> Moving intel_ctx_destroy after igt_spin_end.
>> """
>>
>> Implying moving context close to after spin end fixes things for GuC, 
>> not fd close.
> That's because persistence is currently a big pile of poo and does not 
> work in all the corner cases. The correct solution is to leave the IGT 
> alone and just fix the implementation of persistence. However, the IGT 
> update to not use the broken feature is a trivial test change (two 
> lines?) whereas fixing the broken feature is a significant KMD re-work. 
> It needs to be done but no-one currently has the time to do it. But 
> trivially changing the test allows the test to work and test the 
> features it is meant to be testing (which is not persistence).

Clear as mud. If the statement is that persistence cannot simply be 
removed, then for sure it cannot be said that anything can be fixed or 
unblocked by allowing some test to pass with GuC, by making them avoid 
using persistence (and not even explicitly with a context param). It 
implies persistence does not work with the GuC, which is then in 
contradiction with the statement that we cannot just remove persistence. 
I truly have no idea what the argument is here.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 2/28/2022 08:11, Tvrtko Ursulin wrote:
> On 25/02/2022 17:39, John Harrison wrote:
>> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>>
>>> On 24/02/2022 19:19, John Harrison wrote:
>>>
>>> [snip]
>>>
>>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>>
>>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>>
>>>>>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>>>>>> timeslice_duration_ms;
>>>>>>>>>
>>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>>
>>>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>>>> because of tick conversion internally but because at backend 
>>>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or I 
>>>>>>>>> failed to find where it is handled.
>>>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>>>> allow overflows.
>>>>>>>
>>>>>>> Yes and no, this fixes it, but the first bug was not only due 
>>>>>>> GuC internal tick conversion. It was present ever since the u64 
>>>>>>> from i915 was shoved into u32 sent to GuC. So even if GuC used 
>>>>>>> the value without additional multiplication, bug was be there. 
>>>>>>> My point being when GuC backend was added timeout_ms values 
>>>>>>> should have been limited/clamped to U32_MAX. The tick discovery 
>>>>>>> is additional limit on top.
>>>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>>>> noticed until I actually tried using very long timeouts to debug 
>>>>>> a particular problem. Now that it is noticed, we need some method 
>>>>>> of range checking and this simple clamp solves all the truncation 
>>>>>> problems.
>>>>>
>>>>> Agreed in principle, just please mention in the commit message all 
>>>>> aspects of the problem.
>>>>>
>>>>> I think we can get away without a Fixes: tag since it requires 
>>>>> user fiddling to break things in unexpected ways.
>>>>>
>>>>> I would though put in a code a clamping which expresses both, 
>>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>>>> Perhaps internal ticks go away or anything and we are left with 
>>>>> plain 1:1 millisecond relationship.
>>>> Can certainly add a comment along the lines of "GuC API only takes 
>>>> a 32bit field but that is further reduced to GUC_LIMIT due to 
>>>> internal calculations which would otherwise overflow".
>>>>
>>>> But if the GuC limit is > u32 then, by definition, that means the 
>>>> GuC API has changed to take a u64 instead of a u32. So there will 
>>>> no u32 truncation any more. So I'm not seeing a need to explicitly 
>>>> test the integer size when the value check covers that.
>>>
>>> Hmm I was thinking if the internal conversion in the GuC fw changes 
>>> so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, then to be 
>>> extra safe by documenting in code there is the additional limit of 
>>> the data structure field. Say the field was changed to take some 
>>> unit larger than a millisecond. Then the check against the GuC MAX 
>>> limit define would not be enough, unless that would account both for 
>>> internal implementation and u32 in the protocol. Maybe that is 
>>> overdefensive but I don't see that it harms. 50-50, but it's do it 
>>> once and forget so I'd do it.
>> Huh?
>>
>> How can the limit be greater than a u32 if the interface only takes a 
>> u32? By definition the limit would be clamped to u32 size.
>>
>> If you mean that the GuC policy is in different units and those units 
>> might not overflow but ms units do, then actually that is already the 
>> case. The GuC works in us not ms. That's part of why the wrap around 
>> is so low, we have to multiply by 1000 before sending to GuC. 
>> However, that is actually irrelevant because the comparison is being 
>> done on the i915 side in i915's units. We have to scale the GuC limit 
>> to match what i915 is using. And the i915 side is u64 so if the 
>> scaling to i915 numbers overflows a u32 then who cares because that 
>> comparison can be done at 64 bits wide.
>>
>> If the units change then that is a backwards breaking API change that 
>> will require a manual driver code update. You can't just recompile 
>> with a new header and magically get an ms to us or ms to s conversion 
>> in your a = b assignment. The code will need to be changed to do the 
>> new unit conversion (note we already convert from ms to us, the GuC 
>> API is all expressed in us). And that code change will mean having to 
>> revisit any and all scaling, type conversions, etc. I.e. any 
>> pre-existing checks will not necessarily be valid and will need to be 
>> re-visted anyway. But as above, any scaling to GuC units has to be 
>> incorporated into the limit already because otherwise the limit would 
>> not fit in the GuC's own API.
>
> Yes I get that, I was just worried that u32 field in the protocol and 
> GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the source code 
> and then how to protect against forgetting to update both in sync.
>
> Like if the protocol was changed to take nanoseconds, and firmware 
> implementation changed to support the full range, but define 
> left/forgotten at 100s. That would then overflow u32.
Huh? If the API was updated to 'support the full range' then how can you 
get overflow by forgetting to update the limit? You could get 
unnecessary clamping, which hopefully would be noticed by whoever is 
testing the new API and/or whoever requested the change. But you can't 
get u32 overflow errors if all the code has been updated to u64.

John.

>
> Regards,
>
> Tvrtko
>
>> John.
>>
>>>
>>>>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>> ---
>>>>>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 
>>>>>>>>>> +++++++++++++++
>>>>>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 
>>>>>>>>>> ++++++++++++++
>>>>>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h | 9 +++++++++
>>>>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>>>>
>>>>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == 
>>>>>>>>>> RENDER_CLASS)
>>>>>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>>>>
>>>>>>>>> Hm "wanted".. There's been too much back and forth on the GuC 
>>>>>>>>> load options over the years to keep track.. 
>>>>>>>>> intel_engine_uses_guc work sounds like would work and read nicer.
>>>>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>>>>> existing one. If we want to rename it yet again then that would 
>>>>>>>> be a different patch set.
>>>>>>>
>>>>>>> $ grep intel_engine_uses_guc . -rl
>>>>>>> ./i915_perf.c
>>>>>>> ./i915_request.c
>>>>>>> ./selftests/intel_scheduler_helpers.c
>>>>>>> ./gem/i915_gem_context.c
>>>>>>> ./gt/intel_context.c
>>>>>>> ./gt/intel_engine.h
>>>>>>> ./gt/intel_engine_cs.c
>>>>>>> ./gt/intel_engine_heartbeat.c
>>>>>>> ./gt/intel_engine_pm.c
>>>>>>> ./gt/intel_reset.c
>>>>>>> ./gt/intel_lrc.c
>>>>>>> ./gt/selftest_context.c
>>>>>>> ./gt/selftest_engine_pm.c
>>>>>>> ./gt/selftest_hangcheck.c
>>>>>>> ./gt/selftest_mocs.c
>>>>>>> ./gt/selftest_workarounds.c
>>>>>>>
>>>>>>> Sounds better to me than intel_guc_submission_is_wanted. What 
>>>>>>> does the reader know whether "is wanted" translates to "is 
>>>>>>> actually used". Shrug on "is wanted".
>>>>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it 
>>>>>> too early in the boot up sequence? I never understood why that 
>>>>>> was necessary or why we need so many different ways to ask the 
>>>>>> same question. But this version already exists and definitely 
>>>>>> works without hitting any explosions.
>>>>>
>>>>> No idea if it causes a bug on, doesn't in the helper itself so 
>>>>> maybe you are saying it is called too early? Might be.. I think 
>>>>> over time the nice idea we had that "setup" and "init" phases of 
>>>>> engine setup clearly separated got destroyed a bit. There would 
>>>>> always be an option to move this clamping in a later phase, once 
>>>>> the submission method is known. One could argue that if the 
>>>>> submission method is not yet known at this point, it is even wrong 
>>>>> to clamp based on something which will only be decided later. 
>>>>> Because:
>>>>>
>>>>> int intel_engines_init(struct intel_gt *gt)
>>>>> {
>>>>>     int (*setup)(struct intel_engine_cs *engine);
>>>>>     struct intel_engine_cs *engine;
>>>>>     enum intel_engine_id id;
>>>>>     int err;
>>>>>
>>>>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>>>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>>>>
>>>>> So this uses "uses", not "wanted". Presumably the point for having 
>>>>> "wanted" and "uses" is that they can disagree, in which case if 
>>>>> you clamp early based on "wanted" that suggests it could be wrong.
>>>>
>>>> Okay, looks like I was getting confused with intel_guc_is_used(). 
>>>> That one blows up if called too early.
>>>>
>>>> I'll change it to _uses_ and repost, then.
>>>
>>> Check that it isn't called too early, before gt->submission_setup is 
>>> set.
>> Obviously it is because it blew up. But I am not re-writing the 
>> driver start up sequence just to use the word 'use' instead of 'want'.
>>
>>>
>>>>
>>>>>
>>>>>>>>> And limit to class instead of applying to all engines looks 
>>>>>>>>> like a miss.
>>>>>>>> As per follow up email, the class limit is not applied here.
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>>>>> + GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>>>>> + engine->props.timeslice_duration_ms = 
>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>>>>
>>>>>>>>> I am not sure logging such message during driver load is 
>>>>>>>>> useful. Sounds more like a confused driver which starts with 
>>>>>>>>> one value and then overrides itself. I'd just silently set the 
>>>>>>>>> value appropriate for the active backend. Preemption timeout 
>>>>>>>>> kconfig text already documents the fact timeouts can get 
>>>>>>>>> overriden at runtime depending on platform+engine. So maybe 
>>>>>>>>> just add same text to timeslice kconfig.
>>>>>>>> The point is to make people aware if they compile with 
>>>>>>>> unsupported config options. As far as I know, there is no way 
>>>>>>>> to apply range checking or other limits to config defines. 
>>>>>>>> Which means that a user would silently get unwanted behaviour. 
>>>>>>>> That seems like a bad thing to me. If the driver is confused 
>>>>>>>> because the user built it in a confused manner then we should 
>>>>>>>> let them know.
>>>>>>>
>>>>>>> Okay, but I think make it notice low level.
>>>>>>>
>>>>>>> Also consider in patch 3/3 when you triple it, and then clamp 
>>>>>>> back down here. That's even more confused state since tripling 
>>>>>>> gets nerfed. I think that's also an argument to always account 
>>>>>>> preempt timeout in heartbeat interval calculation. Haven't got 
>>>>>>> to your reply on 2/3 yet though..
>>>>>> That sounds like even more reason to make sure the warning gets 
>>>>>> seen. The more complex the system and the more chances there are 
>>>>>> to get it wrong, the more important it is to have a nice easy to 
>>>>>> see and understand notification that it did go wrong.
>>>>>
>>>>> I did not disagree, just said make it notice, one level higher 
>>>>> than info! :)
>>>> But then it won't appear unless you have explicitly said an 
>>>> elevated debug level. Whereas info appears in dmesg by default (but 
>>>> is still not classed as an error by CI and such).
>>>
>>> Notice is higher than info! :) If info appears by default so does 
>>> notice, warning, err, etc...
>> Doh! I could have sworn those were the other way around.
>>
>> Okay. Will update to use notice :).
>>
>>>
>>> #define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
>>> #define KERN_ALERT      KERN_SOH "1"    /* action must be taken 
>>> immediately */
>>> #define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
>>> #define KERN_ERR        KERN_SOH "3"    /* error conditions */
>>> #define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
>>> #define KERN_NOTICE     KERN_SOH "5"    /* normal but significant 
>>> condition */
>>> #define KERN_INFO       KERN_SOH "6"    /* informational */
>>> #define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */
>>>
>>>>> But also think how, if we agree to go with tripling, that you'd 
>>>>> have to consider that in the sysfs store when hearbeat timeout is 
>>>>> written, to consider whether or not to triple and error out if 
>>>>> preemption timeout is over limit.
>>>> I see this as just setting the default values. If an end user is 
>>>> explicitly overriding the defaults then we should obey what they 
>>>> have requested. If they are changing the heartbeat interval then 
>>>> they can also change the pre-emption timeout appropriately.
>>>
>>> Question is can they unknowingly and without any feedback configure 
>>> a much worse state than they expect? Like when they set heartbeats 
>>> up to some value, everything is configured as you intended - but if 
>>> you go over a certain hidden limit the overall scheme degrades in 
>>> some way. What is the failure mode here if you silently let them do 
>>> that?
>> You can always configure things to be worse than expected. If you 
>> don't understand what you are doing then any control can make things 
>> worse instead of better. The assumption is that if a user is savvy 
>> enough to be writing to sysfs overrides of kernel parameters then 
>> they know what those parameters are and what their implications are. 
>> If they want to set a very short heartbeat with a very long 
>> pre-emption timeout then its their problem if they hit frequent TDRs. 
>> Conversely, if they want to set a very long heartbeat with a very 
>> short pre-emption timeout then its still their problem if they hit 
>> frequent TDRs.
>>
>> But if the user explicitly requests a heartbeat period of 3s and a 
>> pre-emption timeout of 2s and the i915 arbitrarily splats their 2s 
>> and makes it 9s then that is wrong.
>>
>> We should give the driver defaults that work for the majority of 
>> users and then let the minority specify exactly what they need.
>>
>> And there is no silent or hidden limit. If the user specifies a value 
>> too large then they will get -EINVAL. Nothing hidden or silent about 
>> that. Any other values are legal and the behaviour will be whatever 
>> has been requested.
>>
>> John.
>>
>>
>>>
>>> Regards,
>>>
>>> Tvrtko
>>

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 2/28/2022 09:12, Tvrtko Ursulin wrote:
> On 25/02/2022 18:48, John Harrison wrote:
>> On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>>>
>>> I'll try to simplify the discussion here:
>>>
>>> On 24/02/2022 19:45, John Harrison wrote:
>>>> On 2/24/2022 03:41, Tvrtko Ursulin wrote:
>>>>> On 23/02/2022 20:00, John Harrison wrote:
>>>>>> On 2/23/2022 05:58, Tvrtko Ursulin wrote:
>>>>>>> On 23/02/2022 02:45, John Harrison wrote:
>>>>>>>> On 2/22/2022 03:19, Tvrtko Ursulin wrote:
>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>
>>>>>>>>>> Compute workloads are inherantly not pre-emptible for long 
>>>>>>>>>> periods on
>>>>>>>>>> current hardware. As a workaround for this, the pre-emption 
>>>>>>>>>> timeout
>>>>>>>>>> for compute capable engines was disabled. This is undesirable 
>>>>>>>>>> with GuC
>>>>>>>>>> submission as it prevents per engine reset of hung contexts. 
>>>>>>>>>> Hence the
>>>>>>>>>> next patch will re-enable the timeout but bumped up by an 
>>>>>>>>>> order of
>>>>>>>>>> magnititude.
>>>>>>>>>
>>>>>>>>> (Some typos above.)
>>>>>>>> I'm spotting 'inherently' but not anything else.
>>>>>>>
>>>>>>> Magnititude! O;)
>>>>>> Doh!
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>>> Whereas, bumping all heartbeat periods to be greater than the 
>>>>>>>> pre-emption timeout is wasteful and unnecessary. That leads to 
>>>>>>>> a total heartbeat time of about a minute. Which is a very long 
>>>>>>>> time to wait for a hang to be detected and recovered. 
>>>>>>>> Especially when the official limit on a context responding to 
>>>>>>>> an 'are you dead' query is only 7.5 seconds.
>>>>>>>
>>>>>>> Not sure how did you get one minute?
>>>>>> 7.5 * 2 (to be safe) = 15. 15 * 5 (number of heartbeat periods) = 
>>>>>> 75 => 1 minute 15 seconds
>>>>>>
>>>>>> Even ignoring any safety factor and just going with 7.5 * 5 still 
>>>>>> gets you to 37.5 seconds which is over a half a minute and likely 
>>>>>> to race.
>>>>>
>>>>> Ah because my starting point is there should be no preempt timeout 
>>>>> = heartbeat * 3, I just think that's too ugly.
>>>> Then complain at the hardware designers to give us mid-thread 
>>>> pre-emption back. The heartbeat is only one source of pre-emption 
>>>> events. For example, a user can be running multiple contexts in 
>>>> parallel and expecting them to time slice on a single engine. Or 
>>>> maybe a user is just running one compute task in the background but 
>>>> is doing render work in the foreground. Etc.
>>>>
>>>> There was a reason the original hack was to disable pre-emption 
>>>> rather than increase the heartbeat. This is simply a slightly less 
>>>> ugly version of the same hack. And unfortunately, the basic idea of 
>>>> the hack is non-negotiable.
>>>>
>>>> As per other comments, 'tP(RCS) = tH *3' or 'tP(RCS) = tP(default) 
>>>> * 12' or 'tP(RCS) = 7500' are the available options. Given that the 
>>>> heartbeat is the ever present hard limit, it seems most plausible 
>>>> to base the hack on that. Any of the others works, though. Although 
>>>> I think a explicit hardcoded value is the most ugly. I guess the 
>>>> other option is to add CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE and 
>>>> default that to 7500.
>>>>
>>>> Take your pick. But 640ms is not allowed.
>>>>
>>>>>
>>>>>>> Regardless, crux of argument was to avoid GuC engine reset and 
>>>>>>> heartbeat reset racing with each other, and to do that by 
>>>>>>> considering the preempt timeout with the heartbeat interval. I 
>>>>>>> was thinking about this scenario in this series:
>>>>>>>
>>>>>>> [Please use fixed width font and no line wrap to view.]
>>>>>>>
>>>>>>> A)
>>>>>>>
>>>>>>> tP = preempt timeout
>>>>>>> tH = hearbeat interval
>>>>>>>
>>>>>>> tP = 3 * tH
>>>>>>>
>>>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>>>
>>>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 
>>>>>>> <---- [2 * tH] ----> FULL RESET
>>>>>>> |
>>>>>>> \- preemption triggered, tP = 3 * tH ------\
>>>>>>> \-> preempt timeout would hit here
>>>>>>>
>>>>>>> Here we have collateral damage due full reset, since we can't 
>>>>>>> tell GuC to reset just one engine and we fudged tP just to 
>>>>>>> "account" for heartbeats.
>>>>>> You are missing the whole point of the patch series which is that 
>>>>>> the last heartbeat period is '2 * tP' not '2 * tH'.
>>>>>> +        longer = READ_ONCE(engine->props.preempt_timeout_ms) * 2;
>>>>>>
>>>>>> By making the last period double the pre-emption timeout, it is 
>>>>>> guaranteed that the FULL RESET stage cannot be hit before the 
>>>>>> hardware has attempted and timed-out on at least one pre-emption.
>>>>>
>>>>> Oh well :) that probably means the overall scheme is too odd for 
>>>>> me. tp = 3tH and last pulse after 2tP I mean.
>>>> To be accurate, it is 'tP(RCS) = 3 * tH(default); tH(final) = 
>>>> tP(current) * 2;'. Seems fairly straight forward to me. It's not a 
>>>> recursive definition or anything like that. It gives us a total 
>>>> heartbeat timeout that is close to the original version but still 
>>>> allows at least one pre-emption event.
>>>>
>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>
>>>>>>> <-- [tH] --> Pulse1 <-- [tH] --> Pulse2 <-- [tH] --> Pulse3 
>>>>>>> <---- [2 * tH] ----> full reset would be here
>>>>>>>                |
>>>>>>>                \- preemption triggered, tP = 3 * tH 
>>>>>>> ----------------\
>>>>>>> \-> Preempt timeout reset
>>>>>>>
>>>>>>> Here is is kind of least worse, but question is why we fudged tP 
>>>>>>> when it gives us nothing good in this case.
>>>>>>>
>>>>>> The point of fudging tP(RCS) is to give compute workloads longer 
>>>>>> to reach a pre-emptible point (given that EU walkers are 
>>>>>> basically not pre-emptible). The reason for doing the fudge is 
>>>>>> not connected to the heartbeat at all. The fact that it causes 
>>>>>> problems for the heartbeat is an undesired side effect.
>>>>>>
>>>>>> Note that the use of 'tP(RCS) = tH * 3' was just an arbitrary 
>>>>>> calculation that gave us something that all interested parties 
>>>>>> were vaguely happy with. It could just as easily be a fixed, hard 
>>>>>> coded value of 7.5s but having it based on something configurable 
>>>>>> seemed more sensible. The other option was 'tP(RCS) = tP * 12' 
>>>>>> but that felt more arbitrary than basing it on the average 
>>>>>> heartbeat timeout. As in, three heartbeat periods is about what a 
>>>>>> normal prio task gets before it gets pre-empted by the heartbeat. 
>>>>>> So using that for general purpose pre-emptions (e.g. time slicing 
>>>>>> between multiple user apps) seems reasonable.
>>>>>
>>>>> I think the fact you say tP fudge is not related to heartbeats and 
>>>>> then go to mention heartbeat even in the "formula" which uses no 
>>>>> tH is saying something (at least that's how I read the 7.5s 
>>>>> option). :)
>>>> I said the tP fudge is not because of the heartbeat. It is 
>>>> obviously related.
>>>>
>>>> As per comment above, the fudge factor is based on the heartbeat 
>>>> because the heartbeat is the ultimate limit. But the *reason* for 
>>>> the fudge fact has nothing to do with the heartbeat. It is required 
>>>> even if heartbeats are disabled.
>>>>
>>>>>
>>>>>>> B)
>>>>>>>
>>>>>>> Instead, my idea to account for preempt timeout when calculating 
>>>>>>> when to schedule next hearbeat would look like this:
>>>>>>>
>>>>>>> First of all tP can be left at a large value unrelated to tH. 
>>>>>>> Lets say tP = 640ms. tH stays 2.5s.
>>>>>> 640ms is not 'large'. The requirement is either zero (disabled) 
>>>>>> or region of 7.5s. The 640ms figure is the default for 
>>>>>> non-compute engines. Anything that can run EUs needs to be 'huge'.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 1) Background load = I915_PRIORITY_DISPLAY
>>>>>>>
>>>>>>> <-- [tH + tP] --> Pulse1 <-- [tH + tP] --> Pulse2 <-- [tH + tP] 
>>>>>>> --> Pulse3 <-- [tH + tP] --> full reset would be here
>>>>>> Sure, this works but each period is now 2.5 + 7.5 = 10s. The full 
>>>>>> five periods is therefore 50s, which is practically a minute.
>>>>>
>>>>> No, in my proposal it is 3 * (2.5s + 640ms) =~ 9.3s.
>>>> Not good enough. After 2.5s, we send a pulse. After a further 640ms 
>>>> we perform an engine reset. That means your compute workload had 
>>>> only 640ms after being told to pre-empt to reach a pre-emption 
>>>> point. That won't work. It needs to be multiple seconds.
>>>>
>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>> Am I missing some requirement or you see another problem with 
>>>>>>> this idea?
>>>>>>>
>>>>>>>>> On a related topic, if GuC engine resets stop working when 
>>>>>>>>> preempt timeout is set to zero - I think we need to somehow 
>>>>>>>>> let the user know if they try to tweak it via sysfs. Perhaps 
>>>>>>>>> go as far as -EINVAL in GuC mode, if i915.reset has not 
>>>>>>>>> explicitly disabled engine resets.
>>>>>>>> Define 'stops working'. The definition of the sysfs interface 
>>>>>>>> is that a value of zero disables pre-emption. If you don't have 
>>>>>>>> pre-emption and your hang detection mechanism relies on 
>>>>>>>> pre-emption then you don't have a hang detection mechanism 
>>>>>>>> either. If the user really wants to allow 
>>>>>>>
>>>>>>> By stops working I meant that it stops working. :)
>>>>>>>
>>>>>>> With execlist one can disable preempt timeout and "stopped 
>>>>>>> heartbeat" can still reset the stuck engine and so avoid 
>>>>>>> collateral damage. With GuC it appears this is not possible. So 
>>>>>>> I was thinking this is something worthy a log notice.
>>>>>>>
>>>>>>>> their context to run forever and never be pre-empted then that 
>>>>>>>> means they also don't want it to be reset arbitrarily. Which 
>>>>>>>> means they would also be disabling the heartbeat timer as well. 
>>>>>>>> Indeed, this is what we 
>>>>>>>
>>>>>>> I don't think so. Preempt timeout is disabled already on TGL/RCS 
>>>>>>> upstream but hearbeat is not and so hangcheck still works.
>>>>>> The pre-emption disable in upstream is not a valid solution for 
>>>>>> compute customers. It is a worst-of-all-worlds hack for general 
>>>>>> usage. As noted already, any actual compute specific customer is 
>>>>>> advised to disable all forms of reset and do their hang detection 
>>>>>> manually. A slightly less worse hack for customers that are not 
>>>>>> actually running long compute workloads (i.e. the vast majority 
>>>>>> of end users) is to just use a long pre-emption timeout.
>>>>>
>>>>> If disabled preemption timeout is worst of all words and compute 
>>>>> needs to disable heartbeat as well then why did we put it in? 
>>>>> Perhaps it was not know at the time it would not be good enough. 
>>>>> But anyway, do I read you correct that current thinking is it 
>>>>> would be better to leave it at default 640ms?
>>>> No. We cannot have the RCS default to 640ms.
>>>>
>>>> Note that there is a difference between 'general end user who might 
>>>> run some compute' and 'compute focused customer'. The driver 
>>>> defaults (disabled or 7500ms) are for the general user who gets the 
>>>> out-of-the-box experience and expects to be able to run 'normal' 
>>>> workloads without hitting problems. I.e. they expect hung tasks to 
>>>> get reset in a timely manner and while they might run some AI or 
>>>> other compute workloads, they are not a HPC datacenter. Whereas the 
>>>> compute datacenter customer expects their workloads to run for 
>>>> arbitrarily long times (minutes, hours, maybe even days) without 
>>>> being arbitrarily killed. Those customers will be explicitly 
>>>> configuring their datacenter server for that scenario and thus 
>>>> don't care what the defaults are.
>>>
>>> Okay maybe I misunderstood what you were saying earlier about worst 
>>> of all worlds and all. But tell me this, if preemption timeout on 
>>> RCS is not directly related to hearbeats, but to some pessimistic 
>>> expected user workloads, what is wrong with my scheme of calculating 
>>> the next heartbeat pulse as tH + tP?
>>>
>>> We can leave tH as default 2.5s and tP you set for RCS to 12s if 
>>> that is what you say is required. Or whatever long value really.
>>>
>>> Your only objection is that ends up with too long total time before 
>>> reset? Or something else as well?
>> An unnecessarily long total heartbeat timeout is the main objection. 
>> (2.5 + 12) * 5 = 72.5 seconds. That is a massive change from the 
>> current 12.5s.
>>
>> If we are happy with that huge increase then fine. But I'm pretty 
>> sure you are going to get a lot more bug reports about hung systems 
>> not recovering. 10-20s is just about long enough for someone to wait 
>> before leaning on the power button of their machine. Over a minute is 
>> not. That kind of delay is going to cause support issues.
>
> Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, chosen 
> just so it is longer than tH * 3?
>
> And how do you keep coming up with factor of five? Isn't it four 
> periods before "heartbeat stopped"? (Prio normal, hearbeat, barrier 
> and then reset.)
Prio starts at low not normal.

>
> From the point of view of user experience I agree reasonable 
> responsiveness is needed before user "reaches for the power button".
>
> In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.
>
> Question of workloads.. what is the actual preempt timeout compute is 
> happy with? And I don't mean compute setups with disabled hangcheck, 
> which you say they want anyway, but if we target defaults for end 
> users. Do we have some numbers on what they are likely to run?
Not that I have ever seen. This is all just finger in the air stuff. I 
don't recall if we invented the number and the compute people agreed 
with it or if they proposed the number to us.

>
> What if we gave them a default of 2.5s? That would be 4 * (2.5s + 
> 2.5s) = 20s worst case until reset, comparable to your proposal. Are 
> there realistic workloads which are non-preemptable for 2.5s? I am 
> having hard time imagining someone would run them on a general purpose 
> desktop since it would mean frozen and unusable UI anyway.
>
> Advantage still being in my mind that there would be no fudging of 
> timeouts during driver load and heartbeat periods depending on 
> priority. To me it feels more plausible to account for preempt timeout 
> in heartbeat pulses that to calculate preempt timeout to be longer 
> than hearbeat pulses, just to avoid races between the two.
Except that when the user asks for a heartbeat period of 2.5s you are 
actually setting it to 5s. How is that not a major fudge that is totally 
disregarding the user's request?


>
>>> It's long but it is correct in a way. Because we can't expect 
>>> hearbeat to react quicker than the interval + preempt timeout (or 
>>> timeslice for equal priority) + some scheduling latency.
>>>
>>> I conceptually disagree with the last hearbeat pulse being special. 
>>> If the user concept is "after N heartbeats you are out" and you want 
>>> to make it "after N-1 heartbeats plus 2 preemption periods you are 
>>> out", where preemption period actually depends on heartbeat period, 
>>> then that sounds really convoluted to me.
>>>
>>> And we don't know which of the pulses will trigger preemption since 
>>> user priority we don't control. So low priority compute task gets 
>>> reset after 5s, normal priority gets to run for 12s. Effectively 
>>> making context priority a variable in hangcheck.
>> It already is. That is no different. The pre-emption is not triggered 
>> until the pulse is of equal or higher priority than the busy task. 
>> That is the case no matter whether you are running GuC or execlist, 
>> whether you have the original driver or an updated one.
>>
>> And it doesn't matter which pulse triggers the pre-emption. All that 
>> matters is that once a pre-emption is attempted, if the busy context 
>> fails to relinquish the hardware within the pre-emption timeout limit 
>> then it will be forcibly evicted.
>>
>>
>>>
>>>>>
>>>>> If so, if we went with my proposal, would everyone be happy? If 
>>>>> yes, isn't it a simpler scheme? No special casing when setting the 
>>>>> preempt timeout, no special casing of the last heartbeat pulse. 
>>>>> Works predictably regardless of the priority of the hypothetical 
>>>>> non-preemptible workload.
>>>> No, we have to have the increased pre-emption timeout. And that has 
>>>> ripple effects of making very long heartbeats or risking races with 
>>>> the heartbeat beating the per engine reset.
>>>>
>>>>
>>>>>
>>>>>>>> advise compute customers to do. It is then up to the user 
>>>>>>>> themselves to spot a hang and to manually kill (Ctrl+C, kill 
>>>>>>>> ###, etc.) their task. Killing the CPU task will automatically 
>>>>>>>> clear up any GPU resources allocated to that task (excepting 
>>>>>>>> context persistence, which is a) broken and b) something we 
>>>>>>>> also tell compute customers to disable).
>>>>>>>
>>>>>>> What is broken with context persistence? I noticed one patch 
>>>>>>> claiming to be fixing something in that area which looked 
>>>>>>> suspect. Has it been established no userspace relies on it?
>>>>>> One major issue is that it has hooks into the execlist scheduler 
>>>>>> backend. I forget the exact details right now. The implementation 
>>>>>> as a whole is incredibly complex and convoluted :(. But there's 
>>>>>> stuff about what happens when you disable the heartbeat after 
>>>>>> having closed a persistence context's handle (and thus made it 
>>>>>> persisting). There's also things like it sends a super high 
>>>>>> priority heartbeat pulse at the point of becoming persisting. 
>>>>>> That plays havoc for platforms with dependent engines and/or 
>>>>>> compute workloads. A context becomes persisting on RCS and 
>>>>>> results in your unrealted CCS work being reset. It's a mess.
>>>>>>
>>>>>> The comment from Daniel Vetter is that persistence should have no 
>>>>>> connection to the heartbeat at all. All of that dynamic behaviour 
>>>>>> and complexity should just be removed.
>>>>>
>>>>> Dependent engines is definitely a topic on it's own, outside 
>>>>> hearbeats, persistence and all.
>>>> Except that it has implications for persistence which the current 
>>>> driver does not take into account.
>>>
>>> Well current driver does not take RCS+CCS dependency into account so 
>>> that should come first, or all in one package at least.
>> Not sure what you are arguing for here? Simplifying persistence as 
>> described will fix all the problems in one easy go. There is no point 
>> in adding yet more hideously complex code to make one corner case 
>> work when the real fix is to rip it all out.
>
> It sounded like you were arguing there was an issue with context 
> persistence and dependent engines so my reply was that if there are no 
> dependent engines in upstream there is no issue with persistence. Once 
> dependent engines are brought upstream then any reset caused by 
> persistence can be looked at and determined if there is any conceptual 
> difference between that and any random reset.
There are issues with context persistence and *everything*. Dependent 
engines is just one of may problems with persistence.

What I am arguing for is ripping out all the hideously complex 
persistence code that it is extremely fragile and does not work with 
modern platforms. Instead, replacing it with some massively simpler, 
much more maintainable and compatible with all platforms - past, present 
and future.


>
>>>>> Otherwise there is definitely complexity in the execlists backend 
>>>>> but I am not sure if logic persistence and heartbeats are so very 
>>>>> connected. It does send a pulse when heartbeat interval is 
>>>>> changed, which if going to zero, it will kick of closed contexts 
>>>>> if it can:
>>>>>
>>>>> static struct intel_engine_cs *
>>>>> __execlists_schedule_in(struct i915_request *rq)
>>>>> {
>>>>>     struct intel_engine_cs * const engine = rq->engine;
>>>>>     struct intel_context * const ce = rq->context;
>>>>>
>>>>>     intel_context_get(ce);
>>>>>
>>>>>     if (unlikely(intel_context_is_closed(ce) &&
>>>>>              !intel_engine_has_heartbeat(engine)))
>>>>>         intel_context_set_banned(ce);
>>>>>
>>>>>     if (unlikely(intel_context_is_banned(ce) || bad_request(rq)))
>>>>>         reset_active(rq, engine);
>>>>>
>>>>> Is this what you mean? The point of this is to make sure 
>>>>> persistent context does not hog the engine forever if hangcheck 
>>>>> has been disabled.
>>>>>
>>>>> Reminds me of my improvement to customer experience which never 
>>>>> got in 
>>>>> (https://patchwork.freedesktop.org/patch/451491/?series=93420&rev=2). 
>>>>> Point of that one was to avoid engine reset (or worse) after user 
>>>>> presses "Ctrl-C" if something takes just over 1ms to cleanly 
>>>>> complete.
>>>> The plan is that the persistent contexts would still get the 
>>>> default grace period (pre-emption timeout at least) to finish but 
>>>> Ctrl+C will kill it within a timely manner if it does not finish.
>>>
>>> Yes my patch does that. ;) Currently non-persistent is killed to 
>>> quickly triggering pointless and alarming engine resets. Users 
>>> reported this last year and I tried to fix it.
>> Except that your patch is adding yet more complexity to an already 
>> complex feature. The desire is to simplify the driver and make it 
>> more maintainable not complicated it further.
>
> It is fixing an user reported issue and is not really adding 
> complexity apart from enabling us to distinguish the reason for 
> revoking a request (ban vs soft close). I think we should care to fix 
> user reported issues in a reasonable time frame. It's a patch from 
> last summer or so..
>
>>>>>> Persistence itself can stay. There are valid UMD use cases. It is 
>>>>>> just massively over complicated and doesn't work in all corner 
>>>>>> cases when not using execlist submission or on newer platforms. 
>>>>>> The simplification that is planned is to allow contexts to 
>>>>>> persist until the associated DRM master handle is closed. At that 
>>>>>> point, all contexts associated with that DRM handle are killed. 
>>>>>> That is what AMD and others apparently implement.
>>>>>
>>>>> Okay, that goes against one recent IGT patch which appeared to 
>>>>> work around something by moving the position of _context_ close.
>>>> No it does not. The context close is not the trigger. The trigger is 
>>>
>>> Well patch says:
>>> """
>>> The spin all test relied on context persistence unecessarily by trying
>>> to destroy contexts while keeping spinners active.
>>> The current implementation of context persistence in i915 can cause
>>> failures with GuC enabled, and persistence is not needed for this test.
>>>
>>> Moving intel_ctx_destroy after igt_spin_end.
>>> """
>>>
>>> Implying moving context close to after spin end fixes things for 
>>> GuC, not fd close.
>> That's because persistence is currently a big pile of poo and does 
>> not work in all the corner cases. The correct solution is to leave 
>> the IGT alone and just fix the implementation of persistence. 
>> However, the IGT update to not use the broken feature is a trivial 
>> test change (two lines?) whereas fixing the broken feature is a 
>> significant KMD re-work. It needs to be done but no-one currently has 
>> the time to do it. But trivially changing the test allows the test to 
>> work and test the features it is meant to be testing (which is not 
>> persistence).
>
> Clear as mud. If the statement is that persistence cannot simply be 
> removed, then for sure it cannot be said that anything can be fixed or 
> unblocked by allowing some test to pass with GuC, by making them avoid 
> using persistence (and not even explicitly with a context param). It 
> implies persistence does not work with the GuC, which is then in 
> contradiction with the statement that we cannot just remove 
> persistence. I truly have no idea what the argument is here.
Persistence works in the right set of circumstances. Those circumstances 
do not involve dynamically changing heartbeat settings, platforms with 
dependent engines, etc. The correct fix is to leave the IGT test alone 
and fix the persistence implementation. However, that is not trivial and 
we have many other high priority holes still to plug. Whereas changing 
the IGT to not use a feature it is not intended to be testing anyway is 
a trivial change and gets us the test coverage of what that IGT is meant 
to be for.

John.


>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 2/28/2022 07:32, Tvrtko Ursulin wrote:
> On 25/02/2022 19:03, John Harrison wrote:
>> On 2/25/2022 10:29, Tvrtko Ursulin wrote:
>>> On 25/02/2022 18:01, John Harrison wrote:
>>>> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>>>>> On 25/02/2022 17:11, John Harrison wrote:
>>>>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>>>
>>>>>>>>>>>> Compute workloads are inherently not pre-emptible on 
>>>>>>>>>>>> current hardware.
>>>>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround 
>>>>>>>>>>>> to prevent
>>>>>>>>>>>> unwanted resets. Instead, the hang detection was left to 
>>>>>>>>>>>> the heartbeat
>>>>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>>>>> submission as
>>>>>>>>>>>> the heartbeat is a full GT reset rather than a per engine 
>>>>>>>>>>>> reset and so
>>>>>>>>>>>> is much more destructive. Instead, just bump the 
>>>>>>>>>>>> pre-emption timeout
>>>>>>>>>>>
>>>>>>>>>>> Can we have a feature request to allow asking GuC for an 
>>>>>>>>>>> engine reset?
>>>>>>>>>> For what purpose?
>>>>>>>>>
>>>>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>>>>
>>>>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>>>>> context might be executing on. Even without virtual engines, 
>>>>>>>>>> the KMD still has no knowledge of which context is currently 
>>>>>>>>>> executing on any given engine at any given time.
>>>>>>>>>>
>>>>>>>>>> There is a reason why hang detection should be left to the 
>>>>>>>>>> entity that is doing the scheduling. Any other entity is 
>>>>>>>>>> second guessing at best.
>>>>>>>>>>
>>>>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>>>>> submission is enabled is for the case where the KMD/GuC have 
>>>>>>>>>> got out of sync with either other somehow or GuC itself has 
>>>>>>>>>> just crashed. I.e. when no submission at all is working and 
>>>>>>>>>> we need to reset the GuC itself and start over.
>>>>>>>>>
>>>>>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>>>>>> nerfed already in GuC mode.
>>>>>>>> Not sure what you mean by that claim. Engine resets are handled 
>>>>>>>> by GuC because GuC handles the scheduling. You can't do the 
>>>>>>>> former if you aren't doing the latter. However, the heartbeat 
>>>>>>>> is still present and is still the watchdog by which engine 
>>>>>>>> resets are triggered. As per the rest of the submission 
>>>>>>>> process, the hang detection and recovery is split between i915 
>>>>>>>> and GuC.
>>>>>>>
>>>>>>> I meant that "stopped heartbeat on engine XXX" can only do a 
>>>>>>> full GPU reset on GuC.
>>>>>> I mean that there is no 'stopped heartbeat on engine XXX' when 
>>>>>> i915 is not handling the recovery part of the process.
>>>>>
>>>>> Hmmmm?
>>>>>
>>>>> static void
>>>>> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
>>>>> {
>>>>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>>>>         show_heartbeat(rq, engine);
>>>>>
>>>>>     if (intel_engine_uses_guc(engine))
>>>>>         /*
>>>>>          * GuC itself is toast or GuC's hang detection
>>>>>          * is disabled. Either way, need to find the
>>>>>          * hang culprit manually.
>>>>>          */
>>>>>         intel_guc_find_hung_context(engine);
>>>>>
>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>                   I915_ERROR_CAPTURE,
>>>>>                   "stopped heartbeat on %s",
>>>>>                   engine->name);
>>>>> }
>>>>>
>>>>> How there is no "stopped hearbeat" in guc mode? From this code it 
>>>>> certainly looks there is.
>>>> Only when the GuC is toast and it is no longer an engine reset but 
>>>> a full GT reset that is required. So technically, it is not a 
>>>> 'stopped heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
>>>>
>>>>>
>>>>> You say below heartbeats are going in GuC mode. Now I totally 
>>>>> don't understand how they are going but there is allegedly no 
>>>>> "stopped hearbeat".
>>>> Because if GuC is handling the detection and recovery then i915 
>>>> will not reach that point. GuC will do the engine reset and start 
>>>> scheduling the next context before the heartbeat period expires. So 
>>>> the notification will be a G2H about a specific context being reset 
>>>> rather than the i915 notification about a stopped heartbeat.
>>>>
>>>>>
>>>>>>>
>>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>>                   "stopped heartbeat on %s",
>>>>>>>                   engine->name);
>>>>>>>
>>>>>>> intel_gt_handle_error:
>>>>>>>
>>>>>>>     /*
>>>>>>>      * Try engine reset when available. We fall back to full 
>>>>>>> reset if
>>>>>>>      * single reset fails.
>>>>>>>      */
>>>>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>>>>         local_bh_disable();
>>>>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>>>>
>>>>>>> You said "However, the heartbeat is still present and is still 
>>>>>>> the watchdog by which engine resets are triggered", now I don't 
>>>>>>> know what you meant by this. It actually triggers a single 
>>>>>>> engine reset in GuC mode? Where in code does that happen if this 
>>>>>>> block above shows it not taking the engine reset path?
>>>>>> i915 sends down the per engine pulse.
>>>>>> GuC schedules the pulse
>>>>>> GuC attempts to pre-empt the currently active context
>>>>>> GuC detects the pre-emption timeout
>>>>>> GuC resets the engine
>>>>>>
>>>>>> The fundamental process is exactly the same as in execlist mode. 
>>>>>> It's just that the above blocks of code (calls to 
>>>>>> intel_gt_handle_error and such) are now inside the GuC not i915.
>>>>>>
>>>>>> Without the heartbeat going ping, there would be no context 
>>>>>> switching and thus no pre-emption, no pre-emption timeout and so 
>>>>>> no hang and reset recovery. And GuC cannot sent pulses by itself 
>>>>>> - it has no ability to generate context workloads. So we need 
>>>>>> i915 to send the pings and to gradually raise their priority. But 
>>>>>> the back half of the heartbeat code is now inside the GuC. It 
>>>>>> will simply never reach the i915 side timeout if GuC is doing the 
>>>>>> recovery (unless the heartbeat's final period is too short). We 
>>>>>> should only reach the i915 side timeout if GuC itself is toast. 
>>>>>> At which point we need the full GT reset to recover the GuC.
>>>>>
>>>>> If workload is not preempting and reset does not work, like engine 
>>>>> is truly stuck, does the current code hit "stopped heartbeat" or 
>>>>> not in GuC mode?
>>>> Hang on, where did 'reset does not work' come into this?
>>>>
>>>> If GuC is alive and the hardware is not broken then no, it won't. 
>>>> That's the whole point. GuC does the detection and recovery. The 
>>>> KMD will never reach 'stopped heartbeat'.
>>>>
>>>> If the hardware is broken and the reset does not work then GuC will 
>>>> send a 'failed reset' notification to the KMD. The KMD treats that 
>>>> as a major error and immediately does a full GT reset. So there is 
>>>> still no 'stopped heartbeat'.
>>>>
>>>> If GuC has died (or a KMD bug has caused sufficient confusion to 
>>>> make it think the GuC has died) then yes, you will reach that code. 
>>>> But in that case it is not an engine reset that is required, it is 
>>>> a full GT reset including a reset of the GuC.
>>>
>>> Got it, so what is actually wrong is calling intel_gt_handle_error 
>>> with an engine->mask in GuC mode. 
>>> intel_engine_hearbeat.c/reset_engine should fork into two (in some 
>>> way), depending on backend, so in the case of GuC it can call a 
>>> variant of intel_gt_handle_error which would be explicitly about a 
>>> full GPU reset only, instead of a sprinkle of 
>>> intel_uc_uses_guc_submission in that function. Possibly even off 
>>> load the reset to a single per gt worker, so that if multiple active 
>>> engines trigger the reset roughly simultaneously, there is only one 
>>> full GPU reset. And it gets correctly labeled as "dead GuC" or 
>>> something.
>>>
>> Sure. Feel free to re-write the reset code yet again. That's another 
>> exceedingly fragile area of the driver.
>
>> However, that is unrelated to this patch set.
>
> It's still okay to talk about improving things in my opinion. 
> Especially since I think it is the same issue I raised late 2019 
> during internal review.
>
> And I don't think it is fair to say that I should go and rewrite it, 
> since I do not own the GuC backend area. Especially given the above.
>
> If there is no motivation to improve it now then please at least track 
> this, if it isn't already, in that internal Jira which was tracking 
> all the areas of the GuC backend which could be improved.
>
> I am also pretty sure if the design was cleaner it would have been 
> more obvious to me, or anyone who happens to stumble there, what the 
> purpose of intel_engine_heartbeat.c/reset_engine() is in GuC mode. So 
> we wouldn't have to spend this long explaining things.
My point is that redesigning it to be cleaner is not just a GuC task. It 
means redesigning the entire reset sequence to more compatible with 
externally handled resets. That's a big task. Sure it can be added to 
the todo list but it is totally not in the scope of this patch set.

This is purely about enabling per engine resets again so that end users 
have a better experience when GPU hangs occur. Or at least, don't have a 
greatly worse experience on our newest platforms than they did on all 
the previous ones because we have totally hacked that feature out. And 
actually getting that feature enabled before we ship too many products 
and the maintainers/architects decide we are no longer allowed to turn 
it on because it is now a behaviour change that end users are not 
expecting. So forever more ADL-P/DG2 are stuck on full GT only resets.

John.


>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 28/02/2022 18:32, John Harrison wrote:
> On 2/28/2022 08:11, Tvrtko Ursulin wrote:
>> On 25/02/2022 17:39, John Harrison wrote:
>>> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>>>
>>>> On 24/02/2022 19:19, John Harrison wrote:
>>>>
>>>> [snip]
>>>>
>>>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>>>
>>>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>>>
>>>>>>>>>> ./gt/intel_engine_types.h:              unsigned long 
>>>>>>>>>> timeslice_duration_ms;
>>>>>>>>>>
>>>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>>>
>>>>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>>>>> because of tick conversion internally but because at backend 
>>>>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or I 
>>>>>>>>>> failed to find where it is handled.
>>>>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>>>>> allow overflows.
>>>>>>>>
>>>>>>>> Yes and no, this fixes it, but the first bug was not only due 
>>>>>>>> GuC internal tick conversion. It was present ever since the u64 
>>>>>>>> from i915 was shoved into u32 sent to GuC. So even if GuC used 
>>>>>>>> the value without additional multiplication, bug was be there. 
>>>>>>>> My point being when GuC backend was added timeout_ms values 
>>>>>>>> should have been limited/clamped to U32_MAX. The tick discovery 
>>>>>>>> is additional limit on top.
>>>>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>>>>> noticed until I actually tried using very long timeouts to debug 
>>>>>>> a particular problem. Now that it is noticed, we need some method 
>>>>>>> of range checking and this simple clamp solves all the truncation 
>>>>>>> problems.
>>>>>>
>>>>>> Agreed in principle, just please mention in the commit message all 
>>>>>> aspects of the problem.
>>>>>>
>>>>>> I think we can get away without a Fixes: tag since it requires 
>>>>>> user fiddling to break things in unexpected ways.
>>>>>>
>>>>>> I would though put in a code a clamping which expresses both, 
>>>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>>>>> Perhaps internal ticks go away or anything and we are left with 
>>>>>> plain 1:1 millisecond relationship.
>>>>> Can certainly add a comment along the lines of "GuC API only takes 
>>>>> a 32bit field but that is further reduced to GUC_LIMIT due to 
>>>>> internal calculations which would otherwise overflow".
>>>>>
>>>>> But if the GuC limit is > u32 then, by definition, that means the 
>>>>> GuC API has changed to take a u64 instead of a u32. So there will 
>>>>> no u32 truncation any more. So I'm not seeing a need to explicitly 
>>>>> test the integer size when the value check covers that.
>>>>
>>>> Hmm I was thinking if the internal conversion in the GuC fw changes 
>>>> so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, then to be 
>>>> extra safe by documenting in code there is the additional limit of 
>>>> the data structure field. Say the field was changed to take some 
>>>> unit larger than a millisecond. Then the check against the GuC MAX 
>>>> limit define would not be enough, unless that would account both for 
>>>> internal implementation and u32 in the protocol. Maybe that is 
>>>> overdefensive but I don't see that it harms. 50-50, but it's do it 
>>>> once and forget so I'd do it.
>>> Huh?
>>>
>>> How can the limit be greater than a u32 if the interface only takes a 
>>> u32? By definition the limit would be clamped to u32 size.
>>>
>>> If you mean that the GuC policy is in different units and those units 
>>> might not overflow but ms units do, then actually that is already the 
>>> case. The GuC works in us not ms. That's part of why the wrap around 
>>> is so low, we have to multiply by 1000 before sending to GuC. 
>>> However, that is actually irrelevant because the comparison is being 
>>> done on the i915 side in i915's units. We have to scale the GuC limit 
>>> to match what i915 is using. And the i915 side is u64 so if the 
>>> scaling to i915 numbers overflows a u32 then who cares because that 
>>> comparison can be done at 64 bits wide.
>>>
>>> If the units change then that is a backwards breaking API change that 
>>> will require a manual driver code update. You can't just recompile 
>>> with a new header and magically get an ms to us or ms to s conversion 
>>> in your a = b assignment. The code will need to be changed to do the 
>>> new unit conversion (note we already convert from ms to us, the GuC 
>>> API is all expressed in us). And that code change will mean having to 
>>> revisit any and all scaling, type conversions, etc. I.e. any 
>>> pre-existing checks will not necessarily be valid and will need to be 
>>> re-visted anyway. But as above, any scaling to GuC units has to be 
>>> incorporated into the limit already because otherwise the limit would 
>>> not fit in the GuC's own API.
>>
>> Yes I get that, I was just worried that u32 field in the protocol and 
>> GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the source code 
>> and then how to protect against forgetting to update both in sync.
>>
>> Like if the protocol was changed to take nanoseconds, and firmware 
>> implementation changed to support the full range, but define 
>> left/forgotten at 100s. That would then overflow u32.
> Huh? If the API was updated to 'support the full range' then how can you 
> get overflow by forgetting to update the limit? You could get 
> unnecessary clamping, which hopefully would be noticed by whoever is 
> testing the new API and/or whoever requested the change. But you can't 
> get u32 overflow errors if all the code has been updated to u64.

1)
Change the protocol so that "u32 desc->execution_quantum" now takes nano seconds.

This now makes the maximum time 4.29.. seconds.

2)
Forget to update GUC_POLICY_MAX_EXEC_QUANTUM_MS from 100s, since for instance that part at that point still not part of the interface contract.

3)
User passes in 5 seconds.

Clamping check says all is good.

"engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS"

4)

Assignment was updated:

gt/uc/intel_guc_submission.c:

   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;

But someone did not realize field is u32.

   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;

Defensive solution:

   if (overflows_type(engine->props.timeslice_duration_ms * 1e6, desc->execution_quantum))
	drm_WARN_ON...

   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;

Regards,

Tvrtko

  
> John.
> 
>>
>> Regards,
>>
>> Tvrtko
>>
>>> John.
>>>
>>>>
>>>>>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>> ---
>>>>>>>>>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c   | 15 
>>>>>>>>>>> +++++++++++++++
>>>>>>>>>>>   drivers/gpu/drm/i915/gt/sysfs_engines.c     | 14 
>>>>>>>>>>> ++++++++++++++
>>>>>>>>>>>   drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h | 9 +++++++++
>>>>>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == 
>>>>>>>>>>> RENDER_CLASS)
>>>>>>>>>>>           engine->props.preempt_timeout_ms = 0;
>>>>>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>>>>>
>>>>>>>>>> Hm "wanted".. There's been too much back and forth on the GuC 
>>>>>>>>>> load options over the years to keep track.. 
>>>>>>>>>> intel_engine_uses_guc work sounds like would work and read nicer.
>>>>>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>>>>>> existing one. If we want to rename it yet again then that would 
>>>>>>>>> be a different patch set.
>>>>>>>>
>>>>>>>> $ grep intel_engine_uses_guc . -rl
>>>>>>>> ./i915_perf.c
>>>>>>>> ./i915_request.c
>>>>>>>> ./selftests/intel_scheduler_helpers.c
>>>>>>>> ./gem/i915_gem_context.c
>>>>>>>> ./gt/intel_context.c
>>>>>>>> ./gt/intel_engine.h
>>>>>>>> ./gt/intel_engine_cs.c
>>>>>>>> ./gt/intel_engine_heartbeat.c
>>>>>>>> ./gt/intel_engine_pm.c
>>>>>>>> ./gt/intel_reset.c
>>>>>>>> ./gt/intel_lrc.c
>>>>>>>> ./gt/selftest_context.c
>>>>>>>> ./gt/selftest_engine_pm.c
>>>>>>>> ./gt/selftest_hangcheck.c
>>>>>>>> ./gt/selftest_mocs.c
>>>>>>>> ./gt/selftest_workarounds.c
>>>>>>>>
>>>>>>>> Sounds better to me than intel_guc_submission_is_wanted. What 
>>>>>>>> does the reader know whether "is wanted" translates to "is 
>>>>>>>> actually used". Shrug on "is wanted".
>>>>>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call it 
>>>>>>> too early in the boot up sequence? I never understood why that 
>>>>>>> was necessary or why we need so many different ways to ask the 
>>>>>>> same question. But this version already exists and definitely 
>>>>>>> works without hitting any explosions.
>>>>>>
>>>>>> No idea if it causes a bug on, doesn't in the helper itself so 
>>>>>> maybe you are saying it is called too early? Might be.. I think 
>>>>>> over time the nice idea we had that "setup" and "init" phases of 
>>>>>> engine setup clearly separated got destroyed a bit. There would 
>>>>>> always be an option to move this clamping in a later phase, once 
>>>>>> the submission method is known. One could argue that if the 
>>>>>> submission method is not yet known at this point, it is even wrong 
>>>>>> to clamp based on something which will only be decided later. 
>>>>>> Because:
>>>>>>
>>>>>> int intel_engines_init(struct intel_gt *gt)
>>>>>> {
>>>>>>     int (*setup)(struct intel_engine_cs *engine);
>>>>>>     struct intel_engine_cs *engine;
>>>>>>     enum intel_engine_id id;
>>>>>>     int err;
>>>>>>
>>>>>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>>>>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>>>>>
>>>>>> So this uses "uses", not "wanted". Presumably the point for having 
>>>>>> "wanted" and "uses" is that they can disagree, in which case if 
>>>>>> you clamp early based on "wanted" that suggests it could be wrong.
>>>>>
>>>>> Okay, looks like I was getting confused with intel_guc_is_used(). 
>>>>> That one blows up if called too early.
>>>>>
>>>>> I'll change it to _uses_ and repost, then.
>>>>
>>>> Check that it isn't called too early, before gt->submission_setup is 
>>>> set.
>>> Obviously it is because it blew up. But I am not re-writing the 
>>> driver start up sequence just to use the word 'use' instead of 'want'.
>>>
>>>>
>>>>>
>>>>>>
>>>>>>>>>> And limit to class instead of applying to all engines looks 
>>>>>>>>>> like a miss.
>>>>>>>>> As per follow up email, the class limit is not applied here.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>>>>>> + GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>>>>>> + engine->props.timeslice_duration_ms = 
>>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>>>>>
>>>>>>>>>> I am not sure logging such message during driver load is 
>>>>>>>>>> useful. Sounds more like a confused driver which starts with 
>>>>>>>>>> one value and then overrides itself. I'd just silently set the 
>>>>>>>>>> value appropriate for the active backend. Preemption timeout 
>>>>>>>>>> kconfig text already documents the fact timeouts can get 
>>>>>>>>>> overriden at runtime depending on platform+engine. So maybe 
>>>>>>>>>> just add same text to timeslice kconfig.
>>>>>>>>> The point is to make people aware if they compile with 
>>>>>>>>> unsupported config options. As far as I know, there is no way 
>>>>>>>>> to apply range checking or other limits to config defines. 
>>>>>>>>> Which means that a user would silently get unwanted behaviour. 
>>>>>>>>> That seems like a bad thing to me. If the driver is confused 
>>>>>>>>> because the user built it in a confused manner then we should 
>>>>>>>>> let them know.
>>>>>>>>
>>>>>>>> Okay, but I think make it notice low level.
>>>>>>>>
>>>>>>>> Also consider in patch 3/3 when you triple it, and then clamp 
>>>>>>>> back down here. That's even more confused state since tripling 
>>>>>>>> gets nerfed. I think that's also an argument to always account 
>>>>>>>> preempt timeout in heartbeat interval calculation. Haven't got 
>>>>>>>> to your reply on 2/3 yet though..
>>>>>>> That sounds like even more reason to make sure the warning gets 
>>>>>>> seen. The more complex the system and the more chances there are 
>>>>>>> to get it wrong, the more important it is to have a nice easy to 
>>>>>>> see and understand notification that it did go wrong.
>>>>>>
>>>>>> I did not disagree, just said make it notice, one level higher 
>>>>>> than info! :)
>>>>> But then it won't appear unless you have explicitly said an 
>>>>> elevated debug level. Whereas info appears in dmesg by default (but 
>>>>> is still not classed as an error by CI and such).
>>>>
>>>> Notice is higher than info! :) If info appears by default so does 
>>>> notice, warning, err, etc...
>>> Doh! I could have sworn those were the other way around.
>>>
>>> Okay. Will update to use notice :).
>>>
>>>>
>>>> #define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
>>>> #define KERN_ALERT      KERN_SOH "1"    /* action must be taken 
>>>> immediately */
>>>> #define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
>>>> #define KERN_ERR        KERN_SOH "3"    /* error conditions */
>>>> #define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
>>>> #define KERN_NOTICE     KERN_SOH "5"    /* normal but significant 
>>>> condition */
>>>> #define KERN_INFO       KERN_SOH "6"    /* informational */
>>>> #define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */
>>>>
>>>>>> But also think how, if we agree to go with tripling, that you'd 
>>>>>> have to consider that in the sysfs store when hearbeat timeout is 
>>>>>> written, to consider whether or not to triple and error out if 
>>>>>> preemption timeout is over limit.
>>>>> I see this as just setting the default values. If an end user is 
>>>>> explicitly overriding the defaults then we should obey what they 
>>>>> have requested. If they are changing the heartbeat interval then 
>>>>> they can also change the pre-emption timeout appropriately.
>>>>
>>>> Question is can they unknowingly and without any feedback configure 
>>>> a much worse state than they expect? Like when they set heartbeats 
>>>> up to some value, everything is configured as you intended - but if 
>>>> you go over a certain hidden limit the overall scheme degrades in 
>>>> some way. What is the failure mode here if you silently let them do 
>>>> that?
>>> You can always configure things to be worse than expected. If you 
>>> don't understand what you are doing then any control can make things 
>>> worse instead of better. The assumption is that if a user is savvy 
>>> enough to be writing to sysfs overrides of kernel parameters then 
>>> they know what those parameters are and what their implications are. 
>>> If they want to set a very short heartbeat with a very long 
>>> pre-emption timeout then its their problem if they hit frequent TDRs. 
>>> Conversely, if they want to set a very long heartbeat with a very 
>>> short pre-emption timeout then its still their problem if they hit 
>>> frequent TDRs.
>>>
>>> But if the user explicitly requests a heartbeat period of 3s and a 
>>> pre-emption timeout of 2s and the i915 arbitrarily splats their 2s 
>>> and makes it 9s then that is wrong.
>>>
>>> We should give the driver defaults that work for the majority of 
>>> users and then let the minority specify exactly what they need.
>>>
>>> And there is no silent or hidden limit. If the user specifies a value 
>>> too large then they will get -EINVAL. Nothing hidden or silent about 
>>> that. Any other values are legal and the behaviour will be whatever 
>>> has been requested.
>>>
>>> John.
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>> Tvrtko
>>>
> 

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

I'll trim it a bit again..

On 28/02/2022 18:55, John Harrison wrote:
> On 2/28/2022 09:12, Tvrtko Ursulin wrote:
>> On 25/02/2022 18:48, John Harrison wrote:
>>> On 2/25/2022 10:14, Tvrtko Ursulin wrote:

[snip]

>>>> Your only objection is that ends up with too long total time before 
>>>> reset? Or something else as well?
>>> An unnecessarily long total heartbeat timeout is the main objection. 
>>> (2.5 + 12) * 5 = 72.5 seconds. That is a massive change from the 
>>> current 12.5s.
>>>
>>> If we are happy with that huge increase then fine. But I'm pretty 
>>> sure you are going to get a lot more bug reports about hung systems 
>>> not recovering. 10-20s is just about long enough for someone to wait 
>>> before leaning on the power button of their machine. Over a minute is 
>>> not. That kind of delay is going to cause support issues.
>>
>> Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, chosen 
>> just so it is longer than tH * 3?
>>
>> And how do you keep coming up with factor of five? Isn't it four 
>> periods before "heartbeat stopped"? (Prio normal, hearbeat, barrier 
>> and then reset.)
> Prio starts at low not normal.

Right, slipped my mind since I only keep seeing that one priority ladder 
block in intel_engine_heartbeat.c/heartbeat()..

>> From the point of view of user experience I agree reasonable 
>> responsiveness is needed before user "reaches for the power button".
>>
>> In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.
>>
>> Question of workloads.. what is the actual preempt timeout compute is 
>> happy with? And I don't mean compute setups with disabled hangcheck, 
>> which you say they want anyway, but if we target defaults for end 
>> users. Do we have some numbers on what they are likely to run?
> Not that I have ever seen. This is all just finger in the air stuff. I 
> don't recall if we invented the number and the compute people agreed 
> with it or if they proposed the number to us.

Yeah me neither. And found nothing in my email archives. :(

Thinking about it today I don't see that disabled timeout is a practical 
default.

With it, if users have something un-preemptable to run (assuming prio 
normal), it would get killed after ~13s (5 * 2.5).

If we go for my scheme it gets killed in ~17.5s (3 * (2.5 + 2.5) + 2.5 
(third pulse triggers preempt timeout)).

And if we go for your scheme it gets killed in ~22.5s (4 * 2.5 + 2 * 3 * 
2.5).

If I did not confuse any calculation this time round, then the 
differences for default case for normal priority sound pretty immaterial 
to me.

>> What if we gave them a default of 2.5s? That would be 4 * (2.5s + 
>> 2.5s) = 20s worst case until reset, comparable to your proposal. Are 
>> there realistic workloads which are non-preemptable for 2.5s? I am 
>> having hard time imagining someone would run them on a general purpose 
>> desktop since it would mean frozen and unusable UI anyway.
>>
>> Advantage still being in my mind that there would be no fudging of 
>> timeouts during driver load and heartbeat periods depending on 
>> priority. To me it feels more plausible to account for preempt timeout 
>> in heartbeat pulses that to calculate preempt timeout to be longer 
>> than hearbeat pulses, just to avoid races between the two.
> Except that when the user asks for a heartbeat period of 2.5s you are 
> actually setting it to 5s. How is that not a major fudge that is totally 
> disregarding the user's request?

This is indeed the core question. My thinking:

It is not defined in the heartbeat ABI that N pulses should do anything, 
just that they are periodic pulses which check the health of an engine.

If we view user priority as not under our control then we can say that 
any heartbeat pulse can trigger preempt timeout and we should let it do 
that.

 From that it follows that it is justified to account for preempt 
timeout in the decision when to schedule heartbeat pulses and that it is 
legitimate to do it for all of them.

It also avoids the double reset problem, regardless of the backend and 
regardless of how the user configured the timeouts. Without the need to 
fudge them neither during driver load or during sysfs store.

User has configured that heartbeat pulses should be sent every N 
seconds, yes, but I think we are free to account for inherent hardware 
and software latencies in our calculations. Especially since other than 
flawed Gen12 RCS, other engines will be much closer to the configured 
period.

It is just the same as user asking for preempt timeout N and we say on 
driver load "oh no you won't get it". Same for heartbeats, they said 
2.5s, we said 2.5s + broken engine factor...

I don't see a problem there. Nothing timing sensitive relies on the 
heartbeat interval nor we provided any guarantees.

That patch from Chris for instance AFAIR accounted for scheduling or 
context switch latencies. Because what is the point of sending further 
elevated priority pulses if we did not leave enough time to the engine 
to schedule them in, react with preemption, or signalling completion?

>>>>>>> Persistence itself can stay. There are valid UMD use cases. It is 
>>>>>>> just massively over complicated and doesn't work in all corner 
>>>>>>> cases when not using execlist submission or on newer platforms. 
>>>>>>> The simplification that is planned is to allow contexts to 
>>>>>>> persist until the associated DRM master handle is closed. At that 
>>>>>>> point, all contexts associated with that DRM handle are killed. 
>>>>>>> That is what AMD and others apparently implement.
>>>>>>
>>>>>> Okay, that goes against one recent IGT patch which appeared to 
>>>>>> work around something by moving the position of _context_ close.
>>>>> No it does not. The context close is not the trigger. The trigger is 
>>>>
>>>> Well patch says:
>>>> """
>>>> The spin all test relied on context persistence unecessarily by trying
>>>> to destroy contexts while keeping spinners active.
>>>> The current implementation of context persistence in i915 can cause
>>>> failures with GuC enabled, and persistence is not needed for this test.
>>>>
>>>> Moving intel_ctx_destroy after igt_spin_end.
>>>> """
>>>>
>>>> Implying moving context close to after spin end fixes things for 
>>>> GuC, not fd close.
>>> That's because persistence is currently a big pile of poo and does 
>>> not work in all the corner cases. The correct solution is to leave 
>>> the IGT alone and just fix the implementation of persistence. 
>>> However, the IGT update to not use the broken feature is a trivial 
>>> test change (two lines?) whereas fixing the broken feature is a 
>>> significant KMD re-work. It needs to be done but no-one currently has 
>>> the time to do it. But trivially changing the test allows the test to 
>>> work and test the features it is meant to be testing (which is not 
>>> persistence).
>>
>> Clear as mud. If the statement is that persistence cannot simply be 
>> removed, then for sure it cannot be said that anything can be fixed or 
>> unblocked by allowing some test to pass with GuC, by making them avoid 
>> using persistence (and not even explicitly with a context param). It 
>> implies persistence does not work with the GuC, which is then in 
>> contradiction with the statement that we cannot just remove 
>> persistence. I truly have no idea what the argument is here.
> Persistence works in the right set of circumstances. Those circumstances 
> do not involve dynamically changing heartbeat settings, platforms with 
> dependent engines, etc. The correct fix is to leave the IGT test alone 
> and fix the persistence implementation. However, that is not trivial and 
> we have many other high priority holes still to plug. Whereas changing 
> the IGT to not use a feature it is not intended to be testing anyway is 
> a trivial change and gets us the test coverage of what that IGT is meant 
> to be for.

It may be acceptable if someone is reviewing overall coverage and making 
sure all is not removed and so a missing ABI in GuC backend not swept 
under the carpet. That's my main concern. If it is acknowledged 
persistence is a needed ABI, then how can we upstream dependent engine 
support without making sure this ABI is respected? Removing it's use 
from random tests does not fill me with confidence that we are on top of 
this topic.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 3/1/2022 02:50, Tvrtko Ursulin wrote:
> On 28/02/2022 18:32, John Harrison wrote:
>> On 2/28/2022 08:11, Tvrtko Ursulin wrote:
>>> On 25/02/2022 17:39, John Harrison wrote:
>>>> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>>>>
>>>>> On 24/02/2022 19:19, John Harrison wrote:
>>>>>
>>>>> [snip]
>>>>>
>>>>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>>>>
>>>>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>>>>
>>>>>>>>>>> ./gt/intel_engine_types.h: unsigned long timeslice_duration_ms;
>>>>>>>>>>>
>>>>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>>>>
>>>>>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>>>>>> because of tick conversion internally but because at backend 
>>>>>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or 
>>>>>>>>>>> I failed to find where it is handled.
>>>>>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>>>>>> allow overflows.
>>>>>>>>>
>>>>>>>>> Yes and no, this fixes it, but the first bug was not only due 
>>>>>>>>> GuC internal tick conversion. It was present ever since the 
>>>>>>>>> u64 from i915 was shoved into u32 sent to GuC. So even if GuC 
>>>>>>>>> used the value without additional multiplication, bug was be 
>>>>>>>>> there. My point being when GuC backend was added timeout_ms 
>>>>>>>>> values should have been limited/clamped to U32_MAX. The tick 
>>>>>>>>> discovery is additional limit on top.
>>>>>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>>>>>> noticed until I actually tried using very long timeouts to 
>>>>>>>> debug a particular problem. Now that it is noticed, we need 
>>>>>>>> some method of range checking and this simple clamp solves all 
>>>>>>>> the truncation problems.
>>>>>>>
>>>>>>> Agreed in principle, just please mention in the commit message 
>>>>>>> all aspects of the problem.
>>>>>>>
>>>>>>> I think we can get away without a Fixes: tag since it requires 
>>>>>>> user fiddling to break things in unexpected ways.
>>>>>>>
>>>>>>> I would though put in a code a clamping which expresses both, 
>>>>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>>>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>>>>>> Perhaps internal ticks go away or anything and we are left with 
>>>>>>> plain 1:1 millisecond relationship.
>>>>>> Can certainly add a comment along the lines of "GuC API only 
>>>>>> takes a 32bit field but that is further reduced to GUC_LIMIT due 
>>>>>> to internal calculations which would otherwise overflow".
>>>>>>
>>>>>> But if the GuC limit is > u32 then, by definition, that means the 
>>>>>> GuC API has changed to take a u64 instead of a u32. So there will 
>>>>>> no u32 truncation any more. So I'm not seeing a need to 
>>>>>> explicitly test the integer size when the value check covers that.
>>>>>
>>>>> Hmm I was thinking if the internal conversion in the GuC fw 
>>>>> changes so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, 
>>>>> then to be extra safe by documenting in code there is the 
>>>>> additional limit of the data structure field. Say the field was 
>>>>> changed to take some unit larger than a millisecond. Then the 
>>>>> check against the GuC MAX limit define would not be enough, unless 
>>>>> that would account both for internal implementation and u32 in the 
>>>>> protocol. Maybe that is overdefensive but I don't see that it 
>>>>> harms. 50-50, but it's do it once and forget so I'd do it.
>>>> Huh?
>>>>
>>>> How can the limit be greater than a u32 if the interface only takes 
>>>> a u32? By definition the limit would be clamped to u32 size.
>>>>
>>>> If you mean that the GuC policy is in different units and those 
>>>> units might not overflow but ms units do, then actually that is 
>>>> already the case. The GuC works in us not ms. That's part of why 
>>>> the wrap around is so low, we have to multiply by 1000 before 
>>>> sending to GuC. However, that is actually irrelevant because the 
>>>> comparison is being done on the i915 side in i915's units. We have 
>>>> to scale the GuC limit to match what i915 is using. And the i915 
>>>> side is u64 so if the scaling to i915 numbers overflows a u32 then 
>>>> who cares because that comparison can be done at 64 bits wide.
>>>>
>>>> If the units change then that is a backwards breaking API change 
>>>> that will require a manual driver code update. You can't just 
>>>> recompile with a new header and magically get an ms to us or ms to 
>>>> s conversion in your a = b assignment. The code will need to be 
>>>> changed to do the new unit conversion (note we already convert from 
>>>> ms to us, the GuC API is all expressed in us). And that code change 
>>>> will mean having to revisit any and all scaling, type conversions, 
>>>> etc. I.e. any pre-existing checks will not necessarily be valid and 
>>>> will need to be re-visted anyway. But as above, any scaling to GuC 
>>>> units has to be incorporated into the limit already because 
>>>> otherwise the limit would not fit in the GuC's own API.
>>>
>>> Yes I get that, I was just worried that u32 field in the protocol 
>>> and GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the 
>>> source code and then how to protect against forgetting to update 
>>> both in sync.
>>>
>>> Like if the protocol was changed to take nanoseconds, and firmware 
>>> implementation changed to support the full range, but define 
>>> left/forgotten at 100s. That would then overflow u32.
>> Huh? If the API was updated to 'support the full range' then how can 
>> you get overflow by forgetting to update the limit? You could get 
>> unnecessary clamping, which hopefully would be noticed by whoever is 
>> testing the new API and/or whoever requested the change. But you 
>> can't get u32 overflow errors if all the code has been updated to u64.
>
> 1)
> Change the protocol so that "u32 desc->execution_quantum" now takes 
> nano seconds.
>
> This now makes the maximum time 4.29.. seconds.
You seriously think this is likely to happen?

That the GuC people would force an API change on us that is completely 
backwards from what we have been asking? And that such a massive 
backwards step would not get implemented correctly because someone 
didn't notice just how huge an impact it was?

>
> 2)
> Forget to update GUC_POLICY_MAX_EXEC_QUANTUM_MS from 100s, since for 
> instance that part at that point still not part of the interface 
> contract.
There is zero chance of the us -> ns change occurring in the foreseeable 
future whereas the expectation is to have the limits be part of the spec 
in the next firmware release. So this scenario is just not going to 
happen. And as above, it would be such a big change with such a huge 
amount of push back and discussion going on that it would be impossible 
for the limit update to be missed/forgotten.

>
> 3)
> User passes in 5 seconds.
>
> Clamping check says all is good.
>
> "engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS"
>
> 4)
>
> Assignment was updated:
>
> gt/uc/intel_guc_submission.c:
>
>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>
> But someone did not realize field is u32.
>
>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>
> Defensive solution:
>
>   if (overflows_type(engine->props.timeslice_duration_ms * 1e6, 
> desc->execution_quantum))
>     drm_WARN_ON...

All you are saying is that bugs can happen. The above is just one more 
place to have a bug.

The purpose of the limit is to take into account all reasons for there 
being a limit. Having a bunch of different tests that are all testing 
the same thing is pointless.

John.

>
>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>
> Regards,
>
> Tvrtko
>
>
>> John.
>>
>>>
>>> Regards,
>>>
>>> Tvrtko
>>>
>>>> John.
>>>>
>>>>>
>>>>>>>>>>>> Signed-off-by: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>>> ---
>>>>>>>>>>>> drivers/gpu/drm/i915/gt/intel_engine_cs.c | 15 +++++++++++++++
>>>>>>>>>>>> drivers/gpu/drm/i915/gt/sysfs_engines.c | 14 ++++++++++++++
>>>>>>>>>>>> drivers/gpu/drm/i915/gt/uc/intel_guc_fwif.h | 9 +++++++++
>>>>>>>>>>>>   3 files changed, 38 insertions(+)
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>>>>>>>>>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>>> index e53008b4dd05..2a1e9f36e6f5 100644
>>>>>>>>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>>>>>>>>> @@ -389,6 +389,21 @@ static int intel_engine_setup(struct 
>>>>>>>>>>>> intel_gt *gt, enum intel_engine_id id,
>>>>>>>>>>>>       if (GRAPHICS_VER(i915) == 12 && engine->class == 
>>>>>>>>>>>> RENDER_CLASS)
>>>>>>>>>>>> engine->props.preempt_timeout_ms = 0;
>>>>>>>>>>>>   +    /* Cap timeouts to prevent overflow inside GuC */
>>>>>>>>>>>> +    if (intel_guc_submission_is_wanted(&gt->uc.guc)) {
>>>>>>>>>>>> +        if (engine->props.timeslice_duration_ms > 
>>>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS) {
>>>>>>>>>>>
>>>>>>>>>>> Hm "wanted".. There's been too much back and forth on the 
>>>>>>>>>>> GuC load options over the years to keep track.. 
>>>>>>>>>>> intel_engine_uses_guc work sounds like would work and read 
>>>>>>>>>>> nicer.
>>>>>>>>>> I'm not adding a new feature check here. I'm just using the 
>>>>>>>>>> existing one. If we want to rename it yet again then that 
>>>>>>>>>> would be a different patch set.
>>>>>>>>>
>>>>>>>>> $ grep intel_engine_uses_guc . -rl
>>>>>>>>> ./i915_perf.c
>>>>>>>>> ./i915_request.c
>>>>>>>>> ./selftests/intel_scheduler_helpers.c
>>>>>>>>> ./gem/i915_gem_context.c
>>>>>>>>> ./gt/intel_context.c
>>>>>>>>> ./gt/intel_engine.h
>>>>>>>>> ./gt/intel_engine_cs.c
>>>>>>>>> ./gt/intel_engine_heartbeat.c
>>>>>>>>> ./gt/intel_engine_pm.c
>>>>>>>>> ./gt/intel_reset.c
>>>>>>>>> ./gt/intel_lrc.c
>>>>>>>>> ./gt/selftest_context.c
>>>>>>>>> ./gt/selftest_engine_pm.c
>>>>>>>>> ./gt/selftest_hangcheck.c
>>>>>>>>> ./gt/selftest_mocs.c
>>>>>>>>> ./gt/selftest_workarounds.c
>>>>>>>>>
>>>>>>>>> Sounds better to me than intel_guc_submission_is_wanted. What 
>>>>>>>>> does the reader know whether "is wanted" translates to "is 
>>>>>>>>> actually used". Shrug on "is wanted".
>>>>>>>> Yes, but isn't '_uses' the one that hits a BUG_ON if you call 
>>>>>>>> it too early in the boot up sequence? I never understood why 
>>>>>>>> that was necessary or why we need so many different ways to ask 
>>>>>>>> the same question. But this version already exists and 
>>>>>>>> definitely works without hitting any explosions.
>>>>>>>
>>>>>>> No idea if it causes a bug on, doesn't in the helper itself so 
>>>>>>> maybe you are saying it is called too early? Might be.. I think 
>>>>>>> over time the nice idea we had that "setup" and "init" phases of 
>>>>>>> engine setup clearly separated got destroyed a bit. There would 
>>>>>>> always be an option to move this clamping in a later phase, once 
>>>>>>> the submission method is known. One could argue that if the 
>>>>>>> submission method is not yet known at this point, it is even 
>>>>>>> wrong to clamp based on something which will only be decided 
>>>>>>> later. Because:
>>>>>>>
>>>>>>> int intel_engines_init(struct intel_gt *gt)
>>>>>>> {
>>>>>>>     int (*setup)(struct intel_engine_cs *engine);
>>>>>>>     struct intel_engine_cs *engine;
>>>>>>>     enum intel_engine_id id;
>>>>>>>     int err;
>>>>>>>
>>>>>>>     if (intel_uc_uses_guc_submission(&gt->uc)) {
>>>>>>>         gt->submission_method = INTEL_SUBMISSION_GUC;
>>>>>>>
>>>>>>> So this uses "uses", not "wanted". Presumably the point for 
>>>>>>> having "wanted" and "uses" is that they can disagree, in which 
>>>>>>> case if you clamp early based on "wanted" that suggests it could 
>>>>>>> be wrong.
>>>>>>
>>>>>> Okay, looks like I was getting confused with intel_guc_is_used(). 
>>>>>> That one blows up if called too early.
>>>>>>
>>>>>> I'll change it to _uses_ and repost, then.
>>>>>
>>>>> Check that it isn't called too early, before gt->submission_setup 
>>>>> is set.
>>>> Obviously it is because it blew up. But I am not re-writing the 
>>>> driver start up sequence just to use the word 'use' instead of 'want'.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>>>>>> And limit to class instead of applying to all engines looks 
>>>>>>>>>>> like a miss.
>>>>>>>>>> As per follow up email, the class limit is not applied here.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> + drm_info(&engine->i915->drm, "Warning, clamping timeslice 
>>>>>>>>>>>> duration to %d to prevent possibly overflow\n",
>>>>>>>>>>>> + GUC_POLICY_MAX_EXEC_QUANTUM_MS);
>>>>>>>>>>>> + engine->props.timeslice_duration_ms = 
>>>>>>>>>>>> GUC_POLICY_MAX_EXEC_QUANTUM_MS;
>>>>>>>>>>>
>>>>>>>>>>> I am not sure logging such message during driver load is 
>>>>>>>>>>> useful. Sounds more like a confused driver which starts with 
>>>>>>>>>>> one value and then overrides itself. I'd just silently set 
>>>>>>>>>>> the value appropriate for the active backend. Preemption 
>>>>>>>>>>> timeout kconfig text already documents the fact timeouts can 
>>>>>>>>>>> get overriden at runtime depending on platform+engine. So 
>>>>>>>>>>> maybe just add same text to timeslice kconfig.
>>>>>>>>>> The point is to make people aware if they compile with 
>>>>>>>>>> unsupported config options. As far as I know, there is no way 
>>>>>>>>>> to apply range checking or other limits to config defines. 
>>>>>>>>>> Which means that a user would silently get unwanted 
>>>>>>>>>> behaviour. That seems like a bad thing to me. If the driver 
>>>>>>>>>> is confused because the user built it in a confused manner 
>>>>>>>>>> then we should let them know.
>>>>>>>>>
>>>>>>>>> Okay, but I think make it notice low level.
>>>>>>>>>
>>>>>>>>> Also consider in patch 3/3 when you triple it, and then clamp 
>>>>>>>>> back down here. That's even more confused state since tripling 
>>>>>>>>> gets nerfed. I think that's also an argument to always account 
>>>>>>>>> preempt timeout in heartbeat interval calculation. Haven't got 
>>>>>>>>> to your reply on 2/3 yet though..
>>>>>>>> That sounds like even more reason to make sure the warning gets 
>>>>>>>> seen. The more complex the system and the more chances there 
>>>>>>>> are to get it wrong, the more important it is to have a nice 
>>>>>>>> easy to see and understand notification that it did go wrong.
>>>>>>>
>>>>>>> I did not disagree, just said make it notice, one level higher 
>>>>>>> than info! :)
>>>>>> But then it won't appear unless you have explicitly said an 
>>>>>> elevated debug level. Whereas info appears in dmesg by default 
>>>>>> (but is still not classed as an error by CI and such).
>>>>>
>>>>> Notice is higher than info! :) If info appears by default so does 
>>>>> notice, warning, err, etc...
>>>> Doh! I could have sworn those were the other way around.
>>>>
>>>> Okay. Will update to use notice :).
>>>>
>>>>>
>>>>> #define KERN_EMERG      KERN_SOH "0"    /* system is unusable */
>>>>> #define KERN_ALERT      KERN_SOH "1"    /* action must be taken 
>>>>> immediately */
>>>>> #define KERN_CRIT       KERN_SOH "2"    /* critical conditions */
>>>>> #define KERN_ERR        KERN_SOH "3"    /* error conditions */
>>>>> #define KERN_WARNING    KERN_SOH "4"    /* warning conditions */
>>>>> #define KERN_NOTICE     KERN_SOH "5"    /* normal but significant 
>>>>> condition */
>>>>> #define KERN_INFO       KERN_SOH "6"    /* informational */
>>>>> #define KERN_DEBUG      KERN_SOH "7"    /* debug-level messages */
>>>>>
>>>>>>> But also think how, if we agree to go with tripling, that you'd 
>>>>>>> have to consider that in the sysfs store when hearbeat timeout 
>>>>>>> is written, to consider whether or not to triple and error out 
>>>>>>> if preemption timeout is over limit.
>>>>>> I see this as just setting the default values. If an end user is 
>>>>>> explicitly overriding the defaults then we should obey what they 
>>>>>> have requested. If they are changing the heartbeat interval then 
>>>>>> they can also change the pre-emption timeout appropriately.
>>>>>
>>>>> Question is can they unknowingly and without any feedback 
>>>>> configure a much worse state than they expect? Like when they set 
>>>>> heartbeats up to some value, everything is configured as you 
>>>>> intended - but if you go over a certain hidden limit the overall 
>>>>> scheme degrades in some way. What is the failure mode here if you 
>>>>> silently let them do that?
>>>> You can always configure things to be worse than expected. If you 
>>>> don't understand what you are doing then any control can make 
>>>> things worse instead of better. The assumption is that if a user is 
>>>> savvy enough to be writing to sysfs overrides of kernel parameters 
>>>> then they know what those parameters are and what their 
>>>> implications are. If they want to set a very short heartbeat with a 
>>>> very long pre-emption timeout then its their problem if they hit 
>>>> frequent TDRs. Conversely, if they want to set a very long 
>>>> heartbeat with a very short pre-emption timeout then its still 
>>>> their problem if they hit frequent TDRs.
>>>>
>>>> But if the user explicitly requests a heartbeat period of 3s and a 
>>>> pre-emption timeout of 2s and the i915 arbitrarily splats their 2s 
>>>> and makes it 9s then that is wrong.
>>>>
>>>> We should give the driver defaults that work for the majority of 
>>>> users and then let the minority specify exactly what they need.
>>>>
>>>> And there is no silent or hidden limit. If the user specifies a 
>>>> value too large then they will get -EINVAL. Nothing hidden or 
>>>> silent about that. Any other values are legal and the behaviour 
>>>> will be whatever has been requested.
>>>>
>>>> John.
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Tvrtko
>>>>
>>

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 3/1/2022 04:09, Tvrtko Ursulin wrote:
>
> I'll trim it a bit again..
>
> On 28/02/2022 18:55, John Harrison wrote:
>> On 2/28/2022 09:12, Tvrtko Ursulin wrote:
>>> On 25/02/2022 18:48, John Harrison wrote:
>>>> On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>
> [snip]
>
>>>>> Your only objection is that ends up with too long total time 
>>>>> before reset? Or something else as well?
>>>> An unnecessarily long total heartbeat timeout is the main 
>>>> objection. (2.5 + 12) * 5 = 72.5 seconds. That is a massive change 
>>>> from the current 12.5s.
>>>>
>>>> If we are happy with that huge increase then fine. But I'm pretty 
>>>> sure you are going to get a lot more bug reports about hung systems 
>>>> not recovering. 10-20s is just about long enough for someone to 
>>>> wait before leaning on the power button of their machine. Over a 
>>>> minute is not. That kind of delay is going to cause support issues.
>>>
>>> Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, chosen 
>>> just so it is longer than tH * 3?
>>>
>>> And how do you keep coming up with factor of five? Isn't it four 
>>> periods before "heartbeat stopped"? (Prio normal, hearbeat, barrier 
>>> and then reset.)
>> Prio starts at low not normal.
>
> Right, slipped my mind since I only keep seeing that one priority 
> ladder block in intel_engine_heartbeat.c/heartbeat()..
>
>>> From the point of view of user experience I agree reasonable 
>>> responsiveness is needed before user "reaches for the power button".
>>>
>>> In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.
>>>
>>> Question of workloads.. what is the actual preempt timeout compute 
>>> is happy with? And I don't mean compute setups with disabled 
>>> hangcheck, which you say they want anyway, but if we target defaults 
>>> for end users. Do we have some numbers on what they are likely to run?
>> Not that I have ever seen. This is all just finger in the air stuff. 
>> I don't recall if we invented the number and the compute people 
>> agreed with it or if they proposed the number to us.
>
> Yeah me neither. And found nothing in my email archives. :(
>
> Thinking about it today I don't see that disabled timeout is a 
> practical default.
>
> With it, if users have something un-preemptable to run (assuming prio 
> normal), it would get killed after ~13s (5 * 2.5).
>
> If we go for my scheme it gets killed in ~17.5s (3 * (2.5 + 2.5) + 2.5 
> (third pulse triggers preempt timeout)).
>
> And if we go for your scheme it gets killed in ~22.5s (4 * 2.5 + 2 * 3 
> * 2.5).
Erm, that is not an apples to apples comparison. Your 17.5 is for an 
engine reset tripped by the pre-emption timeout, but your 22.5s is for a 
GT reset tripped by the heartbeat reaching the end and nuking the universe.

If you are saying that the first pulse at sufficient priority (third 
being normal prio) is what causes the reset because the system is 
working as expected and the pre-emption timeout trips the reset. In that 
case, you have two periods to get to normal prio plus one pre-emption 
timeout to trip the reset. I.e. (tH * 2) + tP.

Your scheme is then tH(actual) = tH(user) + tP, yes?
So pre-emption based reset is after ((tH(user) + tP) * 2) + tP => (3 * 
tP) + (2 * tH)
And GT based reset is after (tH(user) + tP) * 5 => (5 * tP) + (5 * tH)

My scheme is tH(actual) = tH(user) for first four, then max(tH(user), 
tP) for fifth.
So pre-emption based reset is after tH(user) * 2 + tP = > tP + (2 * tH);
And GT based reset is after (tH(user) * 4) + (max(tH(user), tP) * 1) => 
greater of ((4 * tH) + tP) or (5 * tH)

Either way your scheme is longer. With tH(user) = 2.5s, tP(RCS) = 7.5s, 
we get 27.5s for engine and 50s for GT versus my 12.5s for engine and 
17.5s for GT. With tP(RCS) = 2.5s, yours is 12.5s for engine and 25s for 
GT versus my 7.5s for engine and 12.5s for GT.

Plus, not sure why your calculations above are using 2.5 for tP? Are you 
still arguing that 7.5s is too long? That is a separate issue and not 
related to the heartbeat algorithms. tP must be long enough to allow 
'out of box OpenCL workloads to complete'. That doesn't just mean not 
being killed by the heartbeat, it also means not being killed by running 
two of them concurrently (or one plus desktop OpenGL rendering) and not 
having it killed by basic time slicing between the two contexts. The 
heartbeat is not involved in that process. That is purely the 
pre-emption timeout. And that is the fundamental reason why tP needs to 
be much larger on RCS/CCS.



>
> If I did not confuse any calculation this time round, then the 
> differences for default case for normal priority sound pretty 
> immaterial to me.
>
>>> What if we gave them a default of 2.5s? That would be 4 * (2.5s + 
>>> 2.5s) = 20s worst case until reset, comparable to your proposal. Are 
>>> there realistic workloads which are non-preemptable for 2.5s? I am 
>>> having hard time imagining someone would run them on a general 
>>> purpose desktop since it would mean frozen and unusable UI anyway.
>>>
>>> Advantage still being in my mind that there would be no fudging of 
>>> timeouts during driver load and heartbeat periods depending on 
>>> priority. To me it feels more plausible to account for preempt 
>>> timeout in heartbeat pulses that to calculate preempt timeout to be 
>>> longer than hearbeat pulses, just to avoid races between the two.
>> Except that when the user asks for a heartbeat period of 2.5s you are 
>> actually setting it to 5s. How is that not a major fudge that is 
>> totally disregarding the user's request?
>
> This is indeed the core question. My thinking:
>
> It is not defined in the heartbeat ABI that N pulses should do 
> anything, just that they are periodic pulses which check the health of 
> an engine.
>
> If we view user priority as not under our control then we can say that 
> any heartbeat pulse can trigger preempt timeout and we should let it 
> do that.
>
> From that it follows that it is justified to account for preempt 
> timeout in the decision when to schedule heartbeat pulses and that it 
> is legitimate to do it for all of them.
But it can be optimised to say that it doesn't matter if you bump the 
priority of a pulse before waiting for the pre-emption period to expire. 
If the pulse was already high enough prio then the pre-emption has 
already been triggered and bumping the prio has no effect. If was too 
low then waiting for longer has no benefit at all.

All that matters is that you don't hit the end stop and trigger the GT 
reset too early.

>
> It also avoids the double reset problem, regardless of the backend and 
> regardless of how the user configured the timeouts. Without the need 
> to fudge them neither during driver load or during sysfs store.
>
> User has configured that heartbeat pulses should be sent every N 
> seconds, yes, but I think we are free to account for inherent hardware 
> and software latencies in our calculations. Especially since other 
> than flawed Gen12 RCS, other engines will be much closer to the 
> configured period.
>
> It is just the same as user asking for preempt timeout N and we say on 
> driver load "oh no you won't get it". Same for heartbeats, they said 
> 2.5s, we said 2.5s + broken engine factor...
Why would you not get the pre-emption timeout? Because it is too large? 
That is a physical limitation (of the GuC firmware) not an override 
because we think we know better. If we can obey the user then we should 
do so.

>
> I don't see a problem there. Nothing timing sensitive relies on the 
> heartbeat interval nor we provided any guarantees.
>
> That patch from Chris for instance AFAIR accounted for scheduling or 
> context switch latencies. Because what is the point of sending further 
> elevated priority pulses if we did not leave enough time to the engine 
> to schedule them in, react with preemption, or signalling completion?
>
>>>>>>>> Persistence itself can stay. There are valid UMD use cases. It 
>>>>>>>> is just massively over complicated and doesn't work in all 
>>>>>>>> corner cases when not using execlist submission or on newer 
>>>>>>>> platforms. The simplification that is planned is to allow 
>>>>>>>> contexts to persist until the associated DRM master handle is 
>>>>>>>> closed. At that point, all contexts associated with that DRM 
>>>>>>>> handle are killed. That is what AMD and others apparently 
>>>>>>>> implement.
>>>>>>>
>>>>>>> Okay, that goes against one recent IGT patch which appeared to 
>>>>>>> work around something by moving the position of _context_ close.
>>>>>> No it does not. The context close is not the trigger. The trigger is 
>>>>>
>>>>> Well patch says:
>>>>> """
>>>>> The spin all test relied on context persistence unecessarily by 
>>>>> trying
>>>>> to destroy contexts while keeping spinners active.
>>>>> The current implementation of context persistence in i915 can cause
>>>>> failures with GuC enabled, and persistence is not needed for this 
>>>>> test.
>>>>>
>>>>> Moving intel_ctx_destroy after igt_spin_end.
>>>>> """
>>>>>
>>>>> Implying moving context close to after spin end fixes things for 
>>>>> GuC, not fd close.
>>>> That's because persistence is currently a big pile of poo and does 
>>>> not work in all the corner cases. The correct solution is to leave 
>>>> the IGT alone and just fix the implementation of persistence. 
>>>> However, the IGT update to not use the broken feature is a trivial 
>>>> test change (two lines?) whereas fixing the broken feature is a 
>>>> significant KMD re-work. It needs to be done but no-one currently 
>>>> has the time to do it. But trivially changing the test allows the 
>>>> test to work and test the features it is meant to be testing (which 
>>>> is not persistence).
>>>
>>> Clear as mud. If the statement is that persistence cannot simply be 
>>> removed, then for sure it cannot be said that anything can be fixed 
>>> or unblocked by allowing some test to pass with GuC, by making them 
>>> avoid using persistence (and not even explicitly with a context 
>>> param). It implies persistence does not work with the GuC, which is 
>>> then in contradiction with the statement that we cannot just remove 
>>> persistence. I truly have no idea what the argument is here.
>> Persistence works in the right set of circumstances. Those 
>> circumstances do not involve dynamically changing heartbeat settings, 
>> platforms with dependent engines, etc. The correct fix is to leave 
>> the IGT test alone and fix the persistence implementation. However, 
>> that is not trivial and we have many other high priority holes still 
>> to plug. Whereas changing the IGT to not use a feature it is not 
>> intended to be testing anyway is a trivial change and gets us the 
>> test coverage of what that IGT is meant to be for.
>
> It may be acceptable if someone is reviewing overall coverage and 
> making sure all is not removed and so a missing ABI in GuC backend not 
> swept under the carpet. That's my main concern. If it is acknowledged 
> persistence is a needed ABI, then how can we upstream dependent engine 
> support without making sure this ABI is respected? Removing it's use 
> from random tests does not fill me with confidence that we are on top 
> of this topic.
Maybe if it didn't take three+ weeks to get a trivial change merged then 
I might have time to work on the non-trivial tasks that are in the backlog.

John.


>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[Tvrtko Ursulin]

On 01/03/2022 19:57, John Harrison wrote:
> On 3/1/2022 02:50, Tvrtko Ursulin wrote:
>> On 28/02/2022 18:32, John Harrison wrote:
>>> On 2/28/2022 08:11, Tvrtko Ursulin wrote:
>>>> On 25/02/2022 17:39, John Harrison wrote:
>>>>> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>>>>>
>>>>>> On 24/02/2022 19:19, John Harrison wrote:
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>>>>>
>>>>>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>>>>>
>>>>>>>>>>>> ./gt/intel_engine_types.h: unsigned long timeslice_duration_ms;
>>>>>>>>>>>>
>>>>>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>>>>>
>>>>>>>>>>>> So both kconfig and sysfs can already overflow GuC, not only 
>>>>>>>>>>>> because of tick conversion internally but because at backend 
>>>>>>>>>>>> level nothing was done for assigning 64-bit into 32-bit. Or 
>>>>>>>>>>>> I failed to find where it is handled.
>>>>>>>>>>> That's why I'm adding this range check to make sure we don't 
>>>>>>>>>>> allow overflows.
>>>>>>>>>>
>>>>>>>>>> Yes and no, this fixes it, but the first bug was not only due 
>>>>>>>>>> GuC internal tick conversion. It was present ever since the 
>>>>>>>>>> u64 from i915 was shoved into u32 sent to GuC. So even if GuC 
>>>>>>>>>> used the value without additional multiplication, bug was be 
>>>>>>>>>> there. My point being when GuC backend was added timeout_ms 
>>>>>>>>>> values should have been limited/clamped to U32_MAX. The tick 
>>>>>>>>>> discovery is additional limit on top.
>>>>>>>>> I'm not disagreeing. I'm just saying that the truncation wasn't 
>>>>>>>>> noticed until I actually tried using very long timeouts to 
>>>>>>>>> debug a particular problem. Now that it is noticed, we need 
>>>>>>>>> some method of range checking and this simple clamp solves all 
>>>>>>>>> the truncation problems.
>>>>>>>>
>>>>>>>> Agreed in principle, just please mention in the commit message 
>>>>>>>> all aspects of the problem.
>>>>>>>>
>>>>>>>> I think we can get away without a Fixes: tag since it requires 
>>>>>>>> user fiddling to break things in unexpected ways.
>>>>>>>>
>>>>>>>> I would though put in a code a clamping which expresses both, 
>>>>>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>>>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>>>>>> -EINVAL". Just in case GuC limit one day changes but u32 stays. 
>>>>>>>> Perhaps internal ticks go away or anything and we are left with 
>>>>>>>> plain 1:1 millisecond relationship.
>>>>>>> Can certainly add a comment along the lines of "GuC API only 
>>>>>>> takes a 32bit field but that is further reduced to GUC_LIMIT due 
>>>>>>> to internal calculations which would otherwise overflow".
>>>>>>>
>>>>>>> But if the GuC limit is > u32 then, by definition, that means the 
>>>>>>> GuC API has changed to take a u64 instead of a u32. So there will 
>>>>>>> no u32 truncation any more. So I'm not seeing a need to 
>>>>>>> explicitly test the integer size when the value check covers that.
>>>>>>
>>>>>> Hmm I was thinking if the internal conversion in the GuC fw 
>>>>>> changes so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above u32, 
>>>>>> then to be extra safe by documenting in code there is the 
>>>>>> additional limit of the data structure field. Say the field was 
>>>>>> changed to take some unit larger than a millisecond. Then the 
>>>>>> check against the GuC MAX limit define would not be enough, unless 
>>>>>> that would account both for internal implementation and u32 in the 
>>>>>> protocol. Maybe that is overdefensive but I don't see that it 
>>>>>> harms. 50-50, but it's do it once and forget so I'd do it.
>>>>> Huh?
>>>>>
>>>>> How can the limit be greater than a u32 if the interface only takes 
>>>>> a u32? By definition the limit would be clamped to u32 size.
>>>>>
>>>>> If you mean that the GuC policy is in different units and those 
>>>>> units might not overflow but ms units do, then actually that is 
>>>>> already the case. The GuC works in us not ms. That's part of why 
>>>>> the wrap around is so low, we have to multiply by 1000 before 
>>>>> sending to GuC. However, that is actually irrelevant because the 
>>>>> comparison is being done on the i915 side in i915's units. We have 
>>>>> to scale the GuC limit to match what i915 is using. And the i915 
>>>>> side is u64 so if the scaling to i915 numbers overflows a u32 then 
>>>>> who cares because that comparison can be done at 64 bits wide.
>>>>>
>>>>> If the units change then that is a backwards breaking API change 
>>>>> that will require a manual driver code update. You can't just 
>>>>> recompile with a new header and magically get an ms to us or ms to 
>>>>> s conversion in your a = b assignment. The code will need to be 
>>>>> changed to do the new unit conversion (note we already convert from 
>>>>> ms to us, the GuC API is all expressed in us). And that code change 
>>>>> will mean having to revisit any and all scaling, type conversions, 
>>>>> etc. I.e. any pre-existing checks will not necessarily be valid and 
>>>>> will need to be re-visted anyway. But as above, any scaling to GuC 
>>>>> units has to be incorporated into the limit already because 
>>>>> otherwise the limit would not fit in the GuC's own API.
>>>>
>>>> Yes I get that, I was just worried that u32 field in the protocol 
>>>> and GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the 
>>>> source code and then how to protect against forgetting to update 
>>>> both in sync.
>>>>
>>>> Like if the protocol was changed to take nanoseconds, and firmware 
>>>> implementation changed to support the full range, but define 
>>>> left/forgotten at 100s. That would then overflow u32.
>>> Huh? If the API was updated to 'support the full range' then how can 
>>> you get overflow by forgetting to update the limit? You could get 
>>> unnecessary clamping, which hopefully would be noticed by whoever is 
>>> testing the new API and/or whoever requested the change. But you 
>>> can't get u32 overflow errors if all the code has been updated to u64.
>>
>> 1)
>> Change the protocol so that "u32 desc->execution_quantum" now takes 
>> nano seconds.
>>
>> This now makes the maximum time 4.29.. seconds.
> You seriously think this is likely to happen?
> 
> That the GuC people would force an API change on us that is completely 
> backwards from what we have been asking? And that such a massive 
> backwards step would not get implemented correctly because someone 
> didn't notice just how huge an impact it was?

I don't know what we have been asking or what GuC people would do.

>> 2)
>> Forget to update GUC_POLICY_MAX_EXEC_QUANTUM_MS from 100s, since for 
>> instance that part at that point still not part of the interface 
>> contract.
> There is zero chance of the us -> ns change occurring in the foreseeable 
> future whereas the expectation is to have the limits be part of the spec 
> in the next firmware release. So this scenario is just not going to 
> happen. And as above, it would be such a big change with such a huge 
> amount of push back and discussion going on that it would be impossible 
> for the limit update to be missed/forgotten.
> 
>>
>> 3)
>> User passes in 5 seconds.
>>
>> Clamping check says all is good.
>>
>> "engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS"
>>
>> 4)
>>
>> Assignment was updated:
>>
>> gt/uc/intel_guc_submission.c:
>>
>>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>>
>> But someone did not realize field is u32.
>>
>>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>>
>> Defensive solution:
>>
>>   if (overflows_type(engine->props.timeslice_duration_ms * 1e6, 
>> desc->execution_quantum))
>>     drm_WARN_ON...
> 
> All you are saying is that bugs can happen. The above is just one more 
> place to have a bug.
> 
> The purpose of the limit is to take into account all reasons for there 
> being a limit. Having a bunch of different tests that are all testing 
> the same thing is pointless.

I am saying this:

1)
The code I pointed out is a boundary layer between two components which 
have independent design and development teams.

2)
The limit in question is currently not explicitly defined by the 
interface provider.

3)
The limit in question is also implicitly defined by the hidden internal 
firmware implementation details not relating to the units of the interface.

4)
The source code location of the clamping check is far away (different 
file, different layer) from the assignment to the interface data structure.

 From this it sounds plausible to me to have the check at the assignment 
site and don't have to think about it further.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 01/03/2022 20:59, John Harrison wrote:
> On 3/1/2022 04:09, Tvrtko Ursulin wrote:
>>
>> I'll trim it a bit again..
>>
>> On 28/02/2022 18:55, John Harrison wrote:
>>> On 2/28/2022 09:12, Tvrtko Ursulin wrote:
>>>> On 25/02/2022 18:48, John Harrison wrote:
>>>>> On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>>
>> [snip]
>>
>>>>>> Your only objection is that ends up with too long total time 
>>>>>> before reset? Or something else as well?
>>>>> An unnecessarily long total heartbeat timeout is the main 
>>>>> objection. (2.5 + 12) * 5 = 72.5 seconds. That is a massive change 
>>>>> from the current 12.5s.
>>>>>
>>>>> If we are happy with that huge increase then fine. But I'm pretty 
>>>>> sure you are going to get a lot more bug reports about hung systems 
>>>>> not recovering. 10-20s is just about long enough for someone to 
>>>>> wait before leaning on the power button of their machine. Over a 
>>>>> minute is not. That kind of delay is going to cause support issues.
>>>>
>>>> Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, chosen 
>>>> just so it is longer than tH * 3?
>>>>
>>>> And how do you keep coming up with factor of five? Isn't it four 
>>>> periods before "heartbeat stopped"? (Prio normal, hearbeat, barrier 
>>>> and then reset.)
>>> Prio starts at low not normal.
>>
>> Right, slipped my mind since I only keep seeing that one priority 
>> ladder block in intel_engine_heartbeat.c/heartbeat()..
>>
>>>> From the point of view of user experience I agree reasonable 
>>>> responsiveness is needed before user "reaches for the power button".
>>>>
>>>> In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.
>>>>
>>>> Question of workloads.. what is the actual preempt timeout compute 
>>>> is happy with? And I don't mean compute setups with disabled 
>>>> hangcheck, which you say they want anyway, but if we target defaults 
>>>> for end users. Do we have some numbers on what they are likely to run?
>>> Not that I have ever seen. This is all just finger in the air stuff. 
>>> I don't recall if we invented the number and the compute people 
>>> agreed with it or if they proposed the number to us.
>>
>> Yeah me neither. And found nothing in my email archives. :(
>>
>> Thinking about it today I don't see that disabled timeout is a 
>> practical default.
>>
>> With it, if users have something un-preemptable to run (assuming prio 
>> normal), it would get killed after ~13s (5 * 2.5).
>>
>> If we go for my scheme it gets killed in ~17.5s (3 * (2.5 + 2.5) + 2.5 
>> (third pulse triggers preempt timeout)).
>>
>> And if we go for your scheme it gets killed in ~22.5s (4 * 2.5 + 2 * 3 
>> * 2.5).
> Erm, that is not an apples to apples comparison. Your 17.5 is for an 
> engine reset tripped by the pre-emption timeout, but your 22.5s is for a 
> GT reset tripped by the heartbeat reaching the end and nuking the universe.

Right, in your scheme I did get it wrong. It would wait for GuC to reset 
the engine at the end, rather than hit the fake "hearbeat stopped" in 
that case, full reset path.

4 * 2.5 to trigger a max prio pulse, then 3 * 2.5 preempt timeout for 
GuC to reset (last hearbeat delay extended so it does not trigger). So 
17.5 as well.

> If you are saying that the first pulse at sufficient priority (third 
> being normal prio) is what causes the reset because the system is 
> working as expected and the pre-emption timeout trips the reset. In that 
> case, you have two periods to get to normal prio plus one pre-emption 
> timeout to trip the reset. I.e. (tH * 2) + tP.
> 
> Your scheme is then tH(actual) = tH(user) + tP, yes?
> So pre-emption based reset is after ((tH(user) + tP) * 2) + tP => (3 * 
> tP) + (2 * tH)
> And GT based reset is after (tH(user) + tP) * 5 => (5 * tP) + (5 * tH)
> 
> My scheme is tH(actual) = tH(user) for first four, then max(tH(user), 
> tP) for fifth.
> So pre-emption based reset is after tH(user) * 2 + tP = > tP + (2 * tH);
> And GT based reset is after (tH(user) * 4) + (max(tH(user), tP) * 1) => 
> greater of ((4 * tH) + tP) or (5 * tH)
> 
> Either way your scheme is longer. With tH(user) = 2.5s, tP(RCS) = 7.5s, 
> we get 27.5s for engine and 50s for GT versus my 12.5s for engine and 
> 17.5s for GT. With tP(RCS) = 2.5s, yours is 12.5s for engine and 25s for 
> GT versus my 7.5s for engine and 12.5s for GT.
> 
> Plus, not sure why your calculations above are using 2.5 for tP? Are you 
> still arguing that 7.5s is too long? That is a separate issue and not 
> related to the heartbeat algorithms. tP must be long enough to allow 
> 'out of box OpenCL workloads to complete'. That doesn't just mean not 
> being killed by the heartbeat, it also means not being killed by running 
> two of them concurrently (or one plus desktop OpenGL rendering) and not 
> having it killed by basic time slicing between the two contexts. The 
> heartbeat is not involved in that process. That is purely the 
> pre-emption timeout. And that is the fundamental reason why tP needs to 
> be much larger on RCS/CCS.

I was assuming 2.5s tP is enough and basing all calculation on that. 
Heartbeat or timeslicing regardless. I thought we established neither of 
us knows how long is enough.

Are you now saying 2.5s is definitely not enough? How is that usable for 
a default out of the box desktop?

>> If I did not confuse any calculation this time round, then the 
>> differences for default case for normal priority sound pretty 
>> immaterial to me.
>>
>>>> What if we gave them a default of 2.5s? That would be 4 * (2.5s + 
>>>> 2.5s) = 20s worst case until reset, comparable to your proposal. Are 
>>>> there realistic workloads which are non-preemptable for 2.5s? I am 
>>>> having hard time imagining someone would run them on a general 
>>>> purpose desktop since it would mean frozen and unusable UI anyway.
>>>>
>>>> Advantage still being in my mind that there would be no fudging of 
>>>> timeouts during driver load and heartbeat periods depending on 
>>>> priority. To me it feels more plausible to account for preempt 
>>>> timeout in heartbeat pulses that to calculate preempt timeout to be 
>>>> longer than hearbeat pulses, just to avoid races between the two.
>>> Except that when the user asks for a heartbeat period of 2.5s you are 
>>> actually setting it to 5s. How is that not a major fudge that is 
>>> totally disregarding the user's request?
>>
>> This is indeed the core question. My thinking:
>>
>> It is not defined in the heartbeat ABI that N pulses should do 
>> anything, just that they are periodic pulses which check the health of 
>> an engine.
>>
>> If we view user priority as not under our control then we can say that 
>> any heartbeat pulse can trigger preempt timeout and we should let it 
>> do that.
>>
>> From that it follows that it is justified to account for preempt 
>> timeout in the decision when to schedule heartbeat pulses and that it 
>> is legitimate to do it for all of them.
> But it can be optimised to say that it doesn't matter if you bump the 
> priority of a pulse before waiting for the pre-emption period to expire. 
> If the pulse was already high enough prio then the pre-emption has 
> already been triggered and bumping the prio has no effect. If was too 
> low then waiting for longer has no benefit at all.
> 
> All that matters is that you don't hit the end stop and trigger the GT 
> reset too early.

Yes I agree that it can also be argued what you are saying.

I was trying to weigh pros&cons of both approaches by bringing into the 
discussing the question of what are heartbeats. Given they are 
loosely/vaguely defined I think we have freedom to tweak things.

And I don't have a problem with extending the last pulse. It is 
fundamentally correct to do regardless of the backend. I just raised the 
question of whether to extend all heartbeats to account for preemption 
(and scheduling delays). (What is the point of bumping their priority 
and re-scheduling if we didn't give enough time to the engine to react? 
So opposite of the question you raise.)

What I do have a problem with is deriving the preempt timeout from the 
hearbeat period. Hence I am looking if we can instead find a fixed 
number which works, and so avoid having bi-directional coupling.

>> It also avoids the double reset problem, regardless of the backend and 
>> regardless of how the user configured the timeouts. Without the need 
>> to fudge them neither during driver load or during sysfs store.
>>
>> User has configured that heartbeat pulses should be sent every N 
>> seconds, yes, but I think we are free to account for inherent hardware 
>> and software latencies in our calculations. Especially since other 
>> than flawed Gen12 RCS, other engines will be much closer to the 
>> configured period.
>>
>> It is just the same as user asking for preempt timeout N and we say on 
>> driver load "oh no you won't get it". Same for heartbeats, they said 
>> 2.5s, we said 2.5s + broken engine factor...
> Why would you not get the pre-emption timeout? Because it is too large? 
> That is a physical limitation (of the GuC firmware) not an override 
> because we think we know better. If we can obey the user then we should 
> do so.

I was simply referring to the override in intel_engine_setup.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[Tvrtko Ursulin]

On 28/02/2022 19:17, John Harrison wrote:
> On 2/28/2022 07:32, Tvrtko Ursulin wrote:
>> On 25/02/2022 19:03, John Harrison wrote:
>>> On 2/25/2022 10:29, Tvrtko Ursulin wrote:
>>>> On 25/02/2022 18:01, John Harrison wrote:
>>>>> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>>>>>> On 25/02/2022 17:11, John Harrison wrote:
>>>>>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>>>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Compute workloads are inherently not pre-emptible on 
>>>>>>>>>>>>> current hardware.
>>>>>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround 
>>>>>>>>>>>>> to prevent
>>>>>>>>>>>>> unwanted resets. Instead, the hang detection was left to 
>>>>>>>>>>>>> the heartbeat
>>>>>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>>>>>> submission as
>>>>>>>>>>>>> the heartbeat is a full GT reset rather than a per engine 
>>>>>>>>>>>>> reset and so
>>>>>>>>>>>>> is much more destructive. Instead, just bump the 
>>>>>>>>>>>>> pre-emption timeout
>>>>>>>>>>>>
>>>>>>>>>>>> Can we have a feature request to allow asking GuC for an 
>>>>>>>>>>>> engine reset?
>>>>>>>>>>> For what purpose?
>>>>>>>>>>
>>>>>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>>>>>
>>>>>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>>>>>> context might be executing on. Even without virtual engines, 
>>>>>>>>>>> the KMD still has no knowledge of which context is currently 
>>>>>>>>>>> executing on any given engine at any given time.
>>>>>>>>>>>
>>>>>>>>>>> There is a reason why hang detection should be left to the 
>>>>>>>>>>> entity that is doing the scheduling. Any other entity is 
>>>>>>>>>>> second guessing at best.
>>>>>>>>>>>
>>>>>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>>>>>> submission is enabled is for the case where the KMD/GuC have 
>>>>>>>>>>> got out of sync with either other somehow or GuC itself has 
>>>>>>>>>>> just crashed. I.e. when no submission at all is working and 
>>>>>>>>>>> we need to reset the GuC itself and start over.
>>>>>>>>>>
>>>>>>>>>> .. I wasn't really up to speed to know/remember heartbeats are 
>>>>>>>>>> nerfed already in GuC mode.
>>>>>>>>> Not sure what you mean by that claim. Engine resets are handled 
>>>>>>>>> by GuC because GuC handles the scheduling. You can't do the 
>>>>>>>>> former if you aren't doing the latter. However, the heartbeat 
>>>>>>>>> is still present and is still the watchdog by which engine 
>>>>>>>>> resets are triggered. As per the rest of the submission 
>>>>>>>>> process, the hang detection and recovery is split between i915 
>>>>>>>>> and GuC.
>>>>>>>>
>>>>>>>> I meant that "stopped heartbeat on engine XXX" can only do a 
>>>>>>>> full GPU reset on GuC.
>>>>>>> I mean that there is no 'stopped heartbeat on engine XXX' when 
>>>>>>> i915 is not handling the recovery part of the process.
>>>>>>
>>>>>> Hmmmm?
>>>>>>
>>>>>> static void
>>>>>> reset_engine(struct intel_engine_cs *engine, struct i915_request *rq)
>>>>>> {
>>>>>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>>>>>         show_heartbeat(rq, engine);
>>>>>>
>>>>>>     if (intel_engine_uses_guc(engine))
>>>>>>         /*
>>>>>>          * GuC itself is toast or GuC's hang detection
>>>>>>          * is disabled. Either way, need to find the
>>>>>>          * hang culprit manually.
>>>>>>          */
>>>>>>         intel_guc_find_hung_context(engine);
>>>>>>
>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>                   "stopped heartbeat on %s",
>>>>>>                   engine->name);
>>>>>> }
>>>>>>
>>>>>> How there is no "stopped hearbeat" in guc mode? From this code it 
>>>>>> certainly looks there is.
>>>>> Only when the GuC is toast and it is no longer an engine reset but 
>>>>> a full GT reset that is required. So technically, it is not a 
>>>>> 'stopped heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
>>>>>
>>>>>>
>>>>>> You say below heartbeats are going in GuC mode. Now I totally 
>>>>>> don't understand how they are going but there is allegedly no 
>>>>>> "stopped hearbeat".
>>>>> Because if GuC is handling the detection and recovery then i915 
>>>>> will not reach that point. GuC will do the engine reset and start 
>>>>> scheduling the next context before the heartbeat period expires. So 
>>>>> the notification will be a G2H about a specific context being reset 
>>>>> rather than the i915 notification about a stopped heartbeat.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>>>                   "stopped heartbeat on %s",
>>>>>>>>                   engine->name);
>>>>>>>>
>>>>>>>> intel_gt_handle_error:
>>>>>>>>
>>>>>>>>     /*
>>>>>>>>      * Try engine reset when available. We fall back to full 
>>>>>>>> reset if
>>>>>>>>      * single reset fails.
>>>>>>>>      */
>>>>>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>>>>>         local_bh_disable();
>>>>>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>>>>>
>>>>>>>> You said "However, the heartbeat is still present and is still 
>>>>>>>> the watchdog by which engine resets are triggered", now I don't 
>>>>>>>> know what you meant by this. It actually triggers a single 
>>>>>>>> engine reset in GuC mode? Where in code does that happen if this 
>>>>>>>> block above shows it not taking the engine reset path?
>>>>>>> i915 sends down the per engine pulse.
>>>>>>> GuC schedules the pulse
>>>>>>> GuC attempts to pre-empt the currently active context
>>>>>>> GuC detects the pre-emption timeout
>>>>>>> GuC resets the engine
>>>>>>>
>>>>>>> The fundamental process is exactly the same as in execlist mode. 
>>>>>>> It's just that the above blocks of code (calls to 
>>>>>>> intel_gt_handle_error and such) are now inside the GuC not i915.
>>>>>>>
>>>>>>> Without the heartbeat going ping, there would be no context 
>>>>>>> switching and thus no pre-emption, no pre-emption timeout and so 
>>>>>>> no hang and reset recovery. And GuC cannot sent pulses by itself 
>>>>>>> - it has no ability to generate context workloads. So we need 
>>>>>>> i915 to send the pings and to gradually raise their priority. But 
>>>>>>> the back half of the heartbeat code is now inside the GuC. It 
>>>>>>> will simply never reach the i915 side timeout if GuC is doing the 
>>>>>>> recovery (unless the heartbeat's final period is too short). We 
>>>>>>> should only reach the i915 side timeout if GuC itself is toast. 
>>>>>>> At which point we need the full GT reset to recover the GuC.
>>>>>>
>>>>>> If workload is not preempting and reset does not work, like engine 
>>>>>> is truly stuck, does the current code hit "stopped heartbeat" or 
>>>>>> not in GuC mode?
>>>>> Hang on, where did 'reset does not work' come into this?
>>>>>
>>>>> If GuC is alive and the hardware is not broken then no, it won't. 
>>>>> That's the whole point. GuC does the detection and recovery. The 
>>>>> KMD will never reach 'stopped heartbeat'.
>>>>>
>>>>> If the hardware is broken and the reset does not work then GuC will 
>>>>> send a 'failed reset' notification to the KMD. The KMD treats that 
>>>>> as a major error and immediately does a full GT reset. So there is 
>>>>> still no 'stopped heartbeat'.
>>>>>
>>>>> If GuC has died (or a KMD bug has caused sufficient confusion to 
>>>>> make it think the GuC has died) then yes, you will reach that code. 
>>>>> But in that case it is not an engine reset that is required, it is 
>>>>> a full GT reset including a reset of the GuC.
>>>>
>>>> Got it, so what is actually wrong is calling intel_gt_handle_error 
>>>> with an engine->mask in GuC mode. 
>>>> intel_engine_hearbeat.c/reset_engine should fork into two (in some 
>>>> way), depending on backend, so in the case of GuC it can call a 
>>>> variant of intel_gt_handle_error which would be explicitly about a 
>>>> full GPU reset only, instead of a sprinkle of 
>>>> intel_uc_uses_guc_submission in that function. Possibly even off 
>>>> load the reset to a single per gt worker, so that if multiple active 
>>>> engines trigger the reset roughly simultaneously, there is only one 
>>>> full GPU reset. And it gets correctly labeled as "dead GuC" or 
>>>> something.
>>>>
>>> Sure. Feel free to re-write the reset code yet again. That's another 
>>> exceedingly fragile area of the driver.
>>
>>> However, that is unrelated to this patch set.
>>
>> It's still okay to talk about improving things in my opinion. 
>> Especially since I think it is the same issue I raised late 2019 
>> during internal review.
>>
>> And I don't think it is fair to say that I should go and rewrite it, 
>> since I do not own the GuC backend area. Especially given the above.
>>
>> If there is no motivation to improve it now then please at least track 
>> this, if it isn't already, in that internal Jira which was tracking 
>> all the areas of the GuC backend which could be improved.
>>
>> I am also pretty sure if the design was cleaner it would have been 
>> more obvious to me, or anyone who happens to stumble there, what the 
>> purpose of intel_engine_heartbeat.c/reset_engine() is in GuC mode. So 
>> we wouldn't have to spend this long explaining things.
> My point is that redesigning it to be cleaner is not just a GuC task. It 
> means redesigning the entire reset sequence to more compatible with 
> externally handled resets. That's a big task. Sure it can be added to 
> the todo list but it is totally not in the scope of this patch set.

My point was that was something which was raised years ago ("don't just 
shoe-horn, redesign, refactor"). Anyway, stopping flogging of this dead 
horse.. onto below..
> This is purely about enabling per engine resets again so that end users 
> have a better experience when GPU hangs occur. Or at least, don't have a 
> greatly worse experience on our newest platforms than they did on all 
> the previous ones because we have totally hacked that feature out. And 
> actually getting that feature enabled before we ship too many products 
> and the maintainers/architects decide we are no longer allowed to turn 
> it on because it is now a behaviour change that end users are not 
> expecting. So forever more ADL-P/DG2 are stuck on full GT only resets.

Is any platform with GuC outside force probe already? Either way 
blocking re-addition of engine resets will not be a thing from 
maintainers point of view. Whether or not the fail of not having them is 
a conscious or accidental miss, we certainly want it back ASAP.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 0/3] Improve anti-pre-emption w/a for compute workloads

[John Harrison]

On 3/2/2022 03:21, Tvrtko Ursulin wrote:
> On 28/02/2022 19:17, John Harrison wrote:
>> On 2/28/2022 07:32, Tvrtko Ursulin wrote:
>>> On 25/02/2022 19:03, John Harrison wrote:
>>>> On 2/25/2022 10:29, Tvrtko Ursulin wrote:
>>>>> On 25/02/2022 18:01, John Harrison wrote:
>>>>>> On 2/25/2022 09:39, Tvrtko Ursulin wrote:
>>>>>>> On 25/02/2022 17:11, John Harrison wrote:
>>>>>>>> On 2/25/2022 08:36, Tvrtko Ursulin wrote:
>>>>>>>>> On 24/02/2022 20:02, John Harrison wrote:
>>>>>>>>>> On 2/23/2022 04:00, Tvrtko Ursulin wrote:
>>>>>>>>>>> On 23/02/2022 02:22, John Harrison wrote:
>>>>>>>>>>>> On 2/22/2022 01:53, Tvrtko Ursulin wrote:
>>>>>>>>>>>>> On 18/02/2022 21:33, John.C.Harrison@Intel.com wrote:
>>>>>>>>>>>>>> From: John Harrison <John.C.Harrison@Intel.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Compute workloads are inherently not pre-emptible on 
>>>>>>>>>>>>>> current hardware.
>>>>>>>>>>>>>> Thus the pre-emption timeout was disabled as a workaround 
>>>>>>>>>>>>>> to prevent
>>>>>>>>>>>>>> unwanted resets. Instead, the hang detection was left to 
>>>>>>>>>>>>>> the heartbeat
>>>>>>>>>>>>>> and its (longer) timeout. This is undesirable with GuC 
>>>>>>>>>>>>>> submission as
>>>>>>>>>>>>>> the heartbeat is a full GT reset rather than a per engine 
>>>>>>>>>>>>>> reset and so
>>>>>>>>>>>>>> is much more destructive. Instead, just bump the 
>>>>>>>>>>>>>> pre-emption timeout
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can we have a feature request to allow asking GuC for an 
>>>>>>>>>>>>> engine reset?
>>>>>>>>>>>> For what purpose?
>>>>>>>>>>>
>>>>>>>>>>> To allow "stopped heartbeat" to reset the engine, however..
>>>>>>>>>>>
>>>>>>>>>>>> GuC manages the scheduling of contexts across engines. With 
>>>>>>>>>>>> virtual engines, the KMD has no knowledge of which engine a 
>>>>>>>>>>>> context might be executing on. Even without virtual 
>>>>>>>>>>>> engines, the KMD still has no knowledge of which context is 
>>>>>>>>>>>> currently executing on any given engine at any given time.
>>>>>>>>>>>>
>>>>>>>>>>>> There is a reason why hang detection should be left to the 
>>>>>>>>>>>> entity that is doing the scheduling. Any other entity is 
>>>>>>>>>>>> second guessing at best.
>>>>>>>>>>>>
>>>>>>>>>>>> The reason for keeping the heartbeat around even when GuC 
>>>>>>>>>>>> submission is enabled is for the case where the KMD/GuC 
>>>>>>>>>>>> have got out of sync with either other somehow or GuC 
>>>>>>>>>>>> itself has just crashed. I.e. when no submission at all is 
>>>>>>>>>>>> working and we need to reset the GuC itself and start over.
>>>>>>>>>>>
>>>>>>>>>>> .. I wasn't really up to speed to know/remember heartbeats 
>>>>>>>>>>> are nerfed already in GuC mode.
>>>>>>>>>> Not sure what you mean by that claim. Engine resets are 
>>>>>>>>>> handled by GuC because GuC handles the scheduling. You can't 
>>>>>>>>>> do the former if you aren't doing the latter. However, the 
>>>>>>>>>> heartbeat is still present and is still the watchdog by which 
>>>>>>>>>> engine resets are triggered. As per the rest of the 
>>>>>>>>>> submission process, the hang detection and recovery is split 
>>>>>>>>>> between i915 and GuC.
>>>>>>>>>
>>>>>>>>> I meant that "stopped heartbeat on engine XXX" can only do a 
>>>>>>>>> full GPU reset on GuC.
>>>>>>>> I mean that there is no 'stopped heartbeat on engine XXX' when 
>>>>>>>> i915 is not handling the recovery part of the process.
>>>>>>>
>>>>>>> Hmmmm?
>>>>>>>
>>>>>>> static void
>>>>>>> reset_engine(struct intel_engine_cs *engine, struct i915_request 
>>>>>>> *rq)
>>>>>>> {
>>>>>>>     if (IS_ENABLED(CONFIG_DRM_I915_DEBUG_GEM))
>>>>>>>         show_heartbeat(rq, engine);
>>>>>>>
>>>>>>>     if (intel_engine_uses_guc(engine))
>>>>>>>         /*
>>>>>>>          * GuC itself is toast or GuC's hang detection
>>>>>>>          * is disabled. Either way, need to find the
>>>>>>>          * hang culprit manually.
>>>>>>>          */
>>>>>>>         intel_guc_find_hung_context(engine);
>>>>>>>
>>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>>                   "stopped heartbeat on %s",
>>>>>>>                   engine->name);
>>>>>>> }
>>>>>>>
>>>>>>> How there is no "stopped hearbeat" in guc mode? From this code 
>>>>>>> it certainly looks there is.
>>>>>> Only when the GuC is toast and it is no longer an engine reset 
>>>>>> but a full GT reset that is required. So technically, it is not a 
>>>>>> 'stopped heartbeat on engine XXX' it is 'stopped heartbeat on GT#'.
>>>>>>
>>>>>>>
>>>>>>> You say below heartbeats are going in GuC mode. Now I totally 
>>>>>>> don't understand how they are going but there is allegedly no 
>>>>>>> "stopped hearbeat".
>>>>>> Because if GuC is handling the detection and recovery then i915 
>>>>>> will not reach that point. GuC will do the engine reset and start 
>>>>>> scheduling the next context before the heartbeat period expires. 
>>>>>> So the notification will be a G2H about a specific context being 
>>>>>> reset rather than the i915 notification about a stopped heartbeat.
>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>     intel_gt_handle_error(engine->gt, engine->mask,
>>>>>>>>>                   I915_ERROR_CAPTURE,
>>>>>>>>>                   "stopped heartbeat on %s",
>>>>>>>>>                   engine->name);
>>>>>>>>>
>>>>>>>>> intel_gt_handle_error:
>>>>>>>>>
>>>>>>>>>     /*
>>>>>>>>>      * Try engine reset when available. We fall back to full 
>>>>>>>>> reset if
>>>>>>>>>      * single reset fails.
>>>>>>>>>      */
>>>>>>>>>     if (!intel_uc_uses_guc_submission(&gt->uc) &&
>>>>>>>>>         intel_has_reset_engine(gt) && !intel_gt_is_wedged(gt)) {
>>>>>>>>>         local_bh_disable();
>>>>>>>>>         for_each_engine_masked(engine, gt, engine_mask, tmp) {
>>>>>>>>>
>>>>>>>>> You said "However, the heartbeat is still present and is still 
>>>>>>>>> the watchdog by which engine resets are triggered", now I 
>>>>>>>>> don't know what you meant by this. It actually triggers a 
>>>>>>>>> single engine reset in GuC mode? Where in code does that 
>>>>>>>>> happen if this block above shows it not taking the engine 
>>>>>>>>> reset path?
>>>>>>>> i915 sends down the per engine pulse.
>>>>>>>> GuC schedules the pulse
>>>>>>>> GuC attempts to pre-empt the currently active context
>>>>>>>> GuC detects the pre-emption timeout
>>>>>>>> GuC resets the engine
>>>>>>>>
>>>>>>>> The fundamental process is exactly the same as in execlist 
>>>>>>>> mode. It's just that the above blocks of code (calls to 
>>>>>>>> intel_gt_handle_error and such) are now inside the GuC not i915.
>>>>>>>>
>>>>>>>> Without the heartbeat going ping, there would be no context 
>>>>>>>> switching and thus no pre-emption, no pre-emption timeout and 
>>>>>>>> so no hang and reset recovery. And GuC cannot sent pulses by 
>>>>>>>> itself - it has no ability to generate context workloads. So we 
>>>>>>>> need i915 to send the pings and to gradually raise their 
>>>>>>>> priority. But the back half of the heartbeat code is now inside 
>>>>>>>> the GuC. It will simply never reach the i915 side timeout if 
>>>>>>>> GuC is doing the recovery (unless the heartbeat's final period 
>>>>>>>> is too short). We should only reach the i915 side timeout if 
>>>>>>>> GuC itself is toast. At which point we need the full GT reset 
>>>>>>>> to recover the GuC.
>>>>>>>
>>>>>>> If workload is not preempting and reset does not work, like 
>>>>>>> engine is truly stuck, does the current code hit "stopped 
>>>>>>> heartbeat" or not in GuC mode?
>>>>>> Hang on, where did 'reset does not work' come into this?
>>>>>>
>>>>>> If GuC is alive and the hardware is not broken then no, it won't. 
>>>>>> That's the whole point. GuC does the detection and recovery. The 
>>>>>> KMD will never reach 'stopped heartbeat'.
>>>>>>
>>>>>> If the hardware is broken and the reset does not work then GuC 
>>>>>> will send a 'failed reset' notification to the KMD. The KMD 
>>>>>> treats that as a major error and immediately does a full GT 
>>>>>> reset. So there is still no 'stopped heartbeat'.
>>>>>>
>>>>>> If GuC has died (or a KMD bug has caused sufficient confusion to 
>>>>>> make it think the GuC has died) then yes, you will reach that 
>>>>>> code. But in that case it is not an engine reset that is 
>>>>>> required, it is a full GT reset including a reset of the GuC.
>>>>>
>>>>> Got it, so what is actually wrong is calling intel_gt_handle_error 
>>>>> with an engine->mask in GuC mode. 
>>>>> intel_engine_hearbeat.c/reset_engine should fork into two (in some 
>>>>> way), depending on backend, so in the case of GuC it can call a 
>>>>> variant of intel_gt_handle_error which would be explicitly about a 
>>>>> full GPU reset only, instead of a sprinkle of 
>>>>> intel_uc_uses_guc_submission in that function. Possibly even off 
>>>>> load the reset to a single per gt worker, so that if multiple 
>>>>> active engines trigger the reset roughly simultaneously, there is 
>>>>> only one full GPU reset. And it gets correctly labeled as "dead 
>>>>> GuC" or something.
>>>>>
>>>> Sure. Feel free to re-write the reset code yet again. That's 
>>>> another exceedingly fragile area of the driver.
>>>
>>>> However, that is unrelated to this patch set.
>>>
>>> It's still okay to talk about improving things in my opinion. 
>>> Especially since I think it is the same issue I raised late 2019 
>>> during internal review.
>>>
>>> And I don't think it is fair to say that I should go and rewrite it, 
>>> since I do not own the GuC backend area. Especially given the above.
>>>
>>> If there is no motivation to improve it now then please at least 
>>> track this, if it isn't already, in that internal Jira which was 
>>> tracking all the areas of the GuC backend which could be improved.
>>>
>>> I am also pretty sure if the design was cleaner it would have been 
>>> more obvious to me, or anyone who happens to stumble there, what the 
>>> purpose of intel_engine_heartbeat.c/reset_engine() is in GuC mode. 
>>> So we wouldn't have to spend this long explaining things.
>> My point is that redesigning it to be cleaner is not just a GuC task. 
>> It means redesigning the entire reset sequence to more compatible 
>> with externally handled resets. That's a big task. Sure it can be 
>> added to the todo list but it is totally not in the scope of this 
>> patch set.
>
> My point was that was something which was raised years ago ("don't 
> just shoe-horn, redesign, refactor"). Anyway, stopping flogging of 
> this dead horse.. onto below..
>> This is purely about enabling per engine resets again so that end 
>> users have a better experience when GPU hangs occur. Or at least, 
>> don't have a greatly worse experience on our newest platforms than 
>> they did on all the previous ones because we have totally hacked that 
>> feature out. And actually getting that feature enabled before we ship 
>> too many products and the maintainers/architects decide we are no 
>> longer allowed to turn it on because it is now a behaviour change 
>> that end users are not expecting. So forever more ADL-P/DG2 are stuck 
>> on full GT only resets.
>
> Is any platform with GuC outside force probe already? Either way 
> blocking re-addition of engine resets will not be a thing from 
> maintainers point of view. Whether or not the fail of not having them 
> is a conscious or accidental miss, we certainly want it back ASAP.
>
ADL-P is released and running GuC submission.

John.

> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 3/2/2022 03:07, Tvrtko Ursulin wrote:
> On 01/03/2022 20:59, John Harrison wrote:
>> On 3/1/2022 04:09, Tvrtko Ursulin wrote:
>>>
>>> I'll trim it a bit again..
>>>
>>> On 28/02/2022 18:55, John Harrison wrote:
>>>> On 2/28/2022 09:12, Tvrtko Ursulin wrote:
>>>>> On 25/02/2022 18:48, John Harrison wrote:
>>>>>> On 2/25/2022 10:14, Tvrtko Ursulin wrote:
>>>
>>> [snip]
>>>
>>>>>>> Your only objection is that ends up with too long total time 
>>>>>>> before reset? Or something else as well?
>>>>>> An unnecessarily long total heartbeat timeout is the main 
>>>>>> objection. (2.5 + 12) * 5 = 72.5 seconds. That is a massive 
>>>>>> change from the current 12.5s.
>>>>>>
>>>>>> If we are happy with that huge increase then fine. But I'm pretty 
>>>>>> sure you are going to get a lot more bug reports about hung 
>>>>>> systems not recovering. 10-20s is just about long enough for 
>>>>>> someone to wait before leaning on the power button of their 
>>>>>> machine. Over a minute is not. That kind of delay is going to 
>>>>>> cause support issues.
>>>>>
>>>>> Sorry I wrote 12s, while you actually said tP * 12, so 7.68s, 
>>>>> chosen just so it is longer than tH * 3?
>>>>>
>>>>> And how do you keep coming up with factor of five? Isn't it four 
>>>>> periods before "heartbeat stopped"? (Prio normal, hearbeat, 
>>>>> barrier and then reset.)
>>>> Prio starts at low not normal.
>>>
>>> Right, slipped my mind since I only keep seeing that one priority 
>>> ladder block in intel_engine_heartbeat.c/heartbeat()..
>>>
>>>>> From the point of view of user experience I agree reasonable 
>>>>> responsiveness is needed before user "reaches for the power button".
>>>>>
>>>>> In your proposal we are talking about 3 * 2.5s + 2 * 7.5s, so 22.5s.
>>>>>
>>>>> Question of workloads.. what is the actual preempt timeout compute 
>>>>> is happy with? And I don't mean compute setups with disabled 
>>>>> hangcheck, which you say they want anyway, but if we target 
>>>>> defaults for end users. Do we have some numbers on what they are 
>>>>> likely to run?
>>>> Not that I have ever seen. This is all just finger in the air 
>>>> stuff. I don't recall if we invented the number and the compute 
>>>> people agreed with it or if they proposed the number to us.
>>>
>>> Yeah me neither. And found nothing in my email archives. :(
>>>
>>> Thinking about it today I don't see that disabled timeout is a 
>>> practical default.
>>>
>>> With it, if users have something un-preemptable to run (assuming 
>>> prio normal), it would get killed after ~13s (5 * 2.5).
>>>
>>> If we go for my scheme it gets killed in ~17.5s (3 * (2.5 + 2.5) + 
>>> 2.5 (third pulse triggers preempt timeout)).
>>>
>>> And if we go for your scheme it gets killed in ~22.5s (4 * 2.5 + 2 * 
>>> 3 * 2.5).
>> Erm, that is not an apples to apples comparison. Your 17.5 is for an 
>> engine reset tripped by the pre-emption timeout, but your 22.5s is 
>> for a GT reset tripped by the heartbeat reaching the end and nuking 
>> the universe.
>
> Right, in your scheme I did get it wrong. It would wait for GuC to 
> reset the engine at the end, rather than hit the fake "hearbeat 
> stopped" in that case, full reset path.
>
> 4 * 2.5 to trigger a max prio pulse, then 3 * 2.5 preempt timeout for 
> GuC to reset (last hearbeat delay extended so it does not trigger). So 
> 17.5 as well.

Again, apples or oranges? I was using your tP(RCS) == 2.5s assumption in 
all the above calculations given that the discussion was about the 
heartbeat algorithm, not the choice of pre-emption timeout. In which 
case the last heartbeat is max(tP * 2, tH) == 2 * 2.5s.

>
>> If you are saying that the first pulse at sufficient priority (third 
>> being normal prio) is what causes the reset because the system is 
>> working as expected and the pre-emption timeout trips the reset. In 
>> that case, you have two periods to get to normal prio plus one 
>> pre-emption timeout to trip the reset. I.e. (tH * 2) + tP.
>>
>> Your scheme is then tH(actual) = tH(user) + tP, yes?
>> So pre-emption based reset is after ((tH(user) + tP) * 2) + tP => (3 
>> * tP) + (2 * tH)
>> And GT based reset is after (tH(user) + tP) * 5 => (5 * tP) + (5 * tH)
>>
>> My scheme is tH(actual) = tH(user) for first four, then max(tH(user), 
>> tP) for fifth.
>> So pre-emption based reset is after tH(user) * 2 + tP = > tP + (2 * tH);
>> And GT based reset is after (tH(user) * 4) + (max(tH(user), tP) * 1) 
>> => greater of ((4 * tH) + tP) or (5 * tH)
>>
>> Either way your scheme is longer. With tH(user) = 2.5s, tP(RCS) = 
>> 7.5s, we get 27.5s for engine and 50s for GT versus my 12.5s for 
>> engine and 17.5s for GT. With tP(RCS) = 2.5s, yours is 12.5s for 
>> engine and 25s for GT versus my 7.5s for engine and 12.5s for GT.
>>
>> Plus, not sure why your calculations above are using 2.5 for tP? Are 
>> you still arguing that 7.5s is too long? That is a separate issue and 
>> not related to the heartbeat algorithms. tP must be long enough to 
>> allow 'out of box OpenCL workloads to complete'. That doesn't just 
>> mean not being killed by the heartbeat, it also means not being 
>> killed by running two of them concurrently (or one plus desktop 
>> OpenGL rendering) and not having it killed by basic time slicing 
>> between the two contexts. The heartbeat is not involved in that 
>> process. That is purely the pre-emption timeout. And that is the 
>> fundamental reason why tP needs to be much larger on RCS/CCS.
>
> I was assuming 2.5s tP is enough and basing all calculation on that. 
> Heartbeat or timeslicing regardless. I thought we established neither 
> of us knows how long is enough.
>
> Are you now saying 2.5s is definitely not enough? How is that usable 
> for a default out of the box desktop?
Show me your proof that 2.5s is enough.

7.5s is what we have been using internally for a very long time. It has 
approval from all relevant parties. If you wish to pick a new random 
number then please provide data to back it up along with buy in from all 
UMD teams and project management.

>
>>> If I did not confuse any calculation this time round, then the 
>>> differences for default case for normal priority sound pretty 
>>> immaterial to me.
>>>
>>>>> What if we gave them a default of 2.5s? That would be 4 * (2.5s + 
>>>>> 2.5s) = 20s worst case until reset, comparable to your proposal. 
>>>>> Are there realistic workloads which are non-preemptable for 2.5s? 
>>>>> I am having hard time imagining someone would run them on a 
>>>>> general purpose desktop since it would mean frozen and unusable UI 
>>>>> anyway.
>>>>>
>>>>> Advantage still being in my mind that there would be no fudging of 
>>>>> timeouts during driver load and heartbeat periods depending on 
>>>>> priority. To me it feels more plausible to account for preempt 
>>>>> timeout in heartbeat pulses that to calculate preempt timeout to 
>>>>> be longer than hearbeat pulses, just to avoid races between the two.
>>>> Except that when the user asks for a heartbeat period of 2.5s you 
>>>> are actually setting it to 5s. How is that not a major fudge that 
>>>> is totally disregarding the user's request?
>>>
>>> This is indeed the core question. My thinking:
>>>
>>> It is not defined in the heartbeat ABI that N pulses should do 
>>> anything, just that they are periodic pulses which check the health 
>>> of an engine.
>>>
>>> If we view user priority as not under our control then we can say 
>>> that any heartbeat pulse can trigger preempt timeout and we should 
>>> let it do that.
>>>
>>> From that it follows that it is justified to account for preempt 
>>> timeout in the decision when to schedule heartbeat pulses and that 
>>> it is legitimate to do it for all of them.
>> But it can be optimised to say that it doesn't matter if you bump the 
>> priority of a pulse before waiting for the pre-emption period to 
>> expire. If the pulse was already high enough prio then the 
>> pre-emption has already been triggered and bumping the prio has no 
>> effect. If was too low then waiting for longer has no benefit at all.
>>
>> All that matters is that you don't hit the end stop and trigger the 
>> GT reset too early.
>
> Yes I agree that it can also be argued what you are saying.
>
> I was trying to weigh pros&cons of both approaches by bringing into 
> the discussing the question of what are heartbeats. Given they are 
> loosely/vaguely defined I think we have freedom to tweak things.
>
> And I don't have a problem with extending the last pulse. It is 
> fundamentally correct to do regardless of the backend. I just raised 
> the question of whether to extend all heartbeats to account for 
> preemption (and scheduling delays). (What is the point of bumping 
> their priority and re-scheduling if we didn't give enough time to the 
> engine to react? So opposite of the question you raise.)
The point is that it we are giving enough time to react. Raising the 
priority of a pre-emption that has already been triggered will have no 
effect. So as long as the total time from when the pre-emption is 
triggered (prio becomes sufficiently high) to the point when the reset 
is decided is longer than the pre-emption timeout then it works. Given 
that, it is unnecessary to increase the intermediate periods. It has no 
advantage and has the disadvantage of making the total time unreasonably 
long.

So again, what is the point of making every period longer? What benefit 
does it *actually* give?

>
> What I do have a problem with is deriving the preempt timeout from the 
> hearbeat period. Hence I am looking if we can instead find a fixed 
> number which works, and so avoid having bi-directional coupling.
Fine. "tP(RCS) = 7500;" can I merge the patch now?

>
>>> It also avoids the double reset problem, regardless of the backend 
>>> and regardless of how the user configured the timeouts. Without the 
>>> need to fudge them neither during driver load or during sysfs store.
>>>
>>> User has configured that heartbeat pulses should be sent every N 
>>> seconds, yes, but I think we are free to account for inherent 
>>> hardware and software latencies in our calculations. Especially 
>>> since other than flawed Gen12 RCS, other engines will be much closer 
>>> to the configured period.
>>>
>>> It is just the same as user asking for preempt timeout N and we say 
>>> on driver load "oh no you won't get it". Same for heartbeats, they 
>>> said 2.5s, we said 2.5s + broken engine factor...
>> Why would you not get the pre-emption timeout? Because it is too 
>> large? That is a physical limitation (of the GuC firmware) not an 
>> override because we think we know better. If we can obey the user 
>> then we should do so.
>
> I was simply referring to the override in intel_engine_setup.
Meaning the boost rather than the clamp? The other option would be to 
add a CONFIG_DRM_I915_PREEMPT_TIMEOUT_COMPUTE option for specifying 
tP(RCS). I can do that as a follow up patch if other maintainers agree 
to adding yet more CONFIG options. My understanding was that they are 
frowned upon and only to be added when there is no other way.

John.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 1/3] drm/i915/guc: Limit scheduling properties to avoid overflow

[John Harrison]

On 3/2/2022 01:20, Tvrtko Ursulin wrote:
> On 01/03/2022 19:57, John Harrison wrote:
>> On 3/1/2022 02:50, Tvrtko Ursulin wrote:
>>> On 28/02/2022 18:32, John Harrison wrote:
>>>> On 2/28/2022 08:11, Tvrtko Ursulin wrote:
>>>>> On 25/02/2022 17:39, John Harrison wrote:
>>>>>> On 2/25/2022 09:06, Tvrtko Ursulin wrote:
>>>>>>>
>>>>>>> On 24/02/2022 19:19, John Harrison wrote:
>>>>>>>
>>>>>>> [snip]
>>>>>>>
>>>>>>>>>>>>> ./gt/uc/intel_guc_fwif.h: u32 execution_quantum;
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./gt/uc/intel_guc_submission.c: desc->execution_quantum = 
>>>>>>>>>>>>> engine->props.timeslice_duration_ms * 1000;
>>>>>>>>>>>>>
>>>>>>>>>>>>> ./gt/intel_engine_types.h: unsigned long 
>>>>>>>>>>>>> timeslice_duration_ms;
>>>>>>>>>>>>>
>>>>>>>>>>>>> timeslice_store/preempt_timeout_store:
>>>>>>>>>>>>> err = kstrtoull(buf, 0, &duration);
>>>>>>>>>>>>>
>>>>>>>>>>>>> So both kconfig and sysfs can already overflow GuC, not 
>>>>>>>>>>>>> only because of tick conversion internally but because at 
>>>>>>>>>>>>> backend level nothing was done for assigning 64-bit into 
>>>>>>>>>>>>> 32-bit. Or I failed to find where it is handled.
>>>>>>>>>>>> That's why I'm adding this range check to make sure we 
>>>>>>>>>>>> don't allow overflows.
>>>>>>>>>>>
>>>>>>>>>>> Yes and no, this fixes it, but the first bug was not only 
>>>>>>>>>>> due GuC internal tick conversion. It was present ever since 
>>>>>>>>>>> the u64 from i915 was shoved into u32 sent to GuC. So even 
>>>>>>>>>>> if GuC used the value without additional multiplication, bug 
>>>>>>>>>>> was be there. My point being when GuC backend was added 
>>>>>>>>>>> timeout_ms values should have been limited/clamped to 
>>>>>>>>>>> U32_MAX. The tick discovery is additional limit on top.
>>>>>>>>>> I'm not disagreeing. I'm just saying that the truncation 
>>>>>>>>>> wasn't noticed until I actually tried using very long 
>>>>>>>>>> timeouts to debug a particular problem. Now that it is 
>>>>>>>>>> noticed, we need some method of range checking and this 
>>>>>>>>>> simple clamp solves all the truncation problems.
>>>>>>>>>
>>>>>>>>> Agreed in principle, just please mention in the commit message 
>>>>>>>>> all aspects of the problem.
>>>>>>>>>
>>>>>>>>> I think we can get away without a Fixes: tag since it requires 
>>>>>>>>> user fiddling to break things in unexpected ways.
>>>>>>>>>
>>>>>>>>> I would though put in a code a clamping which expresses both, 
>>>>>>>>> something like min(u32, ..GUC LIMIT..). So the full story is 
>>>>>>>>> documented forever. Or "if > u32 || > ..GUC LIMIT..) return 
>>>>>>>>> -EINVAL". Just in case GuC limit one day changes but u32 
>>>>>>>>> stays. Perhaps internal ticks go away or anything and we are 
>>>>>>>>> left with plain 1:1 millisecond relationship.
>>>>>>>> Can certainly add a comment along the lines of "GuC API only 
>>>>>>>> takes a 32bit field but that is further reduced to GUC_LIMIT 
>>>>>>>> due to internal calculations which would otherwise overflow".
>>>>>>>>
>>>>>>>> But if the GuC limit is > u32 then, by definition, that means 
>>>>>>>> the GuC API has changed to take a u64 instead of a u32. So 
>>>>>>>> there will no u32 truncation any more. So I'm not seeing a need 
>>>>>>>> to explicitly test the integer size when the value check covers 
>>>>>>>> that.
>>>>>>>
>>>>>>> Hmm I was thinking if the internal conversion in the GuC fw 
>>>>>>> changes so that GUC_POLICY_MAX_PREEMPT_TIMEOUT_MS goes above 
>>>>>>> u32, then to be extra safe by documenting in code there is the 
>>>>>>> additional limit of the data structure field. Say the field was 
>>>>>>> changed to take some unit larger than a millisecond. Then the 
>>>>>>> check against the GuC MAX limit define would not be enough, 
>>>>>>> unless that would account both for internal implementation and 
>>>>>>> u32 in the protocol. Maybe that is overdefensive but I don't see 
>>>>>>> that it harms. 50-50, but it's do it once and forget so I'd do it.
>>>>>> Huh?
>>>>>>
>>>>>> How can the limit be greater than a u32 if the interface only 
>>>>>> takes a u32? By definition the limit would be clamped to u32 size.
>>>>>>
>>>>>> If you mean that the GuC policy is in different units and those 
>>>>>> units might not overflow but ms units do, then actually that is 
>>>>>> already the case. The GuC works in us not ms. That's part of why 
>>>>>> the wrap around is so low, we have to multiply by 1000 before 
>>>>>> sending to GuC. However, that is actually irrelevant because the 
>>>>>> comparison is being done on the i915 side in i915's units. We 
>>>>>> have to scale the GuC limit to match what i915 is using. And the 
>>>>>> i915 side is u64 so if the scaling to i915 numbers overflows a 
>>>>>> u32 then who cares because that comparison can be done at 64 bits 
>>>>>> wide.
>>>>>>
>>>>>> If the units change then that is a backwards breaking API change 
>>>>>> that will require a manual driver code update. You can't just 
>>>>>> recompile with a new header and magically get an ms to us or ms 
>>>>>> to s conversion in your a = b assignment. The code will need to 
>>>>>> be changed to do the new unit conversion (note we already convert 
>>>>>> from ms to us, the GuC API is all expressed in us). And that code 
>>>>>> change will mean having to revisit any and all scaling, type 
>>>>>> conversions, etc. I.e. any pre-existing checks will not 
>>>>>> necessarily be valid and will need to be re-visted anyway. But as 
>>>>>> above, any scaling to GuC units has to be incorporated into the 
>>>>>> limit already because otherwise the limit would not fit in the 
>>>>>> GuC's own API.
>>>>>
>>>>> Yes I get that, I was just worried that u32 field in the protocol 
>>>>> and GUC_POLICY_MAX_EXEC_QUANTUM_MS defines are separate in the 
>>>>> source code and then how to protect against forgetting to update 
>>>>> both in sync.
>>>>>
>>>>> Like if the protocol was changed to take nanoseconds, and firmware 
>>>>> implementation changed to support the full range, but define 
>>>>> left/forgotten at 100s. That would then overflow u32.
>>>> Huh? If the API was updated to 'support the full range' then how 
>>>> can you get overflow by forgetting to update the limit? You could 
>>>> get unnecessary clamping, which hopefully would be noticed by 
>>>> whoever is testing the new API and/or whoever requested the change. 
>>>> But you can't get u32 overflow errors if all the code has been 
>>>> updated to u64.
>>>
>>> 1)
>>> Change the protocol so that "u32 desc->execution_quantum" now takes 
>>> nano seconds.
>>>
>>> This now makes the maximum time 4.29.. seconds.
>> You seriously think this is likely to happen?
>>
>> That the GuC people would force an API change on us that is 
>> completely backwards from what we have been asking? And that such a 
>> massive backwards step would not get implemented correctly because 
>> someone didn't notice just how huge an impact it was?
>
> I don't know what we have been asking or what GuC people would do.
Despite the views of some in the community, the GuC team are not evil 
monsters out for world domination. We are their customers and their task 
is to provide a usable offload device that makes the Linux experience 
better not worse.

Just from this discussion alone, ignoring any internal forums, it has 
been made clear that the (long standing) request from the i915 team is 
to support 64bit policy values and (more recently) to officially 
document any and all limits involved in the policies. By definition, 
that also means that there would be significant push back and argument 
if the GuC team proposed making this interface worse.

>
>>> 2)
>>> Forget to update GUC_POLICY_MAX_EXEC_QUANTUM_MS from 100s, since for 
>>> instance that part at that point still not part of the interface 
>>> contract.
>> There is zero chance of the us -> ns change occurring in the 
>> foreseeable future whereas the expectation is to have the limits be 
>> part of the spec in the next firmware release. So this scenario is 
>> just not going to happen. And as above, it would be such a big change 
>> with such a huge amount of push back and discussion going on that it 
>> would be impossible for the limit update to be missed/forgotten.
>>
>>>
>>> 3)
>>> User passes in 5 seconds.
>>>
>>> Clamping check says all is good.
>>>
>>> "engine->props.timeslice_duration_ms > GUC_POLICY_MAX_EXEC_QUANTUM_MS"
>>>
>>> 4)
>>>
>>> Assignment was updated:
>>>
>>> gt/uc/intel_guc_submission.c:
>>>
>>>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>>>
>>> But someone did not realize field is u32.
>>>
>>>   desc->execution_quantum = engine->props.timeslice_duration_ms * 1e6;
>>>
>>> Defensive solution:
>>>
>>>   if (overflows_type(engine->props.timeslice_duration_ms * 1e6, 
>>> desc->execution_quantum))
>>>     drm_WARN_ON...
>>
>> All you are saying is that bugs can happen. The above is just one 
>> more place to have a bug.
>>
>> The purpose of the limit is to take into account all reasons for 
>> there being a limit. Having a bunch of different tests that are all 
>> testing the same thing is pointless.
>
> I am saying this:
>
> 1)
> The code I pointed out is a boundary layer between two components 
> which have independent design and development teams.
>
> 2)
> The limit in question is currently not explicitly defined by the 
> interface provider.
>
> 3)
> The limit in question is also implicitly defined by the hidden 
> internal firmware implementation details not relating to the units of 
> the interface.
>
> 4)
> The source code location of the clamping check is far away (different 
> file, different layer) from the assignment to the interface data 
> structure.
>
> From this it sounds plausible to me to have the check at the 
> assignment site and don't have to think about it further.
It also sounds plausible to use the concept of consolidation. Rather 
than scattering random different limit tests in random different places, 
it all goes into a single helper function that can be used at the top 
level and report any range issues before you get to the lower levels 
where errors might not be allowed. This was your own feedback (and is 
currently implemented in the v2 post).

John.

>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 02/03/2022 17:55, John Harrison wrote:

>> I was assuming 2.5s tP is enough and basing all calculation on that. 
>> Heartbeat or timeslicing regardless. I thought we established neither 
>> of us knows how long is enough.
>>
>> Are you now saying 2.5s is definitely not enough? How is that usable 
>> for a default out of the box desktop?
> Show me your proof that 2.5s is enough.
> 
> 7.5s is what we have been using internally for a very long time. It has 
> approval from all relevant parties. If you wish to pick a new random 
> number then please provide data to back it up along with buy in from all 
> UMD teams and project management.

And upstream disabled preemption has acks from compute. "Internally" is 
as far away from out of the box desktop experiences we have been arguing 
about. In fact you have argued compute disables the hearbeat anyway.

Lets jump to the end of this reply please for actions.

>> And I don't have a problem with extending the last pulse. It is 
>> fundamentally correct to do regardless of the backend. I just raised 
>> the question of whether to extend all heartbeats to account for 
>> preemption (and scheduling delays). (What is the point of bumping 
>> their priority and re-scheduling if we didn't give enough time to the 
>> engine to react? So opposite of the question you raise.)
> The point is that it we are giving enough time to react. Raising the 
> priority of a pre-emption that has already been triggered will have no 
> effect. So as long as the total time from when the pre-emption is 
> triggered (prio becomes sufficiently high) to the point when the reset 
> is decided is longer than the pre-emption timeout then it works. Given 
> that, it is unnecessary to increase the intermediate periods. It has no 
> advantage and has the disadvantage of making the total time unreasonably 
> long.
> 
> So again, what is the point of making every period longer? What benefit 
> does it *actually* give?

Less special casing and pointless prio bumps ahead of giving time to 
engine to even react. You wouldn't have to have the last pulse 2 * tP 
but normal tH + tP. So again, it is nicer for me to derive all heartbeat 
pulses from the same input parameters.

The whole "it is very long" argument is IMO moot because now proposed 
7.5s preempt period is I suspect wholly impractical for desktop. 
Combined with the argument that real compute disables heartbeats anyway 
even extra so.

> Fine. "tP(RCS) = 7500;" can I merge the patch now?
I could live with setting preempt timeout to 7.5s. The downside is 
slower time to restoring frozen desktops. Worst case today 5 * 2.5s, 
with changes 4 * 2.5s + 2 * 7.5s; so from 12.5s to 25s, doubling..

Actions:

1)
Get a number from compute/OpenCL people for what they say is minimum 
preempt timeout for default out of the box Linux desktop experience.

This does not mean them running some tests and can't be bothered to 
setup up the machine for the extreme use cases, but workloads average 
users can realistically be expected to run.

Say for instance some image manipulation software which is OpenCL 
accelerated or similar. How long unpreemptable sections are expected 
there. Or similar. I am not familiar what all OpenCL accelerated use 
cases there are on Linux.

And this number should be purely about minimum preempt timeout, not 
considering heartbeats. This is because preempt timeout may kick in 
sooner than stopped heartbeat if the user workload is low priority.

2)
Commit message should explain the effect on the worst case time until 
engine reset.

3)
OpenCL/compute should ack the change publicly as well since they acked 
the disabling of preemption.

4)
I really want overflows_type in the first patch.

My position is that with the above satisfied it is okay to merge.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[John Harrison]

On 3/3/2022 01:55, Tvrtko Ursulin wrote:
> On 02/03/2022 17:55, John Harrison wrote:
>
>>> I was assuming 2.5s tP is enough and basing all calculation on that. 
>>> Heartbeat or timeslicing regardless. I thought we established 
>>> neither of us knows how long is enough.
>>>
>>> Are you now saying 2.5s is definitely not enough? How is that usable 
>>> for a default out of the box desktop?
>> Show me your proof that 2.5s is enough.
>>
>> 7.5s is what we have been using internally for a very long time. It 
>> has approval from all relevant parties. If you wish to pick a new 
>> random number then please provide data to back it up along with buy 
>> in from all UMD teams and project management.
>
> And upstream disabled preemption has acks from compute. "Internally" 
> is as far away from out of the box desktop experiences we have been 
> arguing about. In fact you have argued compute disables the hearbeat 
> anyway.
>
> Lets jump to the end of this reply please for actions.
>
>>> And I don't have a problem with extending the last pulse. It is 
>>> fundamentally correct to do regardless of the backend. I just raised 
>>> the question of whether to extend all heartbeats to account for 
>>> preemption (and scheduling delays). (What is the point of bumping 
>>> their priority and re-scheduling if we didn't give enough time to 
>>> the engine to react? So opposite of the question you raise.)
>> The point is that it we are giving enough time to react. Raising the 
>> priority of a pre-emption that has already been triggered will have 
>> no effect. So as long as the total time from when the pre-emption is 
>> triggered (prio becomes sufficiently high) to the point when the 
>> reset is decided is longer than the pre-emption timeout then it 
>> works. Given that, it is unnecessary to increase the intermediate 
>> periods. It has no advantage and has the disadvantage of making the 
>> total time unreasonably long.
>>
>> So again, what is the point of making every period longer? What 
>> benefit does it *actually* give?
>
> Less special casing and pointless prio bumps ahead of giving time to 
> engine to even react. You wouldn't have to have the last pulse 2 * tP 
> but normal tH + tP. So again, it is nicer for me to derive all 
> heartbeat pulses from the same input parameters.
>
> The whole "it is very long" argument is IMO moot because now proposed 
> 7.5s preempt period is I suspect wholly impractical for desktop. 
> Combined with the argument that real compute disables heartbeats 
> anyway even extra so.
The whole thing is totally fubar already. Right now pre-emption is 
totally disabled. So you are currently waiting for the entire heartbeat 
sequence to complete and then nuking the entire machine. So arguing that 
7.5s is too long is pointless. 7.5s is a big improvement over what is 
currently enabled.

And 'nice' would be having hardware that worked in a sensible manner. 
There is no nice here. There is only 'what is the least worst option'. 
And the least worst option for an end user is a long pre-emption timeout 
with a not massively long heartbeat. If that means a very slight 
complication in the heartbeat code, that is a trivial concern.


>
>> Fine. "tP(RCS) = 7500;" can I merge the patch now?
> I could live with setting preempt timeout to 7.5s. The downside is 
> slower time to restoring frozen desktops. Worst case today 5 * 2.5s, 
> with changes 4 * 2.5s + 2 * 7.5s; so from 12.5s to 25s, doubling..
But that is worst case scenario (when something much more severe than an 
application hang has occurred). Regular case would be second heartbeat 
period + pre-emption timeout and an engine only reset not a full GT 
reset. So it's still better than what we have at present.

>
> Actions:
>
> 1)
> Get a number from compute/OpenCL people for what they say is minimum 
> preempt timeout for default out of the box Linux desktop experience.
That would be the one that has been agreed upon by both linux software 
arch and all UMD teams and has been in use for the past year or more in 
the internal tree.

>
> This does not mean them running some tests and can't be bothered to 
> setup up the machine for the extreme use cases, but workloads average 
> users can realistically be expected to run.
>
> Say for instance some image manipulation software which is OpenCL 
> accelerated or similar. How long unpreemptable sections are expected 
> there. Or similar. I am not familiar what all OpenCL accelerated use 
> cases there are on Linux.
>
> And this number should be purely about minimum preempt timeout, not 
> considering heartbeats. This is because preempt timeout may kick in 
> sooner than stopped heartbeat if the user workload is low priority.
>
And driver is simply hosed in the intervening six months or more that it 
takes for the right people to find the time to do this.

Right now, it is broken. This patch set improves things. Actual numbers 
can be refined later as/when some random use case that we hadn't 
previously thought of pops up. But not fixing the basic problem at all 
until we have an absolutely perfect for all parties solution is 
pointless. Not least because there is no perfect solution. No matter 
what number you pick it is going to be wrong for someone.

2.5s, 7.5s, X.Ys, I really don't care. 2.5s is a number you seem to have 
picked out of the air totally at random, or maybe based on it being the 
heartbeat period (except that you keep arguing that basing tP on tH is 
wrong). 7.5s is a number that has been in active use for a lot of 
testing for quite some time - KMD CI, UMD CI, E2E, etc. But either way, 
the initial number is almost irrelevant as long as it is not zero. So 
can we please just get something merged now as a starting point?


> 2)
> Commit message should explain the effect on the worst case time until 
> engine reset.
>
> 3)
> OpenCL/compute should ack the change publicly as well since they acked 
> the disabling of preemption.
This patch set has already been publicly acked by the compute team. See 
the 'acked-by' tag.


>
> 4)
> I really want overflows_type in the first patch.
In the final GuC assignment? Only if it is a BUG_ON. If we get a failure 
there it is an internal driver error and cannot be corrected for. It is 
too late for any plausible range check action.

And if you mean in the the actual helper function with the rest of the 
clamping then you are bleeding internal GuC API structure details into 
non-GuC code. Plus the test would be right next to the 'if (size < 
OFFICIAL_GUC_RANGE_LIMIT)' test which just looks dumb as well as being 
redundant duplication - "if ((value < GUC_LIMIT) && (value < 
NO_WE_REALLY_MEAN_IT_GUC_LIMIT))". And putting it inside the GuC limit 
definition looks even worse "#define LIMIT min(MAX_U32, 100*1000) /* 
because the developer doesn't know how big a u32 is */".

John.

>
> My position is that with the above satisfied it is okay to merge.
>
> Regards,
>
> Tvrtko

Re: [Intel-gfx] [PATCH 2/3] drm/i915/gt: Make the heartbeat play nice with long pre-emption timeouts

[Tvrtko Ursulin]

On 03/03/2022 19:09, John Harrison wrote:

>> Actions:
>>
>> 1)
>> Get a number from compute/OpenCL people for what they say is minimum 
>> preempt timeout for default out of the box Linux desktop experience.
> That would be the one that has been agreed upon by both linux software 
> arch and all UMD teams and has been in use for the past year or more in 
> the internal tree.

What has been used in the internal tree is irrelevant when UMD ack is needed for changes which affect upstream shipping platforms like Tigerlake.

>> This does not mean them running some tests and can't be bothered to 
>> setup up the machine for the extreme use cases, but workloads average 
>> users can realistically be expected to run.
>>
>> Say for instance some image manipulation software which is OpenCL 
>> accelerated or similar. How long unpreemptable sections are expected 
>> there. Or similar. I am not familiar what all OpenCL accelerated use 
>> cases there are on Linux.
>>
>> And this number should be purely about minimum preempt timeout, not 
>> considering heartbeats. This is because preempt timeout may kick in 
>> sooner than stopped heartbeat if the user workload is low priority.
>>
> And driver is simply hosed in the intervening six months or more that it 
> takes for the right people to find the time to do this.

What is hosed? Driver currently contains a patch which was acked by the compute UMD to disable preemption. If it takes six months for compute UMD to give us a different number which works for the open source stack and typical use cases then what can we do.

> Right now, it is broken. This patch set improves things. Actual numbers 
> can be refined later as/when some random use case that we hadn't 
> previously thought of pops up. But not fixing the basic problem at all 
> until we have an absolutely perfect for all parties solution is 
> pointless. Not least because there is no perfect solution. No matter 
> what number you pick it is going to be wrong for someone.
> 
> 2.5s, 7.5s, X.Ys, I really don't care. 2.5s is a number you seem to have 
> picked out of the air totally at random, or maybe based on it being the 
> heartbeat period (except that you keep arguing that basing tP on tH is 
> wrong). 7.5s is a number that has been in active use for a lot of 
> testing for quite some time - KMD CI, UMD CI, E2E, etc. But either way, 
> the initial number is almost irrelevant as long as it is not zero. So 
> can we please just get something merged now as a starting point?
> 
> 
>> 2)
>> Commit message should explain the effect on the worst case time until 
>> engine reset.
>>
>> 3)
>> OpenCL/compute should ack the change publicly as well since they acked 
>> the disabling of preemption.
> This patch set has already been publicly acked by the compute team. See 
> the 'acked-by' tag.

I can't find the reply which contained the ack on the mailing list - do you have a msg-id or an archive link?

Also, ack needs to be against the the fixed timeout patch and not one dependent on the heartbeat interval.

>> 4)
>> I really want overflows_type in the first patch.
> In the final GuC assignment? Only if it is a BUG_ON. If we get a failure 
> there it is an internal driver error and cannot be corrected for. It is 
> too late for any plausible range check action.

If you can find a test which exercises setting insane values to the relevant timeouts and so would hit the problem in our CI then BUG_ON is fine. Otherwise I think BUG_ON is too anti-social and prefer drm_warn or drm_WARN_ON. I don't think adding a test is strictly necessary, if we don't already have one, given how unlikely this is too be hit, but if you insist on a BUG_ON instead of a flavour of a warn then I think we need one so we can catch in CI 100% of the time.
  
> And if you mean in the the actual helper function with the rest of the 
> clamping then you are bleeding internal GuC API structure details into 
> non-GuC code. Plus the test would be right next to the 'if (size < 

In my other reply I exactly described that would be a downside and that I prefer checks at the assignment sites.

Also regarding this comment in the relevant patch:

+	/*
+	 * NB: The GuC API only supports 32bit values. However, the limit is further
+	 * reduced due to internal calculations which would otherwise overflow.
+	 */

I would suggest clarifying this as "The GuC API only supports timeouts up to U32_MAX micro-seconds. However, ...". Given the function at hand deals in milliseconds explicitly calling out that additional scaling factor makes sense I think.

Big picture - it's really still very simple. Public ack for a fixed number and a warn on is not really a lot to ask.

Regards,

Tvrtko