[PATCH 0/2] TLB invalidation cleanup

[PATCH 0/2] TLB invalidation cleanup

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

First patch to make the platform check not catch all due current lack of
automated testing.

Second patch to consolidate the code a bit and move invariant setup at engine
init time. Don't think I want to merge this one until some test coverage can be
had.

Tvrtko Ursulin (2):
  drm/i915: Do not cover all future platforms in TLB invalidation
  drm/i915: Consolidate TLB invalidation flow

 drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 ++++++++++++++
 drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
 drivers/gpu/drm/i915/gt/intel_gt.c           | 124 +++----------------
 3 files changed, 128 insertions(+), 104 deletions(-)

-- 
2.34.1

[Intel-gfx] [PATCH 0/2] TLB invalidation cleanup

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

First patch to make the platform check not catch all due current lack of
automated testing.

Second patch to consolidate the code a bit and move invariant setup at engine
init time. Don't think I want to merge this one until some test coverage can be
had.

Tvrtko Ursulin (2):
  drm/i915: Do not cover all future platforms in TLB invalidation
  drm/i915: Consolidate TLB invalidation flow

 drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 ++++++++++++++
 drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
 drivers/gpu/drm/i915/gt/intel_gt.c           | 124 +++----------------
 3 files changed, 128 insertions(+), 104 deletions(-)

-- 
2.34.1

[PATCH 1/2] drm/i915: Do not cover all future platforms in TLB invalidation

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Revert to the original explicit approach and document the reasoning
behind it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Matt Roper <matthew.d.roper@intel.com>
Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Cc: Andrzej Hajda <andrzej.hajda@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_gt.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 7eeee5a7cb33..854841a731cb 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -1070,7 +1070,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 	unsigned int num = 0;
 	unsigned long flags;
 
-	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
+	/*
+	 * New platforms should not be added with catch-all-newer (>=)
+	 * condition so that any later platform added triggers the below warning
+	 * and in turn mandates a human cross-check of whether the invalidation
+	 * flows have compatible semantics.
+	 *
+	 * For instance with the 11.00 -> 12.00 transition three out of five
+	 * respective engine registers were moved to masked type. Then after the
+	 * 12.00 -> 12.50 transition multi cast handling is required too.
+	 */
+
+	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
 		regs = NULL;
 		num = ARRAY_SIZE(xehp_regs);
 	} else if (GRAPHICS_VER(i915) == 12) {
-- 
2.34.1

[Intel-gfx] [PATCH 1/2] drm/i915: Do not cover all future platforms in TLB invalidation

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Revert to the original explicit approach and document the reasoning
behind it.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Matt Roper <matthew.d.roper@intel.com>
Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
Cc: Andrzej Hajda <andrzej.hajda@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_gt.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 7eeee5a7cb33..854841a731cb 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -1070,7 +1070,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 	unsigned int num = 0;
 	unsigned long flags;
 
-	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
+	/*
+	 * New platforms should not be added with catch-all-newer (>=)
+	 * condition so that any later platform added triggers the below warning
+	 * and in turn mandates a human cross-check of whether the invalidation
+	 * flows have compatible semantics.
+	 *
+	 * For instance with the 11.00 -> 12.00 transition three out of five
+	 * respective engine registers were moved to masked type. Then after the
+	 * 12.00 -> 12.50 transition multi cast handling is required too.
+	 */
+
+	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
 		regs = NULL;
 		num = ARRAY_SIZE(xehp_regs);
 	} else if (GRAPHICS_VER(i915) == 12) {
-- 
2.34.1

Re: [Intel-gfx] [PATCH 1/2] drm/i915: Do not cover all future platforms in TLB invalidation

[Andrzej Hajda]

On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> 
> Revert to the original explicit approach and document the reasoning
> behind it.
> 
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> Cc: Matt Roper <matthew.d.roper@intel.com>
> Cc: Balasubramani Vivekanandan <balasubramani.vivekanandan@intel.com>
> Cc: Andrzej Hajda <andrzej.hajda@intel.com>

Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>

Regards
Andrzej

> ---
>   drivers/gpu/drm/i915/gt/intel_gt.c | 13 ++++++++++++-
>   1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> index 7eeee5a7cb33..854841a731cb 100644
> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> @@ -1070,7 +1070,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>   	unsigned int num = 0;
>   	unsigned long flags;
>   
> -	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> +	/*
> +	 * New platforms should not be added with catch-all-newer (>=)
> +	 * condition so that any later platform added triggers the below warning
> +	 * and in turn mandates a human cross-check of whether the invalidation
> +	 * flows have compatible semantics.
> +	 *
> +	 * For instance with the 11.00 -> 12.00 transition three out of five
> +	 * respective engine registers were moved to masked type. Then after the
> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> +	 */
> +
> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>   		regs = NULL;
>   		num = ARRAY_SIZE(xehp_regs);
>   	} else if (GRAPHICS_VER(i915) == 12) {

[PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As the logic for selecting the register and corresponsing values grew, the
code become a bit unsightly. Consolidate by storing the required values at
engine init time in the engine itself, and by doing so minimise the amount
of invariant platform and engine checks during each and every TLB
invalidation.

v2:
 * Fail engine probe if TLB invlidations registers are unknown.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Andrzej Hajda <andrzej.hajda@intel.com>
Cc: Matt Roper <matthew.d.roper@intel.com>
Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
 drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
 drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
 3 files changed, 128 insertions(+), 115 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index 99c4b866addd..d47dadfc25c8 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
 	return ret;
 }
 
+static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
+{
+	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
+		[RENDER_CLASS].reg		= GEN8_RTCR,
+		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
+		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
+	};
+	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
+		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
+	};
+	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
+		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
+	};
+	struct drm_i915_private *i915 = engine->i915;
+	const union intel_engine_tlb_inv_reg *regs;
+	union intel_engine_tlb_inv_reg reg;
+	unsigned int class = engine->class;
+	unsigned int num = 0;
+	u32 val;
+
+	/*
+	 * New platforms should not be added with catch-all-newer (>=)
+	 * condition so that any later platform added triggers the below warning
+	 * and in turn mandates a human cross-check of whether the invalidation
+	 * flows have compatible semantics.
+	 *
+	 * For instance with the 11.00 -> 12.00 transition three out of five
+	 * respective engine registers were moved to masked type. Then after the
+	 * 12.00 -> 12.50 transition multi cast handling is required too.
+	 */
+
+	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
+		regs = xehp_regs;
+		num = ARRAY_SIZE(xehp_regs);
+	} else if (GRAPHICS_VER(i915) == 12) {
+		regs = gen12_regs;
+		num = ARRAY_SIZE(gen12_regs);
+	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
+		regs = gen8_regs;
+		num = ARRAY_SIZE(gen8_regs);
+	} else if (GRAPHICS_VER(i915) < 8) {
+		return 0;
+	}
+
+	if (drm_WARN_ONCE(&i915->drm, !num,
+			  "Platform does not implement TLB invalidation!"))
+		return -ENODEV;
+
+	if (drm_WARN_ON_ONCE(&i915->drm,
+			     class >= num ||
+			     (!regs[class].reg.reg &&
+			      !regs[class].mcr_reg.reg)))
+		return -ERANGE;
+
+	reg = regs[class];
+
+	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
+		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
+		val = 0;
+	} else {
+		val = engine->instance;
+	}
+
+	val = BIT(val);
+
+	engine->tlb_inv.mcr = regs == xehp_regs;
+	engine->tlb_inv.reg = reg;
+	engine->tlb_inv.done = val;
+
+	if (GRAPHICS_VER(i915) >= 12 &&
+	    (engine->class == VIDEO_DECODE_CLASS ||
+	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
+	     engine->class == COMPUTE_CLASS))
+		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
+	else
+		engine->tlb_inv.request = val;
+
+	return 0;
+}
+
 static int engine_setup_common(struct intel_engine_cs *engine)
 {
 	int err;
 
 	init_llist_head(&engine->barrier_tasks);
 
+	err = intel_engine_init_tlb_invalidation(engine);
+	if (err)
+		return err;
+
 	err = init_status_page(engine);
 	if (err)
 		return err;
diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
index 4fd54fb8810f..8c661fe89314 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
+++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
@@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
 	u64 start_gt_clk;
 };
 
+union intel_engine_tlb_inv_reg {
+	i915_reg_t	reg;
+	i915_mcr_reg_t	mcr_reg;
+};
+
+struct intel_engine_tlb_inv
+{
+	bool mcr;
+	union intel_engine_tlb_inv_reg reg;
+	u32 request;
+	u32 done;
+};
+
 struct intel_engine_cs {
 	struct drm_i915_private *i915;
 	struct intel_gt *gt;
@@ -372,6 +385,8 @@ struct intel_engine_cs {
 	u32 context_size;
 	u32 mmio_base;
 
+	struct intel_engine_tlb_inv tlb_inv;
+
 	/*
 	 * Some w/a require forcewake to be held (which prevents RC6) while
 	 * a particular engine is active. If so, we set fw_domain to which
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 854841a731cb..9fb0ac03f51a 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
 	intel_sseu_dump(&info->sseu, p);
 }
 
-struct reg_and_bit {
-	union {
-		i915_reg_t reg;
-		i915_mcr_reg_t mcr_reg;
-	};
-	u32 bit;
-};
-
-static struct reg_and_bit
-get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
-		const i915_reg_t *regs, const unsigned int num)
-{
-	const unsigned int class = engine->class;
-	struct reg_and_bit rb = { };
-
-	if (drm_WARN_ON_ONCE(&engine->i915->drm,
-			     class >= num || !regs[class].reg))
-		return rb;
-
-	rb.reg = regs[class];
-	if (gen8 && class == VIDEO_DECODE_CLASS)
-		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
-	else
-		rb.bit = engine->instance;
-
-	rb.bit = BIT(rb.bit);
-
-	return rb;
-}
-
 /*
  * HW architecture suggest typical invalidation time at 40us,
  * with pessimistic cases up to 100us and a recommendation to
@@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
  * but are now considered MCR registers.  Since they exist within a GAM range,
  * the primary instance of the register rolls up the status from each unit.
  */
-static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
+static int wait_for_invalidate(struct intel_engine_cs *engine)
 {
-	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
-		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
+	if (engine->tlb_inv.mcr)
+		return intel_gt_mcr_wait_for_reg(engine->gt,
+						 engine->tlb_inv.reg.mcr_reg,
+						 engine->tlb_inv.done,
+						 0,
 						 TLB_INVAL_TIMEOUT_US,
 						 TLB_INVAL_TIMEOUT_MS);
 	else
-		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
+		return __intel_wait_for_register_fw(engine->gt->uncore,
+						    engine->tlb_inv.reg.reg,
+						    engine->tlb_inv.done,
+						    0,
 						    TLB_INVAL_TIMEOUT_US,
 						    TLB_INVAL_TIMEOUT_MS,
 						    NULL);
@@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
 
 static void mmio_invalidate_full(struct intel_gt *gt)
 {
-	static const i915_reg_t gen8_regs[] = {
-		[RENDER_CLASS]			= GEN8_RTCR,
-		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
-		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
-	};
-	static const i915_reg_t gen12_regs[] = {
-		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
-	};
-	static const i915_mcr_reg_t xehp_regs[] = {
-		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
-	};
 	struct drm_i915_private *i915 = gt->i915;
 	struct intel_uncore *uncore = gt->uncore;
 	struct intel_engine_cs *engine;
 	intel_engine_mask_t awake, tmp;
 	enum intel_engine_id id;
-	const i915_reg_t *regs;
-	unsigned int num = 0;
 	unsigned long flags;
 
-	/*
-	 * New platforms should not be added with catch-all-newer (>=)
-	 * condition so that any later platform added triggers the below warning
-	 * and in turn mandates a human cross-check of whether the invalidation
-	 * flows have compatible semantics.
-	 *
-	 * For instance with the 11.00 -> 12.00 transition three out of five
-	 * respective engine registers were moved to masked type. Then after the
-	 * 12.00 -> 12.50 transition multi cast handling is required too.
-	 */
-
-	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
-		regs = NULL;
-		num = ARRAY_SIZE(xehp_regs);
-	} else if (GRAPHICS_VER(i915) == 12) {
-		regs = gen12_regs;
-		num = ARRAY_SIZE(gen12_regs);
-	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
-		regs = gen8_regs;
-		num = ARRAY_SIZE(gen8_regs);
-	} else if (GRAPHICS_VER(i915) < 8) {
-		return;
-	}
-
-	if (drm_WARN_ONCE(&i915->drm, !num,
-			  "Platform does not implement TLB invalidation!"))
+	if (GRAPHICS_VER(i915) < 8)
 		return;
 
 	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
@@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 
 	awake = 0;
 	for_each_engine(engine, gt, id) {
-		struct reg_and_bit rb;
-
 		if (!intel_engine_pm_is_awake(engine))
 			continue;
 
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			u32 val = BIT(engine->instance);
-
-			if (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS)
-				val = _MASKED_BIT_ENABLE(val);
+		if (engine->tlb_inv.mcr)
 			intel_gt_mcr_multicast_write_fw(gt,
-							xehp_regs[engine->class],
-							val);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-			if (!i915_mmio_reg_offset(rb.reg))
-				continue;
-
-			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS))
-				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
-
-			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
-		}
+							engine->tlb_inv.reg.mcr_reg,
+							engine->tlb_inv.request);
+		else
+			intel_uncore_write_fw(uncore,
+					      engine->tlb_inv.reg.reg,
+					      engine->tlb_inv.request);
+
 		awake |= engine->mask;
 	}
 
@@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 	intel_gt_mcr_unlock(gt, flags);
 
 	for_each_engine_masked(engine, gt, awake, tmp) {
-		struct reg_and_bit rb;
-
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			rb.mcr_reg = xehp_regs[engine->class];
-			rb.bit = BIT(engine->instance);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-		}
-
-		if (wait_for_invalidate(gt, rb))
+		if (wait_for_invalidate(engine))
 			drm_err_ratelimited(&gt->i915->drm,
 					    "%s TLB invalidation did not complete in %ums!\n",
 					    engine->name, TLB_INVAL_TIMEOUT_MS);
-- 
2.34.1

[Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As the logic for selecting the register and corresponsing values grew, the
code become a bit unsightly. Consolidate by storing the required values at
engine init time in the engine itself, and by doing so minimise the amount
of invariant platform and engine checks during each and every TLB
invalidation.

v2:
 * Fail engine probe if TLB invlidations registers are unknown.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Andrzej Hajda <andrzej.hajda@intel.com>
Cc: Matt Roper <matthew.d.roper@intel.com>
Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
 drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
 drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
 3 files changed, 128 insertions(+), 115 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index 99c4b866addd..d47dadfc25c8 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
 	return ret;
 }
 
+static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
+{
+	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
+		[RENDER_CLASS].reg		= GEN8_RTCR,
+		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
+		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
+	};
+	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
+		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
+	};
+	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
+		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
+	};
+	struct drm_i915_private *i915 = engine->i915;
+	const union intel_engine_tlb_inv_reg *regs;
+	union intel_engine_tlb_inv_reg reg;
+	unsigned int class = engine->class;
+	unsigned int num = 0;
+	u32 val;
+
+	/*
+	 * New platforms should not be added with catch-all-newer (>=)
+	 * condition so that any later platform added triggers the below warning
+	 * and in turn mandates a human cross-check of whether the invalidation
+	 * flows have compatible semantics.
+	 *
+	 * For instance with the 11.00 -> 12.00 transition three out of five
+	 * respective engine registers were moved to masked type. Then after the
+	 * 12.00 -> 12.50 transition multi cast handling is required too.
+	 */
+
+	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
+		regs = xehp_regs;
+		num = ARRAY_SIZE(xehp_regs);
+	} else if (GRAPHICS_VER(i915) == 12) {
+		regs = gen12_regs;
+		num = ARRAY_SIZE(gen12_regs);
+	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
+		regs = gen8_regs;
+		num = ARRAY_SIZE(gen8_regs);
+	} else if (GRAPHICS_VER(i915) < 8) {
+		return 0;
+	}
+
+	if (drm_WARN_ONCE(&i915->drm, !num,
+			  "Platform does not implement TLB invalidation!"))
+		return -ENODEV;
+
+	if (drm_WARN_ON_ONCE(&i915->drm,
+			     class >= num ||
+			     (!regs[class].reg.reg &&
+			      !regs[class].mcr_reg.reg)))
+		return -ERANGE;
+
+	reg = regs[class];
+
+	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
+		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
+		val = 0;
+	} else {
+		val = engine->instance;
+	}
+
+	val = BIT(val);
+
+	engine->tlb_inv.mcr = regs == xehp_regs;
+	engine->tlb_inv.reg = reg;
+	engine->tlb_inv.done = val;
+
+	if (GRAPHICS_VER(i915) >= 12 &&
+	    (engine->class == VIDEO_DECODE_CLASS ||
+	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
+	     engine->class == COMPUTE_CLASS))
+		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
+	else
+		engine->tlb_inv.request = val;
+
+	return 0;
+}
+
 static int engine_setup_common(struct intel_engine_cs *engine)
 {
 	int err;
 
 	init_llist_head(&engine->barrier_tasks);
 
+	err = intel_engine_init_tlb_invalidation(engine);
+	if (err)
+		return err;
+
 	err = init_status_page(engine);
 	if (err)
 		return err;
diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
index 4fd54fb8810f..8c661fe89314 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
+++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
@@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
 	u64 start_gt_clk;
 };
 
+union intel_engine_tlb_inv_reg {
+	i915_reg_t	reg;
+	i915_mcr_reg_t	mcr_reg;
+};
+
+struct intel_engine_tlb_inv
+{
+	bool mcr;
+	union intel_engine_tlb_inv_reg reg;
+	u32 request;
+	u32 done;
+};
+
 struct intel_engine_cs {
 	struct drm_i915_private *i915;
 	struct intel_gt *gt;
@@ -372,6 +385,8 @@ struct intel_engine_cs {
 	u32 context_size;
 	u32 mmio_base;
 
+	struct intel_engine_tlb_inv tlb_inv;
+
 	/*
 	 * Some w/a require forcewake to be held (which prevents RC6) while
 	 * a particular engine is active. If so, we set fw_domain to which
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 854841a731cb..9fb0ac03f51a 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
 	intel_sseu_dump(&info->sseu, p);
 }
 
-struct reg_and_bit {
-	union {
-		i915_reg_t reg;
-		i915_mcr_reg_t mcr_reg;
-	};
-	u32 bit;
-};
-
-static struct reg_and_bit
-get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
-		const i915_reg_t *regs, const unsigned int num)
-{
-	const unsigned int class = engine->class;
-	struct reg_and_bit rb = { };
-
-	if (drm_WARN_ON_ONCE(&engine->i915->drm,
-			     class >= num || !regs[class].reg))
-		return rb;
-
-	rb.reg = regs[class];
-	if (gen8 && class == VIDEO_DECODE_CLASS)
-		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
-	else
-		rb.bit = engine->instance;
-
-	rb.bit = BIT(rb.bit);
-
-	return rb;
-}
-
 /*
  * HW architecture suggest typical invalidation time at 40us,
  * with pessimistic cases up to 100us and a recommendation to
@@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
  * but are now considered MCR registers.  Since they exist within a GAM range,
  * the primary instance of the register rolls up the status from each unit.
  */
-static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
+static int wait_for_invalidate(struct intel_engine_cs *engine)
 {
-	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
-		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
+	if (engine->tlb_inv.mcr)
+		return intel_gt_mcr_wait_for_reg(engine->gt,
+						 engine->tlb_inv.reg.mcr_reg,
+						 engine->tlb_inv.done,
+						 0,
 						 TLB_INVAL_TIMEOUT_US,
 						 TLB_INVAL_TIMEOUT_MS);
 	else
-		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
+		return __intel_wait_for_register_fw(engine->gt->uncore,
+						    engine->tlb_inv.reg.reg,
+						    engine->tlb_inv.done,
+						    0,
 						    TLB_INVAL_TIMEOUT_US,
 						    TLB_INVAL_TIMEOUT_MS,
 						    NULL);
@@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
 
 static void mmio_invalidate_full(struct intel_gt *gt)
 {
-	static const i915_reg_t gen8_regs[] = {
-		[RENDER_CLASS]			= GEN8_RTCR,
-		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
-		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
-	};
-	static const i915_reg_t gen12_regs[] = {
-		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
-	};
-	static const i915_mcr_reg_t xehp_regs[] = {
-		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
-	};
 	struct drm_i915_private *i915 = gt->i915;
 	struct intel_uncore *uncore = gt->uncore;
 	struct intel_engine_cs *engine;
 	intel_engine_mask_t awake, tmp;
 	enum intel_engine_id id;
-	const i915_reg_t *regs;
-	unsigned int num = 0;
 	unsigned long flags;
 
-	/*
-	 * New platforms should not be added with catch-all-newer (>=)
-	 * condition so that any later platform added triggers the below warning
-	 * and in turn mandates a human cross-check of whether the invalidation
-	 * flows have compatible semantics.
-	 *
-	 * For instance with the 11.00 -> 12.00 transition three out of five
-	 * respective engine registers were moved to masked type. Then after the
-	 * 12.00 -> 12.50 transition multi cast handling is required too.
-	 */
-
-	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
-		regs = NULL;
-		num = ARRAY_SIZE(xehp_regs);
-	} else if (GRAPHICS_VER(i915) == 12) {
-		regs = gen12_regs;
-		num = ARRAY_SIZE(gen12_regs);
-	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
-		regs = gen8_regs;
-		num = ARRAY_SIZE(gen8_regs);
-	} else if (GRAPHICS_VER(i915) < 8) {
-		return;
-	}
-
-	if (drm_WARN_ONCE(&i915->drm, !num,
-			  "Platform does not implement TLB invalidation!"))
+	if (GRAPHICS_VER(i915) < 8)
 		return;
 
 	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
@@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 
 	awake = 0;
 	for_each_engine(engine, gt, id) {
-		struct reg_and_bit rb;
-
 		if (!intel_engine_pm_is_awake(engine))
 			continue;
 
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			u32 val = BIT(engine->instance);
-
-			if (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS)
-				val = _MASKED_BIT_ENABLE(val);
+		if (engine->tlb_inv.mcr)
 			intel_gt_mcr_multicast_write_fw(gt,
-							xehp_regs[engine->class],
-							val);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-			if (!i915_mmio_reg_offset(rb.reg))
-				continue;
-
-			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS))
-				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
-
-			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
-		}
+							engine->tlb_inv.reg.mcr_reg,
+							engine->tlb_inv.request);
+		else
+			intel_uncore_write_fw(uncore,
+					      engine->tlb_inv.reg.reg,
+					      engine->tlb_inv.request);
+
 		awake |= engine->mask;
 	}
 
@@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 	intel_gt_mcr_unlock(gt, flags);
 
 	for_each_engine_masked(engine, gt, awake, tmp) {
-		struct reg_and_bit rb;
-
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			rb.mcr_reg = xehp_regs[engine->class];
-			rb.bit = BIT(engine->instance);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-		}
-
-		if (wait_for_invalidate(gt, rb))
+		if (wait_for_invalidate(engine))
 			drm_err_ratelimited(&gt->i915->drm,
 					    "%s TLB invalidation did not complete in %ums!\n",
 					    engine->name, TLB_INVAL_TIMEOUT_MS);
-- 
2.34.1

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Andrzej Hajda]

On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> 
> As the logic for selecting the register and corresponsing values grew, the

corresponding

> code become a bit unsightly. Consolidate by storing the required values at
> engine init time in the engine itself, and by doing so minimise the amount
> of invariant platform and engine checks during each and every TLB
> invalidation.
> 
> v2:
>   * Fail engine probe if TLB invlidations registers are unknown.
> 
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> Cc: Matt Roper <matthew.d.roper@intel.com>
> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
> ---
>   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
>   drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
>   drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
>   3 files changed, 128 insertions(+), 115 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> index 99c4b866addd..d47dadfc25c8 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
>   	return ret;
>   }
>   
> +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> +{
> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> +		[RENDER_CLASS].reg		= GEN8_RTCR,
> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> +	};
> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> +	};
> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> +	};
> +	struct drm_i915_private *i915 = engine->i915;
> +	const union intel_engine_tlb_inv_reg *regs;
> +	union intel_engine_tlb_inv_reg reg;
> +	unsigned int class = engine->class;
> +	unsigned int num = 0;
> +	u32 val;
> +
> +	/*
> +	 * New platforms should not be added with catch-all-newer (>=)
> +	 * condition so that any later platform added triggers the below warning
> +	 * and in turn mandates a human cross-check of whether the invalidation
> +	 * flows have compatible semantics.
> +	 *
> +	 * For instance with the 11.00 -> 12.00 transition three out of five
> +	 * respective engine registers were moved to masked type. Then after the
> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> +	 */
> +
> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> +		regs = xehp_regs;
> +		num = ARRAY_SIZE(xehp_regs);
> +	} else if (GRAPHICS_VER(i915) == 12) {
> +		regs = gen12_regs;
> +		num = ARRAY_SIZE(gen12_regs);
> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> +		regs = gen8_regs;
> +		num = ARRAY_SIZE(gen8_regs);
> +	} else if (GRAPHICS_VER(i915) < 8) {
> +		return 0;
> +	} > +
> +	if (drm_WARN_ONCE(&i915->drm, !num,
> +			  "Platform does not implement TLB invalidation!"))
> +		return -ENODEV;
> +
> +	if (drm_WARN_ON_ONCE(&i915->drm,
> +			     class >= num ||
> +			     (!regs[class].reg.reg &&
> +			      !regs[class].mcr_reg.reg)))
> +		return -ERANGE;

I hope the propagation of -ERANGE to device probe is OK.

Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>

Regards
Andrzej

> +
> +	reg = regs[class];
> +
> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> +		val = 0;
> +	} else {
> +		val = engine->instance;
> +	}
> +
> +	val = BIT(val);
> +
> +	engine->tlb_inv.mcr = regs == xehp_regs;
> +	engine->tlb_inv.reg = reg;
> +	engine->tlb_inv.done = val;
> +
> +	if (GRAPHICS_VER(i915) >= 12 &&
> +	    (engine->class == VIDEO_DECODE_CLASS ||
> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> +	     engine->class == COMPUTE_CLASS))
> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> +	else
> +		engine->tlb_inv.request = val;
> +
> +	return 0;
> +}
> +
>   static int engine_setup_common(struct intel_engine_cs *engine)
>   {
>   	int err;
>   
>   	init_llist_head(&engine->barrier_tasks);
>   
> +	err = intel_engine_init_tlb_invalidation(engine);
> +	if (err)
> +		return err;
> +
>   	err = init_status_page(engine);
>   	if (err)
>   		return err;
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> index 4fd54fb8810f..8c661fe89314 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>   	u64 start_gt_clk;
>   };
>   
> +union intel_engine_tlb_inv_reg {
> +	i915_reg_t	reg;
> +	i915_mcr_reg_t	mcr_reg;
> +};
> +
> +struct intel_engine_tlb_inv
> +{
> +	bool mcr;
> +	union intel_engine_tlb_inv_reg reg;
> +	u32 request;
> +	u32 done;
> +};
> +
>   struct intel_engine_cs {
>   	struct drm_i915_private *i915;
>   	struct intel_gt *gt;
> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>   	u32 context_size;
>   	u32 mmio_base;
>   
> +	struct intel_engine_tlb_inv tlb_inv;
> +
>   	/*
>   	 * Some w/a require forcewake to be held (which prevents RC6) while
>   	 * a particular engine is active. If so, we set fw_domain to which
> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> index 854841a731cb..9fb0ac03f51a 100644
> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>   	intel_sseu_dump(&info->sseu, p);
>   }
>   
> -struct reg_and_bit {
> -	union {
> -		i915_reg_t reg;
> -		i915_mcr_reg_t mcr_reg;
> -	};
> -	u32 bit;
> -};
> -
> -static struct reg_and_bit
> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> -		const i915_reg_t *regs, const unsigned int num)
> -{
> -	const unsigned int class = engine->class;
> -	struct reg_and_bit rb = { };
> -
> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> -			     class >= num || !regs[class].reg))
> -		return rb;
> -
> -	rb.reg = regs[class];
> -	if (gen8 && class == VIDEO_DECODE_CLASS)
> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> -	else
> -		rb.bit = engine->instance;
> -
> -	rb.bit = BIT(rb.bit);
> -
> -	return rb;
> -}
> -
>   /*
>    * HW architecture suggest typical invalidation time at 40us,
>    * with pessimistic cases up to 100us and a recommendation to
> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>    * but are now considered MCR registers.  Since they exist within a GAM range,
>    * the primary instance of the register rolls up the status from each unit.
>    */
> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>   {
> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> +	if (engine->tlb_inv.mcr)
> +		return intel_gt_mcr_wait_for_reg(engine->gt,
> +						 engine->tlb_inv.reg.mcr_reg,
> +						 engine->tlb_inv.done,
> +						 0,
>   						 TLB_INVAL_TIMEOUT_US,
>   						 TLB_INVAL_TIMEOUT_MS);
>   	else
> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> +		return __intel_wait_for_register_fw(engine->gt->uncore,
> +						    engine->tlb_inv.reg.reg,
> +						    engine->tlb_inv.done,
> +						    0,
>   						    TLB_INVAL_TIMEOUT_US,
>   						    TLB_INVAL_TIMEOUT_MS,
>   						    NULL);
> @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>   
>   static void mmio_invalidate_full(struct intel_gt *gt)
>   {
> -	static const i915_reg_t gen8_regs[] = {
> -		[RENDER_CLASS]			= GEN8_RTCR,
> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> -	};
> -	static const i915_reg_t gen12_regs[] = {
> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> -	};
> -	static const i915_mcr_reg_t xehp_regs[] = {
> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> -	};
>   	struct drm_i915_private *i915 = gt->i915;
>   	struct intel_uncore *uncore = gt->uncore;
>   	struct intel_engine_cs *engine;
>   	intel_engine_mask_t awake, tmp;
>   	enum intel_engine_id id;
> -	const i915_reg_t *regs;
> -	unsigned int num = 0;
>   	unsigned long flags;
>   
> -	/*
> -	 * New platforms should not be added with catch-all-newer (>=)
> -	 * condition so that any later platform added triggers the below warning
> -	 * and in turn mandates a human cross-check of whether the invalidation
> -	 * flows have compatible semantics.
> -	 *
> -	 * For instance with the 11.00 -> 12.00 transition three out of five
> -	 * respective engine registers were moved to masked type. Then after the
> -	 * 12.00 -> 12.50 transition multi cast handling is required too.
> -	 */
> -
> -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> -		regs = NULL;
> -		num = ARRAY_SIZE(xehp_regs);
> -	} else if (GRAPHICS_VER(i915) == 12) {
> -		regs = gen12_regs;
> -		num = ARRAY_SIZE(gen12_regs);
> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> -		regs = gen8_regs;
> -		num = ARRAY_SIZE(gen8_regs);
> -	} else if (GRAPHICS_VER(i915) < 8) {
> -		return;
> -	}
> -
> -	if (drm_WARN_ONCE(&i915->drm, !num,
> -			  "Platform does not implement TLB invalidation!"))
> +	if (GRAPHICS_VER(i915) < 8)
>   		return;
>   
>   	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>   
>   	awake = 0;
>   	for_each_engine(engine, gt, id) {
> -		struct reg_and_bit rb;
> -
>   		if (!intel_engine_pm_is_awake(engine))
>   			continue;
>   
> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> -			u32 val = BIT(engine->instance);
> -
> -			if (engine->class == VIDEO_DECODE_CLASS ||
> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> -			    engine->class == COMPUTE_CLASS)
> -				val = _MASKED_BIT_ENABLE(val);
> +		if (engine->tlb_inv.mcr)
>   			intel_gt_mcr_multicast_write_fw(gt,
> -							xehp_regs[engine->class],
> -							val);
> -		} else {
> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> -			if (!i915_mmio_reg_offset(rb.reg))
> -				continue;
> -
> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> -			    engine->class == COMPUTE_CLASS))
> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> -
> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> -		}
> +							engine->tlb_inv.reg.mcr_reg,
> +							engine->tlb_inv.request);
> +		else
> +			intel_uncore_write_fw(uncore,
> +					      engine->tlb_inv.reg.reg,
> +					      engine->tlb_inv.request);
> +
>   		awake |= engine->mask;
>   	}
>   
> @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>   	intel_gt_mcr_unlock(gt, flags);
>   
>   	for_each_engine_masked(engine, gt, awake, tmp) {
> -		struct reg_and_bit rb;
> -
> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> -			rb.mcr_reg = xehp_regs[engine->class];
> -			rb.bit = BIT(engine->instance);
> -		} else {
> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> -		}
> -
> -		if (wait_for_invalidate(gt, rb))
> +		if (wait_for_invalidate(engine))
>   			drm_err_ratelimited(&gt->i915->drm,
>   					    "%s TLB invalidation did not complete in %ums!\n",
>   					    engine->name, TLB_INVAL_TIMEOUT_MS);

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Matt Roper]

On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> > From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > 
> > As the logic for selecting the register and corresponsing values grew, the
> 
> corresponding
> 
> > code become a bit unsightly. Consolidate by storing the required values at
> > engine init time in the engine itself, and by doing so minimise the amount
> > of invariant platform and engine checks during each and every TLB
> > invalidation.
> > 
> > v2:
> >   * Fail engine probe if TLB invlidations registers are unknown.
> > 
> > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> > Cc: Matt Roper <matthew.d.roper@intel.com>
> > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
> > ---
> >   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
> >   drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
> >   drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
> >   3 files changed, 128 insertions(+), 115 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > index 99c4b866addd..d47dadfc25c8 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
> >   	return ret;
> >   }
> > +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> > +{
> > +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> > +		[RENDER_CLASS].reg		= GEN8_RTCR,
> > +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> > +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> > +	};
> > +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> > +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> > +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> > +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> > +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> > +	};
> > +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> > +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> > +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> > +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> > +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> > +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> > +	};
> > +	struct drm_i915_private *i915 = engine->i915;
> > +	const union intel_engine_tlb_inv_reg *regs;
> > +	union intel_engine_tlb_inv_reg reg;
> > +	unsigned int class = engine->class;
> > +	unsigned int num = 0;
> > +	u32 val;
> > +
> > +	/*
> > +	 * New platforms should not be added with catch-all-newer (>=)
> > +	 * condition so that any later platform added triggers the below warning
> > +	 * and in turn mandates a human cross-check of whether the invalidation
> > +	 * flows have compatible semantics.
> > +	 *
> > +	 * For instance with the 11.00 -> 12.00 transition three out of five
> > +	 * respective engine registers were moved to masked type. Then after the
> > +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > +	 */
> > +
> > +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {

This is bad...it only captures XEHPSDV and breaks the handling of DG2
(12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
hitting the warning as expected since those are all now being captured
by the next case of the if/else ladder.  With the way GMD_ID works, we
may also get new version numbers that silently show up in hardware too
at some point (e.g., 12.73, 12.74, etc.)

> > +		regs = xehp_regs;
> > +		num = ARRAY_SIZE(xehp_regs);
> > +	} else if (GRAPHICS_VER(i915) == 12) {

You'd want to change this to 

        GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)

to get the behavior you expected.


Matt

> > +		regs = gen12_regs;
> > +		num = ARRAY_SIZE(gen12_regs);
> > +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > +		regs = gen8_regs;
> > +		num = ARRAY_SIZE(gen8_regs);
> > +	} else if (GRAPHICS_VER(i915) < 8) {
> > +		return 0;
> > +	} > +
> > +	if (drm_WARN_ONCE(&i915->drm, !num,
> > +			  "Platform does not implement TLB invalidation!"))
> > +		return -ENODEV;
> > +
> > +	if (drm_WARN_ON_ONCE(&i915->drm,
> > +			     class >= num ||
> > +			     (!regs[class].reg.reg &&
> > +			      !regs[class].mcr_reg.reg)))
> > +		return -ERANGE;
> 
> I hope the propagation of -ERANGE to device probe is OK.
> 
> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
> 
> Regards
> Andrzej
> 
> > +
> > +	reg = regs[class];
> > +
> > +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> > +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > +		val = 0;
> > +	} else {
> > +		val = engine->instance;
> > +	}
> > +
> > +	val = BIT(val);
> > +
> > +	engine->tlb_inv.mcr = regs == xehp_regs;
> > +	engine->tlb_inv.reg = reg;
> > +	engine->tlb_inv.done = val;
> > +
> > +	if (GRAPHICS_VER(i915) >= 12 &&
> > +	    (engine->class == VIDEO_DECODE_CLASS ||
> > +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > +	     engine->class == COMPUTE_CLASS))
> > +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> > +	else
> > +		engine->tlb_inv.request = val;
> > +
> > +	return 0;
> > +}
> > +
> >   static int engine_setup_common(struct intel_engine_cs *engine)
> >   {
> >   	int err;
> >   	init_llist_head(&engine->barrier_tasks);
> > +	err = intel_engine_init_tlb_invalidation(engine);
> > +	if (err)
> > +		return err;
> > +
> >   	err = init_status_page(engine);
> >   	if (err)
> >   		return err;
> > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > index 4fd54fb8810f..8c661fe89314 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
> >   	u64 start_gt_clk;
> >   };
> > +union intel_engine_tlb_inv_reg {
> > +	i915_reg_t	reg;
> > +	i915_mcr_reg_t	mcr_reg;
> > +};
> > +
> > +struct intel_engine_tlb_inv
> > +{
> > +	bool mcr;
> > +	union intel_engine_tlb_inv_reg reg;
> > +	u32 request;
> > +	u32 done;
> > +};
> > +
> >   struct intel_engine_cs {
> >   	struct drm_i915_private *i915;
> >   	struct intel_gt *gt;
> > @@ -372,6 +385,8 @@ struct intel_engine_cs {
> >   	u32 context_size;
> >   	u32 mmio_base;
> > +	struct intel_engine_tlb_inv tlb_inv;
> > +
> >   	/*
> >   	 * Some w/a require forcewake to be held (which prevents RC6) while
> >   	 * a particular engine is active. If so, we set fw_domain to which
> > diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> > index 854841a731cb..9fb0ac03f51a 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> > +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> > @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
> >   	intel_sseu_dump(&info->sseu, p);
> >   }
> > -struct reg_and_bit {
> > -	union {
> > -		i915_reg_t reg;
> > -		i915_mcr_reg_t mcr_reg;
> > -	};
> > -	u32 bit;
> > -};
> > -
> > -static struct reg_and_bit
> > -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > -		const i915_reg_t *regs, const unsigned int num)
> > -{
> > -	const unsigned int class = engine->class;
> > -	struct reg_and_bit rb = { };
> > -
> > -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> > -			     class >= num || !regs[class].reg))
> > -		return rb;
> > -
> > -	rb.reg = regs[class];
> > -	if (gen8 && class == VIDEO_DECODE_CLASS)
> > -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > -	else
> > -		rb.bit = engine->instance;
> > -
> > -	rb.bit = BIT(rb.bit);
> > -
> > -	return rb;
> > -}
> > -
> >   /*
> >    * HW architecture suggest typical invalidation time at 40us,
> >    * with pessimistic cases up to 100us and a recommendation to
> > @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> >    * but are now considered MCR registers.  Since they exist within a GAM range,
> >    * the primary instance of the register rolls up the status from each unit.
> >    */
> > -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > +static int wait_for_invalidate(struct intel_engine_cs *engine)
> >   {
> > -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> > -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> > +	if (engine->tlb_inv.mcr)
> > +		return intel_gt_mcr_wait_for_reg(engine->gt,
> > +						 engine->tlb_inv.reg.mcr_reg,
> > +						 engine->tlb_inv.done,
> > +						 0,
> >   						 TLB_INVAL_TIMEOUT_US,
> >   						 TLB_INVAL_TIMEOUT_MS);
> >   	else
> > -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> > +		return __intel_wait_for_register_fw(engine->gt->uncore,
> > +						    engine->tlb_inv.reg.reg,
> > +						    engine->tlb_inv.done,
> > +						    0,
> >   						    TLB_INVAL_TIMEOUT_US,
> >   						    TLB_INVAL_TIMEOUT_MS,
> >   						    NULL);
> > @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> >   static void mmio_invalidate_full(struct intel_gt *gt)
> >   {
> > -	static const i915_reg_t gen8_regs[] = {
> > -		[RENDER_CLASS]			= GEN8_RTCR,
> > -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> > -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> > -	};
> > -	static const i915_reg_t gen12_regs[] = {
> > -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> > -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> > -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> > -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> > -	};
> > -	static const i915_mcr_reg_t xehp_regs[] = {
> > -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> > -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> > -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> > -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> > -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> > -	};
> >   	struct drm_i915_private *i915 = gt->i915;
> >   	struct intel_uncore *uncore = gt->uncore;
> >   	struct intel_engine_cs *engine;
> >   	intel_engine_mask_t awake, tmp;
> >   	enum intel_engine_id id;
> > -	const i915_reg_t *regs;
> > -	unsigned int num = 0;
> >   	unsigned long flags;
> > -	/*
> > -	 * New platforms should not be added with catch-all-newer (>=)
> > -	 * condition so that any later platform added triggers the below warning
> > -	 * and in turn mandates a human cross-check of whether the invalidation
> > -	 * flows have compatible semantics.
> > -	 *
> > -	 * For instance with the 11.00 -> 12.00 transition three out of five
> > -	 * respective engine registers were moved to masked type. Then after the
> > -	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > -	 */
> > -
> > -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > -		regs = NULL;
> > -		num = ARRAY_SIZE(xehp_regs);
> > -	} else if (GRAPHICS_VER(i915) == 12) {
> > -		regs = gen12_regs;
> > -		num = ARRAY_SIZE(gen12_regs);
> > -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > -		regs = gen8_regs;
> > -		num = ARRAY_SIZE(gen8_regs);
> > -	} else if (GRAPHICS_VER(i915) < 8) {
> > -		return;
> > -	}
> > -
> > -	if (drm_WARN_ONCE(&i915->drm, !num,
> > -			  "Platform does not implement TLB invalidation!"))
> > +	if (GRAPHICS_VER(i915) < 8)
> >   		return;
> >   	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> > @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> >   	awake = 0;
> >   	for_each_engine(engine, gt, id) {
> > -		struct reg_and_bit rb;
> > -
> >   		if (!intel_engine_pm_is_awake(engine))
> >   			continue;
> > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > -			u32 val = BIT(engine->instance);
> > -
> > -			if (engine->class == VIDEO_DECODE_CLASS ||
> > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > -			    engine->class == COMPUTE_CLASS)
> > -				val = _MASKED_BIT_ENABLE(val);
> > +		if (engine->tlb_inv.mcr)
> >   			intel_gt_mcr_multicast_write_fw(gt,
> > -							xehp_regs[engine->class],
> > -							val);
> > -		} else {
> > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > -			if (!i915_mmio_reg_offset(rb.reg))
> > -				continue;
> > -
> > -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > -			    engine->class == COMPUTE_CLASS))
> > -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> > -
> > -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> > -		}
> > +							engine->tlb_inv.reg.mcr_reg,
> > +							engine->tlb_inv.request);
> > +		else
> > +			intel_uncore_write_fw(uncore,
> > +					      engine->tlb_inv.reg.reg,
> > +					      engine->tlb_inv.request);
> > +
> >   		awake |= engine->mask;
> >   	}
> > @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> >   	intel_gt_mcr_unlock(gt, flags);
> >   	for_each_engine_masked(engine, gt, awake, tmp) {
> > -		struct reg_and_bit rb;
> > -
> > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > -			rb.mcr_reg = xehp_regs[engine->class];
> > -			rb.bit = BIT(engine->instance);
> > -		} else {
> > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > -		}
> > -
> > -		if (wait_for_invalidate(gt, rb))
> > +		if (wait_for_invalidate(engine))
> >   			drm_err_ratelimited(&gt->i915->drm,
> >   					    "%s TLB invalidation did not complete in %ums!\n",
> >   					    engine->name, TLB_INVAL_TIMEOUT_MS);
> 

-- 
Matt Roper
Graphics Software Engineer
VTT-OSGC Platform Enablement
Intel Corporation

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Matt Roper]

On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> > From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > 
> > As the logic for selecting the register and corresponsing values grew, the
> 
> corresponding
> 
> > code become a bit unsightly. Consolidate by storing the required values at
> > engine init time in the engine itself, and by doing so minimise the amount
> > of invariant platform and engine checks during each and every TLB
> > invalidation.
> > 
> > v2:
> >   * Fail engine probe if TLB invlidations registers are unknown.
> > 
> > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> > Cc: Matt Roper <matthew.d.roper@intel.com>
> > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
> > ---
> >   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
> >   drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
> >   drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
> >   3 files changed, 128 insertions(+), 115 deletions(-)
> > 
> > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > index 99c4b866addd..d47dadfc25c8 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
> >   	return ret;
> >   }
> > +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> > +{
> > +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> > +		[RENDER_CLASS].reg		= GEN8_RTCR,
> > +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> > +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> > +	};
> > +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> > +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> > +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> > +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> > +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> > +	};
> > +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> > +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> > +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> > +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> > +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> > +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> > +	};
> > +	struct drm_i915_private *i915 = engine->i915;
> > +	const union intel_engine_tlb_inv_reg *regs;
> > +	union intel_engine_tlb_inv_reg reg;
> > +	unsigned int class = engine->class;
> > +	unsigned int num = 0;
> > +	u32 val;
> > +
> > +	/*
> > +	 * New platforms should not be added with catch-all-newer (>=)
> > +	 * condition so that any later platform added triggers the below warning
> > +	 * and in turn mandates a human cross-check of whether the invalidation
> > +	 * flows have compatible semantics.
> > +	 *
> > +	 * For instance with the 11.00 -> 12.00 transition three out of five
> > +	 * respective engine registers were moved to masked type. Then after the
> > +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > +	 */
> > +
> > +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {

This is bad...it only captures XEHPSDV and breaks the handling of DG2
(12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
hitting the warning as expected since those are all now being captured
by the next case of the if/else ladder.  With the way GMD_ID works, we
may also get new version numbers that silently show up in hardware too
at some point (e.g., 12.73, 12.74, etc.)

> > +		regs = xehp_regs;
> > +		num = ARRAY_SIZE(xehp_regs);
> > +	} else if (GRAPHICS_VER(i915) == 12) {

You'd want to change this to 

        GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)

to get the behavior you expected.


Matt

> > +		regs = gen12_regs;
> > +		num = ARRAY_SIZE(gen12_regs);
> > +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > +		regs = gen8_regs;
> > +		num = ARRAY_SIZE(gen8_regs);
> > +	} else if (GRAPHICS_VER(i915) < 8) {
> > +		return 0;
> > +	} > +
> > +	if (drm_WARN_ONCE(&i915->drm, !num,
> > +			  "Platform does not implement TLB invalidation!"))
> > +		return -ENODEV;
> > +
> > +	if (drm_WARN_ON_ONCE(&i915->drm,
> > +			     class >= num ||
> > +			     (!regs[class].reg.reg &&
> > +			      !regs[class].mcr_reg.reg)))
> > +		return -ERANGE;
> 
> I hope the propagation of -ERANGE to device probe is OK.
> 
> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
> 
> Regards
> Andrzej
> 
> > +
> > +	reg = regs[class];
> > +
> > +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> > +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > +		val = 0;
> > +	} else {
> > +		val = engine->instance;
> > +	}
> > +
> > +	val = BIT(val);
> > +
> > +	engine->tlb_inv.mcr = regs == xehp_regs;
> > +	engine->tlb_inv.reg = reg;
> > +	engine->tlb_inv.done = val;
> > +
> > +	if (GRAPHICS_VER(i915) >= 12 &&
> > +	    (engine->class == VIDEO_DECODE_CLASS ||
> > +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > +	     engine->class == COMPUTE_CLASS))
> > +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> > +	else
> > +		engine->tlb_inv.request = val;
> > +
> > +	return 0;
> > +}
> > +
> >   static int engine_setup_common(struct intel_engine_cs *engine)
> >   {
> >   	int err;
> >   	init_llist_head(&engine->barrier_tasks);
> > +	err = intel_engine_init_tlb_invalidation(engine);
> > +	if (err)
> > +		return err;
> > +
> >   	err = init_status_page(engine);
> >   	if (err)
> >   		return err;
> > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > index 4fd54fb8810f..8c661fe89314 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
> >   	u64 start_gt_clk;
> >   };
> > +union intel_engine_tlb_inv_reg {
> > +	i915_reg_t	reg;
> > +	i915_mcr_reg_t	mcr_reg;
> > +};
> > +
> > +struct intel_engine_tlb_inv
> > +{
> > +	bool mcr;
> > +	union intel_engine_tlb_inv_reg reg;
> > +	u32 request;
> > +	u32 done;
> > +};
> > +
> >   struct intel_engine_cs {
> >   	struct drm_i915_private *i915;
> >   	struct intel_gt *gt;
> > @@ -372,6 +385,8 @@ struct intel_engine_cs {
> >   	u32 context_size;
> >   	u32 mmio_base;
> > +	struct intel_engine_tlb_inv tlb_inv;
> > +
> >   	/*
> >   	 * Some w/a require forcewake to be held (which prevents RC6) while
> >   	 * a particular engine is active. If so, we set fw_domain to which
> > diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> > index 854841a731cb..9fb0ac03f51a 100644
> > --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> > +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> > @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
> >   	intel_sseu_dump(&info->sseu, p);
> >   }
> > -struct reg_and_bit {
> > -	union {
> > -		i915_reg_t reg;
> > -		i915_mcr_reg_t mcr_reg;
> > -	};
> > -	u32 bit;
> > -};
> > -
> > -static struct reg_and_bit
> > -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > -		const i915_reg_t *regs, const unsigned int num)
> > -{
> > -	const unsigned int class = engine->class;
> > -	struct reg_and_bit rb = { };
> > -
> > -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> > -			     class >= num || !regs[class].reg))
> > -		return rb;
> > -
> > -	rb.reg = regs[class];
> > -	if (gen8 && class == VIDEO_DECODE_CLASS)
> > -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > -	else
> > -		rb.bit = engine->instance;
> > -
> > -	rb.bit = BIT(rb.bit);
> > -
> > -	return rb;
> > -}
> > -
> >   /*
> >    * HW architecture suggest typical invalidation time at 40us,
> >    * with pessimistic cases up to 100us and a recommendation to
> > @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> >    * but are now considered MCR registers.  Since they exist within a GAM range,
> >    * the primary instance of the register rolls up the status from each unit.
> >    */
> > -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > +static int wait_for_invalidate(struct intel_engine_cs *engine)
> >   {
> > -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> > -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> > +	if (engine->tlb_inv.mcr)
> > +		return intel_gt_mcr_wait_for_reg(engine->gt,
> > +						 engine->tlb_inv.reg.mcr_reg,
> > +						 engine->tlb_inv.done,
> > +						 0,
> >   						 TLB_INVAL_TIMEOUT_US,
> >   						 TLB_INVAL_TIMEOUT_MS);
> >   	else
> > -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> > +		return __intel_wait_for_register_fw(engine->gt->uncore,
> > +						    engine->tlb_inv.reg.reg,
> > +						    engine->tlb_inv.done,
> > +						    0,
> >   						    TLB_INVAL_TIMEOUT_US,
> >   						    TLB_INVAL_TIMEOUT_MS,
> >   						    NULL);
> > @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> >   static void mmio_invalidate_full(struct intel_gt *gt)
> >   {
> > -	static const i915_reg_t gen8_regs[] = {
> > -		[RENDER_CLASS]			= GEN8_RTCR,
> > -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> > -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> > -	};
> > -	static const i915_reg_t gen12_regs[] = {
> > -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> > -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> > -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> > -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> > -	};
> > -	static const i915_mcr_reg_t xehp_regs[] = {
> > -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> > -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> > -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> > -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> > -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> > -	};
> >   	struct drm_i915_private *i915 = gt->i915;
> >   	struct intel_uncore *uncore = gt->uncore;
> >   	struct intel_engine_cs *engine;
> >   	intel_engine_mask_t awake, tmp;
> >   	enum intel_engine_id id;
> > -	const i915_reg_t *regs;
> > -	unsigned int num = 0;
> >   	unsigned long flags;
> > -	/*
> > -	 * New platforms should not be added with catch-all-newer (>=)
> > -	 * condition so that any later platform added triggers the below warning
> > -	 * and in turn mandates a human cross-check of whether the invalidation
> > -	 * flows have compatible semantics.
> > -	 *
> > -	 * For instance with the 11.00 -> 12.00 transition three out of five
> > -	 * respective engine registers were moved to masked type. Then after the
> > -	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > -	 */
> > -
> > -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > -		regs = NULL;
> > -		num = ARRAY_SIZE(xehp_regs);
> > -	} else if (GRAPHICS_VER(i915) == 12) {
> > -		regs = gen12_regs;
> > -		num = ARRAY_SIZE(gen12_regs);
> > -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > -		regs = gen8_regs;
> > -		num = ARRAY_SIZE(gen8_regs);
> > -	} else if (GRAPHICS_VER(i915) < 8) {
> > -		return;
> > -	}
> > -
> > -	if (drm_WARN_ONCE(&i915->drm, !num,
> > -			  "Platform does not implement TLB invalidation!"))
> > +	if (GRAPHICS_VER(i915) < 8)
> >   		return;
> >   	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> > @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> >   	awake = 0;
> >   	for_each_engine(engine, gt, id) {
> > -		struct reg_and_bit rb;
> > -
> >   		if (!intel_engine_pm_is_awake(engine))
> >   			continue;
> > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > -			u32 val = BIT(engine->instance);
> > -
> > -			if (engine->class == VIDEO_DECODE_CLASS ||
> > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > -			    engine->class == COMPUTE_CLASS)
> > -				val = _MASKED_BIT_ENABLE(val);
> > +		if (engine->tlb_inv.mcr)
> >   			intel_gt_mcr_multicast_write_fw(gt,
> > -							xehp_regs[engine->class],
> > -							val);
> > -		} else {
> > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > -			if (!i915_mmio_reg_offset(rb.reg))
> > -				continue;
> > -
> > -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > -			    engine->class == COMPUTE_CLASS))
> > -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> > -
> > -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> > -		}
> > +							engine->tlb_inv.reg.mcr_reg,
> > +							engine->tlb_inv.request);
> > +		else
> > +			intel_uncore_write_fw(uncore,
> > +					      engine->tlb_inv.reg.reg,
> > +					      engine->tlb_inv.request);
> > +
> >   		awake |= engine->mask;
> >   	}
> > @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> >   	intel_gt_mcr_unlock(gt, flags);
> >   	for_each_engine_masked(engine, gt, awake, tmp) {
> > -		struct reg_and_bit rb;
> > -
> > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > -			rb.mcr_reg = xehp_regs[engine->class];
> > -			rb.bit = BIT(engine->instance);
> > -		} else {
> > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > -		}
> > -
> > -		if (wait_for_invalidate(gt, rb))
> > +		if (wait_for_invalidate(engine))
> >   			drm_err_ratelimited(&gt->i915->drm,
> >   					    "%s TLB invalidation did not complete in %ums!\n",
> >   					    engine->name, TLB_INVAL_TIMEOUT_MS);
> 

-- 
Matt Roper
Graphics Software Engineer
VTT-OSGC Platform Enablement
Intel Corporation

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Andrzej Hajda]

On 03.01.2023 20:57, Matt Roper wrote:
> On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
>> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
>>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>
>>> As the logic for selecting the register and corresponsing values grew, the
>>
>> corresponding
>>
>>> code become a bit unsightly. Consolidate by storing the required values at
>>> engine init time in the engine itself, and by doing so minimise the amount
>>> of invariant platform and engine checks during each and every TLB
>>> invalidation.
>>>
>>> v2:
>>>    * Fail engine probe if TLB invlidations registers are unknown.
>>>
>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>>> Cc: Matt Roper <matthew.d.roper@intel.com>
>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
>>> ---
>>>    drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
>>>    drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
>>>    drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
>>>    3 files changed, 128 insertions(+), 115 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> index 99c4b866addd..d47dadfc25c8 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
>>>    	return ret;
>>>    }
>>> +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
>>> +{
>>> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>>> +		[RENDER_CLASS].reg		= GEN8_RTCR,
>>> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
>>> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
>>> +	};
>>> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>>> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
>>> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
>>> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
>>> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
>>> +	};
>>> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>>> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
>>> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
>>> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>>> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
>>> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
>>> +	};
>>> +	struct drm_i915_private *i915 = engine->i915;
>>> +	const union intel_engine_tlb_inv_reg *regs;
>>> +	union intel_engine_tlb_inv_reg reg;
>>> +	unsigned int class = engine->class;
>>> +	unsigned int num = 0;
>>> +	u32 val;
>>> +
>>> +	/*
>>> +	 * New platforms should not be added with catch-all-newer (>=)
>>> +	 * condition so that any later platform added triggers the below warning
>>> +	 * and in turn mandates a human cross-check of whether the invalidation
>>> +	 * flows have compatible semantics.
>>> +	 *
>>> +	 * For instance with the 11.00 -> 12.00 transition three out of five
>>> +	 * respective engine registers were moved to masked type. Then after the
>>> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>> +	 */
>>> +
>>> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> 
> This is bad...it only captures XEHPSDV and breaks the handling of DG2
> (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
> hitting the warning as expected since those are all now being captured
> by the next case of the if/else ladder.  With the way GMD_ID works, we
> may also get new version numbers that silently show up in hardware too
> at some point (e.g., 12.73, 12.74, etc.)
> 
>>> +		regs = xehp_regs;
>>> +		num = ARRAY_SIZE(xehp_regs);
>>> +	} else if (GRAPHICS_VER(i915) == 12) {
> 
> You'd want to change this to
> 
>          GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
> 
> to get the behavior you expected.

According to dg1_info dg1 has IP_VER(12, 10), it will not fit into this 
bucket.

Regards
Andrzej

> 
> 
> Matt
> 
>>> +		regs = gen12_regs;
>>> +		num = ARRAY_SIZE(gen12_regs);
>>> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>> +		regs = gen8_regs;
>>> +		num = ARRAY_SIZE(gen8_regs);
>>> +	} else if (GRAPHICS_VER(i915) < 8) {
>>> +		return 0;
>>> +	} > +
>>> +	if (drm_WARN_ONCE(&i915->drm, !num,
>>> +			  "Platform does not implement TLB invalidation!"))
>>> +		return -ENODEV;
>>> +
>>> +	if (drm_WARN_ON_ONCE(&i915->drm,
>>> +			     class >= num ||
>>> +			     (!regs[class].reg.reg &&
>>> +			      !regs[class].mcr_reg.reg)))
>>> +		return -ERANGE;
>>
>> I hope the propagation of -ERANGE to device probe is OK.
>>
>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
>>
>> Regards
>> Andrzej
>>
>>> +
>>> +	reg = regs[class];
>>> +
>>> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>>> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> +		val = 0;
>>> +	} else {
>>> +		val = engine->instance;
>>> +	}
>>> +
>>> +	val = BIT(val);
>>> +
>>> +	engine->tlb_inv.mcr = regs == xehp_regs;
>>> +	engine->tlb_inv.reg = reg;
>>> +	engine->tlb_inv.done = val;
>>> +
>>> +	if (GRAPHICS_VER(i915) >= 12 &&
>>> +	    (engine->class == VIDEO_DECODE_CLASS ||
>>> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> +	     engine->class == COMPUTE_CLASS))
>>> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>>> +	else
>>> +		engine->tlb_inv.request = val;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>>    static int engine_setup_common(struct intel_engine_cs *engine)
>>>    {
>>>    	int err;
>>>    	init_llist_head(&engine->barrier_tasks);
>>> +	err = intel_engine_init_tlb_invalidation(engine);
>>> +	if (err)
>>> +		return err;
>>> +
>>>    	err = init_status_page(engine);
>>>    	if (err)
>>>    		return err;
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> index 4fd54fb8810f..8c661fe89314 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>>    	u64 start_gt_clk;
>>>    };
>>> +union intel_engine_tlb_inv_reg {
>>> +	i915_reg_t	reg;
>>> +	i915_mcr_reg_t	mcr_reg;
>>> +};
>>> +
>>> +struct intel_engine_tlb_inv
>>> +{
>>> +	bool mcr;
>>> +	union intel_engine_tlb_inv_reg reg;
>>> +	u32 request;
>>> +	u32 done;
>>> +};
>>> +
>>>    struct intel_engine_cs {
>>>    	struct drm_i915_private *i915;
>>>    	struct intel_gt *gt;
>>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>>    	u32 context_size;
>>>    	u32 mmio_base;
>>> +	struct intel_engine_tlb_inv tlb_inv;
>>> +
>>>    	/*
>>>    	 * Some w/a require forcewake to be held (which prevents RC6) while
>>>    	 * a particular engine is active. If so, we set fw_domain to which
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> index 854841a731cb..9fb0ac03f51a 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>>>    	intel_sseu_dump(&info->sseu, p);
>>>    }
>>> -struct reg_and_bit {
>>> -	union {
>>> -		i915_reg_t reg;
>>> -		i915_mcr_reg_t mcr_reg;
>>> -	};
>>> -	u32 bit;
>>> -};
>>> -
>>> -static struct reg_and_bit
>>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>> -		const i915_reg_t *regs, const unsigned int num)
>>> -{
>>> -	const unsigned int class = engine->class;
>>> -	struct reg_and_bit rb = { };
>>> -
>>> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
>>> -			     class >= num || !regs[class].reg))
>>> -		return rb;
>>> -
>>> -	rb.reg = regs[class];
>>> -	if (gen8 && class == VIDEO_DECODE_CLASS)
>>> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> -	else
>>> -		rb.bit = engine->instance;
>>> -
>>> -	rb.bit = BIT(rb.bit);
>>> -
>>> -	return rb;
>>> -}
>>> -
>>>    /*
>>>     * HW architecture suggest typical invalidation time at 40us,
>>>     * with pessimistic cases up to 100us and a recommendation to
>>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>     * but are now considered MCR registers.  Since they exist within a GAM range,
>>>     * the primary instance of the register rolls up the status from each unit.
>>>     */
>>> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>>    {
>>> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>>> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>>> +	if (engine->tlb_inv.mcr)
>>> +		return intel_gt_mcr_wait_for_reg(engine->gt,
>>> +						 engine->tlb_inv.reg.mcr_reg,
>>> +						 engine->tlb_inv.done,
>>> +						 0,
>>>    						 TLB_INVAL_TIMEOUT_US,
>>>    						 TLB_INVAL_TIMEOUT_MS);
>>>    	else
>>> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
>>> +		return __intel_wait_for_register_fw(engine->gt->uncore,
>>> +						    engine->tlb_inv.reg.reg,
>>> +						    engine->tlb_inv.done,
>>> +						    0,
>>>    						    TLB_INVAL_TIMEOUT_US,
>>>    						    TLB_INVAL_TIMEOUT_MS,
>>>    						    NULL);
>>> @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>    static void mmio_invalidate_full(struct intel_gt *gt)
>>>    {
>>> -	static const i915_reg_t gen8_regs[] = {
>>> -		[RENDER_CLASS]			= GEN8_RTCR,
>>> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
>>> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
>>> -	};
>>> -	static const i915_reg_t gen12_regs[] = {
>>> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
>>> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
>>> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
>>> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
>>> -	};
>>> -	static const i915_mcr_reg_t xehp_regs[] = {
>>> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
>>> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
>>> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
>>> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
>>> -	};
>>>    	struct drm_i915_private *i915 = gt->i915;
>>>    	struct intel_uncore *uncore = gt->uncore;
>>>    	struct intel_engine_cs *engine;
>>>    	intel_engine_mask_t awake, tmp;
>>>    	enum intel_engine_id id;
>>> -	const i915_reg_t *regs;
>>> -	unsigned int num = 0;
>>>    	unsigned long flags;
>>> -	/*
>>> -	 * New platforms should not be added with catch-all-newer (>=)
>>> -	 * condition so that any later platform added triggers the below warning
>>> -	 * and in turn mandates a human cross-check of whether the invalidation
>>> -	 * flows have compatible semantics.
>>> -	 *
>>> -	 * For instance with the 11.00 -> 12.00 transition three out of five
>>> -	 * respective engine registers were moved to masked type. Then after the
>>> -	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>> -	 */
>>> -
>>> -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>> -		regs = NULL;
>>> -		num = ARRAY_SIZE(xehp_regs);
>>> -	} else if (GRAPHICS_VER(i915) == 12) {
>>> -		regs = gen12_regs;
>>> -		num = ARRAY_SIZE(gen12_regs);
>>> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>> -		regs = gen8_regs;
>>> -		num = ARRAY_SIZE(gen8_regs);
>>> -	} else if (GRAPHICS_VER(i915) < 8) {
>>> -		return;
>>> -	}
>>> -
>>> -	if (drm_WARN_ONCE(&i915->drm, !num,
>>> -			  "Platform does not implement TLB invalidation!"))
>>> +	if (GRAPHICS_VER(i915) < 8)
>>>    		return;
>>>    	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>>> @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>    	awake = 0;
>>>    	for_each_engine(engine, gt, id) {
>>> -		struct reg_and_bit rb;
>>> -
>>>    		if (!intel_engine_pm_is_awake(engine))
>>>    			continue;
>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -			u32 val = BIT(engine->instance);
>>> -
>>> -			if (engine->class == VIDEO_DECODE_CLASS ||
>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -			    engine->class == COMPUTE_CLASS)
>>> -				val = _MASKED_BIT_ENABLE(val);
>>> +		if (engine->tlb_inv.mcr)
>>>    			intel_gt_mcr_multicast_write_fw(gt,
>>> -							xehp_regs[engine->class],
>>> -							val);
>>> -		} else {
>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -			if (!i915_mmio_reg_offset(rb.reg))
>>> -				continue;
>>> -
>>> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -			    engine->class == COMPUTE_CLASS))
>>> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>>> -
>>> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>>> -		}
>>> +							engine->tlb_inv.reg.mcr_reg,
>>> +							engine->tlb_inv.request);
>>> +		else
>>> +			intel_uncore_write_fw(uncore,
>>> +					      engine->tlb_inv.reg.reg,
>>> +					      engine->tlb_inv.request);
>>> +
>>>    		awake |= engine->mask;
>>>    	}
>>> @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>    	intel_gt_mcr_unlock(gt, flags);
>>>    	for_each_engine_masked(engine, gt, awake, tmp) {
>>> -		struct reg_and_bit rb;
>>> -
>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -			rb.mcr_reg = xehp_regs[engine->class];
>>> -			rb.bit = BIT(engine->instance);
>>> -		} else {
>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -		}
>>> -
>>> -		if (wait_for_invalidate(gt, rb))
>>> +		if (wait_for_invalidate(engine))
>>>    			drm_err_ratelimited(&gt->i915->drm,
>>>    					    "%s TLB invalidation did not complete in %ums!\n",
>>>    					    engine->name, TLB_INVAL_TIMEOUT_MS);
>>
> 

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

On 03/01/2023 19:57, Matt Roper wrote:
> On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
>> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
>>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>
>>> As the logic for selecting the register and corresponsing values grew, the
>>
>> corresponding
>>
>>> code become a bit unsightly. Consolidate by storing the required values at
>>> engine init time in the engine itself, and by doing so minimise the amount
>>> of invariant platform and engine checks during each and every TLB
>>> invalidation.
>>>
>>> v2:
>>>    * Fail engine probe if TLB invlidations registers are unknown.
>>>
>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>>> Cc: Matt Roper <matthew.d.roper@intel.com>
>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
>>> ---
>>>    drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
>>>    drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
>>>    drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
>>>    3 files changed, 128 insertions(+), 115 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> index 99c4b866addd..d47dadfc25c8 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
>>>    	return ret;
>>>    }
>>> +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
>>> +{
>>> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>>> +		[RENDER_CLASS].reg		= GEN8_RTCR,
>>> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
>>> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
>>> +	};
>>> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>>> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
>>> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
>>> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
>>> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
>>> +	};
>>> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>>> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
>>> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
>>> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>>> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
>>> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
>>> +	};
>>> +	struct drm_i915_private *i915 = engine->i915;
>>> +	const union intel_engine_tlb_inv_reg *regs;
>>> +	union intel_engine_tlb_inv_reg reg;
>>> +	unsigned int class = engine->class;
>>> +	unsigned int num = 0;
>>> +	u32 val;
>>> +
>>> +	/*
>>> +	 * New platforms should not be added with catch-all-newer (>=)
>>> +	 * condition so that any later platform added triggers the below warning
>>> +	 * and in turn mandates a human cross-check of whether the invalidation
>>> +	 * flows have compatible semantics.
>>> +	 *
>>> +	 * For instance with the 11.00 -> 12.00 transition three out of five
>>> +	 * respective engine registers were moved to masked type. Then after the
>>> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>> +	 */
>>> +
>>> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> 
> This is bad...it only captures XEHPSDV and breaks the handling of DG2
> (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
> hitting the warning as expected since those are all now being captured
> by the next case of the if/else ladder.  With the way GMD_ID works, we
> may also get new version numbers that silently show up in hardware too
> at some point (e.g., 12.73, 12.74, etc.)

Great (on multiple counts) ...

> 
>>> +		regs = xehp_regs;
>>> +		num = ARRAY_SIZE(xehp_regs);
>>> +	} else if (GRAPHICS_VER(i915) == 12) {
> 
> You'd want to change this to
> 
>          GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
> 
> to get the behavior you expected.

Okay, that, and then to be as safe as I intended, ie. warn on every new 
platforms so developers *must* check registers are still compatible 
during platform enablement, we would need a full ver range check 
something like:

	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&
	    GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)) {
		regs = xehp_regs;
		num = ARRAY_SIZE(xehp_regs);
	} else if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)) {
		regs = gen12_regs;
		num = ARRAY_SIZE(gen12_regs);

What do you think about that?

Or you are saying new GMD IDs may appear in the field without first 
having passed the new platform enablemend process? That would be 
horrible so I hope not.

Regards,

Tvrtko

>>> +		regs = gen12_regs;
>>> +		num = ARRAY_SIZE(gen12_regs);
>>> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>> +		regs = gen8_regs;
>>> +		num = ARRAY_SIZE(gen8_regs);
>>> +	} else if (GRAPHICS_VER(i915) < 8) {
>>> +		return 0;
>>> +	} > +
>>> +	if (drm_WARN_ONCE(&i915->drm, !num,
>>> +			  "Platform does not implement TLB invalidation!"))
>>> +		return -ENODEV;
>>> +
>>> +	if (drm_WARN_ON_ONCE(&i915->drm,
>>> +			     class >= num ||
>>> +			     (!regs[class].reg.reg &&
>>> +			      !regs[class].mcr_reg.reg)))
>>> +		return -ERANGE;
>>
>> I hope the propagation of -ERANGE to device probe is OK.
>>
>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
>>
>> Regards
>> Andrzej
>>
>>> +
>>> +	reg = regs[class];
>>> +
>>> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>>> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> +		val = 0;
>>> +	} else {
>>> +		val = engine->instance;
>>> +	}
>>> +
>>> +	val = BIT(val);
>>> +
>>> +	engine->tlb_inv.mcr = regs == xehp_regs;
>>> +	engine->tlb_inv.reg = reg;
>>> +	engine->tlb_inv.done = val;
>>> +
>>> +	if (GRAPHICS_VER(i915) >= 12 &&
>>> +	    (engine->class == VIDEO_DECODE_CLASS ||
>>> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> +	     engine->class == COMPUTE_CLASS))
>>> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>>> +	else
>>> +		engine->tlb_inv.request = val;
>>> +
>>> +	return 0;
>>> +}
>>> +
>>>    static int engine_setup_common(struct intel_engine_cs *engine)
>>>    {
>>>    	int err;
>>>    	init_llist_head(&engine->barrier_tasks);
>>> +	err = intel_engine_init_tlb_invalidation(engine);
>>> +	if (err)
>>> +		return err;
>>> +
>>>    	err = init_status_page(engine);
>>>    	if (err)
>>>    		return err;
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> index 4fd54fb8810f..8c661fe89314 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>>    	u64 start_gt_clk;
>>>    };
>>> +union intel_engine_tlb_inv_reg {
>>> +	i915_reg_t	reg;
>>> +	i915_mcr_reg_t	mcr_reg;
>>> +};
>>> +
>>> +struct intel_engine_tlb_inv
>>> +{
>>> +	bool mcr;
>>> +	union intel_engine_tlb_inv_reg reg;
>>> +	u32 request;
>>> +	u32 done;
>>> +};
>>> +
>>>    struct intel_engine_cs {
>>>    	struct drm_i915_private *i915;
>>>    	struct intel_gt *gt;
>>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>>    	u32 context_size;
>>>    	u32 mmio_base;
>>> +	struct intel_engine_tlb_inv tlb_inv;
>>> +
>>>    	/*
>>>    	 * Some w/a require forcewake to be held (which prevents RC6) while
>>>    	 * a particular engine is active. If so, we set fw_domain to which
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> index 854841a731cb..9fb0ac03f51a 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>>>    	intel_sseu_dump(&info->sseu, p);
>>>    }
>>> -struct reg_and_bit {
>>> -	union {
>>> -		i915_reg_t reg;
>>> -		i915_mcr_reg_t mcr_reg;
>>> -	};
>>> -	u32 bit;
>>> -};
>>> -
>>> -static struct reg_and_bit
>>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>> -		const i915_reg_t *regs, const unsigned int num)
>>> -{
>>> -	const unsigned int class = engine->class;
>>> -	struct reg_and_bit rb = { };
>>> -
>>> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
>>> -			     class >= num || !regs[class].reg))
>>> -		return rb;
>>> -
>>> -	rb.reg = regs[class];
>>> -	if (gen8 && class == VIDEO_DECODE_CLASS)
>>> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> -	else
>>> -		rb.bit = engine->instance;
>>> -
>>> -	rb.bit = BIT(rb.bit);
>>> -
>>> -	return rb;
>>> -}
>>> -
>>>    /*
>>>     * HW architecture suggest typical invalidation time at 40us,
>>>     * with pessimistic cases up to 100us and a recommendation to
>>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>     * but are now considered MCR registers.  Since they exist within a GAM range,
>>>     * the primary instance of the register rolls up the status from each unit.
>>>     */
>>> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>>    {
>>> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>>> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>>> +	if (engine->tlb_inv.mcr)
>>> +		return intel_gt_mcr_wait_for_reg(engine->gt,
>>> +						 engine->tlb_inv.reg.mcr_reg,
>>> +						 engine->tlb_inv.done,
>>> +						 0,
>>>    						 TLB_INVAL_TIMEOUT_US,
>>>    						 TLB_INVAL_TIMEOUT_MS);
>>>    	else
>>> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
>>> +		return __intel_wait_for_register_fw(engine->gt->uncore,
>>> +						    engine->tlb_inv.reg.reg,
>>> +						    engine->tlb_inv.done,
>>> +						    0,
>>>    						    TLB_INVAL_TIMEOUT_US,
>>>    						    TLB_INVAL_TIMEOUT_MS,
>>>    						    NULL);
>>> @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>    static void mmio_invalidate_full(struct intel_gt *gt)
>>>    {
>>> -	static const i915_reg_t gen8_regs[] = {
>>> -		[RENDER_CLASS]			= GEN8_RTCR,
>>> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
>>> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
>>> -	};
>>> -	static const i915_reg_t gen12_regs[] = {
>>> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
>>> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
>>> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
>>> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
>>> -	};
>>> -	static const i915_mcr_reg_t xehp_regs[] = {
>>> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
>>> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
>>> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
>>> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
>>> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
>>> -	};
>>>    	struct drm_i915_private *i915 = gt->i915;
>>>    	struct intel_uncore *uncore = gt->uncore;
>>>    	struct intel_engine_cs *engine;
>>>    	intel_engine_mask_t awake, tmp;
>>>    	enum intel_engine_id id;
>>> -	const i915_reg_t *regs;
>>> -	unsigned int num = 0;
>>>    	unsigned long flags;
>>> -	/*
>>> -	 * New platforms should not be added with catch-all-newer (>=)
>>> -	 * condition so that any later platform added triggers the below warning
>>> -	 * and in turn mandates a human cross-check of whether the invalidation
>>> -	 * flows have compatible semantics.
>>> -	 *
>>> -	 * For instance with the 11.00 -> 12.00 transition three out of five
>>> -	 * respective engine registers were moved to masked type. Then after the
>>> -	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>> -	 */
>>> -
>>> -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>> -		regs = NULL;
>>> -		num = ARRAY_SIZE(xehp_regs);
>>> -	} else if (GRAPHICS_VER(i915) == 12) {
>>> -		regs = gen12_regs;
>>> -		num = ARRAY_SIZE(gen12_regs);
>>> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>> -		regs = gen8_regs;
>>> -		num = ARRAY_SIZE(gen8_regs);
>>> -	} else if (GRAPHICS_VER(i915) < 8) {
>>> -		return;
>>> -	}
>>> -
>>> -	if (drm_WARN_ONCE(&i915->drm, !num,
>>> -			  "Platform does not implement TLB invalidation!"))
>>> +	if (GRAPHICS_VER(i915) < 8)
>>>    		return;
>>>    	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>>> @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>    	awake = 0;
>>>    	for_each_engine(engine, gt, id) {
>>> -		struct reg_and_bit rb;
>>> -
>>>    		if (!intel_engine_pm_is_awake(engine))
>>>    			continue;
>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -			u32 val = BIT(engine->instance);
>>> -
>>> -			if (engine->class == VIDEO_DECODE_CLASS ||
>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -			    engine->class == COMPUTE_CLASS)
>>> -				val = _MASKED_BIT_ENABLE(val);
>>> +		if (engine->tlb_inv.mcr)
>>>    			intel_gt_mcr_multicast_write_fw(gt,
>>> -							xehp_regs[engine->class],
>>> -							val);
>>> -		} else {
>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -			if (!i915_mmio_reg_offset(rb.reg))
>>> -				continue;
>>> -
>>> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -			    engine->class == COMPUTE_CLASS))
>>> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>>> -
>>> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>>> -		}
>>> +							engine->tlb_inv.reg.mcr_reg,
>>> +							engine->tlb_inv.request);
>>> +		else
>>> +			intel_uncore_write_fw(uncore,
>>> +					      engine->tlb_inv.reg.reg,
>>> +					      engine->tlb_inv.request);
>>> +
>>>    		awake |= engine->mask;
>>>    	}
>>> @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>    	intel_gt_mcr_unlock(gt, flags);
>>>    	for_each_engine_masked(engine, gt, awake, tmp) {
>>> -		struct reg_and_bit rb;
>>> -
>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -			rb.mcr_reg = xehp_regs[engine->class];
>>> -			rb.bit = BIT(engine->instance);
>>> -		} else {
>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -		}
>>> -
>>> -		if (wait_for_invalidate(gt, rb))
>>> +		if (wait_for_invalidate(engine))
>>>    			drm_err_ratelimited(&gt->i915->drm,
>>>    					    "%s TLB invalidation did not complete in %ums!\n",
>>>    					    engine->name, TLB_INVAL_TIMEOUT_MS);
>>
> 

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Matt Roper]

On Wed, Jan 04, 2023 at 10:08:29AM +0000, Tvrtko Ursulin wrote:
> 
> On 03/01/2023 19:57, Matt Roper wrote:
> > On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
> > > On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> > > > From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > > > 
> > > > As the logic for selecting the register and corresponsing values grew, the
> > > 
> > > corresponding
> > > 
> > > > code become a bit unsightly. Consolidate by storing the required values at
> > > > engine init time in the engine itself, and by doing so minimise the amount
> > > > of invariant platform and engine checks during each and every TLB
> > > > invalidation.
> > > > 
> > > > v2:
> > > >    * Fail engine probe if TLB invlidations registers are unknown.
> > > > 
> > > > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > > > Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> > > > Cc: Matt Roper <matthew.d.roper@intel.com>
> > > > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
> > > > ---
> > > >    drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
> > > >    drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
> > > >    drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
> > > >    3 files changed, 128 insertions(+), 115 deletions(-)
> > > > 
> > > > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > index 99c4b866addd..d47dadfc25c8 100644
> > > > --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
> > > >    	return ret;
> > > >    }
> > > > +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> > > > +{
> > > > +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> > > > +		[RENDER_CLASS].reg		= GEN8_RTCR,
> > > > +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> > > > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> > > > +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> > > > +	};
> > > > +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> > > > +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> > > > +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> > > > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> > > > +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> > > > +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> > > > +	};
> > > > +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> > > > +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> > > > +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> > > > +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> > > > +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> > > > +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> > > > +	};
> > > > +	struct drm_i915_private *i915 = engine->i915;
> > > > +	const union intel_engine_tlb_inv_reg *regs;
> > > > +	union intel_engine_tlb_inv_reg reg;
> > > > +	unsigned int class = engine->class;
> > > > +	unsigned int num = 0;
> > > > +	u32 val;
> > > > +
> > > > +	/*
> > > > +	 * New platforms should not be added with catch-all-newer (>=)
> > > > +	 * condition so that any later platform added triggers the below warning
> > > > +	 * and in turn mandates a human cross-check of whether the invalidation
> > > > +	 * flows have compatible semantics.
> > > > +	 *
> > > > +	 * For instance with the 11.00 -> 12.00 transition three out of five
> > > > +	 * respective engine registers were moved to masked type. Then after the
> > > > +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > > > +	 */
> > > > +
> > > > +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > 
> > This is bad...it only captures XEHPSDV and breaks the handling of DG2
> > (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
> > hitting the warning as expected since those are all now being captured
> > by the next case of the if/else ladder.  With the way GMD_ID works, we
> > may also get new version numbers that silently show up in hardware too
> > at some point (e.g., 12.73, 12.74, etc.)
> 
> Great (on multiple counts) ...
> 
> > 
> > > > +		regs = xehp_regs;
> > > > +		num = ARRAY_SIZE(xehp_regs);
> > > > +	} else if (GRAPHICS_VER(i915) == 12) {
> > 
> > You'd want to change this to
> > 
> >          GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
> > 
> > to get the behavior you expected.
> 
> Okay, that, and then to be as safe as I intended, ie. warn on every new
> platforms so developers *must* check registers are still compatible during
> platform enablement, we would need a full ver range check something like:
> 
> 	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&
> 	    GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)) {
> 		regs = xehp_regs;
> 		num = ARRAY_SIZE(xehp_regs);
> 	} else if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)) {
> 		regs = gen12_regs;
> 		num = ARRAY_SIZE(gen12_regs);
> 
> What do you think about that?

What about just keeping the code the way it is now, but adding a new
error condition at the *top* of the ladder?

        if (GRAPHICS_VER_FULL(i915) > IP_VER(12, 72)) {
                /* Error, something new we haven't considered */
        } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
                ...

I guess that won't catch cases where they go back and insert a new
version or IP refresh that's older than the latest version number, but
usually refresh platforms aren't going to be adding the kind of
architectural changes that require a new if/else arm anyway.

Also, as Andrzej pointed out on a different reply, we classified DG1 as
12.10 (because of the interrupt handling changes), so I guess we'd need
a further adjustment to the "gen12" branch to cover that one too.

Honestly, I'm not super wild about the general plan of intentionally
breaking inheritance here; that runs counter to how we usually try to do
things in i915.  In most places we always try to assume that future
platforms will behave the same way as the latest one, and then we update
those assumptions as part of the initial bringup when a future platform
comes along that breaks them.  Most of the places where we do specific
checks and a MISSING_CASE warning are where we have a strong reason to
believe that each platform will always need a new implementation (e.g.,
stuff like workaround lists).  I don't see a reason to expect major
changes in MMIO-based invalidation for a platform n+1, n+2, etc.  The
change that I _do_ expect to show up (even for some of these current
platforms) is that a GuC-based TLB invalidation path will be added since
that's mandatory for a lot of modern features, but that will be a
parallel codepath, not part of the if/else ladder we're editing here.

> 
> Or you are saying new GMD IDs may appear in the field without first having
> passed the new platform enablemend process? That would be horrible so I hope
> not.

They won't just show up in the wild, we'll know they're coming.  But
they'll be used for stuff like how DG2-G12 showed up after we'd already
enabled DG2-G10 and DG2-G11.  Or how we picked up a couple more ADL
variants that our driver treats as "ADL-P," even though there are
technically other SKU letters associated with them.  So they're things
that we wouldn't consider to be a whole new platform, just a new
subplatform at best.


Matt

> 
> Regards,
> 
> Tvrtko
> 
> > > > +		regs = gen12_regs;
> > > > +		num = ARRAY_SIZE(gen12_regs);
> > > > +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > > > +		regs = gen8_regs;
> > > > +		num = ARRAY_SIZE(gen8_regs);
> > > > +	} else if (GRAPHICS_VER(i915) < 8) {
> > > > +		return 0;
> > > > +	} > +
> > > > +	if (drm_WARN_ONCE(&i915->drm, !num,
> > > > +			  "Platform does not implement TLB invalidation!"))
> > > > +		return -ENODEV;
> > > > +
> > > > +	if (drm_WARN_ON_ONCE(&i915->drm,
> > > > +			     class >= num ||
> > > > +			     (!regs[class].reg.reg &&
> > > > +			      !regs[class].mcr_reg.reg)))
> > > > +		return -ERANGE;
> > > 
> > > I hope the propagation of -ERANGE to device probe is OK.
> > > 
> > > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
> > > 
> > > Regards
> > > Andrzej
> > > 
> > > > +
> > > > +	reg = regs[class];
> > > > +
> > > > +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> > > > +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > > > +		val = 0;
> > > > +	} else {
> > > > +		val = engine->instance;
> > > > +	}
> > > > +
> > > > +	val = BIT(val);
> > > > +
> > > > +	engine->tlb_inv.mcr = regs == xehp_regs;
> > > > +	engine->tlb_inv.reg = reg;
> > > > +	engine->tlb_inv.done = val;
> > > > +
> > > > +	if (GRAPHICS_VER(i915) >= 12 &&
> > > > +	    (engine->class == VIDEO_DECODE_CLASS ||
> > > > +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > +	     engine->class == COMPUTE_CLASS))
> > > > +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> > > > +	else
> > > > +		engine->tlb_inv.request = val;
> > > > +
> > > > +	return 0;
> > > > +}
> > > > +
> > > >    static int engine_setup_common(struct intel_engine_cs *engine)
> > > >    {
> > > >    	int err;
> > > >    	init_llist_head(&engine->barrier_tasks);
> > > > +	err = intel_engine_init_tlb_invalidation(engine);
> > > > +	if (err)
> > > > +		return err;
> > > > +
> > > >    	err = init_status_page(engine);
> > > >    	if (err)
> > > >    		return err;
> > > > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > index 4fd54fb8810f..8c661fe89314 100644
> > > > --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
> > > >    	u64 start_gt_clk;
> > > >    };
> > > > +union intel_engine_tlb_inv_reg {
> > > > +	i915_reg_t	reg;
> > > > +	i915_mcr_reg_t	mcr_reg;
> > > > +};
> > > > +
> > > > +struct intel_engine_tlb_inv
> > > > +{
> > > > +	bool mcr;
> > > > +	union intel_engine_tlb_inv_reg reg;
> > > > +	u32 request;
> > > > +	u32 done;
> > > > +};
> > > > +
> > > >    struct intel_engine_cs {
> > > >    	struct drm_i915_private *i915;
> > > >    	struct intel_gt *gt;
> > > > @@ -372,6 +385,8 @@ struct intel_engine_cs {
> > > >    	u32 context_size;
> > > >    	u32 mmio_base;
> > > > +	struct intel_engine_tlb_inv tlb_inv;
> > > > +
> > > >    	/*
> > > >    	 * Some w/a require forcewake to be held (which prevents RC6) while
> > > >    	 * a particular engine is active. If so, we set fw_domain to which
> > > > diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > index 854841a731cb..9fb0ac03f51a 100644
> > > > --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
> > > >    	intel_sseu_dump(&info->sseu, p);
> > > >    }
> > > > -struct reg_and_bit {
> > > > -	union {
> > > > -		i915_reg_t reg;
> > > > -		i915_mcr_reg_t mcr_reg;
> > > > -	};
> > > > -	u32 bit;
> > > > -};
> > > > -
> > > > -static struct reg_and_bit
> > > > -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > > > -		const i915_reg_t *regs, const unsigned int num)
> > > > -{
> > > > -	const unsigned int class = engine->class;
> > > > -	struct reg_and_bit rb = { };
> > > > -
> > > > -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> > > > -			     class >= num || !regs[class].reg))
> > > > -		return rb;
> > > > -
> > > > -	rb.reg = regs[class];
> > > > -	if (gen8 && class == VIDEO_DECODE_CLASS)
> > > > -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > > > -	else
> > > > -		rb.bit = engine->instance;
> > > > -
> > > > -	rb.bit = BIT(rb.bit);
> > > > -
> > > > -	return rb;
> > > > -}
> > > > -
> > > >    /*
> > > >     * HW architecture suggest typical invalidation time at 40us,
> > > >     * with pessimistic cases up to 100us and a recommendation to
> > > > @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > > >     * but are now considered MCR registers.  Since they exist within a GAM range,
> > > >     * the primary instance of the register rolls up the status from each unit.
> > > >     */
> > > > -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > > > +static int wait_for_invalidate(struct intel_engine_cs *engine)
> > > >    {
> > > > -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> > > > -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> > > > +	if (engine->tlb_inv.mcr)
> > > > +		return intel_gt_mcr_wait_for_reg(engine->gt,
> > > > +						 engine->tlb_inv.reg.mcr_reg,
> > > > +						 engine->tlb_inv.done,
> > > > +						 0,
> > > >    						 TLB_INVAL_TIMEOUT_US,
> > > >    						 TLB_INVAL_TIMEOUT_MS);
> > > >    	else
> > > > -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> > > > +		return __intel_wait_for_register_fw(engine->gt->uncore,
> > > > +						    engine->tlb_inv.reg.reg,
> > > > +						    engine->tlb_inv.done,
> > > > +						    0,
> > > >    						    TLB_INVAL_TIMEOUT_US,
> > > >    						    TLB_INVAL_TIMEOUT_MS,
> > > >    						    NULL);
> > > > @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > > >    static void mmio_invalidate_full(struct intel_gt *gt)
> > > >    {
> > > > -	static const i915_reg_t gen8_regs[] = {
> > > > -		[RENDER_CLASS]			= GEN8_RTCR,
> > > > -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> > > > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> > > > -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> > > > -	};
> > > > -	static const i915_reg_t gen12_regs[] = {
> > > > -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> > > > -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> > > > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> > > > -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> > > > -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> > > > -	};
> > > > -	static const i915_mcr_reg_t xehp_regs[] = {
> > > > -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> > > > -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> > > > -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> > > > -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> > > > -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> > > > -	};
> > > >    	struct drm_i915_private *i915 = gt->i915;
> > > >    	struct intel_uncore *uncore = gt->uncore;
> > > >    	struct intel_engine_cs *engine;
> > > >    	intel_engine_mask_t awake, tmp;
> > > >    	enum intel_engine_id id;
> > > > -	const i915_reg_t *regs;
> > > > -	unsigned int num = 0;
> > > >    	unsigned long flags;
> > > > -	/*
> > > > -	 * New platforms should not be added with catch-all-newer (>=)
> > > > -	 * condition so that any later platform added triggers the below warning
> > > > -	 * and in turn mandates a human cross-check of whether the invalidation
> > > > -	 * flows have compatible semantics.
> > > > -	 *
> > > > -	 * For instance with the 11.00 -> 12.00 transition three out of five
> > > > -	 * respective engine registers were moved to masked type. Then after the
> > > > -	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > > > -	 */
> > > > -
> > > > -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > > > -		regs = NULL;
> > > > -		num = ARRAY_SIZE(xehp_regs);
> > > > -	} else if (GRAPHICS_VER(i915) == 12) {
> > > > -		regs = gen12_regs;
> > > > -		num = ARRAY_SIZE(gen12_regs);
> > > > -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > > > -		regs = gen8_regs;
> > > > -		num = ARRAY_SIZE(gen8_regs);
> > > > -	} else if (GRAPHICS_VER(i915) < 8) {
> > > > -		return;
> > > > -	}
> > > > -
> > > > -	if (drm_WARN_ONCE(&i915->drm, !num,
> > > > -			  "Platform does not implement TLB invalidation!"))
> > > > +	if (GRAPHICS_VER(i915) < 8)
> > > >    		return;
> > > >    	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> > > > @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> > > >    	awake = 0;
> > > >    	for_each_engine(engine, gt, id) {
> > > > -		struct reg_and_bit rb;
> > > > -
> > > >    		if (!intel_engine_pm_is_awake(engine))
> > > >    			continue;
> > > > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > > > -			u32 val = BIT(engine->instance);
> > > > -
> > > > -			if (engine->class == VIDEO_DECODE_CLASS ||
> > > > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > -			    engine->class == COMPUTE_CLASS)
> > > > -				val = _MASKED_BIT_ENABLE(val);
> > > > +		if (engine->tlb_inv.mcr)
> > > >    			intel_gt_mcr_multicast_write_fw(gt,
> > > > -							xehp_regs[engine->class],
> > > > -							val);
> > > > -		} else {
> > > > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > > > -			if (!i915_mmio_reg_offset(rb.reg))
> > > > -				continue;
> > > > -
> > > > -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> > > > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > -			    engine->class == COMPUTE_CLASS))
> > > > -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> > > > -
> > > > -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> > > > -		}
> > > > +							engine->tlb_inv.reg.mcr_reg,
> > > > +							engine->tlb_inv.request);
> > > > +		else
> > > > +			intel_uncore_write_fw(uncore,
> > > > +					      engine->tlb_inv.reg.reg,
> > > > +					      engine->tlb_inv.request);
> > > > +
> > > >    		awake |= engine->mask;
> > > >    	}
> > > > @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> > > >    	intel_gt_mcr_unlock(gt, flags);
> > > >    	for_each_engine_masked(engine, gt, awake, tmp) {
> > > > -		struct reg_and_bit rb;
> > > > -
> > > > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > > > -			rb.mcr_reg = xehp_regs[engine->class];
> > > > -			rb.bit = BIT(engine->instance);
> > > > -		} else {
> > > > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > > > -		}
> > > > -
> > > > -		if (wait_for_invalidate(gt, rb))
> > > > +		if (wait_for_invalidate(engine))
> > > >    			drm_err_ratelimited(&gt->i915->drm,
> > > >    					    "%s TLB invalidation did not complete in %ums!\n",
> > > >    					    engine->name, TLB_INVAL_TIMEOUT_MS);
> > > 
> > 

-- 
Matt Roper
Graphics Software Engineer
VTT-OSGC Platform Enablement
Intel Corporation

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Andrzej Hajda]

On 04.01.2023 18:41, Matt Roper wrote:
> On Wed, Jan 04, 2023 at 10:08:29AM +0000, Tvrtko Ursulin wrote:
>>
>> On 03/01/2023 19:57, Matt Roper wrote:
>>> On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
>>>> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
>>>>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>>>
>>>>> As the logic for selecting the register and corresponsing values grew, the
>>>>
>>>> corresponding
>>>>
>>>>> code become a bit unsightly. Consolidate by storing the required values at
>>>>> engine init time in the engine itself, and by doing so minimise the amount
>>>>> of invariant platform and engine checks during each and every TLB
>>>>> invalidation.
>>>>>
>>>>> v2:
>>>>>     * Fail engine probe if TLB invlidations registers are unknown.
>>>>>
>>>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>>>>> Cc: Matt Roper <matthew.d.roper@intel.com>
>>>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
>>>>> ---
>>>>>     drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
>>>>>     drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
>>>>>     drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
>>>>>     3 files changed, 128 insertions(+), 115 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> index 99c4b866addd..d47dadfc25c8 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
>>>>>     	return ret;
>>>>>     }
>>>>> +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
>>>>> +{
>>>>> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>>>>> +		[RENDER_CLASS].reg		= GEN8_RTCR,
>>>>> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
>>>>> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
>>>>> +	};
>>>>> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>>>>> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
>>>>> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
>>>>> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
>>>>> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
>>>>> +	};
>>>>> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>>>>> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
>>>>> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>>>>> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
>>>>> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
>>>>> +	};
>>>>> +	struct drm_i915_private *i915 = engine->i915;
>>>>> +	const union intel_engine_tlb_inv_reg *regs;
>>>>> +	union intel_engine_tlb_inv_reg reg;
>>>>> +	unsigned int class = engine->class;
>>>>> +	unsigned int num = 0;
>>>>> +	u32 val;
>>>>> +
>>>>> +	/*
>>>>> +	 * New platforms should not be added with catch-all-newer (>=)
>>>>> +	 * condition so that any later platform added triggers the below warning
>>>>> +	 * and in turn mandates a human cross-check of whether the invalidation
>>>>> +	 * flows have compatible semantics.
>>>>> +	 *
>>>>> +	 * For instance with the 11.00 -> 12.00 transition three out of five
>>>>> +	 * respective engine registers were moved to masked type. Then after the
>>>>> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>>>> +	 */
>>>>> +
>>>>> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>>
>>> This is bad...it only captures XEHPSDV and breaks the handling of DG2
>>> (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
>>> hitting the warning as expected since those are all now being captured
>>> by the next case of the if/else ladder.  With the way GMD_ID works, we
>>> may also get new version numbers that silently show up in hardware too
>>> at some point (e.g., 12.73, 12.74, etc.)
>>
>> Great (on multiple counts) ...
>>
>>>
>>>>> +		regs = xehp_regs;
>>>>> +		num = ARRAY_SIZE(xehp_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) == 12) {
>>>
>>> You'd want to change this to
>>>
>>>           GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
>>>
>>> to get the behavior you expected.
>>
>> Okay, that, and then to be as safe as I intended, ie. warn on every new
>> platforms so developers *must* check registers are still compatible during
>> platform enablement, we would need a full ver range check something like:
>>
>> 	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&
>> 	    GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)) {
>> 		regs = xehp_regs;
>> 		num = ARRAY_SIZE(xehp_regs);
>> 	} else if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)) {
>> 		regs = gen12_regs;
>> 		num = ARRAY_SIZE(gen12_regs);
>>
>> What do you think about that?
> 
> What about just keeping the code the way it is now, but adding a new
> error condition at the *top* of the ladder?
> 
>          if (GRAPHICS_VER_FULL(i915) > IP_VER(12, 72)) {
>                  /* Error, something new we haven't considered */
>          } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>                  ...
> 
> I guess that won't catch cases where they go back and insert a new
> version or IP refresh that's older than the latest version number, but
> usually refresh platforms aren't going to be adding the kind of
> architectural changes that require a new if/else arm anyway.
> 
> Also, as Andrzej pointed out on a different reply, we classified DG1 as
> 12.10 (because of the interrupt handling changes), so I guess we'd need
> a further adjustment to the "gen12" branch to cover that one too.


My 2 cents.
There seems to be two more or less elegant ways IMO:
1. The ladder for choosing range, covering all values in top-down order, 
in such case it could be simplified (only one check per if):
	if (ver > IP_VER(12, 72)) {
		error
	} else if (ver >= IP_VER(12, 50)) {
	} else if (ver >= ...) {
	} else {
		return;
	}
This clearly covers whole range, no possible holes/overlaps, but can be 
little bit problematic if there will be 'exceptional' platforms, but 
this is not the case for now.

2. Explicit checking for supported versions:
- either switch, either ifs specifiying all supported versions (no 
ranges) - ok this is not elegant but at least very strict.
- adding tlb_inv_type field to struct intel_device_info, this in theory 
will be quite restrictive, but copy/paste technique can ruin it :).

Current approach is also OK (after fixing bugs) but for me seems not 
consistent - dubious double checks, upper not-supported range tested at 
the end differently.

Regards
Andrzej



> 
> Honestly, I'm not super wild about the general plan of intentionally
> breaking inheritance here; that runs counter to how we usually try to do
> things in i915.  In most places we always try to assume that future
> platforms will behave the same way as the latest one, and then we update
> those assumptions as part of the initial bringup when a future platform
> comes along that breaks them.  Most of the places where we do specific
> checks and a MISSING_CASE warning are where we have a strong reason to
> believe that each platform will always need a new implementation (e.g.,
> stuff like workaround lists).  I don't see a reason to expect major
> changes in MMIO-based invalidation for a platform n+1, n+2, etc.  The
> change that I _do_ expect to show up (even for some of these current
> platforms) is that a GuC-based TLB invalidation path will be added since
> that's mandatory for a lot of modern features, but that will be a
> parallel codepath, not part of the if/else ladder we're editing here.
> 
>>
>> Or you are saying new GMD IDs may appear in the field without first having
>> passed the new platform enablemend process? That would be horrible so I hope
>> not.
> 
> They won't just show up in the wild, we'll know they're coming.  But
> they'll be used for stuff like how DG2-G12 showed up after we'd already
> enabled DG2-G10 and DG2-G11.  Or how we picked up a couple more ADL
> variants that our driver treats as "ADL-P," even though there are
> technically other SKU letters associated with them.  So they're things
> that we wouldn't consider to be a whole new platform, just a new
> subplatform at best.
> 
> 
> Matt
> 
>>
>> Regards,
>>
>> Tvrtko
>>
>>>>> +		regs = gen12_regs;
>>>>> +		num = ARRAY_SIZE(gen12_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>>>> +		regs = gen8_regs;
>>>>> +		num = ARRAY_SIZE(gen8_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) < 8) {
>>>>> +		return 0;
>>>>> +	} > +
>>>>> +	if (drm_WARN_ONCE(&i915->drm, !num,
>>>>> +			  "Platform does not implement TLB invalidation!"))
>>>>> +		return -ENODEV;
>>>>> +
>>>>> +	if (drm_WARN_ON_ONCE(&i915->drm,
>>>>> +			     class >= num ||
>>>>> +			     (!regs[class].reg.reg &&
>>>>> +			      !regs[class].mcr_reg.reg)))
>>>>> +		return -ERANGE;
>>>>
>>>> I hope the propagation of -ERANGE to device probe is OK.
>>>>
>>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
>>>>
>>>> Regards
>>>> Andrzej
>>>>
>>>>> +
>>>>> +	reg = regs[class];
>>>>> +
>>>>> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>>>>> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>>>> +		val = 0;
>>>>> +	} else {
>>>>> +		val = engine->instance;
>>>>> +	}
>>>>> +
>>>>> +	val = BIT(val);
>>>>> +
>>>>> +	engine->tlb_inv.mcr = regs == xehp_regs;
>>>>> +	engine->tlb_inv.reg = reg;
>>>>> +	engine->tlb_inv.done = val;
>>>>> +
>>>>> +	if (GRAPHICS_VER(i915) >= 12 &&
>>>>> +	    (engine->class == VIDEO_DECODE_CLASS ||
>>>>> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> +	     engine->class == COMPUTE_CLASS))
>>>>> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>>>>> +	else
>>>>> +		engine->tlb_inv.request = val;
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>>     static int engine_setup_common(struct intel_engine_cs *engine)
>>>>>     {
>>>>>     	int err;
>>>>>     	init_llist_head(&engine->barrier_tasks);
>>>>> +	err = intel_engine_init_tlb_invalidation(engine);
>>>>> +	if (err)
>>>>> +		return err;
>>>>> +
>>>>>     	err = init_status_page(engine);
>>>>>     	if (err)
>>>>>     		return err;
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> index 4fd54fb8810f..8c661fe89314 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>>>>     	u64 start_gt_clk;
>>>>>     };
>>>>> +union intel_engine_tlb_inv_reg {
>>>>> +	i915_reg_t	reg;
>>>>> +	i915_mcr_reg_t	mcr_reg;
>>>>> +};
>>>>> +
>>>>> +struct intel_engine_tlb_inv
>>>>> +{
>>>>> +	bool mcr;
>>>>> +	union intel_engine_tlb_inv_reg reg;
>>>>> +	u32 request;
>>>>> +	u32 done;
>>>>> +};
>>>>> +
>>>>>     struct intel_engine_cs {
>>>>>     	struct drm_i915_private *i915;
>>>>>     	struct intel_gt *gt;
>>>>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>>>>     	u32 context_size;
>>>>>     	u32 mmio_base;
>>>>> +	struct intel_engine_tlb_inv tlb_inv;
>>>>> +
>>>>>     	/*
>>>>>     	 * Some w/a require forcewake to be held (which prevents RC6) while
>>>>>     	 * a particular engine is active. If so, we set fw_domain to which
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> index 854841a731cb..9fb0ac03f51a 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>>>>>     	intel_sseu_dump(&info->sseu, p);
>>>>>     }
>>>>> -struct reg_and_bit {
>>>>> -	union {
>>>>> -		i915_reg_t reg;
>>>>> -		i915_mcr_reg_t mcr_reg;
>>>>> -	};
>>>>> -	u32 bit;
>>>>> -};
>>>>> -
>>>>> -static struct reg_and_bit
>>>>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>>> -		const i915_reg_t *regs, const unsigned int num)
>>>>> -{
>>>>> -	const unsigned int class = engine->class;
>>>>> -	struct reg_and_bit rb = { };
>>>>> -
>>>>> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
>>>>> -			     class >= num || !regs[class].reg))
>>>>> -		return rb;
>>>>> -
>>>>> -	rb.reg = regs[class];
>>>>> -	if (gen8 && class == VIDEO_DECODE_CLASS)
>>>>> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>>>> -	else
>>>>> -		rb.bit = engine->instance;
>>>>> -
>>>>> -	rb.bit = BIT(rb.bit);
>>>>> -
>>>>> -	return rb;
>>>>> -}
>>>>> -
>>>>>     /*
>>>>>      * HW architecture suggest typical invalidation time at 40us,
>>>>>      * with pessimistic cases up to 100us and a recommendation to
>>>>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>>>      * but are now considered MCR registers.  Since they exist within a GAM range,
>>>>>      * the primary instance of the register rolls up the status from each unit.
>>>>>      */
>>>>> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>>>>     {
>>>>> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>>>>> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>>>>> +	if (engine->tlb_inv.mcr)
>>>>> +		return intel_gt_mcr_wait_for_reg(engine->gt,
>>>>> +						 engine->tlb_inv.reg.mcr_reg,
>>>>> +						 engine->tlb_inv.done,
>>>>> +						 0,
>>>>>     						 TLB_INVAL_TIMEOUT_US,
>>>>>     						 TLB_INVAL_TIMEOUT_MS);
>>>>>     	else
>>>>> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
>>>>> +		return __intel_wait_for_register_fw(engine->gt->uncore,
>>>>> +						    engine->tlb_inv.reg.reg,
>>>>> +						    engine->tlb_inv.done,
>>>>> +						    0,
>>>>>     						    TLB_INVAL_TIMEOUT_US,
>>>>>     						    TLB_INVAL_TIMEOUT_MS,
>>>>>     						    NULL);
>>>>> @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>>>     static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     {
>>>>> -	static const i915_reg_t gen8_regs[] = {
>>>>> -		[RENDER_CLASS]			= GEN8_RTCR,
>>>>> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
>>>>> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
>>>>> -	};
>>>>> -	static const i915_reg_t gen12_regs[] = {
>>>>> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
>>>>> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
>>>>> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
>>>>> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
>>>>> -	};
>>>>> -	static const i915_mcr_reg_t xehp_regs[] = {
>>>>> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
>>>>> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
>>>>> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
>>>>> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
>>>>> -	};
>>>>>     	struct drm_i915_private *i915 = gt->i915;
>>>>>     	struct intel_uncore *uncore = gt->uncore;
>>>>>     	struct intel_engine_cs *engine;
>>>>>     	intel_engine_mask_t awake, tmp;
>>>>>     	enum intel_engine_id id;
>>>>> -	const i915_reg_t *regs;
>>>>> -	unsigned int num = 0;
>>>>>     	unsigned long flags;
>>>>> -	/*
>>>>> -	 * New platforms should not be added with catch-all-newer (>=)
>>>>> -	 * condition so that any later platform added triggers the below warning
>>>>> -	 * and in turn mandates a human cross-check of whether the invalidation
>>>>> -	 * flows have compatible semantics.
>>>>> -	 *
>>>>> -	 * For instance with the 11.00 -> 12.00 transition three out of five
>>>>> -	 * respective engine registers were moved to masked type. Then after the
>>>>> -	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>>>> -	 */
>>>>> -
>>>>> -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>>>> -		regs = NULL;
>>>>> -		num = ARRAY_SIZE(xehp_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) == 12) {
>>>>> -		regs = gen12_regs;
>>>>> -		num = ARRAY_SIZE(gen12_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>>>> -		regs = gen8_regs;
>>>>> -		num = ARRAY_SIZE(gen8_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) < 8) {
>>>>> -		return;
>>>>> -	}
>>>>> -
>>>>> -	if (drm_WARN_ONCE(&i915->drm, !num,
>>>>> -			  "Platform does not implement TLB invalidation!"))
>>>>> +	if (GRAPHICS_VER(i915) < 8)
>>>>>     		return;
>>>>>     	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>>>>> @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     	awake = 0;
>>>>>     	for_each_engine(engine, gt, id) {
>>>>> -		struct reg_and_bit rb;
>>>>> -
>>>>>     		if (!intel_engine_pm_is_awake(engine))
>>>>>     			continue;
>>>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>>>> -			u32 val = BIT(engine->instance);
>>>>> -
>>>>> -			if (engine->class == VIDEO_DECODE_CLASS ||
>>>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> -			    engine->class == COMPUTE_CLASS)
>>>>> -				val = _MASKED_BIT_ENABLE(val);
>>>>> +		if (engine->tlb_inv.mcr)
>>>>>     			intel_gt_mcr_multicast_write_fw(gt,
>>>>> -							xehp_regs[engine->class],
>>>>> -							val);
>>>>> -		} else {
>>>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>>>> -			if (!i915_mmio_reg_offset(rb.reg))
>>>>> -				continue;
>>>>> -
>>>>> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
>>>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> -			    engine->class == COMPUTE_CLASS))
>>>>> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>>>>> -
>>>>> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>>>>> -		}
>>>>> +							engine->tlb_inv.reg.mcr_reg,
>>>>> +							engine->tlb_inv.request);
>>>>> +		else
>>>>> +			intel_uncore_write_fw(uncore,
>>>>> +					      engine->tlb_inv.reg.reg,
>>>>> +					      engine->tlb_inv.request);
>>>>> +
>>>>>     		awake |= engine->mask;
>>>>>     	}
>>>>> @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     	intel_gt_mcr_unlock(gt, flags);
>>>>>     	for_each_engine_masked(engine, gt, awake, tmp) {
>>>>> -		struct reg_and_bit rb;
>>>>> -
>>>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>>>> -			rb.mcr_reg = xehp_regs[engine->class];
>>>>> -			rb.bit = BIT(engine->instance);
>>>>> -		} else {
>>>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>>>> -		}
>>>>> -
>>>>> -		if (wait_for_invalidate(gt, rb))
>>>>> +		if (wait_for_invalidate(engine))
>>>>>     			drm_err_ratelimited(&gt->i915->drm,
>>>>>     					    "%s TLB invalidation did not complete in %ums!\n",
>>>>>     					    engine->name, TLB_INVAL_TIMEOUT_MS);
>>>>
>>>
> 

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

On 04/01/2023 17:41, Matt Roper wrote:
> On Wed, Jan 04, 2023 at 10:08:29AM +0000, Tvrtko Ursulin wrote:
>>
>> On 03/01/2023 19:57, Matt Roper wrote:
>>> On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
>>>> On 19.12.2022 11:13, Tvrtko Ursulin wrote:
>>>>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>>>
>>>>> As the logic for selecting the register and corresponsing values grew, the
>>>>
>>>> corresponding
>>>>
>>>>> code become a bit unsightly. Consolidate by storing the required values at
>>>>> engine init time in the engine itself, and by doing so minimise the amount
>>>>> of invariant platform and engine checks during each and every TLB
>>>>> invalidation.
>>>>>
>>>>> v2:
>>>>>     * Fail engine probe if TLB invlidations registers are unknown.
>>>>>
>>>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>>>>> Cc: Matt Roper <matthew.d.roper@intel.com>
>>>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
>>>>> ---
>>>>>     drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
>>>>>     drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
>>>>>     drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
>>>>>     3 files changed, 128 insertions(+), 115 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> index 99c4b866addd..d47dadfc25c8 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>>>> @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
>>>>>     	return ret;
>>>>>     }
>>>>> +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
>>>>> +{
>>>>> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>>>>> +		[RENDER_CLASS].reg		= GEN8_RTCR,
>>>>> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
>>>>> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
>>>>> +	};
>>>>> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>>>>> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
>>>>> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
>>>>> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
>>>>> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
>>>>> +	};
>>>>> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>>>>> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
>>>>> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
>>>>> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>>>>> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
>>>>> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
>>>>> +	};
>>>>> +	struct drm_i915_private *i915 = engine->i915;
>>>>> +	const union intel_engine_tlb_inv_reg *regs;
>>>>> +	union intel_engine_tlb_inv_reg reg;
>>>>> +	unsigned int class = engine->class;
>>>>> +	unsigned int num = 0;
>>>>> +	u32 val;
>>>>> +
>>>>> +	/*
>>>>> +	 * New platforms should not be added with catch-all-newer (>=)
>>>>> +	 * condition so that any later platform added triggers the below warning
>>>>> +	 * and in turn mandates a human cross-check of whether the invalidation
>>>>> +	 * flows have compatible semantics.
>>>>> +	 *
>>>>> +	 * For instance with the 11.00 -> 12.00 transition three out of five
>>>>> +	 * respective engine registers were moved to masked type. Then after the
>>>>> +	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>>>> +	 */
>>>>> +
>>>>> +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>>
>>> This is bad...it only captures XEHPSDV and breaks the handling of DG2
>>> (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
>>> hitting the warning as expected since those are all now being captured
>>> by the next case of the if/else ladder.  With the way GMD_ID works, we
>>> may also get new version numbers that silently show up in hardware too
>>> at some point (e.g., 12.73, 12.74, etc.)
>>
>> Great (on multiple counts) ...
>>
>>>
>>>>> +		regs = xehp_regs;
>>>>> +		num = ARRAY_SIZE(xehp_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) == 12) {
>>>
>>> You'd want to change this to
>>>
>>>           GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
>>>
>>> to get the behavior you expected.
>>
>> Okay, that, and then to be as safe as I intended, ie. warn on every new
>> platforms so developers *must* check registers are still compatible during
>> platform enablement, we would need a full ver range check something like:
>>
>> 	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&
>> 	    GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)) {
>> 		regs = xehp_regs;
>> 		num = ARRAY_SIZE(xehp_regs);
>> 	} else if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)) {
>> 		regs = gen12_regs;
>> 		num = ARRAY_SIZE(gen12_regs);
>>
>> What do you think about that?
> 
> What about just keeping the code the way it is now, but adding a new
> error condition at the *top* of the ladder?
> 
>          if (GRAPHICS_VER_FULL(i915) > IP_VER(12, 72)) {
>                  /* Error, something new we haven't considered */
>          } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>                  ...
> 
> I guess that won't catch cases where they go back and insert a new
> version or IP refresh that's older than the latest version number, but
> usually refresh platforms aren't going to be adding the kind of
> architectural changes that require a new if/else arm anyway.
> 
> Also, as Andrzej pointed out on a different reply, we classified DG1 as
> 12.10 (because of the interrupt handling changes), so I guess we'd need
> a further adjustment to the "gen12" branch to cover that one too.
> 
> Honestly, I'm not super wild about the general plan of intentionally
> breaking inheritance here; that runs counter to how we usually try to do
> things in i915.  In most places we always try to assume that future
> platforms will behave the same way as the latest one, and then we update
> those assumptions as part of the initial bringup when a future platform
> comes along that breaks them.  Most of the places where we do specific
> checks and a MISSING_CASE warning are where we have a strong reason to
> believe that each platform will always need a new implementation (e.g.,
> stuff like workaround lists).  I don't see a reason to expect major
> changes in MMIO-based invalidation for a platform n+1, n+2, etc.  The
> change that I _do_ expect to show up (even for some of these current
> platforms) is that a GuC-based TLB invalidation path will be added since
> that's mandatory for a lot of modern features, but that will be a
> parallel codepath, not part of the if/else ladder we're editing here.

The strong reason you mention is currently the lack of _any_ test 
coverage for this. In other words you'd never know a new platform does 
not work properly (imaging one more register becomes masked, like three 
have become from Gen11 to Gen12). Or if refactoring this area of the 
code broke an existing platform. And both cases lead to CVEs so IMO 
safety trumps our usual patterns.

As I said earlier, I am not going to merge anything in this area, apart 
that I want "drm/i915: Do not cover all future platforms in TLB 
invalidation" in, until test coverage materialises.

Once we have test coverage I am okay with catch all future platforms.

So regarding "drm/i915: Do not cover all future platforms in TLB 
invalidation".. minimal change is then to have it like:

   if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&) {
       GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)
	...
   } else if (GRAPHICS_VER(i915) == 12) {
	...
   ...

Is this agreeable? That covers XEHPSDV and DG2. PVC and MTL, if they are 
intended to be supported, someone (not exclusing myself here) needs to 
check bspec before extending the condition with an explicit patch.

Otherwise 77fa9efc16a9 ("drm/i915/xehp: Create separate reg definitions 
for new MCR registers") was about Xe_HP, already wrong and Andrzej fixed 
it in 4d5cf7b1680a ("drm/i915: fix TLB invalidation for Gen12.50 video 
and compute engines").

Further work can happen once test coverage appears.

Regards,

Tvrtko

>>
>> Or you are saying new GMD IDs may appear in the field without first having
>> passed the new platform enablemend process? That would be horrible so I hope
>> not.
> 
> They won't just show up in the wild, we'll know they're coming.  But
> they'll be used for stuff like how DG2-G12 showed up after we'd already
> enabled DG2-G10 and DG2-G11.  Or how we picked up a couple more ADL
> variants that our driver treats as "ADL-P," even though there are
> technically other SKU letters associated with them.  So they're things
> that we wouldn't consider to be a whole new platform, just a new
> subplatform at best.
> 
> 
> Matt
> 
>>
>> Regards,
>>
>> Tvrtko
>>
>>>>> +		regs = gen12_regs;
>>>>> +		num = ARRAY_SIZE(gen12_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>>>> +		regs = gen8_regs;
>>>>> +		num = ARRAY_SIZE(gen8_regs);
>>>>> +	} else if (GRAPHICS_VER(i915) < 8) {
>>>>> +		return 0;
>>>>> +	} > +
>>>>> +	if (drm_WARN_ONCE(&i915->drm, !num,
>>>>> +			  "Platform does not implement TLB invalidation!"))
>>>>> +		return -ENODEV;
>>>>> +
>>>>> +	if (drm_WARN_ON_ONCE(&i915->drm,
>>>>> +			     class >= num ||
>>>>> +			     (!regs[class].reg.reg &&
>>>>> +			      !regs[class].mcr_reg.reg)))
>>>>> +		return -ERANGE;
>>>>
>>>> I hope the propagation of -ERANGE to device probe is OK.
>>>>
>>>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
>>>>
>>>> Regards
>>>> Andrzej
>>>>
>>>>> +
>>>>> +	reg = regs[class];
>>>>> +
>>>>> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>>>>> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>>>> +		val = 0;
>>>>> +	} else {
>>>>> +		val = engine->instance;
>>>>> +	}
>>>>> +
>>>>> +	val = BIT(val);
>>>>> +
>>>>> +	engine->tlb_inv.mcr = regs == xehp_regs;
>>>>> +	engine->tlb_inv.reg = reg;
>>>>> +	engine->tlb_inv.done = val;
>>>>> +
>>>>> +	if (GRAPHICS_VER(i915) >= 12 &&
>>>>> +	    (engine->class == VIDEO_DECODE_CLASS ||
>>>>> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> +	     engine->class == COMPUTE_CLASS))
>>>>> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>>>>> +	else
>>>>> +		engine->tlb_inv.request = val;
>>>>> +
>>>>> +	return 0;
>>>>> +}
>>>>> +
>>>>>     static int engine_setup_common(struct intel_engine_cs *engine)
>>>>>     {
>>>>>     	int err;
>>>>>     	init_llist_head(&engine->barrier_tasks);
>>>>> +	err = intel_engine_init_tlb_invalidation(engine);
>>>>> +	if (err)
>>>>> +		return err;
>>>>> +
>>>>>     	err = init_status_page(engine);
>>>>>     	if (err)
>>>>>     		return err;
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> index 4fd54fb8810f..8c661fe89314 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>>>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>>>>     	u64 start_gt_clk;
>>>>>     };
>>>>> +union intel_engine_tlb_inv_reg {
>>>>> +	i915_reg_t	reg;
>>>>> +	i915_mcr_reg_t	mcr_reg;
>>>>> +};
>>>>> +
>>>>> +struct intel_engine_tlb_inv
>>>>> +{
>>>>> +	bool mcr;
>>>>> +	union intel_engine_tlb_inv_reg reg;
>>>>> +	u32 request;
>>>>> +	u32 done;
>>>>> +};
>>>>> +
>>>>>     struct intel_engine_cs {
>>>>>     	struct drm_i915_private *i915;
>>>>>     	struct intel_gt *gt;
>>>>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>>>>     	u32 context_size;
>>>>>     	u32 mmio_base;
>>>>> +	struct intel_engine_tlb_inv tlb_inv;
>>>>> +
>>>>>     	/*
>>>>>     	 * Some w/a require forcewake to be held (which prevents RC6) while
>>>>>     	 * a particular engine is active. If so, we set fw_domain to which
>>>>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> index 854841a731cb..9fb0ac03f51a 100644
>>>>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>>>>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>>>>>     	intel_sseu_dump(&info->sseu, p);
>>>>>     }
>>>>> -struct reg_and_bit {
>>>>> -	union {
>>>>> -		i915_reg_t reg;
>>>>> -		i915_mcr_reg_t mcr_reg;
>>>>> -	};
>>>>> -	u32 bit;
>>>>> -};
>>>>> -
>>>>> -static struct reg_and_bit
>>>>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>>> -		const i915_reg_t *regs, const unsigned int num)
>>>>> -{
>>>>> -	const unsigned int class = engine->class;
>>>>> -	struct reg_and_bit rb = { };
>>>>> -
>>>>> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
>>>>> -			     class >= num || !regs[class].reg))
>>>>> -		return rb;
>>>>> -
>>>>> -	rb.reg = regs[class];
>>>>> -	if (gen8 && class == VIDEO_DECODE_CLASS)
>>>>> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>>>> -	else
>>>>> -		rb.bit = engine->instance;
>>>>> -
>>>>> -	rb.bit = BIT(rb.bit);
>>>>> -
>>>>> -	return rb;
>>>>> -}
>>>>> -
>>>>>     /*
>>>>>      * HW architecture suggest typical invalidation time at 40us,
>>>>>      * with pessimistic cases up to 100us and a recommendation to
>>>>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>>>>      * but are now considered MCR registers.  Since they exist within a GAM range,
>>>>>      * the primary instance of the register rolls up the status from each unit.
>>>>>      */
>>>>> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>>>>     {
>>>>> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>>>>> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>>>>> +	if (engine->tlb_inv.mcr)
>>>>> +		return intel_gt_mcr_wait_for_reg(engine->gt,
>>>>> +						 engine->tlb_inv.reg.mcr_reg,
>>>>> +						 engine->tlb_inv.done,
>>>>> +						 0,
>>>>>     						 TLB_INVAL_TIMEOUT_US,
>>>>>     						 TLB_INVAL_TIMEOUT_MS);
>>>>>     	else
>>>>> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
>>>>> +		return __intel_wait_for_register_fw(engine->gt->uncore,
>>>>> +						    engine->tlb_inv.reg.reg,
>>>>> +						    engine->tlb_inv.done,
>>>>> +						    0,
>>>>>     						    TLB_INVAL_TIMEOUT_US,
>>>>>     						    TLB_INVAL_TIMEOUT_MS,
>>>>>     						    NULL);
>>>>> @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>>>>>     static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     {
>>>>> -	static const i915_reg_t gen8_regs[] = {
>>>>> -		[RENDER_CLASS]			= GEN8_RTCR,
>>>>> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
>>>>> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
>>>>> -	};
>>>>> -	static const i915_reg_t gen12_regs[] = {
>>>>> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
>>>>> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
>>>>> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
>>>>> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
>>>>> -	};
>>>>> -	static const i915_mcr_reg_t xehp_regs[] = {
>>>>> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
>>>>> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
>>>>> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
>>>>> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
>>>>> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
>>>>> -	};
>>>>>     	struct drm_i915_private *i915 = gt->i915;
>>>>>     	struct intel_uncore *uncore = gt->uncore;
>>>>>     	struct intel_engine_cs *engine;
>>>>>     	intel_engine_mask_t awake, tmp;
>>>>>     	enum intel_engine_id id;
>>>>> -	const i915_reg_t *regs;
>>>>> -	unsigned int num = 0;
>>>>>     	unsigned long flags;
>>>>> -	/*
>>>>> -	 * New platforms should not be added with catch-all-newer (>=)
>>>>> -	 * condition so that any later platform added triggers the below warning
>>>>> -	 * and in turn mandates a human cross-check of whether the invalidation
>>>>> -	 * flows have compatible semantics.
>>>>> -	 *
>>>>> -	 * For instance with the 11.00 -> 12.00 transition three out of five
>>>>> -	 * respective engine registers were moved to masked type. Then after the
>>>>> -	 * 12.00 -> 12.50 transition multi cast handling is required too.
>>>>> -	 */
>>>>> -
>>>>> -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
>>>>> -		regs = NULL;
>>>>> -		num = ARRAY_SIZE(xehp_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) == 12) {
>>>>> -		regs = gen12_regs;
>>>>> -		num = ARRAY_SIZE(gen12_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>>>> -		regs = gen8_regs;
>>>>> -		num = ARRAY_SIZE(gen8_regs);
>>>>> -	} else if (GRAPHICS_VER(i915) < 8) {
>>>>> -		return;
>>>>> -	}
>>>>> -
>>>>> -	if (drm_WARN_ONCE(&i915->drm, !num,
>>>>> -			  "Platform does not implement TLB invalidation!"))
>>>>> +	if (GRAPHICS_VER(i915) < 8)
>>>>>     		return;
>>>>>     	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>>>>> @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     	awake = 0;
>>>>>     	for_each_engine(engine, gt, id) {
>>>>> -		struct reg_and_bit rb;
>>>>> -
>>>>>     		if (!intel_engine_pm_is_awake(engine))
>>>>>     			continue;
>>>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>>>> -			u32 val = BIT(engine->instance);
>>>>> -
>>>>> -			if (engine->class == VIDEO_DECODE_CLASS ||
>>>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> -			    engine->class == COMPUTE_CLASS)
>>>>> -				val = _MASKED_BIT_ENABLE(val);
>>>>> +		if (engine->tlb_inv.mcr)
>>>>>     			intel_gt_mcr_multicast_write_fw(gt,
>>>>> -							xehp_regs[engine->class],
>>>>> -							val);
>>>>> -		} else {
>>>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>>>> -			if (!i915_mmio_reg_offset(rb.reg))
>>>>> -				continue;
>>>>> -
>>>>> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
>>>>> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>>>> -			    engine->class == COMPUTE_CLASS))
>>>>> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>>>>> -
>>>>> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>>>>> -		}
>>>>> +							engine->tlb_inv.reg.mcr_reg,
>>>>> +							engine->tlb_inv.request);
>>>>> +		else
>>>>> +			intel_uncore_write_fw(uncore,
>>>>> +					      engine->tlb_inv.reg.reg,
>>>>> +					      engine->tlb_inv.request);
>>>>> +
>>>>>     		awake |= engine->mask;
>>>>>     	}
>>>>> @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>>>>>     	intel_gt_mcr_unlock(gt, flags);
>>>>>     	for_each_engine_masked(engine, gt, awake, tmp) {
>>>>> -		struct reg_and_bit rb;
>>>>> -
>>>>> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>>>> -			rb.mcr_reg = xehp_regs[engine->class];
>>>>> -			rb.bit = BIT(engine->instance);
>>>>> -		} else {
>>>>> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>>>> -		}
>>>>> -
>>>>> -		if (wait_for_invalidate(gt, rb))
>>>>> +		if (wait_for_invalidate(engine))
>>>>>     			drm_err_ratelimited(&gt->i915->drm,
>>>>>     					    "%s TLB invalidation did not complete in %ums!\n",
>>>>>     					    engine->name, TLB_INVAL_TIMEOUT_MS);
>>>>
>>>
> 

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Matt Roper]

On Thu, Jan 05, 2023 at 01:06:37PM +0000, Tvrtko Ursulin wrote:
> 
> On 04/01/2023 17:41, Matt Roper wrote:
> > On Wed, Jan 04, 2023 at 10:08:29AM +0000, Tvrtko Ursulin wrote:
> > > 
> > > On 03/01/2023 19:57, Matt Roper wrote:
> > > > On Mon, Dec 19, 2022 at 05:10:02PM +0100, Andrzej Hajda wrote:
> > > > > On 19.12.2022 11:13, Tvrtko Ursulin wrote:
> > > > > > From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > > > > > 
> > > > > > As the logic for selecting the register and corresponsing values grew, the
> > > > > 
> > > > > corresponding
> > > > > 
> > > > > > code become a bit unsightly. Consolidate by storing the required values at
> > > > > > engine init time in the engine itself, and by doing so minimise the amount
> > > > > > of invariant platform and engine checks during each and every TLB
> > > > > > invalidation.
> > > > > > 
> > > > > > v2:
> > > > > >     * Fail engine probe if TLB invlidations registers are unknown.
> > > > > > 
> > > > > > Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> > > > > > Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> > > > > > Cc: Matt Roper <matthew.d.roper@intel.com>
> > > > > > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com> # v1
> > > > > > ---
> > > > > >     drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  93 +++++++++++++
> > > > > >     drivers/gpu/drm/i915/gt/intel_engine_types.h |  15 +++
> > > > > >     drivers/gpu/drm/i915/gt/intel_gt.c           | 135 +++----------------
> > > > > >     3 files changed, 128 insertions(+), 115 deletions(-)
> > > > > > 
> > > > > > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > > > index 99c4b866addd..d47dadfc25c8 100644
> > > > > > --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > > > +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> > > > > > @@ -1143,12 +1143,105 @@ static int init_status_page(struct intel_engine_cs *engine)
> > > > > >     	return ret;
> > > > > >     }
> > > > > > +static int intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> > > > > > +{
> > > > > > +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> > > > > > +		[RENDER_CLASS].reg		= GEN8_RTCR,
> > > > > > +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> > > > > > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> > > > > > +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> > > > > > +	};
> > > > > > +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> > > > > > +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> > > > > > +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> > > > > > +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> > > > > > +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> > > > > > +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> > > > > > +	};
> > > > > > +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> > > > > > +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> > > > > > +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> > > > > > +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> > > > > > +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> > > > > > +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> > > > > > +	};
> > > > > > +	struct drm_i915_private *i915 = engine->i915;
> > > > > > +	const union intel_engine_tlb_inv_reg *regs;
> > > > > > +	union intel_engine_tlb_inv_reg reg;
> > > > > > +	unsigned int class = engine->class;
> > > > > > +	unsigned int num = 0;
> > > > > > +	u32 val;
> > > > > > +
> > > > > > +	/*
> > > > > > +	 * New platforms should not be added with catch-all-newer (>=)
> > > > > > +	 * condition so that any later platform added triggers the below warning
> > > > > > +	 * and in turn mandates a human cross-check of whether the invalidation
> > > > > > +	 * flows have compatible semantics.
> > > > > > +	 *
> > > > > > +	 * For instance with the 11.00 -> 12.00 transition three out of five
> > > > > > +	 * respective engine registers were moved to masked type. Then after the
> > > > > > +	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > > > > > +	 */
> > > > > > +
> > > > > > +	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > > > 
> > > > This is bad...it only captures XEHPSDV and breaks the handling of DG2
> > > > (12.55), PVC (12.60), and MTL (12.70, 12.71, and 12.72).  You're not
> > > > hitting the warning as expected since those are all now being captured
> > > > by the next case of the if/else ladder.  With the way GMD_ID works, we
> > > > may also get new version numbers that silently show up in hardware too
> > > > at some point (e.g., 12.73, 12.74, etc.)
> > > 
> > > Great (on multiple counts) ...
> > > 
> > > > 
> > > > > > +		regs = xehp_regs;
> > > > > > +		num = ARRAY_SIZE(xehp_regs);
> > > > > > +	} else if (GRAPHICS_VER(i915) == 12) {
> > > > 
> > > > You'd want to change this to
> > > > 
> > > >           GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)
> > > > 
> > > > to get the behavior you expected.
> > > 
> > > Okay, that, and then to be as safe as I intended, ie. warn on every new
> > > platforms so developers *must* check registers are still compatible during
> > > platform enablement, we would need a full ver range check something like:
> > > 
> > > 	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&
> > > 	    GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)) {
> > > 		regs = xehp_regs;
> > > 		num = ARRAY_SIZE(xehp_regs);
> > > 	} else if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 0)) {
> > > 		regs = gen12_regs;
> > > 		num = ARRAY_SIZE(gen12_regs);
> > > 
> > > What do you think about that?
> > 
> > What about just keeping the code the way it is now, but adding a new
> > error condition at the *top* of the ladder?
> > 
> >          if (GRAPHICS_VER_FULL(i915) > IP_VER(12, 72)) {
> >                  /* Error, something new we haven't considered */
> >          } else if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> >                  ...
> > 
> > I guess that won't catch cases where they go back and insert a new
> > version or IP refresh that's older than the latest version number, but
> > usually refresh platforms aren't going to be adding the kind of
> > architectural changes that require a new if/else arm anyway.
> > 
> > Also, as Andrzej pointed out on a different reply, we classified DG1 as
> > 12.10 (because of the interrupt handling changes), so I guess we'd need
> > a further adjustment to the "gen12" branch to cover that one too.
> > 
> > Honestly, I'm not super wild about the general plan of intentionally
> > breaking inheritance here; that runs counter to how we usually try to do
> > things in i915.  In most places we always try to assume that future
> > platforms will behave the same way as the latest one, and then we update
> > those assumptions as part of the initial bringup when a future platform
> > comes along that breaks them.  Most of the places where we do specific
> > checks and a MISSING_CASE warning are where we have a strong reason to
> > believe that each platform will always need a new implementation (e.g.,
> > stuff like workaround lists).  I don't see a reason to expect major
> > changes in MMIO-based invalidation for a platform n+1, n+2, etc.  The
> > change that I _do_ expect to show up (even for some of these current
> > platforms) is that a GuC-based TLB invalidation path will be added since
> > that's mandatory for a lot of modern features, but that will be a
> > parallel codepath, not part of the if/else ladder we're editing here.
> 
> The strong reason you mention is currently the lack of _any_ test coverage
> for this. In other words you'd never know a new platform does not work
> properly (imaging one more register becomes masked, like three have become
> from Gen11 to Gen12). Or if refactoring this area of the code broke an
> existing platform. And both cases lead to CVEs so IMO safety trumps our
> usual patterns.
> 
> As I said earlier, I am not going to merge anything in this area, apart that
> I want "drm/i915: Do not cover all future platforms in TLB invalidation" in,
> until test coverage materialises.
> 
> Once we have test coverage I am okay with catch all future platforms.
> 
> So regarding "drm/i915: Do not cover all future platforms in TLB
> invalidation".. minimal change is then to have it like:
> 
>   if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50) &&) {
>       GRAPHICS_VER_FULL(i915) <= IP_VER(12, 55)
> 	...
>   } else if (GRAPHICS_VER(i915) == 12) {
> 	...
>   ...
> 
> Is this agreeable? That covers XEHPSDV and DG2. PVC and MTL, if they are
> intended to be supported, someone (not exclusing myself here) needs to check
> bspec before extending the condition with an explicit patch.

Yeah, that's probably fine (although it does put a hard stop to any hope
of getting preliminary/partial CI coverage of some platforms that are
still in the process of being enabled).

PVC needs the GuC-based invalidation handling (which hasn't been posted
yet); I don't think it even has MMIO invalidation registers like these
so it will never enter this function once the platform is fully enabled.
For MTL I already noted during review of Daniele's GSC series that we'd
need some kind of an update here to include the invalidation register
for the GSC engine (which is of class 'other'), but the change for that
is still coming as well.  So in general I think we do already know when
updates like this are needed for a platform and we should be able to
make sure we have those updates in place before force_probe is removed,
but your concern about being extra paranoid on areas of the code that
can lead to CVEs makes sense as well, so your proposal sounds okay to
me.

> 
> Otherwise 77fa9efc16a9 ("drm/i915/xehp: Create separate reg definitions for
> new MCR registers") was about Xe_HP, already wrong and Andrzej fixed it in
> 4d5cf7b1680a ("drm/i915: fix TLB invalidation for Gen12.50 video and compute
> engines").

I think the problem there was more an artifact of the original fix
04aa64375f48a ("drm/i915: fix TLB invalidation for Gen12 video and
compute engines") silently landing through a non-gfx tree.  That fix was
good on a mainline kernel release at that point in time, but was no
longer a complete solution once the pending DRM tree pull request
landed.  A follow-up after the merge was necessary to bring the second
half of the fix.


Matt

> 
> Further work can happen once test coverage appears.
> 
> Regards,
> 
> Tvrtko
> 
> > > 
> > > Or you are saying new GMD IDs may appear in the field without first having
> > > passed the new platform enablemend process? That would be horrible so I hope
> > > not.
> > 
> > They won't just show up in the wild, we'll know they're coming.  But
> > they'll be used for stuff like how DG2-G12 showed up after we'd already
> > enabled DG2-G10 and DG2-G11.  Or how we picked up a couple more ADL
> > variants that our driver treats as "ADL-P," even though there are
> > technically other SKU letters associated with them.  So they're things
> > that we wouldn't consider to be a whole new platform, just a new
> > subplatform at best.
> > 
> > 
> > Matt
> > 
> > > 
> > > Regards,
> > > 
> > > Tvrtko
> > > 
> > > > > > +		regs = gen12_regs;
> > > > > > +		num = ARRAY_SIZE(gen12_regs);
> > > > > > +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > > > > > +		regs = gen8_regs;
> > > > > > +		num = ARRAY_SIZE(gen8_regs);
> > > > > > +	} else if (GRAPHICS_VER(i915) < 8) {
> > > > > > +		return 0;
> > > > > > +	} > +
> > > > > > +	if (drm_WARN_ONCE(&i915->drm, !num,
> > > > > > +			  "Platform does not implement TLB invalidation!"))
> > > > > > +		return -ENODEV;
> > > > > > +
> > > > > > +	if (drm_WARN_ON_ONCE(&i915->drm,
> > > > > > +			     class >= num ||
> > > > > > +			     (!regs[class].reg.reg &&
> > > > > > +			      !regs[class].mcr_reg.reg)))
> > > > > > +		return -ERANGE;
> > > > > 
> > > > > I hope the propagation of -ERANGE to device probe is OK.
> > > > > 
> > > > > Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
> > > > > 
> > > > > Regards
> > > > > Andrzej
> > > > > 
> > > > > > +
> > > > > > +	reg = regs[class];
> > > > > > +
> > > > > > +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> > > > > > +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > > > > > +		val = 0;
> > > > > > +	} else {
> > > > > > +		val = engine->instance;
> > > > > > +	}
> > > > > > +
> > > > > > +	val = BIT(val);
> > > > > > +
> > > > > > +	engine->tlb_inv.mcr = regs == xehp_regs;
> > > > > > +	engine->tlb_inv.reg = reg;
> > > > > > +	engine->tlb_inv.done = val;
> > > > > > +
> > > > > > +	if (GRAPHICS_VER(i915) >= 12 &&
> > > > > > +	    (engine->class == VIDEO_DECODE_CLASS ||
> > > > > > +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > > > +	     engine->class == COMPUTE_CLASS))
> > > > > > +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> > > > > > +	else
> > > > > > +		engine->tlb_inv.request = val;
> > > > > > +
> > > > > > +	return 0;
> > > > > > +}
> > > > > > +
> > > > > >     static int engine_setup_common(struct intel_engine_cs *engine)
> > > > > >     {
> > > > > >     	int err;
> > > > > >     	init_llist_head(&engine->barrier_tasks);
> > > > > > +	err = intel_engine_init_tlb_invalidation(engine);
> > > > > > +	if (err)
> > > > > > +		return err;
> > > > > > +
> > > > > >     	err = init_status_page(engine);
> > > > > >     	if (err)
> > > > > >     		return err;
> > > > > > diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > > > index 4fd54fb8810f..8c661fe89314 100644
> > > > > > --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > > > +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> > > > > > @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
> > > > > >     	u64 start_gt_clk;
> > > > > >     };
> > > > > > +union intel_engine_tlb_inv_reg {
> > > > > > +	i915_reg_t	reg;
> > > > > > +	i915_mcr_reg_t	mcr_reg;
> > > > > > +};
> > > > > > +
> > > > > > +struct intel_engine_tlb_inv
> > > > > > +{
> > > > > > +	bool mcr;
> > > > > > +	union intel_engine_tlb_inv_reg reg;
> > > > > > +	u32 request;
> > > > > > +	u32 done;
> > > > > > +};
> > > > > > +
> > > > > >     struct intel_engine_cs {
> > > > > >     	struct drm_i915_private *i915;
> > > > > >     	struct intel_gt *gt;
> > > > > > @@ -372,6 +385,8 @@ struct intel_engine_cs {
> > > > > >     	u32 context_size;
> > > > > >     	u32 mmio_base;
> > > > > > +	struct intel_engine_tlb_inv tlb_inv;
> > > > > > +
> > > > > >     	/*
> > > > > >     	 * Some w/a require forcewake to be held (which prevents RC6) while
> > > > > >     	 * a particular engine is active. If so, we set fw_domain to which
> > > > > > diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > > > index 854841a731cb..9fb0ac03f51a 100644
> > > > > > --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > > > +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> > > > > > @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
> > > > > >     	intel_sseu_dump(&info->sseu, p);
> > > > > >     }
> > > > > > -struct reg_and_bit {
> > > > > > -	union {
> > > > > > -		i915_reg_t reg;
> > > > > > -		i915_mcr_reg_t mcr_reg;
> > > > > > -	};
> > > > > > -	u32 bit;
> > > > > > -};
> > > > > > -
> > > > > > -static struct reg_and_bit
> > > > > > -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > > > > > -		const i915_reg_t *regs, const unsigned int num)
> > > > > > -{
> > > > > > -	const unsigned int class = engine->class;
> > > > > > -	struct reg_and_bit rb = { };
> > > > > > -
> > > > > > -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> > > > > > -			     class >= num || !regs[class].reg))
> > > > > > -		return rb;
> > > > > > -
> > > > > > -	rb.reg = regs[class];
> > > > > > -	if (gen8 && class == VIDEO_DECODE_CLASS)
> > > > > > -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> > > > > > -	else
> > > > > > -		rb.bit = engine->instance;
> > > > > > -
> > > > > > -	rb.bit = BIT(rb.bit);
> > > > > > -
> > > > > > -	return rb;
> > > > > > -}
> > > > > > -
> > > > > >     /*
> > > > > >      * HW architecture suggest typical invalidation time at 40us,
> > > > > >      * with pessimistic cases up to 100us and a recommendation to
> > > > > > @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> > > > > >      * but are now considered MCR registers.  Since they exist within a GAM range,
> > > > > >      * the primary instance of the register rolls up the status from each unit.
> > > > > >      */
> > > > > > -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > > > > > +static int wait_for_invalidate(struct intel_engine_cs *engine)
> > > > > >     {
> > > > > > -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> > > > > > -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> > > > > > +	if (engine->tlb_inv.mcr)
> > > > > > +		return intel_gt_mcr_wait_for_reg(engine->gt,
> > > > > > +						 engine->tlb_inv.reg.mcr_reg,
> > > > > > +						 engine->tlb_inv.done,
> > > > > > +						 0,
> > > > > >     						 TLB_INVAL_TIMEOUT_US,
> > > > > >     						 TLB_INVAL_TIMEOUT_MS);
> > > > > >     	else
> > > > > > -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> > > > > > +		return __intel_wait_for_register_fw(engine->gt->uncore,
> > > > > > +						    engine->tlb_inv.reg.reg,
> > > > > > +						    engine->tlb_inv.done,
> > > > > > +						    0,
> > > > > >     						    TLB_INVAL_TIMEOUT_US,
> > > > > >     						    TLB_INVAL_TIMEOUT_MS,
> > > > > >     						    NULL);
> > > > > > @@ -1041,61 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> > > > > >     static void mmio_invalidate_full(struct intel_gt *gt)
> > > > > >     {
> > > > > > -	static const i915_reg_t gen8_regs[] = {
> > > > > > -		[RENDER_CLASS]			= GEN8_RTCR,
> > > > > > -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> > > > > > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> > > > > > -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> > > > > > -	};
> > > > > > -	static const i915_reg_t gen12_regs[] = {
> > > > > > -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> > > > > > -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> > > > > > -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> > > > > > -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> > > > > > -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> > > > > > -	};
> > > > > > -	static const i915_mcr_reg_t xehp_regs[] = {
> > > > > > -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> > > > > > -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> > > > > > -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> > > > > > -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> > > > > > -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> > > > > > -	};
> > > > > >     	struct drm_i915_private *i915 = gt->i915;
> > > > > >     	struct intel_uncore *uncore = gt->uncore;
> > > > > >     	struct intel_engine_cs *engine;
> > > > > >     	intel_engine_mask_t awake, tmp;
> > > > > >     	enum intel_engine_id id;
> > > > > > -	const i915_reg_t *regs;
> > > > > > -	unsigned int num = 0;
> > > > > >     	unsigned long flags;
> > > > > > -	/*
> > > > > > -	 * New platforms should not be added with catch-all-newer (>=)
> > > > > > -	 * condition so that any later platform added triggers the below warning
> > > > > > -	 * and in turn mandates a human cross-check of whether the invalidation
> > > > > > -	 * flows have compatible semantics.
> > > > > > -	 *
> > > > > > -	 * For instance with the 11.00 -> 12.00 transition three out of five
> > > > > > -	 * respective engine registers were moved to masked type. Then after the
> > > > > > -	 * 12.00 -> 12.50 transition multi cast handling is required too.
> > > > > > -	 */
> > > > > > -
> > > > > > -	if (GRAPHICS_VER_FULL(i915) == IP_VER(12, 50)) {
> > > > > > -		regs = NULL;
> > > > > > -		num = ARRAY_SIZE(xehp_regs);
> > > > > > -	} else if (GRAPHICS_VER(i915) == 12) {
> > > > > > -		regs = gen12_regs;
> > > > > > -		num = ARRAY_SIZE(gen12_regs);
> > > > > > -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> > > > > > -		regs = gen8_regs;
> > > > > > -		num = ARRAY_SIZE(gen8_regs);
> > > > > > -	} else if (GRAPHICS_VER(i915) < 8) {
> > > > > > -		return;
> > > > > > -	}
> > > > > > -
> > > > > > -	if (drm_WARN_ONCE(&i915->drm, !num,
> > > > > > -			  "Platform does not implement TLB invalidation!"))
> > > > > > +	if (GRAPHICS_VER(i915) < 8)
> > > > > >     		return;
> > > > > >     	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> > > > > > @@ -1105,33 +1034,18 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> > > > > >     	awake = 0;
> > > > > >     	for_each_engine(engine, gt, id) {
> > > > > > -		struct reg_and_bit rb;
> > > > > > -
> > > > > >     		if (!intel_engine_pm_is_awake(engine))
> > > > > >     			continue;
> > > > > > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > > > > > -			u32 val = BIT(engine->instance);
> > > > > > -
> > > > > > -			if (engine->class == VIDEO_DECODE_CLASS ||
> > > > > > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > > > -			    engine->class == COMPUTE_CLASS)
> > > > > > -				val = _MASKED_BIT_ENABLE(val);
> > > > > > +		if (engine->tlb_inv.mcr)
> > > > > >     			intel_gt_mcr_multicast_write_fw(gt,
> > > > > > -							xehp_regs[engine->class],
> > > > > > -							val);
> > > > > > -		} else {
> > > > > > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > > > > > -			if (!i915_mmio_reg_offset(rb.reg))
> > > > > > -				continue;
> > > > > > -
> > > > > > -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> > > > > > -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> > > > > > -			    engine->class == COMPUTE_CLASS))
> > > > > > -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> > > > > > -
> > > > > > -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> > > > > > -		}
> > > > > > +							engine->tlb_inv.reg.mcr_reg,
> > > > > > +							engine->tlb_inv.request);
> > > > > > +		else
> > > > > > +			intel_uncore_write_fw(uncore,
> > > > > > +					      engine->tlb_inv.reg.reg,
> > > > > > +					      engine->tlb_inv.request);
> > > > > > +
> > > > > >     		awake |= engine->mask;
> > > > > >     	}
> > > > > > @@ -1150,16 +1064,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
> > > > > >     	intel_gt_mcr_unlock(gt, flags);
> > > > > >     	for_each_engine_masked(engine, gt, awake, tmp) {
> > > > > > -		struct reg_and_bit rb;
> > > > > > -
> > > > > > -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> > > > > > -			rb.mcr_reg = xehp_regs[engine->class];
> > > > > > -			rb.bit = BIT(engine->instance);
> > > > > > -		} else {
> > > > > > -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> > > > > > -		}
> > > > > > -
> > > > > > -		if (wait_for_invalidate(gt, rb))
> > > > > > +		if (wait_for_invalidate(engine))
> > > > > >     			drm_err_ratelimited(&gt->i915->drm,
> > > > > >     					    "%s TLB invalidation did not complete in %ums!\n",
> > > > > >     					    engine->name, TLB_INVAL_TIMEOUT_MS);
> > > > > 
> > > > 
> > 

-- 
Matt Roper
Graphics Software Engineer
Linux GPU Platform Enablement
Intel Corporation

[Intel-gfx] ✗ Fi.CI.SPARSE: warning for TLB invalidation cleanup

[Patchwork]

== Series Details ==

Series: TLB invalidation cleanup
URL   : https://patchwork.freedesktop.org/series/112047/
State : warning

== Summary ==

Error: dim sparse failed
Sparse version: v0.6.2
Fast mode used, each commit won't be checked separately.

[Intel-gfx] ✓ Fi.CI.BAT: success for TLB invalidation cleanup

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 11581 bytes --]

== Series Details ==

Series: TLB invalidation cleanup
URL   : https://patchwork.freedesktop.org/series/112047/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_12515 -> Patchwork_112047v1
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/index.html

Participating hosts (40 -> 46)
------------------------------

  Additional (6): fi-bwr-2160 bat-adlp-4 fi-apl-guc fi-snb-2520m fi-cfl-8109u fi-blb-e6850 

Known issues
------------

  Here are the changes found in Patchwork_112047v1 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@debugfs_test@basic-hwmon:
    - bat-adlp-4:         NOTRUN -> [SKIP][1] ([i915#7456])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@debugfs_test@basic-hwmon.html

  * igt@gem_huc_copy@huc-copy:
    - fi-cfl-8109u:       NOTRUN -> [SKIP][2] ([fdo#109271] / [i915#2190])
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-cfl-8109u/igt@gem_huc_copy@huc-copy.html

  * igt@gem_lmem_swapping@basic:
    - fi-apl-guc:         NOTRUN -> [SKIP][3] ([fdo#109271] / [i915#4613]) +3 similar issues
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-apl-guc/igt@gem_lmem_swapping@basic.html

  * igt@gem_lmem_swapping@parallel-random-engines:
    - bat-adlp-4:         NOTRUN -> [SKIP][4] ([i915#4613]) +3 similar issues
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@gem_lmem_swapping@parallel-random-engines.html

  * igt@gem_lmem_swapping@verify-random:
    - fi-cfl-8109u:       NOTRUN -> [SKIP][5] ([fdo#109271] / [i915#4613]) +3 similar issues
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-cfl-8109u/igt@gem_lmem_swapping@verify-random.html

  * igt@gem_tiled_pread_basic:
    - bat-adlp-4:         NOTRUN -> [SKIP][6] ([i915#3282])
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@gem_tiled_pread_basic.html

  * igt@i915_pm_rpm@module-reload:
    - fi-blb-e6850:       NOTRUN -> [SKIP][7] ([fdo#109271]) +51 similar issues
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-blb-e6850/igt@i915_pm_rpm@module-reload.html

  * igt@i915_pm_rps@basic-api:
    - bat-adlp-4:         NOTRUN -> [SKIP][8] ([i915#6621])
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@i915_pm_rps@basic-api.html

  * igt@i915_selftest@live@gt_heartbeat:
    - fi-apl-guc:         NOTRUN -> [DMESG-FAIL][9] ([i915#5334])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-apl-guc/igt@i915_selftest@live@gt_heartbeat.html

  * igt@kms_chamelium@common-hpd-after-suspend:
    - fi-hsw-4770:        NOTRUN -> [SKIP][10] ([fdo#109271] / [fdo#111827])
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-hsw-4770/igt@kms_chamelium@common-hpd-after-suspend.html

  * igt@kms_chamelium@dp-crc-fast:
    - bat-adlp-4:         NOTRUN -> [SKIP][11] ([fdo#111827]) +8 similar issues
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@kms_chamelium@dp-crc-fast.html

  * igt@kms_chamelium@hdmi-crc-fast:
    - fi-apl-guc:         NOTRUN -> [SKIP][12] ([fdo#109271] / [fdo#111827]) +8 similar issues
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-apl-guc/igt@kms_chamelium@hdmi-crc-fast.html
    - fi-snb-2520m:       NOTRUN -> [SKIP][13] ([fdo#109271] / [fdo#111827]) +8 similar issues
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-snb-2520m/igt@kms_chamelium@hdmi-crc-fast.html

  * igt@kms_chamelium@hdmi-edid-read:
    - fi-cfl-8109u:       NOTRUN -> [SKIP][14] ([fdo#109271] / [fdo#111827]) +8 similar issues
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-cfl-8109u/igt@kms_chamelium@hdmi-edid-read.html

  * igt@kms_cursor_legacy@basic-busy-flip-before-cursor:
    - bat-adlp-4:         NOTRUN -> [SKIP][15] ([i915#4103])
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@kms_cursor_legacy@basic-busy-flip-before-cursor.html

  * igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size:
    - fi-bsw-kefka:       [PASS][16] -> [FAIL][17] ([i915#6298])
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/fi-bsw-kefka/igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size.html
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-bsw-kefka/igt@kms_cursor_legacy@basic-busy-flip-before-cursor@atomic-transitions-varying-size.html

  * igt@kms_force_connector_basic@force-load-detect:
    - bat-adlp-4:         NOTRUN -> [SKIP][18] ([i915#4093]) +3 similar issues
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@kms_force_connector_basic@force-load-detect.html

  * igt@kms_pipe_crc_basic@suspend-read-crc@pipe-a-hdmi-a-1:
    - fi-skl-6700k2:      [PASS][19] -> [INCOMPLETE][20] ([i915#7266])
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/fi-skl-6700k2/igt@kms_pipe_crc_basic@suspend-read-crc@pipe-a-hdmi-a-1.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-skl-6700k2/igt@kms_pipe_crc_basic@suspend-read-crc@pipe-a-hdmi-a-1.html

  * igt@kms_psr@primary_mmap_gtt:
    - fi-bwr-2160:        NOTRUN -> [SKIP][21] ([fdo#109271]) +54 similar issues
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-bwr-2160/igt@kms_psr@primary_mmap_gtt.html

  * igt@kms_psr@sprite_plane_onoff:
    - fi-apl-guc:         NOTRUN -> [SKIP][22] ([fdo#109271]) +12 similar issues
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-apl-guc/igt@kms_psr@sprite_plane_onoff.html

  * igt@kms_setmode@basic-clone-single-crtc:
    - bat-adlp-4:         NOTRUN -> [SKIP][23] ([i915#3555] / [i915#4579])
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@kms_setmode@basic-clone-single-crtc.html

  * igt@prime_vgem@basic-fence-flip:
    - fi-snb-2520m:       NOTRUN -> [SKIP][24] ([fdo#109271]) +23 similar issues
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-snb-2520m/igt@prime_vgem@basic-fence-flip.html

  * igt@prime_vgem@basic-userptr:
    - fi-cfl-8109u:       NOTRUN -> [SKIP][25] ([fdo#109271]) +9 similar issues
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-cfl-8109u/igt@prime_vgem@basic-userptr.html
    - bat-adlp-4:         NOTRUN -> [SKIP][26] ([fdo#109295] / [i915#3301] / [i915#3708])
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@prime_vgem@basic-userptr.html

  * igt@prime_vgem@basic-write:
    - bat-adlp-4:         NOTRUN -> [SKIP][27] ([fdo#109295] / [i915#3291] / [i915#3708]) +2 similar issues
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-4/igt@prime_vgem@basic-write.html

  
#### Possible fixes ####

  * igt@i915_selftest@live@hangcheck:
    - fi-hsw-4770:        [INCOMPLETE][28] ([i915#4785]) -> [PASS][29]
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/fi-hsw-4770/igt@i915_selftest@live@hangcheck.html
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/fi-hsw-4770/igt@i915_selftest@live@hangcheck.html
    - {bat-adlm-1}:       [INCOMPLETE][30] ([i915#4983]) -> [PASS][31]
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/bat-adlm-1/igt@i915_selftest@live@hangcheck.html
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlm-1/igt@i915_selftest@live@hangcheck.html

  * igt@i915_selftest@live@reset:
    - {bat-rpls-1}:       [DMESG-FAIL][32] ([i915#4983]) -> [PASS][33]
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/bat-rpls-1/igt@i915_selftest@live@reset.html
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-rpls-1/igt@i915_selftest@live@reset.html

  * igt@i915_selftest@live@workarounds:
    - {bat-dg1-7}:        [DMESG-FAIL][34] -> [PASS][35]
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/bat-dg1-7/igt@i915_selftest@live@workarounds.html
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-dg1-7/igt@i915_selftest@live@workarounds.html

  * igt@i915_suspend@basic-s3-without-i915:
    - {bat-adln-1}:       [DMESG-WARN][36] ([i915#2867]) -> [PASS][37]
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/bat-adln-1/igt@i915_suspend@basic-s3-without-i915.html
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adln-1/igt@i915_suspend@basic-s3-without-i915.html

  * igt@kms_pipe_crc_basic@suspend-read-crc@pipe-b-edp-1:
    - {bat-adlp-6}:       [DMESG-WARN][38] ([i915#2867]) -> [PASS][39]
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/bat-adlp-6/igt@kms_pipe_crc_basic@suspend-read-crc@pipe-b-edp-1.html
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/bat-adlp-6/igt@kms_pipe_crc_basic@suspend-read-crc@pipe-b-edp-1.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [fdo#109295]: https://bugs.freedesktop.org/show_bug.cgi?id=109295
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
  [i915#2867]: https://gitlab.freedesktop.org/drm/intel/issues/2867
  [i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
  [i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
  [i915#3301]: https://gitlab.freedesktop.org/drm/intel/issues/3301
  [i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
  [i915#3708]: https://gitlab.freedesktop.org/drm/intel/issues/3708
  [i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
  [i915#4093]: https://gitlab.freedesktop.org/drm/intel/issues/4093
  [i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
  [i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
  [i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
  [i915#4785]: https://gitlab.freedesktop.org/drm/intel/issues/4785
  [i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
  [i915#5334]: https://gitlab.freedesktop.org/drm/intel/issues/5334
  [i915#6298]: https://gitlab.freedesktop.org/drm/intel/issues/6298
  [i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
  [i915#6621]: https://gitlab.freedesktop.org/drm/intel/issues/6621
  [i915#7266]: https://gitlab.freedesktop.org/drm/intel/issues/7266
  [i915#7456]: https://gitlab.freedesktop.org/drm/intel/issues/7456


Build changes
-------------

  * Linux: CI_DRM_12515 -> Patchwork_112047v1

  CI-20190529: 20190529
  CI_DRM_12515: 25905f33e9cb1630d94ce04688d5a6c756f96e9d @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_7098: d24d605d1b94d404e5ccea13eed607b81e233696 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_112047v1: 25905f33e9cb1630d94ce04688d5a6c756f96e9d @ git://anongit.freedesktop.org/gfx-ci/linux


### Linux commits

4ff6afd59ad9 drm/i915: Consolidate TLB invalidation flow
94f490bf6324 drm/i915: Do not cover all future platforms in TLB invalidation

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/index.html

[-- Attachment #2: Type: text/html, Size: 14148 bytes --]

[Intel-gfx] ✓ Fi.CI.IGT: success for TLB invalidation cleanup

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 17830 bytes --]

== Series Details ==

Series: TLB invalidation cleanup
URL   : https://patchwork.freedesktop.org/series/112047/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_12515_full -> Patchwork_112047v1_full
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/index.html

Participating hosts (14 -> 10)
------------------------------

  Missing    (4): shard-rkl0 pig-kbl-iris pig-glk-j5005 pig-skl-6260u 

Known issues
------------

  Here are the changes found in Patchwork_112047v1_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@kms_cursor_legacy@flip-vs-cursor@atomic-transitions:
    - shard-glk:          [PASS][1] -> [FAIL][2] ([i915#2346])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-glk3/igt@kms_cursor_legacy@flip-vs-cursor@atomic-transitions.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-glk9/igt@kms_cursor_legacy@flip-vs-cursor@atomic-transitions.html

  
#### Possible fixes ####

  * igt@api_intel_bb@object-reloc-keep-cache:
    - {shard-rkl}:        [SKIP][3] ([i915#3281]) -> [PASS][4] +3 similar issues
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@api_intel_bb@object-reloc-keep-cache.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-5/igt@api_intel_bb@object-reloc-keep-cache.html

  * igt@fbdev@eof:
    - {shard-rkl}:        [SKIP][5] ([i915#2582]) -> [PASS][6] +1 similar issue
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@fbdev@eof.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@fbdev@eof.html

  * igt@fbdev@info:
    - {shard-tglu}:       [SKIP][7] ([i915#2582]) -> [PASS][8]
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@fbdev@info.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@fbdev@info.html

  * igt@gem_ctx_isolation@preservation@bcs0:
    - {shard-rkl}:        [FAIL][9] ([i915#7673]) -> [PASS][10] +3 similar issues
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@gem_ctx_isolation@preservation@bcs0.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-5/igt@gem_ctx_isolation@preservation@bcs0.html

  * igt@gem_exec_fair@basic-deadline:
    - shard-glk:          [FAIL][11] ([i915#2846]) -> [PASS][12]
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-glk3/igt@gem_exec_fair@basic-deadline.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-glk9/igt@gem_exec_fair@basic-deadline.html

  * igt@i915_pipe_stress@stress-xrgb8888-untiled:
    - {shard-tglu}:       [SKIP][13] ([i915#1845]) -> [PASS][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@i915_pipe_stress@stress-xrgb8888-untiled.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@i915_pipe_stress@stress-xrgb8888-untiled.html

  * igt@i915_pm_rpm@cursor-dpms:
    - {shard-rkl}:        [SKIP][15] ([i915#1849]) -> [PASS][16] +1 similar issue
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@i915_pm_rpm@cursor-dpms.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@i915_pm_rpm@cursor-dpms.html

  * igt@i915_pm_rpm@system-suspend-modeset:
    - {shard-rkl}:        [SKIP][17] ([fdo#109308]) -> [PASS][18]
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-2/igt@i915_pm_rpm@system-suspend-modeset.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@i915_pm_rpm@system-suspend-modeset.html

  * igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-1:
    - shard-glk:          [FAIL][19] ([i915#2521]) -> [PASS][20] +1 similar issue
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-glk7/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-1.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-glk1/igt@kms_async_flips@alternate-sync-async-flip@pipe-a-hdmi-a-1.html

  * igt@kms_big_fb@linear-32bpp-rotate-0:
    - {shard-rkl}:        [SKIP][21] ([i915#1845] / [i915#4098]) -> [PASS][22] +28 similar issues
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@kms_big_fb@linear-32bpp-rotate-0.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_big_fb@linear-32bpp-rotate-0.html

  * igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc:
    - {shard-tglu}:       [SKIP][23] ([i915#1849]) -> [PASS][24] +1 similar issue
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc.html
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-indfb-draw-mmap-wc.html

  * igt@kms_frontbuffer_tracking@psr-modesetfrombusy:
    - {shard-rkl}:        [SKIP][25] ([i915#1849] / [i915#4098]) -> [PASS][26] +14 similar issues
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@kms_frontbuffer_tracking@psr-modesetfrombusy.html
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_frontbuffer_tracking@psr-modesetfrombusy.html

  * igt@kms_plane@pixel-format-source-clamping@pipe-b-planes:
    - {shard-rkl}:        [SKIP][27] ([i915#1849] / [i915#3558]) -> [PASS][28] +1 similar issue
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-2/igt@kms_plane@pixel-format-source-clamping@pipe-b-planes.html
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_plane@pixel-format-source-clamping@pipe-b-planes.html

  * igt@kms_plane@plane-position-hole-dpms@pipe-a-planes:
    - {shard-tglu}:       [SKIP][29] ([i915#3558]) -> [PASS][30] +1 similar issue
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@kms_plane@plane-position-hole-dpms@pipe-a-planes.html
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@kms_plane@plane-position-hole-dpms@pipe-a-planes.html

  * igt@kms_psr@cursor_render:
    - {shard-rkl}:        [SKIP][31] ([i915#1072]) -> [PASS][32] +2 similar issues
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@kms_psr@cursor_render.html
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_psr@cursor_render.html

  * igt@kms_universal_plane@disable-primary-vs-flip-pipe-b:
    - {shard-rkl}:        [SKIP][33] ([i915#1845] / [i915#4070] / [i915#4098]) -> [PASS][34] +1 similar issue
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-2/igt@kms_universal_plane@disable-primary-vs-flip-pipe-b.html
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_universal_plane@disable-primary-vs-flip-pipe-b.html

  * igt@kms_universal_plane@universal-plane-pageflip-windowed-pipe-a:
    - {shard-rkl}:        [SKIP][35] ([i915#4070] / [i915#4098]) -> [PASS][36]
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-2/igt@kms_universal_plane@universal-plane-pageflip-windowed-pipe-a.html
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-6/igt@kms_universal_plane@universal-plane-pageflip-windowed-pipe-a.html

  * igt@kms_vblank@pipe-a-query-idle:
    - {shard-tglu}:       [SKIP][37] -> [PASS][38] +7 similar issues
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@kms_vblank@pipe-a-query-idle.html
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@kms_vblank@pipe-a-query-idle.html

  * igt@perf@mi-rpc:
    - {shard-rkl}:        [SKIP][39] ([i915#2434]) -> [PASS][40]
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-4/igt@perf@mi-rpc.html
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-5/igt@perf@mi-rpc.html

  * igt@prime_mmap_kms@buffer-sharing:
    - {shard-tglu}:       [SKIP][41] ([fdo#109274]) -> [PASS][42]
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-tglu-6/igt@prime_mmap_kms@buffer-sharing.html
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-tglu-8/igt@prime_mmap_kms@buffer-sharing.html

  * igt@prime_vgem@basic-read:
    - {shard-rkl}:        [SKIP][43] ([fdo#109295] / [i915#3291] / [i915#3708]) -> [PASS][44]
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12515/shard-rkl-1/igt@prime_vgem@basic-read.html
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/shard-rkl-5/igt@prime_vgem@basic-read.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109274]: https://bugs.freedesktop.org/show_bug.cgi?id=109274
  [fdo#109279]: https://bugs.freedesktop.org/show_bug.cgi?id=109279
  [fdo#109280]: https://bugs.freedesktop.org/show_bug.cgi?id=109280
  [fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
  [fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
  [fdo#109291]: https://bugs.freedesktop.org/show_bug.cgi?id=109291
  [fdo#109295]: https://bugs.freedesktop.org/show_bug.cgi?id=109295
  [fdo#109307]: https://bugs.freedesktop.org/show_bug.cgi?id=109307
  [fdo#109308]: https://bugs.freedesktop.org/show_bug.cgi?id=109308
  [fdo#109309]: https://bugs.freedesktop.org/show_bug.cgi?id=109309
  [fdo#109313]: https://bugs.freedesktop.org/show_bug.cgi?id=109313
  [fdo#109315]: https://bugs.freedesktop.org/show_bug.cgi?id=109315
  [fdo#109506]: https://bugs.freedesktop.org/show_bug.cgi?id=109506
  [fdo#109642]: https://bugs.freedesktop.org/show_bug.cgi?id=109642
  [fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
  [fdo#110542]: https://bugs.freedesktop.org/show_bug.cgi?id=110542
  [fdo#110723]: https://bugs.freedesktop.org/show_bug.cgi?id=110723
  [fdo#111068]: https://bugs.freedesktop.org/show_bug.cgi?id=111068
  [fdo#111614]: https://bugs.freedesktop.org/show_bug.cgi?id=111614
  [fdo#111615]: https://bugs.freedesktop.org/show_bug.cgi?id=111615
  [fdo#111656]: https://bugs.freedesktop.org/show_bug.cgi?id=111656
  [fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [fdo#112054]: https://bugs.freedesktop.org/show_bug.cgi?id=112054
  [fdo#112283]: https://bugs.freedesktop.org/show_bug.cgi?id=112283
  [i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
  [i915#1257]: https://gitlab.freedesktop.org/drm/intel/issues/1257
  [i915#132]: https://gitlab.freedesktop.org/drm/intel/issues/132
  [i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
  [i915#1825]: https://gitlab.freedesktop.org/drm/intel/issues/1825
  [i915#1839]: https://gitlab.freedesktop.org/drm/intel/issues/1839
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#1849]: https://gitlab.freedesktop.org/drm/intel/issues/1849
  [i915#2346]: https://gitlab.freedesktop.org/drm/intel/issues/2346
  [i915#2434]: https://gitlab.freedesktop.org/drm/intel/issues/2434
  [i915#2437]: https://gitlab.freedesktop.org/drm/intel/issues/2437
  [i915#2521]: https://gitlab.freedesktop.org/drm/intel/issues/2521
  [i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
  [i915#2575]: https://gitlab.freedesktop.org/drm/intel/issues/2575
  [i915#2582]: https://gitlab.freedesktop.org/drm/intel/issues/2582
  [i915#2587]: https://gitlab.freedesktop.org/drm/intel/issues/2587
  [i915#2658]: https://gitlab.freedesktop.org/drm/intel/issues/2658
  [i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
  [i915#280]: https://gitlab.freedesktop.org/drm/intel/issues/280
  [i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
  [i915#2846]: https://gitlab.freedesktop.org/drm/intel/issues/2846
  [i915#2856]: https://gitlab.freedesktop.org/drm/intel/issues/2856
  [i915#2920]: https://gitlab.freedesktop.org/drm/intel/issues/2920
  [i915#2994]: https://gitlab.freedesktop.org/drm/intel/issues/2994
  [i915#3002]: https://gitlab.freedesktop.org/drm/intel/issues/3002
  [i915#3116]: https://gitlab.freedesktop.org/drm/intel/issues/3116
  [i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
  [i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
  [i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
  [i915#3297]: https://gitlab.freedesktop.org/drm/intel/issues/3297
  [i915#3299]: https://gitlab.freedesktop.org/drm/intel/issues/3299
  [i915#3323]: https://gitlab.freedesktop.org/drm/intel/issues/3323
  [i915#3359]: https://gitlab.freedesktop.org/drm/intel/issues/3359
  [i915#3469]: https://gitlab.freedesktop.org/drm/intel/issues/3469
  [i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
  [i915#3547]: https://gitlab.freedesktop.org/drm/intel/issues/3547
  [i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
  [i915#3558]: https://gitlab.freedesktop.org/drm/intel/issues/3558
  [i915#3637]: https://gitlab.freedesktop.org/drm/intel/issues/3637
  [i915#3638]: https://gitlab.freedesktop.org/drm/intel/issues/3638
  [i915#3639]: https://gitlab.freedesktop.org/drm/intel/issues/3639
  [i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
  [i915#3708]: https://gitlab.freedesktop.org/drm/intel/issues/3708
  [i915#3734]: https://gitlab.freedesktop.org/drm/intel/issues/3734
  [i915#3742]: https://gitlab.freedesktop.org/drm/intel/issues/3742
  [i915#3804]: https://gitlab.freedesktop.org/drm/intel/issues/3804
  [i915#3825]: https://gitlab.freedesktop.org/drm/intel/issues/3825
  [i915#3840]: https://gitlab.freedesktop.org/drm/intel/issues/3840
  [i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
  [i915#4070]: https://gitlab.freedesktop.org/drm/intel/issues/4070
  [i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
  [i915#4098]: https://gitlab.freedesktop.org/drm/intel/issues/4098
  [i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
  [i915#426]: https://gitlab.freedesktop.org/drm/intel/issues/426
  [i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
  [i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
  [i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
  [i915#4991]: https://gitlab.freedesktop.org/drm/intel/issues/4991
  [i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
  [i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
  [i915#5286]: https://gitlab.freedesktop.org/drm/intel/issues/5286
  [i915#5288]: https://gitlab.freedesktop.org/drm/intel/issues/5288
  [i915#5289]: https://gitlab.freedesktop.org/drm/intel/issues/5289
  [i915#533]: https://gitlab.freedesktop.org/drm/intel/issues/533
  [i915#5439]: https://gitlab.freedesktop.org/drm/intel/issues/5439
  [i915#5461]: https://gitlab.freedesktop.org/drm/intel/issues/5461
  [i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
  [i915#6245]: https://gitlab.freedesktop.org/drm/intel/issues/6245
  [i915#6248]: https://gitlab.freedesktop.org/drm/intel/issues/6248
  [i915#6268]: https://gitlab.freedesktop.org/drm/intel/issues/6268
  [i915#6301]: https://gitlab.freedesktop.org/drm/intel/issues/6301
  [i915#6334]: https://gitlab.freedesktop.org/drm/intel/issues/6334
  [i915#6335]: https://gitlab.freedesktop.org/drm/intel/issues/6335
  [i915#6412]: https://gitlab.freedesktop.org/drm/intel/issues/6412
  [i915#6433]: https://gitlab.freedesktop.org/drm/intel/issues/6433
  [i915#6497]: https://gitlab.freedesktop.org/drm/intel/issues/6497
  [i915#6524]: https://gitlab.freedesktop.org/drm/intel/issues/6524
  [i915#658]: https://gitlab.freedesktop.org/drm/intel/issues/658
  [i915#6590]: https://gitlab.freedesktop.org/drm/intel/issues/6590
  [i915#6768]: https://gitlab.freedesktop.org/drm/intel/issues/6768
  [i915#6944]: https://gitlab.freedesktop.org/drm/intel/issues/6944
  [i915#6946]: https://gitlab.freedesktop.org/drm/intel/issues/6946
  [i915#6953]: https://gitlab.freedesktop.org/drm/intel/issues/6953
  [i915#7037]: https://gitlab.freedesktop.org/drm/intel/issues/7037
  [i915#7116]: https://gitlab.freedesktop.org/drm/intel/issues/7116
  [i915#7118]: https://gitlab.freedesktop.org/drm/intel/issues/7118
  [i915#7456]: https://gitlab.freedesktop.org/drm/intel/issues/7456
  [i915#7561]: https://gitlab.freedesktop.org/drm/intel/issues/7561
  [i915#7651]: https://gitlab.freedesktop.org/drm/intel/issues/7651
  [i915#7673]: https://gitlab.freedesktop.org/drm/intel/issues/7673
  [i915#7681]: https://gitlab.freedesktop.org/drm/intel/issues/7681
  [i915#7697]: https://gitlab.freedesktop.org/drm/intel/issues/7697
  [i915#7701]: https://gitlab.freedesktop.org/drm/intel/issues/7701
  [i915#7707]: https://gitlab.freedesktop.org/drm/intel/issues/7707


Build changes
-------------

  * Linux: CI_DRM_12515 -> Patchwork_112047v1
  * Piglit: piglit_4509 -> None

  CI-20190529: 20190529
  CI_DRM_12515: 25905f33e9cb1630d94ce04688d5a6c756f96e9d @ git://anongit.freedesktop.org/gfx-ci/linux
  IGT_7098: d24d605d1b94d404e5ccea13eed607b81e233696 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  Patchwork_112047v1: 25905f33e9cb1630d94ce04688d5a6c756f96e9d @ git://anongit.freedesktop.org/gfx-ci/linux
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_112047v1/index.html

[-- Attachment #2: Type: text/html, Size: 12851 bytes --]

[PATCH 1/2] drm/i915: fix TLB invalidation for Gen12.50 video and compute engines

[Tvrtko Ursulin]

From: Andrzej Hajda <andrzej.hajda@intel.com>

In case of Gen12.50 video and compute engines, TLB_INV registers are
masked - to modify one bit, corresponding bit in upper half of the register
must be enabled, otherwise nothing happens.

Fixes: 77fa9efc16a9 ("drm/i915/xehp: Create separate reg definitions for new MCR registers")
Signed-off-by: Andrzej Hajda <andrzej.hajda@intel.com>
Reviewed-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
---
 drivers/gpu/drm/i915/gt/intel_gt.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 63f95c5f3614..7eeee5a7cb33 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -1100,9 +1100,15 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 			continue;
 
 		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
+			u32 val = BIT(engine->instance);
+
+			if (engine->class == VIDEO_DECODE_CLASS ||
+			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
+			    engine->class == COMPUTE_CLASS)
+				val = _MASKED_BIT_ENABLE(val);
 			intel_gt_mcr_multicast_write_fw(gt,
 							xehp_regs[engine->class],
-							BIT(engine->instance));
+							val);
 		} else {
 			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
 			if (!i915_mmio_reg_offset(rb.reg))
-- 
2.34.1

[Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

As the logic for selecting the register and corresponsing values grew, the
code become a bit unsightly. Consolidate by storing the required values at
engine init time in the engine itself, and by doing so minimise the amount
of invariant platform and engine checks during each and every TLB
invalidation.

Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
Cc: Andrzej Hajda <andrzej.hajda@intel.com>
---
I think this looks nicer, but I don't really want to merge any flavour of
consolidation until we get IGT coverage for the issue upstreamed.
---
 drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  81 ++++++++++++
 drivers/gpu/drm/i915/gt/intel_engine_types.h |  16 +++
 drivers/gpu/drm/i915/gt/intel_gt.c           | 126 ++++---------------
 3 files changed, 120 insertions(+), 103 deletions(-)

diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
index 99c4b866addd..97cdd9853e38 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
+++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
@@ -1143,6 +1143,84 @@ static int init_status_page(struct intel_engine_cs *engine)
 	return ret;
 }
 
+static bool intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
+{
+	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
+		[RENDER_CLASS].reg		= GEN8_RTCR,
+		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
+		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
+	};
+	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
+		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
+	};
+	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
+		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
+		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
+		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
+		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
+		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
+	};
+	struct drm_i915_private *i915 = engine->i915;
+	const union intel_engine_tlb_inv_reg *regs;
+	union intel_engine_tlb_inv_reg reg;
+	unsigned int class = engine->class;
+	unsigned int num = 0;
+	u32 val;
+
+	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
+		regs = xehp_regs;
+		num = ARRAY_SIZE(xehp_regs);
+	} else if (GRAPHICS_VER(i915) == 12) {
+		regs = gen12_regs;
+		num = ARRAY_SIZE(gen12_regs);
+	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
+		regs = gen8_regs;
+		num = ARRAY_SIZE(gen8_regs);
+	} else if (GRAPHICS_VER(i915) < 8) {
+		return false;
+	}
+
+	if (drm_WARN_ONCE(&i915->drm, !num,
+			  "Platform does not implement TLB invalidation!"))
+		return false;
+
+	if (drm_WARN_ON_ONCE(&i915->drm,
+			     class >= num ||
+			     (!regs[class].reg.reg &&
+			      !regs[class].mcr_reg.reg)))
+		return false;
+
+	reg = regs[class];
+
+	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
+		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
+		val = 0;
+	} else {
+		val = engine->instance;
+	}
+
+	val = BIT(val);
+
+	engine->tlb_inv.mcr = regs == xehp_regs;
+	engine->tlb_inv.reg = reg;
+	engine->tlb_inv.done = val;
+
+	if (GRAPHICS_VER(i915) >= 12 &&
+	    (engine->class == VIDEO_DECODE_CLASS ||
+	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
+	     engine->class == COMPUTE_CLASS))
+		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
+	else
+		engine->tlb_inv.request = val;
+
+	return true;
+}
+
 static int engine_setup_common(struct intel_engine_cs *engine)
 {
 	int err;
@@ -1182,6 +1260,9 @@ static int engine_setup_common(struct intel_engine_cs *engine)
 	intel_engine_init_whitelist(engine);
 	intel_engine_init_ctx_wa(engine);
 
+	if (intel_engine_init_tlb_invalidation(engine))
+		engine->flags |= I915_ENGINE_HAS_TLB_INVALIDATION;
+
 	if (GRAPHICS_VER(engine->i915) >= 12)
 		engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
 
diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
index 4fd54fb8810f..8df4a09a6022 100644
--- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
+++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
@@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
 	u64 start_gt_clk;
 };
 
+union intel_engine_tlb_inv_reg {
+	i915_reg_t	reg;
+	i915_mcr_reg_t	mcr_reg;
+};
+
+struct intel_engine_tlb_inv
+{
+	bool mcr;
+	union intel_engine_tlb_inv_reg reg;
+	u32 request;
+	u32 done;
+};
+
 struct intel_engine_cs {
 	struct drm_i915_private *i915;
 	struct intel_gt *gt;
@@ -372,6 +385,8 @@ struct intel_engine_cs {
 	u32 context_size;
 	u32 mmio_base;
 
+	struct intel_engine_tlb_inv tlb_inv;
+
 	/*
 	 * Some w/a require forcewake to be held (which prevents RC6) while
 	 * a particular engine is active. If so, we set fw_domain to which
@@ -556,6 +571,7 @@ struct intel_engine_cs {
 #define I915_ENGINE_HAS_EU_PRIORITY    BIT(10)
 #define I915_ENGINE_FIRST_RENDER_COMPUTE BIT(11)
 #define I915_ENGINE_USES_WA_HOLD_CCS_SWITCHOUT BIT(12)
+#define I915_ENGINE_HAS_TLB_INVALIDATION BIT(13)
 	unsigned int flags;
 
 	/*
diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
index 7eeee5a7cb33..df7afff16fd6 100644
--- a/drivers/gpu/drm/i915/gt/intel_gt.c
+++ b/drivers/gpu/drm/i915/gt/intel_gt.c
@@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
 	intel_sseu_dump(&info->sseu, p);
 }
 
-struct reg_and_bit {
-	union {
-		i915_reg_t reg;
-		i915_mcr_reg_t mcr_reg;
-	};
-	u32 bit;
-};
-
-static struct reg_and_bit
-get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
-		const i915_reg_t *regs, const unsigned int num)
-{
-	const unsigned int class = engine->class;
-	struct reg_and_bit rb = { };
-
-	if (drm_WARN_ON_ONCE(&engine->i915->drm,
-			     class >= num || !regs[class].reg))
-		return rb;
-
-	rb.reg = regs[class];
-	if (gen8 && class == VIDEO_DECODE_CLASS)
-		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
-	else
-		rb.bit = engine->instance;
-
-	rb.bit = BIT(rb.bit);
-
-	return rb;
-}
-
 /*
  * HW architecture suggest typical invalidation time at 40us,
  * with pessimistic cases up to 100us and a recommendation to
@@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
  * but are now considered MCR registers.  Since they exist within a GAM range,
  * the primary instance of the register rolls up the status from each unit.
  */
-static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
+static int wait_for_invalidate(struct intel_engine_cs *engine)
 {
-	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
-		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
+	if (engine->tlb_inv.mcr)
+		return intel_gt_mcr_wait_for_reg(engine->gt,
+						 engine->tlb_inv.reg.mcr_reg,
+						 engine->tlb_inv.done,
+						 0,
 						 TLB_INVAL_TIMEOUT_US,
 						 TLB_INVAL_TIMEOUT_MS);
 	else
-		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
+		return __intel_wait_for_register_fw(engine->gt->uncore,
+						    engine->tlb_inv.reg.reg,
+						    engine->tlb_inv.done,
+						    0,
 						    TLB_INVAL_TIMEOUT_US,
 						    TLB_INVAL_TIMEOUT_MS,
 						    NULL);
@@ -1041,50 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
 
 static void mmio_invalidate_full(struct intel_gt *gt)
 {
-	static const i915_reg_t gen8_regs[] = {
-		[RENDER_CLASS]			= GEN8_RTCR,
-		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
-		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
-	};
-	static const i915_reg_t gen12_regs[] = {
-		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
-	};
-	static const i915_mcr_reg_t xehp_regs[] = {
-		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
-		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
-		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
-		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
-		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
-	};
 	struct drm_i915_private *i915 = gt->i915;
 	struct intel_uncore *uncore = gt->uncore;
 	struct intel_engine_cs *engine;
 	intel_engine_mask_t awake, tmp;
 	enum intel_engine_id id;
-	const i915_reg_t *regs;
-	unsigned int num = 0;
 	unsigned long flags;
 
-	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-		regs = NULL;
-		num = ARRAY_SIZE(xehp_regs);
-	} else if (GRAPHICS_VER(i915) == 12) {
-		regs = gen12_regs;
-		num = ARRAY_SIZE(gen12_regs);
-	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
-		regs = gen8_regs;
-		num = ARRAY_SIZE(gen8_regs);
-	} else if (GRAPHICS_VER(i915) < 8) {
-		return;
-	}
-
-	if (drm_WARN_ONCE(&i915->drm, !num,
-			  "Platform does not implement TLB invalidation!"))
+	if (GRAPHICS_VER(i915) < 8)
 		return;
 
 	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
@@ -1094,33 +1034,22 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 
 	awake = 0;
 	for_each_engine(engine, gt, id) {
-		struct reg_and_bit rb;
-
 		if (!intel_engine_pm_is_awake(engine))
 			continue;
 
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			u32 val = BIT(engine->instance);
+		if (drm_WARN_ON_ONCE(&i915->drm,
+				     !(engine->flags & I915_ENGINE_HAS_TLB_INVALIDATION)))
+			continue;
 
-			if (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS)
-				val = _MASKED_BIT_ENABLE(val);
+		if (engine->tlb_inv.mcr)
 			intel_gt_mcr_multicast_write_fw(gt,
-							xehp_regs[engine->class],
-							val);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-			if (!i915_mmio_reg_offset(rb.reg))
-				continue;
-
-			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
-			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
-			    engine->class == COMPUTE_CLASS))
-				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
-
-			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
-		}
+							engine->tlb_inv.reg.mcr_reg,
+							engine->tlb_inv.request);
+		else
+			intel_uncore_write_fw(uncore,
+					      engine->tlb_inv.reg.reg,
+					      engine->tlb_inv.request);
+
 		awake |= engine->mask;
 	}
 
@@ -1139,16 +1068,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
 	intel_gt_mcr_unlock(gt, flags);
 
 	for_each_engine_masked(engine, gt, awake, tmp) {
-		struct reg_and_bit rb;
-
-		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
-			rb.mcr_reg = xehp_regs[engine->class];
-			rb.bit = BIT(engine->instance);
-		} else {
-			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
-		}
-
-		if (wait_for_invalidate(gt, rb))
+		if (wait_for_invalidate(engine))
 			drm_err_ratelimited(&gt->i915->drm,
 					    "%s TLB invalidation did not complete in %ums!\n",
 					    engine->name, TLB_INVAL_TIMEOUT_MS);
-- 
2.34.1

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Andrzej Hajda]

On 13.12.2022 13:39, Tvrtko Ursulin wrote:
> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> 
> As the logic for selecting the register and corresponsing values grew, the
> code become a bit unsightly. Consolidate by storing the required values at
> engine init time in the engine itself, and by doing so minimise the amount
> of invariant platform and engine checks during each and every TLB
> invalidation.
> 
> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
> ---
> I think this looks nicer, but I don't really want to merge any flavour of
> consolidation until we get IGT coverage for the issue upstreamed.


Yep, the important is to have 1st patch merged.


> ---
>   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  81 ++++++++++++
>   drivers/gpu/drm/i915/gt/intel_engine_types.h |  16 +++
>   drivers/gpu/drm/i915/gt/intel_gt.c           | 126 ++++---------------
>   3 files changed, 120 insertions(+), 103 deletions(-)
> 
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> index 99c4b866addd..97cdd9853e38 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
> @@ -1143,6 +1143,84 @@ static int init_status_page(struct intel_engine_cs *engine)
>   	return ret;
>   }
>   
> +static bool intel_engine_init_tlb_invalidation(struct intel_engine_cs *engine)
> +{
> +	static const union intel_engine_tlb_inv_reg gen8_regs[] = {
> +		[RENDER_CLASS].reg		= GEN8_RTCR,
> +		[VIDEO_DECODE_CLASS].reg	= GEN8_M1TCR, /* , GEN8_M2TCR */
> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN8_VTCR,
> +		[COPY_ENGINE_CLASS].reg		= GEN8_BTCR,
> +	};
> +	static const union intel_engine_tlb_inv_reg gen12_regs[] = {
> +		[RENDER_CLASS].reg		= GEN12_GFX_TLB_INV_CR,
> +		[VIDEO_DECODE_CLASS].reg	= GEN12_VD_TLB_INV_CR,
> +		[VIDEO_ENHANCEMENT_CLASS].reg	= GEN12_VE_TLB_INV_CR,
> +		[COPY_ENGINE_CLASS].reg		= GEN12_BLT_TLB_INV_CR,
> +		[COMPUTE_CLASS].reg		= GEN12_COMPCTX_TLB_INV_CR,
> +	};
> +	static const union intel_engine_tlb_inv_reg xehp_regs[] = {
> +		[RENDER_CLASS].mcr_reg		  = XEHP_GFX_TLB_INV_CR,
> +		[VIDEO_DECODE_CLASS].mcr_reg	  = XEHP_VD_TLB_INV_CR,
> +		[VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
> +		[COPY_ENGINE_CLASS].mcr_reg	  = XEHP_BLT_TLB_INV_CR,
> +		[COMPUTE_CLASS].mcr_reg		  = XEHP_COMPCTX_TLB_INV_CR,
> +	};
> +	struct drm_i915_private *i915 = engine->i915;
> +	const union intel_engine_tlb_inv_reg *regs;
> +	union intel_engine_tlb_inv_reg reg;
> +	unsigned int class = engine->class;
> +	unsigned int num = 0;
> +	u32 val;
> +
> +	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> +		regs = xehp_regs;
> +		num = ARRAY_SIZE(xehp_regs);
> +	} else if (GRAPHICS_VER(i915) == 12) {
> +		regs = gen12_regs;
> +		num = ARRAY_SIZE(gen12_regs);
> +	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {

"GRAPHICS_VER(i915) <= 11" seems redundant.

> +		regs = gen8_regs;
> +		num = ARRAY_SIZE(gen8_regs);
> +	} else if (GRAPHICS_VER(i915) < 8) {

ditto

> +		return false;
> +	}
> +
> +	if (drm_WARN_ONCE(&i915->drm, !num,
> +			  "Platform does not implement TLB invalidation!"))
> +		return false;

It never happens.

> +
> +	if (drm_WARN_ON_ONCE(&i915->drm,
> +			     class >= num ||
> +			     (!regs[class].reg.reg &&
> +			      !regs[class].mcr_reg.reg)))
> +		return false;
> +
> +	reg = regs[class];
> +
> +	if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
> +		reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> +		val = 0;
> +	} else {
> +		val = engine->instance;
> +	}
> +
> +	val = BIT(val);
> +
> +	engine->tlb_inv.mcr = regs == xehp_regs;
> +	engine->tlb_inv.reg = reg;
> +	engine->tlb_inv.done = val;
> +
> +	if (GRAPHICS_VER(i915) >= 12 &&
> +	    (engine->class == VIDEO_DECODE_CLASS ||
> +	     engine->class == VIDEO_ENHANCEMENT_CLASS ||
> +	     engine->class == COMPUTE_CLASS))
> +		engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
> +	else
> +		engine->tlb_inv.request = val;
> +
> +	return true;
> +}
> +
>   static int engine_setup_common(struct intel_engine_cs *engine)
>   {
>   	int err;
> @@ -1182,6 +1260,9 @@ static int engine_setup_common(struct intel_engine_cs *engine)
>   	intel_engine_init_whitelist(engine);
>   	intel_engine_init_ctx_wa(engine);
>   
> +	if (intel_engine_init_tlb_invalidation(engine))
> +		engine->flags |= I915_ENGINE_HAS_TLB_INVALIDATION;
> +
>   	if (GRAPHICS_VER(engine->i915) >= 12)
>   		engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
>   
> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> index 4fd54fb8810f..8df4a09a6022 100644
> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>   	u64 start_gt_clk;
>   };
>   
> +union intel_engine_tlb_inv_reg {
> +	i915_reg_t	reg;
> +	i915_mcr_reg_t	mcr_reg;
> +};
> +
> +struct intel_engine_tlb_inv
> +{
> +	bool mcr;
> +	union intel_engine_tlb_inv_reg reg;
> +	u32 request;
> +	u32 done;
> +};
> +
>   struct intel_engine_cs {
>   	struct drm_i915_private *i915;
>   	struct intel_gt *gt;
> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>   	u32 context_size;
>   	u32 mmio_base;
>   
> +	struct intel_engine_tlb_inv tlb_inv;
> +
>   	/*
>   	 * Some w/a require forcewake to be held (which prevents RC6) while
>   	 * a particular engine is active. If so, we set fw_domain to which
> @@ -556,6 +571,7 @@ struct intel_engine_cs {
>   #define I915_ENGINE_HAS_EU_PRIORITY    BIT(10)
>   #define I915_ENGINE_FIRST_RENDER_COMPUTE BIT(11)
>   #define I915_ENGINE_USES_WA_HOLD_CCS_SWITCHOUT BIT(12)
> +#define I915_ENGINE_HAS_TLB_INVALIDATION BIT(13)
>   	unsigned int flags;
>   
>   	/*
> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c b/drivers/gpu/drm/i915/gt/intel_gt.c
> index 7eeee5a7cb33..df7afff16fd6 100644
> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct intel_gt_info *info,
>   	intel_sseu_dump(&info->sseu, p);
>   }
>   
> -struct reg_and_bit {
> -	union {
> -		i915_reg_t reg;
> -		i915_mcr_reg_t mcr_reg;
> -	};
> -	u32 bit;
> -};
> -
> -static struct reg_and_bit
> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
> -		const i915_reg_t *regs, const unsigned int num)
> -{
> -	const unsigned int class = engine->class;
> -	struct reg_and_bit rb = { };
> -
> -	if (drm_WARN_ON_ONCE(&engine->i915->drm,
> -			     class >= num || !regs[class].reg))
> -		return rb;
> -
> -	rb.reg = regs[class];
> -	if (gen8 && class == VIDEO_DECODE_CLASS)
> -		rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
> -	else
> -		rb.bit = engine->instance;
> -
> -	rb.bit = BIT(rb.bit);
> -
> -	return rb;
> -}
> -
>   /*
>    * HW architecture suggest typical invalidation time at 40us,
>    * with pessimistic cases up to 100us and a recommendation to
> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>    * but are now considered MCR registers.  Since they exist within a GAM range,
>    * the primary instance of the register rolls up the status from each unit.
>    */
> -static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>   {
> -	if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
> -		return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
> +	if (engine->tlb_inv.mcr)
> +		return intel_gt_mcr_wait_for_reg(engine->gt,
> +						 engine->tlb_inv.reg.mcr_reg,
> +						 engine->tlb_inv.done,
> +						 0,
>   						 TLB_INVAL_TIMEOUT_US,
>   						 TLB_INVAL_TIMEOUT_MS);
>   	else
> -		return __intel_wait_for_register_fw(gt->uncore, rb.reg, rb.bit, 0,
> +		return __intel_wait_for_register_fw(engine->gt->uncore,
> +						    engine->tlb_inv.reg.reg,
> +						    engine->tlb_inv.done,
> +						    0,
>   						    TLB_INVAL_TIMEOUT_US,
>   						    TLB_INVAL_TIMEOUT_MS,
>   						    NULL);
> @@ -1041,50 +1017,14 @@ static int wait_for_invalidate(struct intel_gt *gt, struct reg_and_bit rb)
>   
>   static void mmio_invalidate_full(struct intel_gt *gt)
>   {
> -	static const i915_reg_t gen8_regs[] = {
> -		[RENDER_CLASS]			= GEN8_RTCR,
> -		[VIDEO_DECODE_CLASS]		= GEN8_M1TCR, /* , GEN8_M2TCR */
> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN8_VTCR,
> -		[COPY_ENGINE_CLASS]		= GEN8_BTCR,
> -	};
> -	static const i915_reg_t gen12_regs[] = {
> -		[RENDER_CLASS]			= GEN12_GFX_TLB_INV_CR,
> -		[VIDEO_DECODE_CLASS]		= GEN12_VD_TLB_INV_CR,
> -		[VIDEO_ENHANCEMENT_CLASS]	= GEN12_VE_TLB_INV_CR,
> -		[COPY_ENGINE_CLASS]		= GEN12_BLT_TLB_INV_CR,
> -		[COMPUTE_CLASS]			= GEN12_COMPCTX_TLB_INV_CR,
> -	};
> -	static const i915_mcr_reg_t xehp_regs[] = {
> -		[RENDER_CLASS]			= XEHP_GFX_TLB_INV_CR,
> -		[VIDEO_DECODE_CLASS]		= XEHP_VD_TLB_INV_CR,
> -		[VIDEO_ENHANCEMENT_CLASS]	= XEHP_VE_TLB_INV_CR,
> -		[COPY_ENGINE_CLASS]		= XEHP_BLT_TLB_INV_CR,
> -		[COMPUTE_CLASS]			= XEHP_COMPCTX_TLB_INV_CR,
> -	};
>   	struct drm_i915_private *i915 = gt->i915;
>   	struct intel_uncore *uncore = gt->uncore;
>   	struct intel_engine_cs *engine;
>   	intel_engine_mask_t awake, tmp;
>   	enum intel_engine_id id;
> -	const i915_reg_t *regs;
> -	unsigned int num = 0;
>   	unsigned long flags;
>   
> -	if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> -		regs = NULL;
> -		num = ARRAY_SIZE(xehp_regs);
> -	} else if (GRAPHICS_VER(i915) == 12) {
> -		regs = gen12_regs;
> -		num = ARRAY_SIZE(gen12_regs);
> -	} else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> -		regs = gen8_regs;
> -		num = ARRAY_SIZE(gen8_regs);
> -	} else if (GRAPHICS_VER(i915) < 8) {
> -		return;
> -	}
> -
> -	if (drm_WARN_ONCE(&i915->drm, !num,
> -			  "Platform does not implement TLB invalidation!"))
> +	if (GRAPHICS_VER(i915) < 8)
>   		return;
>   
>   	intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
> @@ -1094,33 +1034,22 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>   
>   	awake = 0;
>   	for_each_engine(engine, gt, id) {
> -		struct reg_and_bit rb;
> -
>   		if (!intel_engine_pm_is_awake(engine))
>   			continue;
>   
> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> -			u32 val = BIT(engine->instance);
> +		if (drm_WARN_ON_ONCE(&i915->drm,
> +				     !(engine->flags & I915_ENGINE_HAS_TLB_INVALIDATION)))
> +			continue;

Hmm, can this flag change dynamically? If not why not put it in 
initialization phase.

>   
> -			if (engine->class == VIDEO_DECODE_CLASS ||
> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> -			    engine->class == COMPUTE_CLASS)
> -				val = _MASKED_BIT_ENABLE(val);
> +		if (engine->tlb_inv.mcr)
>   			intel_gt_mcr_multicast_write_fw(gt,
> -							xehp_regs[engine->class],
> -							val);
> -		} else {
> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> -			if (!i915_mmio_reg_offset(rb.reg))
> -				continue;
> -
> -			if (GRAPHICS_VER(i915) == 12 && (engine->class == VIDEO_DECODE_CLASS ||
> -			    engine->class == VIDEO_ENHANCEMENT_CLASS ||
> -			    engine->class == COMPUTE_CLASS))
> -				rb.bit = _MASKED_BIT_ENABLE(rb.bit);
> -
> -			intel_uncore_write_fw(uncore, rb.reg, rb.bit);
> -		}
> +							engine->tlb_inv.reg.mcr_reg,
> +							engine->tlb_inv.request);
> +		else
> +			intel_uncore_write_fw(uncore,
> +					      engine->tlb_inv.reg.reg,
> +					      engine->tlb_inv.request);
> +
>   		awake |= engine->mask;
>   	}
>   
> @@ -1139,16 +1068,7 @@ static void mmio_invalidate_full(struct intel_gt *gt)
>   	intel_gt_mcr_unlock(gt, flags);
>   
>   	for_each_engine_masked(engine, gt, awake, tmp) {
> -		struct reg_and_bit rb;
> -
> -		if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
> -			rb.mcr_reg = xehp_regs[engine->class];
> -			rb.bit = BIT(engine->instance);
> -		} else {
> -			rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
> -		}
> -
> -		if (wait_for_invalidate(gt, rb))
> +		if (wait_for_invalidate(engine))
>   			drm_err_ratelimited(&gt->i915->drm,
>   					    "%s TLB invalidation did not complete in %ums!\n",
>   					    engine->name, TLB_INVAL_TIMEOUT_MS);

With minor comments addressed:

Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>

Regards
Andrzej

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Tvrtko Ursulin]

On 13/12/2022 14:52, Andrzej Hajda wrote:
> On 13.12.2022 13:39, Tvrtko Ursulin wrote:
>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>
>> As the logic for selecting the register and corresponsing values grew, 
>> the
>> code become a bit unsightly. Consolidate by storing the required 
>> values at
>> engine init time in the engine itself, and by doing so minimise the 
>> amount
>> of invariant platform and engine checks during each and every TLB
>> invalidation.
>>
>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>> ---
>> I think this looks nicer, but I don't really want to merge any flavour of
>> consolidation until we get IGT coverage for the issue upstreamed.
> 
> 
> Yep, the important is to have 1st patch merged.

Agreed.

1)
Can you send it standalone so it gets CI results and can get merged?

2)
Could you check internally where did gem_exec_tlb get stuck?

>> ---
>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  81 ++++++++++++
>>   drivers/gpu/drm/i915/gt/intel_engine_types.h |  16 +++
>>   drivers/gpu/drm/i915/gt/intel_gt.c           | 126 ++++---------------
>>   3 files changed, 120 insertions(+), 103 deletions(-)
>>
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> index 99c4b866addd..97cdd9853e38 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>> @@ -1143,6 +1143,84 @@ static int init_status_page(struct 
>> intel_engine_cs *engine)
>>       return ret;
>>   }
>> +static bool intel_engine_init_tlb_invalidation(struct intel_engine_cs 
>> *engine)
>> +{
>> +    static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>> +        [RENDER_CLASS].reg        = GEN8_RTCR,
>> +        [VIDEO_DECODE_CLASS].reg    = GEN8_M1TCR, /* , GEN8_M2TCR */
>> +        [VIDEO_ENHANCEMENT_CLASS].reg    = GEN8_VTCR,
>> +        [COPY_ENGINE_CLASS].reg        = GEN8_BTCR,
>> +    };
>> +    static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>> +        [RENDER_CLASS].reg        = GEN12_GFX_TLB_INV_CR,
>> +        [VIDEO_DECODE_CLASS].reg    = GEN12_VD_TLB_INV_CR,
>> +        [VIDEO_ENHANCEMENT_CLASS].reg    = GEN12_VE_TLB_INV_CR,
>> +        [COPY_ENGINE_CLASS].reg        = GEN12_BLT_TLB_INV_CR,
>> +        [COMPUTE_CLASS].reg        = GEN12_COMPCTX_TLB_INV_CR,
>> +    };
>> +    static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>> +        [RENDER_CLASS].mcr_reg          = XEHP_GFX_TLB_INV_CR,
>> +        [VIDEO_DECODE_CLASS].mcr_reg      = XEHP_VD_TLB_INV_CR,
>> +        [VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>> +        [COPY_ENGINE_CLASS].mcr_reg      = XEHP_BLT_TLB_INV_CR,
>> +        [COMPUTE_CLASS].mcr_reg          = XEHP_COMPCTX_TLB_INV_CR,
>> +    };
>> +    struct drm_i915_private *i915 = engine->i915;
>> +    const union intel_engine_tlb_inv_reg *regs;
>> +    union intel_engine_tlb_inv_reg reg;
>> +    unsigned int class = engine->class;
>> +    unsigned int num = 0;
>> +    u32 val;
>> +
>> +    if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>> +        regs = xehp_regs;
>> +        num = ARRAY_SIZE(xehp_regs);
>> +    } else if (GRAPHICS_VER(i915) == 12) {
>> +        regs = gen12_regs;
>> +        num = ARRAY_SIZE(gen12_regs);
>> +    } else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
> 
> "GRAPHICS_VER(i915) <= 11" seems redundant.

Code movement... Fixes: "patch which added >= 12,50" to clean just that 
part up. Before it was deliberately made to hit the below warn on so 
that any new platform added requires a human checking if the registers 
are still same and compatible. (Instead of assuming future platforms 
remain compatible - which is IMO too big risk to take.)

> 
>> +        regs = gen8_regs;
>> +        num = ARRAY_SIZE(gen8_regs);
>> +    } else if (GRAPHICS_VER(i915) < 8) {
> 
> ditto

How? We want to do nothing before gen 8.

>> +        return false;
>> +    }
>> +
>> +    if (drm_WARN_ONCE(&i915->drm, !num,
>> +              "Platform does not implement TLB invalidation!"))
>> +        return false;
> 
> It never happens.

As above a) it's just code movement and b) it would be safer if it could 
happen.

>> +
>> +    if (drm_WARN_ON_ONCE(&i915->drm,
>> +                 class >= num ||
>> +                 (!regs[class].reg.reg &&
>> +                  !regs[class].mcr_reg.reg)))
>> +        return false;
>> +
>> +    reg = regs[class];
>> +
>> +    if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>> +        reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>> +        val = 0;
>> +    } else {
>> +        val = engine->instance;
>> +    }
>> +
>> +    val = BIT(val);
>> +
>> +    engine->tlb_inv.mcr = regs == xehp_regs;
>> +    engine->tlb_inv.reg = reg;
>> +    engine->tlb_inv.done = val;
>> +
>> +    if (GRAPHICS_VER(i915) >= 12 &&
>> +        (engine->class == VIDEO_DECODE_CLASS ||
>> +         engine->class == VIDEO_ENHANCEMENT_CLASS ||
>> +         engine->class == COMPUTE_CLASS))
>> +        engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>> +    else
>> +        engine->tlb_inv.request = val;
>> +
>> +    return true;
>> +}
>> +
>>   static int engine_setup_common(struct intel_engine_cs *engine)
>>   {
>>       int err;
>> @@ -1182,6 +1260,9 @@ static int engine_setup_common(struct 
>> intel_engine_cs *engine)
>>       intel_engine_init_whitelist(engine);
>>       intel_engine_init_ctx_wa(engine);
>> +    if (intel_engine_init_tlb_invalidation(engine))
>> +        engine->flags |= I915_ENGINE_HAS_TLB_INVALIDATION;
>> +
>>       if (GRAPHICS_VER(engine->i915) >= 12)
>>           engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h 
>> b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>> index 4fd54fb8810f..8df4a09a6022 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>       u64 start_gt_clk;
>>   };
>> +union intel_engine_tlb_inv_reg {
>> +    i915_reg_t    reg;
>> +    i915_mcr_reg_t    mcr_reg;
>> +};
>> +
>> +struct intel_engine_tlb_inv
>> +{
>> +    bool mcr;
>> +    union intel_engine_tlb_inv_reg reg;
>> +    u32 request;
>> +    u32 done;
>> +};
>> +
>>   struct intel_engine_cs {
>>       struct drm_i915_private *i915;
>>       struct intel_gt *gt;
>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>       u32 context_size;
>>       u32 mmio_base;
>> +    struct intel_engine_tlb_inv tlb_inv;
>> +
>>       /*
>>        * Some w/a require forcewake to be held (which prevents RC6) while
>>        * a particular engine is active. If so, we set fw_domain to which
>> @@ -556,6 +571,7 @@ struct intel_engine_cs {
>>   #define I915_ENGINE_HAS_EU_PRIORITY    BIT(10)
>>   #define I915_ENGINE_FIRST_RENDER_COMPUTE BIT(11)
>>   #define I915_ENGINE_USES_WA_HOLD_CCS_SWITCHOUT BIT(12)
>> +#define I915_ENGINE_HAS_TLB_INVALIDATION BIT(13)
>>       unsigned int flags;
>>       /*
>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c 
>> b/drivers/gpu/drm/i915/gt/intel_gt.c
>> index 7eeee5a7cb33..df7afff16fd6 100644
>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct 
>> intel_gt_info *info,
>>       intel_sseu_dump(&info->sseu, p);
>>   }
>> -struct reg_and_bit {
>> -    union {
>> -        i915_reg_t reg;
>> -        i915_mcr_reg_t mcr_reg;
>> -    };
>> -    u32 bit;
>> -};
>> -
>> -static struct reg_and_bit
>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>> -        const i915_reg_t *regs, const unsigned int num)
>> -{
>> -    const unsigned int class = engine->class;
>> -    struct reg_and_bit rb = { };
>> -
>> -    if (drm_WARN_ON_ONCE(&engine->i915->drm,
>> -                 class >= num || !regs[class].reg))
>> -        return rb;
>> -
>> -    rb.reg = regs[class];
>> -    if (gen8 && class == VIDEO_DECODE_CLASS)
>> -        rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>> -    else
>> -        rb.bit = engine->instance;
>> -
>> -    rb.bit = BIT(rb.bit);
>> -
>> -    return rb;
>> -}
>> -
>>   /*
>>    * HW architecture suggest typical invalidation time at 40us,
>>    * with pessimistic cases up to 100us and a recommendation to
>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs 
>> *engine, const bool gen8,
>>    * but are now considered MCR registers.  Since they exist within a 
>> GAM range,
>>    * the primary instance of the register rolls up the status from 
>> each unit.
>>    */
>> -static int wait_for_invalidate(struct intel_gt *gt, struct 
>> reg_and_bit rb)
>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>   {
>> -    if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>> -        return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>> +    if (engine->tlb_inv.mcr)
>> +        return intel_gt_mcr_wait_for_reg(engine->gt,
>> +                         engine->tlb_inv.reg.mcr_reg,
>> +                         engine->tlb_inv.done,
>> +                         0,
>>                            TLB_INVAL_TIMEOUT_US,
>>                            TLB_INVAL_TIMEOUT_MS);
>>       else
>> -        return __intel_wait_for_register_fw(gt->uncore, rb.reg, 
>> rb.bit, 0,
>> +        return __intel_wait_for_register_fw(engine->gt->uncore,
>> +                            engine->tlb_inv.reg.reg,
>> +                            engine->tlb_inv.done,
>> +                            0,
>>                               TLB_INVAL_TIMEOUT_US,
>>                               TLB_INVAL_TIMEOUT_MS,
>>                               NULL);
>> @@ -1041,50 +1017,14 @@ static int wait_for_invalidate(struct intel_gt 
>> *gt, struct reg_and_bit rb)
>>   static void mmio_invalidate_full(struct intel_gt *gt)
>>   {
>> -    static const i915_reg_t gen8_regs[] = {
>> -        [RENDER_CLASS]            = GEN8_RTCR,
>> -        [VIDEO_DECODE_CLASS]        = GEN8_M1TCR, /* , GEN8_M2TCR */
>> -        [VIDEO_ENHANCEMENT_CLASS]    = GEN8_VTCR,
>> -        [COPY_ENGINE_CLASS]        = GEN8_BTCR,
>> -    };
>> -    static const i915_reg_t gen12_regs[] = {
>> -        [RENDER_CLASS]            = GEN12_GFX_TLB_INV_CR,
>> -        [VIDEO_DECODE_CLASS]        = GEN12_VD_TLB_INV_CR,
>> -        [VIDEO_ENHANCEMENT_CLASS]    = GEN12_VE_TLB_INV_CR,
>> -        [COPY_ENGINE_CLASS]        = GEN12_BLT_TLB_INV_CR,
>> -        [COMPUTE_CLASS]            = GEN12_COMPCTX_TLB_INV_CR,
>> -    };
>> -    static const i915_mcr_reg_t xehp_regs[] = {
>> -        [RENDER_CLASS]            = XEHP_GFX_TLB_INV_CR,
>> -        [VIDEO_DECODE_CLASS]        = XEHP_VD_TLB_INV_CR,
>> -        [VIDEO_ENHANCEMENT_CLASS]    = XEHP_VE_TLB_INV_CR,
>> -        [COPY_ENGINE_CLASS]        = XEHP_BLT_TLB_INV_CR,
>> -        [COMPUTE_CLASS]            = XEHP_COMPCTX_TLB_INV_CR,
>> -    };
>>       struct drm_i915_private *i915 = gt->i915;
>>       struct intel_uncore *uncore = gt->uncore;
>>       struct intel_engine_cs *engine;
>>       intel_engine_mask_t awake, tmp;
>>       enum intel_engine_id id;
>> -    const i915_reg_t *regs;
>> -    unsigned int num = 0;
>>       unsigned long flags;
>> -    if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>> -        regs = NULL;
>> -        num = ARRAY_SIZE(xehp_regs);
>> -    } else if (GRAPHICS_VER(i915) == 12) {
>> -        regs = gen12_regs;
>> -        num = ARRAY_SIZE(gen12_regs);
>> -    } else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>> -        regs = gen8_regs;
>> -        num = ARRAY_SIZE(gen8_regs);
>> -    } else if (GRAPHICS_VER(i915) < 8) {
>> -        return;
>> -    }
>> -
>> -    if (drm_WARN_ONCE(&i915->drm, !num,
>> -              "Platform does not implement TLB invalidation!"))
>> +    if (GRAPHICS_VER(i915) < 8)
>>           return;
>>       intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>> @@ -1094,33 +1034,22 @@ static void mmio_invalidate_full(struct 
>> intel_gt *gt)
>>       awake = 0;
>>       for_each_engine(engine, gt, id) {
>> -        struct reg_and_bit rb;
>> -
>>           if (!intel_engine_pm_is_awake(engine))
>>               continue;
>> -        if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>> -            u32 val = BIT(engine->instance);
>> +        if (drm_WARN_ON_ONCE(&i915->drm,
>> +                     !(engine->flags & 
>> I915_ENGINE_HAS_TLB_INVALIDATION)))
>> +            continue;
> 
> Hmm, can this flag change dynamically? If not why not put it in 
> initialization phase.

Could do. In that case I couldn't have any asserts that engine->tlb_inv 
is valid but perhaps that is okay. Could just fail engine probe from 
there if register table is incomplete. Sounds better indeed, I'll change it.

> 
>> -            if (engine->class == VIDEO_DECODE_CLASS ||
>> -                engine->class == VIDEO_ENHANCEMENT_CLASS ||
>> -                engine->class == COMPUTE_CLASS)
>> -                val = _MASKED_BIT_ENABLE(val);
>> +        if (engine->tlb_inv.mcr)
>>               intel_gt_mcr_multicast_write_fw(gt,
>> -                            xehp_regs[engine->class],
>> -                            val);
>> -        } else {
>> -            rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>> -            if (!i915_mmio_reg_offset(rb.reg))
>> -                continue;
>> -
>> -            if (GRAPHICS_VER(i915) == 12 && (engine->class == 
>> VIDEO_DECODE_CLASS ||
>> -                engine->class == VIDEO_ENHANCEMENT_CLASS ||
>> -                engine->class == COMPUTE_CLASS))
>> -                rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>> -
>> -            intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>> -        }
>> +                            engine->tlb_inv.reg.mcr_reg,
>> +                            engine->tlb_inv.request);
>> +        else
>> +            intel_uncore_write_fw(uncore,
>> +                          engine->tlb_inv.reg.reg,
>> +                          engine->tlb_inv.request);
>> +
>>           awake |= engine->mask;
>>       }
>> @@ -1139,16 +1068,7 @@ static void mmio_invalidate_full(struct 
>> intel_gt *gt)
>>       intel_gt_mcr_unlock(gt, flags);
>>       for_each_engine_masked(engine, gt, awake, tmp) {
>> -        struct reg_and_bit rb;
>> -
>> -        if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>> -            rb.mcr_reg = xehp_regs[engine->class];
>> -            rb.bit = BIT(engine->instance);
>> -        } else {
>> -            rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>> -        }
>> -
>> -        if (wait_for_invalidate(gt, rb))
>> +        if (wait_for_invalidate(engine))
>>               drm_err_ratelimited(&gt->i915->drm,
>>                           "%s TLB invalidation did not complete in 
>> %ums!\n",
>>                           engine->name, TLB_INVAL_TIMEOUT_MS);
> 
> With minor comments addressed:
> 
> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>

I'll send v2 at some point, please stand by.

Regards,

Tvrtko

Re: [Intel-gfx] [PATCH 2/2] drm/i915: Consolidate TLB invalidation flow

[Andrzej Hajda]

On 13.12.2022 16:22, Tvrtko Ursulin wrote:
> 
> On 13/12/2022 14:52, Andrzej Hajda wrote:
>> On 13.12.2022 13:39, Tvrtko Ursulin wrote:
>>> From: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>>
>>> As the logic for selecting the register and corresponsing values 
>>> grew, the
>>> code become a bit unsightly. Consolidate by storing the required 
>>> values at
>>> engine init time in the engine itself, and by doing so minimise the 
>>> amount
>>> of invariant platform and engine checks during each and every TLB
>>> invalidation.
>>>
>>> Signed-off-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>
>>> Cc: Andrzej Hajda <andrzej.hajda@intel.com>
>>> ---
>>> I think this looks nicer, but I don't really want to merge any 
>>> flavour of
>>> consolidation until we get IGT coverage for the issue upstreamed.
>>
>>
>> Yep, the important is to have 1st patch merged.
> 
> Agreed.
> 
> 1)
> Can you send it standalone so it gets CI results and can get merged?

Done.

> 
> 2)
> Could you check internally where did gem_exec_tlb get stuck? >
>>> ---
>>>   drivers/gpu/drm/i915/gt/intel_engine_cs.c    |  81 ++++++++++++
>>>   drivers/gpu/drm/i915/gt/intel_engine_types.h |  16 +++
>>>   drivers/gpu/drm/i915/gt/intel_gt.c           | 126 ++++---------------
>>>   3 files changed, 120 insertions(+), 103 deletions(-)
>>>
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_cs.c 
>>> b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> index 99c4b866addd..97cdd9853e38 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_cs.c
>>> @@ -1143,6 +1143,84 @@ static int init_status_page(struct 
>>> intel_engine_cs *engine)
>>>       return ret;
>>>   }
>>> +static bool intel_engine_init_tlb_invalidation(struct 
>>> intel_engine_cs *engine)
>>> +{
>>> +    static const union intel_engine_tlb_inv_reg gen8_regs[] = {
>>> +        [RENDER_CLASS].reg        = GEN8_RTCR,
>>> +        [VIDEO_DECODE_CLASS].reg    = GEN8_M1TCR, /* , GEN8_M2TCR */
>>> +        [VIDEO_ENHANCEMENT_CLASS].reg    = GEN8_VTCR,
>>> +        [COPY_ENGINE_CLASS].reg        = GEN8_BTCR,
>>> +    };
>>> +    static const union intel_engine_tlb_inv_reg gen12_regs[] = {
>>> +        [RENDER_CLASS].reg        = GEN12_GFX_TLB_INV_CR,
>>> +        [VIDEO_DECODE_CLASS].reg    = GEN12_VD_TLB_INV_CR,
>>> +        [VIDEO_ENHANCEMENT_CLASS].reg    = GEN12_VE_TLB_INV_CR,
>>> +        [COPY_ENGINE_CLASS].reg        = GEN12_BLT_TLB_INV_CR,
>>> +        [COMPUTE_CLASS].reg        = GEN12_COMPCTX_TLB_INV_CR,
>>> +    };
>>> +    static const union intel_engine_tlb_inv_reg xehp_regs[] = {
>>> +        [RENDER_CLASS].mcr_reg          = XEHP_GFX_TLB_INV_CR,
>>> +        [VIDEO_DECODE_CLASS].mcr_reg      = XEHP_VD_TLB_INV_CR,
>>> +        [VIDEO_ENHANCEMENT_CLASS].mcr_reg = XEHP_VE_TLB_INV_CR,
>>> +        [COPY_ENGINE_CLASS].mcr_reg      = XEHP_BLT_TLB_INV_CR,
>>> +        [COMPUTE_CLASS].mcr_reg          = XEHP_COMPCTX_TLB_INV_CR,
>>> +    };
>>> +    struct drm_i915_private *i915 = engine->i915;
>>> +    const union intel_engine_tlb_inv_reg *regs;
>>> +    union intel_engine_tlb_inv_reg reg;
>>> +    unsigned int class = engine->class;
>>> +    unsigned int num = 0;
>>> +    u32 val;
>>> +
>>> +    if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> +        regs = xehp_regs;
>>> +        num = ARRAY_SIZE(xehp_regs);
>>> +    } else if (GRAPHICS_VER(i915) == 12) {
>>> +        regs = gen12_regs;
>>> +        num = ARRAY_SIZE(gen12_regs);
>>> +    } else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>
>> "GRAPHICS_VER(i915) <= 11" seems redundant.
> 
> Code movement... Fixes: "patch which added >= 12,50" to clean just that 
> part up. Before it was deliberately made to hit the below warn on so 
> that any new platform added requires a human checking if the registers 
> are still same and compatible. (Instead of assuming future platforms 
> remain compatible - which is IMO too big risk to take.)
> 
>>
>>> +        regs = gen8_regs;
>>> +        num = ARRAY_SIZE(gen8_regs);
>>> +    } else if (GRAPHICS_VER(i915) < 8) {
>>
>> ditto
> 
> How? We want to do nothing before gen 8.

} else {
	return false;
}

?

As I understand there are three 'rules' we want to follow:
1. start from the newest hw, end at the oldest.
2. warn for not yet supported platforms, lets assume it is ver >= 13.
3. cover all cases

Why not use >= consistently, till exhaustion:

if (ver >= 13) {
	warn
	return false;
} else if (ver >= 12.50) {
	...
} else if (ver >= 12) {
	...
} else if (ver >= 8) {
	...
} else {
	return false;
}

It looks cleaner, ordered, no redundant checks, no doubts, hopefuly 
future-proof.

Regards
Andrzej

> 
>>> +        return false;
>>> +    }
>>> +
>>> +    if (drm_WARN_ONCE(&i915->drm, !num,
>>> +              "Platform does not implement TLB invalidation!"))
>>> +        return false;
>>
>> It never happens.
> 
> As above a) it's just code movement and b) it would be safer if it could 
> happen.
> 
>>> +
>>> +    if (drm_WARN_ON_ONCE(&i915->drm,
>>> +                 class >= num ||
>>> +                 (!regs[class].reg.reg &&
>>> +                  !regs[class].mcr_reg.reg)))
>>> +        return false;
>>> +
>>> +    reg = regs[class];
>>> +
>>> +    if (GRAPHICS_VER(i915) == 8 && class == VIDEO_DECODE_CLASS) {
>>> +        reg.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> +        val = 0;
>>> +    } else {
>>> +        val = engine->instance;
>>> +    }
>>> +
>>> +    val = BIT(val);
>>> +
>>> +    engine->tlb_inv.mcr = regs == xehp_regs;
>>> +    engine->tlb_inv.reg = reg;
>>> +    engine->tlb_inv.done = val;
>>> +
>>> +    if (GRAPHICS_VER(i915) >= 12 &&
>>> +        (engine->class == VIDEO_DECODE_CLASS ||
>>> +         engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> +         engine->class == COMPUTE_CLASS))
>>> +        engine->tlb_inv.request = _MASKED_BIT_ENABLE(val);
>>> +    else
>>> +        engine->tlb_inv.request = val;
>>> +
>>> +    return true;
>>> +}
>>> +
>>>   static int engine_setup_common(struct intel_engine_cs *engine)
>>>   {
>>>       int err;
>>> @@ -1182,6 +1260,9 @@ static int engine_setup_common(struct 
>>> intel_engine_cs *engine)
>>>       intel_engine_init_whitelist(engine);
>>>       intel_engine_init_ctx_wa(engine);
>>> +    if (intel_engine_init_tlb_invalidation(engine))
>>> +        engine->flags |= I915_ENGINE_HAS_TLB_INVALIDATION;
>>> +
>>>       if (GRAPHICS_VER(engine->i915) >= 12)
>>>           engine->flags |= I915_ENGINE_HAS_RELATIVE_MMIO;
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_engine_types.h 
>>> b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> index 4fd54fb8810f..8df4a09a6022 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> +++ b/drivers/gpu/drm/i915/gt/intel_engine_types.h
>>> @@ -341,6 +341,19 @@ struct intel_engine_guc_stats {
>>>       u64 start_gt_clk;
>>>   };
>>> +union intel_engine_tlb_inv_reg {
>>> +    i915_reg_t    reg;
>>> +    i915_mcr_reg_t    mcr_reg;
>>> +};
>>> +
>>> +struct intel_engine_tlb_inv
>>> +{
>>> +    bool mcr;
>>> +    union intel_engine_tlb_inv_reg reg;
>>> +    u32 request;
>>> +    u32 done;
>>> +};
>>> +
>>>   struct intel_engine_cs {
>>>       struct drm_i915_private *i915;
>>>       struct intel_gt *gt;
>>> @@ -372,6 +385,8 @@ struct intel_engine_cs {
>>>       u32 context_size;
>>>       u32 mmio_base;
>>> +    struct intel_engine_tlb_inv tlb_inv;
>>> +
>>>       /*
>>>        * Some w/a require forcewake to be held (which prevents RC6) 
>>> while
>>>        * a particular engine is active. If so, we set fw_domain to which
>>> @@ -556,6 +571,7 @@ struct intel_engine_cs {
>>>   #define I915_ENGINE_HAS_EU_PRIORITY    BIT(10)
>>>   #define I915_ENGINE_FIRST_RENDER_COMPUTE BIT(11)
>>>   #define I915_ENGINE_USES_WA_HOLD_CCS_SWITCHOUT BIT(12)
>>> +#define I915_ENGINE_HAS_TLB_INVALIDATION BIT(13)
>>>       unsigned int flags;
>>>       /*
>>> diff --git a/drivers/gpu/drm/i915/gt/intel_gt.c 
>>> b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> index 7eeee5a7cb33..df7afff16fd6 100644
>>> --- a/drivers/gpu/drm/i915/gt/intel_gt.c
>>> +++ b/drivers/gpu/drm/i915/gt/intel_gt.c
>>> @@ -983,36 +983,6 @@ void intel_gt_info_print(const struct 
>>> intel_gt_info *info,
>>>       intel_sseu_dump(&info->sseu, p);
>>>   }
>>> -struct reg_and_bit {
>>> -    union {
>>> -        i915_reg_t reg;
>>> -        i915_mcr_reg_t mcr_reg;
>>> -    };
>>> -    u32 bit;
>>> -};
>>> -
>>> -static struct reg_and_bit
>>> -get_reg_and_bit(const struct intel_engine_cs *engine, const bool gen8,
>>> -        const i915_reg_t *regs, const unsigned int num)
>>> -{
>>> -    const unsigned int class = engine->class;
>>> -    struct reg_and_bit rb = { };
>>> -
>>> -    if (drm_WARN_ON_ONCE(&engine->i915->drm,
>>> -                 class >= num || !regs[class].reg))
>>> -        return rb;
>>> -
>>> -    rb.reg = regs[class];
>>> -    if (gen8 && class == VIDEO_DECODE_CLASS)
>>> -        rb.reg.reg += 4 * engine->instance; /* GEN8_M2TCR */
>>> -    else
>>> -        rb.bit = engine->instance;
>>> -
>>> -    rb.bit = BIT(rb.bit);
>>> -
>>> -    return rb;
>>> -}
>>> -
>>>   /*
>>>    * HW architecture suggest typical invalidation time at 40us,
>>>    * with pessimistic cases up to 100us and a recommendation to
>>> @@ -1026,14 +996,20 @@ get_reg_and_bit(const struct intel_engine_cs 
>>> *engine, const bool gen8,
>>>    * but are now considered MCR registers.  Since they exist within a 
>>> GAM range,
>>>    * the primary instance of the register rolls up the status from 
>>> each unit.
>>>    */
>>> -static int wait_for_invalidate(struct intel_gt *gt, struct 
>>> reg_and_bit rb)
>>> +static int wait_for_invalidate(struct intel_engine_cs *engine)
>>>   {
>>> -    if (GRAPHICS_VER_FULL(gt->i915) >= IP_VER(12, 50))
>>> -        return intel_gt_mcr_wait_for_reg(gt, rb.mcr_reg, rb.bit, 0,
>>> +    if (engine->tlb_inv.mcr)
>>> +        return intel_gt_mcr_wait_for_reg(engine->gt,
>>> +                         engine->tlb_inv.reg.mcr_reg,
>>> +                         engine->tlb_inv.done,
>>> +                         0,
>>>                            TLB_INVAL_TIMEOUT_US,
>>>                            TLB_INVAL_TIMEOUT_MS);
>>>       else
>>> -        return __intel_wait_for_register_fw(gt->uncore, rb.reg, 
>>> rb.bit, 0,
>>> +        return __intel_wait_for_register_fw(engine->gt->uncore,
>>> +                            engine->tlb_inv.reg.reg,
>>> +                            engine->tlb_inv.done,
>>> +                            0,
>>>                               TLB_INVAL_TIMEOUT_US,
>>>                               TLB_INVAL_TIMEOUT_MS,
>>>                               NULL);
>>> @@ -1041,50 +1017,14 @@ static int wait_for_invalidate(struct 
>>> intel_gt *gt, struct reg_and_bit rb)
>>>   static void mmio_invalidate_full(struct intel_gt *gt)
>>>   {
>>> -    static const i915_reg_t gen8_regs[] = {
>>> -        [RENDER_CLASS]            = GEN8_RTCR,
>>> -        [VIDEO_DECODE_CLASS]        = GEN8_M1TCR, /* , GEN8_M2TCR */
>>> -        [VIDEO_ENHANCEMENT_CLASS]    = GEN8_VTCR,
>>> -        [COPY_ENGINE_CLASS]        = GEN8_BTCR,
>>> -    };
>>> -    static const i915_reg_t gen12_regs[] = {
>>> -        [RENDER_CLASS]            = GEN12_GFX_TLB_INV_CR,
>>> -        [VIDEO_DECODE_CLASS]        = GEN12_VD_TLB_INV_CR,
>>> -        [VIDEO_ENHANCEMENT_CLASS]    = GEN12_VE_TLB_INV_CR,
>>> -        [COPY_ENGINE_CLASS]        = GEN12_BLT_TLB_INV_CR,
>>> -        [COMPUTE_CLASS]            = GEN12_COMPCTX_TLB_INV_CR,
>>> -    };
>>> -    static const i915_mcr_reg_t xehp_regs[] = {
>>> -        [RENDER_CLASS]            = XEHP_GFX_TLB_INV_CR,
>>> -        [VIDEO_DECODE_CLASS]        = XEHP_VD_TLB_INV_CR,
>>> -        [VIDEO_ENHANCEMENT_CLASS]    = XEHP_VE_TLB_INV_CR,
>>> -        [COPY_ENGINE_CLASS]        = XEHP_BLT_TLB_INV_CR,
>>> -        [COMPUTE_CLASS]            = XEHP_COMPCTX_TLB_INV_CR,
>>> -    };
>>>       struct drm_i915_private *i915 = gt->i915;
>>>       struct intel_uncore *uncore = gt->uncore;
>>>       struct intel_engine_cs *engine;
>>>       intel_engine_mask_t awake, tmp;
>>>       enum intel_engine_id id;
>>> -    const i915_reg_t *regs;
>>> -    unsigned int num = 0;
>>>       unsigned long flags;
>>> -    if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -        regs = NULL;
>>> -        num = ARRAY_SIZE(xehp_regs);
>>> -    } else if (GRAPHICS_VER(i915) == 12) {
>>> -        regs = gen12_regs;
>>> -        num = ARRAY_SIZE(gen12_regs);
>>> -    } else if (GRAPHICS_VER(i915) >= 8 && GRAPHICS_VER(i915) <= 11) {
>>> -        regs = gen8_regs;
>>> -        num = ARRAY_SIZE(gen8_regs);
>>> -    } else if (GRAPHICS_VER(i915) < 8) {
>>> -        return;
>>> -    }
>>> -
>>> -    if (drm_WARN_ONCE(&i915->drm, !num,
>>> -              "Platform does not implement TLB invalidation!"))
>>> +    if (GRAPHICS_VER(i915) < 8)
>>>           return;
>>>       intel_uncore_forcewake_get(uncore, FORCEWAKE_ALL);
>>> @@ -1094,33 +1034,22 @@ static void mmio_invalidate_full(struct 
>>> intel_gt *gt)
>>>       awake = 0;
>>>       for_each_engine(engine, gt, id) {
>>> -        struct reg_and_bit rb;
>>> -
>>>           if (!intel_engine_pm_is_awake(engine))
>>>               continue;
>>> -        if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -            u32 val = BIT(engine->instance);
>>> +        if (drm_WARN_ON_ONCE(&i915->drm,
>>> +                     !(engine->flags & 
>>> I915_ENGINE_HAS_TLB_INVALIDATION)))
>>> +            continue;
>>
>> Hmm, can this flag change dynamically? If not why not put it in 
>> initialization phase.
> 
> Could do. In that case I couldn't have any asserts that engine->tlb_inv 
> is valid but perhaps that is okay. Could just fail engine probe from 
> there if register table is incomplete. Sounds better indeed, I'll change 
> it.
> 
>>
>>> -            if (engine->class == VIDEO_DECODE_CLASS ||
>>> -                engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -                engine->class == COMPUTE_CLASS)
>>> -                val = _MASKED_BIT_ENABLE(val);
>>> +        if (engine->tlb_inv.mcr)
>>>               intel_gt_mcr_multicast_write_fw(gt,
>>> -                            xehp_regs[engine->class],
>>> -                            val);
>>> -        } else {
>>> -            rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -            if (!i915_mmio_reg_offset(rb.reg))
>>> -                continue;
>>> -
>>> -            if (GRAPHICS_VER(i915) == 12 && (engine->class == 
>>> VIDEO_DECODE_CLASS ||
>>> -                engine->class == VIDEO_ENHANCEMENT_CLASS ||
>>> -                engine->class == COMPUTE_CLASS))
>>> -                rb.bit = _MASKED_BIT_ENABLE(rb.bit);
>>> -
>>> -            intel_uncore_write_fw(uncore, rb.reg, rb.bit);
>>> -        }
>>> +                            engine->tlb_inv.reg.mcr_reg,
>>> +                            engine->tlb_inv.request);
>>> +        else
>>> +            intel_uncore_write_fw(uncore,
>>> +                          engine->tlb_inv.reg.reg,
>>> +                          engine->tlb_inv.request);
>>> +
>>>           awake |= engine->mask;
>>>       }
>>> @@ -1139,16 +1068,7 @@ static void mmio_invalidate_full(struct 
>>> intel_gt *gt)
>>>       intel_gt_mcr_unlock(gt, flags);
>>>       for_each_engine_masked(engine, gt, awake, tmp) {
>>> -        struct reg_and_bit rb;
>>> -
>>> -        if (GRAPHICS_VER_FULL(i915) >= IP_VER(12, 50)) {
>>> -            rb.mcr_reg = xehp_regs[engine->class];
>>> -            rb.bit = BIT(engine->instance);
>>> -        } else {
>>> -            rb = get_reg_and_bit(engine, regs == gen8_regs, regs, num);
>>> -        }
>>> -
>>> -        if (wait_for_invalidate(gt, rb))
>>> +        if (wait_for_invalidate(engine))
>>>               drm_err_ratelimited(&gt->i915->drm,
>>>                           "%s TLB invalidation did not complete in 
>>> %ums!\n",
>>>                           engine->name, TLB_INVAL_TIMEOUT_MS);
>>
>> With minor comments addressed:
>>
>> Reviewed-by: Andrzej Hajda <andrzej.hajda@intel.com>
> 
> I'll send v2 at some point, please stand by.
> 
> Regards,
> 
> Tvrtko