All of lore.kernel.org
 help / color / mirror / Atom feed
* [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time
@ 2022-07-08 12:24 Zbigniew Kempczyński
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection Zbigniew Kempczyński
                   ` (3 more replies)
  0 siblings, 4 replies; 8+ messages in thread
From: Zbigniew Kempczyński @ 2022-07-08 12:24 UTC (permalink / raw)
  To: igt-dev

Some tests can be affected by unresponsive driver when fault injection
would leave loaded driver in fault state. Series decreases number of
checkpoints (they are get randomly some during module reload) and
ensures driver is unloaded when test completes.

Fixes: https://gitlab.freedesktop.org/drm/intel/-/issues/6393

Chris Wilson (2):
  i915/i915_module_load: Randomise fault injection
  i915/i915_module_load: Raise a fatal-error on failing to unload a
    module

 tests/i915/i915_module_load.c | 54 +++++++++++++++++++++++++++++++----
 1 file changed, 48 insertions(+), 6 deletions(-)

-- 
2.34.1

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection
  2022-07-08 12:24 [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time Zbigniew Kempczyński
@ 2022-07-08 12:24 ` Zbigniew Kempczyński
  2022-07-12 13:57   ` Kamil Konieczny
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module Zbigniew Kempczyński
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 8+ messages in thread
From: Zbigniew Kempczyński @ 2022-07-08 12:24 UTC (permalink / raw)
  To: igt-dev; +Cc: Chris Wilson

From: Chris Wilson <chris.p.wilson@linux.intel.com>

Reduce the number of faults we inject during module reload to bring the
test runtime back to a reasonable level, and rely on multiple CI runs to
establish the full picture. This reduces the efficacy of testing (i.e.
premerge will probably not prevent inclusion of a buggy patch), but
error handling on module reload is not a critical factor for driver
health -- a broken patch that explodes in an error path is unlikely to
prevent CI from detecting critical issues, and with multiple runs we
should still be able to detect and fix issues before merging.

Fixes: https://gitlab.freedesktop.org/drm/intel/-/issues/6393

Signed-off-by: Chris Wilson <chris.p.wilson@linux.intel.com>
Signed-off-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
Cc: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
---
 tests/i915/i915_module_load.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/tests/i915/i915_module_load.c b/tests/i915/i915_module_load.c
index 4705f3d6c4..a922af0064 100644
--- a/tests/i915/i915_module_load.c
+++ b/tests/i915/i915_module_load.c
@@ -302,21 +302,28 @@ igt_main
 		     " with fault injection.");
 	igt_subtest("reload-with-fault-injection") {
 		const char *param;
-		int i = 0;
+		int i;
 
 		igt_i915_driver_unload();
 
+		i = 0;
+		param = getenv("IGT_SRANDOM");
+		if (param)
+			i = atoi(param);
+		if (!i)
+			i = time(NULL);
+		igt_info("Using IGT_SRANDOM=%d for randomised faults\n", i);
+		srandom(i);
+
 		param = "inject_probe_failure";
 		if (!igt_kmod_has_param("i915", param))
 			param = "inject_load_failure";
 		igt_require(igt_kmod_has_param("i915", param));
 
-		while (inject_fault("i915", param, ++i) == 0)
+		i = 0;
+		while (inject_fault("i915", param, i += 1 + random() % 17) == 0)
 			;
 
-		/* We expect to hit at least one fault! */
-		igt_assert(i > 1);
-
 		/* inject_fault() leaves the module unloaded */
 	}
 
-- 
2.34.1

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module
  2022-07-08 12:24 [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time Zbigniew Kempczyński
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection Zbigniew Kempczyński
@ 2022-07-08 12:24 ` Zbigniew Kempczyński
  2022-07-12 14:29   ` Kamil Konieczny
  2022-07-08 12:57 ` [igt-dev] ✓ Fi.CI.BAT: success for Decrease module reload fault injection time Patchwork
  2022-07-08 23:27 ` [igt-dev] ✓ Fi.CI.IGT: " Patchwork
  3 siblings, 1 reply; 8+ messages in thread
From: Zbigniew Kempczyński @ 2022-07-08 12:24 UTC (permalink / raw)
  To: igt-dev; +Cc: Jonathan Cavitt, Chris Wilson, Petri Latvala

From: Chris Wilson <chris.p.wilson@linux.intel.com>

If we fail to unload the module after the fault-injection test, we leave
behind a dangerous modparam that will affect all subsequent tests.
Better to declare the run void rather than have to triage all the bizarre
bugs.

Somes the fault injection module load test timeouts out, causing
igt_runner to cancel the test with a SIGQUIT. This unfortunately causes
us to abort in the middle of the test leaving the module in a broken
state. That broken module is then used for subsequent tests, casuing a
cascade of failures.

Signed-off-by: Chris Wilson <chris.p.wilson@linux.intel.com>
Signed-off-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
Cc: Petri Latvala <petri.latvala@intel.com>
Cc: Jonathan Cavitt <jonathan.cavitt@intel.com>
---
 tests/i915/i915_module_load.c | 37 ++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/tests/i915/i915_module_load.c b/tests/i915/i915_module_load.c
index a922af0064..170900b1ab 100644
--- a/tests/i915/i915_module_load.c
+++ b/tests/i915/i915_module_load.c
@@ -163,6 +163,33 @@ static int open_parameters(const char *module_name)
 	return open(path, O_RDONLY);
 }
 
+static void unload_or_die(const char *module_name)
+{
+	int err, loop;
+
+	/* should be unloaded, so expect a no-op */
+	for (loop = 0;; loop++) {
+		err = igt_kmod_unload(module_name, 0);
+		if (err && err == -ENOENT) /* -ENOENT == unloaded already */
+			err = 0;
+		if (!err || loop >= 10)
+			break;
+
+		sleep(1); /* wait for external clients to drop */
+		if (!strcmp(module_name, "i915"))
+			igt_i915_driver_unload();
+	}
+
+	igt_abort_on_f(err,
+		       "Failed to unload '%s' err:%d after %ds, leaving dangerous modparams intact!\n",
+		       module_name, err, loop);
+}
+
+static void must_unload(int sig)
+{
+	unload_or_die("i915");
+}
+
 static int
 inject_fault(const char *module_name, const char *opt, int fault)
 {
@@ -306,6 +333,14 @@ igt_main
 
 		igt_i915_driver_unload();
 
+		/*
+		 * inject_fault() leaves the module unloaded, but if that fails
+		 * we must abort the run. Otherwise, we leave a dangerous
+		 * modparam affecting all subsequent tests causing bizarre
+		 * failures.
+		 */
+		igt_install_exit_handler(must_unload);
+
 		i = 0;
 		param = getenv("IGT_SRANDOM");
 		if (param)
@@ -324,7 +359,7 @@ igt_main
 		while (inject_fault("i915", param, i += 1 + random() % 17) == 0)
 			;
 
-		/* inject_fault() leaves the module unloaded */
+		unload_or_die("i915");
 	}
 
 	/* Subtests should unload the module themselves if they use modparams */
-- 
2.34.1

^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [igt-dev] ✓ Fi.CI.BAT: success for Decrease module reload fault injection time
  2022-07-08 12:24 [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time Zbigniew Kempczyński
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection Zbigniew Kempczyński
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module Zbigniew Kempczyński
@ 2022-07-08 12:57 ` Patchwork
  2022-07-08 23:27 ` [igt-dev] ✓ Fi.CI.IGT: " Patchwork
  3 siblings, 0 replies; 8+ messages in thread
From: Patchwork @ 2022-07-08 12:57 UTC (permalink / raw)
  To: Zbigniew Kempczyński; +Cc: igt-dev

[-- Attachment #1: Type: text/plain, Size: 5490 bytes --]

== Series Details ==

Series: Decrease module reload fault injection time
URL   : https://patchwork.freedesktop.org/series/106108/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_11862 -> IGTPW_7487
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html

Participating hosts (45 -> 39)
------------------------------

  Additional (1): bat-dg2-9 
  Missing    (7): fi-cml-u2 fi-bxt-dsi fi-hsw-4200u bat-dg2-8 fi-icl-u2 bat-adlp-4 fi-ctg-p8600 

Known issues
------------

  Here are the changes found in IGTPW_7487 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@i915_module_load@load:
    - fi-kbl-soraka:      [PASS][1] -> [DMESG-WARN][2] ([i915#1982])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/fi-kbl-soraka/igt@i915_module_load@load.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-kbl-soraka/igt@i915_module_load@load.html

  * igt@i915_selftest@live@gem:
    - fi-blb-e6850:       NOTRUN -> [DMESG-FAIL][3] ([i915#4528])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-blb-e6850/igt@i915_selftest@live@gem.html

  * igt@kms_chamelium@common-hpd-after-suspend:
    - fi-snb-2600:        NOTRUN -> [SKIP][4] ([fdo#109271] / [fdo#111827])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-snb-2600/igt@kms_chamelium@common-hpd-after-suspend.html

  
#### Possible fixes ####

  * igt@i915_selftest@live@hangcheck:
    - fi-snb-2600:        [INCOMPLETE][5] ([i915#3921]) -> [PASS][6]
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/fi-snb-2600/igt@i915_selftest@live@hangcheck.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-snb-2600/igt@i915_selftest@live@hangcheck.html
    - bat-dg1-6:          [DMESG-FAIL][7] ([i915#4494] / [i915#4957]) -> [PASS][8]
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/bat-dg1-6/igt@i915_selftest@live@hangcheck.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/bat-dg1-6/igt@i915_selftest@live@hangcheck.html

  * igt@i915_selftest@live@requests:
    - fi-blb-e6850:       [DMESG-FAIL][9] ([i915#4528]) -> [PASS][10]
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/fi-blb-e6850/igt@i915_selftest@live@requests.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-blb-e6850/igt@i915_selftest@live@requests.html

  * igt@vgem_basic@setversion:
    - fi-kbl-soraka:      [INCOMPLETE][11] -> [PASS][12]
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/fi-kbl-soraka/igt@vgem_basic@setversion.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/fi-kbl-soraka/igt@vgem_basic@setversion.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
  [i915#1155]: https://gitlab.freedesktop.org/drm/intel/issues/1155
  [i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
  [i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
  [i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
  [i915#3576]: https://gitlab.freedesktop.org/drm/intel/issues/3576
  [i915#3595]: https://gitlab.freedesktop.org/drm/intel/issues/3595
  [i915#3708]: https://gitlab.freedesktop.org/drm/intel/issues/3708
  [i915#3921]: https://gitlab.freedesktop.org/drm/intel/issues/3921
  [i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
  [i915#4079]: https://gitlab.freedesktop.org/drm/intel/issues/4079
  [i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
  [i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
  [i915#4212]: https://gitlab.freedesktop.org/drm/intel/issues/4212
  [i915#4213]: https://gitlab.freedesktop.org/drm/intel/issues/4213
  [i915#4215]: https://gitlab.freedesktop.org/drm/intel/issues/4215
  [i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
  [i915#4494]: https://gitlab.freedesktop.org/drm/intel/issues/4494
  [i915#4528]: https://gitlab.freedesktop.org/drm/intel/issues/4528
  [i915#4579]: https://gitlab.freedesktop.org/drm/intel/issues/4579
  [i915#4873]: https://gitlab.freedesktop.org/drm/intel/issues/4873
  [i915#4957]: https://gitlab.freedesktop.org/drm/intel/issues/4957
  [i915#5122]: https://gitlab.freedesktop.org/drm/intel/issues/5122
  [i915#5174]: https://gitlab.freedesktop.org/drm/intel/issues/5174
  [i915#5190]: https://gitlab.freedesktop.org/drm/intel/issues/5190
  [i915#5274]: https://gitlab.freedesktop.org/drm/intel/issues/5274
  [i915#5763]: https://gitlab.freedesktop.org/drm/intel/issues/5763


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_6563 -> IGTPW_7487

  CI-20190529: 20190529
  CI_DRM_11862: ffee806d103b9604db7eb9cd689c098aca1ffa96 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_7487: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html
  IGT_6563: 7d43b49bf10788d4870668f93a800888fc8ab339 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html

[-- Attachment #2: Type: text/html, Size: 4766 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* [igt-dev] ✓ Fi.CI.IGT: success for Decrease module reload fault injection time
  2022-07-08 12:24 [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time Zbigniew Kempczyński
                   ` (2 preceding siblings ...)
  2022-07-08 12:57 ` [igt-dev] ✓ Fi.CI.BAT: success for Decrease module reload fault injection time Patchwork
@ 2022-07-08 23:27 ` Patchwork
  3 siblings, 0 replies; 8+ messages in thread
From: Patchwork @ 2022-07-08 23:27 UTC (permalink / raw)
  To: Zbigniew Kempczyński; +Cc: igt-dev

[-- Attachment #1: Type: text/plain, Size: 44048 bytes --]

== Series Details ==

Series: Decrease module reload fault injection time
URL   : https://patchwork.freedesktop.org/series/106108/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_11862_full -> IGTPW_7487_full
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html

Participating hosts (10 -> 10)
------------------------------

  Additional (3): shard-rkl shard-dg1 shard-tglu 
  Missing    (3): pig-skl-6260u pig-kbl-iris pig-glk-j5005 

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_7487_full:

### IGT changes ###

#### Suppressed ####

  The following results come from untrusted machines, tests, or statuses.
  They do not affect the overall result.

  * igt@i915_pm_rc6_residency@rc6-idle@vcs0:
    - {shard-rkl}:        NOTRUN -> [WARN][1]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-rkl-5/igt@i915_pm_rc6_residency@rc6-idle@vcs0.html

  * igt@kms_frontbuffer_tracking@fbc-suspend:
    - {shard-tglu}:       NOTRUN -> [DMESG-WARN][2]
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglu-4/igt@kms_frontbuffer_tracking@fbc-suspend.html

  * {igt@kms_invalid_mode@clock-too-high@edp-1-pipe-d}:
    - shard-tglb:         NOTRUN -> [SKIP][3] +3 similar issues
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb7/igt@kms_invalid_mode@clock-too-high@edp-1-pipe-d.html

  
New tests
---------

  New tests have been introduced between CI_DRM_11862_full and IGTPW_7487_full:

### New IGT tests (1) ###

  * igt@kms_legacy_colorkey@basic:
    - Statuses :
    - Exec time: [None] s

  

Known issues
------------

  Here are the changes found in IGTPW_7487_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_create@create-massive:
    - shard-apl:          NOTRUN -> [DMESG-WARN][4] ([i915#4991])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl4/igt@gem_create@create-massive.html

  * igt@gem_ctx_isolation@preservation-s3@vcs0:
    - shard-kbl:          [PASS][5] -> [DMESG-WARN][6] ([i915#180]) +7 similar issues
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@gem_ctx_isolation@preservation-s3@vcs0.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@gem_ctx_isolation@preservation-s3@vcs0.html

  * igt@gem_ctx_persistence@engines-hostile-preempt:
    - shard-snb:          NOTRUN -> [SKIP][7] ([fdo#109271] / [i915#1099]) +2 similar issues
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb7/igt@gem_ctx_persistence@engines-hostile-preempt.html

  * igt@gem_ctx_persistence@hostile:
    - shard-tglb:         NOTRUN -> [FAIL][8] ([i915#2410])
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb1/igt@gem_ctx_persistence@hostile.html

  * igt@gem_exec_balancer@parallel-contexts:
    - shard-iclb:         [PASS][9] -> [SKIP][10] ([i915#4525])
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb4/igt@gem_exec_balancer@parallel-contexts.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb8/igt@gem_exec_balancer@parallel-contexts.html

  * igt@gem_exec_endless@dispatch@bcs0:
    - shard-tglb:         [PASS][11] -> [INCOMPLETE][12] ([i915#3778])
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb3/igt@gem_exec_endless@dispatch@bcs0.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb7/igt@gem_exec_endless@dispatch@bcs0.html

  * igt@gem_exec_fair@basic-flow@rcs0:
    - shard-tglb:         [PASS][13] -> [FAIL][14] ([i915#2842])
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb3/igt@gem_exec_fair@basic-flow@rcs0.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb5/igt@gem_exec_fair@basic-flow@rcs0.html

  * igt@gem_exec_fair@basic-none-solo@rcs0:
    - shard-apl:          NOTRUN -> [FAIL][15] ([i915#2842])
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl8/igt@gem_exec_fair@basic-none-solo@rcs0.html
    - shard-tglb:         NOTRUN -> [FAIL][16] ([i915#2842])
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb2/igt@gem_exec_fair@basic-none-solo@rcs0.html
    - shard-glk:          NOTRUN -> [FAIL][17] ([i915#2842])
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk8/igt@gem_exec_fair@basic-none-solo@rcs0.html
    - shard-iclb:         NOTRUN -> [FAIL][18] ([i915#2842])
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb4/igt@gem_exec_fair@basic-none-solo@rcs0.html
    - shard-kbl:          NOTRUN -> [FAIL][19] ([i915#2842])
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@gem_exec_fair@basic-none-solo@rcs0.html

  * igt@gem_exec_fair@basic-pace@bcs0:
    - shard-iclb:         [PASS][20] -> [FAIL][21] ([i915#2842])
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb1/igt@gem_exec_fair@basic-pace@bcs0.html
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb1/igt@gem_exec_fair@basic-pace@bcs0.html

  * igt@gem_exec_fair@basic-pace@vcs1:
    - shard-kbl:          [PASS][22] -> [FAIL][23] ([i915#2842])
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@gem_exec_fair@basic-pace@vcs1.html
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@gem_exec_fair@basic-pace@vcs1.html

  * igt@gem_huc_copy@huc-copy:
    - shard-tglb:         [PASS][24] -> [SKIP][25] ([i915#2190])
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb8/igt@gem_huc_copy@huc-copy.html
   [25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb7/igt@gem_huc_copy@huc-copy.html

  * igt@gem_lmem_swapping@verify-random:
    - shard-kbl:          NOTRUN -> [SKIP][26] ([fdo#109271] / [i915#4613]) +2 similar issues
   [26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@gem_lmem_swapping@verify-random.html

  * igt@gem_pxp@create-regular-buffer:
    - shard-iclb:         NOTRUN -> [SKIP][27] ([i915#4270])
   [27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb4/igt@gem_pxp@create-regular-buffer.html
    - shard-tglb:         NOTRUN -> [SKIP][28] ([i915#4270]) +1 similar issue
   [28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb2/igt@gem_pxp@create-regular-buffer.html

  * igt@gem_softpin@evict-single-offset:
    - shard-kbl:          NOTRUN -> [FAIL][29] ([i915#4171])
   [29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@gem_softpin@evict-single-offset.html

  * igt@gem_softpin@evict-snoop-interruptible:
    - shard-tglb:         NOTRUN -> [SKIP][30] ([fdo#109312])
   [30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb5/igt@gem_softpin@evict-snoop-interruptible.html

  * igt@gem_vm_create@invalid-create:
    - shard-snb:          NOTRUN -> [SKIP][31] ([fdo#109271]) +51 similar issues
   [31]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb2/igt@gem_vm_create@invalid-create.html

  * igt@i915_pm_rc6_residency@rc6-idle@rcs0:
    - shard-tglb:         NOTRUN -> [WARN][32] ([i915#2681]) +3 similar issues
   [32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb1/igt@i915_pm_rc6_residency@rc6-idle@rcs0.html

  * igt@i915_suspend@debugfs-reader:
    - shard-kbl:          NOTRUN -> [DMESG-WARN][33] ([i915#180]) +4 similar issues
   [33]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@i915_suspend@debugfs-reader.html

  * igt@i915_suspend@fence-restore-tiled2untiled:
    - shard-apl:          [PASS][34] -> [DMESG-WARN][35] ([i915#180]) +2 similar issues
   [34]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-apl6/igt@i915_suspend@fence-restore-tiled2untiled.html
   [35]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl8/igt@i915_suspend@fence-restore-tiled2untiled.html

  * igt@kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0:
    - shard-tglb:         NOTRUN -> [SKIP][36] ([fdo#111615])
   [36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb3/igt@kms_big_fb@yf-tiled-max-hw-stride-32bpp-rotate-0.html

  * igt@kms_ccs@pipe-a-bad-pixel-format-y_tiled_gen12_rc_ccs_cc:
    - shard-iclb:         NOTRUN -> [SKIP][37] ([fdo#109278] / [i915#3886])
   [37]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb1/igt@kms_ccs@pipe-a-bad-pixel-format-y_tiled_gen12_rc_ccs_cc.html
    - shard-apl:          NOTRUN -> [SKIP][38] ([fdo#109271] / [i915#3886])
   [38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl7/igt@kms_ccs@pipe-a-bad-pixel-format-y_tiled_gen12_rc_ccs_cc.html
    - shard-glk:          NOTRUN -> [SKIP][39] ([fdo#109271] / [i915#3886])
   [39]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk9/igt@kms_ccs@pipe-a-bad-pixel-format-y_tiled_gen12_rc_ccs_cc.html

  * igt@kms_ccs@pipe-a-ccs-on-another-bo-y_tiled_gen12_mc_ccs:
    - shard-tglb:         NOTRUN -> [SKIP][40] ([i915#3689] / [i915#3886])
   [40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_ccs@pipe-a-ccs-on-another-bo-y_tiled_gen12_mc_ccs.html

  * igt@kms_ccs@pipe-b-bad-pixel-format-y_tiled_ccs:
    - shard-tglb:         NOTRUN -> [SKIP][41] ([i915#3689]) +2 similar issues
   [41]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb2/igt@kms_ccs@pipe-b-bad-pixel-format-y_tiled_ccs.html

  * igt@kms_ccs@pipe-b-bad-pixel-format-y_tiled_gen12_rc_ccs_cc:
    - shard-kbl:          NOTRUN -> [SKIP][42] ([fdo#109271] / [i915#3886]) +6 similar issues
   [42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@kms_ccs@pipe-b-bad-pixel-format-y_tiled_gen12_rc_ccs_cc.html

  * igt@kms_ccs@pipe-c-bad-aux-stride-y_tiled_ccs:
    - shard-kbl:          NOTRUN -> [SKIP][43] ([fdo#109271]) +173 similar issues
   [43]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@kms_ccs@pipe-c-bad-aux-stride-y_tiled_ccs.html

  * igt@kms_ccs@pipe-c-bad-pixel-format-4_tiled_dg2_rc_ccs:
    - shard-tglb:         NOTRUN -> [SKIP][44] ([i915#6095]) +2 similar issues
   [44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_ccs@pipe-c-bad-pixel-format-4_tiled_dg2_rc_ccs.html

  * igt@kms_ccs@pipe-c-bad-pixel-format-yf_tiled_ccs:
    - shard-apl:          NOTRUN -> [SKIP][45] ([fdo#109271]) +15 similar issues
   [45]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl3/igt@kms_ccs@pipe-c-bad-pixel-format-yf_tiled_ccs.html
    - shard-tglb:         NOTRUN -> [SKIP][46] ([fdo#111615] / [i915#3689]) +1 similar issue
   [46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb3/igt@kms_ccs@pipe-c-bad-pixel-format-yf_tiled_ccs.html

  * igt@kms_ccs@pipe-d-bad-pixel-format-4_tiled_dg2_rc_ccs_cc:
    - shard-iclb:         NOTRUN -> [SKIP][47] ([fdo#109278]) +2 similar issues
   [47]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb3/igt@kms_ccs@pipe-d-bad-pixel-format-4_tiled_dg2_rc_ccs_cc.html

  * igt@kms_chamelium@dp-hpd-fast:
    - shard-glk:          NOTRUN -> [SKIP][48] ([fdo#109271] / [fdo#111827]) +3 similar issues
   [48]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk5/igt@kms_chamelium@dp-hpd-fast.html

  * igt@kms_chamelium@hdmi-audio:
    - shard-kbl:          NOTRUN -> [SKIP][49] ([fdo#109271] / [fdo#111827]) +12 similar issues
   [49]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@kms_chamelium@hdmi-audio.html

  * igt@kms_color_chamelium@pipe-a-ctm-0-25:
    - shard-snb:          NOTRUN -> [SKIP][50] ([fdo#109271] / [fdo#111827]) +1 similar issue
   [50]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb5/igt@kms_color_chamelium@pipe-a-ctm-0-25.html
    - shard-apl:          NOTRUN -> [SKIP][51] ([fdo#109271] / [fdo#111827])
   [51]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl3/igt@kms_color_chamelium@pipe-a-ctm-0-25.html
    - shard-tglb:         NOTRUN -> [SKIP][52] ([fdo#109284] / [fdo#111827])
   [52]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb2/igt@kms_color_chamelium@pipe-a-ctm-0-25.html
    - shard-iclb:         NOTRUN -> [SKIP][53] ([fdo#109284] / [fdo#111827])
   [53]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb5/igt@kms_color_chamelium@pipe-a-ctm-0-25.html

  * igt@kms_content_protection@atomic-dpms:
    - shard-kbl:          NOTRUN -> [TIMEOUT][54] ([i915#1319])
   [54]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@kms_content_protection@atomic-dpms.html

  * igt@kms_content_protection@lic:
    - shard-tglb:         NOTRUN -> [SKIP][55] ([i915#1063])
   [55]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_content_protection@lic.html

  * igt@kms_cursor_legacy@2x-flip-vs-cursor-legacy:
    - shard-tglb:         NOTRUN -> [SKIP][56] ([fdo#109274] / [fdo#111825])
   [56]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb2/igt@kms_cursor_legacy@2x-flip-vs-cursor-legacy.html

  * igt@kms_fbcon_fbt@fbc-suspend:
    - shard-kbl:          NOTRUN -> [INCOMPLETE][57] ([i915#180] / [i915#4939])
   [57]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@kms_fbcon_fbt@fbc-suspend.html

  * igt@kms_flip@2x-modeset-vs-vblank-race:
    - shard-tglb:         NOTRUN -> [SKIP][58] ([fdo#109274] / [fdo#111825] / [i915#3637])
   [58]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb1/igt@kms_flip@2x-modeset-vs-vblank-race.html

  * igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling@pipe-a-valid-mode:
    - shard-iclb:         NOTRUN -> [SKIP][59] ([i915#2672]) +6 similar issues
   [59]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb7/igt@kms_flip_scaled_crc@flip-32bpp-yftileccs-to-64bpp-yftile-upscaling@pipe-a-valid-mode.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-downscaling@pipe-a-valid-mode:
    - shard-tglb:         NOTRUN -> [SKIP][60] ([i915#2672])
   [60]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytileccs-downscaling@pipe-a-valid-mode.html

  * igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-downscaling@pipe-a-default-mode:
    - shard-iclb:         NOTRUN -> [SKIP][61] ([i915#2672] / [i915#3555])
   [61]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@kms_flip_scaled_crc@flip-32bpp-ytile-to-32bpp-ytilegen12rcccs-downscaling@pipe-a-default-mode.html

  * igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling@pipe-a-default-mode:
    - shard-iclb:         NOTRUN -> [SKIP][62] ([i915#3555])
   [62]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@kms_flip_scaled_crc@flip-64bpp-ytile-to-32bpp-ytile-downscaling@pipe-a-default-mode.html

  * igt@kms_frontbuffer_tracking@fbc-tiling-4:
    - shard-tglb:         NOTRUN -> [SKIP][63] ([i915#5439])
   [63]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb5/igt@kms_frontbuffer_tracking@fbc-tiling-4.html

  * igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-pwrite:
    - shard-iclb:         NOTRUN -> [SKIP][64] ([fdo#109280])
   [64]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb1/igt@kms_frontbuffer_tracking@fbcpsr-2p-primscrn-pri-shrfb-draw-pwrite.html

  * igt@kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-blt:
    - shard-tglb:         NOTRUN -> [SKIP][65] ([fdo#109280] / [fdo#111825]) +3 similar issues
   [65]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_frontbuffer_tracking@fbcpsr-2p-scndscrn-cur-indfb-draw-blt.html

  * igt@kms_pipe_b_c_ivb@enable-pipe-c-while-b-has-3-lanes:
    - shard-tglb:         NOTRUN -> [SKIP][66] ([fdo#109289]) +2 similar issues
   [66]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_pipe_b_c_ivb@enable-pipe-c-while-b-has-3-lanes.html

  * igt@kms_psr2_sf@overlay-plane-update-continuous-sf:
    - shard-kbl:          NOTRUN -> [SKIP][67] ([fdo#109271] / [i915#658])
   [67]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@kms_psr2_sf@overlay-plane-update-continuous-sf.html

  * igt@kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb:
    - shard-apl:          NOTRUN -> [SKIP][68] ([fdo#109271] / [i915#658])
   [68]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl4/igt@kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb.html
    - shard-tglb:         NOTRUN -> [SKIP][69] ([i915#2920])
   [69]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb.html
    - shard-glk:          NOTRUN -> [SKIP][70] ([fdo#109271] / [i915#658]) +1 similar issue
   [70]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk7/igt@kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb.html
    - shard-iclb:         NOTRUN -> [SKIP][71] ([i915#658])
   [71]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb5/igt@kms_psr2_sf@primary-plane-update-sf-dmg-area-big-fb.html

  * igt@kms_psr@psr2_no_drrs:
    - shard-iclb:         [PASS][72] -> [SKIP][73] ([fdo#109441]) +1 similar issue
   [72]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb2/igt@kms_psr@psr2_no_drrs.html
   [73]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb1/igt@kms_psr@psr2_no_drrs.html

  * igt@kms_vblank@pipe-b-ts-continuation-suspend:
    - shard-snb:          [PASS][74] -> [SKIP][75] ([fdo#109271])
   [74]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-snb4/igt@kms_vblank@pipe-b-ts-continuation-suspend.html
   [75]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb7/igt@kms_vblank@pipe-b-ts-continuation-suspend.html

  * igt@kms_vblank@pipe-d-wait-busy-hang:
    - shard-glk:          NOTRUN -> [SKIP][76] ([fdo#109271]) +28 similar issues
   [76]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk5/igt@kms_vblank@pipe-d-wait-busy-hang.html

  * igt@kms_writeback@writeback-fb-id:
    - shard-kbl:          NOTRUN -> [SKIP][77] ([fdo#109271] / [i915#2437])
   [77]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@kms_writeback@writeback-fb-id.html

  * igt@perf@unprivileged-single-ctx-counters:
    - shard-iclb:         NOTRUN -> [SKIP][78] ([fdo#109289])
   [78]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb6/igt@perf@unprivileged-single-ctx-counters.html

  * igt@perf_pmu@rc6-suspend:
    - shard-kbl:          NOTRUN -> [INCOMPLETE][79] ([i915#4939] / [i915#794])
   [79]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@perf_pmu@rc6-suspend.html

  * igt@prime_nv_api@i915_nv_import_twice:
    - shard-tglb:         NOTRUN -> [SKIP][80] ([fdo#109291]) +1 similar issue
   [80]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb1/igt@prime_nv_api@i915_nv_import_twice.html

  * igt@prime_nv_api@i915_nv_import_vs_close:
    - shard-iclb:         NOTRUN -> [SKIP][81] ([fdo#109291])
   [81]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb3/igt@prime_nv_api@i915_nv_import_vs_close.html

  * igt@sysfs_clients@sema-25:
    - shard-kbl:          NOTRUN -> [SKIP][82] ([fdo#109271] / [i915#2994])
   [82]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@sysfs_clients@sema-25.html

  
#### Possible fixes ####

  * igt@gem_busy@close-race:
    - shard-snb:          [TIMEOUT][83] ([i915#5748]) -> [PASS][84]
   [83]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-snb6/igt@gem_busy@close-race.html
   [84]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb6/igt@gem_busy@close-race.html

  * igt@gem_exec_balancer@parallel-keep-in-fence:
    - shard-iclb:         [SKIP][85] ([i915#4525]) -> [PASS][86] +1 similar issue
   [85]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb8/igt@gem_exec_balancer@parallel-keep-in-fence.html
   [86]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@gem_exec_balancer@parallel-keep-in-fence.html

  * igt@gem_exec_fair@basic-throttle@rcs0:
    - shard-iclb:         [FAIL][87] ([i915#2849]) -> [PASS][88]
   [87]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb6/igt@gem_exec_fair@basic-throttle@rcs0.html
   [88]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb3/igt@gem_exec_fair@basic-throttle@rcs0.html

  * igt@gem_exec_whisper@basic-contexts-forked-all:
    - shard-glk:          [DMESG-WARN][89] ([i915#118]) -> [PASS][90]
   [89]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-glk6/igt@gem_exec_whisper@basic-contexts-forked-all.html
   [90]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk3/igt@gem_exec_whisper@basic-contexts-forked-all.html

  * igt@i915_module_load@reload-with-fault-injection:
    - shard-tglb:         [TIMEOUT][91] ([i915#3953]) -> [PASS][92]
   [91]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb8/igt@i915_module_load@reload-with-fault-injection.html
   [92]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb8/igt@i915_module_load@reload-with-fault-injection.html

  * igt@i915_selftest@live@hangcheck:
    - shard-snb:          [INCOMPLETE][93] ([i915#3921]) -> [PASS][94]
   [93]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-snb6/igt@i915_selftest@live@hangcheck.html
   [94]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-snb7/igt@i915_selftest@live@hangcheck.html
    - shard-tglb:         [DMESG-WARN][95] ([i915#5591]) -> [PASS][96]
   [95]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb3/igt@i915_selftest@live@hangcheck.html
   [96]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb1/igt@i915_selftest@live@hangcheck.html

  * igt@i915_suspend@fence-restore-untiled:
    - shard-apl:          [DMESG-WARN][97] ([i915#180]) -> [PASS][98]
   [97]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-apl4/igt@i915_suspend@fence-restore-untiled.html
   [98]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-apl2/igt@i915_suspend@fence-restore-untiled.html

  * igt@kms_flip@plain-flip-ts-check-interruptible@c-dp1:
    - shard-kbl:          [FAIL][99] ([i915#2122]) -> [PASS][100]
   [99]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl6/igt@kms_flip@plain-flip-ts-check-interruptible@c-dp1.html
   [100]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@kms_flip@plain-flip-ts-check-interruptible@c-dp1.html

  * igt@kms_plane@plane-panning-bottom-right-suspend@pipe-a-planes:
    - shard-kbl:          [DMESG-WARN][101] ([i915#180]) -> [PASS][102] +1 similar issue
   [101]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl1/igt@kms_plane@plane-panning-bottom-right-suspend@pipe-a-planes.html
   [102]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@kms_plane@plane-panning-bottom-right-suspend@pipe-a-planes.html

  * igt@kms_plane@plane-position-covered@pipe-b-planes:
    - shard-tglb:         [INCOMPLETE][103] -> [PASS][104]
   [103]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb8/igt@kms_plane@plane-position-covered@pipe-b-planes.html
   [104]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb3/igt@kms_plane@plane-position-covered@pipe-b-planes.html

  * igt@kms_plane_lowres@tiling-y@pipe-a-hdmi-a-1:
    - shard-glk:          [DMESG-FAIL][105] ([i915#118] / [i915#1888]) -> [PASS][106]
   [105]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-glk7/igt@kms_plane_lowres@tiling-y@pipe-a-hdmi-a-1.html
   [106]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk9/igt@kms_plane_lowres@tiling-y@pipe-a-hdmi-a-1.html

  * igt@kms_plane_lowres@tiling-y@pipe-c-hdmi-a-2:
    - shard-glk:          [FAIL][107] ([i915#1888]) -> [PASS][108]
   [107]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-glk7/igt@kms_plane_lowres@tiling-y@pipe-c-hdmi-a-2.html
   [108]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-glk9/igt@kms_plane_lowres@tiling-y@pipe-c-hdmi-a-2.html

  * igt@kms_psr2_su@frontbuffer-xrgb8888:
    - shard-iclb:         [SKIP][109] ([fdo#109642] / [fdo#111068] / [i915#658]) -> [PASS][110]
   [109]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb3/igt@kms_psr2_su@frontbuffer-xrgb8888.html
   [110]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@kms_psr2_su@frontbuffer-xrgb8888.html

  * igt@kms_psr@psr2_sprite_render:
    - shard-iclb:         [SKIP][111] ([fdo#109441]) -> [PASS][112] +1 similar issue
   [111]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb4/igt@kms_psr@psr2_sprite_render.html
   [112]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@kms_psr@psr2_sprite_render.html

  * igt@kms_psr_stress_test@flip-primary-invalidate-overlay:
    - shard-tglb:         [SKIP][113] ([i915#5519]) -> [PASS][114]
   [113]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-tglb5/igt@kms_psr_stress_test@flip-primary-invalidate-overlay.html
   [114]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-tglb5/igt@kms_psr_stress_test@flip-primary-invalidate-overlay.html

  
#### Warnings ####

  * igt@gem_exec_balancer@parallel-ordering:
    - shard-iclb:         [SKIP][115] ([i915#4525]) -> [FAIL][116] ([i915#6117])
   [115]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb6/igt@gem_exec_balancer@parallel-ordering.html
   [116]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb4/igt@gem_exec_balancer@parallel-ordering.html

  * igt@i915_pm_rc6_residency@rc6-idle@vcs0:
    - shard-iclb:         [FAIL][117] ([i915#2684]) -> [WARN][118] ([i915#2684])
   [117]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb3/igt@i915_pm_rc6_residency@rc6-idle@vcs0.html
   [118]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb3/igt@i915_pm_rc6_residency@rc6-idle@vcs0.html

  * igt@kms_psr2_sf@cursor-plane-move-continuous-sf:
    - shard-iclb:         [SKIP][119] ([i915#2920]) -> [SKIP][120] ([i915#658])
   [119]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb2/igt@kms_psr2_sf@cursor-plane-move-continuous-sf.html
   [120]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb5/igt@kms_psr2_sf@cursor-plane-move-continuous-sf.html

  * igt@kms_psr2_sf@overlay-plane-move-continuous-exceed-sf:
    - shard-iclb:         [SKIP][121] ([i915#658]) -> [SKIP][122] ([i915#2920]) +1 similar issue
   [121]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-iclb6/igt@kms_psr2_sf@overlay-plane-move-continuous-exceed-sf.html
   [122]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-iclb2/igt@kms_psr2_sf@overlay-plane-move-continuous-exceed-sf.html

  * igt@runner@aborted:
    - shard-kbl:          ([FAIL][123], [FAIL][124], [FAIL][125], [FAIL][126], [FAIL][127], [FAIL][128], [FAIL][129], [FAIL][130], [FAIL][131]) ([i915#180] / [i915#3002] / [i915#4312] / [i915#5257]) -> ([FAIL][132], [FAIL][133], [FAIL][134], [FAIL][135], [FAIL][136], [FAIL][137], [FAIL][138], [FAIL][139], [FAIL][140], [FAIL][141], [FAIL][142], [FAIL][143], [FAIL][144], [FAIL][145], [FAIL][146]) ([i915#180] / [i915#4312] / [i915#5257] / [i915#92])
   [123]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@runner@aborted.html
   [124]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl4/igt@runner@aborted.html
   [125]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@runner@aborted.html
   [126]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl1/igt@runner@aborted.html
   [127]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@runner@aborted.html
   [128]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl4/igt@runner@aborted.html
   [129]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl1/igt@runner@aborted.html
   [130]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl7/igt@runner@aborted.html
   [131]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11862/shard-kbl1/igt@runner@aborted.html
   [132]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@runner@aborted.html
   [133]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@runner@aborted.html
   [134]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@runner@aborted.html
   [135]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl1/igt@runner@aborted.html
   [136]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@runner@aborted.html
   [137]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@runner@aborted.html
   [138]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@runner@aborted.html
   [139]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@runner@aborted.html
   [140]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@runner@aborted.html
   [141]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@runner@aborted.html
   [142]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl7/igt@runner@aborted.html
   [143]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@runner@aborted.html
   [144]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@runner@aborted.html
   [145]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl6/igt@runner@aborted.html
   [146]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/shard-kbl4/igt@runner@aborted.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#103375]: https://bugs.freedesktop.org/show_bug.cgi?id=103375
  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [fdo#109274]: https://bugs.freedesktop.org/show_bug.cgi?id=109274
  [fdo#109278]: https://bugs.freedesktop.org/show_bug.cgi?id=109278
  [fdo#109280]: https://bugs.freedesktop.org/show_bug.cgi?id=109280
  [fdo#109283]: https://bugs.freedesktop.org/show_bug.cgi?id=109283
  [fdo#109284]: https://bugs.freedesktop.org/show_bug.cgi?id=109284
  [fdo#109289]: https://bugs.freedesktop.org/show_bug.cgi?id=109289
  [fdo#109291]: https://bugs.freedesktop.org/show_bug.cgi?id=109291
  [fdo#109295]: https://bugs.freedesktop.org/show_bug.cgi?id=109295
  [fdo#109300]: https://bugs.freedesktop.org/show_bug.cgi?id=109300
  [fdo#109307]: https://bugs.freedesktop.org/show_bug.cgi?id=109307
  [fdo#109308]: https://bugs.freedesktop.org/show_bug.cgi?id=109308
  [fdo#109309]: https://bugs.freedesktop.org/show_bug.cgi?id=109309
  [fdo#109312]: https://bugs.freedesktop.org/show_bug.cgi?id=109312
  [fdo#109313]: https://bugs.freedesktop.org/show_bug.cgi?id=109313
  [fdo#109314]: https://bugs.freedesktop.org/show_bug.cgi?id=109314
  [fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
  [fdo#109506]: https://bugs.freedesktop.org/show_bug.cgi?id=109506
  [fdo#109642]: https://bugs.freedesktop.org/show_bug.cgi?id=109642
  [fdo#110189]: https://bugs.freedesktop.org/show_bug.cgi?id=110189
  [fdo#110542]: https://bugs.freedesktop.org/show_bug.cgi?id=110542
  [fdo#110723]: https://bugs.freedesktop.org/show_bug.cgi?id=110723
  [fdo#111068]: https://bugs.freedesktop.org/show_bug.cgi?id=111068
  [fdo#111314]: https://bugs.freedesktop.org/show_bug.cgi?id=111314
  [fdo#111614]: https://bugs.freedesktop.org/show_bug.cgi?id=111614
  [fdo#111615]: https://bugs.freedesktop.org/show_bug.cgi?id=111615
  [fdo#111644]: https://bugs.freedesktop.org/show_bug.cgi?id=111644
  [fdo#111656]: https://bugs.freedesktop.org/show_bug.cgi?id=111656
  [fdo#111825]: https://bugs.freedesktop.org/show_bug.cgi?id=111825
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [fdo#112283]: https://bugs.freedesktop.org/show_bug.cgi?id=112283
  [i915#1063]: https://gitlab.freedesktop.org/drm/intel/issues/1063
  [i915#1072]: https://gitlab.freedesktop.org/drm/intel/issues/1072
  [i915#1099]: https://gitlab.freedesktop.org/drm/intel/issues/1099
  [i915#1155]: https://gitlab.freedesktop.org/drm/intel/issues/1155
  [i915#118]: https://gitlab.freedesktop.org/drm/intel/issues/118
  [i915#1257]: https://gitlab.freedesktop.org/drm/intel/issues/1257
  [i915#1319]: https://gitlab.freedesktop.org/drm/intel/issues/1319
  [i915#132]: https://gitlab.freedesktop.org/drm/intel/issues/132
  [i915#1397]: https://gitlab.freedesktop.org/drm/intel/issues/1397
  [i915#1722]: https://gitlab.freedesktop.org/drm/intel/issues/1722
  [i915#1755]: https://gitlab.freedesktop.org/drm/intel/issues/1755
  [i915#1769]: https://gitlab.freedesktop.org/drm/intel/issues/1769
  [i915#180]: https://gitlab.freedesktop.org/drm/intel/issues/180
  [i915#1825]: https://gitlab.freedesktop.org/drm/intel/issues/1825
  [i915#1836]: https://gitlab.freedesktop.org/drm/intel/issues/1836
  [i915#1839]: https://gitlab.freedesktop.org/drm/intel/issues/1839
  [i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
  [i915#1849]: https://gitlab.freedesktop.org/drm/intel/issues/1849
  [i915#1850]: https://gitlab.freedesktop.org/drm/intel/issues/1850
  [i915#1888]: https://gitlab.freedesktop.org/drm/intel/issues/1888
  [i915#1902]: https://gitlab.freedesktop.org/drm/intel/issues/1902
  [i915#1911]: https://gitlab.freedesktop.org/drm/intel/issues/1911
  [i915#2029]: https://gitlab.freedesktop.org/drm/intel/issues/2029
  [i915#2122]: https://gitlab.freedesktop.org/drm/intel/issues/2122
  [i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
  [i915#2410]: https://gitlab.freedesktop.org/drm/intel/issues/2410
  [i915#2433]: https://gitlab.freedesktop.org/drm/intel/issues/2433
  [i915#2434]: https://gitlab.freedesktop.org/drm/intel/issues/2434
  [i915#2436]: https://gitlab.freedesktop.org/drm/intel/issues/2436
  [i915#2437]: https://gitlab.freedesktop.org/drm/intel/issues/2437
  [i915#2527]: https://gitlab.freedesktop.org/drm/intel/issues/2527
  [i915#2530]: https://gitlab.freedesktop.org/drm/intel/issues/2530
  [i915#2582]: https://gitlab.freedesktop.org/drm/intel/issues/2582
  [i915#2658]: https://gitlab.freedesktop.org/drm/intel/issues/2658
  [i915#2672]: https://gitlab.freedesktop.org/drm/intel/issues/2672
  [i915#2681]: https://gitlab.freedesktop.org/drm/intel/issues/2681
  [i915#2684]: https://gitlab.freedesktop.org/drm/intel/issues/2684
  [i915#2705]: https://gitlab.freedesktop.org/drm/intel/issues/2705
  [i915#280]: https://gitlab.freedesktop.org/drm/intel/issues/280
  [i915#284]: https://gitlab.freedesktop.org/drm/intel/issues/284
  [i915#2842]: https://gitlab.freedesktop.org/drm/intel/issues/2842
  [i915#2849]: https://gitlab.freedesktop.org/drm/intel/issues/2849
  [i915#2856]: https://gitlab.freedesktop.org/drm/intel/issues/2856
  [i915#2920]: https://gitlab.freedesktop.org/drm/intel/issues/2920
  [i915#2994]: https://gitlab.freedesktop.org/drm/intel/issues/2994
  [i915#3002]: https://gitlab.freedesktop.org/drm/intel/issues/3002
  [i915#3012]: https://gitlab.freedesktop.org/drm/intel/issues/3012
  [i915#3116]: https://gitlab.freedesktop.org/drm/intel/issues/3116
  [i915#3281]: https://gitlab.freedesktop.org/drm/intel/issues/3281
  [i915#3282]: https://gitlab.freedesktop.org/drm/intel/issues/3282
  [i915#3291]: https://gitlab.freedesktop.org/drm/intel/issues/3291
  [i915#3297]: https://gitlab.freedesktop.org/drm/intel/issues/3297
  [i915#3299]: https://gitlab.freedesktop.org/drm/intel/issues/3299
  [i915#3301]: https://gitlab.freedesktop.org/drm/intel/issues/3301
  [i915#3318]: https://gitlab.freedesktop.org/drm/intel/issues/3318
  [i915#3359]: https://gitlab.freedesktop.org/drm/intel/issues/3359
  [i915#3361]: https://gitlab.freedesktop.org/drm/intel/issues/3361
  [i915#3458]: https://gitlab.freedesktop.org/drm/intel/issues/3458
  [i915#3469]: https://gitlab.freedesktop.org/drm/intel/issues/3469
  [i915#3528]: https://gitlab.freedesktop.org/drm/intel/issues/3528
  [i915#3536]: https://gitlab.freedesktop.org/drm/intel/issues/3536
  [i915#3539]: https://gitlab.freedesktop.org/drm/intel/issues/3539
  [i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
  [i915#3558]: https://gitlab.freedesktop.org/drm/intel/issues/3558
  [i915#3591]: https://gitlab.freedesktop.org/drm/intel/issues/3591
  [i915#3637]: https://gitlab.freedesktop.org/drm/intel/issues/3637
  [i915#3638]: https://gitlab.freedesktop.org/drm/intel/issues/3638
  [i915#3689]: https://gitlab.freedesktop.org/drm/intel/issues/3689
  [i915#3708]: https://gitlab.freedesktop.org/drm/intel/issues/3708
  [i915#3734]: https://gitlab.freedesktop.org/drm/intel/issues/3734
  [i915#3742]: https://gitlab.freedesktop.org/drm/intel/issues/3742
  [i915#3778]: https://gitlab.freedesktop.org/drm/intel/issues/3778
  [i915#3804]: https://gitlab.freedesktop.org/drm/intel/issues/3804
  [i915#3826]: https://gitlab.freedesktop.org/drm/intel/issues/3826
  [i915#3828]: https://gitlab.freedesktop.org/drm/intel/issues/3828
  [i915#3886]: https://gitlab.freedesktop.org/drm/intel/issues/3886
  [i915#3921]: https://gitlab.freedesktop.org/drm/intel/issues/3921
  [i915#3936]: https://gitlab.freedesktop.org/drm/intel/issues/3936
  [i915#3938]: https://gitlab.freedesktop.org/drm/intel/issues/3938
  [i915#3952]: https://gitlab.freedesktop.org/drm/intel/issues/3952
  [i915#3953]: https://gitlab.freedesktop.org/drm/intel/issues/3953
  [i915#3966]: https://gitlab.freedesktop.org/drm/intel/issues/3966
  [i915#3989]: https://gitlab.freedesktop.org/drm/intel/issues/3989
  [i915#4016]: https://gitlab.freedesktop.org/drm/intel/issues/4016
  [i915#404]: https://gitlab.freedesktop.org/drm/intel/issues/404
  [i915#4070]: https://gitlab.freedesktop.org/drm/intel/issues/4070
  [i915#4077]: https://gitlab.freedesktop.org/drm/intel/issues/4077
  [i915#4078]: https://gitlab.freedesktop.org/drm/intel/issues/4078
  [i915#4079]: https://gitlab.freedesktop.org/drm/intel/issues/4079
  [i915#4083]: https://gitlab.freedesktop.org/drm/intel/issues/4083
  [i915#4098]: https://gitlab.freedesktop.org/drm/intel/issues/4098
  [i915#4103]: https://gitlab.freedesktop.org/drm/intel/issues/4103
  [i915#4171]: https://gitlab.freedesktop.org/drm/intel/issues/4171
  [i915#4212]: https://gitlab.freedesktop.org/drm/intel/issues/4212
  [i915#4213]: https://gitlab.freedesktop.org/drm/intel/issues/4213
  [i915#4215]: https://gitlab.freedesktop.org/drm/intel/issues/4215
  [i915#426]: https://gitlab.freedesktop.org/drm/intel/issues/426
  [i915#4270]: https://gitlab.freedesktop.org/drm/intel/issues/4270
  [i915#4281]: https://gitlab.freedesktop.org/drm/intel/issues/4281
  [i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
  [i915#4349]: https://gitlab.freedesktop.org/drm/intel/issues/4349
  [i915#4369]: https://gitlab.freedesktop.org/drm/intel/issues/4369
  [i915#4387]: https://gitlab.freedesktop.org/drm/intel/issues/4387
  [i915#4462]: https://gitlab.freedesktop.org/drm/intel/issues/4462
  [i915#4525]: https://gitlab.freedesktop.org/drm/intel/issues/4525
  [i915#4538]: https://gitlab.freedesktop.org/drm/intel/issues/4538
  [i915#454]: https://gitlab.freedesktop.org/drm/intel/issues/454
  [i915#4565]: https://gitlab.freedesktop.org/drm/intel/issues/4565
  [i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
  [i915#4771]: https://gitlab.freedesktop.org/drm/intel/issues/4771
  [i915#4812]: https://gitlab.freedesktop.org/drm/intel/issues/4812
  [i915#4818]: https://gitlab.freedesktop.org/drm/intel/issues/4818
  [i915#4833]: https://gitlab.freedesktop.org/drm/intel/issues/4833
  [i915#4852]: https://gitlab.freedesktop.org/drm/intel/issues/4852
  [i915#4853]: https://gitlab.freedesktop.org/drm/intel/issues/4853
  [i915#4854]: https://gitlab.freedesktop.org/drm/intel/issues/4854
  [i915#4855]: https://gitlab.freedesktop.org/drm/intel/issues/4855
  [i915#4859]: https://gitlab.freedesktop.org/drm/intel/issues/4859
  [i915#4860]: https://gitlab.freedesktop.org/drm/intel/issues/4860
  [i915#4873]: https://gitlab.freedesktop.org/drm/intel/issues/4873
  [i915#4874]: https://gitlab.freedesktop.org/drm/intel/issues/4874
  [i915#4877]: https://gitlab.freedesktop.org/drm/intel/issues/4877
  [i915#4879]: https://gitlab.freedesktop.org/drm/intel/issues/4879
  [i915#4880]: https://gitlab.freedesktop.org/drm/intel/issues/4880
  [i915#4881]: https://gitlab.freedesktop.org/drm/intel/issues/4881
  [i915#4883]: https://gitlab.freedesktop.org/drm/intel/issues/4883
  [i915#4885]: https://gitlab.freedesktop.org/drm/intel/issues/4885
  [i915#4893]: https://gitlab.freedesktop.org/drm/intel/issues/4893
  [i915#4904]: https://gitlab.freedesktop.org/drm/intel/issues/4904
  [i915#4939]: https://gitlab.freedesktop.org/drm/intel/issues/4939
  [i915#4941]: https://gitlab.freedesktop.org/drm/intel/issues/4941
  [i915#4991]: https://gitlab.freedesktop.org/drm/intel/issues/4991
  [i915#5030]: https://gitlab.freedesktop.org/drm/intel/issues/5030
  [i915#5176]: https://gitlab.freedesktop.org/drm/intel/issues/5176
  [i915#5235]: https://gitlab.freedesktop.org/drm/intel/issues/5235
  [i915#5257]: https://gitlab.freedesktop.org/drm/intel/issues/5257
  [i915#5286]: https://gitlab.freedesktop.org/drm/intel/issues/5286
  [i915#5287]: https://gitlab.freedesktop.org/drm/intel/issues/5287
  [i915#5288]: https://gitlab.freedesktop.org/drm/intel/issues/5288
  [i915#5289]: https://gitlab.freedesktop.org/drm/intel/issues/5289
  [i915#5325]: https://gitlab.freedesktop.org/drm/intel/issues/5325
  [i915#5327]: https://gitlab.freedesktop.org/drm/intel/issues/5327
  [i915#533]: https://gitlab.freedesktop.org/drm/intel/issues/533
  [i915#5439]: https://gitlab.freedesktop.org/drm/intel/issues/5439
  [i915#5461]: https://gitlab.freedesktop.org/drm/intel/issues/5461
  [i915#5519]: https://gitlab.freedesktop.org/drm/intel/issues/5519
  [i915#5563]: https://gitlab.freedesktop.org/drm/intel/issues/5563
  [i915#5591]: https://gitlab.freedesktop.org/drm/intel/issues/5591
  [i915#5723]: https://gitlab.freedesktop.org/drm/intel/issues/5723
  [i915#5748]: https://gitlab.freedesktop.org/drm/intel/issues/5748
  [i915#5784]: https://gitlab.freedesktop.org/drm/intel/issues/5784
  [i915#5903]: https://gitlab.freedesktop.org/drm/intel/issues/5903
  [i915#6011]: https://gitlab.freedesktop.org/drm/intel/issues/6011
  [i915#6095]: https://gitlab.freedesktop.org/drm/intel/issues/6095
  [i915#6117]: https://gitlab.freedesktop.org/drm/intel/issues/6117
  [i915#6140]: https://gitlab.freedesktop.org/drm/intel/issues/6140
  [i915#6227]: https://gitlab.freedesktop.org/drm/intel/issues/6227
  [i915#6230]: https://gitlab.freedesktop.org/drm/intel/issues/6230
  [i915#6247]: https://gitlab.freedesktop.org/drm/intel/issues/6247
  [i915#6301]: https://gitlab.freedesktop.org/drm/intel/issues/6301
  [i915#6334]: https://gitlab.freedesktop.org/drm/intel/issues/6334
  [i915#6335]: https://gitlab.freedesktop.org/drm/intel/issues/6335
  [i915#6344]: https://gitlab.freedesktop.org/drm/intel/issues/6344
  [i915#6355]: https://gitlab.freedesktop.org/drm/intel/issues/6355
  [i915#658]: https://gitlab.freedesktop.org/drm/intel/issues/658
  [i915#794]: https://gitlab.freedesktop.org/drm/intel/issues/794
  [i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_6563 -> IGTPW_7487
  * Piglit: piglit_4509 -> None

  CI-20190529: 20190529
  CI_DRM_11862: ffee806d103b9604db7eb9cd689c098aca1ffa96 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_7487: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html
  IGT_6563: 7d43b49bf10788d4870668f93a800888fc8ab339 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7487/index.html

[-- Attachment #2: Type: text/html, Size: 42517 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection Zbigniew Kempczyński
@ 2022-07-12 13:57   ` Kamil Konieczny
  2022-07-12 16:29     ` Zbigniew Kempczyński
  0 siblings, 1 reply; 8+ messages in thread
From: Kamil Konieczny @ 2022-07-12 13:57 UTC (permalink / raw)
  To: igt-dev; +Cc: Chris Wilson

Hi,

On 2022-07-08 at 14:24:21 +0200, Zbigniew Kempczyński wrote:
> From: Chris Wilson <chris.p.wilson@linux.intel.com>
> 
> Reduce the number of faults we inject during module reload to bring the
> test runtime back to a reasonable level, and rely on multiple CI runs to
> establish the full picture. This reduces the efficacy of testing (i.e.
--------------------------------------------------- ^
s/efficacy/efficency/

> premerge will probably not prevent inclusion of a buggy patch), but
> error handling on module reload is not a critical factor for driver
> health -- a broken patch that explodes in an error path is unlikely to
> prevent CI from detecting critical issues, and with multiple runs we
> should still be able to detect and fix issues before merging.
> 
> Fixes: https://gitlab.freedesktop.org/drm/intel/-/issues/6393
> 
> Signed-off-by: Chris Wilson <chris.p.wilson@linux.intel.com>
> Signed-off-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
> Cc: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
> ---
>  tests/i915/i915_module_load.c | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)
> 
> diff --git a/tests/i915/i915_module_load.c b/tests/i915/i915_module_load.c
> index 4705f3d6c4..a922af0064 100644
> --- a/tests/i915/i915_module_load.c
> +++ b/tests/i915/i915_module_load.c
> @@ -302,21 +302,28 @@ igt_main
>  		     " with fault injection.");
>  	igt_subtest("reload-with-fault-injection") {
>  		const char *param;
> -		int i = 0;
> +		int i;
>  
>  		igt_i915_driver_unload();
>  
> +		i = 0;
> +		param = getenv("IGT_SRANDOM");
> +		if (param)
> +			i = atoi(param);
> +		if (!i)
> +			i = time(NULL);
> +		igt_info("Using IGT_SRANDOM=%d for randomised faults\n", i);
> +		srandom(i);
> +
>  		param = "inject_probe_failure";
>  		if (!igt_kmod_has_param("i915", param))
>  			param = "inject_load_failure";
>  		igt_require(igt_kmod_has_param("i915", param));
>  
> -		while (inject_fault("i915", param, ++i) == 0)
> +		i = 0;
> +		while (inject_fault("i915", param, i += 1 + random() % 17) == 0)
----------------------------------------------------------------------- ^ --- ^

imho this should be !inject_fault instead of == 0.
Second point is using constant 17, as it depends on i915 driver
but I assume it is enough for now.

>  			;

Setting var i before while() makes it a for loop, so maybe:
		for (i = 1; !inject_fault("i915", param, i); )
			i += 1 + random() % 17;

As side effect, this will always start from injecting at 1, only
after that will increase param i randomly. You can use
do/while if you want to start randomly:
		do
			i += 1 + random() % 17;
		while (!inject_fault("i915", param, i));

It is not strong opinion, you can leave code as is, your choice.

>  
> -		/* We expect to hit at least one fault! */
> -		igt_assert(i > 1);
> -

This is good to remove, as the only thing tested here was local
variable incrementing above zero.

Please rebase before resending or merge. With that fixed you can
add my r-b tag.

Regards,
Kamil

>  		/* inject_fault() leaves the module unloaded */
>  	}
>  
> -- 
> 2.34.1
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module
  2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module Zbigniew Kempczyński
@ 2022-07-12 14:29   ` Kamil Konieczny
  0 siblings, 0 replies; 8+ messages in thread
From: Kamil Konieczny @ 2022-07-12 14:29 UTC (permalink / raw)
  To: igt-dev; +Cc: Jonathan Cavitt, Chris Wilson, Petri Latvala

Hi,

On 2022-07-08 at 14:24:22 +0200, Zbigniew Kempczyński wrote:
> From: Chris Wilson <chris.p.wilson@linux.intel.com>
> 
> If we fail to unload the module after the fault-injection test, we leave
> behind a dangerous modparam that will affect all subsequent tests.
> Better to declare the run void rather than have to triage all the bizarre
--------------------------- ^
maybe s/void/fails/ ?

> bugs.
> 
> Somes the fault injection module load test timeouts out, causing
- ^
s/Somes/Sometimes/

> igt_runner to cancel the test with a SIGQUIT. This unfortunately causes
> us to abort in the middle of the test leaving the module in a broken
> state. That broken module is then used for subsequent tests, casuing a
-------------------------------------------------------------- ^
s/casuing/causing/

> cascade of failures.
> 
> Signed-off-by: Chris Wilson <chris.p.wilson@linux.intel.com>
> Signed-off-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
> Cc: Petri Latvala <petri.latvala@intel.com>
> Cc: Jonathan Cavitt <jonathan.cavitt@intel.com>
> ---
>  tests/i915/i915_module_load.c | 37 ++++++++++++++++++++++++++++++++++-
>  1 file changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/i915/i915_module_load.c b/tests/i915/i915_module_load.c
> index a922af0064..170900b1ab 100644
> --- a/tests/i915/i915_module_load.c
> +++ b/tests/i915/i915_module_load.c
> @@ -163,6 +163,33 @@ static int open_parameters(const char *module_name)
>  	return open(path, O_RDONLY);
>  }
>  
> +static void unload_or_die(const char *module_name)
> +{
> +	int err, loop;
> +
> +	/* should be unloaded, so expect a no-op */
> +	for (loop = 0;; loop++) {
> +		err = igt_kmod_unload(module_name, 0);
> +		if (err && err == -ENOENT) /* -ENOENT == unloaded already */
------------------- ^^^^^^
Drop this.

> +			err = 0;
> +		if (!err || loop >= 10)
--------------------------  ^
Put this at for loop test, e.g.: for(loop = 0; loop <= 10; ++loop)
Btw why 10 ? imho 3 should be more than enough.

> +			break;
> +
> +		sleep(1); /* wait for external clients to drop */
> +		if (!strcmp(module_name, "i915"))
> +			igt_i915_driver_unload();

We are calling this from one place only, with "i915" as param.
Do we plan to change this in near future ? If not, drop if() and
just call igt_i915_driver_unload() (you can add a comment here).

With that fixed you can add my r-b tag.

Regards,
Kamil

> +	}
> +
> +	igt_abort_on_f(err,
> +		       "Failed to unload '%s' err:%d after %ds, leaving dangerous modparams intact!\n",
> +		       module_name, err, loop);
> +}
> +
> +static void must_unload(int sig)
> +{
> +	unload_or_die("i915");
> +}
> +
>  static int
>  inject_fault(const char *module_name, const char *opt, int fault)
>  {
> @@ -306,6 +333,14 @@ igt_main
>  
>  		igt_i915_driver_unload();
>  
> +		/*
> +		 * inject_fault() leaves the module unloaded, but if that fails
> +		 * we must abort the run. Otherwise, we leave a dangerous
> +		 * modparam affecting all subsequent tests causing bizarre
> +		 * failures.
> +		 */
> +		igt_install_exit_handler(must_unload);
> +
>  		i = 0;
>  		param = getenv("IGT_SRANDOM");
>  		if (param)
> @@ -324,7 +359,7 @@ igt_main
>  		while (inject_fault("i915", param, i += 1 + random() % 17) == 0)
>  			;
>  
> -		/* inject_fault() leaves the module unloaded */
> +		unload_or_die("i915");
>  	}
>  
>  	/* Subtests should unload the module themselves if they use modparams */
> -- 
> 2.34.1
> 

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection
  2022-07-12 13:57   ` Kamil Konieczny
@ 2022-07-12 16:29     ` Zbigniew Kempczyński
  0 siblings, 0 replies; 8+ messages in thread
From: Zbigniew Kempczyński @ 2022-07-12 16:29 UTC (permalink / raw)
  To: Kamil Konieczny, igt-dev, Chris Wilson, Janusz Krzysztofik

On Tue, Jul 12, 2022 at 03:57:45PM +0200, Kamil Konieczny wrote:
> Hi,
> 
> On 2022-07-08 at 14:24:21 +0200, Zbigniew Kempczyński wrote:
> > From: Chris Wilson <chris.p.wilson@linux.intel.com>
> > 
> > Reduce the number of faults we inject during module reload to bring the
> > test runtime back to a reasonable level, and rely on multiple CI runs to
> > establish the full picture. This reduces the efficacy of testing (i.e.
> --------------------------------------------------- ^
> s/efficacy/efficency/

Efficacy is correct in this case. Increasing efficiency of the test would
reduce runtime but decrease the efficacy (more fault injections in this case
gives your better detection, but we get timeout).

> 
> > premerge will probably not prevent inclusion of a buggy patch), but
> > error handling on module reload is not a critical factor for driver
> > health -- a broken patch that explodes in an error path is unlikely to
> > prevent CI from detecting critical issues, and with multiple runs we
> > should still be able to detect and fix issues before merging.
> > 
> > Fixes: https://gitlab.freedesktop.org/drm/intel/-/issues/6393
> > 
> > Signed-off-by: Chris Wilson <chris.p.wilson@linux.intel.com>
> > Signed-off-by: Zbigniew Kempczyński <zbigniew.kempczynski@intel.com>
> > Cc: Janusz Krzysztofik <janusz.krzysztofik@linux.intel.com>
> > ---
> >  tests/i915/i915_module_load.c | 17 ++++++++++++-----
> >  1 file changed, 12 insertions(+), 5 deletions(-)
> > 
> > diff --git a/tests/i915/i915_module_load.c b/tests/i915/i915_module_load.c
> > index 4705f3d6c4..a922af0064 100644
> > --- a/tests/i915/i915_module_load.c
> > +++ b/tests/i915/i915_module_load.c
> > @@ -302,21 +302,28 @@ igt_main
> >  		     " with fault injection.");
> >  	igt_subtest("reload-with-fault-injection") {
> >  		const char *param;
> > -		int i = 0;
> > +		int i;
> >  
> >  		igt_i915_driver_unload();
> >  
> > +		i = 0;
> > +		param = getenv("IGT_SRANDOM");
> > +		if (param)
> > +			i = atoi(param);
> > +		if (!i)
> > +			i = time(NULL);
> > +		igt_info("Using IGT_SRANDOM=%d for randomised faults\n", i);
> > +		srandom(i);
> > +
> >  		param = "inject_probe_failure";
> >  		if (!igt_kmod_has_param("i915", param))
> >  			param = "inject_load_failure";
> >  		igt_require(igt_kmod_has_param("i915", param));
> >  
> > -		while (inject_fault("i915", param, ++i) == 0)
> > +		i = 0;
> > +		while (inject_fault("i915", param, i += 1 + random() % 17) == 0)
> ----------------------------------------------------------------------- ^ --- ^
> 
> imho this should be !inject_fault instead of == 0.
> Second point is using constant 17, as it depends on i915 driver
> but I assume it is enough for now.

Starting at 17 - there's no possibility to probe/calculate this value 
so it was chosen as a reasonable compromise. Test is completing in ~15s
instead 130s on machine where I was testing it.

> 
> >  			;
> 
> Setting var i before while() makes it a for loop, so maybe:
> 		for (i = 1; !inject_fault("i915", param, i); )
> 			i += 1 + random() % 17;

What I don't like in this condition (!inject_fault()) is it suggests
success where we cannot inject fault. That is not true and == 0 is more
explicit. We want to loop until inject_fault() == 0. Reading such I
understand when function succeed it returns 0. Having !inject_fault()
suggests - if we're not able to inject fault.

> 
> As side effect, this will always start from injecting at 1, only
> after that will increase param i randomly. You can use
> do/while if you want to start randomly:
> 		do
> 			i += 1 + random() % 17;
> 		while (!inject_fault("i915", param, i));
>

Assuming both I would rewrite this to make everyone happy:

i = 1;
while (inject_fault("i915", param, i) == 0)
	i += 1 + random() % 17;
 
> It is not strong opinion, you can leave code as is, your choice.
> 
> >  
> > -		/* We expect to hit at least one fault! */
> > -		igt_assert(i > 1);
> > -
> 
> This is good to remove, as the only thing tested here was local
> variable incrementing above zero.
> 
> Please rebase before resending or merge. With that fixed you can
> add my r-b tag.

Will resend v2 soon.

--
Zbigniew

> 
> Regards,
> Kamil
> 
> >  		/* inject_fault() leaves the module unloaded */
> >  	}
> >  
> > -- 
> > 2.34.1
> > 

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2022-07-12 16:29 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-08 12:24 [igt-dev] [PATCH i-g-t 0/2] Decrease module reload fault injection time Zbigniew Kempczyński
2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 1/2] i915/i915_module_load: Randomise fault injection Zbigniew Kempczyński
2022-07-12 13:57   ` Kamil Konieczny
2022-07-12 16:29     ` Zbigniew Kempczyński
2022-07-08 12:24 ` [igt-dev] [PATCH i-g-t 2/2] i915/i915_module_load: Raise a fatal-error on failing to unload a module Zbigniew Kempczyński
2022-07-12 14:29   ` Kamil Konieczny
2022-07-08 12:57 ` [igt-dev] ✓ Fi.CI.BAT: success for Decrease module reload fault injection time Patchwork
2022-07-08 23:27 ` [igt-dev] ✓ Fi.CI.IGT: " Patchwork

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.