[igt-dev] [PATCH i-g-t v3 1/2] scripts/verify-blacklist: Script for checking blacklist files

[igt-dev] [PATCH i-g-t v3 1/2] scripts/verify-blacklist: Script for checking blacklist files

[Petri Latvala]

tests/intel-ci/blacklist*.txt files can collect bitrot unless there's
an easy way to check for lines that are no longer needed due to the
tests being renamed or removed. Therefore, a script just for that.

v2: Use long options for readability, exit with 1 if something found

v3: Verify manually against the list of all tests instead of trying
    with igt_runner's --include-tests

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com> #v1
---
 scripts/verify-blacklist.sh | 54 +++++++++++++++++++++++++++++++++++++
 1 file changed, 54 insertions(+)
 create mode 100755 scripts/verify-blacklist.sh

diff --git a/scripts/verify-blacklist.sh b/scripts/verify-blacklist.sh
new file mode 100755
index 00000000..93dca495
--- /dev/null
+++ b/scripts/verify-blacklist.sh
@@ -0,0 +1,54 @@
+#!/bin/sh
+
+# Verify that all entries in a blacklist file are still valid
+
+usage() {
+    echo "Usage: $0 <path-to-igt-runner> <test-binary-directory> <blacklist-file>"
+    echo
+    echo " path-to-igt-runner: For example build/runner/igt_runner"
+    echo " test-binary-directory: For example build/tests"
+    echo " blacklist-file: For example tests/intel-ci/blacklist.txt"
+    exit 2
+}
+
+if [ $# -ne 3 ]; then
+    usage
+fi
+
+RUNNER="$1"
+BINDIR="$2"
+BLFILE="$3"
+
+if [ ! -x "$RUNNER" ]; then
+    echo "$RUNNER not found"
+    echo
+    usage
+fi
+
+if [ ! -f "$BINDIR/test-list.txt" ]; then
+    echo "$BINDIR doesn't look like a test-binary directory"
+    echo
+    usage
+fi
+
+if [ ! -f "$BLFILE" ]; then
+    echo "$BLFILE not found"
+    echo
+    usage
+fi
+
+STATUS=0
+
+TESTLIST="$("$RUNNER" --list-all "$BINDIR")"
+
+cat "$BLFILE" | while read line; do
+    blentry=$(echo "$line" | sed 's/#.*//' | tr -d '[:space:]')
+    if [ "$blentry" = "" ]; then continue; fi
+
+    if ! (echo "$TESTLIST" | grep -Pq "$blentry") >/dev/null 2>/dev/null; then
+	echo Useless blacklist entry: "$blentry"
+	STATUS=1
+    fi
+done
+
+exit $STATUS
-- 
2.20.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] [PATCH i-g-t 2/2] gitlab-ci: Verify blacklist files with verify-blacklist.sh

[Petri Latvala]

Signed-off-by: Petri Latvala <petri.latvala@intel.com>
Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
---

Now finished in 47 seconds in gitlab.


 .gitlab-ci.yml | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c5a6bd9e..d7fdbfde 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -252,6 +252,12 @@ test:list-undocumented-tests:
     paths:
       - undocumented_tests.txt
 
+test:verify-blacklists:
+  dependencies:
+    - build:tests-fedora
+  stage: test
+  script: for bl in tests/intel-ci/blacklist{,-pre-merge}.txt; do scripts/verify-blacklist.sh build/runner/igt_runner build/tests "$bl" || exit 1; done
+
 ################### DEPLOY #########################
 
 pages:
-- 
2.20.1

_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Re: [igt-dev] [PATCH i-g-t 2/2] gitlab-ci: Verify blacklist files with verify-blacklist.sh

[Arkadiusz Hiler]

On Mon, Jun 15, 2020 at 01:25:16PM +0300, Petri Latvala wrote:
> Signed-off-by: Petri Latvala <petri.latvala@intel.com>
> Cc: Arkadiusz Hiler <arkadiusz.hiler@intel.com>
> ---
> 
> Now finished in 47 seconds in gitlab.
> 
> 
>  .gitlab-ci.yml | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index c5a6bd9e..d7fdbfde 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -252,6 +252,12 @@ test:list-undocumented-tests:
>      paths:
>        - undocumented_tests.txt
>  
> +test:verify-blacklists:
> +  dependencies:
> +    - build:tests-fedora
> +  stage: test
> +  script: for bl in tests/intel-ci/blacklist{,-pre-merge}.txt; do scripts/verify-blacklist.sh build/runner/igt_runner build/tests "$bl" || exit 1; done
> +
>  ################### DEPLOY #########################
>  
>  pages:
> -- 
> 2.20.1

For both patches,

Reviewed-by: Arkadiusz Hiler <arkadiusz.hiler@intel.com>

We still should do something about the assume implicit ^$ that are not
there.
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✗ Fi.CI.BAT: failure for series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files

[Patchwork]

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files
URL   : https://patchwork.freedesktop.org/series/78368/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_8626 -> IGTPW_4675
====================================================

Summary
-------

  **FAILURE**

  Serious unknown changes coming with IGTPW_4675 absolutely need to be
  verified manually.
  
  If you think the reported changes have nothing to do with the changes
  introduced in IGTPW_4675, please notify your bug team to allow them
  to document this new failure mode, which will reduce false positives in CI.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html

Possible new issues
-------------------

  Here are the unknown changes that may have been introduced in IGTPW_4675:

### IGT changes ###

#### Possible regressions ####

  * igt@gem_exec_parallel@engines@fds:
    - fi-tgl-u2:          [PASS][1] -> [INCOMPLETE][2]
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-tgl-u2/igt@gem_exec_parallel@engines@fds.html
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-tgl-u2/igt@gem_exec_parallel@engines@fds.html

  
Known issues
------------

  Here are the changes found in IGTPW_4675 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_exec_suspend@basic-s0:
    - fi-apl-guc:         [PASS][3] -> [INCOMPLETE][4] ([i915#1242])
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-apl-guc/igt@gem_exec_suspend@basic-s0.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-apl-guc/igt@gem_exec_suspend@basic-s0.html

  * igt@i915_module_load@reload:
    - fi-byt-j1900:       [PASS][5] -> [DMESG-WARN][6] ([i915#1982])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-byt-j1900/igt@i915_module_load@reload.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-byt-j1900/igt@i915_module_load@reload.html
    - fi-byt-n2820:       [PASS][7] -> [DMESG-WARN][8] ([i915#1982])
   [7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-byt-n2820/igt@i915_module_load@reload.html
   [8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-byt-n2820/igt@i915_module_load@reload.html

  * igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic:
    - fi-bsw-kefka:       [PASS][9] -> [DMESG-WARN][10] ([i915#1982]) +1 similar issue
   [9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-bsw-kefka/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html
   [10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-bsw-kefka/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html

  * igt@kms_flip@basic-flip-vs-wf_vblank@c-edp1:
    - fi-icl-u2:          [PASS][11] -> [DMESG-WARN][12] ([i915#1982]) +1 similar issue
   [11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@c-edp1.html
   [12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@c-edp1.html

  
#### Possible fixes ####

  * igt@kms_cursor_legacy@basic-flip-before-cursor-atomic:
    - fi-icl-guc:         [DMESG-WARN][13] ([i915#1982]) -> [PASS][14]
   [13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-guc/igt@kms_cursor_legacy@basic-flip-before-cursor-atomic.html
   [14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-guc/igt@kms_cursor_legacy@basic-flip-before-cursor-atomic.html

  * igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1:
    - fi-icl-u2:          [DMESG-WARN][15] ([i915#1982]) -> [PASS][16] +2 similar issues
   [15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1.html
   [16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-icl-u2/igt@kms_flip@basic-flip-vs-wf_vblank@b-edp1.html

  * igt@kms_flip@basic-plain-flip@d-dsi1:
    - {fi-tgl-dsi}:       [DMESG-WARN][17] ([i915#1982]) -> [PASS][18]
   [17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-tgl-dsi/igt@kms_flip@basic-plain-flip@d-dsi1.html
   [18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-tgl-dsi/igt@kms_flip@basic-plain-flip@d-dsi1.html

  
#### Warnings ####

  * igt@gem_exec_suspend@basic-s3:
    - fi-kbl-x1275:       [DMESG-WARN][19] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][20] ([i915#1982] / [i915#62] / [i915#92] / [i915#95])
   [19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@gem_exec_suspend@basic-s3.html
   [20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@gem_exec_suspend@basic-s3.html

  * igt@kms_flip@basic-flip-vs-modeset@a-dp1:
    - fi-kbl-x1275:       [DMESG-WARN][21] ([i915#62] / [i915#92]) -> [DMESG-WARN][22] ([i915#62] / [i915#92] / [i915#95]) +5 similar issues
   [21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@kms_flip@basic-flip-vs-modeset@a-dp1.html
   [22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@kms_flip@basic-flip-vs-modeset@a-dp1.html

  * igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
    - fi-kbl-x1275:       [DMESG-WARN][23] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][24] ([i915#62] / [i915#92]) +4 similar issues
   [23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_8626/fi-kbl-x1275/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html
   [24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/fi-kbl-x1275/igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [i915#1242]: https://gitlab.freedesktop.org/drm/intel/issues/1242
  [i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
  [i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62
  [i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92
  [i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95


Participating hosts (44 -> 41)
------------------------------

  Additional (3): fi-whl-u fi-bdw-5557u fi-skl-6600u 
  Missing    (6): fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-byt-clapper fi-bdw-samus 


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_5710 -> IGTPW_4675

  CI-20190529: 20190529
  CI_DRM_8626: 563d95505a36489373782ca98d67897d4b338f94 @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_4675: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html
  IGT_5710: f524eee47930601ad7b4cba9d40c26d68dc7d250 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_4675/index.html
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev

[igt-dev] ✗ GitLab.Pipeline: warning for series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files

[Patchwork]

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] scripts/verify-blacklist: Script for checking blacklist files
URL   : https://patchwork.freedesktop.org/series/78368/
State : warning

== Summary ==

Did not get list of undocumented tests for this run, something is wrong!

Other than that, pipeline status: FAILED.

see https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/pipelines/160705 for the overview.

build-containers:build-debian-arm64 has failed (https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/-/jobs/3098121):
  Using docker image sha256:594aa868d31ee3304dee8cae8a3433c89a6fcfcf6c7d420c04cce22f60147176 for registry.freedesktop.org/wayland/ci-templates/buildah:2019-08-13.0 ...
  section_end:1592218945:prepare_executor
  section_start:1592218945:prepare_script
  Preparing environment
  Running on runner-j4Lrg1oF-project-3185-concurrent-1 via gst-htz-1...
  section_end:1592218947:prepare_script
  section_start:1592218947:get_sources
  Getting source from Git repository
  Fetching changes...
  Initialized empty Git repository in /builds/gfx-ci/igt-ci-tags/.git/
  Created fresh repository.
  warning: redirecting to https://gitlab-ci-hetzner.freedesktop.org/gfx-ci/igt-ci-tags.git/
  error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
  fatal: the remote end hung up unexpectedly
  section_end:1592219009:get_sources
  section_start:1592219009:upload_artifacts_on_failure
  Uploading artifacts for failed job
  section_end:1592219011:upload_artifacts_on_failure
  ERROR: Job failed: exit code 1
  

build-containers:build-debian-armhf has failed (https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/-/jobs/3098120):
  Using docker image sha256:594aa868d31ee3304dee8cae8a3433c89a6fcfcf6c7d420c04cce22f60147176 for registry.freedesktop.org/wayland/ci-templates/buildah:2019-08-13.0 ...
  section_end:1592218945:prepare_executor
  section_start:1592218945:prepare_script
  Preparing environment
  Running on runner-z2CifdYy-project-3185-concurrent-0 via gst-htz-3...
  section_end:1592218946:prepare_script
  section_start:1592218946:get_sources
  Getting source from Git repository
  Fetching changes...
  Initialized empty Git repository in /builds/gfx-ci/igt-ci-tags/.git/
  Created fresh repository.
  warning: redirecting to https://gitlab-ci-hetzner.freedesktop.org/gfx-ci/igt-ci-tags.git/
  error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
  fatal: the remote end hung up unexpectedly
  section_end:1592219009:get_sources
  section_start:1592219009:upload_artifacts_on_failure
  Uploading artifacts for failed job
  section_end:1592219010:upload_artifacts_on_failure
  ERROR: Job failed: exit code 1

== Logs ==

For more details see: https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags/pipelines/160705
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev