* [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
@ 2020-11-02 9:02 Anand Moon
2020-11-03 10:27 ` [igt-dev] ✓ Fi.CI.BAT: success for tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2) Patchwork
2020-11-03 16:04 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
0 siblings, 2 replies; 9+ messages in thread
From: Anand Moon @ 2020-11-02 9:02 UTC (permalink / raw)
To: igt-dev, Dorota Czaplejewicz; +Cc: Anand Moon
Using confidence interval formula accuracy is estimates approximately
at 95% when {X} is the average of a sample of size n.
Use this confidence interval formula as used in statistics
to estimate the accuracy.
Cc: Dorota Czaplejewicz <dorota.czaplejewicz@collabora.co.uk>
Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
---
See below link.
https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulative_distribution_function
---
tests/kms_setmode.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c
index 92f3ead2..58814a13 100644
--- a/tests/kms_setmode.c
+++ b/tests/kms_setmode.c
@@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
* which depend on timing, giving the lowest acceptable MTBF of 5.6s
* for 60Hz sampling rate.
*/
- accuracy = 3. * stddev;
+ accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
igt_info("Expected frametime: %.0fus; measured %.1fus +- %.3fus accuracy %.2f%% [%.2f scanlines]\n",
expected, mean, stddev,
--
2.25.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [igt-dev] ✓ Fi.CI.BAT: success for tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2)
2020-11-02 9:02 [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval Anand Moon
@ 2020-11-03 10:27 ` Patchwork
2020-11-03 16:04 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
1 sibling, 0 replies; 9+ messages in thread
From: Patchwork @ 2020-11-03 10:27 UTC (permalink / raw)
To: Anand Moon; +Cc: igt-dev
[-- Attachment #1.1: Type: text/plain, Size: 4415 bytes --]
== Series Details ==
Series: tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2)
URL : https://patchwork.freedesktop.org/series/83411/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_9250 -> IGTPW_5120
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
Known issues
------------
Here are the changes found in IGTPW_5120 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@i915_module_load@reload:
- fi-icl-u2: [PASS][1] -> [DMESG-WARN][2] ([i915#1982]) +1 similar issue
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-icl-u2/igt@i915_module_load@reload.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-icl-u2/igt@i915_module_load@reload.html
- fi-icl-y: [PASS][3] -> [DMESG-WARN][4] ([i915#1982])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-icl-y/igt@i915_module_load@reload.html
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-icl-y/igt@i915_module_load@reload.html
* igt@i915_pm_rpm@module-reload:
- fi-skl-lmem: [PASS][5] -> [DMESG-WARN][6] ([i915#2605])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-skl-lmem/igt@i915_pm_rpm@module-reload.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-skl-lmem/igt@i915_pm_rpm@module-reload.html
* igt@i915_selftest@live@gt_timelines:
- fi-apl-guc: [PASS][7] -> [INCOMPLETE][8] ([i915#1635])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-apl-guc/igt@i915_selftest@live@gt_timelines.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-apl-guc/igt@i915_selftest@live@gt_timelines.html
* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic:
- fi-byt-j1900: [PASS][9] -> [DMESG-WARN][10] ([i915#1982])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-byt-j1900/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-byt-j1900/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html
#### Possible fixes ####
* igt@core_hotunplug@unbind-rebind:
- fi-icl-y: [DMESG-WARN][11] ([i915#1982]) -> [PASS][12]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-icl-y/igt@core_hotunplug@unbind-rebind.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-icl-y/igt@core_hotunplug@unbind-rebind.html
* igt@kms_chamelium@dp-crc-fast:
- fi-cml-u2: [DMESG-WARN][13] ([i915#1982]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-cml-u2/igt@kms_chamelium@dp-crc-fast.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-cml-u2/igt@kms_chamelium@dp-crc-fast.html
* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic:
- fi-apl-guc: [DMESG-WARN][15] ([i915#1635] / [i915#1982]) -> [PASS][16]
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/fi-apl-guc/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/fi-apl-guc/igt@kms_cursor_legacy@basic-busy-flip-before-cursor-atomic.html
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[i915#1635]: https://gitlab.freedesktop.org/drm/intel/issues/1635
[i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
[i915#2411]: https://gitlab.freedesktop.org/drm/intel/issues/2411
[i915#2605]: https://gitlab.freedesktop.org/drm/intel/issues/2605
Participating hosts (43 -> 39)
------------------------------
Missing (4): fi-ilk-m540 fi-bsw-cyan fi-bdw-samus fi-hsw-4200u
Build changes
-------------
* CI: CI-20190529 -> None
* IGT: IGT_5830 -> IGTPW_5120
CI-20190529: 20190529
CI_DRM_9250: 0ddbbf768a932ea9ca82a5872fd83977e27347c0 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_5120: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
IGT_5830: 12d370cb57e0cfcb781c87ad9e15e68b17a1f41f @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
[-- Attachment #1.2: Type: text/html, Size: 5544 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* [igt-dev] ✗ Fi.CI.IGT: failure for tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2)
2020-11-02 9:02 [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval Anand Moon
2020-11-03 10:27 ` [igt-dev] ✓ Fi.CI.BAT: success for tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2) Patchwork
@ 2020-11-03 16:04 ` Patchwork
1 sibling, 0 replies; 9+ messages in thread
From: Patchwork @ 2020-11-03 16:04 UTC (permalink / raw)
To: Anand Moon; +Cc: igt-dev
[-- Attachment #1.1: Type: text/plain, Size: 24899 bytes --]
== Series Details ==
Series: tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2)
URL : https://patchwork.freedesktop.org/series/83411/
State : failure
== Summary ==
CI Bug Log - changes from CI_DRM_9250_full -> IGTPW_5120_full
====================================================
Summary
-------
**FAILURE**
Serious unknown changes coming with IGTPW_5120_full absolutely need to be
verified manually.
If you think the reported changes have nothing to do with the changes
introduced in IGTPW_5120_full, please notify your bug team to allow them
to document this new failure mode, which will reduce false positives in CI.
External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
Possible new issues
-------------------
Here are the unknown changes that may have been introduced in IGTPW_5120_full:
### IGT changes ###
#### Possible regressions ####
* igt@gem_eio@in-flight-suspend:
- shard-hsw: [PASS][1] -> [DMESG-WARN][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw1/igt@gem_eio@in-flight-suspend.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw4/igt@gem_eio@in-flight-suspend.html
- shard-glk: [PASS][3] -> [DMESG-WARN][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk5/igt@gem_eio@in-flight-suspend.html
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk6/igt@gem_eio@in-flight-suspend.html
* igt@gen7_exec_parse@chained-batch:
- shard-hsw: [PASS][5] -> [FAIL][6]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw1/igt@gen7_exec_parse@chained-batch.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@gen7_exec_parse@chained-batch.html
Known issues
------------
Here are the changes found in IGTPW_5120_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_eio@in-flight-suspend:
- shard-kbl: [PASS][7] -> [INCOMPLETE][8] ([i915#155])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl3/igt@gem_eio@in-flight-suspend.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl4/igt@gem_eio@in-flight-suspend.html
- shard-apl: [PASS][9] -> [INCOMPLETE][10] ([i915#1635])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl7/igt@gem_eio@in-flight-suspend.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl6/igt@gem_eio@in-flight-suspend.html
* igt@gem_exec_whisper@basic-forked:
- shard-glk: [PASS][11] -> [DMESG-WARN][12] ([i915#118] / [i915#95])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk8/igt@gem_exec_whisper@basic-forked.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk7/igt@gem_exec_whisper@basic-forked.html
* igt@gem_partial_pwrite_pread@writes-after-reads-uncached:
- shard-snb: [PASS][13] -> [INCOMPLETE][14] ([i915#82])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-snb7/igt@gem_partial_pwrite_pread@writes-after-reads-uncached.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-snb4/igt@gem_partial_pwrite_pread@writes-after-reads-uncached.html
* igt@i915_pm_backlight@fade_with_suspend:
- shard-iclb: [PASS][15] -> [INCOMPLETE][16] ([i915#1185])
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb1/igt@i915_pm_backlight@fade_with_suspend.html
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb4/igt@i915_pm_backlight@fade_with_suspend.html
* igt@i915_pm_rpm@sysfs-read:
- shard-iclb: [PASS][17] -> [SKIP][18] ([i915#579])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@i915_pm_rpm@sysfs-read.html
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb2/igt@i915_pm_rpm@sysfs-read.html
- shard-apl: [PASS][19] -> [SKIP][20] ([fdo#109271] / [i915#1635])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl2/igt@i915_pm_rpm@sysfs-read.html
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl6/igt@i915_pm_rpm@sysfs-read.html
- shard-glk: [PASS][21] -> [SKIP][22] ([fdo#109271])
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk9/igt@i915_pm_rpm@sysfs-read.html
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk2/igt@i915_pm_rpm@sysfs-read.html
- shard-kbl: [PASS][23] -> [SKIP][24] ([fdo#109271])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl7/igt@i915_pm_rpm@sysfs-read.html
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl7/igt@i915_pm_rpm@sysfs-read.html
* igt@kms_cursor_edge_walk@pipe-b-64x64-left-edge:
- shard-apl: [PASS][25] -> [DMESG-WARN][26] ([i915#1635] / [i915#1982]) +5 similar issues
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl3/igt@kms_cursor_edge_walk@pipe-b-64x64-left-edge.html
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl7/igt@kms_cursor_edge_walk@pipe-b-64x64-left-edge.html
- shard-tglb: [PASS][27] -> [DMESG-WARN][28] ([i915#1982]) +3 similar issues
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb2/igt@kms_cursor_edge_walk@pipe-b-64x64-left-edge.html
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb8/igt@kms_cursor_edge_walk@pipe-b-64x64-left-edge.html
* igt@kms_flip@2x-flip-vs-panning@ab-vga1-hdmi-a1:
- shard-hsw: [PASS][29] -> [DMESG-WARN][30] ([i915#1982]) +1 similar issue
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw1/igt@kms_flip@2x-flip-vs-panning@ab-vga1-hdmi-a1.html
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw4/igt@kms_flip@2x-flip-vs-panning@ab-vga1-hdmi-a1.html
* igt@kms_flip@flip-vs-expired-vblank-interruptible@a-edp1:
- shard-tglb: [PASS][31] -> [FAIL][32] ([i915#2598])
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb2/igt@kms_flip@flip-vs-expired-vblank-interruptible@a-edp1.html
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb7/igt@kms_flip@flip-vs-expired-vblank-interruptible@a-edp1.html
* igt@kms_flip@plain-flip-fb-recreate@d-edp1:
- shard-tglb: [PASS][33] -> [FAIL][34] ([i915#2122])
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb8/igt@kms_flip@plain-flip-fb-recreate@d-edp1.html
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb1/igt@kms_flip@plain-flip-fb-recreate@d-edp1.html
* igt@kms_frontbuffer_tracking@fbc-1p-rte:
- shard-glk: [PASS][35] -> [DMESG-WARN][36] ([i915#1982]) +6 similar issues
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk8/igt@kms_frontbuffer_tracking@fbc-1p-rte.html
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk2/igt@kms_frontbuffer_tracking@fbc-1p-rte.html
* igt@kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt:
- shard-snb: [PASS][37] -> [FAIL][38] ([i915#2546])
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-snb7/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt.html
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-snb4/igt@kms_frontbuffer_tracking@fbc-rgb565-draw-mmap-gtt.html
* igt@kms_psr@psr2_primary_mmap_cpu:
- shard-iclb: [PASS][39] -> [SKIP][40] ([fdo#109441]) +2 similar issues
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb2/igt@kms_psr@psr2_primary_mmap_cpu.html
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb8/igt@kms_psr@psr2_primary_mmap_cpu.html
* igt@kms_universal_plane@universal-plane-gen9-features-pipe-a:
- shard-kbl: [PASS][41] -> [DMESG-WARN][42] ([i915#1982]) +6 similar issues
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl7/igt@kms_universal_plane@universal-plane-gen9-features-pipe-a.html
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl3/igt@kms_universal_plane@universal-plane-gen9-features-pipe-a.html
* igt@perf@invalid-oa-format-id:
- shard-kbl: [PASS][43] -> [SKIP][44] ([fdo#109271] / [i915#1354])
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl1/igt@perf@invalid-oa-format-id.html
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl2/igt@perf@invalid-oa-format-id.html
- shard-hsw: [PASS][45] -> [SKIP][46] ([fdo#109271]) +1 similar issue
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw6/igt@perf@invalid-oa-format-id.html
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@perf@invalid-oa-format-id.html
- shard-tglb: [PASS][47] -> [SKIP][48] ([i915#1354])
[47]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb2/igt@perf@invalid-oa-format-id.html
[48]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb8/igt@perf@invalid-oa-format-id.html
- shard-iclb: [PASS][49] -> [SKIP][50] ([i915#1354])
[49]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb1/igt@perf@invalid-oa-format-id.html
[50]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb2/igt@perf@invalid-oa-format-id.html
- shard-glk: [PASS][51] -> [SKIP][52] ([fdo#109271] / [i915#1354])
[51]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk8/igt@perf@invalid-oa-format-id.html
[52]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk9/igt@perf@invalid-oa-format-id.html
- shard-apl: [PASS][53] -> [SKIP][54] ([fdo#109271] / [i915#1354] / [i915#1635])
[53]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl4/igt@perf@invalid-oa-format-id.html
[54]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl2/igt@perf@invalid-oa-format-id.html
* igt@perf_pmu@module-unload:
- shard-iclb: [PASS][55] -> [DMESG-WARN][56] ([i915#1982]) +1 similar issue
[55]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb8/igt@perf_pmu@module-unload.html
[56]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb7/igt@perf_pmu@module-unload.html
#### Possible fixes ####
* igt@gem_exec_reloc@basic-many-active@rcs0:
- shard-glk: [FAIL][57] ([i915#2389]) -> [PASS][58]
[57]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk2/igt@gem_exec_reloc@basic-many-active@rcs0.html
[58]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk3/igt@gem_exec_reloc@basic-many-active@rcs0.html
* igt@gem_partial_pwrite_pread@writes-after-reads-uncached:
- shard-hsw: [FAIL][59] ([i915#1888]) -> [PASS][60]
[59]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw1/igt@gem_partial_pwrite_pread@writes-after-reads-uncached.html
[60]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@gem_partial_pwrite_pread@writes-after-reads-uncached.html
* igt@i915_pm_rpm@pm-tiling:
- shard-hsw: [SKIP][61] ([fdo#109271]) -> [PASS][62] +2 similar issues
[61]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw4/igt@i915_pm_rpm@pm-tiling.html
[62]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@i915_pm_rpm@pm-tiling.html
- shard-kbl: [SKIP][63] ([fdo#109271]) -> [PASS][64] +1 similar issue
[63]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl6/igt@i915_pm_rpm@pm-tiling.html
[64]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl2/igt@i915_pm_rpm@pm-tiling.html
- shard-apl: [SKIP][65] ([fdo#109271] / [i915#1635]) -> [PASS][66] +1 similar issue
[65]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl6/igt@i915_pm_rpm@pm-tiling.html
[66]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl7/igt@i915_pm_rpm@pm-tiling.html
- shard-glk: [SKIP][67] ([fdo#109271]) -> [PASS][68] +1 similar issue
[67]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk1/igt@i915_pm_rpm@pm-tiling.html
[68]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk5/igt@i915_pm_rpm@pm-tiling.html
- shard-iclb: [SKIP][69] ([i915#579]) -> [PASS][70]
[69]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb8/igt@i915_pm_rpm@pm-tiling.html
[70]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb8/igt@i915_pm_rpm@pm-tiling.html
* igt@i915_suspend@fence-restore-untiled:
- shard-kbl: [DMESG-WARN][71] ([i915#180]) -> [PASS][72]
[71]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl2/igt@i915_suspend@fence-restore-untiled.html
[72]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl6/igt@i915_suspend@fence-restore-untiled.html
* igt@kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing@vga-1-pipe-a:
- shard-hsw: [DMESG-WARN][73] ([i915#1982]) -> [PASS][74] +2 similar issues
[73]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw6/igt@kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing@vga-1-pipe-a.html
[74]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw1/igt@kms_atomic_transition@plane-use-after-nonblocking-unbind-fencing@vga-1-pipe-a.html
* igt@kms_busy@basic-flip-pipe-b:
- shard-tglb: [DMESG-WARN][75] ([i915#1982]) -> [PASS][76] +2 similar issues
[75]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb7/igt@kms_busy@basic-flip-pipe-b.html
[76]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb8/igt@kms_busy@basic-flip-pipe-b.html
* igt@kms_draw_crc@draw-method-xrgb2101010-render-ytiled:
- shard-kbl: [DMESG-WARN][77] ([i915#1982]) -> [PASS][78] +4 similar issues
[77]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl1/igt@kms_draw_crc@draw-method-xrgb2101010-render-ytiled.html
[78]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl7/igt@kms_draw_crc@draw-method-xrgb2101010-render-ytiled.html
* igt@kms_draw_crc@draw-method-xrgb8888-render-untiled:
- shard-glk: [DMESG-WARN][79] ([i915#1982]) -> [PASS][80] +6 similar issues
[79]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk2/igt@kms_draw_crc@draw-method-xrgb8888-render-untiled.html
[80]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk7/igt@kms_draw_crc@draw-method-xrgb8888-render-untiled.html
* igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-render:
- shard-snb: [FAIL][81] ([i915#2546]) -> [PASS][82]
[81]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-snb7/igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-render.html
[82]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-snb2/igt@kms_frontbuffer_tracking@fbc-rgb101010-draw-render.html
* igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt:
- shard-iclb: [DMESG-WARN][83] ([i915#1982]) -> [PASS][84]
[83]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt.html
[84]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb5/igt@kms_frontbuffer_tracking@psr-1p-offscren-pri-indfb-draw-blt.html
* igt@kms_psr@psr2_sprite_plane_move:
- shard-iclb: [SKIP][85] ([fdo#109441]) -> [PASS][86] +1 similar issue
[85]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb3/igt@kms_psr@psr2_sprite_plane_move.html
[86]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb2/igt@kms_psr@psr2_sprite_plane_move.html
* igt@kms_psr@suspend:
- shard-iclb: [INCOMPLETE][87] ([i915#1185]) -> [PASS][88]
[87]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@kms_psr@suspend.html
[88]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb4/igt@kms_psr@suspend.html
* igt@kms_setmode@basic:
- shard-tglb: [FAIL][89] ([i915#31]) -> [PASS][90]
[89]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb7/igt@kms_setmode@basic.html
[90]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb5/igt@kms_setmode@basic.html
- shard-glk: [FAIL][91] ([i915#31]) -> [PASS][92]
[91]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk4/igt@kms_setmode@basic.html
[92]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk8/igt@kms_setmode@basic.html
- shard-hsw: [FAIL][93] ([i915#31]) -> [PASS][94]
[93]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw6/igt@kms_setmode@basic.html
[94]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@kms_setmode@basic.html
- shard-kbl: [FAIL][95] ([i915#31]) -> [PASS][96]
[95]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl4/igt@kms_setmode@basic.html
[96]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl6/igt@kms_setmode@basic.html
- shard-iclb: [FAIL][97] ([i915#31]) -> [PASS][98]
[97]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb4/igt@kms_setmode@basic.html
[98]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb3/igt@kms_setmode@basic.html
* igt@kms_sysfs_edid_timing:
- shard-hsw: [FAIL][99] ([IGT#2]) -> [PASS][100]
[99]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw1/igt@kms_sysfs_edid_timing.html
[100]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw1/igt@kms_sysfs_edid_timing.html
* igt@kms_vblank@pipe-c-accuracy-idle:
- shard-apl: [DMESG-WARN][101] ([i915#1635] / [i915#1982]) -> [PASS][102] +5 similar issues
[101]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl8/igt@kms_vblank@pipe-c-accuracy-idle.html
[102]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl2/igt@kms_vblank@pipe-c-accuracy-idle.html
* igt@perf@non-system-wide-paranoid:
- shard-apl: [SKIP][103] ([fdo#109271] / [i915#1354] / [i915#1635]) -> [PASS][104]
[103]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-apl8/igt@perf@non-system-wide-paranoid.html
[104]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-apl6/igt@perf@non-system-wide-paranoid.html
- shard-glk: [SKIP][105] ([fdo#109271] / [i915#1354]) -> [PASS][106]
[105]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-glk4/igt@perf@non-system-wide-paranoid.html
[106]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-glk2/igt@perf@non-system-wide-paranoid.html
- shard-iclb: [SKIP][107] ([i915#1354]) -> [PASS][108]
[107]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@perf@non-system-wide-paranoid.html
[108]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb4/igt@perf@non-system-wide-paranoid.html
- shard-kbl: [SKIP][109] ([fdo#109271] / [i915#1354]) -> [PASS][110]
[109]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-kbl6/igt@perf@non-system-wide-paranoid.html
[110]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-kbl7/igt@perf@non-system-wide-paranoid.html
- shard-tglb: [SKIP][111] ([i915#1354]) -> [PASS][112]
[111]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb1/igt@perf@non-system-wide-paranoid.html
[112]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb3/igt@perf@non-system-wide-paranoid.html
* igt@perf@polling-parameterized:
- shard-hsw: [FAIL][113] ([i915#1542]) -> [PASS][114]
[113]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-hsw6/igt@perf@polling-parameterized.html
[114]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-hsw6/igt@perf@polling-parameterized.html
#### Warnings ####
* igt@i915_pm_rpm@dpms-mode-unset-non-lpsp:
- shard-iclb: [SKIP][115] ([i915#579]) -> [SKIP][116] ([fdo#110892])
[115]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@i915_pm_rpm@dpms-mode-unset-non-lpsp.html
[116]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb1/igt@i915_pm_rpm@dpms-mode-unset-non-lpsp.html
* igt@i915_pm_rpm@pm-tiling:
- shard-tglb: [SKIP][117] ([i915#579]) -> [DMESG-WARN][118] ([i915#2411])
[117]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb8/igt@i915_pm_rpm@pm-tiling.html
[118]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb5/igt@i915_pm_rpm@pm-tiling.html
* igt@i915_pm_rpm@sysfs-read:
- shard-tglb: [DMESG-WARN][119] ([i915#2411]) -> [SKIP][120] ([i915#579])
[119]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb6/igt@i915_pm_rpm@sysfs-read.html
[120]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb3/igt@i915_pm_rpm@sysfs-read.html
* igt@kms_vblank@pipe-d-ts-continuation-dpms-suspend:
- shard-tglb: [DMESG-WARN][121] ([i915#2411]) -> [INCOMPLETE][122] ([i915#1436] / [i915#1887] / [i915#1982])
[121]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb3/igt@kms_vblank@pipe-d-ts-continuation-dpms-suspend.html
[122]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb3/igt@kms_vblank@pipe-d-ts-continuation-dpms-suspend.html
* igt@runner@aborted:
- shard-iclb: [FAIL][123] ([i915#1814]) -> [FAIL][124] ([i915#1814] / [i915#2439])
[123]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-iclb6/igt@runner@aborted.html
[124]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-iclb4/igt@runner@aborted.html
- shard-tglb: [FAIL][125] ([i915#2439]) -> [FAIL][126] ([i915#1602])
[125]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_9250/shard-tglb8/igt@runner@aborted.html
[126]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/shard-tglb3/igt@runner@aborted.html
{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).
[IGT#2]: https://gitlab.freedesktop.org/drm/igt-gpu-tools/issues/2
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
[fdo#110892]: https://bugs.freedesktop.org/show_bug.cgi?id=110892
[i915#118]: https://gitlab.freedesktop.org/drm/intel/issues/118
[i915#1185]: https://gitlab.freedesktop.org/drm/intel/issues/1185
[i915#1354]: https://gitlab.freedesktop.org/drm/intel/issues/1354
[i915#1436]: https://gitlab.freedesktop.org/drm/intel/issues/1436
[i915#1542]: https://gitlab.freedesktop.org/drm/intel/issues/1542
[i915#155]: https://gitlab.freedesktop.org/drm/intel/issues/155
[i915#1602]: https://gitlab.freedesktop.org/drm/intel/issues/1602
[i915#1635]: https://gitlab.freedesktop.org/drm/intel/issues/1635
[i915#180]: https://gitlab.freedesktop.org/drm/intel/issues/180
[i915#1814]: https://gitlab.freedesktop.org/drm/intel/issues/1814
[i915#1887]: https://gitlab.freedesktop.org/drm/intel/issues/1887
[i915#1888]: https://gitlab.freedesktop.org/drm/intel/issues/1888
[i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
[i915#2122]: https://gitlab.freedesktop.org/drm/intel/issues/2122
[i915#2389]: https://gitlab.freedesktop.org/drm/intel/issues/2389
[i915#2411]: https://gitlab.freedesktop.org/drm/intel/issues/2411
[i915#2439]: https://gitlab.freedesktop.org/drm/intel/issues/2439
[i915#2521]: https://gitlab.freedesktop.org/drm/intel/issues/2521
[i915#2546]: https://gitlab.freedesktop.org/drm/intel/issues/2546
[i915#2598]: https://gitlab.freedesktop.org/drm/intel/issues/2598
[i915#31]: https://gitlab.freedesktop.org/drm/intel/issues/31
[i915#579]: https://gitlab.freedesktop.org/drm/intel/issues/579
[i915#82]: https://gitlab.freedesktop.org/drm/intel/issues/82
[i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95
Participating hosts (11 -> 8)
------------------------------
Missing (3): pig-skl-6260u pig-glk-j5005 pig-icl-1065g7
Build changes
-------------
* CI: CI-20190529 -> None
* IGT: IGT_5830 -> IGTPW_5120
* Piglit: piglit_4509 -> None
CI-20190529: 20190529
CI_DRM_9250: 0ddbbf768a932ea9ca82a5872fd83977e27347c0 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_5120: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
IGT_5830: 12d370cb57e0cfcb781c87ad9e15e68b17a1f41f @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_5120/index.html
[-- Attachment #1.2: Type: text/html, Size: 30377 bytes --]
[-- Attachment #2: Type: text/plain, Size: 154 bytes --]
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
2020-11-20 15:05 ` Chris Wilson
@ 2020-11-23 13:57 ` Ram Moon, AnandX
0 siblings, 0 replies; 9+ messages in thread
From: Ram Moon, AnandX @ 2020-11-23 13:57 UTC (permalink / raw)
To: Chris Wilson, igt-dev
Hi Chris,
-----Original Message-----
From: Chris Wilson <chris@chris-wilson.co.uk>
Sent: Friday, November 20, 2020 8:36 PM
To: Ram Moon, AnandX <anandx.ram.moon@intel.com>; igt-dev@lists.freedesktop.org
Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
Quoting Ram Moon, AnandX (2020-11-20 09:12:40)
> Hi Chris,
>
> -----Original Message-----
> From: Ram Moon, AnandX
> Sent: Tuesday, November 3, 2020 11:28 PM
> To: Chris Wilson <chris@chris-wilson.co.uk>;
> igt-dev@lists.freedesktop.org
> Subject: RE: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve
> accuracy with using of confidence interval
>
> Hi Chris,
>
> Thanks for your review comment.
>
> -----Original Message-----
> From: Chris Wilson <chris@chris-wilson.co.uk>
> Sent: Tuesday, November 3, 2020 4:19 PM
> To: Ram Moon, AnandX <anandx.ram.moon@intel.com>;
> igt-dev@lists.freedesktop.org
> Cc: Ram Moon, AnandX <anandx.ram.moon@intel.com>
> Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve
> accuracy with using of confidence interval
>
> Quoting Anand Moon (2020-11-02 09:08:36)
> > Using confidence interval formula accuracy is estimates
> > approximately at 95% when {X} is the average of a sample of size n.
> > Use this confidence interval formula as used in statistics to
> > estimate the accuracy.
> >
> > Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
> > ---
> > See below link.
> > https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumula
> > ti
> > ve_distribution_function
> > ---
> > tests/kms_setmode.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> > 92f3ead2..58814a13 100644
> > --- a/tests/kms_setmode.c
> > +++ b/tests/kms_setmode.c
> > @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
> > * which depend on timing, giving the lowest acceptable MTBF of 5.6s
> > * for 60Hz sampling rate.
> > */
> > - accuracy = 3. * stddev;
> > + accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
>
> Oops, yes, this should be standard error rather than deviation. Could you add igt_stats_get_std_error() so we don't hardcode N?
> -Chris
>
> Do you want me to rename accuracy to std_error and use stats.n_values instead of CALIBRATE_TS_STEPS I could not find the definition of igt_stats_get_std_error function.
>
> -Anand
>
> Thanks for fixing the code changes.
>
> Actually above changes fix the issue to improve the accuracy calculations.
> But now I am able to reproduce the original bug using 4K monitors on 5.8.x kernel, See below logs.
>
> CRTC[98] [Pipe A] Mode: 3840x2160@60Hz Connectors: DP-2[330]
> Expected frametime: 16668us; measured 16672.3us +- 1.805us accuracy
> 0.00% [0.07 scanlines]
> (kms_setmode:14315) CRITICAL: Test assertion failure function check_timings, file ../tests/kms_setmode.c:544:
> (kms_setmode:14315) CRITICAL: Failed assertion: fabs(mean - expected)
> < 1.718 * stddev
> (kms_setmode:14315) CRITICAL: vblank interval differs from modeline! expected 16667.6us, measured 16672us +- 1.805us, difference 4.7us (2.6 sigma) Stack trace:
> #0 ../lib/igt_core.c:1734 __igt_fail_assert()
> #1 ../tests/kms_setmode.c:544 test_one_combination()
> #2 ../tests/kms_setmode.c:834 __real_main883()
> #3 ../tests/kms_setmode.c:883 main()
> #4 ../csu/libc-start.c:342 __libc_start_main()
> #5 [_start+0x2e]
> Subtest basic failed.
> **** DEBUG ****
>
> So after in digging into the below comments it turns out that
> *confidence interval* with respect to standard deviation is calculated
> with *Cumulative distribution function* (formula)
>
> As per the Wikipedia table
> For various values of z, the percentage of values expected to lie in and outside the symmetric interval, CI = (−zσ, zσ), are as follows:
> [0]
> https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_di
> stributed_data
>
> It turns out that *3 σ* gives around proportion percentage 99.7300204% which is according to the CI = (−zσ, zσ) table.
>
> I would like to submit a below patch as this changes fix the overall bug.
> Please share your thought and suggestion on any alternative approach I should try.
You would also have to update the comments as it no longer matches the broaden 90% critical value.
Still debating what is a sensible error rate, but in the meantime I would suggest a different approach: ignore the deviation if it is within a single scanline.
Something like:
- igt_assert_f(fabs(mean - expected) < 1.718 * stddev,
- "vblank interval differs from modeline! expected %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma)\n",
+ igt_assert_f(fabs(mean - expected) < max(line_time(kmode), 1.718 * stddev),
+ "vblank interval differs from modeline! expected
+ %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma, %.1f
+ scanlines)\n",
expected, mean, stddev,
- fabs(mean - expected), fabs(mean - expected) / stddev);
+ fabs(mean - expected),
+ fabs(mean - expected) / stddev,
+ fabs(mean - expected) / line_time(kmode));
Thanks for this new approach, I have tested this patch on different machines
with different modes as below.
Mode: 1920x1080@60Hz Connectors: HDMI-A-2
Mode: 1920x1080@60Hz Connectors: DP-1
Mode: 1920x1080@60Hz Connectors: DP-2
Mode: 2560x1440@60Hz Connectors: eDP-1
Mode: 3840x2160@60Hz Connectors: DP-2
And all the testcase regarding kms_setmode seems to passing on 5.8.x and 5.10-0-rcx kernel.
Thanks
-Anand
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
2020-11-20 9:12 ` Ram Moon, AnandX
@ 2020-11-20 15:05 ` Chris Wilson
2020-11-23 13:57 ` Ram Moon, AnandX
0 siblings, 1 reply; 9+ messages in thread
From: Chris Wilson @ 2020-11-20 15:05 UTC (permalink / raw)
To: Ram Moon, AnandX, igt-dev
Quoting Ram Moon, AnandX (2020-11-20 09:12:40)
> Hi Chris,
>
> -----Original Message-----
> From: Ram Moon, AnandX
> Sent: Tuesday, November 3, 2020 11:28 PM
> To: Chris Wilson <chris@chris-wilson.co.uk>; igt-dev@lists.freedesktop.org
> Subject: RE: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
>
> Hi Chris,
>
> Thanks for your review comment.
>
> -----Original Message-----
> From: Chris Wilson <chris@chris-wilson.co.uk>
> Sent: Tuesday, November 3, 2020 4:19 PM
> To: Ram Moon, AnandX <anandx.ram.moon@intel.com>; igt-dev@lists.freedesktop.org
> Cc: Ram Moon, AnandX <anandx.ram.moon@intel.com>
> Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
>
> Quoting Anand Moon (2020-11-02 09:08:36)
> > Using confidence interval formula accuracy is estimates approximately
> > at 95% when {X} is the average of a sample of size n.
> > Use this confidence interval formula as used in statistics to estimate
> > the accuracy.
> >
> > Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
> > ---
> > See below link.
> > https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulati
> > ve_distribution_function
> > ---
> > tests/kms_setmode.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> > 92f3ead2..58814a13 100644
> > --- a/tests/kms_setmode.c
> > +++ b/tests/kms_setmode.c
> > @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
> > * which depend on timing, giving the lowest acceptable MTBF of 5.6s
> > * for 60Hz sampling rate.
> > */
> > - accuracy = 3. * stddev;
> > + accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
>
> Oops, yes, this should be standard error rather than deviation. Could you add igt_stats_get_std_error() so we don't hardcode N?
> -Chris
>
> Do you want me to rename accuracy to std_error and use stats.n_values instead of CALIBRATE_TS_STEPS I could not find the definition of igt_stats_get_std_error function.
>
> -Anand
>
> Thanks for fixing the code changes.
>
> Actually above changes fix the issue to improve the accuracy calculations.
> But now I am able to reproduce the original bug using 4K monitors on 5.8.x kernel, See below logs.
>
> CRTC[98] [Pipe A] Mode: 3840x2160@60Hz Connectors: DP-2[330] Expected frametime: 16668us; measured 16672.3us +- 1.805us accuracy 0.00% [0.07 scanlines]
> (kms_setmode:14315) CRITICAL: Test assertion failure function check_timings, file ../tests/kms_setmode.c:544:
> (kms_setmode:14315) CRITICAL: Failed assertion: fabs(mean - expected) < 1.718 * stddev
> (kms_setmode:14315) CRITICAL: vblank interval differs from modeline! expected 16667.6us, measured 16672us +- 1.805us, difference 4.7us (2.6 sigma) Stack trace:
> #0 ../lib/igt_core.c:1734 __igt_fail_assert()
> #1 ../tests/kms_setmode.c:544 test_one_combination()
> #2 ../tests/kms_setmode.c:834 __real_main883()
> #3 ../tests/kms_setmode.c:883 main()
> #4 ../csu/libc-start.c:342 __libc_start_main()
> #5 [_start+0x2e]
> Subtest basic failed.
> **** DEBUG ****
>
> So after in digging into the below comments it turns out that *confidence interval* with respect to standard deviation is calculated with *Cumulative distribution function* (formula)
>
> As per the Wikipedia table
> For various values of z, the percentage of values expected to lie in and outside the symmetric interval, CI = (−zσ, zσ), are as follows:
> [0] https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data
>
> It turns out that *3 σ* gives around proportion percentage 99.7300204% which is according to the CI = (−zσ, zσ) table.
>
> I would like to submit a below patch as this changes fix the overall bug.
> Please share your thought and suggestion on any alternative approach I should try.
You would also have to update the comments as it no longer matches the
broaden 90% critical value.
Still debating what is a sensible error rate, but in the meantime I
would suggest a different approach: ignore the deviation if it is within
a single scanline.
Something like:
- igt_assert_f(fabs(mean - expected) < 1.718 * stddev,
- "vblank interval differs from modeline! expected %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma)\n",
+ igt_assert_f(fabs(mean - expected) < max(line_time(kmode), 1.718 * stddev),
+ "vblank interval differs from modeline! expected %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma, %.1f scanlines)\n",
expected, mean, stddev,
- fabs(mean - expected), fabs(mean - expected) / stddev);
+ fabs(mean - expected),
+ fabs(mean - expected) / stddev,
+ fabs(mean - expected) / line_time(kmode));
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
2020-11-03 17:58 ` Ram Moon, AnandX
@ 2020-11-20 9:12 ` Ram Moon, AnandX
2020-11-20 15:05 ` Chris Wilson
0 siblings, 1 reply; 9+ messages in thread
From: Ram Moon, AnandX @ 2020-11-20 9:12 UTC (permalink / raw)
To: Chris Wilson, igt-dev
Hi Chris,
-----Original Message-----
From: Ram Moon, AnandX
Sent: Tuesday, November 3, 2020 11:28 PM
To: Chris Wilson <chris@chris-wilson.co.uk>; igt-dev@lists.freedesktop.org
Subject: RE: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
Hi Chris,
Thanks for your review comment.
-----Original Message-----
From: Chris Wilson <chris@chris-wilson.co.uk>
Sent: Tuesday, November 3, 2020 4:19 PM
To: Ram Moon, AnandX <anandx.ram.moon@intel.com>; igt-dev@lists.freedesktop.org
Cc: Ram Moon, AnandX <anandx.ram.moon@intel.com>
Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
Quoting Anand Moon (2020-11-02 09:08:36)
> Using confidence interval formula accuracy is estimates approximately
> at 95% when {X} is the average of a sample of size n.
> Use this confidence interval formula as used in statistics to estimate
> the accuracy.
>
> Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
> ---
> See below link.
> https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulati
> ve_distribution_function
> ---
> tests/kms_setmode.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> 92f3ead2..58814a13 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
> * which depend on timing, giving the lowest acceptable MTBF of 5.6s
> * for 60Hz sampling rate.
> */
> - accuracy = 3. * stddev;
> + accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
Oops, yes, this should be standard error rather than deviation. Could you add igt_stats_get_std_error() so we don't hardcode N?
-Chris
Do you want me to rename accuracy to std_error and use stats.n_values instead of CALIBRATE_TS_STEPS I could not find the definition of igt_stats_get_std_error function.
-Anand
Thanks for fixing the code changes.
Actually above changes fix the issue to improve the accuracy calculations.
But now I am able to reproduce the original bug using 4K monitors on 5.8.x kernel, See below logs.
CRTC[98] [Pipe A] Mode: 3840x2160@60Hz Connectors: DP-2[330] Expected frametime: 16668us; measured 16672.3us +- 1.805us accuracy 0.00% [0.07 scanlines]
(kms_setmode:14315) CRITICAL: Test assertion failure function check_timings, file ../tests/kms_setmode.c:544:
(kms_setmode:14315) CRITICAL: Failed assertion: fabs(mean - expected) < 1.718 * stddev
(kms_setmode:14315) CRITICAL: vblank interval differs from modeline! expected 16667.6us, measured 16672us +- 1.805us, difference 4.7us (2.6 sigma) Stack trace:
#0 ../lib/igt_core.c:1734 __igt_fail_assert()
#1 ../tests/kms_setmode.c:544 test_one_combination()
#2 ../tests/kms_setmode.c:834 __real_main883()
#3 ../tests/kms_setmode.c:883 main()
#4 ../csu/libc-start.c:342 __libc_start_main()
#5 [_start+0x2e]
Subtest basic failed.
**** DEBUG ****
So after in digging into the below comments it turns out that *confidence interval* with respect to standard deviation is calculated with *Cumulative distribution function* (formula)
As per the Wikipedia table
For various values of z, the percentage of values expected to lie in and outside the symmetric interval, CI = (−zσ, zσ), are as follows:
[0] https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data
It turns out that *3 σ* gives around proportion percentage 99.7300204% which is according to the CI = (−zσ, zσ) table.
I would like to submit a below patch as this changes fix the overall bug.
Please share your thought and suggestion on any alternative approach I should try.
Thanks
Anand
naveen@naveen-GB-Z390-Garuda:~/anand/igt-gpu-tools.internal-rebasing$ git diff diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index 928a5a8cf..58da36985 100644
--- a/tests/kms_setmode.c
+++ b/tests/kms_setmode.c
@@ -526,7 +526,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
* See:
* https://en.wikipedia.org/wiki/Standard_deviation#Rules_for_normally_distributed_data
*/
- igt_assert_f(fabs(mean - expected) < 1.718 * stddev,
+ igt_assert_f(fabs(mean - expected) < 3. * stddev,
"vblank interval differs from modeline! expected %.1fus, measured %1.fus +- %.3fus, difference %.1fus (%.1f sigma)\n",
expected, mean, stddev,
fabs(mean - expected), fabs(mean - expected) / stddev);
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
2020-11-03 10:49 ` Chris Wilson
@ 2020-11-03 17:58 ` Ram Moon, AnandX
2020-11-20 9:12 ` Ram Moon, AnandX
0 siblings, 1 reply; 9+ messages in thread
From: Ram Moon, AnandX @ 2020-11-03 17:58 UTC (permalink / raw)
To: Chris Wilson, igt-dev
Hi Chris,
Thanks for your review comment.
-----Original Message-----
From: Chris Wilson <chris@chris-wilson.co.uk>
Sent: Tuesday, November 3, 2020 4:19 PM
To: Ram Moon, AnandX <anandx.ram.moon@intel.com>; igt-dev@lists.freedesktop.org
Cc: Ram Moon, AnandX <anandx.ram.moon@intel.com>
Subject: Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
Quoting Anand Moon (2020-11-02 09:08:36)
> Using confidence interval formula accuracy is estimates approximately
> at 95% when {X} is the average of a sample of size n.
> Use this confidence interval formula as used in statistics to estimate
> the accuracy.
>
> Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
> ---
> See below link.
> https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulati
> ve_distribution_function
> ---
> tests/kms_setmode.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c index
> 92f3ead2..58814a13 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
> * which depend on timing, giving the lowest acceptable MTBF of 5.6s
> * for 60Hz sampling rate.
> */
> - accuracy = 3. * stddev;
> + accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
Oops, yes, this should be standard error rather than deviation. Could you add igt_stats_get_std_error() so we don't hardcode N?
-Chris
Do you want me to rename accuracy to std_error and use stats.n_values instead of CALIBRATE_TS_STEPS
I could not find the definition of igt_stats_get_std_error function.
-Anand
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
2020-11-02 9:08 [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval Anand Moon
@ 2020-11-03 10:49 ` Chris Wilson
2020-11-03 17:58 ` Ram Moon, AnandX
0 siblings, 1 reply; 9+ messages in thread
From: Chris Wilson @ 2020-11-03 10:49 UTC (permalink / raw)
To: Anand Moon, igt-dev; +Cc: Anand Moon
Quoting Anand Moon (2020-11-02 09:08:36)
> Using confidence interval formula accuracy is estimates approximately
> at 95% when {X} is the average of a sample of size n.
> Use this confidence interval formula as used in statistics
> to estimate the accuracy.
>
> Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
> ---
> See below link.
> https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulative_distribution_function
> ---
> tests/kms_setmode.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c
> index 92f3ead2..58814a13 100644
> --- a/tests/kms_setmode.c
> +++ b/tests/kms_setmode.c
> @@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
> * which depend on timing, giving the lowest acceptable MTBF of 5.6s
> * for 60Hz sampling rate.
> */
> - accuracy = 3. * stddev;
> + accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
Oops, yes, this should be standard error rather than deviation. Could you
add igt_stats_get_std_error() so we don't hardcode N?
-Chris
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval
@ 2020-11-02 9:08 Anand Moon
2020-11-03 10:49 ` Chris Wilson
0 siblings, 1 reply; 9+ messages in thread
From: Anand Moon @ 2020-11-02 9:08 UTC (permalink / raw)
To: igt-dev; +Cc: Anand Moon
Using confidence interval formula accuracy is estimates approximately
at 95% when {X} is the average of a sample of size n.
Use this confidence interval formula as used in statistics
to estimate the accuracy.
Signed-off-by: Anand Moon <anandx.ram.moon@intel.com>
---
See below link.
https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule#Cumulative_distribution_function
---
tests/kms_setmode.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/kms_setmode.c b/tests/kms_setmode.c
index 92f3ead2..58814a13 100644
--- a/tests/kms_setmode.c
+++ b/tests/kms_setmode.c
@@ -494,7 +494,7 @@ static void check_timings(int crtc_idx, const drmModeModeInfo *kmode)
* which depend on timing, giving the lowest acceptable MTBF of 5.6s
* for 60Hz sampling rate.
*/
- accuracy = 3. * stddev;
+ accuracy = 3. * stddev / sqrt(CALIBRATE_TS_STEPS);
igt_info("Expected frametime: %.0fus; measured %.1fus +- %.3fus accuracy %.2f%% [%.2f scanlines]\n",
expected, mean, stddev,
--
2.25.1
_______________________________________________
igt-dev mailing list
igt-dev@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/igt-dev
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-11-23 13:57 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-02 9:02 [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval Anand Moon
2020-11-03 10:27 ` [igt-dev] ✓ Fi.CI.BAT: success for tests/kms_setmode: basic Improve accuracy with using of confidence interval (rev2) Patchwork
2020-11-03 16:04 ` [igt-dev] ✗ Fi.CI.IGT: failure " Patchwork
2020-11-02 9:08 [igt-dev] [i-g-t] tests/kms_setmode: basic Improve accuracy with using of confidence interval Anand Moon
2020-11-03 10:49 ` Chris Wilson
2020-11-03 17:58 ` Ram Moon, AnandX
2020-11-20 9:12 ` Ram Moon, AnandX
2020-11-20 15:05 ` Chris Wilson
2020-11-23 13:57 ` Ram Moon, AnandX
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.