[PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Imre Deak]

The CI software watchdog (owatch) will timeout if the test doesn't
output anything for a long time on standard out or error. At least the
plane-all-modeset-transition and plane-all-modeset-transition-fences
subtests run without any output longer than the watchdog timeout, so
output some more progress info.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103334
Signed-off-by: Imre Deak <imre.deak@intel.com>
---
 tests/kms_atomic_transition.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tests/kms_atomic_transition.c b/tests/kms_atomic_transition.c
index 2ae75f2d..db43d2c9 100644
--- a/tests/kms_atomic_transition.c
+++ b/tests/kms_atomic_transition.c
@@ -525,6 +525,9 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 	}
 
 	for (i = 0; i < iter_max; i++) {
+		igt_info(".");
+		fflush(stdout);
+
 		igt_output_set_pipe(output, pipe);
 
 		wm_setup_plane(display, pipe, i, parms, fencing);
@@ -544,6 +547,9 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 
 			/* i -> i+1 will be done when i increases, can be skipped here */
 			for (j = iter_max - 1; j > i + 1; j--) {
+				igt_info(".");
+				fflush(stdout);
+
 				wm_setup_plane(display, pipe, j, parms, fencing);
 
 				if (type == TRANSITION_MODESET)
@@ -562,6 +568,8 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 		}
 	}
 
+	igt_info("\n");
+
 cleanup:
 	if (fencing)
 		unprepare_fencing(display, pipe);
@@ -780,6 +788,9 @@ static void run_modeset_tests(igt_display_t *display, int howmany, bool nonblock
 			if (!event_mask)
 				continue;
 
+			igt_info(".");
+			fflush(stdout);
+
 			commit_display(display, event_mask, nonblocking);
 
 			collect_crcs_mask(pipe_crcs, j, crcs[1]);
@@ -814,6 +825,8 @@ static void run_modeset_tests(igt_display_t *display, int howmany, bool nonblock
 		}
 	}
 
+	igt_info("\n");
+
 cleanup:
 	set_combinations(display, 0, NULL);
 	igt_display_commit2(display, COMMIT_ATOMIC);
-- 
2.11.0

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Chris Wilson]

Quoting Imre Deak (2017-10-18 12:29:33)
> The CI software watchdog (owatch) will timeout if the test doesn't
> output anything for a long time on standard out or error. At least the
> plane-all-modeset-transition and plane-all-modeset-transition-fences
> subtests run without any output longer than the watchdog timeout, so
> output some more progress info.

Then why are we using owatch? We have a test suite that is quite capable
of supporting /dev/watchdog and providing a ping function.
-Chris
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Petri Latvala]

On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> The CI software watchdog (owatch) will timeout if the test doesn't
> output anything for a long time on standard out or error. At least the
> plane-all-modeset-transition and plane-all-modeset-transition-fences
> subtests run without any output longer than the watchdog timeout, so
> output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_
prints anything. Which it does on start/exit of a test.


-- 
Petri Latvala
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Imre Deak]

On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
> On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> > The CI software watchdog (owatch) will timeout if the test doesn't
> > output anything for a long time on standard out or error. At least the
> > plane-all-modeset-transition and plane-all-modeset-transition-fences
> > subtests run without any output longer than the watchdog timeout, so
> > output some more progress info.
> 
> No, owatch is wrapping piglit, and pings the watchdog if _piglit_
> prints anything. Which it does on start/exit of a test.

Yes, but the subtest itself takes more time than owatch's timeout,
during which there is no output. So owatch will trigger a reboot before
the subtest ends, even though it's making progress.

--Imre

> 
> 
> -- 
> Petri Latvala
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Imre Deak]

On Wed, Oct 18, 2017 at 02:47:17PM +0300, Imre Deak wrote:
> On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
> > On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> > > The CI software watchdog (owatch) will timeout if the test doesn't
> > > output anything for a long time on standard out or error. At least the
> > > plane-all-modeset-transition and plane-all-modeset-transition-fences
> > > subtests run without any output longer than the watchdog timeout, so
> > > output some more progress info.
> > 
> > No, owatch is wrapping piglit, and pings the watchdog if _piglit_
> > prints anything. Which it does on start/exit of a test.
> 
> Yes, but the subtest itself takes more time than owatch's timeout,
> during which there is no output. So owatch will trigger a reboot before
> the subtest ends, even though it's making progress.

Or you mean piglit doesn't forward the test's stdout, stderr? In that
case this won't work.

--Imre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.BAT: success for kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Patchwork]

== Series Details ==

Series: kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout
URL   : https://patchwork.freedesktop.org/series/32206/
State : success

== Summary ==

IGT patchset tested on top of latest successful build
9a3f6f59d4c208219f7d3178106bdef628ff68d0 tests/intel-ci: Remove fast-feedback-simulation.testlist

with latest DRM-Tip kernel build CI_DRM_3258
8b18ad103908 drm-tip: 2017y-10m-18d-08h-27m-31s UTC integration manifest

No testlist changes.

Test chamelium:
        Subgroup dp-crc-fast:
                fail       -> PASS       (fi-kbl-7500u) fdo#102514

fdo#102514 https://bugs.freedesktop.org/show_bug.cgi?id=102514

fi-bdw-5557u     total:289  pass:268  dwarn:0   dfail:0   fail:0   skip:21  time:442s
fi-blb-e6850     total:289  pass:223  dwarn:1   dfail:0   fail:0   skip:65  time:371s
fi-bsw-n3050     total:289  pass:243  dwarn:0   dfail:0   fail:0   skip:46  time:531s
fi-bwr-2160      total:289  pass:183  dwarn:0   dfail:0   fail:0   skip:106 time:270s
fi-bxt-dsi       total:289  pass:259  dwarn:0   dfail:0   fail:0   skip:30  time:504s
fi-bxt-j4205     total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:498s
fi-byt-j1900     total:289  pass:253  dwarn:1   dfail:0   fail:0   skip:35  time:499s
fi-byt-n2820     total:289  pass:249  dwarn:1   dfail:0   fail:0   skip:39  time:483s
fi-elk-e7500     total:289  pass:229  dwarn:0   dfail:0   fail:0   skip:60  time:424s
fi-gdg-551       total:289  pass:178  dwarn:1   dfail:0   fail:1   skip:109 time:252s
fi-glk-1         total:289  pass:261  dwarn:0   dfail:0   fail:0   skip:28  time:582s
fi-hsw-4770      total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:445s
fi-hsw-4770r     total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:429s
fi-ilk-650       total:289  pass:228  dwarn:0   dfail:0   fail:0   skip:61  time:438s
fi-ivb-3520m     total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:487s
fi-ivb-3770      total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:463s
fi-kbl-7500u     total:289  pass:264  dwarn:1   dfail:0   fail:0   skip:24  time:491s
fi-kbl-7560u     total:289  pass:270  dwarn:0   dfail:0   fail:0   skip:19  time:577s
fi-kbl-7567u     total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:477s
fi-kbl-r         total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:584s
fi-pnv-d510      total:289  pass:222  dwarn:1   dfail:0   fail:0   skip:66  time:551s
fi-skl-6260u     total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:451s
fi-skl-6700hq    total:289  pass:263  dwarn:0   dfail:0   fail:0   skip:26  time:643s
fi-skl-6700k     total:289  pass:265  dwarn:0   dfail:0   fail:0   skip:24  time:515s
fi-skl-6770hq    total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:504s
fi-skl-gvtdvm    total:289  pass:266  dwarn:0   dfail:0   fail:0   skip:23  time:459s
fi-snb-2520m     total:289  pass:250  dwarn:0   dfail:0   fail:0   skip:39  time:571s
fi-snb-2600      total:289  pass:249  dwarn:0   dfail:0   fail:0   skip:40  time:427s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_375/
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Petri Latvala]

On 10/18/2017 02:50 PM, Imre Deak wrote:
> On Wed, Oct 18, 2017 at 02:47:17PM +0300, Imre Deak wrote:
>> On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
>>> On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
>>>> The CI software watchdog (owatch) will timeout if the test doesn't
>>>> output anything for a long time on standard out or error. At least the
>>>> plane-all-modeset-transition and plane-all-modeset-transition-fences
>>>> subtests run without any output longer than the watchdog timeout, so
>>>> output some more progress info.
>>> No, owatch is wrapping piglit, and pings the watchdog if _piglit_
>>> prints anything. Which it does on start/exit of a test.
>> Yes, but the subtest itself takes more time than owatch's timeout,
>> during which there is no output. So owatch will trigger a reboot before
>> the subtest ends, even though it's making progress.
> Or you mean piglit doesn't forward the test's stdout, stderr? In that
> case this won't work.

Yeah, sorry I was too terse. Piglit redirects the test output (out and 
err) to its own files. Printing from the test does not trigger owatch's 
heartbeat.

-- 
Petri Latvala

_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Daniel Vetter]

On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
> On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> > The CI software watchdog (owatch) will timeout if the test doesn't
> > output anything for a long time on standard out or error. At least the
> > plane-all-modeset-transition and plane-all-modeset-transition-fences
> > subtests run without any output longer than the watchdog timeout, so
> > output some more progress info.
> 
> No, owatch is wrapping piglit, and pings the watchdog if _piglit_
> prints anything. Which it does on start/exit of a test.

tbh this sounds like owatch being dense and it shouldn't try to reboot
this quickly. What's the current owatch timeout?

Aside: What exactly does owatch give us? I thought jenkins also watches
machines and reboots them using the ac switch ... And owatch provides
spurious reboots?
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

✓ Fi.CI.IGT: success for kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Patchwork]

== Series Details ==

Series: kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout
URL   : https://patchwork.freedesktop.org/series/32206/
State : success

== Summary ==

Test kms_setmode:
        Subgroup basic:
                pass       -> FAIL       (shard-hsw) fdo#99912
Test perf:
        Subgroup polling:
                fail       -> PASS       (shard-hsw) fdo#102252

fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912
fdo#102252 https://bugs.freedesktop.org/show_bug.cgi?id=102252

shard-hsw        total:2540 pass:1430 dwarn:0   dfail:0   fail:9   skip:1101 time:9226s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_375/shards.html
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Lofstedt, Marta]

> -----Original Message-----
> From: Intel-gfx [mailto:intel-gfx-bounces@lists.freedesktop.org] On Behalf
> Of Daniel Vetter
> Sent: Wednesday, October 18, 2017 5:36 PM
> To: Latvala, Petri <petri.latvala@intel.com>
> Cc: intel-gfx@lists.freedesktop.org
> Subject: Re: [Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Output more
> finegrained progress info to avoid CI watchdog timeout
> 
> On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
> > On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> > > The CI software watchdog (owatch) will timeout if the test doesn't
> > > output anything for a long time on standard out or error. At least
> > > the plane-all-modeset-transition and
> > > plane-all-modeset-transition-fences
> > > subtests run without any output longer than the watchdog timeout, so
> > > output some more progress info.
> >
> > No, owatch is wrapping piglit, and pings the watchdog if _piglit_
> > prints anything. Which it does on start/exit of a test.
> 
> tbh this sounds like owatch being dense and it shouldn't try to reboot this
> quickly. What's the current owatch timeout?
> 
> Aside: What exactly does owatch give us? I thought jenkins also watches
> machines and reboots them using the ac switch ... And owatch provides
> spurious reboots?
Daniel,
Owatch gives us the knowledge that is was a test that took too long. I.e. we will know that it was not a system hang. 
 We also know that the NMI watchdog didn't trigger. 
I believe this is extremely useful information when you are starting to debug the issue

Imre if you believe that owatch is preventing you from getting information to debug why these test are taking so extremely long time, it would be easy to increase the timeout or even do runs without it being enabled.

/Marta

> -Daniel
> --
> Daniel Vetter
> Software Engineer, Intel Corporation
> http://blog.ffwll.ch
> _______________________________________________
> Intel-gfx mailing list
> Intel-gfx@lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

[Imre Deak]

On Thu, Oct 19, 2017 at 06:48:30AM +0000, Lofstedt, Marta wrote:
> 
> 
> > -----Original Message-----
> > From: Intel-gfx [mailto:intel-gfx-bounces@lists.freedesktop.org] On Behalf
> > Of Daniel Vetter
> > Sent: Wednesday, October 18, 2017 5:36 PM
> > To: Latvala, Petri <petri.latvala@intel.com>
> > Cc: intel-gfx@lists.freedesktop.org
> > Subject: Re: [Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Output more
> > finegrained progress info to avoid CI watchdog timeout
> > 
> > On Wed, Oct 18, 2017 at 02:43:38PM +0300, Petri Latvala wrote:
> > > On Wed, Oct 18, 2017 at 02:29:33PM +0300, Imre Deak wrote:
> > > > The CI software watchdog (owatch) will timeout if the test doesn't
> > > > output anything for a long time on standard out or error. At least
> > > > the plane-all-modeset-transition and
> > > > plane-all-modeset-transition-fences
> > > > subtests run without any output longer than the watchdog timeout, so
> > > > output some more progress info.
> > >
> > > No, owatch is wrapping piglit, and pings the watchdog if _piglit_
> > > prints anything. Which it does on start/exit of a test.
> > 
> > tbh this sounds like owatch being dense and it shouldn't try to reboot this
> > quickly. What's the current owatch timeout?
> > 
> > Aside: What exactly does owatch give us? I thought jenkins also watches
> > machines and reboots them using the ac switch ... And owatch provides
> > spurious reboots?
> Daniel,
> Owatch gives us the knowledge that is was a test that took too long. I.e. we will know that it was not a system hang. 
>  We also know that the NMI watchdog didn't trigger. 
> I believe this is extremely useful information when you are starting to debug the issue
> 
> Imre if you believe that owatch is preventing you from getting
> information to debug why these test are taking so extremely long time,
> it would be easy to increase the timeout or even do runs without it
> being enabled.

It would be useful to have a clear indication if the test really hang or
it just took too long. Right now this isn't obvious without going
through the logs.

--Imre
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx