Re: [Intel-gfx] ✗ Fi.CI.IGT: failure for use DYNAMIC_DEBUG to implement DRM.debug (rev2)

["Gupta, Anshuman" <anshuman.gupta@intel.com>]

> -----Original Message-----
> From: Latvala, Petri <petri.latvala@intel.com>
> Sent: Monday, September 6, 2021 4:56 PM
> To: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
> Cc: jim.cromie@gmail.com; Intel Graphics Development <intel-
> gfx@lists.freedesktop.org>; Bhatt, Jigar <jigar.bhatt@intel.com>; Gupta,
> Anshuman <anshuman.gupta@intel.com>
> Subject: Re: [Intel-gfx] ✗ Fi.CI.IGT: failure for use DYNAMIC_DEBUG to
> implement DRM.debug (rev2)
> 
> On Mon, Sep 06, 2021 at 11:04:13AM +0100, Tvrtko Ursulin wrote:
> >
> > On 03/09/2021 14:01, Petri Latvala wrote:
> > > On Fri, Sep 03, 2021 at 12:29:51PM +0100, Tvrtko Ursulin wrote:
> > > >
> > > > On 03/09/2021 01:31, jim.cromie@gmail.com wrote:
> > > > >
> > > > >
> > > > > On Tue, Aug 31, 2021 at 5:38 PM Patchwork
> > > > > <patchwork@emeril.freedesktop.org
> > > > > <mailto:patchwork@emeril.freedesktop.org>> wrote:
> > > > >
> > > > >      __
> > > > >      *Patch Details*
> > > > >      *Series:*	use DYNAMIC_DEBUG to implement DRM.debug (rev2)
> > > > >      *URL:*	https://patchwork.freedesktop.org/series/93914/
> > > > >      <https://patchwork.freedesktop.org/series/93914/>
> > > > >      *State:*	failure
> > > > >      *Details:*
> > > > >      https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20931/index.html
> > > > >
> > > > > <https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20931/index.
> > > > > html>
> > > > >
> > > > >
> > > > >        CI Bug Log - changes from CI_DRM_10541_full ->
> > > > > Patchwork_20931_full
> > > > >
> > > > >
> > > > >          Summary
> > > > >
> > > > >      *FAILURE*
> > > > >
> > > > >      Serious unknown changes coming with Patchwork_20931_full
> absolutely
> > > > >      need to be
> > > > >      verified manually.
> > > > >
> > > > >      If you think the reported changes have nothing to do with the changes
> > > > >      introduced in Patchwork_20931_full, please notify your bug team to
> > > > >      allow them
> > > > >      to document this new failure mode, which will reduce false positives
> > > > >      in CI.
> > > > >
> > > > >
> > > > > hi Team !
> > > > >
> > > > > I think I need a bit of orientation.
> > > > >
> > > > >
> > > > >          Possible new issues
> > > > >
> > > > >      Here are the unknown changes that may have been introduced in
> > > > >      Patchwork_20931_full:
> > > > >
> > > > >
> > > > >            IGT changes
> > > > >
> > > > >
> > > > >              Possible regressions
> > > > >
> > > > >        * igt@gem_exec_schedule@u-submit-golden-slice@vcs0:
> > > > >            o shard-skl: NOTRUN -> INCOMPLETE
> > > > >
> > > > > <https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20931/shard-
> > > > > skl10/igt@gem_exec_schedule@u-submit-golden-slice@vcs0.html>
> > > > >
> > > > >
> > > > >              Warnings
> > > > >
> > > > >        * igt@i915_pm_dc@dc9-dpms:
> > > > >            o shard-skl: SKIP
> > > > >              <https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_10541/shard-
> skl6/igt@i915_pm_dc@dc9-dpms.html>
> > > > >              ([fdo#109271]) -> FAIL
I915_pm_dc test failures are not related to your series , probably  this failure can be ignored.
Br,
Anshuman Gupta.
> > > > >
> > > > > <https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_20931/shard-
> > > > > skl8/igt@i915_pm_dc@dc9-dpms.html>
> > > > >
> > > > >
> > > > >
> > > > > Im assuming the FAIL is the sticking point,
> > > >
> > > > Both INCOMPLETE and FAIL will cause a failure to be declared, but in this
> case it looks to me this series is not at fault.
> > > >
> > > > 1)
> > > >
> > > > The gem_exec_schedule failure looks like a test runner timeout issue (and
> apparently test does not handle well running the the fence timeout enabled).
> > > >
> > > > @Petri - does the below look like IGT runner running our of time
> > > > budget for the test run? Would it log
> > > >
> > > > [1051.943629] [114/138] ( 11s left) gem_exec_schedule
> > > > (u-submit-golden-slice) Starting subtest: u-submit-golden-slice
> > > > Starting dynamic subtest: rcs0 Dynamic subtest rcs0: SUCCESS
> > > > (80.175s) Starting dynamic subtest: bcs0 Dynamic subtest bcs0:
> > > > SUCCESS (80.195s) Starting dynamic subtest: vcs0 Dynamic subtest
> > > > vcs0: SUCCESS (80.243s) Starting dynamic subtest: vecs0
> > > >
> > > > Interesting part is that according to dmesg it never got to the vecs0
> dynamic subtest - any idea what happened there?
> > >
> > > Yep, we ran out of time. We still had 11 seconds left to execute
> > > something but then that test took centuries.
> >
> > Okay at least that's explained then.
> >
> > Perhaps could make that act of termination logged in igt_runner log?
> 
> We do log everything we can, but unfortunately this was such an extreme case
> that it hit the timeout that just cuts off AC power.
> 
> run.log has this act logged.
> 
> >
> > Also, any explanation on why dmesg and igt_runner log disagree on how
> > far the test progressed (in terms of which subtest was active when
> > things ended)?
> >
> 
> Looks like a race condition with the above mentioned AC cutoff. The test printed
> that vcs0 is finished and vecs0 starts, that info was printed to igt_runner's
> stdout, but the write() to the test logs didn't get called before cutoff.
> 
> 
> --
> Petri Latvala
> 
> 
> > Regards,
> >
> > Tvrtko
> >
> >
> > >
> > >
> > > >
> > > > 2)
> > > >
> > > > I915_pm_dc I'd say you just gotten unlucky that test went from always
> skipping on SKL to trying to run it and then it failed. But I don't know enough
> about the test to tell you why. Adding Jigar and Anshuman as test author and
> reviewer who might be able to shed some light here.
> > > >
> > > > Regards,
> > > >
> > > > Tvrtko
> > > >
> > > > > I found code that seemed to be relevant
> > > > >
> > > > > [jimc@frodo igt-ci-tags.git]$ git remote -v
> > > > > origin https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags.git
> > > > > <https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags.git> (fetch)
> > > > > origin https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags.git
> > > > > <https://gitlab.freedesktop.org/gfx-ci/igt-ci-tags.git> (push)
> > > > >
> > > > > I built it, got an error, threw that to google,
> > > > > found a patch on i-g-t from
> > > > > commit 1ff3e5ae99ceb66d2926d58635d0379ce971065a (HEAD ->
> master)
> > > > > Author: Lyude Paul <lyude@redhat.com <mailto:lyude@redhat.com>>
> > > > > Date:   Mon Apr 15 14:57:23 2019 -0400
> > > > >
> > > > > and applied it
> > > > > it fixed the one problem
> > > > >
> > > > > then I looked at previous head
> > > > >
> > > > > commit f052e49a43cc9704ea5f240df15dd9d3dfed68ab (origin/master,
> origin/HEAD)
> > > > > Author: Simon Ser <simon.ser@intel.com
> <mailto:simon.ser@intel.com>>
> > > > > Date:   Wed Apr 24 19:15:26 2019 +0300
> > > > >
> > > > > It sure seems that tree is stale.
> > >
> > > That tree's master ref does not get updated. It's only for storing tags.
> > >
> > > That test result comparison was too long to fit into patchwork so the
> > > build information at the bottom is missing, but the BAT results have
> > > it:
> > >
> > > IGT_6193: 080869f804cb86b25a38889e5ce9a870571cd8c4 @
> https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
> > >
> > >
> > >