Re: [PATCH] [RFC i-g-t] Test Design to verify mipi enable/disable sequence.

From: Daniel Vetter <daniel@ffwll.ch>
To: Jani Nikula <jani.nikula@intel.com>
Cc: intel-gfx@lists.freedesktop.org, ville.syrjala@intel.com,
	Jenkins Val <jenkins@intel.com>
Subject: Re: [PATCH] [RFC i-g-t] Test Design to verify mipi enable/disable sequence.
Date: Wed, 11 Jan 2017 09:14:15 +0100	[thread overview]
Message-ID: <20170111081415.siywqriv4xdbo7sq@phenom.ffwll.local> (raw)
In-Reply-To: <87o9zgfw19.fsf@intel.com>

On Mon, Jan 09, 2017 at 11:00:02AM +0200, Jani Nikula wrote:
> On Sat, 07 Jan 2017, Yadav Jyoti <jyoti.r.yadav@intel.com> wrote:
> > From: Jenkins Val <jenkins@intel.com>
> >
> 
> This place here is for the commit message, where you should explain
> *why* we need this change.
> 
> Where do you get the XML file? Do you write it manually? How do you
> manage them? The kernel will execute the sequences from the VBT, not
> from your XML file, so you'll have a problem of maintaining XML files
> for each machine you ever run this test on.
> 
> I'm also not thrilled about adding special debug messages that the test
> depends on finding in dmesg. The test also doesn't actually do anything
> to cause the sequences to be run, so you expect some other, undefined
> tests to have been run, the dmesg from that run captured, and saved to a
> file that you feed to this test.
> 
> I think the design is rather fragile.

Also, igt are black-box testcases, they should not assume any specific
implementation. Every time we break that, we are adding api (even if it's
just for tests in debugfs), and that means coordination issues.

On top of that Chris is building up a neat selftest infrastructure which
helps to cover anything which cannot easily be tested using a blackbox
approach.

Furthermore writing the same stuff twice (like the xml and vbt sequence
this test seems to rely) on isn't validation, it's just typing stuff
twice. Real validation tries to verify (preferrably orthogonal)
invariants, or at least entirely indepent approachs to the implementation.
Another similar case was the color manager testcase, which did not check
functional outcome using crc, but instead checked that the kernel wrote
the right register values in the right places. That's not independent
validation, an hence not really useful as a testcase.

If you want to validate dsi, then either there needs to be some indication
from the sink (on edp we have sink crcs and status flags) that thing went
well, or we need a special testing board like chamelium (but that doesn't
do dsi unfortunately). Everything else is already covered by the generic
modeset testcases and the kernel's selftest.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx