Re: [PATCH 1/2] kselftests: timers: Make set-timer-lat fail more gracefully for !CAP_WAKE_ALARM

[Prarit Bhargava <prarit@redhat.com>]

On 04/02/2015 02:02 PM, John Stultz wrote:
> On Thu, Apr 2, 2015 at 3:18 AM, Prarit Bhargava <prarit@redhat.com> wrote:
>> On 03/26/2015 01:33 PM, Tyler Baker wrote:
>>> I realize this may be a good amount of work, so I'd like to help out.
>>> Perhaps working John to convert his timer tests to use TAP output
>>> would be a good starting point?
>>
>> John, I could probably do that for you.  I'm always willing to give it a shot.
> 
> I took a quick look into it, since I'm definitely interested in
> improving output formatting, but man, TAP is a fairly ugly output
> format if you ask me.
> 
> It only has binary "ok" or "not ok" (why not "fail", or something else
> that's exclusively grep-able, I don't know). So I'm not sure if cases
> where functionality is unsupported should be a pass or fail.
> 
> Most problematically: It seems to want enumeration in the test output
> (so test 2 needs to print: "ok 2 Test complete") which means either
> there needs to be a wrapper that does the TAP output knowing which
> test of N its currently running, or the test number needs to be
> submitted as an runtime argument to the test, and the test then has to
> add it to its output line.
> 
> Anyway, if we do want to go with that format,  I suspect it should be
> something we add to the kselftest pass/fail hooks, rather then to the
> individual tests. Then its just a matter of prefixing normal test
> output with #'s so they can be ignored by the parser.

I've been looking into this and while TAP is useful, I find myself with a few
questions:

1.  I want the output to be both human and machine readable.  I find that "not
okay" vs "okay" is confusing at best for humans to read.  I suppose that is a
minor issue, but it is a real issue when laymen are asked to execute a test.

2.  I think we can get around the TAP numbering, however, maybe a better
approach is to limit tests to doing one, and only one, test?  ie) something like
rtctest.c would have be be broken up into individual tests.

3.  TAP appears to have a BSD-style license?  Not knowing enough about cross
licensing issues like this, is that going to be a problem?

4.  I think John is moving in the right direction by suggesting that we move to
putting this into kselftest.h as hooks, rather than modifying each test.
Another idea is to include a script that allows the running of multiple tests
from the top level tests directory.

Let me muck around with the timer tests and see what I can come up with.  I'm
trying to keep in mind that we're going to be asking people to potentially
modify their tests, as well as what we're going to be telling people to do when
they add new tests, while I make modifications.

P.


> 
> thanks
> -john
>