[LTP] realtime measurement tests: approach to criteria

[LTP] realtime measurement tests: approach to criteria

[Darren Hart]

The current ltp/testcases/realtime tests belong to one of func, perf, or 
stress.  While strict pass/fail criteria make sense for functional tests 
(did the tasks wake up in priority order?), the others use "arbitrary" 
values and compare those against the whatever is being measured (wakeup 
latency, etc.) and then determine pass/fail.  Ideally the tests 
themselves would not determine the pass/fail criteria, and would instead 
simply report on their measurements since the criteria will vary in 
every use-case based on requirements, workload, hardware, etc.

I'd like to propose an approach where the tests only report their 
measured values (with the exception of the func/* tests which will 
maintain their pass/fail criteria).  Users should be able to populate a 
criteria.conf file that specified the criteria of each test.  The 
results could then be parsed, compared against the results, and a 
pass/fail determined from there.  I suspect it would be best for the .c 
tests to just report the numbers and the statistics in a common format 
and rely on python parser scripts to read the config file and determine 
pass/fail from there.

I'd like users thoughts on this approach before we jump in and start 
changing things (as this is a fairly invasive change).

-- 
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: realtime measurement tests: approach to criteria

[Subrata Modak]

Hi Darren,

On Mon, 2009-08-03 at 16:30 -0700, Darren Hart wrote: 
> The current ltp/testcases/realtime tests belong to one of func, perf, or 
> stress.  While strict pass/fail criteria make sense for functional tests 
> (did the tasks wake up in priority order?), the others use "arbitrary" 
> values and compare those against the whatever is being measured (wakeup 
> latency, etc.) and then determine pass/fail.  Ideally the tests 
> themselves would not determine the pass/fail criteria, and would instead 
> simply report on their measurements since the criteria will vary in 
> every use-case based on requirements, workload, hardware, etc.
> 
> I'd like to propose an approach where the tests only report their 
> measured values (with the exception of the func/* tests which will 
> maintain their pass/fail criteria).  Users should be able to populate a 
> criteria.conf file that specified the criteria of each test.  The 
> results could then be parsed, compared against the results, and a 
> pass/fail determined from there.  I suspect it would be best for the .c 
> tests to just report the numbers and the statistics in a common format 
> and rely on python parser scripts to read the config file and determine 
> pass/fail from there.
> 
> I'd like users thoughts on this approach before we jump in and start 
> changing things (as this is a fairly invasive change).

This is indeed a good approach. Should we also ask the RT-USERS, who
might be interested to comment on this ?

Regards--
Subrata

> 

Re: [LTP] realtime measurement tests: approach to criteria

[Subrata Modak]

Hi Darren,

On Mon, 2009-08-03 at 16:30 -0700, Darren Hart wrote: 
> The current ltp/testcases/realtime tests belong to one of func, perf, or 
> stress.  While strict pass/fail criteria make sense for functional tests 
> (did the tasks wake up in priority order?), the others use "arbitrary" 
> values and compare those against the whatever is being measured (wakeup 
> latency, etc.) and then determine pass/fail.  Ideally the tests 
> themselves would not determine the pass/fail criteria, and would instead 
> simply report on their measurements since the criteria will vary in 
> every use-case based on requirements, workload, hardware, etc.
> 
> I'd like to propose an approach where the tests only report their 
> measured values (with the exception of the func/* tests which will 
> maintain their pass/fail criteria).  Users should be able to populate a 
> criteria.conf file that specified the criteria of each test.  The 
> results could then be parsed, compared against the results, and a 
> pass/fail determined from there.  I suspect it would be best for the .c 
> tests to just report the numbers and the statistics in a common format 
> and rely on python parser scripts to read the config file and determine 
> pass/fail from there.
> 
> I'd like users thoughts on this approach before we jump in and start 
> changing things (as this is a fairly invasive change).

This is indeed a good approach. Should we also ask the RT-USERS, who
might be interested to comment on this ?

Regards--
Subrata

> 


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list

Re: realtime measurement tests: approach to criteria

[Darren Hart]

Subrata Modak wrote:
> Hi Darren,
> 
> On Mon, 2009-08-03 at 16:30 -0700, Darren Hart wrote: 
>> The current ltp/testcases/realtime tests belong to one of func, perf, or 
>> stress.  While strict pass/fail criteria make sense for functional tests 
>> (did the tasks wake up in priority order?), the others use "arbitrary" 
>> values and compare those against the whatever is being measured (wakeup 
>> latency, etc.) and then determine pass/fail.  Ideally the tests 
>> themselves would not determine the pass/fail criteria, and would instead 
>> simply report on their measurements since the criteria will vary in 
>> every use-case based on requirements, workload, hardware, etc.
>>
>> I'd like to propose an approach where the tests only report their 
>> measured values (with the exception of the func/* tests which will 
>> maintain their pass/fail criteria).  Users should be able to populate a 
>> criteria.conf file that specified the criteria of each test.  The 
>> results could then be parsed, compared against the results, and a 
>> pass/fail determined from there.  I suspect it would be best for the .c 
>> tests to just report the numbers and the statistics in a common format 
>> and rely on python parser scripts to read the config file and determine 
>> pass/fail from there.
>>
>> I'd like users thoughts on this approach before we jump in and start 
>> changing things (as this is a fairly invasive change).
> 
> This is indeed a good approach. Should we also ask the RT-USERS, who
> might be interested to comment on this ?

Thanks for including the rt-users list, yes I should have done that 
originally as well.

-- 
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team

Re: [LTP] realtime measurement tests: approach to criteria

[Darren Hart]

Subrata Modak wrote:
> Hi Darren,
> 
> On Mon, 2009-08-03 at 16:30 -0700, Darren Hart wrote: 
>> The current ltp/testcases/realtime tests belong to one of func, perf, or 
>> stress.  While strict pass/fail criteria make sense for functional tests 
>> (did the tasks wake up in priority order?), the others use "arbitrary" 
>> values and compare those against the whatever is being measured (wakeup 
>> latency, etc.) and then determine pass/fail.  Ideally the tests 
>> themselves would not determine the pass/fail criteria, and would instead 
>> simply report on their measurements since the criteria will vary in 
>> every use-case based on requirements, workload, hardware, etc.
>>
>> I'd like to propose an approach where the tests only report their 
>> measured values (with the exception of the func/* tests which will 
>> maintain their pass/fail criteria).  Users should be able to populate a 
>> criteria.conf file that specified the criteria of each test.  The 
>> results could then be parsed, compared against the results, and a 
>> pass/fail determined from there.  I suspect it would be best for the .c 
>> tests to just report the numbers and the statistics in a common format 
>> and rely on python parser scripts to read the config file and determine 
>> pass/fail from there.
>>
>> I'd like users thoughts on this approach before we jump in and start 
>> changing things (as this is a fairly invasive change).
> 
> This is indeed a good approach. Should we also ask the RT-USERS, who
> might be interested to comment on this ?

Thanks for including the rt-users list, yes I should have done that 
originally as well.

-- 
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list