[Fuego] How to return test_processing result if multiple "log_compare" exist

[Fuego] How to return test_processing result if multiple "log_compare" exist

[Liu, Wenlong]

Hi

I've been doing the OpenSSL test recently and find some interesting log as below.
...
##### doing fuego phase: post_test ########
Teardown board link
##### doing fuego phase: processing ########
ERROR: log_compare: pattern 'passed|ok' found 169 times (expected greater or equal than 176)
log_compare: pattern 'skip' found 0 times (expected less or equal than 86)
running python with PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
WARNING: missing or faulty criteria.json file (/fuego-core/engine/tests/Functional.OpenSSL/criteria.json), looking for reference.log
WARNING: No reference.log found. Using default criteria.
No reference.json available
ref={'test_sets': [{'test_cases': [{'status': 'PASS', 'name': 'OpenSSL'}], 'name': 'default'}]}
Applying criterion {'max_fail': 0, 'tguid': 'default'}
Applying criterion {'max_fail': 0, 'tguid': 'Functional.OpenSSL'}
Writing run data to /fuego-rw/logs/Functional.OpenSSL/DB410c-dab2.default.7.7/run.json
...

So, if multiple "log_compare" exist, how should we return the final log processing result?

Best regards
Liu 

Re: [Fuego] How to return test_processing result if multiple "log_compare" exist

[Daniel Sangorrin]

Hi,

In Functional.bzip2 I used to log_copares. Please check commit 01808b7

Regards,
Daniel

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Liu, Wenlong
> Sent: Monday, December 04, 2017 3:18 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] How to return test_processing result if multiple "log_compare" exist
> 
> Hi
> 
> I've been doing the OpenSSL test recently and find some interesting log as below.
> ...
> ##### doing fuego phase: post_test ########
> Teardown board link
> ##### doing fuego phase: processing ########
> ERROR: log_compare: pattern 'passed|ok' found 169 times (expected greater or equal than 176)
> log_compare: pattern 'skip' found 0 times (expected less or equal than 86)
> running python with PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
> WARNING: missing or faulty criteria.json file (/fuego-core/engine/tests/Functional.OpenSSL/criteria.json), looking for reference.log
> WARNING: No reference.log found. Using default criteria.
> No reference.json available
> ref={'test_sets': [{'test_cases': [{'status': 'PASS', 'name': 'OpenSSL'}], 'name': 'default'}]}
> Applying criterion {'max_fail': 0, 'tguid': 'default'}
> Applying criterion {'max_fail': 0, 'tguid': 'Functional.OpenSSL'}
> Writing run data to /fuego-rw/logs/Functional.OpenSSL/DB410c-dab2.default.7.7/run.json
> ...
> 
> So, if multiple "log_compare" exist, how should we return the final log processing result?
> 
> Best regards
> Liu
> 
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] How to return test_processing result if multiple "log_compare" exist

[Liu, Wenlong]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Monday, December 04, 2017 2:24 PM
> To: Liu, Wenlong/刘 文龙 <liuwl.fnst@cn.fujitsu.com>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] How to return test_processing result if multiple
> "log_compare" exist
> 
> Hi,
> 
> In Functional.bzip2 I used to log_copares. Please check commit 01808b7

I checked that commit, Good. Thanks for your reply.

> 
> Regards,
> Daniel
> 
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org
> > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Liu,
> > Wenlong
> > Sent: Monday, December 04, 2017 3:18 PM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] How to return test_processing result if multiple
> > "log_compare" exist
> >
> > Hi
> >
> > I've been doing the OpenSSL test recently and find some interesting log
> as below.
> > ...
> > ##### doing fuego phase: post_test ######## Teardown board link #####
> > doing fuego phase: processing ########
> > ERROR: log_compare: pattern 'passed|ok' found 169 times (expected
> > greater or equal than 176)
> > log_compare: pattern 'skip' found 0 times (expected less or equal than
> > 86) running python with
> >
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr
> > /games
> > WARNING: missing or faulty criteria.json file
> > (/fuego-core/engine/tests/Functional.OpenSSL/criteria.json), looking
> > for reference.log
> > WARNING: No reference.log found. Using default criteria.
> > No reference.json available
> > ref={'test_sets': [{'test_cases': [{'status': 'PASS', 'name':
> > 'OpenSSL'}], 'name': 'default'}]} Applying criterion {'max_fail': 0,
> > 'tguid': 'default'} Applying criterion {'max_fail': 0, 'tguid':
> > 'Functional.OpenSSL'} Writing run data to
> > /fuego-rw/logs/Functional.OpenSSL/DB410c-dab2.default.7.7/run.json
> > ...
> >
> > So, if multiple "log_compare" exist, how should we return the final log
> processing result?
> >
> > Best regards
> > Liu
> >
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> 

Re: [Fuego] How to return test_processing result if multiple "log_compare" exist

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, December 04, 2017 3:24 PM
> In Functional.bzip2 I used to log_copares. Please check commit 01808b7

Hmmm.  This type of handling didn't used to be needed, because we
are running with set -e in the shell, so the first error (non-zero return)
would terminate test_processing, and the program would exit
at that point.

This changed with the refactoring we did for post-processing.

In git commit 309afa2f we modified the code so that test_processing
returning non-zero does NOT terminate the program.  This is the
correct behavior, given that we have additional work to do after this.
(We used to do additional post-processing in a separate Jenkins
action.)

In general I think that doing multiple log_compares is not the normal
case, or the preferred solution.
Using only counts of success and failure for evaluating the result
is pretty brittle, and is subject to missing an important transition
(e.g. a trivial failure starts succeeding at the same time a critical
test, that was passing previously, starts failing.)

I need to review this more, but I suspect it would be better to switch
the Functional.OpenSSL test case to using a parser.py and a criteria.json file.

But at least handling the multiple log_compares properly is an improvement.
 -- Tim