Re: [Fuego] How to return test_processing result if multiple "log_compare" exist

["Bird, Timothy" <Tim.Bird@sony.com>]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, December 04, 2017 3:24 PM
> In Functional.bzip2 I used to log_copares. Please check commit 01808b7

Hmmm.  This type of handling didn't used to be needed, because we
are running with set -e in the shell, so the first error (non-zero return)
would terminate test_processing, and the program would exit
at that point.

This changed with the refactoring we did for post-processing.

In git commit 309afa2f we modified the code so that test_processing
returning non-zero does NOT terminate the program.  This is the
correct behavior, given that we have additional work to do after this.
(We used to do additional post-processing in a separate Jenkins
action.)

In general I think that doing multiple log_compares is not the normal
case, or the preferred solution.
Using only counts of success and failure for evaluating the result
is pretty brittle, and is subject to missing an important transition
(e.g. a trivial failure starts succeeding at the same time a critical
test, that was passing previously, starts failing.)

I need to review this more, but I suspect it would be better to switch
the Functional.OpenSSL test case to using a parser.py and a criteria.json file.

But at least handling the multiple log_compares properly is an improvement.
 -- Tim