[LTP] TWARN and "ltp-pan reported PASS"

[LTP] TWARN and "ltp-pan reported PASS"

[Petr Vorel]

Hi,

our shell scripts wrapping ltp-pan binary report failure when there is any TWARN.

Similar code in these scripts:
runltp
runltplite.sh
testscripts/diskio.sh
testscripts/network.sh
testscripts/runEALtests.sh

if [ $? -eq "0" ]; then
  echo ltp-pan reported PASS
else
  echo ltp-pan reported FAIL
fi

Does using TWARN always always mean failure? I mean, it's good that ltp-pan returns 4
instead of 0, but wrapper script should return something like:

if [ $? -eq 0 ]; then
  echo ltp-pan reported PASS
elif [ $? -eq 4 ]; then
  echo ltp-pan reported PASS with warnings
else
  echo ltp-pan reported FAIL
fi


Kind regards,
Petr

[LTP] TWARN and "ltp-pan reported PASS"

[Cyril Hrubis]

Hi!
> our shell scripts wrapping ltp-pan binary report failure when there is any TWARN.
> 
> Similar code in these scripts:
> runltp
> runltplite.sh
> testscripts/diskio.sh
> testscripts/network.sh
> testscripts/runEALtests.sh
> 
> if [ $? -eq "0" ]; then
>   echo ltp-pan reported PASS
> else
>   echo ltp-pan reported FAIL
> fi
>
> Does using TWARN always always mean failure? I mean, it's good that ltp-pan returns 4
> instead of 0, but wrapper script should return something like:
>
> if [ $? -eq 0 ]; then
>   echo ltp-pan reported PASS
> elif [ $? -eq 4 ]; then
>   echo ltp-pan reported PASS with warnings
> else
>   echo ltp-pan reported FAIL
> fi

Are you sure that ltp-pan actually passes the test return value? As far
as I can tell it actually increments the exit_stat variable on error
then does exit(exit_stat).

So I would say that the only guarantee that you have is that non-zero
return value from ltp-pan means "everything went ok" and non-zero
"something went wrong".

Frankly at this point I would rather kept the old test execution scripts
as they are and put more effort into the new execution framework.
Hopefully I will have a bit time to invest into it once we are done with
the release.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] TWARN and "ltp-pan reported PASS"

[Petr Vorel]

Hi Cyril!
> Are you sure that ltp-pan actually passes the test return value? As far
> as I can tell it actually increments the exit_stat variable on error
> then does exit(exit_stat).

> So I would say that the only guarantee that you have is that non-zero
> return value from ltp-pan means "everything went ok" and non-zero
> "something went wrong".

do_test() in shell testcases/network/multicast/mc_cmds/mc_cmds with TWARN:
		tst_resm TWARN "'ip maddr show $(tst_iface)' failed," \
			       " parsing 'ip maddr show'"

reports termination_id=4 => FAIL:

$ /opt/ltp/testscripts/network.sh -m
network_settings 1 TINFO: initialize 'lhost' 'ltp_ns_veth2' interface
network_settings 1 TINFO: set local addr 10.0.0.2/24
network_settings 1 TINFO: set local addr fd00:1:1:1::2/64
network_settings 1 TINFO: initialize 'rhost' 'ltp_ns_veth1' interface
network_settings 1 TINFO: set remote addr 10.0.0.1/24
network_settings 1 TINFO: set remote addr fd00:1:1:1::1/64
network_settings 1 TINFO: wait for IPv6 DAD completion 1/5 sec
<<<test_start>>>
tag=mc_cmds stime=1494410089
cmdline="mc_cmds"
contacts=""
analysis=exit
<<<test_output>>>
incrementing stop
mc_cmds 1 TWARN: 'ip maddr show ltp_ns_veth2' failed,  parsing 'ip maddr show'
mc_cmds 1 TINFO: Ping all-host-groups over specified interface
ping: unknown iface 10.0.0.2
mc_cmds 1 TINFO: Trying to ping with ltp_ns_veth2 with the -I option instead of IP address
mc_cmds 1 TPASS: Test Successful
<<<execution_status>>>
initiation_status="ok"
duration=1 termination_type=exited termination_id=4 corefile=no
cutime=3 cstime=1
<<<test_end>>>
ltp-pan reported FAIL

Changing it into TINFO it returns termination_id=0 => PASS:
-----
diff --git testcases/network/multicast/mc_cmds/mc_cmds testcases/network/multicast/mc_cmds/mc_cmds
index 8077b1f15..e66890019 100755
--- testcases/network/multicast/mc_cmds/mc_cmds
+++ testcases/network/multicast/mc_cmds/mc_cmds
@@ -51,7 +51,7 @@ do_test()
 
        ip maddr show $(tst_iface) | grep -q '224.0.0.1'
        if [ $? -ne 0 ]; then
-               tst_resm TWARN "'ip maddr show $(tst_iface)' failed," \
+               tst_resm TINFO "'ip maddr show $(tst_iface)' failed," \
                               " parsing 'ip maddr show'"
                ip maddr show | sed -ne "/\s$(tst_iface)/,/^[0-9]/p" | \
                        grep -q 224.0.0.1 || \
-----

$ /opt/ltp/testscripts/network.sh -m
<snip>
mc_cmds 1 TINFO: Trying to ping with ltp_ns_veth2 with the -I option instead of IP address
mc_cmds 1 TPASS: Test Successful
<<<execution_status>>>
initiation_status="ok"
duration=1 termination_type=exited termination_id=0 corefile=no
cutime=2 cstime=1
<<<test_end>>>
ltp-pan reported PASS
-----
So I don't see any other reason than TWARN causing the exit value.

> Frankly at this point I would rather kept the old test execution scripts
> as they are and put more effort into the new execution framework.
> Hopefully I will have a bit time to invest into it once we are done with
> the release.
I understand and agree with that. I'll be also glad when the old scripts will go away. I
just wanted to consult what is the expected behaviour.


Kind regards,
Petr

[LTP] TWARN and "ltp-pan reported PASS"

[Cyril Hrubis]

Hi!
> Changing it into TINFO it returns termination_id=0 => PASS:
> -----
> diff --git testcases/network/multicast/mc_cmds/mc_cmds testcases/network/multicast/mc_cmds/mc_cmds
> index 8077b1f15..e66890019 100755
> --- testcases/network/multicast/mc_cmds/mc_cmds
> +++ testcases/network/multicast/mc_cmds/mc_cmds
> @@ -51,7 +51,7 @@ do_test()
>  
>         ip maddr show $(tst_iface) | grep -q '224.0.0.1'
>         if [ $? -ne 0 ]; then
> -               tst_resm TWARN "'ip maddr show $(tst_iface)' failed," \
> +               tst_resm TINFO "'ip maddr show $(tst_iface)' failed," \
>                                " parsing 'ip maddr show'"
>                 ip maddr show | sed -ne "/\s$(tst_iface)/,/^[0-9]/p" | \
>                         grep -q 224.0.0.1 || \
> -----
> 
> $ /opt/ltp/testscripts/network.sh -m
> <snip>
> mc_cmds 1 TINFO: Trying to ping with ltp_ns_veth2 with the -I option instead of IP address
> mc_cmds 1 TPASS: Test Successful
> <<<execution_status>>>
> initiation_status="ok"
> duration=1 termination_type=exited termination_id=0 corefile=no
> cutime=2 cstime=1
> <<<test_end>>>
> ltp-pan reported PASS
> -----
> So I don't see any other reason than TWARN causing the exit value.

Looks like I've haven't been clear enough. The ltp-pan mapped any
non-zero test exit value to a failure. Partly becasuse there were (and
possibly still may be some) testcases that do not use LTP exit bitflags
and partly because ltp-pan predates LTP, at least that's what I remeber
from some CVS archeology.

Then we added special support for the TCONF exit status, since otherwise
skipped testcases were listed as failed as well.

So yes the TWARN still maps to a failure in ltp-pan and we use TWARN
mostly for non-fatal (to the test execution) failures, mostly in test
cleanups. Is that clear enough?

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] TWARN and "ltp-pan reported PASS"

[Petr Vorel]

Hi Cyril,

> Looks like I've haven't been clear enough. The ltp-pan mapped any
> non-zero test exit value to a failure. Partly becasuse there were (and
> possibly still may be some) testcases that do not use LTP exit bitflags
> and partly because ltp-pan predates LTP, at least that's what I remeber
> from some CVS archeology.

> Then we added special support for the TCONF exit status, since otherwise
> skipped testcases were listed as failed as well.

> So yes the TWARN still maps to a failure in ltp-pan and we use TWARN
> mostly for non-fatal (to the test execution) failures, mostly in test
> cleanups. Is that clear enough?

Thanks for an explanation.
I still wish TWARN wouldn't be considered as error (and find and fix these tests which
don't exit bitflags). And I don't wouldn't like to introduce some kind of soft warning flag,
which would lead to exit 0 which could be an alternative solution.


Kind regards,
Petr

[LTP] TWARN and "ltp-pan reported PASS"

[Cyril Hrubis]

Hi!
> > Looks like I've haven't been clear enough. The ltp-pan mapped any
> > non-zero test exit value to a failure. Partly becasuse there were (and
> > possibly still may be some) testcases that do not use LTP exit bitflags
> > and partly because ltp-pan predates LTP, at least that's what I remeber
> > from some CVS archeology.
> 
> > Then we added special support for the TCONF exit status, since otherwise
> > skipped testcases were listed as failed as well.
> 
> > So yes the TWARN still maps to a failure in ltp-pan and we use TWARN
> > mostly for non-fatal (to the test execution) failures, mostly in test
> > cleanups. Is that clear enough?
> 
> Thanks for an explanation.
> I still wish TWARN wouldn't be considered as error (and find and fix these tests which
> don't exit bitflags).

Well before that we would have to review all uses of TWARN in the source
tree, for some TBROK would possibly be a better fit. Generally we use
TWARN for things that are not easily recoverable i.e. we were not able
to restore the system state which is mostly used in the test cleanup.
And that is something that should be reported to the user since it may
and will affect subsequent tests. Maybe we can simply map all these to
TBROK.

> And I don't wouldn't like to introduce some kind of soft warning flag,
> which would lead to exit 0 which could be an alternative solution.

We are using TINFO for both informational messasges as well as for a
warnings that are not supposed to fail the test. Technically speaking
the only difference between using TWARN and TINFO is that the former
fails the test and that the log shows TWARN in yellow rather than TINFO
in blue. So if you have a message that should not fail the test consider
changing it to TINFO for now and we can always figure out something
better later.

Moreover the new test runner I'm working on counts warnings correctly
so I'm against filtering out TWARN from the exit value even if we decide
to use that only for a minor/recoverable errors. Anbody can filter it
out easily himself if needed.

Another solution would be removing TWARN completely since it's semantics
is not well defined that would leave us with TBROK for non-recoverable
problems and TINFO for the rest of the messages.

-- 
Cyril Hrubis
chrubis@suse.cz