[LTP] [RFC][PATCH 0/1] cgroup_regression_test.sh fix

[LTP] [RFC][PATCH 0/1] cgroup_regression_test.sh fix

[Cristian Marussi]

Hi

while working on this LTP tip (LTP Jan19 REL + fixes):

81e28eab1 memcg_stress_test.sh: fix memory usage
b25c719de memcg_stress_test.sh: Further cleanup + print info
a6d7d16f4 memcg_stress_test.sh: ported to newlib
6a651fe16 controllers/cpuset: cpuset.cpus file doesn't exist which triggers TFAIL
fa049ec9f controllers/cgroup_regression_test.sh: Fix two issues
a6a5caef1 (tag: 20190115, LTP_20190115) LTP 20190115

I realized 'cgroup' testcase is now wrongly reported FAIL even if NO failures
happens in the sub testcases:


Running tests.......
<<<test_start>>>
tag=cgroup stim[  125.419776] LTP: starting cgroup (            cgroup_regression_test.sh)
e=1550611029
cmdline="               cgroup_regression_test.sh"
contacts=""
analysis=exit
<<<test_output>>>
incrementing stop
cgroup_regression_test 1 TINFO: timeout per run is 0h 5m 0s
cgroup_regression_test 1 TPASS: no kernel bug was found
cgroup_regression_test 2 TPASS: notify_on_release is inherited
cgroup_regression_test 3 TCONF: CONFIG_SCHED_DEBUG is not enabled
cgroup_regression_test 4 TCONF: CONFIG_LOCKDEP is not enabled
cgroup_regression_test 5 TPASS: no kernel bug was found
cgroup_regression_test 6 TCONF: CONFIG_CGROUP_NS is NOT supported in Kernels >= 3.0
cgroup_regression_test 7 TPASS: no kernel bug was found for test 1
[  126.879398] option changes via remount are deprecated (pid=8105 comm=mount)
cgroup_regression_test 7 TWARN: skip rest of testing due possible oops triggered by reading /proc/sched_debug
cgroup_regression_test 7 TPASS: no kernel bug was found for test 2
cgroup_regression_test 8 TPASS: no kernel bug was found
cgroup_regression_test 9 TPASS: no kernel bug was found
mkdir: cannot create directory 'cgroup/0': File exists
cgroup_regression_test 10 TPASS: no kernel bug was found

Summary:
passed   8
failed   0
skipped  3
warnings 1
<<<execution_status>>>
initiation_status="ok"
duration=61 termination_type=exited termination_id=36 corefile=no
cutime=4314 cstime=5792
<<<test_end>>>
INFO: ltp-pan reported some tests FAIL
LTP Version: 20190115

       ###############################################################

            Done executing testcases.
            LTP Version:  20190115
       ###############################################################


This seems to be due to the TWARN added in 78bfa7e785e87d5e7e4e6ebd518ca9a67c78a1f3 which leads
to an exit code interpreted by ltp-pan as failure.
Is this expected ? An ltp-pan issue ? Is there something wrong in my setup ?

A possibly trivial patch follows, BUT anyway there still an unresolved problem: in
the above test run even using TINFO instead of TWARN, I'll have the whole
cgroup_regression_test.sh test case reported as TCONF, since some of the
contained sub test cases have been in fact skipped: is there any way to address
this kind of situation in the LTP framework ? (beside using TINFO everywhere)
I understand the scenario is tricky: a test case with many sub-testcases that
could each PASS/FAIL/SKIP. Any thought/guideline on this ?

Thanks

Cristian

Cristian Marussi (1):
  cgroup_regression_test.sh: Fix TWARN usage

 testcases/kernel/controllers/cgroup/cgroup_regression_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.17.1

[LTP] [PATCH 1/1] cgroup_regression_test.sh: Fix TWARN usage

[Cristian Marussi]

Using TWARN to report that a specific sub-testcase is skipped
caused the whole cgroup_regression_test.sh to be reported as
FAILING. Using TCONF instead.

Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
---
 testcases/kernel/controllers/cgroup/cgroup_regression_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
index 686860923..e197f5d3f 100755
--- a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
+++ b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
@@ -408,7 +408,7 @@ test_7_2()
 
 	grep -q -w "cpu" /proc/cgroups
 	if [ $? -ne 0 -o ! -e /proc/sched_debug ]; then
-		tst_res TWARN "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
+		tst_res TCONF "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
 		return
 	fi
 
-- 
2.17.1

[LTP] [PATCH 1/1] cgroup_regression_test.sh: Fix TWARN usage

[Petr Vorel]

Hi Cristian,

> Using TWARN to report that a specific sub-testcase is skipped
> caused the whole cgroup_regression_test.sh to be reported as
> FAILING. Using TCONF instead.

Fixes: 78bfa7e78 ("cgroup: Simplify check_kernel_bug usage")
My commit, sorry :(.

> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
> ---
>  testcases/kernel/controllers/cgroup/cgroup_regression_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

> diff --git a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
> index 686860923..e197f5d3f 100755
> --- a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
> +++ b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
> @@ -408,7 +408,7 @@ test_7_2()

>  	grep -q -w "cpu" /proc/cgroups
>  	if [ $? -ne 0 -o ! -e /proc/sched_debug ]; then
> -		tst_res TWARN "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
> +		tst_res TCONF "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
Right, TWARN is not good (IMHO TWARN is not popular in LTP testsuites due marking test failing),
but if you don't mind I'd merge it with TBROK (more appropriate in this case).

Other option would be to use TINFO and don't skip test (we're trying to find
kernel crashes, don't we), that's probably a bit crazy.

Kind regards,
Petr

[LTP] [PATCH 1/1] cgroup_regression_test.sh: Fix TWARN usage

[Cristian Marussi]

On 26/02/2019 18:42, Petr Vorel wrote:
> Hi Cristian,
> 
>> Using TWARN to report that a specific sub-testcase is skipped
>> caused the whole cgroup_regression_test.sh to be reported as
>> FAILING. Using TCONF instead.
> 
> Fixes: 78bfa7e78 ("cgroup: Simplify check_kernel_bug usage")
> My commit, sorry :(.
> 

No worries...:>

>> Signed-off-by: Cristian Marussi <cristian.marussi@arm.com>
>> ---
>>  testcases/kernel/controllers/cgroup/cgroup_regression_test.sh | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
>> diff --git a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
>> index 686860923..e197f5d3f 100755
>> --- a/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
>> +++ b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
>> @@ -408,7 +408,7 @@ test_7_2()
> 
>>  	grep -q -w "cpu" /proc/cgroups
>>  	if [ $? -ne 0 -o ! -e /proc/sched_debug ]; then
>> -		tst_res TWARN "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
>> +		tst_res TCONF "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
> Right, TWARN is not good (IMHO TWARN is not popular in LTP testsuites due marking test failing),
> but if you don't mind I'd merge it with TBROK (more appropriate in this case).

mmm...I'm NOT so sure that a FAIL should be the outcome here ... we are skipping
if cpu is not amongst cgroups or sched_debug is NOT available...BUT at this
point we have run (or skipped) 7 tests so far and there are 3 more to go...so
why can't we just TCONF ?
(beside the fact that as I said in the cover-letter one TCONF will lead to the
whole 10-subtest croup suite to be reported as TCONF...)

Regards

Cristian

> 
> Other option would be to use TINFO and don't skip test (we're trying to find
> kernel crashes, don't we), that's probably a bit crazy.
> 
> Kind regards,
> Petr
> 

[LTP] [RFC][PATCH 0/1] cgroup_regression_test.sh fix

[Petr Vorel]

Hi Cristian,

I overlooked this cover letter (read the patch [1] via patchwork and didn't expect
cover letter for single patch, I usually put info below additional ---).
> Hi

> while working on this LTP tip (LTP Jan19 REL + fixes):

> 81e28eab1 memcg_stress_test.sh: fix memory usage
> b25c719de memcg_stress_test.sh: Further cleanup + print info
> a6d7d16f4 memcg_stress_test.sh: ported to newlib
> 6a651fe16 controllers/cpuset: cpuset.cpus file doesn't exist which triggers TFAIL
> fa049ec9f controllers/cgroup_regression_test.sh: Fix two issues
> a6a5caef1 (tag: 20190115, LTP_20190115) LTP 20190115

> I realized 'cgroup' testcase is now wrongly reported FAIL even if NO failures
> happens in the sub testcases:


> Running tests.......
> <<<test_start>>>
> tag=cgroup stim[  125.419776] LTP: starting cgroup (            cgroup_regression_test.sh)
> e=1550611029
> cmdline="               cgroup_regression_test.sh"
> contacts=""
> analysis=exit
> <<<test_output>>>
> incrementing stop
> cgroup_regression_test 1 TINFO: timeout per run is 0h 5m 0s
> cgroup_regression_test 1 TPASS: no kernel bug was found
> cgroup_regression_test 2 TPASS: notify_on_release is inherited
> cgroup_regression_test 3 TCONF: CONFIG_SCHED_DEBUG is not enabled
> cgroup_regression_test 4 TCONF: CONFIG_LOCKDEP is not enabled
> cgroup_regression_test 5 TPASS: no kernel bug was found
> cgroup_regression_test 6 TCONF: CONFIG_CGROUP_NS is NOT supported in Kernels >= 3.0
> cgroup_regression_test 7 TPASS: no kernel bug was found for test 1
> [  126.879398] option changes via remount are deprecated (pid=8105 comm=mount)
> cgroup_regression_test 7 TWARN: skip rest of testing due possible oops triggered by reading /proc/sched_debug
> cgroup_regression_test 7 TPASS: no kernel bug was found for test 2
> cgroup_regression_test 8 TPASS: no kernel bug was found
> cgroup_regression_test 9 TPASS: no kernel bug was found
> mkdir: cannot create directory 'cgroup/0': File exists
> cgroup_regression_test 10 TPASS: no kernel bug was found

> Summary:
> passed   8
> failed   0
> skipped  3
> warnings 1
> <<<execution_status>>>
> initiation_status="ok"
> duration=61 termination_type=exited termination_id=36 corefile=no
> cutime=4314 cstime=5792
> <<<test_end>>>
> INFO: ltp-pan reported some tests FAIL
> LTP Version: 20190115



>             Done executing testcases.
>             LTP Version:  20190115



> This seems to be due to the TWARN added in 78bfa7e785e87d5e7e4e6ebd518ca9a67c78a1f3 which leads
> to an exit code interpreted by ltp-pan as failure.
> Is this expected ? An ltp-pan issue ? Is there something wrong in my setup ?
This is maybe surprising, but deliberate behavior.
IMHO this leads to to the fact that TWARN is fewer used than the others.
@metan: Maybe it'd make sense to change TWARN not producing FAIL.

> A possibly trivial patch follows, BUT anyway there still an unresolved problem: in
> the above test run even using TINFO instead of TWARN, I'll have the whole
> cgroup_regression_test.sh test case reported as TCONF, since some of the
> contained sub test cases have been in fact skipped: is there any way to address
> this kind of situation in the LTP framework ? (beside using TINFO everywhere)
> I understand the scenario is tricky: a test case with many sub-testcases that
> could each PASS/FAIL/SKIP. Any thought/guideline on this ?
FAIL is IMHO in TWARN, TFAIL and TBROK.

TCONF is taken as PASS, see:

cgroup_regression_test 1 TINFO: timeout per run is 0h 5m 0s
cgroup_regression_test 1 TPASS: no kernel bug was found
cgroup_regression_test 2 TPASS: notify_on_release is inherited
cgroup_regression_test 3 TPASS: no kernel bug was found
cgroup_regression_test 4 TCONF: CONFIG_LOCKDEP is not enabled
cgroup_regression_test 5 TPASS: no kernel bug was found
cgroup_regression_test 6 TCONF: CONFIG_CGROUP_NS is NOT supported in Kernels >= 3.0
cgroup_regression_test 7 TPASS: no kernel bug was found for test 1
cgroup_regression_test 7 TPASS: no kernel bug was found for test 2
cgroup_regression_test 8 TPASS: no kernel bug was found
cgroup_regression_test 9 TPASS: no kernel bug was found
cgroup_regression_test 10 TPASS: no kernel bug was found

Summary:
passed   9
failed   0
skipped  2
warnings 0
incrementing stop
<<<execution_status>>>
initiation_status="ok"
duration=92 termination_type=exited termination_id=32 corefile=no
cutime=12606 cstime=5151
<<<test_end>>>
INFO: ltp-pan reported all tests PASS


Kind regards,
Petr

[1] https://patchwork.ozlabs.org/patch/1042942/


> Cristian Marussi (1):
>   cgroup_regression_test.sh: Fix TWARN usage

>  testcases/kernel/controllers/cgroup/cgroup_regression_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

[LTP] [PATCH 1/1] cgroup_regression_test.sh: Fix TWARN usage

[Petr Vorel]

Hi Cristian,

...
> >> +++ b/testcases/kernel/controllers/cgroup/cgroup_regression_test.sh
> >> @@ -408,7 +408,7 @@ test_7_2()

> >>  	grep -q -w "cpu" /proc/cgroups
> >>  	if [ $? -ne 0 -o ! -e /proc/sched_debug ]; then
> >> -		tst_res TWARN "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
> >> +		tst_res TCONF "skip rest of testing due possible oops triggered by reading /proc/sched_debug"
> > Right, TWARN is not good (IMHO TWARN is not popular in LTP testsuites due marking test failing),
> > but if you don't mind I'd merge it with TBROK (more appropriate in this case).

> mmm...I'm NOT so sure that a FAIL should be the outcome here ... we are skipping
> if cpu is not amongst cgroups or sched_debug is NOT available...BUT at this
> point we have run (or skipped) 7 tests so far and there are 3 more to go...so
> why can't we just TCONF ?
OK I got that :) /proc/sched_debug is enabled by CONFIG_SCHED_DEBUG, os it's
kind of configuration issue. Before I thought that failing "grep -q -w "cpu"
/proc/cgroups" means failure/bug found by test, but it's not.

So I'm ok with this change.
Acked-by: Petr Vorel <pvorel@suse.cz>

> (beside the fact that as I said in the cover-letter one TCONF will lead to the
> whole 10-subtest croup suite to be reported as TCONF...)
IMHO that's not true (see my reply to cover letter).

Kind regards,
Petr

[LTP] [PATCH 1/1] cgroup_regression_test.sh: Fix TWARN usage

[Petr Vorel]

Hi Cristian,

thanks for your work, merged (with my comments in commit message).

Kind regards,
Petr