[LTP] [RFC] Unique test case ids in runtest files ?

[LTP] [RFC] Unique test case ids in runtest files ?

[Stanislav Kholmanskikh]

Hi!

Here is a list of test cases which are referenced in
scenario_groups/default multiple times:

cve-2012-0957
cve-2016-4997
cve-2017-5669
cve-2017-6951
keyctl01
move_pages01
move_pages02
move_pages03
move_pages04
move_pages05
move_pages06
move_pages07
move_pages08
move_pages09
move_pages10
move_pages11
move_pages12
pipeio_1
#pipeio_2
pipeio_3
pipeio_4
pipeio_5
pipeio_6
#pipeio_7
pipeio_8
sched_getattr01
sched_getattr02
sched_setattr01
sysctl01
unshare01

I.e. if we run LTP with the default scenario then
 1) most of the test cases above are executed twice.
 2) keyctl01, sysctl01, unshare01 are executed twice.
    Each time they actually point to a different binary/command.

This causes some ambiguity in parsing LTP output logs/results,
especially in group 2.

The proposed solution to this problem is
 a) let tests of group 1 be defined only in one runtest file (referenced
    by the default scenario file)
 b) rename tests of group2

The issue is not new, and it actually hit me only after keyctl01, sysctl01,
unshare01 were introduced in runtest/commands.

It's no doubt, that other names are required for that keyctl01, sysctl01,
unshare01 (be it b) or another patch). However, changes in a)
are more questionable.

I'd like to hear your opinion about this approach.

Thanks a lot.
 

[LTP] [RFC PATCH 1/5] Remove the pipes runtest file

[Stanislav Kholmanskikh]

We have two almost identical runtest files:

[root@localhost runtest]# diff ipc pipes
1c1
< #DESCRIPTION:Interprocess communication stress
---
> #DESCRIPTION:IPC pipes stress
30,32d29
<
< sem01 sem01
< sem02 sem02
[root@localhost runtest]#

Since both are part of the default scenario, their common tests
are executed twice by LTP run. Let's fix that by removing the pipes file.
If one needs to execute only pipeio* test cases, he/she will still be able
to do it by executing the ipc file.

Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
---
 runtest/pipes           | 29 -----------------------------
 scenario_groups/default |  1 -
 2 files changed, 30 deletions(-)
 delete mode 100644 runtest/pipes

diff --git a/runtest/pipes b/runtest/pipes
deleted file mode 100644
index 900c9a2..0000000
--- a/runtest/pipes
+++ /dev/null
@@ -1,29 +0,0 @@
-#DESCRIPTION:IPC pipes stress
-# These tests use tests/pipeio to put pipes (named or unnamed) through a workout
-#
-pipeio_1 pipeio -T pipeio_1 -c 5 -s 4090 -i 100 -b -f x80
-# spawns 5 children to write 100 chunks of 4090 bytes to a named pipe
-# using blocking I/O
-#pipeio_2 pipeio -T pipeio_2 -c 5 -s 4090 -i 100 -f x80
-# spawns 5 children to write 100 chunks of 4090 bytes to a named pipe
-# using non-blocking I/O
-# This test hits EAGAIN, which pipeio doesn't handle at the moment
-pipeio_3 pipeio -T pipeio_3 -c 5 -s 4090 -i 100 -u -b -f x80
-# spawns 5 children to write 100 chunks of 4090 bytes to an unnamed pipe
-# using blocking I/O
-pipeio_4 pipeio -T pipeio_4 -c 5 -s 4090 -i 100 -u -f x80
-# spawns 5 children to write 100 chunks of 4090 bytes to an unnamed pipe
-# using non-blocking I/O
-pipeio_5 pipeio -T pipeio_5 -c 5 -s 5000 -i 10 -b -f x80
-# spawns 5 children to write 10 chunks of 5000 bytes to a named pipe
-# using blocking I/O
-pipeio_6 pipeio -T pipeio_6 -c 5 -s 5000 -i 10 -b -u -f x80
-# spawns 5 children to write 10 chunks of 5000 bytes to an unnamed pipe
-# using blocking I/O
-#pipeio_7 pipeio -T pipeio_7 -c 5 -s 5000 -i 10 -f x80
-# spawns 5 children to write 10 chunks of 5000 bytes to a named pipe
-# using non-blocking I/O
-# This test hits EAGAIN, which pipeio doesn't handle at the moment
-pipeio_8 pipeio -T pipeio_8 -c 5 -s 5000 -i 10 -u -f x80
-# spawns 5 children to write 10 chunks of 5000 bytes to an unnamed pipe
-# using non-blocking I/O
diff --git a/scenario_groups/default b/scenario_groups/default
index 5658a61..a0179c2 100644
--- a/scenario_groups/default
+++ b/scenario_groups/default
@@ -27,7 +27,6 @@ hyperthreading
 kernel_misc
 modules
 fs_ext4
-pipes
 can
 cpuhotplug
 net.ipv6_lib
-- 
1.8.3.1

[LTP] [RFC PATCH 2/5] runtest/syscalls: rename CVE test cases

[Stanislav Kholmanskikh]

Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
---
 runtest/syscalls | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/runtest/syscalls b/runtest/syscalls
index 0d0be77..b68eda7 100644
--- a/runtest/syscalls
+++ b/runtest/syscalls
@@ -950,8 +950,8 @@ renameat202 renameat202 -i 10
 request_key01 request_key01
 request_key02 request_key02
 request_key03 request_key03
-cve-2017-6951 cve-2017-6951
 request_key04 request_key04
+request_key05 cve-2017-6951
 
 rmdir01 rmdir01
 rmdir02 rmdir02
@@ -1177,7 +1177,7 @@ setsid01 setsid01
 
 setsockopt01 setsockopt01
 setsockopt02 setsockopt02
-cve-2016-4997 cve-2016-4997
+setsockopt03 cve-2016-4997
 
 settimeofday01 settimeofday01
 settimeofday02 settimeofday02
@@ -1195,7 +1195,7 @@ setxattr03 setxattr03
 
 shmat01 shmat01
 shmat02 shmat02
-cve-2017-5669 cve-2017-5669
+shmat03 cve-2017-5669
 
 shmctl01 shmctl01
 shmctl02 shmctl02
@@ -1390,7 +1390,7 @@ umask01 umask01
 uname01 uname01
 uname02 uname02
 uname03 uname03
-cve-2012-0957 cve-2012-0957
+uname04 cve-2012-0957
 
 unlink01 symlink01 -T unlink01
 unlink05 unlink05
-- 
1.8.3.1

[LTP] [RFC PATCH 3/5] runtest/sched: removed sched_{set, get}attr test cases

[Stanislav Kholmanskikh]

All these test cases do not fall into the category of
"Scheduler Stress Tests", so their better location is in runtest/syscalls,
where they are already defined.

Both runtest/sched and runtest/syscalls are part of the default
scenario, these test cases will be executed in a default run anyway.

Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
---
 runtest/sched | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/runtest/sched b/runtest/sched
index 774d11f..bfc4f27 100644
--- a/runtest/sched
+++ b/runtest/sched
@@ -9,10 +9,6 @@ trace_sched01		trace_sched -c 1
 hackbench01 hackbench 50 process 1000
 hackbench02 hackbench 20 thread 1000
 
-sched_setattr01 sched_setattr01
-sched_getattr01 sched_getattr01
-sched_getattr02 sched_getattr02
-
 sched_cli_serv run_sched_cliserv.sh
 # Run this stress test for 2 minutes
 sched_stress sched_stress.sh
-- 
1.8.3.1

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Stanislav Kholmanskikh]

runtest/syscalls already defines tests with these ids, but there
they point to different commands. In order to avoid name conflicts
we need to rename tests either in runtest/commands or in runtest/syscalls.
Since the tests in runtest/syscalls are much older, I chose to rename
the tests in runtest/commands.

Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
---
 runtest/commands | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/runtest/commands b/runtest/commands
index ee7e9f4..f27d2fa 100644
--- a/runtest/commands
+++ b/runtest/commands
@@ -37,7 +37,7 @@ which01 which01.sh
 lsmod01 lsmod01.sh
 insmod01 insmod01.sh
 wc01 wc01.sh
-keyctl01 keyctl01.sh
+keyctl_01 keyctl01.sh
 gdb01 gdb01.sh
-unshare01 unshare01.sh
-sysctl01 sysctl01.sh
+unshare_01 unshare01.sh
+sysctl_01 sysctl01.sh
-- 
1.8.3.1

[LTP] [RFC PATCH 5/5] runtest/numa: remove move_pages* test cases

[Stanislav Kholmanskikh]

since they are already defined in runtest/syscalls.

Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
---
 runtest/numa | 12 ------------
 1 file changed, 12 deletions(-)

diff --git a/runtest/numa b/runtest/numa
index 12aedbb..510c353 100644
--- a/runtest/numa
+++ b/runtest/numa
@@ -1,13 +1 @@
 Numa-testcases numa01.sh
-move_pages01 move_pages01
-move_pages02 move_pages02
-move_pages03 move_pages03
-move_pages04 move_pages04
-move_pages05 move_pages05
-move_pages06 move_pages06
-move_pages07 move_pages07
-move_pages08 move_pages08
-move_pages09 move_pages09
-move_pages10 move_pages10
-move_pages11 move_pages11
-move_pages12 move_pages12
-- 
1.8.3.1

[LTP] [RFC] Unique test case ids in runtest files ?

[Cyril Hrubis]

Hi!
> I.e. if we run LTP with the default scenario then
>  1) most of the test cases above are executed twice.

This is problem that would be hard to fix. We would have to rethink how
we organize testcases in the runtest files in the long term to
completely avoid situations like this.

>  2) keyctl01, sysctl01, unshare01 are executed twice.
>     Each time they actually point to a different binary/command.

This is a bug that has to be fixed.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 2/5] runtest/syscalls: rename CVE test cases

[Cyril Hrubis]

Hi!
Can we do this the other way around?

I.e. rename the source files to match the syscall name and then patch
the cve runtest file to map the cve-* identifiers to the test names?

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 3/5] runtest/sched: removed sched_{set, get}attr test cases

[Cyril Hrubis]

Hi!
Looks good, acked.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Cyril Hrubis]

Hi!
> runtest/syscalls already defines tests with these ids, but there
> they point to different commands. In order to avoid name conflicts
> we need to rename tests either in runtest/commands or in runtest/syscalls.
> Since the tests in runtest/syscalls are much older, I chose to rename
> the tests in runtest/commands.
> 
> Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
> ---
>  runtest/commands | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/runtest/commands b/runtest/commands
> index ee7e9f4..f27d2fa 100644
> --- a/runtest/commands
> +++ b/runtest/commands
> @@ -37,7 +37,7 @@ which01 which01.sh
>  lsmod01 lsmod01.sh
>  insmod01 insmod01.sh
>  wc01 wc01.sh
> -keyctl01 keyctl01.sh
> +keyctl_01 keyctl01.sh
>  gdb01 gdb01.sh
> -unshare01 unshare01.sh
> -sysctl01 sysctl01.sh
> +unshare_01 unshare01.sh
> +sysctl_01 sysctl01.sh

Hmm, that's a bit hacky. Maybe we should add a _sh suffix to all the
testcases in commands so that it's clear that these are shell scripts.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 5/5] runtest/numa: remove move_pages* test cases

[Cyril Hrubis]

Hi!
> since they are already defined in runtest/syscalls.
> 
> Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
> ---
>  runtest/numa | 12 ------------
>  1 file changed, 12 deletions(-)
> 
> diff --git a/runtest/numa b/runtest/numa
> index 12aedbb..510c353 100644
> --- a/runtest/numa
> +++ b/runtest/numa
> @@ -1,13 +1 @@
>  Numa-testcases numa01.sh
> -move_pages01 move_pages01
> -move_pages02 move_pages02
> -move_pages03 move_pages03
> -move_pages04 move_pages04
> -move_pages05 move_pages05
> -move_pages06 move_pages06
> -move_pages07 move_pages07
> -move_pages08 move_pages08
> -move_pages09 move_pages09
> -move_pages10 move_pages10
> -move_pages11 move_pages11
> -move_pages12 move_pages12

Or we may remove the numa runtest file from the default scenario, given
that the move_pages tests are in syscalls and the numa01.sh is known to
be broken there is a little value in keeping it in default scenario.

And once I finish the numa01.sh rewrite, we will get set_mempolicy
syscall tests in syscalls runtest file anyways, so we may keep the numa
to be a separate runtest file for a case that someone wants to run only
numa related tests.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 1/5] Remove the pipes runtest file

[Cyril Hrubis]

Hi!
I guess this is OK, acked.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 2/5] runtest/syscalls: rename CVE test cases

[Stanislav Kholmanskikh]

On 10/02/2018 03:53 PM, Cyril Hrubis wrote:
> Hi!
> Can we do this the other way around?
> 
> I.e. rename the source files to match the syscall name and then patch
> the cve runtest file to map the cve-* identifiers to the test names?
> 

Yes. No objection at all.

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Stanislav Kholmanskikh]

On 10/02/2018 04:03 PM, Cyril Hrubis wrote:
> Hi!
>> runtest/syscalls already defines tests with these ids, but there
>> they point to different commands. In order to avoid name conflicts
>> we need to rename tests either in runtest/commands or in runtest/syscalls.
>> Since the tests in runtest/syscalls are much older, I chose to rename
>> the tests in runtest/commands.
>>
>> Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
>> ---
>>  runtest/commands | 6 +++---
>>  1 file changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/runtest/commands b/runtest/commands
>> index ee7e9f4..f27d2fa 100644
>> --- a/runtest/commands
>> +++ b/runtest/commands
>> @@ -37,7 +37,7 @@ which01 which01.sh
>>  lsmod01 lsmod01.sh
>>  insmod01 insmod01.sh
>>  wc01 wc01.sh
>> -keyctl01 keyctl01.sh
>> +keyctl_01 keyctl01.sh
>>  gdb01 gdb01.sh
>> -unshare01 unshare01.sh
>> -sysctl01 sysctl01.sh
>> +unshare_01 unshare01.sh
>> +sysctl_01 sysctl01.sh
> 
> Hmm, that's a bit hacky. Maybe we should add a _sh suffix to all the
> testcases in commands so that it's clear that these are shell scripts.
> 

As if

keyctl01 keyctl01.sh

was changed to

keyctl01_sh keyctl01.sh

Correct?

[LTP] [RFC PATCH 5/5] runtest/numa: remove move_pages* test cases

[Stanislav Kholmanskikh]

On 10/02/2018 04:09 PM, Cyril Hrubis wrote:
> Hi!
>> since they are already defined in runtest/syscalls.
>>
>> Signed-off-by: Stanislav Kholmanskikh <stanislav.kholmanskikh@oracle.com>
>> ---
>>  runtest/numa | 12 ------------
>>  1 file changed, 12 deletions(-)
>>
>> diff --git a/runtest/numa b/runtest/numa
>> index 12aedbb..510c353 100644
>> --- a/runtest/numa
>> +++ b/runtest/numa
>> @@ -1,13 +1 @@
>>  Numa-testcases numa01.sh
>> -move_pages01 move_pages01
>> -move_pages02 move_pages02
>> -move_pages03 move_pages03
>> -move_pages04 move_pages04
>> -move_pages05 move_pages05
>> -move_pages06 move_pages06
>> -move_pages07 move_pages07
>> -move_pages08 move_pages08
>> -move_pages09 move_pages09
>> -move_pages10 move_pages10
>> -move_pages11 move_pages11
>> -move_pages12 move_pages12
> 
> Or we may remove the numa runtest file from the default scenario, given
> that the move_pages tests are in syscalls and the numa01.sh is known to
> be broken there is a little value in keeping it in default scenario.
> 
> And once I finish the numa01.sh rewrite, we will get set_mempolicy
> syscall tests in syscalls runtest file anyways, so we may keep the numa
> to be a separate runtest file for a case that someone wants to run only
> numa related tests.
> 

Alright, I will remove the numa runtest file from the default scenario
in the next series.

Thank you.

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Cyril Hrubis]

Hi!
> As if
> 
> keyctl01 keyctl01.sh
> 
> was changed to
> 
> keyctl01_sh keyctl01.sh
> 
> Correct?

Or we can, for the cases where command has the same name as syscall,
increment the numbers for the command tests by 50, that should avoid the
collisions.

So the keyctl01.sh will become keyctl51.sh.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Stanislav Kholmanskikh]

Hi

On 10/03/2018 10:42 AM, Cyril Hrubis wrote:
> Hi!
>> As if
>>
>> keyctl01 keyctl01.sh
>>
>> was changed to
>>
>> keyctl01_sh keyctl01.sh
>>
>> Correct?
> 
> Or we can, for the cases where command has the same name as syscall,
> increment the numbers for the command tests by 50, that should avoid the
> collisions.
> 
> So the keyctl01.sh will become keyctl51.sh.
> 

Maybe we better stick to your previous proposal, i.e. adding _sh to the
end of a name (in runtest files)? In my opinion, it's more clear and
straightforward, i.e. if one has a look at a runtest file, it will take
little time to understand why some tests have _sh at the end. Plus it
doesn't require renaming of the sh test cases.

Shifting the numbering by a big number looks a bit tricky.

[LTP] [RFC PATCH 4/5] runtest/commands: adjust names for keyctl01, sysctl01, unshare01

[Cyril Hrubis]

Hi!
> Maybe we better stick to your previous proposal, i.e. adding _sh to the
> end of a name (in runtest files)? In my opinion, it's more clear and
> straightforward, i.e. if one has a look at a runtest file, it will take
> little time to understand why some tests have _sh at the end. Plus it
> doesn't require renaming of the sh test cases.
> 
> Shifting the numbering by a big number looks a bit tricky.

Ok.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] [RFC PATCH 2/5] runtest/syscalls: rename CVE test cases

[Stanislav Kholmanskikh]

On 10/02/2018 03:53 PM, Cyril Hrubis wrote:
> Hi!
> Can we do this the other way around?
> 
> I.e. rename the source files to match the syscall name and then patch
> the cve runtest file to map the cve-* identifiers to the test names?
> 

By the way,

should I rename them right into testcases/cve or move to a corresponding
directory in testcases/kernel/syscalls?

I.e.

testcases/cve/cve-2017-6951.c -> testcases/cve/request_key05.c

or

testcases/cve/cve-2017-6951.c ->
testcases/kernel/syscalls/request_key/request_key05.c

?

I assume you propose the latter option, correct?


Thanks.

[LTP] [RFC PATCH 2/5] runtest/syscalls: rename CVE test cases

[Cyril Hrubis]

Hi!
> should I rename them right into testcases/cve or move to a corresponding
> directory in testcases/kernel/syscalls?
> 
> I.e.
> 
> testcases/cve/cve-2017-6951.c -> testcases/cve/request_key05.c
> 
> or
> 
> testcases/cve/cve-2017-6951.c ->
> testcases/kernel/syscalls/request_key/request_key05.c
> 
> ?
> 
> I assume you propose the latter option, correct?

Yes.

-- 
Cyril Hrubis
chrubis@suse.cz