[LTP] runltp-ng hackweek experiment

[LTP] runltp-ng hackweek experiment

[Cyril Hrubis]

Hi!
I've tried to experiment with a proof of concept for new LTP test
execution framework during Hackweek[1]. And I've ended up with something
that is more or less usable and has features missing in runltp + ltp-pan
script we have now.

It's in the proof of concept state at this point and polishing it would
require more work, but it looks promising at this point.

The key desing points are:

* We use the ltp-wrapper[2] instead of ltp-pan, which runs oldlib and
  openposix testcases from a test() function from a newlib test.

  This provides us test timeouts and killing whole test process tree,
  etc. basically for free.

* The ltp-runner[3] then does the runtest files parsing, openposix test
  discovery, test execution, writing test results, etc.

  - The one intrusive change is that I've added code to both oldlib and
    newlib to support '-q' flag, so that we can query test for details
    i.e. if it's oldlib/newlib test, if it requires root, etc. This is
    would be also needed for parallel test execution, where the test
    would report if it can be executed in parallel and/or report what
    kind of system resources it monopolizes. The bottom line is that
    all testcases should understand the -q option and at least exit
    without writing any output, which is not the case at the moment.

  - The code is split into several modules, the parsers prepare list of
    tests to be executed, then there is code to execute the tests and
    store logs, then the output writers can produce logs in several
    different formats. I've experimented a bit with a better html
    output[4][5] (table sorting is not implemented yet, but planned).

  - The code is prepared for parallel test execution which is not
    implemented at the moment but basically boils down to marking
    down testcases that cannot be executed concurently. As I wrote
    I would like to keep this information inside the test code and
    export it via the -q options.

  - We miss random test execution and execute tests for defined amount
    of time at this point, but other than that the basic features are
    there.

And as usuall any comments are welcomed ;-).

[1] https://hackweek.suse.com/
[2] https://github.com/metan-ucw/ltp/blob/master/tools/runltp-ng/bin/ltp-wrapper.c
[3] https://github.com/metan-ucw/ltp/blob/master/tools/runltp-ng/bin/ltp-runner.c
[4] http://metan.ucw.cz/outgoing/ltp-openposix.html
[5] http://metan.ucw.cz/outgoing/ltp-syscalls.html

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] runltp-ng hackweek experiment

[Petr Vorel]

Hi Cyril,

> I've tried to experiment with a proof of concept for new LTP test
> execution framework during Hackweek[1]. And I've ended up with something
> that is more or less usable and has features missing in runltp + ltp-pan
> script we have now.

> It's in the proof of concept state at this point and polishing it would
> require more work, but it looks promising at this point.

It looks really interesting.

BTW while Open Posix Testcases (-o switch) are working, "normal" tests doesn't. Do I miss
something?

$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/io

Overall results:
-------------------------------------

Passed 0 out of 0


And shouldn't there be a different check as otherwise it segfaults?

$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/mm
Segmentation fault (core dumped)

--- tools/runltp-ng/bin/runner_test_filter.c
+++ tools/runltp-ng/bin/runner_test_filter.c
@@ -100,7 +100,7 @@ static void trie_insert(const char *str)
 
 	str = trie_search(str, &node);
 
-	while (*str) {
+	while (str) {
 		int key = char2key(*str);
 
 		if (key == -1)


>   - The code is split into several modules, the parsers prepare list of
>     tests to be executed, then there is code to execute the tests and
>     store logs, then the output writers can produce logs in several
>     different formats. I've experimented a bit with a better html
>     output[4][5] (table sorting is not implemented yet, but planned).

Tables looks very nice. I'd be for more features like hover highlighting a and slightly
different color for odd and even rows, but thats a detail (see
http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).


Kind regards,
Petr

[LTP] runltp-ng hackweek experiment

[Cyril Hrubis]

Hi!
> It looks really interesting.
> 
> BTW while Open Posix Testcases (-o switch) are working, "normal" tests doesn't. Do I miss
> something?
> 
> $ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/io

The -S option sets a skipfile, runtest files are passed as positional
parameters.

> Overall results:
> -------------------------------------
> 
> Passed 0 out of 0
> 
> 
> And shouldn't there be a different check as otherwise it segfaults?
> 
> $ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/mm
> Segmentation fault (core dumped)

Hmm, looks like a bug.

> --- tools/runltp-ng/bin/runner_test_filter.c
> +++ tools/runltp-ng/bin/runner_test_filter.c
> @@ -100,7 +100,7 @@ static void trie_insert(const char *str)
>  
>  	str = trie_search(str, &node);
>  
> -	while (*str) {
> +	while (str) {
>  		int key = char2key(*str);
>  
>  		if (key == -1)

Does not look correct, we are looping over the string characters here
inserting nodes into the tree. Here we likely segfault since the
trie_search() encountered unexpected character and returned NULL since I
deliberately allow only printable ascii characters in the test id. So we
should do something as:

	str = trie_search(str, &node);

	if (!str) {
		WARN("Invalid character found in '%s'", str);
		return;
	}

> >   - The code is split into several modules, the parsers prepare list of
> >     tests to be executed, then there is code to execute the tests and
> >     store logs, then the output writers can produce logs in several
> >     different formats. I've experimented a bit with a better html
> >     output[4][5] (table sorting is not implemented yet, but planned).
> 
> Tables looks very nice. I'd be for more features like hover highlighting a and slightly
> different color for odd and even rows, but thats a detail (see
> http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).

I have something like that in mind as well, the table was done in a half
of a day and I've spend most of the time looking into javascript
documentation...

But I've managed to write a table sorting function yesterday evenning, I
will push that code into my github repo soon enough.

-- 
Cyril Hrubis
chrubis@suse.cz

[LTP] runltp-ng hackweek experiment

[Petr Vorel]

Hi,

> > Tables looks very nice. I'd be for more features like hover highlighting a and slightly
> > different color for odd and even rows, but thats a detail (see
> > http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).

> I have something like that in mind as well, the table was done in a half
> of a day and I've spend most of the time looking into javascript
> documentation...
Sure, web UI development is a bit different from low level C code :-). This reporting
feature is really nice.

> But I've managed to write a table sorting function yesterday evenning, I
> will push that code into my github repo soon enough.
Great.


Kind regards,
Petr