[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Tim.Bird]

Hey Daniel and Liu,

Here is a test that runs an individual LTP test.  It's now in my master
branch.  The reason for pointing this out is to get your feedback
on the approach of writing a test that uses a sub-set of LTP.
Note the use of 'ftc run-test' with the '-p  b' parameter, to build
LTP if needed.  This avoids having to share build script details
between this test and LTP.

I'm thinking this approach might be useful to split out
the OpenPosix and realtime subsets of LTP.  Also, it might
work for handling setup and cleanup for subtest groups or
individual tests, like the network-related tests or smack.

For example, instead of doing smack-specific machine setup
in the LTP test, I think we should make something like
Functional.LTP-smack, use the techniques demonstrated here
for building it.  But then have  pre_test, deply, run, and processing
phases be customized for that test's needs. in Functional.LTP-smack's
fuego_test.sh.

Let me know what you think.
   -- Tim

Patch follows:
----------------------------
LTP is a ginormous collection of tests.  When something fails,
it is very handy to be able to run a single test in isolation.
This test allows you to do that.  To run an individual test,
just add a new spec to the spec file (or, when ftc supports
setting a test variable on the command line, use that to
specify the name of the individual test program to run).

This test also demonstrates how one test can use ftc and phases
to execute just the build of another test.  And how one test
can re-use the build directory of another test.

Signed-off-by: Tim Bird <tim.bird@sony.com>
---
 engine/tests/Functional.LTP_one_test/fuego_test.sh | 40 +++++++++++++++++
 engine/tests/Functional.LTP_one_test/parser.py     | 51 ++++++++++++++++++++++
 engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
 engine/tests/Functional.LTP_one_test/test.yaml     | 31 +++++++++++++
 4 files changed, 134 insertions(+)
 create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
 create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
 create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
 create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml

diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh b/engine/tests/Functional.LTP_one_test/fuego_test.sh
new file mode 100755
index 0000000..2ebcfc5
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
@@ -0,0 +1,40 @@
+one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
+
+# Don't allow jobs to share build directories
+# the "test_successfully_built" flag for one spec
+# is not accurate for another spec, which has a different
+# individual program to deploy.
+FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
+
+function test_pre_check {
+    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
+}
+
+function test_build {
+    # check for LTP build directory
+    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed s/$TESTSPEC/default/)
+
+    # if not already built, build LTP
+    if [ ! -e ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ; then
+        echo "Building parent LTP test..."
+	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
+        # NOTE: vars used in ftc run-test should not leak into this environment
+        # that is, none of our test vars should have changed.
+    fi
+
+    cp ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
+}
+
+function test_deploy {
+	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
+}
+
+function test_run {
+    # FIXTHIS - deal with arguments better
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
+        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
+}
+
+function test_processing {
+    return
+}
diff --git a/engine/tests/Functional.LTP_one_test/parser.py b/engine/tests/Functional.LTP_one_test/parser.py
new file mode 100755
index 0000000..fe2336f
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/parser.py
@@ -0,0 +1,51 @@
+#!/usr/bin/python
+# See common.py for description of command-line arguments
+
+import os
+import sys
+import collections
+import re
+
+sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
+import common as plib
+
+results = collections.OrderedDict()
+
+# LTP results come in largely 2 formats:
+# sched_getattr01    1  TPASS  :  attributes were read back correctly
+# request_key01.c:49: PASS: request_key() succeed
+
+log_lines = open(plib.TEST_LOG, "r").readlines()
+re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
+
+testcase_count = 1
+for line in log_lines:
+    #print "line=", line
+    if ".c:" in line:
+        parts = line.split(":")
+        filename = parts[0]
+        line_no = parts[1]
+        testcase_num = str(testcase_count)
+        testcase_count += 1
+        result = parts[2].strip()
+        message = parts[3].strip()
+    else:
+        m = re.match(re_pattern, line)
+        #print "m=", m
+        if m:
+            test_name = m.group(1)
+            testcase_num = m.group(2)
+            result = m.group(3)[1:] # strip off the leading 'T'
+            message = m.group(4).strip()
+
+    # translate to Fuego results
+    if result == 'PASS':
+        results[testcase_num] = 'PASS'
+    elif result == 'FAIL':
+        results[testcase_num] = 'FAIL'
+    elif result == 'CONF':
+        results[testcase_num] = 'SKIP'
+    else:
+        results[testcase_num] = 'ERR'
+
+sys.exit(plib.process(results))
diff --git a/engine/tests/Functional.LTP_one_test/spec.json b/engine/tests/Functional.LTP_one_test/spec.json
new file mode 100644
index 0000000..9f0b325
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/spec.json
@@ -0,0 +1,12 @@
+{
+    "testName": "Functional.LTP_one_test",
+    "specs": {
+        "default": {
+            "TEST":"brk01"
+        },
+        "fanotify07": {
+            "TEST":"fanotify07"
+        }
+    }
+}
+
diff --git a/engine/tests/Functional.LTP_one_test/test.yaml b/engine/tests/Functional.LTP_one_test/test.yaml
new file mode 100644
index 0000000..765c98d
--- /dev/null
+++ b/engine/tests/Functional.LTP_one_test/test.yaml
@@ -0,0 +1,31 @@
+fuego_package_version: 1
+name: Functional.LTP_one_test
+description: |
+    This is a convenience test.  LTP is a large collection of tests, which
+    takes a lot of infrastructure to set up and execute.  Sometimes it is
+    handy to run a single LTP test program by itself.  This Fuego test
+    exists to allow you to easily do that.
+
+    To run an individual test, add a new spec for it in the spec.json file.
+    Specify the individual test name to run in the TEST variable.  By
+    convention the name of the spec should match the name of the test
+    to run.
+license: multiple (same as LTP)
+author: Tim Bird
+maintainer: Tim Bird <tim.bird@sony.com>
+version: 1.0
+fuego_release: 1
+type: Functional
+tags: ['general']
+params:
+    - TEST:
+        description: Name of the individual LTP test program to run.
+        example: fanotify07
+        optional: no
+    - ARGS:
+        description: Command line arguments for the LTP test program.
+        example: -s foo
+        optional: yes
+data_files:
+    - spec.json
+    - parser.py
-- 
1.9.1

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Saturday, April 7, 2018 4:11 AM
> To: liuwl.fnst@cn.fujitsu.com; daniel.sangorrin@toshiba.co.jp;
> fuego@lists.linuxfoundation.org
> Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP test
> 
> Hey Daniel and Liu,
> 
> Here is a test that runs an individual LTP test.  It's now in my master
> branch.  The reason for pointing this out is to get your feedback
> on the approach of writing a test that uses a sub-set of LTP.

Let me add some background for everyone to understand.

- On Functional.LTP, it is possible to run sub-sets of LTP by using a skip list. In other words,
you first select the test sets (groups of test cases) that you want to run, and then you 
apply a skip list to mask the execution of some of the test cases. Let's call this a blacklisting
approach. The main advantage is that you only need to know the name of the test cases
that you want to skip.
- On Functional.LTP_one_test, it is possible to specify what test case to execute directly. For that
reason there is no need to create any skip list. The main advantage is that it allows the user
to quickly run any of the tests inside LTP.

Let me jump into one technical detail though. I see that you are calling the test cases directly
from /testcases/bin/. I don't think that works in general because some of the test cases
require a bit of setup before they are executed. Fortunately, this is already taken care by
runltp through the "-s" option.
    $ ./runltp -q -f math -s abs01
    [Note] remove -q to see the test output as well

> Note the use of 'ftc run-test' with the '-p  b' parameter, to build
> LTP if needed.  This avoids having to share build script details
> between this test and LTP.

Maybe it's because the recent update broke everything, but "ftc run-test" and "ftc build-jobs" didn't work on my machine.
Other than that, I like the ability to use "ftc run-test" within the tests. In this case for the building phase, but I want to use
it for creating background loads as well. 

Do you think there could be some conflict with the nesting of tests such as variable
overwriting (Fuego using all these global shell variables is a bit scary) or deadlocks? 

> I'm thinking this approach might be useful to split out
> the OpenPosix and realtime subsets of LTP.  
Sounds good.

> Also, it might
> work for handling setup and cleanup for subtest groups or
> individual tests, like the network-related tests or smack.
> For example, instead of doing smack-specific machine setup
> in the LTP test, I think we should make something like
> Functional.LTP-smack, use the techniques demonstrated here
> for building it.  But then have  pre_test, deply, run, and processing
> phases be customized for that test's needs. in Functional.LTP-smack's
> fuego_test.sh.

Yeah, let's split corner cases. Otherwise things will get too complicated to handle.

Thanks,
Daniel

> 
> Let me know what you think.
>    -- Tim
> 
> Patch follows:
> ----------------------------
> LTP is a ginormous collection of tests.  When something fails,
> it is very handy to be able to run a single test in isolation.
> This test allows you to do that.  To run an individual test,
> just add a new spec to the spec file (or, when ftc supports
> setting a test variable on the command line, use that to
> specify the name of the individual test program to run).
> 
> This test also demonstrates how one test can use ftc and phases
> to execute just the build of another test.  And how one test
> can re-use the build directory of another test.
> 
> Signed-off-by: Tim Bird <tim.bird@sony.com>
> ---
>  engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> +++++++++++++++++
>  engine/tests/Functional.LTP_one_test/parser.py     | 51
> ++++++++++++++++++++++
>  engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
>  engine/tests/Functional.LTP_one_test/test.yaml     | 31 +++++++++++++
>  4 files changed, 134 insertions(+)
>  create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
>  create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
>  create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
>  create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
> 
> diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> new file mode 100755
> index 0000000..2ebcfc5
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> @@ -0,0 +1,40 @@
> +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> +
> +# Don't allow jobs to share build directories
> +# the "test_successfully_built" flag for one spec
> +# is not accurate for another spec, which has a different
> +# individual program to deploy.
> +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> +
> +function test_pre_check {
> +    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
> +}
> +
> +function test_build {
> +    # check for LTP build directory
> +    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> s/$TESTSPEC/default/)
> +
> +    # if not already built, build LTP
> +    if [ ! -e ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ;
> then
> +        echo "Building parent LTP test..."
> +	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> +        # NOTE: vars used in ftc run-test should not leak into this environment
> +        # that is, none of our test vars should have changed.
> +    fi
> +
> +    cp ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> +}
> +
> +function test_deploy {
> +	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> +}
> +
> +function test_run {
> +    # FIXTHIS - deal with arguments better
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> +        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> +}
> +
> +function test_processing {
> +    return
> +}
> diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> b/engine/tests/Functional.LTP_one_test/parser.py
> new file mode 100755
> index 0000000..fe2336f
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/parser.py
> @@ -0,0 +1,51 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os
> +import sys
> +import collections
> +import re
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +results = collections.OrderedDict()
> +
> +# LTP results come in largely 2 formats:
> +# sched_getattr01    1  TPASS  :  attributes were read back correctly
> +# request_key01.c:49: PASS: request_key() succeed
> +
> +log_lines = open(plib.TEST_LOG, "r").readlines()
> +re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> +
> +testcase_count = 1
> +for line in log_lines:
> +    #print "line=", line
> +    if ".c:" in line:
> +        parts = line.split(":")
> +        filename = parts[0]
> +        line_no = parts[1]
> +        testcase_num = str(testcase_count)
> +        testcase_count += 1
> +        result = parts[2].strip()
> +        message = parts[3].strip()
> +    else:
> +        m = re.match(re_pattern, line)
> +        #print "m=", m
> +        if m:
> +            test_name = m.group(1)
> +            testcase_num = m.group(2)
> +            result = m.group(3)[1:] # strip off the leading 'T'
> +            message = m.group(4).strip()
> +
> +    # translate to Fuego results
> +    if result == 'PASS':
> +        results[testcase_num] = 'PASS'
> +    elif result == 'FAIL':
> +        results[testcase_num] = 'FAIL'
> +    elif result == 'CONF':
> +        results[testcase_num] = 'SKIP'
> +    else:
> +        results[testcase_num] = 'ERR'
> +
> +sys.exit(plib.process(results))
> diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> b/engine/tests/Functional.LTP_one_test/spec.json
> new file mode 100644
> index 0000000..9f0b325
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/spec.json
> @@ -0,0 +1,12 @@
> +{
> +    "testName": "Functional.LTP_one_test",
> +    "specs": {
> +        "default": {
> +            "TEST":"brk01"
> +        },
> +        "fanotify07": {
> +            "TEST":"fanotify07"
> +        }
> +    }
> +}
> +
> diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> b/engine/tests/Functional.LTP_one_test/test.yaml
> new file mode 100644
> index 0000000..765c98d
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> @@ -0,0 +1,31 @@
> +fuego_package_version: 1
> +name: Functional.LTP_one_test
> +description: |
> +    This is a convenience test.  LTP is a large collection of tests, which
> +    takes a lot of infrastructure to set up and execute.  Sometimes it is
> +    handy to run a single LTP test program by itself.  This Fuego test
> +    exists to allow you to easily do that.
> +
> +    To run an individual test, add a new spec for it in the spec.json file.
> +    Specify the individual test name to run in the TEST variable.  By
> +    convention the name of the spec should match the name of the test
> +    to run.
> +license: multiple (same as LTP)
> +author: Tim Bird
> +maintainer: Tim Bird <tim.bird@sony.com>
> +version: 1.0
> +fuego_release: 1
> +type: Functional
> +tags: ['general']
> +params:
> +    - TEST:
> +        description: Name of the individual LTP test program to run.
> +        example: fanotify07
> +        optional: no
> +    - ARGS:
> +        description: Command line arguments for the LTP test program.
> +        example: -s foo
> +        optional: yes
> +data_files:
> +    - spec.json
> +    - parser.py
> --
> 1.9.1
> 

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Liu, Wenlong]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> Sent: Tuesday, April 10, 2018 10:38 AM
> To: Tim.Bird@sony.com; Liu, Wenlong/刘 文龙 <liuwl.fnst@cn.fujitsu.com>;
> fuego@lists.linuxfoundation.org
> Subject: RE: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> test
> 
> Hi Tim,
> 
> > -----Original Message-----
> > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> > Sent: Saturday, April 7, 2018 4:11 AM
> > To: liuwl.fnst@cn.fujitsu.com; daniel.sangorrin@toshiba.co.jp;
> > fuego@lists.linuxfoundation.org
> > Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP
> > test
> >
> > Hey Daniel and Liu,
> >
> > Here is a test that runs an individual LTP test.  It's now in my
> > master branch.  The reason for pointing this out is to get your
> > feedback on the approach of writing a test that uses a sub-set of LTP.
> 
> Let me add some background for everyone to understand.
> 
> - On Functional.LTP, it is possible to run sub-sets of LTP by using a skip
> list. In other words, you first select the test sets (groups of test cases)
> that you want to run, and then you apply a skip list to mask the execution
> of some of the test cases. Let's call this a blacklisting approach. The
> main advantage is that you only need to know the name of the test cases
> that you want to skip.
> - On Functional.LTP_one_test, it is possible to specify what test case to
> execute directly. For that reason there is no need to create any skip list.
> The main advantage is that it allows the user to quickly run any of the
> tests inside LTP.
> 
> Let me jump into one technical detail though. I see that you are calling
> the test cases directly from /testcases/bin/. I don't think that works in
> general because some of the test cases require a bit of setup before they
> are executed. Fortunately, this is already taken care by runltp through
> the "-s" option.
>     $ ./runltp -q -f math -s abs01
>     [Note] remove -q to see the test output as well

Yes, Daniel, I have the same thought with you.

> > Note the use of 'ftc run-test' with the '-p  b' parameter, to build
> > LTP if needed.  This avoids having to share build script details
> > between this test and LTP.
> 
> Maybe it's because the recent update broke everything, but "ftc run-test"
> and "ftc build-jobs" didn't work on my machine.
> Other than that, I like the ability to use "ftc run-test" within the tests.
> In this case for the building phase, but I want to use it for creating
> background loads as well.
> 
> Do you think there could be some conflict with the nesting of tests such
> as variable overwriting (Fuego using all these global shell variables is
> a bit scary) or deadlocks?
> 
> > I'm thinking this approach might be useful to split out the OpenPosix
> > and realtime subsets of LTP.
> Sounds good.
> 
> > Also, it might
> > work for handling setup and cleanup for subtest groups or
> > individual tests, like the network-related tests or smack.
> > For example, instead of doing smack-specific machine setup
> > in the LTP test, I think we should make something like
> > Functional.LTP-smack, use the techniques demonstrated here
> > for building it.  But then have  pre_test, deply, run, and processing
> > phases be customized for that test's needs. in Functional.LTP-smack's
> > fuego_test.sh.
> 
> Yeah, let's split corner cases. Otherwise things will get too complicated
> to handle.
> 
> Thanks,
> Daniel
> 
> >
> > Let me know what you think.
> >    -- Tim
> >
> > Patch follows:
> > ----------------------------
> > LTP is a ginormous collection of tests.  When something fails,
> > it is very handy to be able to run a single test in isolation.
> > This test allows you to do that.  To run an individual test,
> > just add a new spec to the spec file (or, when ftc supports
> > setting a test variable on the command line, use that to
> > specify the name of the individual test program to run).
> >
> > This test also demonstrates how one test can use ftc and phases
> > to execute just the build of another test.  And how one test
> > can re-use the build directory of another test.
> >
> > Signed-off-by: Tim Bird <tim.bird@sony.com>
> > ---
> >  engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> > +++++++++++++++++
> >  engine/tests/Functional.LTP_one_test/parser.py     | 51
> > ++++++++++++++++++++++
> >  engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
> >  engine/tests/Functional.LTP_one_test/test.yaml     | 31
> +++++++++++++
> >  4 files changed, 134 insertions(+)
> >  create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
> >  create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
> >  create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
> >  create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
> >
> > diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > new file mode 100755
> > index 0000000..2ebcfc5
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> > @@ -0,0 +1,40 @@
> > +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> > +
> > +# Don't allow jobs to share build directories
> > +# the "test_successfully_built" flag for one spec
> > +# is not accurate for another spec, which has a different
> > +# individual program to deploy.
> > +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> > +
> > +function test_pre_check {
> > +    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST
> > +}
> > +
> > +function test_build {
> > +    # check for LTP build directory
> > +    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> > s/$TESTSPEC/default/)
> > +
> > +    # if not already built, build LTP
> > +    if [ ! -e
> ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ;
> > then
> > +        echo "Building parent LTP test..."
> > +	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> > +        # NOTE: vars used in ftc run-test should not leak into this
> environment
> > +        # that is, none of our test vars should have changed.
> > +    fi
> > +
> > +    cp
> ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> > +}
> > +
> > +function test_deploy {
> > +	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/
> > +}
> > +
> > +function test_run {
> > +    # FIXTHIS - deal with arguments better
> > +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> > +        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> > +}
> > +
> > +function test_processing {
> > +    return
> > +}
> > diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> > b/engine/tests/Functional.LTP_one_test/parser.py
> > new file mode 100755
> > index 0000000..fe2336f
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/parser.py
> > @@ -0,0 +1,51 @@
> > +#!/usr/bin/python
> > +# See common.py for description of command-line arguments
> > +
> > +import os
> > +import sys
> > +import collections
> > +import re
> > +
> > +sys.path.insert(0, os.environ['FUEGO_CORE'] +
> '/engine/scripts/parser')
> > +import common as plib
> > +
> > +results = collections.OrderedDict()
> > +
> > +# LTP results come in largely 2 formats:
> > +# sched_getattr01    1  TPASS  :  attributes were read back correctly
> > +# request_key01.c:49: PASS: request_key() succeed
> > +
> > +log_lines = open(plib.TEST_LOG, "r").readlines()
> > +re_pattern = r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> > +
> > +testcase_count = 1
> > +for line in log_lines:
> > +    #print "line=", line
> > +    if ".c:" in line:
> > +        parts = line.split(":")
> > +        filename = parts[0]
> > +        line_no = parts[1]
> > +        testcase_num = str(testcase_count)
> > +        testcase_count += 1
> > +        result = parts[2].strip()
> > +        message = parts[3].strip()
> > +    else:
> > +        m = re.match(re_pattern, line)
> > +        #print "m=", m
> > +        if m:
> > +            test_name = m.group(1)
> > +            testcase_num = m.group(2)
> > +            result = m.group(3)[1:] # strip off the leading 'T'
> > +            message = m.group(4).strip()
> > +
> > +    # translate to Fuego results
> > +    if result == 'PASS':
> > +        results[testcase_num] = 'PASS'
> > +    elif result == 'FAIL':
> > +        results[testcase_num] = 'FAIL'
> > +    elif result == 'CONF':
> > +        results[testcase_num] = 'SKIP'
> > +    else:
> > +        results[testcase_num] = 'ERR'
> > +
> > +sys.exit(plib.process(results))
> > diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> > b/engine/tests/Functional.LTP_one_test/spec.json
> > new file mode 100644
> > index 0000000..9f0b325
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/spec.json
> > @@ -0,0 +1,12 @@
> > +{
> > +    "testName": "Functional.LTP_one_test",
> > +    "specs": {
> > +        "default": {
> > +            "TEST":"brk01"
> > +        },
> > +        "fanotify07": {
> > +            "TEST":"fanotify07"
> > +        }
> > +    }
> > +}
> > +
> > diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> > b/engine/tests/Functional.LTP_one_test/test.yaml
> > new file mode 100644
> > index 0000000..765c98d
> > --- /dev/null
> > +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> > @@ -0,0 +1,31 @@
> > +fuego_package_version: 1
> > +name: Functional.LTP_one_test
> > +description: |
> > +    This is a convenience test.  LTP is a large collection of tests,
> which
> > +    takes a lot of infrastructure to set up and execute.  Sometimes it
> is
> > +    handy to run a single LTP test program by itself.  This Fuego test
> > +    exists to allow you to easily do that.
> > +
> > +    To run an individual test, add a new spec for it in the spec.json
> file.
> > +    Specify the individual test name to run in the TEST variable.  By
> > +    convention the name of the spec should match the name of the test
> > +    to run.
> > +license: multiple (same as LTP)
> > +author: Tim Bird
> > +maintainer: Tim Bird <tim.bird@sony.com>
> > +version: 1.0
> > +fuego_release: 1
> > +type: Functional
> > +tags: ['general']
> > +params:
> > +    - TEST:
> > +        description: Name of the individual LTP test program to run.
> > +        example: fanotify07
> > +        optional: no
> > +    - ARGS:
> > +        description: Command line arguments for the LTP test program.
> > +        example: -s foo
> > +        optional: yes
> > +data_files:
> > +    - spec.json
> > +    - parser.py
> > --
> > 1.9.1
> >
> 
> 
> 
> 

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Liu, Wenlong]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Saturday, April 07, 2018 3:11 AM
> To: Liu, Wenlong/刘 文龙 <liuwl.fnst@cn.fujitsu.com>;
> daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: [PATCH] LTP_one_test: create a Fuego test for a single LTP test
> 
> Hey Daniel and Liu,
> 
> Here is a test that runs an individual LTP test.  It's now in my master
> branch.  The reason for pointing this out is to get your feedback on the
> approach of writing a test that uses a sub-set of LTP.
> Note the use of 'ftc run-test' with the '-p  b' parameter, to build LTP
> if needed.  This avoids having to share build script details between this
> test and LTP.
> 
> I'm thinking this approach might be useful to split out the OpenPosix and
> realtime subsets of LTP.  Also, it might work for handling setup and cleanup
> for subtest groups or individual tests, like the network-related tests or
> smack.
> 
> For example, instead of doing smack-specific machine setup in the LTP test,
> I think we should make something like Functional.LTP-smack, use the
> techniques demonstrated here for building it.  But then have  pre_test,
> deply, run, and processing phases be customized for that test's needs. in
> Functional.LTP-smack's fuego_test.sh.

I didn't have some time to do some other tests like this way on my side yet.
But, I think it'll be a good instance which can help us to split those special cases/testset.

> Let me know what you think.
>    -- Tim
> 
> Patch follows:
> ----------------------------
> LTP is a ginormous collection of tests.  When something fails, it is very
> handy to be able to run a single test in isolation.
> This test allows you to do that.  To run an individual test, just add a
> new spec to the spec file (or, when ftc supports setting a test variable
> on the command line, use that to specify the name of the individual test
> program to run).
> 
> This test also demonstrates how one test can use ftc and phases to execute
> just the build of another test.  And how one test can re-use the build
> directory of another test.
> 
> Signed-off-by: Tim Bird <tim.bird@sony.com>
> ---
>  engine/tests/Functional.LTP_one_test/fuego_test.sh | 40
> +++++++++++++++++
>  engine/tests/Functional.LTP_one_test/parser.py     | 51
> ++++++++++++++++++++++
>  engine/tests/Functional.LTP_one_test/spec.json     | 12 +++++
>  engine/tests/Functional.LTP_one_test/test.yaml     | 31 +++++++++++++
>  4 files changed, 134 insertions(+)
>  create mode 100755 engine/tests/Functional.LTP_one_test/fuego_test.sh
>  create mode 100755 engine/tests/Functional.LTP_one_test/parser.py
>  create mode 100644 engine/tests/Functional.LTP_one_test/spec.json
>  create mode 100644 engine/tests/Functional.LTP_one_test/test.yaml
> 
> diff --git a/engine/tests/Functional.LTP_one_test/fuego_test.sh
> b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> new file mode 100755
> index 0000000..2ebcfc5
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/fuego_test.sh
> @@ -0,0 +1,40 @@
> +one_test=$FUNCTIONAL_LTP_ONE_TEST_TEST
> +
> +# Don't allow jobs to share build directories # the
> +"test_successfully_built" flag for one spec # is not accurate for
> +another spec, which has a different # individual program to deploy.
> +FUNCTIONAL_LTP_ONE_TEST_PER_JOB_BUILD="true"
> +
> +function test_pre_check {
> +    assert_define FUNCTIONAL_LTP_ONE_TEST_TEST }
> +
> +function test_build {
> +    # check for LTP build directory
> +    LTP_BUILD_DIR=$(echo $JOB_BUILD_DIR | sed s/LTP_one_test/LTP/ | sed
> +s/$TESTSPEC/default/)
> +
> +    # if not already built, build LTP
> +    if [ ! -e
> ${WORKSPACE}/${LTP_BUILD_DIR}/fuego_test_succesfully_built ] ; then
> +        echo "Building parent LTP test..."
> +	    ftc run-test -b $NODE_NAME -t Functional.LTP -p b
> +        # NOTE: vars used in ftc run-test should not leak into this
> environment
> +        # that is, none of our test vars should have changed.
> +    fi
> +
> +    cp
> ${WORKSPACE}/${LTP_BUILD_DIR}/target_bin/testcases/bin/$one_test .
> +}
> +
> +function test_deploy {
> +	put $one_test $BOARD_TESTDIR/fuego.$TESTDIR/ }
> +
> +function test_run {
> +    # FIXTHIS - deal with arguments better
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
> +        ./$one_test $FUNCTIONAL_LTP_ONE_TEST_ARGS"
> +}
> +
> +function test_processing {
> +    return
> +}
> diff --git a/engine/tests/Functional.LTP_one_test/parser.py
> b/engine/tests/Functional.LTP_one_test/parser.py
> new file mode 100755
> index 0000000..fe2336f
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/parser.py
> @@ -0,0 +1,51 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os
> +import sys
> +import collections
> +import re
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +results = collections.OrderedDict()
> +
> +# LTP results come in largely 2 formats:
> +# sched_getattr01    1  TPASS  :  attributes were read back correctly
> +# request_key01.c:49: PASS: request_key() succeed
> +
> +log_lines = open(plib.TEST_LOG, "r").readlines() re_pattern =
> +r"(\S+)\s+(\S)+\s+(TPASS|TCONF)\s+:(.*)"
> +
> +testcase_count = 1
> +for line in log_lines:
> +    #print "line=", line
> +    if ".c:" in line:
> +        parts = line.split(":")
> +        filename = parts[0]
> +        line_no = parts[1]
> +        testcase_num = str(testcase_count)
> +        testcase_count += 1
> +        result = parts[2].strip()
> +        message = parts[3].strip()
> +    else:
> +        m = re.match(re_pattern, line)
> +        #print "m=", m
> +        if m:
> +            test_name = m.group(1)
> +            testcase_num = m.group(2)
> +            result = m.group(3)[1:] # strip off the leading 'T'
> +            message = m.group(4).strip()
> +
> +    # translate to Fuego results
> +    if result == 'PASS':
> +        results[testcase_num] = 'PASS'
> +    elif result == 'FAIL':
> +        results[testcase_num] = 'FAIL'
> +    elif result == 'CONF':
> +        results[testcase_num] = 'SKIP'
> +    else:
> +        results[testcase_num] = 'ERR'
> +
> +sys.exit(plib.process(results))
> diff --git a/engine/tests/Functional.LTP_one_test/spec.json
> b/engine/tests/Functional.LTP_one_test/spec.json
> new file mode 100644
> index 0000000..9f0b325
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/spec.json
> @@ -0,0 +1,12 @@
> +{
> +    "testName": "Functional.LTP_one_test",
> +    "specs": {
> +        "default": {
> +            "TEST":"brk01"
> +        },
> +        "fanotify07": {
> +            "TEST":"fanotify07"
> +        }
> +    }
> +}
> +
> diff --git a/engine/tests/Functional.LTP_one_test/test.yaml
> b/engine/tests/Functional.LTP_one_test/test.yaml
> new file mode 100644
> index 0000000..765c98d
> --- /dev/null
> +++ b/engine/tests/Functional.LTP_one_test/test.yaml
> @@ -0,0 +1,31 @@
> +fuego_package_version: 1
> +name: Functional.LTP_one_test
> +description: |
> +    This is a convenience test.  LTP is a large collection of tests, which
> +    takes a lot of infrastructure to set up and execute.  Sometimes it
> is
> +    handy to run a single LTP test program by itself.  This Fuego test
> +    exists to allow you to easily do that.
> +
> +    To run an individual test, add a new spec for it in the spec.json file.
> +    Specify the individual test name to run in the TEST variable.  By
> +    convention the name of the spec should match the name of the test
> +    to run.
> +license: multiple (same as LTP)
> +author: Tim Bird
> +maintainer: Tim Bird <tim.bird@sony.com>
> +version: 1.0
> +fuego_release: 1
> +type: Functional
> +tags: ['general']
> +params:
> +    - TEST:
> +        description: Name of the individual LTP test program to run.
> +        example: fanotify07
> +        optional: no
> +    - ARGS:
> +        description: Command line arguments for the LTP test program.
> +        example: -s foo
> +        optional: yes
> +data_files:
> +    - spec.json
> +    - parser.py
> --
> 1.9.1
> 
> 

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin 
> 
> Hi Tim,
> 
> > -----Original Message-----
> > From: Tim Bird
> >
> > Hey Daniel and Liu,
> >
> > Here is a test that runs an individual LTP test.  It's now in my master
> > branch.  The reason for pointing this out is to get your feedback
> > on the approach of writing a test that uses a sub-set of LTP.
> 
> Let me add some background for everyone to understand.
> 
> - On Functional.LTP, it is possible to run sub-sets of LTP by using a skip list. In
> other words,
> you first select the test sets (groups of test cases) that you want to run, and
> then you
> apply a skip list to mask the execution of some of the test cases. Let's call this
> a blacklisting
> approach. The main advantage is that you only need to know the name of
> the test cases
> that you want to skip.
> - On Functional.LTP_one_test, it is possible to specify what test case to
> execute directly. For that
> reason there is no need to create any skip list. The main advantage is that it
> allows the user
> to quickly run any of the tests inside LTP.
> 
> Let me jump into one technical detail though. I see that you are calling the
> test cases directly
> from /testcases/bin/. I don't think that works in general because some of the
> test cases
> require a bit of setup before they are executed. Fortunately, this is already
> taken care by
> runltp through the "-s" option.
>     $ ./runltp -q -f math -s abs01
>     [Note] remove -q to see the test output as well
> 
> > Note the use of 'ftc run-test' with the '-p  b' parameter, to build
> > LTP if needed.  This avoids having to share build script details
> > between this test and LTP.
> 
> Maybe it's because the recent update broke everything, but "ftc run-test"
> and "ftc build-jobs" didn't work on my machine.
That's a problem.  I see you have one bugfix in a later message.  If you are still 
seeing issues, please let me know and I'll try to help fix them.

> Other than that, I like the ability to use "ftc run-test" within the tests. In this
> case for the building phase, but I want to use
> it for creating background loads as well.
> 
> Do you think there could be some conflict with the nesting of tests such as
> variable
> overwriting (Fuego using all these global shell variables is a bit scary) or
> deadlocks?

I worried about both of those issues as well.  I checked ftc, and it doesn't
read much from the environment (only JENKINS_URL, FUEGO_CONTAINER,
and PS1).  It sets a whole bunch of environment variables when it runs a test,
and it looks like it sets everything that is used by the core scripts and the parsers.
(specifically overriding any values that were in the environment when ftc started).
But it's possible I missed something - so this *is* a concern.

The core scripts have a few cases where we only set a variable if it's not
already defined in the environment
(like BUILD_URL, FUEGO_HOST, BUILD_TIMESTAMP or FUEGO_START_TIME, in common.sh),
or we define a variable using shell default values
(like MAX_REBOOT_RETRIES or FUEGO_TARGET_TMP) in functions.sh).

But from what I can tell, both of these sets of variables are handled OK for nested builds.

The parser modules (especially common.py) read a bunch of variables from the environment, but
these should all have been set correctly (and unconditionally) by ftc.

With regard to locks, I already took care of one issue with nested use of the build lock.
There may be other resources that pop up in the future, that a parent and child build
might contend on. But it seems to work correctly now.


> > I'm thinking this approach might be useful to split out
> > the OpenPosix and realtime subsets of LTP.
> Sounds good.
> 
> > Also, it might
> > work for handling setup and cleanup for subtest groups or
> > individual tests, like the network-related tests or smack.
> > For example, instead of doing smack-specific machine setup
> > in the LTP test, I think we should make something like
> > Functional.LTP-smack, use the techniques demonstrated here
> > for building it.  But then have  pre_test, deply, run, and processing
> > phases be customized for that test's needs. in Functional.LTP-smack's
> > fuego_test.sh.
> 
> Yeah, let's split corner cases. Otherwise things will get too complicated to
> handle.

Agreed - thanks for the feedback.
 -- Tim

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Tim.Bird]

> -----Original Message-----
> From: Liu, Wenlong 
> > -----Original Message-----
> > From: Daniel Sangorrin 
> > > -----Original Message-----
> > Let me jump into one technical detail though. I see that you are calling
> > the test cases directly from /testcases/bin/. I don't think that works in
> > general because some of the test cases require a bit of setup before they
> > are executed. Fortunately, this is already taken care by runltp through
> > the "-s" option.
> >     $ ./runltp -q -f math -s abs01
> >     [Note] remove -q to see the test output as well
> 
> Yes, Daniel, I have the same thought with you.

OK - let me take a look at this today, and see if that's a better way to run the test.
I was worried about the default arguments to the test, and maybe this will solve
that problem.

Thanks for the feedback!
 -- Tim

Re: [Fuego] [PATCH] LTP_one_test: create a Fuego test for a single LTP test

[Daniel Sangorrin]

> > Maybe it's because the recent update broke everything, but "ftc run-test"
> > and "ftc build-jobs" didn't work on my machine.
> That's a problem.  I see you have one bugfix in a later message.  If you are still
> seeing issues, please let me know and I'll try to help fix them.

That was one of the problems.
The second one was that LTP wouldn't build. I just sent a patch that fixes that.
Probably you were testing LTP_one_test after having built LTP from Jenkins.
Now, it seems to work fine.

> 
> > Other than that, I like the ability to use "ftc run-test" within the tests. In this
> > case for the building phase, but I want to use
> > it for creating background loads as well.
> >
> > Do you think there could be some conflict with the nesting of tests such as
> > variable
> > overwriting (Fuego using all these global shell variables is a bit scary) or
> > deadlocks?
> 
> I worried about both of those issues as well.  I checked ftc, and it doesn't
> read much from the environment (only JENKINS_URL, FUEGO_CONTAINER,
> and PS1).  It sets a whole bunch of environment variables when it runs a test,
> and it looks like it sets everything that is used by the core scripts and the parsers.
> (specifically overriding any values that were in the environment when ftc started).
> But it's possible I missed something - so this *is* a concern.
> 
> The core scripts have a few cases where we only set a variable if it's not
> already defined in the environment
> (like BUILD_URL, FUEGO_HOST, BUILD_TIMESTAMP or FUEGO_START_TIME, in
> common.sh),
> or we define a variable using shell default values
> (like MAX_REBOOT_RETRIES or FUEGO_TARGET_TMP) in functions.sh).
> 
> But from what I can tell, both of these sets of variables are handled OK for nested
> builds.
> 
> The parser modules (especially common.py) read a bunch of variables from the
> environment, but
> these should all have been set correctly (and unconditionally) by ftc.
> 
> With regard to locks, I already took care of one issue with nested use of the build lock.
> There may be other resources that pop up in the future, that a parent and child build
> might contend on. But it seems to work correctly now.
> 

Thanks for the explanation, I will keep an eye on it.

Regards,
Daniel 

> 
> > > I'm thinking this approach might be useful to split out
> > > the OpenPosix and realtime subsets of LTP.
> > Sounds good.
> >
> > > Also, it might
> > > work for handling setup and cleanup for subtest groups or
> > > individual tests, like the network-related tests or smack.
> > > For example, instead of doing smack-specific machine setup
> > > in the LTP test, I think we should make something like
> > > Functional.LTP-smack, use the techniques demonstrated here
> > > for building it.  But then have  pre_test, deply, run, and processing
> > > phases be customized for that test's needs. in Functional.LTP-smack's
> > > fuego_test.sh.
> >
> > Yeah, let's split corner cases. Otherwise things will get too complicated to
> > handle.
> 
> Agreed - thanks for the feedback.
>  -- Tim