Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Wang, Mingyu]

[-- Attachment #1: Type: text/plain, Size: 1876 bytes --]

Hi Tim,

I'm very sorry, there is a problem with the previous patch, please ignore it. The modified patch has been sent as an attachment. Please confirm it.

 Best regards.



-----Original Message-----
From: Wang, Mingyu/王 鸣瑜 
Sent: Friday, May 18, 2018 10:39 PM
To: fuego@lists.linuxfoundation.org
Cc: Wang, Mingyu/王 鸣瑜 <wangmy@cn.fujitsu.com>
Subject: [PATCH] Add a new test suite called Functional.busybox.



Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
---
 engine/tests/Functional.busybox/fuego_test.sh | 17 +++++++++++++++++
 engine/tests/Functional.busybox/spec.json     |  7 +++++++
 2 files changed, 24 insertions(+)
 create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
 create mode 100644 engine/tests/Functional.busybox/spec.json

diff --git a/engine/tests/Functional.busybox/fuego_test.sh b/engine/tests/Functional.busybox/fuego_test.sh
new file mode 100755
index 0000000..3cc2b85
--- /dev/null
+++ b/engine/tests/Functional.busybox/fuego_test.sh
@@ -0,0 +1,17 @@
+tarball=busybox.tar.gz
+
+
+function test_deploy {
+    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
+    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/ }
+
+function test_run {
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; export test_tcpip_host=$SRV_IP; export test_port_host=$SRV_PORT; export httproot_dir_host=fuego/userContent/readme.txt; sh -v busybox_test.sh"
+}
+
+function test_processing {
+    log_compare "$TESTDIR" "89" "pass" "p"
+}
+
+
diff --git a/engine/tests/Functional.busybox/spec.json b/engine/tests/Functional.busybox/spec.json
new file mode 100644
index 0000000..a71218a
--- /dev/null
+++ b/engine/tests/Functional.busybox/spec.json
@@ -0,0 +1,7 @@
+{
+    "testName": "Functional.busybox",
+    "specs": {
+        "default": {}
+    }
+}
+
--
1.8.3.1




[-- Attachment #2: 0001-Add-a-new-test-suite-called-Functional.busybox.patch --]
[-- Type: application/octet-stream, Size: 887 bytes --]

From a035c791df125260735a7fe09313db76bfe6c4e4 Mon Sep 17 00:00:00 2001
From: Wang Mingyu <wangmy@cn.fujitsu.com>
Date: Fri, 18 May 2018 09:23:58 -0700
Subject: [PATCH] Add a new test suite called Functional.busybox.

Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
---
 engine/tests/Functional.busybox/fuego_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/tests/Functional.busybox/fuego_test.sh b/engine/tests/Functional.busybox/fuego_test.sh
index 3cc2b85..6b8d858 100755
--- a/engine/tests/Functional.busybox/fuego_test.sh
+++ b/engine/tests/Functional.busybox/fuego_test.sh
@@ -2,7 +2,7 @@ tarball=busybox.tar.gz
 
 
 function test_deploy {
-    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
+    put busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
     put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/
 }
 
-- 
1.8.3.1

[Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Wang Mingyu]

Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
---
 engine/tests/Functional.busybox/fuego_test.sh | 17 +++++++++++++++++
 engine/tests/Functional.busybox/spec.json     |  7 +++++++
 2 files changed, 24 insertions(+)
 create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
 create mode 100644 engine/tests/Functional.busybox/spec.json

diff --git a/engine/tests/Functional.busybox/fuego_test.sh b/engine/tests/Functional.busybox/fuego_test.sh
new file mode 100755
index 0000000..3cc2b85
--- /dev/null
+++ b/engine/tests/Functional.busybox/fuego_test.sh
@@ -0,0 +1,17 @@
+tarball=busybox.tar.gz
+
+
+function test_deploy {
+    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
+    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/
+}
+
+function test_run {
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; export test_tcpip_host=$SRV_IP; export test_port_host=$SRV_PORT; export httproot_dir_host=fuego/userContent/readme.txt; sh -v busybox_test.sh"
+}
+
+function test_processing {
+    log_compare "$TESTDIR" "89" "pass" "p"
+}
+
+
diff --git a/engine/tests/Functional.busybox/spec.json b/engine/tests/Functional.busybox/spec.json
new file mode 100644
index 0000000..a71218a
--- /dev/null
+++ b/engine/tests/Functional.busybox/spec.json
@@ -0,0 +1,7 @@
+{
+    "testName": "Functional.busybox",
+    "specs": {
+        "default": {}
+    }
+}
+
-- 
1.8.3.1

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Tim.Bird]

Thanks for the patches!  We always appreciate contributions.
Please see my comments inline below.

> -----Original Message-----
> From: Wang, Mingyu
> I'm very sorry, there is a problem with the previous patch, please ignore it.
> The modified patch has been sent as an attachment. Please confirm it.

It sounded like the new patch would be a complete replacement of the earlier
patch, but the patch that was attached was just a change of part of the commit.
It did not apply by itself.

In any event, here is some feedback...

> -----Original Message-----
> From: Wang, Mingyu/王 鸣瑜
> Sent: Friday, May 18, 2018 10:39 PM
> To: fuego@lists.linuxfoundation.org
> Cc: Wang, Mingyu/王 鸣瑜 <wangmy@cn.fujitsu.com>
> Subject: [PATCH] Add a new test suite called Functional.busybox.
> 
> 
> 
> Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
> ---
>  engine/tests/Functional.busybox/fuego_test.sh | 17 +++++++++++++++++
>  engine/tests/Functional.busybox/spec.json     |  7 +++++++
>  2 files changed, 24 insertions(+)
>  create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
>  create mode 100644 engine/tests/Functional.busybox/spec.json
> 
> diff --git a/engine/tests/Functional.busybox/fuego_test.sh
> b/engine/tests/Functional.busybox/fuego_test.sh
> new file mode 100755
> index 0000000..3cc2b85
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/fuego_test.sh
> @@ -0,0 +1,17 @@
> +tarball=busybox.tar.gz
I looked at the contents of the tarball.  This particular tarball only holds
a single shell script.  In this case, it is better to just have it in the $TEST_HOME
directory, and not use a tarball at all.  My recommendation would be to 
add busybox_test.sh to $TESTHOME, and remove this tarball line.

> +
> +
(minor) my preference would be only 1 blank line here.

> +function test_deploy {
> +    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
I'd leave this as is (not change it like your followup patch)

> +    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/ }
> +
> +function test_run {
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; export
> test_tcpip_host=$SRV_IP; export test_port_host=$SRV_PORT; export
> httproot_dir_host=fuego/userContent/readme.txt; sh -v busybox_test.sh"
Some line continuations would be good here to keep this from getting too long.
e.g.
     report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
        export test_tcpip_host=$SRV_IP ; \
        export test_port_host=$SRV_PORT; \
        export httproot_dir_host=fuego/userContent/readme.txt; \
        sh -v busybox_test.sh"

I know other tests have some pretty long lines, but I'd like for new tests to
be better at having shorter lines.

> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "89" "pass" "p"
Is 89 the right number for the tests you have submitted so far?

> +}
> +
> +
> diff --git a/engine/tests/Functional.busybox/spec.json
> b/engine/tests/Functional.busybox/spec.json
> new file mode 100644
> index 0000000..a71218a
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/spec.json
> @@ -0,0 +1,7 @@
> +{
> +    "testName": "Functional.busybox",
> +    "specs": {
> +        "default": {}
> +    }
> +}
> +
> --
> 1.8.3.1
> 

Looks good.

Did you want me to apply this, and fix it up myself, or do you want to fix it
and send new patches?
 -- Tim

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Wang, Mingyu]

Hi Tim,

Thank you for your comments, they are helpful.

>I looked at the contents of the tarball.  This particular tarball only holds a single shell 
>script.  In this case, it is better to just have it in the $TEST_HOME directory, and not use 
>a tarball at all.  My recommendation would be to add busybox_test.sh to $TESTHOME, 
>and remove this tarball line.
The tarball has been deleted.

>> +
>> +
>(minor) my preference would be only 1 blank line here.
One blank line have been deleted.

>Some line continuations would be good here to keep this from getting too long.
>e.g.
 >    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
 >       export test_tcpip_host=$SRV_IP ; \
 >       export test_port_host=$SRV_PORT; \
 >       export httproot_dir_host=fuego/userContent/readme.txt; \
 >      sh -v busybox_test.sh"
>I know other tests have some pretty long lines, but I'd like for new tests to be better at 
>having shorter lines.
It has been revised to short lines according to the opinion.

> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "89" "pass" "p"
>Is 89 the right number for the tests you have submitted so far?
There are  89 test cases that I want to submit. This time only submitted 4 of them. The number has been modified to 4. and when other test cases are submitted, the number will be modified according to the actual number of cases that are submitted.

>Did you want me to apply this, and fix it up myself, or do you want to fix it and send 
>new patches?
I will fix it by myself and send new patches.

--Wangmy

-----Original Message-----
From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com] 
Sent: Saturday, May 19, 2018 9:23 AM
To: Wang, Mingyu/王 鸣瑜 <wangmy@cn.fujitsu.com>; fuego@lists.linuxfoundation.org
Subject: RE: [PATCH] Add a new test suite called Functional.busybox.

Thanks for the patches!  We always appreciate contributions.
Please see my comments inline below.

> -----Original Message-----
> From: Wang, Mingyu
> I'm very sorry, there is a problem with the previous patch, please ignore it.
> The modified patch has been sent as an attachment. Please confirm it.

It sounded like the new patch would be a complete replacement of the earlier patch, but the patch that was attached was just a change of part of the commit.
It did not apply by itself.

In any event, here is some feedback...

> -----Original Message-----
> From: Wang, Mingyu/王 鸣瑜
> Sent: Friday, May 18, 2018 10:39 PM
> To: fuego@lists.linuxfoundation.org
> Cc: Wang, Mingyu/王 鸣瑜 <wangmy@cn.fujitsu.com>
> Subject: [PATCH] Add a new test suite called Functional.busybox.
> 
> 
> 
> Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
> ---
>  engine/tests/Functional.busybox/fuego_test.sh | 17 +++++++++++++++++
>  engine/tests/Functional.busybox/spec.json     |  7 +++++++
>  2 files changed, 24 insertions(+)
>  create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
>  create mode 100644 engine/tests/Functional.busybox/spec.json
> 
> diff --git a/engine/tests/Functional.busybox/fuego_test.sh
> b/engine/tests/Functional.busybox/fuego_test.sh
> new file mode 100755
> index 0000000..3cc2b85
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/fuego_test.sh
> @@ -0,0 +1,17 @@
> +tarball=busybox.tar.gz
I looked at the contents of the tarball.  This particular tarball only holds a single shell script.  In this case, it is better to just have it in the $TEST_HOME directory, and not use a tarball at all.  My recommendation would be to add busybox_test.sh to $TESTHOME, and remove this tarball line.

> +
> +
(minor) my preference would be only 1 blank line here.

> +function test_deploy {
> +    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
I'd leave this as is (not change it like your followup patch)

> +    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/ }
> +
> +function test_run {
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; export
> test_tcpip_host=$SRV_IP; export test_port_host=$SRV_PORT; export 
> httproot_dir_host=fuego/userContent/readme.txt; sh -v busybox_test.sh"
Some line continuations would be good here to keep this from getting too long.
e.g.
     report "cd $BOARD_TESTDIR/fuego.$TESTDIR; \
        export test_tcpip_host=$SRV_IP ; \
        export test_port_host=$SRV_PORT; \
        export httproot_dir_host=fuego/userContent/readme.txt; \
        sh -v busybox_test.sh"

I know other tests have some pretty long lines, but I'd like for new tests to be better at having shorter lines.

> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "89" "pass" "p"
Is 89 the right number for the tests you have submitted so far?

> +}
> +
> +
> diff --git a/engine/tests/Functional.busybox/spec.json
> b/engine/tests/Functional.busybox/spec.json
> new file mode 100644
> index 0000000..a71218a
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/spec.json
> @@ -0,0 +1,7 @@
> +{
> +    "testName": "Functional.busybox",
> +    "specs": {
> +        "default": {}
> +    }
> +}
> +
> --
> 1.8.3.1
> 

Looks good.

Did you want me to apply this, and fix it up myself, or do you want to fix it and send new patches?
 -- Tim

[Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Wang Mingyu]

Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
---
 engine/tests/Functional.busybox/fuego_test.sh | 51 +++++++++++++++++++++++++++
 engine/tests/Functional.busybox/parser.py     | 22 ++++++++++++
 engine/tests/Functional.busybox/spec.json     |  7 ++++
 3 files changed, 80 insertions(+)
 create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
 create mode 100755 engine/tests/Functional.busybox/parser.py
 create mode 100644 engine/tests/Functional.busybox/spec.json

diff --git a/engine/tests/Functional.busybox/fuego_test.sh b/engine/tests/Functional.busybox/fuego_test.sh
new file mode 100755
index 0000000..0b8ea54
--- /dev/null
+++ b/engine/tests/Functional.busybox/fuego_test.sh
@@ -0,0 +1,51 @@
+# $1: string with tests to skip separated by spaces
+# $2: absolute path to the skipfile (default: ${LOGDIR}/skiplist.txt)
+function skip_tests {
+    # split $1 on whitespace, without file globbing
+    set -f
+    local TESTS=($1)
+    local SKIPFILE="${LOGDIR}/skiplist.txt"
+    set +f
+    echo "Skipping tests: ${TESTS[@]}."
+    for testname in "${TESTS[@]}"; do
+        echo "$testname" >> ${SKIPFILE}
+    done
+}
+
+# $1: command/program to look for
+# $2: string with busybox test names separated by spaces
+# $3: This function may also be used in other tests, which can be considered as a general function of fuego.
+function skip_if_command_unavailable {
+    local PROGRAM="$1"
+    local TESTS="$2"
+    export FOUND=""
+
+    is_on_target_path ${PROGRAM} FOUND
+    if [ -z "${FOUND}" ]; then
+        echo "WARNING: ${PROGRAM} is not installed on the target."
+        skip_tests "${TESTS}"
+    fi
+}
+
+function test_pre_check {
+    echo "Tests skipped depending on the availability of a command on the target"
+    touch ${LOGDIR}/skiplist.txt
+    skip_if_command_unavailable expect "busybox_ash.sh"
+}
+
+function test_deploy {
+    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
+    put -r $TEST_HOME/tests $BOARD_TESTDIR/fuego.$TESTDIR/
+    put ${LOGDIR}/skiplist.txt $BOARD_TESTDIR/fuego.$TESTDIR/
+}
+
+function test_run {
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR;\
+    export test_tcpip_host=$SRV_IP;\
+    export test_port_host=$SRV_PORT;\
+    sh -v busybox_test.sh"
+}
+
+function test_processing {
+    log_compare "$TESTDIR" "4" "TEST-PASS" "p"
+}
diff --git a/engine/tests/Functional.busybox/parser.py b/engine/tests/Functional.busybox/parser.py
new file mode 100755
index 0000000..d85abd7
--- /dev/null
+++ b/engine/tests/Functional.busybox/parser.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python
+# See common.py for description of command-line arguments
+
+import os, sys, collections
+
+sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
+import common as plib
+
+measurements = {}
+measurements = collections.OrderedDict()
+
+regex_string = '^ -> (.*): TEST-(.*)$'
+matches = plib.parse_log(regex_string)
+
+if matches:
+    for m in matches:
+        measurements['default.' + m[0]] = 'PASS' if m[1] == 'PASS' else 'FAIL'
+
+# split the output for each testcase
+plib.split_output_per_testcase(regex_string, measurements)
+
+sys.exit(plib.process(measurements))
diff --git a/engine/tests/Functional.busybox/spec.json b/engine/tests/Functional.busybox/spec.json
new file mode 100644
index 0000000..a71218a
--- /dev/null
+++ b/engine/tests/Functional.busybox/spec.json
@@ -0,0 +1,7 @@
+{
+    "testName": "Functional.busybox",
+    "specs": {
+        "default": {}
+    }
+}
+
-- 
1.8.3.1

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Tim.Bird]

> -----Original Message-----
> This aborts the whole test is busybox is not on the board.
Should be:

... if busybox is not on the board.

('if', not 'is', for the word before 'busybox').
 -- Tim

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Tim.Bird]

Applied and pushed, with a fixup patch applied on top.

Comments on what I changed are inline below.

> -----Original Message-----
> From: Wang Mingyu
> 
> Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
> ---
>  engine/tests/Functional.busybox/fuego_test.sh | 51
> +++++++++++++++++++++++++++
>  engine/tests/Functional.busybox/parser.py     | 22 ++++++++++++
>  engine/tests/Functional.busybox/spec.json     |  7 ++++
>  3 files changed, 80 insertions(+)
>  create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
>  create mode 100755 engine/tests/Functional.busybox/parser.py
>  create mode 100644 engine/tests/Functional.busybox/spec.json
> 
> diff --git a/engine/tests/Functional.busybox/fuego_test.sh
> b/engine/tests/Functional.busybox/fuego_test.sh
> new file mode 100755
> index 0000000..0b8ea54
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/fuego_test.sh
> @@ -0,0 +1,51 @@
> +# $1: string with tests to skip separated by spaces
> +# $2: absolute path to the skipfile (default: ${LOGDIR}/skiplist.txt)
> +function skip_tests {
> +    # split $1 on whitespace, without file globbing
> +    set -f
> +    local TESTS=($1)
> +    local SKIPFILE="${LOGDIR}/skiplist.txt"
> +    set +f
> +    echo "Skipping tests: ${TESTS[@]}."
> +    for testname in "${TESTS[@]}"; do
> +        echo "$testname" >> ${SKIPFILE}
> +    done
> +}
> +
> +# $1: command/program to look for
> +# $2: string with busybox test names separated by spaces
> +# $3: This function may also be used in other tests, which can be considered
> as a general function of fuego.
> +function skip_if_command_unavailable {
> +    local PROGRAM="$1"
> +    local TESTS="$2"
> +    export FOUND=""
> +
> +    is_on_target_path ${PROGRAM} FOUND
> +    if [ -z "${FOUND}" ]; then
> +        echo "WARNING: ${PROGRAM} is not installed on the target."
> +        skip_tests "${TESTS}"
> +    fi
> +}
> +
> +function test_pre_check {
I added the following here:
    is_on_target_path busybox PROGRAM_BUSYBOX
    assert_define PROGRAM_BUSYBOX "Missing 'busybox' program on target board"
This aborts the whole test is busybox is not on the board.

> +    echo "Tests skipped depending on the availability of a command on the
> target"
> +    touch ${LOGDIR}/skiplist.txt
> +    skip_if_command_unavailable expect "busybox_ash.sh"
> +}
> +
> +function test_deploy {
> +    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put -r $TEST_HOME/tests $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put ${LOGDIR}/skiplist.txt $BOARD_TESTDIR/fuego.$TESTDIR/
> +}
> +
> +function test_run {
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR;\
> +    export test_tcpip_host=$SRV_IP;\
> +    export test_port_host=$SRV_PORT;\
I added a default value here for SRV_PORT:
   export test_port_host=${SRV_PORT:=8080};

> +    sh -v busybox_test.sh"
I indented these lines 1 more indent (4 spaces).

> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "4" "TEST-PASS" "p"
I switched this to a "negative" compare.  That will make
is so that if tests are skipped, the log_compare still works.
Otherwise, if a test is skipped, the number of PASSes is not right.
Also, as tests are added, this line doesn't need to be
updated.

The new line is:
log_compare "$TESTDIR" "0" "TEST-FAIL" "n"

> +}
> diff --git a/engine/tests/Functional.busybox/parser.py
> b/engine/tests/Functional.busybox/parser.py
> new file mode 100755
> index 0000000..d85abd7
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/parser.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os, sys, collections
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +measurements = {}
> +measurements = collections.OrderedDict()
> +
> +regex_string = '^ -> (.*): TEST-(.*)$'
> +matches = plib.parse_log(regex_string)
> +
> +if matches:
> +    for m in matches:
> +        measurements['default.' + m[0]] = 'PASS' if m[1] == 'PASS' else 'FAIL'
> +
> +# split the output for each testcase
> +plib.split_output_per_testcase(regex_string, measurements)
> +
> +sys.exit(plib.process(measurements))
> diff --git a/engine/tests/Functional.busybox/spec.json
> b/engine/tests/Functional.busybox/spec.json
> new file mode 100644
> index 0000000..a71218a
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/spec.json
> @@ -0,0 +1,7 @@
> +{
> +    "testName": "Functional.busybox",
> +    "specs": {
> +        "default": {}
> +    }
> +}
> +
> --
> 1.8.3.1

You may want to pull my latest fuego-core, and see the fixup patch I did.

This test helped me find a bug in ov_transport_put (for TRANSPORT 'local')!
That was a nice find.  :-)

Thanks!
 -- Tim

Re: [Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Tim.Bird]

Please see comments inline below.

> -----Original Message-----
> From: Wang Mingyu
> Sent: Tuesday, May 22, 2018 8:59 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH] Add a new test suite called Functional.busybox.
> 
> Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
> ---
>  engine/tests/Functional.busybox/fuego_test.sh | 51
> +++++++++++++++++++++++++++
>  engine/tests/Functional.busybox/parser.py     | 22 ++++++++++++
>  engine/tests/Functional.busybox/spec.json     |  7 ++++
>  3 files changed, 80 insertions(+)
>  create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
>  create mode 100755 engine/tests/Functional.busybox/parser.py
>  create mode 100644 engine/tests/Functional.busybox/spec.json
> 
> diff --git a/engine/tests/Functional.busybox/fuego_test.sh
> b/engine/tests/Functional.busybox/fuego_test.sh
> new file mode 100755
> index 0000000..3140680
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/fuego_test.sh
> @@ -0,0 +1,51 @@
> +# $1: string with tests to skip separated by spaces
> +# $2: absolute path to the skipfile (default: ${LOGDIR}/skiplist.txt)
> +function skip_tests {
> +    # split $1 on whitespace, without file globbing
> +    set -f
> +    local TESTS=($1)
> +    local SKIPFILE="${LOGDIR}/skiplist.txt"
> +    set +f
> +    echo "Skipping tests: ${TESTS[@]}."
> +    for testname in "${TESTS[@]}"; do
> +        echo "$testname" >> ${SKIPFILE}
> +    done
> +}
> +
> +# $1: command/program to look for
> +# $2: string with busybox test names separated by spaces
> +# $3: This function may also be used in other tests, which can be considered
> as a general function of fuego.
> +function skip_if_command_unavailable {
> +    local PROGRAM="$1"
> +    local TESTS="$2"
> +    export FOUND=""
> +
> +    is_on_target ${PROGRAM} FOUND /bin:/usr/bin:/usr/sbin:/usr/local/bin
You should use is_on_target_path here, which uses the search PATH on the board.
This is more reliable than specifying your own path.
See http://fuegotest.org/wiki/function_is_on_target_path

> +    if [ -z "${FOUND}" ]; then
> +        echo "WARNING: ${PROGRAM} is not installed on the target."
> +        skip_tests "${TESTS}"
> +    fi
> +}

This is a nice way to handle this automatically.  Thanks.

> +
> +function test_pre_check {
> +    echo "Tests skipped depending on the availability of a command on the
> target"
> +    touch ${LOGDIR}/skiplist.txt
> +    skip_if_command_unavailable expect "busybox_ash.sh"
> +}
> +
> +function test_deploy {
> +    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/
Sorry I didn't catch this earlier.  'testsuits' is mis-spelled.  It should be 'testsuites'.
Or you could just use 'tests'. This should be renamed everywhere it is used.

> +    put ${LOGDIR}/skiplist.txt $BOARD_TESTDIR/fuego.$TESTDIR/
> +}
> +
> +function test_run {
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR;\
> +    export test_tcpip_host=$SRV_IP;\
> +    export test_port_host=$SRV_PORT;\
> +    sh -v busybox_test.sh"
> +}
> +
> +function test_processing {
> +    log_compare "$TESTDIR" "4" "TEST-PASS" "p"
> +}
> diff --git a/engine/tests/Functional.busybox/parser.py
> b/engine/tests/Functional.busybox/parser.py
> new file mode 100755
> index 0000000..d85abd7
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/parser.py
> @@ -0,0 +1,22 @@
> +#!/usr/bin/python
> +# See common.py for description of command-line arguments
> +
> +import os, sys, collections
> +
> +sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
> +import common as plib
> +
> +measurements = {}
> +measurements = collections.OrderedDict()
> +
> +regex_string = '^ -> (.*): TEST-(.*)$'
> +matches = plib.parse_log(regex_string)
> +
> +if matches:
> +    for m in matches:
> +        measurements['default.' + m[0]] = 'PASS' if m[1] == 'PASS' else 'FAIL'
> +
> +# split the output for each testcase
> +plib.split_output_per_testcase(regex_string, measurements)
> +
> +sys.exit(plib.process(measurements))
> diff --git a/engine/tests/Functional.busybox/spec.json
> b/engine/tests/Functional.busybox/spec.json
> new file mode 100644
> index 0000000..a71218a
> --- /dev/null
> +++ b/engine/tests/Functional.busybox/spec.json
> @@ -0,0 +1,7 @@
> +{
> +    "testName": "Functional.busybox",
> +    "specs": {
> +        "default": {}
> +    }
> +}
> +
> --
> 1.8.3.1

Looks good.  Please fix the two issues mentioned, and I think it will be
good to go.
 -- Tim

[Fuego] [PATCH] Add a new test suite called Functional.busybox.

[Wang Mingyu]

Signed-off-by: Wang Mingyu <wangmy@cn.fujitsu.com>
---
 engine/tests/Functional.busybox/fuego_test.sh | 51 +++++++++++++++++++++++++++
 engine/tests/Functional.busybox/parser.py     | 22 ++++++++++++
 engine/tests/Functional.busybox/spec.json     |  7 ++++
 3 files changed, 80 insertions(+)
 create mode 100755 engine/tests/Functional.busybox/fuego_test.sh
 create mode 100755 engine/tests/Functional.busybox/parser.py
 create mode 100644 engine/tests/Functional.busybox/spec.json

diff --git a/engine/tests/Functional.busybox/fuego_test.sh b/engine/tests/Functional.busybox/fuego_test.sh
new file mode 100755
index 0000000..3140680
--- /dev/null
+++ b/engine/tests/Functional.busybox/fuego_test.sh
@@ -0,0 +1,51 @@
+# $1: string with tests to skip separated by spaces
+# $2: absolute path to the skipfile (default: ${LOGDIR}/skiplist.txt)
+function skip_tests {
+    # split $1 on whitespace, without file globbing
+    set -f
+    local TESTS=($1)
+    local SKIPFILE="${LOGDIR}/skiplist.txt"
+    set +f
+    echo "Skipping tests: ${TESTS[@]}."
+    for testname in "${TESTS[@]}"; do
+        echo "$testname" >> ${SKIPFILE}
+    done
+}
+
+# $1: command/program to look for
+# $2: string with busybox test names separated by spaces
+# $3: This function may also be used in other tests, which can be considered as a general function of fuego.
+function skip_if_command_unavailable {
+    local PROGRAM="$1"
+    local TESTS="$2"
+    export FOUND=""
+
+    is_on_target ${PROGRAM} FOUND /bin:/usr/bin:/usr/sbin:/usr/local/bin
+    if [ -z "${FOUND}" ]; then
+        echo "WARNING: ${PROGRAM} is not installed on the target."
+        skip_tests "${TESTS}"
+    fi
+}
+
+function test_pre_check {
+    echo "Tests skipped depending on the availability of a command on the target"
+    touch ${LOGDIR}/skiplist.txt
+    skip_if_command_unavailable expect "busybox_ash.sh"
+}
+
+function test_deploy {
+    put $TEST_HOME/busybox_test.sh $BOARD_TESTDIR/fuego.$TESTDIR/
+    put -r $TEST_HOME/testsuits $BOARD_TESTDIR/fuego.$TESTDIR/
+    put ${LOGDIR}/skiplist.txt $BOARD_TESTDIR/fuego.$TESTDIR/
+}
+
+function test_run {
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR;\
+    export test_tcpip_host=$SRV_IP;\
+    export test_port_host=$SRV_PORT;\
+    sh -v busybox_test.sh"
+}
+
+function test_processing {
+    log_compare "$TESTDIR" "4" "TEST-PASS" "p"
+}
diff --git a/engine/tests/Functional.busybox/parser.py b/engine/tests/Functional.busybox/parser.py
new file mode 100755
index 0000000..d85abd7
--- /dev/null
+++ b/engine/tests/Functional.busybox/parser.py
@@ -0,0 +1,22 @@
+#!/usr/bin/python
+# See common.py for description of command-line arguments
+
+import os, sys, collections
+
+sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
+import common as plib
+
+measurements = {}
+measurements = collections.OrderedDict()
+
+regex_string = '^ -> (.*): TEST-(.*)$'
+matches = plib.parse_log(regex_string)
+
+if matches:
+    for m in matches:
+        measurements['default.' + m[0]] = 'PASS' if m[1] == 'PASS' else 'FAIL'
+
+# split the output for each testcase
+plib.split_output_per_testcase(regex_string, measurements)
+
+sys.exit(plib.process(measurements))
diff --git a/engine/tests/Functional.busybox/spec.json b/engine/tests/Functional.busybox/spec.json
new file mode 100644
index 0000000..a71218a
--- /dev/null
+++ b/engine/tests/Functional.busybox/spec.json
@@ -0,0 +1,7 @@
+{
+    "testName": "Functional.busybox",
+    "specs": {
+        "default": {}
+    }
+}
+
-- 
1.8.3.1