[Fuego] [fuego-core 0/7] upstreaming work

[Fuego] [fuego-core 0/7] upstreaming work

[venkata.pyla]

From: venkata pyla <venkata.pyla@toshiba-tsip.com>

Hi Tim,

Below are some patches that we want to upstream our work from long time,
the below patches fixes some issues in ftc and LTP functional test

Daniel Sangorrin (1):
  ftc: docker ps will not work on the local board

Nguyen Dat Tho (2):
  LTP: execute all tests when spec specifies "all"
  LTP: automatically obtain the list of tests

Tran Hai Binh (1):
  LTP: Add the new tests and skip non-installed tests

nguyen thi huong (2):
  ftc: fix test process is not killed when fuego test times out
  LTP: fix fuego test could not skip list of test cases

yoshida toshiko (1):
  testplan_smoketest.json: ftc fails to add job for tesplan_smoketest

 overlays/testplans/testplan_smoketest.json |  2 +-
 scripts/ftc                                | 19 +++--
 tests/Functional.LTP/fuego_test.sh         | 91 +++++++---------------
 tests/Functional.LTP/ltp_target_run.sh     | 23 +++---
 4 files changed, 56 insertions(+), 79 deletions(-)

-- 
2.20.1

[Fuego] [fuego-core 1/7] ftc: docker ps will not work on the local board

[venkata.pyla]

From: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>

When running fuego directly on the local board, instead of
using a host-target approach, the check for a fuego
container caused an error.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/ftc | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/scripts/ftc b/scripts/ftc
index 4430dd2..2a6a640 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -5153,11 +5153,15 @@ def get_running_fuego_container_name():
         return cached_container_name
 
     # return the first container with "fuego" in the image or container name
-    dps_lines = subprocess.check_output("sudo docker ps", shell=True).split('\n')
-    for line in dps_lines:
-        if "fuego" in line:
-            cached_container_name = line.strip().split(" ")[-1]
-            break
+    try:
+        with open(os.devnull, 'w') as devnull:
+            dps_lines = subprocess.check_output("sudo docker ps", shell=True, stderr=devnull).split('\n')
+        for line in dps_lines:
+            if "fuego" in line:
+                cached_container_name = line.strip().split(" ")[-1]
+                break
+    except subprocess.CalledProcessError:
+        return None
 
     return cached_container_name
 
-- 
2.20.1

[Fuego] [fuego-core 2/7] ftc: fix test process is not killed when fuego test times out

[venkata.pyla]

From: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>

- Currently, fuego kill main process and does not kill test process when test times out
  In that case, test process still runs but fuego test result is not available
- Revise implementation to kill all test processes and printout the result

Signed-off-by: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 scripts/ftc | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/ftc b/scripts/ftc
index 2a6a640..5c300d0 100755
--- a/scripts/ftc
+++ b/scripts/ftc
@@ -3914,7 +3914,7 @@ def ftc_exec_command(command, timeout):
 
     dprint("ftc_exec_command: command=%s" % command)
 
-    p = subprocess.Popen(command.split(), stdout=log, stderr=log)
+    p = subprocess.Popen(command.split(), stdout=log, stderr=log, preexec_fn=os.setpgrp)
 
     # specify timeout for command operation
     signal.signal(signal.SIGALRM, alarm_handler)
@@ -3960,7 +3960,8 @@ def ftc_exec_command(command, timeout):
         #    p.kill()
 
         # abort with prejudice...
-        p.kill()
+        pgrp = os.getpgid(p.pid)
+        os.killpg(pgrp, signal.SIGALRM)
         timed_out = True
 
     finally:
-- 
2.20.1

[Fuego] [fuego-core 3/7] testplan_smoketest.json: ftc fails to add job for tesplan_smoketest

[venkata.pyla]

From: yoshida toshiko <toshiko.yoshida@toshiba.co.jp>

 add missing comma after Benchmark.dbench4

 e.g: ftc add-jobs -b local -p testplan_smoketest

Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 overlays/testplans/testplan_smoketest.json | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/overlays/testplans/testplan_smoketest.json b/overlays/testplans/testplan_smoketest.json
index 8c856c1..4bbbe4f 100644
--- a/overlays/testplans/testplan_smoketest.json
+++ b/overlays/testplans/testplan_smoketest.json
@@ -12,7 +12,7 @@
             "testName": "Benchmark.Dhrystone"
         },
         {
-            "testName": "Benchmark.dbench4"
+            "testName": "Benchmark.dbench4",
             "timeout": "40m"
         },
         {
-- 
2.20.1

[Fuego] [fuego-core 4/7] LTP: Add the new tests and skip non-installed tests

[venkata.pyla]

From: Tran Hai Binh <binh1.tranhai@toshiba.co.jp>

 - Add the new tests in 20200930 that are not in ALLTESTS:
	s390x_tests, uevent, crypto
 - Skip non-installed tests in LTP

Signed-off-by: Tran Hai Binh <binh1.tranhai@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 tests/Functional.LTP/fuego_test.sh     |  6 +++---
 tests/Functional.LTP/ltp_target_run.sh | 23 ++++++++++++++---------
 2 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index a4f551b..6a84d1c 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -18,9 +18,9 @@ cve                 io_floppy             net.ipv6_lib          net.tirpc_tests
 dio                 ipc                   net.multicast         network_commands                  timers
 dma_thread_diotest  kernel_misc           net.nfs               nptl                              tpm_tools
 fcntl-locktests     ltp-aiodio.part1      net.rpc               numa                              tracing
-filecaps            ltp-aiodio.part2      net.rpc_tests         pipes
-fs                  ltp-aiodio.part3      net.sctp              power_management_tests
-fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive
+filecaps            ltp-aiodio.part2      net.rpc_tests         pipes                             s390x_tests
+fs                  ltp-aiodio.part3      net.sctp              power_management_tests            uevent
+fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive  crypto
 fs_ext4             ltp-aio-stress.part1  net_stress.broken_ip  pty
 smoketest"
 
diff --git a/tests/Functional.LTP/ltp_target_run.sh b/tests/Functional.LTP/ltp_target_run.sh
index 52a006a..ded81de 100755
--- a/tests/Functional.LTP/ltp_target_run.sh
+++ b/tests/Functional.LTP/ltp_target_run.sh
@@ -5,6 +5,7 @@
 
 OUTPUT_DIR=${PWD}/result
 TMP_DIR=${PWD}/tmp
+RUNTEST_DIR=${PWD}/runtest
 
 [ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}
 mkdir -p ${TMP_DIR}
@@ -18,15 +19,19 @@ echo "ltp_target_run: ${TESTS} | ${PTSTESTS} | ${RTTESTS}"
 
 # FIXTHIS: add -t option for limiting the duration of each test group execution
 for i in ${TESTS}; do
-    echo "ltp_target_run: doing test $i"
-    mkdir -p ${OUTPUT_DIR}/${i}
-    ./runltp -C ${OUTPUT_DIR}/${i}/failed.log \
-             -l ${OUTPUT_DIR}/${i}/result.log \
-             -o ${OUTPUT_DIR}/${i}/output.log \
-             -d ${TMP_DIR} \
-             -S ./skiplist.txt \
-             -f $i > ${OUTPUT_DIR}/${i}/head.log 2>&1
-    rm -rf ${TMP_DIR}/*
+    if [ -f ${RUNTEST_DIR}/${i} ]; then
+        echo "ltp_target_run: doing test $i"
+        mkdir -p ${OUTPUT_DIR}/${i}
+        ./runltp -C ${OUTPUT_DIR}/${i}/failed.log \
+                 -l ${OUTPUT_DIR}/${i}/result.log \
+                 -o ${OUTPUT_DIR}/${i}/output.log \
+                 -d ${TMP_DIR} \
+                 -S ./skiplist.txt \
+                 -f $i > ${OUTPUT_DIR}/${i}/head.log 2>&1
+        rm -rf ${TMP_DIR}/*
+    else
+        echo "Warning: ${i} is not available in the installed LTP"
+    fi
 done
 
 # gather posix results into pts.log
-- 
2.20.1

[Fuego] [fuego-core 5/7] LTP: fix fuego test could not skip list of test cases

[venkata.pyla]

From: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>

Currently It can skip one test case or one file that definied in 'skiplist'
Result of skipped test cases are marked as CONF ('stat=32')
However it could not skip a list of LTP test cases
Correct the syntax of FUEGO_LTP_SKIPLIST variable to parse all items of 'skiplist'

Signed-off-by: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 tests/Functional.LTP/fuego_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index 6a84d1c..e620b17 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -380,7 +380,7 @@ function test_deploy {
     # to skipfiles (text files containing a list of LTP test case names)
     # usually located under /fuego-rw/boards/
     if [ -n "${FUNCTIONAL_LTP_SKIPLIST}" ]; then
-        for item in "${FUNCTIONAL_LTP_SKIPLIST}"; do
+        for item in ${FUNCTIONAL_LTP_SKIPLIST}; do
             if [ -f "$item" ]; then
                 cat "$item" >> ${LOGDIR}/skiplist.txt
             else
-- 
2.20.1

[Fuego] [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[venkata.pyla]

From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>

Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
 1 file changed, 24 insertions(+), 18 deletions(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index e620b17..16d556f 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -188,7 +188,6 @@ function test_pre_check {
     assert_define AR
     assert_define RANLIB
     #assert_define LDFLAGS
-    assert_define FUNCTIONAL_LTP_TESTS
 
     # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
     echo "Tests skipped by default in Fuego for now"
@@ -453,25 +452,32 @@ function test_run {
         # on the type of the test (regular, posize, or realtime)
         # Separate the test list by type, and pass the tests in different
         # variables
-        for a in $FUNCTIONAL_LTP_TESTS; do
-            for b in $ALLTESTS; do
-                if [ "$a" == "$b" ]; then
-                    TESTS+="$a "
-                fi
-            done
-
-            for b in $ALLPTSTESTS; do
-                if [ "$a" == "$b" ]; then
-                    PTSTESTS+="$a "
-                fi
-            done
 
-            for b in $ALLRTTESTS; do
-                if [ "$a" == "$b" ]; then
-                    RTTESTS+="$a "
-                fi
+        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
+            for a in $FUNCTIONAL_LTP_TESTS; do
+                for b in $ALLTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        TESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLPTSTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        PTSTESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLRTTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        RTTESTS+="$a "
+                    fi
+                done
             done
-        done
+        else
+            TESTS=$ALLTESTS
+            PTSTESTS=$ALLPTSTESTS
+            RTTESTS=$ALLRTTESTS
+        fi
 
         # Let some of the tests fail, the information will be in the result xlsx file
         report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\"; ./ltp_target_run.sh"
-- 
2.20.1

[Fuego] [fuego-core 7/7] LTP: automatically obtain the list of tests

[venkata.pyla]

From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>

 set ALLTESTS, ALLPTSTESTS and ALLRTTESTS automatically by
 listing all possible runtests

Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 tests/Functional.LTP/fuego_test.sh | 47 +++---------------------------
 1 file changed, 4 insertions(+), 43 deletions(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index 16d556f..23dfc6c 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -4,49 +4,6 @@ tarball=ltp-full-20210524.tar.bz2
 
 NEED_ROOT=1
 
-ALLTESTS="
-admin_tools         fs_perms_simple       ltp-aio-stress.part2  net_stress.interface
-can                 fs_readonly           ltplite               net_stress.ipsec_dccp             sched
-cap_bounds          fsx                   lvm.part1             net_stress.ipsec_icmp             scsi_debug.part1
-commands            hugetlb               lvm.part2             net_stress.ipsec_sctp             securebits
-connectors          hyperthreading        math                  net_stress.ipsec_tcp              smack
-containers          ima                   mm                    net_stress.ipsec_udp              stress.part1
-controllers         input                 modules               net_stress.multicast              stress.part2
-cpuhotplug          io                    net.features          net_stress.route                  stress.part3
-crashme             io_cd                 net.ipv6              net.tcp_cmds                      syscalls
-cve                 io_floppy             net.ipv6_lib          net.tirpc_tests                   syscalls-ipc
-dio                 ipc                   net.multicast         network_commands                  timers
-dma_thread_diotest  kernel_misc           net.nfs               nptl                              tpm_tools
-fcntl-locktests     ltp-aiodio.part1      net.rpc               numa                              tracing
-filecaps            ltp-aiodio.part2      net.rpc_tests         pipes                             s390x_tests
-fs                  ltp-aiodio.part3      net.sctp              power_management_tests            uevent
-fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive  crypto
-fs_ext4             ltp-aio-stress.part1  net_stress.broken_ip  pty
-smoketest"
-
-ALLPTSTESTS="AIO MEM MSG SEM SIG THR TMR TPS"
-
-# This list can be obtained by doing ./testscripts/test_realtime.sh -t list
-ALLRTTESTS="
-perf/latency
-func/measurement
-func/hrtimer-prio
-func/gtod_latency
-func/periodic_cpu_load
-func/pthread_kill_latency
-func/sched_football
-func/pi-tests
-func/thread_clock
-func/rt-migrate
-func/matrix_mult
-func/prio-preempt
-func/prio-wake
-func/pi_perf
-func/sched_latency
-func/async_handler
-func/sched_jitter
-"
-
 # OK - the logic here is a bit complicated
 # We support several different usage scenarios:
 #   1: fuego builds, deploys and installs LTP
@@ -444,6 +401,10 @@ function test_run {
         cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30 diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
         cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/' $LTP_DESTDIR/runtest/syscalls"
 
+        ALLTESTS=$(ls $LTP_DESTDIR/runtest)
+        ALLPTSTESTS=$(grep 'usage: $(basename "$0")' $LTP_DESTDIR/bin/run-posix-option-group-test.sh | sed -e 's/^.*\[\(.*\)\]/\1/g' -e 's/|/ /g')
+        ALLRTTESTS=$(find $LTP_DESTDIR/testcases/realtime -type f -name run_auto.sh -printf '%h\n' | sed 's/^.*realtime\///g')
+
         TESTS=""
         PTSTESTS=""
         RTTESTS=""
-- 
2.20.1

Re: [Fuego] [fuego-core 0/7] upstreaming work

[Tim.Bird]

Hey Venkata,

Thanks very much for these patches.  I looked at them quickly, and they look
pretty good.  I'll do a more thorough review and try to get them applied
later this week.  Some of these items are quite nice.

I'll let you know when they are applied.

Thanks!
 -- Tim


> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> Hi Tim,
> 
> Below are some patches that we want to upstream our work from long time,
> the below patches fixes some issues in ftc and LTP functional test
> 
> Daniel Sangorrin (1):
>   ftc: docker ps will not work on the local board
> 
> Nguyen Dat Tho (2):
>   LTP: execute all tests when spec specifies "all"
>   LTP: automatically obtain the list of tests
> 
> Tran Hai Binh (1):
>   LTP: Add the new tests and skip non-installed tests
> 
> nguyen thi huong (2):
>   ftc: fix test process is not killed when fuego test times out
>   LTP: fix fuego test could not skip list of test cases
> 
> yoshida toshiko (1):
>   testplan_smoketest.json: ftc fails to add job for tesplan_smoketest
> 
>  overlays/testplans/testplan_smoketest.json |  2 +-
>  scripts/ftc                                | 19 +++--
>  tests/Functional.LTP/fuego_test.sh         | 91 +++++++---------------
>  tests/Functional.LTP/ltp_target_run.sh     | 23 +++---
>  4 files changed, 56 insertions(+), 79 deletions(-)
> 
> --
> 2.20.1
> 

Re: [Fuego] [fuego-core 1/7] ftc: docker ps will not work on the local board

[Tim.Bird]

Looks good. Applied.

Thanks!
 -- Tim


> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> 
> When running fuego directly on the local board, instead of
> using a host-target approach, the check for a fuego
> container caused an error.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/ftc | 14 +++++++++-----
>  1 file changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index 4430dd2..2a6a640 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -5153,11 +5153,15 @@ def get_running_fuego_container_name():
>          return cached_container_name
> 
>      # return the first container with "fuego" in the image or container name
> -    dps_lines = subprocess.check_output("sudo docker ps", shell=True).split('\n')
> -    for line in dps_lines:
> -        if "fuego" in line:
> -            cached_container_name = line.strip().split(" ")[-1]
> -            break
> +    try:
> +        with open(os.devnull, 'w') as devnull:
I was not familiar with this idiom for executing a subprocess with 2>/dev/null.
It's interesting...

> +            dps_lines = subprocess.check_output("sudo docker ps", shell=True, stderr=devnull).split('\n')
> +        for line in dps_lines:
> +            if "fuego" in line:
> +                cached_container_name = line.strip().split(" ")[-1]
> +                break
> +    except subprocess.CalledProcessError:
> +        return None
> 
>      return cached_container_name
> 
> --
> 2.20.1
> 

Re: [Fuego] [fuego-core 2/7] ftc: fix test process is not killed when fuego test times out

[Tim.Bird]

OK - I have a question on this one.

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> 
> - Currently, fuego kill main process and does not kill test process when test times out
>   In that case, test process still runs but fuego test result is not available
> - Revise implementation to kill all test processes and printout the result
> 
> Signed-off-by: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/ftc | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/ftc b/scripts/ftc
> index 2a6a640..5c300d0 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -3914,7 +3914,7 @@ def ftc_exec_command(command, timeout):
> 
>      dprint("ftc_exec_command: command=%s" % command)
> 
> -    p = subprocess.Popen(command.split(), stdout=log, stderr=log)
> +    p = subprocess.Popen(command.split(), stdout=log, stderr=log, preexec_fn=os.setpgrp)
> 
>      # specify timeout for command operation
>      signal.signal(signal.SIGALRM, alarm_handler)
> @@ -3960,7 +3960,8 @@ def ftc_exec_command(command, timeout):
>          #    p.kill()
> 
>          # abort with prejudice...
> -        p.kill()
> +        pgrp = os.getpgid(p.pid)
> +        os.killpg(pgrp, signal.SIGALRM)

Why are you sending a SIGALARM instead of a SIGKILL?
This is a weaker signal, that can be ignored or masked.

>          timed_out = True
> 
>      finally:
> --
> 2.20.1
> 

I've applied this patch, but I'd like to know if you have some reason for weakening the
signal sent to kill the process.

Thanks.
 -- Tim

Re: [Fuego] [fuego-core 3/7] testplan_smoketest.json: ftc fails to add job for tesplan_smoketest

[Tim.Bird]

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: yoshida toshiko <toshiko.yoshida@toshiba.co.jp>
> 
>  add missing comma after Benchmark.dbench4
> 
>  e.g: ftc add-jobs -b local -p testplan_smoketest
> 
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  overlays/testplans/testplan_smoketest.json | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/overlays/testplans/testplan_smoketest.json b/overlays/testplans/testplan_smoketest.json
> index 8c856c1..4bbbe4f 100644
> --- a/overlays/testplans/testplan_smoketest.json
> +++ b/overlays/testplans/testplan_smoketest.json
> @@ -12,7 +12,7 @@
>              "testName": "Benchmark.Dhrystone"
>          },
>          {
> -            "testName": "Benchmark.dbench4"
> +            "testName": "Benchmark.dbench4",
>              "timeout": "40m"
>          },
>          {
> --
> 2.20.1
> 

FYI - the standalone testplans are now deprecated.
The new method of doing this is with a Functional.batch job, like so:
 $ ftc add-jobs -b local -t batch_smoketest

The 'batch' jobs have a testplan embedded in them (in json format
as a HERE document in the fuego_test.sh script for the test.

However, standalone testplans are still supported for backwards compatibility,
and this *is* a bug, so I have applied this.

Thanks!
 -- Tim

Re: [Fuego] [fuego-core 4/7] LTP: Add the new tests and skip non-installed tests

[Tim.Bird]

Kind of weird to add these, and then remove them in patch 7, but OK.
It makes the patch series application easier, but you guys could have
done a git rebase and eliminated this one.

Applied.
 -- Tim


> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: Tran Hai Binh <binh1.tranhai@toshiba.co.jp>
> 
>  - Add the new tests in 20200930 that are not in ALLTESTS:
> 	s390x_tests, uevent, crypto
>  - Skip non-installed tests in LTP
> 
> Signed-off-by: Tran Hai Binh <binh1.tranhai@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  tests/Functional.LTP/fuego_test.sh     |  6 +++---
>  tests/Functional.LTP/ltp_target_run.sh | 23 ++++++++++++++---------
>  2 files changed, 17 insertions(+), 12 deletions(-)
> 
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index a4f551b..6a84d1c 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -18,9 +18,9 @@ cve                 io_floppy             net.ipv6_lib          net.tirpc_tests
>  dio                 ipc                   net.multicast         network_commands                  timers
>  dma_thread_diotest  kernel_misc           net.nfs               nptl                              tpm_tools
>  fcntl-locktests     ltp-aiodio.part1      net.rpc               numa                              tracing
> -filecaps            ltp-aiodio.part2      net.rpc_tests         pipes
> -fs                  ltp-aiodio.part3      net.sctp              power_management_tests
> -fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive
> +filecaps            ltp-aiodio.part2      net.rpc_tests         pipes                             s390x_tests
> +fs                  ltp-aiodio.part3      net.sctp              power_management_tests            uevent
> +fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive  crypto
>  fs_ext4             ltp-aio-stress.part1  net_stress.broken_ip  pty
>  smoketest"
> 
> diff --git a/tests/Functional.LTP/ltp_target_run.sh b/tests/Functional.LTP/ltp_target_run.sh
> index 52a006a..ded81de 100755
> --- a/tests/Functional.LTP/ltp_target_run.sh
> +++ b/tests/Functional.LTP/ltp_target_run.sh
> @@ -5,6 +5,7 @@
> 
>  OUTPUT_DIR=${PWD}/result
>  TMP_DIR=${PWD}/tmp
> +RUNTEST_DIR=${PWD}/runtest
> 
>  [ -d ${TMP_DIR} ] && rm -rf ${TMP_DIR}
>  mkdir -p ${TMP_DIR}
> @@ -18,15 +19,19 @@ echo "ltp_target_run: ${TESTS} | ${PTSTESTS} | ${RTTESTS}"
> 
>  # FIXTHIS: add -t option for limiting the duration of each test group execution
>  for i in ${TESTS}; do
> -    echo "ltp_target_run: doing test $i"
> -    mkdir -p ${OUTPUT_DIR}/${i}
> -    ./runltp -C ${OUTPUT_DIR}/${i}/failed.log \
> -             -l ${OUTPUT_DIR}/${i}/result.log \
> -             -o ${OUTPUT_DIR}/${i}/output.log \
> -             -d ${TMP_DIR} \
> -             -S ./skiplist.txt \
> -             -f $i > ${OUTPUT_DIR}/${i}/head.log 2>&1
> -    rm -rf ${TMP_DIR}/*
> +    if [ -f ${RUNTEST_DIR}/${i} ]; then
> +        echo "ltp_target_run: doing test $i"
> +        mkdir -p ${OUTPUT_DIR}/${i}
> +        ./runltp -C ${OUTPUT_DIR}/${i}/failed.log \
> +                 -l ${OUTPUT_DIR}/${i}/result.log \
> +                 -o ${OUTPUT_DIR}/${i}/output.log \
> +                 -d ${TMP_DIR} \
> +                 -S ./skiplist.txt \
> +                 -f $i > ${OUTPUT_DIR}/${i}/head.log 2>&1
> +        rm -rf ${TMP_DIR}/*
> +    else
> +        echo "Warning: ${i} is not available in the installed LTP"
> +    fi
>  done
> 
>  # gather posix results into pts.log
> --
> 2.20.1
> 

Re: [Fuego] [fuego-core 5/7] LTP: fix fuego test could not skip list of test cases

[Tim.Bird]

Nice catch!! Thanks.

Applied.
 -- Tim

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> Sent: Sunday, July 18, 2021 5:10 AM
> To: Bird, Tim <Tim.Bird@sony.com>
> Cc: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>; daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org; venkata pyla
> <venkata.pyla@toshiba-tsip.com>
> Subject: [fuego-core 5/7] LTP: fix fuego test could not skip list of test cases
> 
> From: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> 
> Currently It can skip one test case or one file that definied in 'skiplist'
> Result of skipped test cases are marked as CONF ('stat=32')
> However it could not skip a list of LTP test cases
> Correct the syntax of FUEGO_LTP_SKIPLIST variable to parse all items of 'skiplist'
> 
> Signed-off-by: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  tests/Functional.LTP/fuego_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index 6a84d1c..e620b17 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -380,7 +380,7 @@ function test_deploy {
>      # to skipfiles (text files containing a list of LTP test case names)
>      # usually located under /fuego-rw/boards/
>      if [ -n "${FUNCTIONAL_LTP_SKIPLIST}" ]; then
> -        for item in "${FUNCTIONAL_LTP_SKIPLIST}"; do
> +        for item in ${FUNCTIONAL_LTP_SKIPLIST}; do
>              if [ -f "$item" ]; then
>                  cat "$item" >> ${LOGDIR}/skiplist.txt
>              else
> --
> 2.20.1
> 

Re: [Fuego] [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[Tim.Bird]

This one has problems.

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> Subject: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> 
I'm not sure that the summary line of this change matches what it does.

From reading the code, it appears that "tests" in the spec needs to be
missing or empty, in order to activate this.  If "tests" is "all", 
then it appears that none of the tests will match, and you'd just
end up with empty lists.

Also, I had to intuit the reasoning here, since there is no description
for this.  I don't mind the concept of being able to easily run all
LTP tests, but it would be nice to use the string "all" to signal
this instead of just leaving FUNCTIONAL_LTP_TESTS empty.

It should probably go into a comment in the code as well, and maybe
in the test.yaml file, so that users and future developers will see
it and understand the behaviour.

Also, you might want to add a spec to the spec.json file for "all", to
show it's usage.

> From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> 
> Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
>  1 file changed, 24 insertions(+), 18 deletions(-)
> 
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index e620b17..16d556f 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -188,7 +188,6 @@ function test_pre_check {
>      assert_define AR
>      assert_define RANLIB
>      #assert_define LDFLAGS
> -    assert_define FUNCTIONAL_LTP_TESTS
> 
>      # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
>      echo "Tests skipped by default in Fuego for now"
> @@ -453,25 +452,32 @@ function test_run {
>          # on the type of the test (regular, posize, or realtime)
>          # Separate the test list by type, and pass the tests in different
>          # variables
> -        for a in $FUNCTIONAL_LTP_TESTS; do
> -            for b in $ALLTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    TESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLPTSTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    PTSTESTS+="$a "
> -                fi
> -            done
> 
> -            for b in $ALLRTTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    RTTESTS+="$a "
> -                fi
> +        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
> +            for a in $FUNCTIONAL_LTP_TESTS; do
> +                for b in $ALLTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        TESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLPTSTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        PTSTESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLRTTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        RTTESTS+="$a "
> +                    fi
> +                done
>              done
> -        done
> +        else
> +            TESTS=$ALLTESTS
> +            PTSTESTS=$ALLPTSTESTS
> +            RTTESTS=$ALLRTTESTS
> +        fi
> 
>          # Let some of the tests fail, the information will be in the result xlsx file
>          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> ./ltp_target_run.sh"
> --
> 2.20.1
> 

I have NOT applied this one.  Please change this to check for the string "all"
in FUNCTIONAL_LTP_TESTS (and please submit a sample spec to go with this).

Thanks,
 -- Tim

Re: [Fuego] [fuego-core 7/7] LTP: automatically obtain the list of tests

[Tim.Bird]

I love this concept, but the implementation had problems.
See below.

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> 
>  set ALLTESTS, ALLPTSTESTS and ALLRTTESTS automatically by
>  listing all possible runtests
> 
> Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  tests/Functional.LTP/fuego_test.sh | 47 +++---------------------------
>  1 file changed, 4 insertions(+), 43 deletions(-)
> 
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index 16d556f..23dfc6c 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -4,49 +4,6 @@ tarball=ltp-full-20210524.tar.bz2
> 
>  NEED_ROOT=1
> 
> -ALLTESTS="
> -admin_tools         fs_perms_simple       ltp-aio-stress.part2  net_stress.interface
> -can                 fs_readonly           ltplite               net_stress.ipsec_dccp             sched
> -cap_bounds          fsx                   lvm.part1             net_stress.ipsec_icmp             scsi_debug.part1
> -commands            hugetlb               lvm.part2             net_stress.ipsec_sctp             securebits
> -connectors          hyperthreading        math                  net_stress.ipsec_tcp              smack
> -containers          ima                   mm                    net_stress.ipsec_udp              stress.part1
> -controllers         input                 modules               net_stress.multicast              stress.part2
> -cpuhotplug          io                    net.features          net_stress.route                  stress.part3
> -crashme             io_cd                 net.ipv6              net.tcp_cmds                      syscalls
> -cve                 io_floppy             net.ipv6_lib          net.tirpc_tests                   syscalls-ipc
> -dio                 ipc                   net.multicast         network_commands                  timers
> -dma_thread_diotest  kernel_misc           net.nfs               nptl                              tpm_tools
> -fcntl-locktests     ltp-aiodio.part1      net.rpc               numa                              tracing
> -filecaps            ltp-aiodio.part2      net.rpc_tests         pipes                             s390x_tests
> -fs                  ltp-aiodio.part3      net.sctp              power_management_tests            uevent
> -fs_bind             ltp-aiodio.part4      net_stress.appl       power_management_tests_exclusive  crypto
> -fs_ext4             ltp-aio-stress.part1  net_stress.broken_ip  pty
> -smoketest"
> -
> -ALLPTSTESTS="AIO MEM MSG SEM SIG THR TMR TPS"
> -
> -# This list can be obtained by doing ./testscripts/test_realtime.sh -t list
> -ALLRTTESTS="
> -perf/latency
> -func/measurement
> -func/hrtimer-prio
> -func/gtod_latency
> -func/periodic_cpu_load
> -func/pthread_kill_latency
> -func/sched_football
> -func/pi-tests
> -func/thread_clock
> -func/rt-migrate
> -func/matrix_mult
> -func/prio-preempt
> -func/prio-wake
> -func/pi_perf
> -func/sched_latency
> -func/async_handler
> -func/sched_jitter
> -"
> -
>  # OK - the logic here is a bit complicated
>  # We support several different usage scenarios:
>  #   1: fuego builds, deploys and installs LTP
> @@ -444,6 +401,10 @@ function test_run {
>          cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30 diotest6 -b 65536 -n 5 -i 100 -o 1024000/'
> $LTP_DESTDIR/runtest/dio"
>          cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/' $LTP_DESTDIR/runtest/syscalls"
> 
> +        ALLTESTS=$(ls $LTP_DESTDIR/runtest)
> +        ALLPTSTESTS=$(grep 'usage: $(basename "$0")' $LTP_DESTDIR/bin/run-posix-option-group-test.sh | sed -e 's/^.*\[\(.*\)\]/\1/g' -e
> 's/|/ /g')
> +        ALLRTTESTS=$(find $LTP_DESTDIR/testcases/realtime -type f -name run_auto.sh -printf '%h\n' | sed 's/^.*realtime\///g')
> +
The above commands only work for a local board.  LTP_DESTDIR is on the target.
These should use 'cmd ...' to execute the operation on the board, instead of running
the command locally.

I made this change, and modified the ALLPTSTESTS check slightly, to make it easier to read.

>          TESTS=""
>          PTSTESTS=""
>          RTTESTS=""
> --
> 2.20.1
> 
Thanks.  This is applied.

I'm going to push these changes to the fuego-core master branch.  Can you please
pull them and check that they work for you?
Thanks.

 -- Tim

Re: [Fuego] [fuego-core 2/7] ftc: fix test process is not killed when fuego test times out

[huong4.nguyenthi]

Dear Tim,

Thanks for your feedback.

The function handle timeout event so I think send SIGALARM is more reasonable.
Test driver or subprocesses will decide what to do with timeout signal.

For now, SIGALARM is trapped in fuego functions.sh
Signal handler will do post_tests and processing before exiting main process
When main process exit, all subprocesses that belong to group process will be exited.

Best Regards,
Huong
-----Original Message-----
From: Tim.Bird@sony.com <Tim.Bird@sony.com> 
Sent: Friday, July 23, 2021 1:35 AM
To: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
Cc: nguyen thi huong(ＴＳＤＶ Ｅｎｇ　１) <huong4.nguyenthi@toshiba.co.jp>; sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org
Subject: RE: [fuego-core 2/7] ftc: fix test process is not killed when fuego test times out

OK - I have a question on this one.

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> 
> From: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> 
> - Currently, fuego kill main process and does not kill test process when test times out
>   In that case, test process still runs but fuego test result is not 
> available
> - Revise implementation to kill all test processes and printout the 
> result
> 
> Signed-off-by: nguyen thi huong <huong4.nguyenthi@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  scripts/ftc | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/ftc b/scripts/ftc index 2a6a640..5c300d0 100755
> --- a/scripts/ftc
> +++ b/scripts/ftc
> @@ -3914,7 +3914,7 @@ def ftc_exec_command(command, timeout):
> 
>      dprint("ftc_exec_command: command=%s" % command)
> 
> -    p = subprocess.Popen(command.split(), stdout=log, stderr=log)
> +    p = subprocess.Popen(command.split(), stdout=log, stderr=log, 
> + preexec_fn=os.setpgrp)
> 
>      # specify timeout for command operation
>      signal.signal(signal.SIGALRM, alarm_handler) @@ -3960,7 +3960,8 
> @@ def ftc_exec_command(command, timeout):
>          #    p.kill()
> 
>          # abort with prejudice...
> -        p.kill()
> +        pgrp = os.getpgid(p.pid)
> +        os.killpg(pgrp, signal.SIGALRM)

Why are you sending a SIGALARM instead of a SIGKILL?
This is a weaker signal, that can be ignored or masked.

>          timed_out = True
> 
>      finally:
> --
> 2.20.1
> 

I've applied this patch, but I'd like to know if you have some reason for weakening the signal sent to kill the process.

Thanks.
 -- Tim

[Fuego] Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[tho1.nguyendat]

[-- Attachment #1: Type: text/plain, Size: 7700 bytes --]

Dear Tim,

I would like to send my update.

From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>

Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>

---
 tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
 tests/Functional.LTP/spec.json     |  6 +++++
 2 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index 85adabf..689abfe 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -414,25 +414,31 @@ function test_run {
         # on the type of the test (regular, posize, or realtime)
         # Separate the test list by type, and pass the tests in different
         # variables
-        for a in $FUNCTIONAL_LTP_TESTS; do
-            for b in $ALLTESTS; do
-                if [ "$a" == "$b" ]; then
-                    TESTS+="$a "
-                fi
-            done
-
-            for b in $ALLPTSTESTS; do
-                if [ "$a" == "$b" ]; then
-                    PTSTESTS+="$a "
-                fi
-            done
-
-            for b in $ALLRTTESTS; do
-                if [ "$a" == "$b" ]; then
-                    RTTESTS+="$a "
-                fi
+        if [ "$FUNCTIONAL_LTP_TESTS" == "all" ]; then
+            TESTS=$ALLTESTS
+            PTSTESTS=$ALLPTSTESTS
+            RTTESTS=$ALLRTTESTS
+        else
+            for a in $FUNCTIONAL_LTP_TESTS; do
+                for b in $ALLTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        TESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLPTSTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        PTSTESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLRTTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        RTTESTS+="$a "
+                    fi
+                done
             done
-        done
+        fi

         # Let some of the tests fail, the information will be in the result xlsx file
         report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\"; ./ltp_target_run.sh"
diff --git a/tests/Functional.LTP/spec.json b/tests/Functional.LTP/spec.json
index d30ba4c..53ee115 100644
--- a/tests/Functional.LTP/spec.json
+++ b/tests/Functional.LTP/spec.json
@@ -1,6 +1,12 @@
 {
     "testName": "Functional.LTP",
     "specs": {
+        "all": {
+            "tests": "all",
+            "noautoskip": "true",
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
+        },
         "default": {
             "tests": "syscalls SEM",
             "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
--
2.20.1
________________________________
Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
Đã gửi: 23 Tháng Bảy 2021 2:03 SA
Đến: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
Cc: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>; sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org <fuego@lists.linuxfoundation.org>
Chủ đề: RE: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

This one has problems.

> -----Original Message-----
> From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> Subject: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
>
I'm not sure that the summary line of this change matches what it does.

From reading the code, it appears that "tests" in the spec needs to be
missing or empty, in order to activate this.  If "tests" is "all",
then it appears that none of the tests will match, and you'd just
end up with empty lists.

Also, I had to intuit the reasoning here, since there is no description
for this.  I don't mind the concept of being able to easily run all
LTP tests, but it would be nice to use the string "all" to signal
this instead of just leaving FUNCTIONAL_LTP_TESTS empty.

It should probably go into a comment in the code as well, and maybe
in the test.yaml file, so that users and future developers will see
it and understand the behaviour.

Also, you might want to add a spec to the spec.json file for "all", to
show it's usage.

> From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
>
> Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> ---
>  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
>  1 file changed, 24 insertions(+), 18 deletions(-)
>
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index e620b17..16d556f 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -188,7 +188,6 @@ function test_pre_check {
>      assert_define AR
>      assert_define RANLIB
>      #assert_define LDFLAGS
> -    assert_define FUNCTIONAL_LTP_TESTS
>
>      # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
>      echo "Tests skipped by default in Fuego for now"
> @@ -453,25 +452,32 @@ function test_run {
>          # on the type of the test (regular, posize, or realtime)
>          # Separate the test list by type, and pass the tests in different
>          # variables
> -        for a in $FUNCTIONAL_LTP_TESTS; do
> -            for b in $ALLTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    TESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLPTSTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    PTSTESTS+="$a "
> -                fi
> -            done
>
> -            for b in $ALLRTTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    RTTESTS+="$a "
> -                fi
> +        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
> +            for a in $FUNCTIONAL_LTP_TESTS; do
> +                for b in $ALLTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        TESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLPTSTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        PTSTESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLRTTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        RTTESTS+="$a "
> +                    fi
> +                done
>              done
> -        done
> +        else
> +            TESTS=$ALLTESTS
> +            PTSTESTS=$ALLPTSTESTS
> +            RTTESTS=$ALLRTTESTS
> +        fi
>
>          # Let some of the tests fail, the information will be in the result xlsx file
>          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> ./ltp_target_run.sh"
> --
> 2.20.1
>

I have NOT applied this one.  Please change this to check for the string "all"
in FUNCTIONAL_LTP_TESTS (and please submit a sample spec to go with this).

Thanks,
 -- Tim


[-- Attachment #2: Type: text/html, Size: 17408 bytes --]

Re: [Fuego] Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[Tim.Bird]

The patch looks good. Thanks for the changes.
But I can't figure out how to apply it.

The patch content is encoded in base64, and when I tried to manually 
decode it at the command line, I got an error "invalid input".

Can you please re-send this patch as an attachment instead of inline
in the message body?  Something is going wrong with the mail
handling of the text.
 -- Tim

> -----Original Message-----
> From: tho1.nguyendat@toshiba.co.jp <tho1.nguyendat@toshiba.co.jp>
> Dear Tim,
> 
> I would like to send my update.
> 
> From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> 
> 
> Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> 
> 
> ---
>  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
>  tests/Functional.LTP/spec.json     |  6 +++++
>  2 files changed, 30 insertions(+), 18 deletions(-)
> 
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index 85adabf..689abfe 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -414,25 +414,31 @@ function test_run {
>          # on the type of the test (regular, posize, or realtime)
>          # Separate the test list by type, and pass the tests in different
>          # variables
> -        for a in $FUNCTIONAL_LTP_TESTS; do
> -            for b in $ALLTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    TESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLPTSTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    PTSTESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLRTTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    RTTESTS+="$a "
> -                fi
> +        if [ "$FUNCTIONAL_LTP_TESTS" == "all" ]; then
> +            TESTS=$ALLTESTS
> +            PTSTESTS=$ALLPTSTESTS
> +            RTTESTS=$ALLRTTESTS
> +        else
> +            for a in $FUNCTIONAL_LTP_TESTS; do
> +                for b in $ALLTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        TESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLPTSTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        PTSTESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLRTTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        RTTESTS+="$a "
> +                    fi
> +                done
>              done
> -        done
> +        fi
> 
>          # Let some of the tests fail, the information will be in the result xlsx file
>          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> ./ltp_target_run.sh"
> diff --git a/tests/Functional.LTP/spec.json b/tests/Functional.LTP/spec.json
> index d30ba4c..53ee115 100644
> --- a/tests/Functional.LTP/spec.json
> +++ b/tests/Functional.LTP/spec.json
> @@ -1,6 +1,12 @@
>  {
>      "testName": "Functional.LTP",
>      "specs": {
> +        "all": {
> +            "tests": "all",
> +            "noautoskip": "true",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
>          "default": {
>              "tests": "syscalls SEM",
>              "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> --
> 2.20.1
> 
> ________________________________
> 
> Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Đã gửi: 23 Tháng Bảy 2021 2:03 SA
> Đến: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
> Cc: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>; sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ)
> <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org <fuego@lists.linuxfoundation.org>
> Chủ đề: RE: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> 
> This one has problems.
> 
> > -----Original Message-----
> > From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> > Subject: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> >
> I'm not sure that the summary line of this change matches what it does.
> 
> From reading the code, it appears that "tests" in the spec needs to be
> missing or empty, in order to activate this.  If "tests" is "all",
> then it appears that none of the tests will match, and you'd just
> end up with empty lists.
> 
> Also, I had to intuit the reasoning here, since there is no description
> for this.  I don't mind the concept of being able to easily run all
> LTP tests, but it would be nice to use the string "all" to signal
> this instead of just leaving FUNCTIONAL_LTP_TESTS empty.
> 
> It should probably go into a comment in the code as well, and maybe
> in the test.yaml file, so that users and future developers will see
> it and understand the behaviour.
> 
> Also, you might want to add a spec to the spec.json file for "all", to
> show it's usage.
> 
> > From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> >
> > Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > ---
> >  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
> >  1 file changed, 24 insertions(+), 18 deletions(-)
> >
> > diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> > index e620b17..16d556f 100755
> > --- a/tests/Functional.LTP/fuego_test.sh
> > +++ b/tests/Functional.LTP/fuego_test.sh
> > @@ -188,7 +188,6 @@ function test_pre_check {
> >      assert_define AR
> >      assert_define RANLIB
> >      #assert_define LDFLAGS
> > -    assert_define FUNCTIONAL_LTP_TESTS
> >
> >      # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
> >      echo "Tests skipped by default in Fuego for now"
> > @@ -453,25 +452,32 @@ function test_run {
> >          # on the type of the test (regular, posize, or realtime)
> >          # Separate the test list by type, and pass the tests in different
> >          # variables
> > -        for a in $FUNCTIONAL_LTP_TESTS; do
> > -            for b in $ALLTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    TESTS+="$a "
> > -                fi
> > -            done
> > -
> > -            for b in $ALLPTSTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    PTSTESTS+="$a "
> > -                fi
> > -            done
> >
> > -            for b in $ALLRTTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    RTTESTS+="$a "
> > -                fi
> > +        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
> > +            for a in $FUNCTIONAL_LTP_TESTS; do
> > +                for b in $ALLTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        TESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLPTSTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        PTSTESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLRTTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        RTTESTS+="$a "
> > +                    fi
> > +                done
> >              done
> > -        done
> > +        else
> > +            TESTS=$ALLTESTS
> > +            PTSTESTS=$ALLPTSTESTS
> > +            RTTESTS=$ALLRTTESTS
> > +        fi
> >
> >          # Let some of the tests fail, the information will be in the result xlsx file
> >          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> > ./ltp_target_run.sh"
> > --
> > 2.20.1
> >
> 
> I have NOT applied this one.  Please change this to check for the string "all"
> in FUNCTIONAL_LTP_TESTS (and please submit a sample spec to go with this).
> 
> Thanks,
>  -- Tim
> 

[Fuego] Trả lời: Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[tho1.nguyendat]

[-- Attachment #1.1: Type: text/plain, Size: 9222 bytes --]

Hi Tim,

I attached the patch file.
Could you please try it?

Thanks,
________________________________
Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
Đã gửi: 28 Tháng Bảy 2021 6:00 SA
Đến: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>
Cc: sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org <fuego@lists.linuxfoundation.org>; pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
Chủ đề: RE: Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

The patch looks good. Thanks for the changes.
But I can't figure out how to apply it.

The patch content is encoded in base64, and when I tried to manually
decode it at the command line, I got an error "invalid input".

Can you please re-send this patch as an attachment instead of inline
in the message body?  Something is going wrong with the mail
handling of the text.
 -- Tim

> -----Original Message-----
> From: tho1.nguyendat@toshiba.co.jp <tho1.nguyendat@toshiba.co.jp>
> Dear Tim,
>
> I would like to send my update.
>
> From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
>
>
> Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
>
>
> ---
>  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
>  tests/Functional.LTP/spec.json     |  6 +++++
>  2 files changed, 30 insertions(+), 18 deletions(-)
>
> diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> index 85adabf..689abfe 100755
> --- a/tests/Functional.LTP/fuego_test.sh
> +++ b/tests/Functional.LTP/fuego_test.sh
> @@ -414,25 +414,31 @@ function test_run {
>          # on the type of the test (regular, posize, or realtime)
>          # Separate the test list by type, and pass the tests in different
>          # variables
> -        for a in $FUNCTIONAL_LTP_TESTS; do
> -            for b in $ALLTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    TESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLPTSTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    PTSTESTS+="$a "
> -                fi
> -            done
> -
> -            for b in $ALLRTTESTS; do
> -                if [ "$a" == "$b" ]; then
> -                    RTTESTS+="$a "
> -                fi
> +        if [ "$FUNCTIONAL_LTP_TESTS" == "all" ]; then
> +            TESTS=$ALLTESTS
> +            PTSTESTS=$ALLPTSTESTS
> +            RTTESTS=$ALLRTTESTS
> +        else
> +            for a in $FUNCTIONAL_LTP_TESTS; do
> +                for b in $ALLTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        TESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLPTSTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        PTSTESTS+="$a "
> +                    fi
> +                done
> +
> +                for b in $ALLRTTESTS; do
> +                    if [ "$a" == "$b" ]; then
> +                        RTTESTS+="$a "
> +                    fi
> +                done
>              done
> -        done
> +        fi
>
>          # Let some of the tests fail, the information will be in the result xlsx file
>          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> ./ltp_target_run.sh"
> diff --git a/tests/Functional.LTP/spec.json b/tests/Functional.LTP/spec.json
> index d30ba4c..53ee115 100644
> --- a/tests/Functional.LTP/spec.json
> +++ b/tests/Functional.LTP/spec.json
> @@ -1,6 +1,12 @@
>  {
>      "testName": "Functional.LTP",
>      "specs": {
> +        "all": {
> +            "tests": "all",
> +            "noautoskip": "true",
> +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> +        },
>          "default": {
>              "tests": "syscalls SEM",
>              "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> --
> 2.20.1
>
> ________________________________
>
> Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Đã gửi: 23 Tháng Bảy 2021 2:03 SA
> Đến: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
> Cc: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>; sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ)
> <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org <fuego@lists.linuxfoundation.org>
> Chủ đề: RE: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
>
> This one has problems.
>
> > -----Original Message-----
> > From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> > Subject: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> >
> I'm not sure that the summary line of this change matches what it does.
>
> From reading the code, it appears that "tests" in the spec needs to be
> missing or empty, in order to activate this.  If "tests" is "all",
> then it appears that none of the tests will match, and you'd just
> end up with empty lists.
>
> Also, I had to intuit the reasoning here, since there is no description
> for this.  I don't mind the concept of being able to easily run all
> LTP tests, but it would be nice to use the string "all" to signal
> this instead of just leaving FUNCTIONAL_LTP_TESTS empty.
>
> It should probably go into a comment in the code as well, and maybe
> in the test.yaml file, so that users and future developers will see
> it and understand the behaviour.
>
> Also, you might want to add a spec to the spec.json file for "all", to
> show it's usage.
>
> > From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> >
> > Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > ---
> >  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
> >  1 file changed, 24 insertions(+), 18 deletions(-)
> >
> > diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> > index e620b17..16d556f 100755
> > --- a/tests/Functional.LTP/fuego_test.sh
> > +++ b/tests/Functional.LTP/fuego_test.sh
> > @@ -188,7 +188,6 @@ function test_pre_check {
> >      assert_define AR
> >      assert_define RANLIB
> >      #assert_define LDFLAGS
> > -    assert_define FUNCTIONAL_LTP_TESTS
> >
> >      # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
> >      echo "Tests skipped by default in Fuego for now"
> > @@ -453,25 +452,32 @@ function test_run {
> >          # on the type of the test (regular, posize, or realtime)
> >          # Separate the test list by type, and pass the tests in different
> >          # variables
> > -        for a in $FUNCTIONAL_LTP_TESTS; do
> > -            for b in $ALLTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    TESTS+="$a "
> > -                fi
> > -            done
> > -
> > -            for b in $ALLPTSTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    PTSTESTS+="$a "
> > -                fi
> > -            done
> >
> > -            for b in $ALLRTTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    RTTESTS+="$a "
> > -                fi
> > +        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
> > +            for a in $FUNCTIONAL_LTP_TESTS; do
> > +                for b in $ALLTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        TESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLPTSTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        PTSTESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLRTTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        RTTESTS+="$a "
> > +                    fi
> > +                done
> >              done
> > -        done
> > +        else
> > +            TESTS=$ALLTESTS
> > +            PTSTESTS=$ALLPTSTESTS
> > +            RTTESTS=$ALLRTTESTS
> > +        fi
> >
> >          # Let some of the tests fail, the information will be in the result xlsx file
> >          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> > ./ltp_target_run.sh"
> > --
> > 2.20.1
> >
>
> I have NOT applied this one.  Please change this to check for the string "all"
> in FUNCTIONAL_LTP_TESTS (and please submit a sample spec to go with this).
>
> Thanks,
>  -- Tim
>


[-- Attachment #1.2: Type: text/html, Size: 20093 bytes --]

[-- Attachment #2: 0001-LTP-execute-all-tests-when-spec-specifies-all.patch --]
[-- Type: application/octet-stream, Size: 3182 bytes --]

From b0077de9fa9ba8d1ad18b282ba7d3ec6ece6532f Mon Sep 17 00:00:00 2001
From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
Date: Tue, 27 Jul 2021 08:39:42 +0700
Subject: [PATCH] LTP: execute all tests when spec specifies "all"

Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
---
 tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
 tests/Functional.LTP/spec.json     |  6 +++++
 2 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
index 85adabf..689abfe 100755
--- a/tests/Functional.LTP/fuego_test.sh
+++ b/tests/Functional.LTP/fuego_test.sh
@@ -414,25 +414,31 @@ function test_run {
         # on the type of the test (regular, posize, or realtime)
         # Separate the test list by type, and pass the tests in different
         # variables
-        for a in $FUNCTIONAL_LTP_TESTS; do
-            for b in $ALLTESTS; do
-                if [ "$a" == "$b" ]; then
-                    TESTS+="$a "
-                fi
-            done
-
-            for b in $ALLPTSTESTS; do
-                if [ "$a" == "$b" ]; then
-                    PTSTESTS+="$a "
-                fi
-            done
-
-            for b in $ALLRTTESTS; do
-                if [ "$a" == "$b" ]; then
-                    RTTESTS+="$a "
-                fi
+        if [ "$FUNCTIONAL_LTP_TESTS" == "all" ]; then
+            TESTS=$ALLTESTS
+            PTSTESTS=$ALLPTSTESTS
+            RTTESTS=$ALLRTTESTS
+        else
+            for a in $FUNCTIONAL_LTP_TESTS; do
+                for b in $ALLTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        TESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLPTSTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        PTSTESTS+="$a "
+                    fi
+                done
+
+                for b in $ALLRTTESTS; do
+                    if [ "$a" == "$b" ]; then
+                        RTTESTS+="$a "
+                    fi
+                done
             done
-        done
+        fi
 
         # Let some of the tests fail, the information will be in the result xlsx file
         report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\"; ./ltp_target_run.sh"
diff --git a/tests/Functional.LTP/spec.json b/tests/Functional.LTP/spec.json
index d30ba4c..53ee115 100644
--- a/tests/Functional.LTP/spec.json
+++ b/tests/Functional.LTP/spec.json
@@ -1,6 +1,12 @@
 {
     "testName": "Functional.LTP",
     "specs": {
+        "all": {
+            "tests": "all",
+            "noautoskip": "true",
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
+        },
         "default": {
             "tests": "syscalls SEM",
             "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
-- 
2.20.1

Re: [Fuego] Trả lời: Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"

[Tim.Bird]

> -----Original Message-----
> From: tho1.nguyendat@toshiba.co.jp <tho1.nguyendat@toshiba.co.jp>
> 
> Hi Tim,
> 
> I attached the patch file.
> Could you please try it?

This worked.  This patch is applied and pushed.  Can you please pull the latest
master branch, and test this out in your lab?

Thanks,
 -- Tim

> ________________________________
> 
> Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Đã gửi: 28 Tháng Bảy 2021 6:00 SA
> Đến: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>
> Cc: sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ) <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org
> <fuego@lists.linuxfoundation.org>; pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
> Chủ đề: RE: Trả lời: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> 
> The patch looks good. Thanks for the changes.
> But I can't figure out how to apply it.
> 
> The patch content is encoded in base64, and when I tried to manually
> decode it at the command line, I got an error "invalid input".
> 
> Can you please re-send this patch as an attachment instead of inline
> in the message body?  Something is going wrong with the mail
> handling of the text.
>  -- Tim
> 
> > -----Original Message-----
> > From: tho1.nguyendat@toshiba.co.jp <tho1.nguyendat@toshiba.co.jp>
> > Dear Tim,
> >
> > I would like to send my update.
> >
> > From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> >
> >
> > Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> >
> >
> > ---
> >  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
> >  tests/Functional.LTP/spec.json     |  6 +++++
> >  2 files changed, 30 insertions(+), 18 deletions(-)
> >
> > diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> > index 85adabf..689abfe 100755
> > --- a/tests/Functional.LTP/fuego_test.sh
> > +++ b/tests/Functional.LTP/fuego_test.sh
> > @@ -414,25 +414,31 @@ function test_run {
> >          # on the type of the test (regular, posize, or realtime)
> >          # Separate the test list by type, and pass the tests in different
> >          # variables
> > -        for a in $FUNCTIONAL_LTP_TESTS; do
> > -            for b in $ALLTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    TESTS+="$a "
> > -                fi
> > -            done
> > -
> > -            for b in $ALLPTSTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    PTSTESTS+="$a "
> > -                fi
> > -            done
> > -
> > -            for b in $ALLRTTESTS; do
> > -                if [ "$a" == "$b" ]; then
> > -                    RTTESTS+="$a "
> > -                fi
> > +        if [ "$FUNCTIONAL_LTP_TESTS" == "all" ]; then
> > +            TESTS=$ALLTESTS
> > +            PTSTESTS=$ALLPTSTESTS
> > +            RTTESTS=$ALLRTTESTS
> > +        else
> > +            for a in $FUNCTIONAL_LTP_TESTS; do
> > +                for b in $ALLTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        TESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLPTSTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        PTSTESTS+="$a "
> > +                    fi
> > +                done
> > +
> > +                for b in $ALLRTTESTS; do
> > +                    if [ "$a" == "$b" ]; then
> > +                        RTTESTS+="$a "
> > +                    fi
> > +                done
> >              done
> > -        done
> > +        fi
> >
> >          # Let some of the tests fail, the information will be in the result xlsx file
> >          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> > ./ltp_target_run.sh"
> > diff --git a/tests/Functional.LTP/spec.json b/tests/Functional.LTP/spec.json
> > index d30ba4c..53ee115 100644
> > --- a/tests/Functional.LTP/spec.json
> > +++ b/tests/Functional.LTP/spec.json
> > @@ -1,6 +1,12 @@
> >  {
> >      "testName": "Functional.LTP",
> >      "specs": {
> > +        "all": {
> > +            "tests": "all",
> > +            "noautoskip": "true",
> > +            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> > +            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
> > +        },
> >          "default": {
> >              "tests": "syscalls SEM",
> >              "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
> > --
> > 2.20.1
> >
> > ________________________________
> >
> > Từ: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > Đã gửi: 23 Tháng Bảy 2021 2:03 SA
> > Đến: pyla venkata(ＴＳＩＰ) <Venkata.Pyla@toshiba-tsip.com>
> > Cc: nguyen dat tho(ＴＳＤＶ Ｅｎｇ　１) <tho1.nguyendat@toshiba.co.jp>; sangorrin daniel(サンゴリン ダニエル □ＳＷＣ◯ＡＣＴ)
> > <daniel.sangorrin@toshiba.co.jp>; fuego@lists.linuxfoundation.org <fuego@lists.linuxfoundation.org>
> > Chủ đề: RE: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> >
> > This one has problems.
> >
> > > -----Original Message-----
> > > From: venkata.pyla@toshiba-tsip.com <venkata.pyla@toshiba-tsip.com>
> > > Subject: [fuego-core 6/7] LTP: execute all tests when spec specifies "all"
> > >
> > I'm not sure that the summary line of this change matches what it does.
> >
> > From reading the code, it appears that "tests" in the spec needs to be
> > missing or empty, in order to activate this.  If "tests" is "all",
> > then it appears that none of the tests will match, and you'd just
> > end up with empty lists.
> >
> > Also, I had to intuit the reasoning here, since there is no description
> > for this.  I don't mind the concept of being able to easily run all
> > LTP tests, but it would be nice to use the string "all" to signal
> > this instead of just leaving FUNCTIONAL_LTP_TESTS empty.
> >
> > It should probably go into a comment in the code as well, and maybe
> > in the test.yaml file, so that users and future developers will see
> > it and understand the behaviour.
> >
> > Also, you might want to add a spec to the spec.json file for "all", to
> > show it's usage.
> >
> > > From: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> > >
> > > Signed-off-by: Nguyen Dat Tho <tho1.nguyendat@toshiba.co.jp>
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > Signed-off-by: venkata pyla <venkata.pyla@toshiba-tsip.com>
> > > ---
> > >  tests/Functional.LTP/fuego_test.sh | 42 +++++++++++++++++-------------
> > >  1 file changed, 24 insertions(+), 18 deletions(-)
> > >
> > > diff --git a/tests/Functional.LTP/fuego_test.sh b/tests/Functional.LTP/fuego_test.sh
> > > index e620b17..16d556f 100755
> > > --- a/tests/Functional.LTP/fuego_test.sh
> > > +++ b/tests/Functional.LTP/fuego_test.sh
> > > @@ -188,7 +188,6 @@ function test_pre_check {
> > >      assert_define AR
> > >      assert_define RANLIB
> > >      #assert_define LDFLAGS
> > > -    assert_define FUNCTIONAL_LTP_TESTS
> > >
> > >      # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
> > >      echo "Tests skipped by default in Fuego for now"
> > > @@ -453,25 +452,32 @@ function test_run {
> > >          # on the type of the test (regular, posize, or realtime)
> > >          # Separate the test list by type, and pass the tests in different
> > >          # variables
> > > -        for a in $FUNCTIONAL_LTP_TESTS; do
> > > -            for b in $ALLTESTS; do
> > > -                if [ "$a" == "$b" ]; then
> > > -                    TESTS+="$a "
> > > -                fi
> > > -            done
> > > -
> > > -            for b in $ALLPTSTESTS; do
> > > -                if [ "$a" == "$b" ]; then
> > > -                    PTSTESTS+="$a "
> > > -                fi
> > > -            done
> > >
> > > -            for b in $ALLRTTESTS; do
> > > -                if [ "$a" == "$b" ]; then
> > > -                    RTTESTS+="$a "
> > > -                fi
> > > +        if [ -n "$FUNCTIONAL_LTP_TESTS" ]; then
> > > +            for a in $FUNCTIONAL_LTP_TESTS; do
> > > +                for b in $ALLTESTS; do
> > > +                    if [ "$a" == "$b" ]; then
> > > +                        TESTS+="$a "
> > > +                    fi
> > > +                done
> > > +
> > > +                for b in $ALLPTSTESTS; do
> > > +                    if [ "$a" == "$b" ]; then
> > > +                        PTSTESTS+="$a "
> > > +                    fi
> > > +                done
> > > +
> > > +                for b in $ALLRTTESTS; do
> > > +                    if [ "$a" == "$b" ]; then
> > > +                        RTTESTS+="$a "
> > > +                    fi
> > > +                done
> > >              done
> > > -        done
> > > +        else
> > > +            TESTS=$ALLTESTS
> > > +            PTSTESTS=$ALLPTSTESTS
> > > +            RTTESTS=$ALLRTTESTS
> > > +        fi
> > >
> > >          # Let some of the tests fail, the information will be in the result xlsx file
> > >          report "cd $LTP_DESTDIR; export TESTS=\"$TESTS\"; export PTSTESTS=\"$PTSTESTS\"; export RTTESTS=\"$RTTESTS\";
> > > ./ltp_target_run.sh"
> > > --
> > > 2.20.1
> > >
> >
> > I have NOT applied this one.  Please change this to check for the string "all"
> > in FUNCTIONAL_LTP_TESTS (and please submit a sample spec to go with this).
> >
> > Thanks,
> >  -- Tim
> >
>