[Fuego] LTP prechecks

[Fuego] LTP prechecks

[Daniel Sangorrin]

Hi Tim,

I send you a batch of patches that do various prechecks and
autogenerate a skiplist when you run LTP on a target.
The list is not exhausitive yet but I'd like to have a first
review and hopefully have other members contribute with
their experencies or knowledge.

[PATCH 01/18] need_check: correct errata
[PATCH 02/18] need_check: cache the kconfig file
[PATCH 03/18] need_check: no need for testing if the file exists
[PATCH 04/18] need_check: check_kconfig should return 1 on first
[PATCH 05/18] LTP: we need root permissions
[PATCH 06/18] LTP: patch runtests before running
[PATCH 07/18] LTP: reduce the time required for two more tests
[PATCH 08/18] LTP: put the skiplist on the log directory
[PATCH 09/18] LTP: show the generated skiplist file on the jenkins
[PATCH 10/18] LTP: skip a few problematic tests for now
[PATCH 11/18] LTP: add tests to skip if tmp is mounted on tmpfs
[PATCH 12/18] LTP: skip tests that depend on specific kernel commits
[PATCH 13/18] LTP: skip tests depending on the architecture
[PATCH 14/18] LTP: skip tests depending on the target kernel config
[PATCH 15/18] LTP: skip tests that depend on unavailable commands
[PATCH 16/18] LTP: skip tests depending on the config.h generated by
[PATCH 17/18] LTP: add note about typical LTP build failure on arm
[PATCH 18/18] LTP: remove trailing space

Thanks,
Daniel

[Fuego] [PATCH 01/18] need_check: correct errata

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/need_check.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
index 1fd6d6d..04c551b 100755
--- a/engine/scripts/need_check.sh
+++ b/engine/scripts/need_check.sh
@@ -219,7 +219,7 @@ function get_full_kconfig {
   return 1
 }
 
-# #1 has a single kconfig entry to check
+# $1 has a single kconfig entry to check
 function check_one_kconfig {
   local cfg=$1
 
-- 
2.7.4

[Fuego] [PATCH 02/18] need_check: cache the kconfig file

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/need_check.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
index 04c551b..58f76c3 100755
--- a/engine/scripts/need_check.sh
+++ b/engine/scripts/need_check.sh
@@ -265,8 +265,10 @@ function check_kconfig {
   arg_array=($1)
   set +f
 
-  get_full_kconfig $LOGDIR
   kconf_filename=$LOGDIR/kconfig
+  if [ ! -f $kconf_filename ]; then
+    get_full_kconfig $LOGDIR
+  fi
 
   for cfg in "${arg_array[@]}" ; do
     echo "Checking cfg: [$cfg]"
-- 
2.7.4

[Fuego] [PATCH 03/18] need_check: no need for testing if the file exists

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/need_check.sh | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
index 58f76c3..d85e89a 100755
--- a/engine/scripts/need_check.sh
+++ b/engine/scripts/need_check.sh
@@ -173,22 +173,18 @@ function get_full_kconfig {
   gfk_board_tmpfile=$(mktemp)
   gfk_host_tmpfile=$(mktemp)
 
-  # try /proc/config.gz first
-  if cmd "test -f /proc/config.gz" ; then
-    if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
-      get $gfk_board_tmpfile $out_filename
-      return 0
-    fi
-  else
-    echo "/proc/config.gz not found"
+  if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
+    get $gfk_board_tmpfile $out_filename
+    return 0
   fi
 
-  # try /boot directory
   cmd "uname -r >$gfk_board_tmpfile"
   get $gfk_board_tmpfile $gfk_host_tmpfile
   kver=$(cat $gfk_host_tmpfile)
   conf_filename="/boot/config-$kver"
 
+  echo "/proc/config.gz not found, trying $conf_filename"
+
   if cmd "test -f $conf_filename" ; then
     get $conf_filename $out_filename
     return 0
-- 
2.7.4

[Fuego] [PATCH 04/18] need_check: check_kconfig should return 1 on first failure

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/need_check.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
index d85e89a..aa96cbe 100755
--- a/engine/scripts/need_check.sh
+++ b/engine/scripts/need_check.sh
@@ -268,7 +268,9 @@ function check_kconfig {
 
   for cfg in "${arg_array[@]}" ; do
     echo "Checking cfg: [$cfg]"
-    check_one_kconfig $cfg
+    if ! check_one_kconfig $cfg; then
+        return 1
+    fi
   done
 }
 
-- 
2.7.4

[Fuego] [PATCH 05/18] LTP: we need root permissions

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 2473cf1..1305421 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -1,6 +1,8 @@
 # FIXTHIS: use https://github.com/linux-test-project/ltp.git instead of a tarball
 tarball=ltp-7eb77fbfd80.tar.bz2
 
+NEED_ROOT=1
+
 ALLTESTS="
 admin_tools         fs_perms_simple       ltp-aiodio.part3  net_stress.appl                   quickhit
 can                 fs_readonly           ltp-aiodio.part4  net_stress.broken_ip              rpc_tests
-- 
2.7.4

[Fuego] [PATCH 06/18] LTP: patch runtests before running

[Daniel Sangorrin]

The reason is that we want to support use cases where the
LTP is installed by the user.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh        |  5 +++--
 engine/tests/Functional.LTP/shorter-fork13.patch | 11 -----------
 2 files changed, 3 insertions(+), 13 deletions(-)
 delete mode 100644 engine/tests/Functional.LTP/shorter-fork13.patch

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 1305421..8278e98 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -117,8 +117,6 @@ function test_pre_check {
 function test_build {
     if [[ "$FUNCTIONAL_LTP_PHASES" == *build* ]] ; then
         echo "Building LTP"
-        patch -p1 -N -s < $TEST_HOME/shorter-fork13.patch
-
         # Build the LTP tests
         make autotools
         ./configure CC="${CC}" AR="${AR}" RANLIB="${RANLIB}" LDFLAGS="$LDFLAGS" SYSROOT="${SDKROOT}" \
@@ -190,6 +188,9 @@ function test_run {
     if [[ "$FUNCTIONAL_LTP_PHASES" == *run* ]] ; then
         echo "Running LTP"
 
+        # we patch them here because of the different phases
+        cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/' $LTP_DESTDIR/runtest/syscalls"
+
         TESTS=""
         PTSTESTS=""
         RTTESTS=""
diff --git a/engine/tests/Functional.LTP/shorter-fork13.patch b/engine/tests/Functional.LTP/shorter-fork13.patch
deleted file mode 100644
index 09e8365..0000000
--- a/engine/tests/Functional.LTP/shorter-fork13.patch
+++ /dev/null
@@ -1,11 +0,0 @@
---- a/runtest/syscalls	2017-08-25 00:15:46.584034299 +0000
-+++ b/runtest/syscalls	2017-08-25 00:15:33.880034153 +0000
-@@ -295,7 +295,7 @@ fork08 fork08
- fork09 fork09
- fork10 fork10
- fork11 fork11
--fork13 fork13 -i 1000000
-+fork13 fork13 -i 100000
- fork14 fork14
- 
- fpathconf01 fpathconf01
-- 
2.7.4

[Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 8278e98..ae85bac 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -190,6 +190,8 @@ function test_run {
 
         # we patch them here because of the different phases
         cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/' $LTP_DESTDIR/runtest/syscalls"
+        cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30 diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
+        cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/' $LTP_DESTDIR/runtest/syscalls"
 
         TESTS=""
         PTSTESTS=""
-- 
2.7.4

[Fuego] [PATCH 08/18] LTP: put the skiplist on the log directory

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index ae85bac..faec158 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -138,11 +138,10 @@ function test_build {
 }
 
 function test_deploy {
-    # generate the skip list for this test
-    echo "# skip these test cases" > skiplist.txt
+    # add tests to skip if defined on the spec
     if [ -n "${FUNCTIONAL_LTP_SKIPLIST}" ]; then
         for item in $FUNCTIONAL_LTP_SKIPLIST; do
-            echo $item >> skiplist.txt
+            echo "$item" >> ${LOGDIR}/skiplist.txt
         done
     fi
 
@@ -159,7 +158,7 @@ function test_deploy {
             awk '/^[^#]/ { print "rm -f ltp/testcases/bin/" $2 }' ltp/runtest/syscalls | sh
         fi
 
-        cp skiplist.txt ltp
+        cp ${LOGDIR}/skiplist.txt ltp
 
         put ltp/* $LTP_DESTDIR
         rm -rf ltp
@@ -180,7 +179,7 @@ function test_deploy {
        cmd "mkdir -p $FUNCTIONAL_LTP_HOMEDIR/testcases/realtime/scripts"
        put testcases/realtime/scripts/setenv.sh $FUNCTIONAL_LTP_HOMEDIR/testcases/realtime/scripts
        put $TEST_HOME/ltp_target_run.sh $FUNCTIONAL_LTP_HOMEDIR
-       put skiplist.txt $FUNCTIONAL_LTP_HOMEDIR
+       put ${LOGDIR}/skiplist.txt $FUNCTIONAL_LTP_HOMEDIR
     fi
 }
 
-- 
2.7.4

[Fuego] [PATCH 09/18] LTP: show the generated skiplist file on the jenkins results

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/spec.json | 44 +++++++++++++++++------------------
 1 file changed, 22 insertions(+), 22 deletions(-)

diff --git a/engine/tests/Functional.LTP/spec.json b/engine/tests/Functional.LTP/spec.json
index 528d9bc..065ee99 100644
--- a/engine/tests/Functional.LTP/spec.json
+++ b/engine/tests/Functional.LTP/spec.json
@@ -3,19 +3,19 @@
     "specs": {
         "default": {
             "tests": "syscalls SEM",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "docker": {
             "tests": "syscalls",
             "skiplist": "kill10 keyctl01",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "selection": {
             "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "install": {
             "tests": "syscalls",
@@ -29,44 +29,44 @@
         "selectionwithrt": {
             "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter",
             "skiplist": "kill10 keyctl01 inotify06",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "ltplite": {
             "tests": "ltplite",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "ptsonly": {
             "tests": "AIO MEM MSG SEM SIG THR TMR TPS",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "smoketest": {
             "tests": "fs_bind",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "quickhit": {
             "tests": "quickhit",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "rtonly": {
             "tests": "func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "somefail": {
             "tests": "timers pty",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         },
         "quickhitwithskips": {
             "tests": "quickhit",
             "skiplist": "abort01",
-            "extra_success_links": {"xlsx": "results.xlsx"},
-            "extra_fail_links": {"xlsx": "results.xlsx"}
+            "extra_success_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"},
+            "extra_fail_links": {"xlsx": "results.xlsx", "skiplist": "skiplist.txt"}
         }
     }
 }
-- 
2.7.4

[Fuego] [PATCH 10/18] LTP: skip a few problematic tests for now

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index faec158..5b9cb2b 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -80,6 +80,19 @@ func/sched_jitter
 #  user should copy tar to board, and install it, then set board file's
 #    FUNCTIONAL_LTP_HOMEDIR var
 
+# $1: string with tests to skip separated by spaces
+function skip_tests {
+    # split $1 on whitespace, without file globbing
+    set -f
+    local TESTS=($1)
+    set +f
+
+    echo "Skipping tests: ${TESTS[@]}."
+    for testname in "${TESTS[@]}"; do
+        echo "$testname" >> ${LOGDIR}/skiplist.txt
+    done
+}
+
 function test_pre_check {
     if [ -n "${FUNCTIONAL_LTP_HOMEDIR}" ] ; then
         # user or spec has specified a home directory for LTP.
@@ -112,6 +125,21 @@ function test_pre_check {
     assert_define RANLIB
     assert_define LDFLAGS
     assert_define FUNCTIONAL_LTP_TESTS
+
+    # FIXTHIS: use regex for selecting tests to skip once merged on LTP upstream
+    echo "Tests skipped by default in Fuego for now"
+    echo "# skip these test cases" > ${LOGDIR}/skiplist.txt
+    skip_tests "su01" # too complicated to setup
+    skip_tests "sbrk03" # Only works in 32bit on s390 series system
+    skip_tests "switch01" # Requires a 64-bit processor that supports little-endian mode,such as POWER6.
+    skip_tests "ptrace04" # Requires blackfin processor
+    skip_tests "cacheflush01" # Available only on MIPS-based systems
+    skip_tests "bdflush01" # bdflush was deprecated by pdflush and pdflush removed from the kernel (https://lwn.net/Articles/508212/)
+    skip_tests "fcntl06" # Linux kernel doesn't implement R_GETLK/R_SETLK
+    skip_tests "fcntl06_64" # Linux kernel doesn't implement R_GETLK/R_SETLK
+    skip_tests "munlockall02" # munlockall can be called form user space
+    skip_tests "openat02" # Requires a filesystem with noatime and relatime
+    skip_tests "utimensat01" # kernel patch f2b20f6ee842313a changed return value from -EACCESS to EPERM when a file is immutable and update time is NULL or UTIME_NOW
 }
 
 function test_build {
-- 
2.7.4

[Fuego] [PATCH 11/18] LTP: add tests to skip if tmp is mounted on tmpfs

[Daniel Sangorrin]

Note: there are more tests to be added here but at least
we have the skeleton.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 5b9cb2b..e516dd8 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -140,6 +140,18 @@ function test_pre_check {
     skip_tests "munlockall02" # munlockall can be called form user space
     skip_tests "openat02" # Requires a filesystem with noatime and relatime
     skip_tests "utimensat01" # kernel patch f2b20f6ee842313a changed return value from -EACCESS to EPERM when a file is immutable and update time is NULL or UTIME_NOW
+
+    echo "Tests skipped if the target /tmp folder is mounted on tmpfs"
+    target_mounts=$(mktemp)
+    if get "/proc/mounts" $target_mounts; then
+        if grep "^tmpfs /tmp " $target_mounts; then
+            echo "WARNING: /tmp is mounted using tmpfs."
+            skip_tests "fallocate04"
+        fi
+    else
+        echo "WARNING: could not check if /tmp is mounted using tmpfs"
+    fi
+    rm -f $target_mounts
 }
 
 function test_build {
-- 
2.7.4

[Fuego] [PATCH 12/18] LTP: skip tests that depend on specific kernel commits

[Daniel Sangorrin]

Note: this should be fixed in the future with a check
on the kernel version.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index e516dd8..cd64867 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -152,6 +152,16 @@ function test_pre_check {
         echo "WARNING: could not check if /tmp is mounted using tmpfs"
     fi
     rm -f $target_mounts
+
+    # Tests skipped by Linux kernel version
+    # FIXTHIS: compare with the kernel version before adding tests to the skiplist
+    kver=$(cmd "uname -r")
+    echo "kernel version: $kver"
+    skip_tests "adjtimex02" # needs a kernel that doesn't normalize buf.offset if it is outside the acceptable range
+    skip_tests "fallocate04" # needs Linux 3.15 or newer
+    skip_tests "add_key02" # needs a kernel with the 5649645d725c commit
+    skip_tests "inotify06" # will loop/crash kernels that dont have commit 8f2f3eb59dff (<4.2)
+    skip_tests "ptrace03" # Only works for <2.6.26, above that the kernel allows to trace init
 }
 
 function test_build {
-- 
2.7.4

[Fuego] [PATCH 13/18] LTP: skip tests depending on the architecture

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index cd64867..d54b299 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -162,6 +162,31 @@ function test_pre_check {
     skip_tests "add_key02" # needs a kernel with the 5649645d725c commit
     skip_tests "inotify06" # will loop/crash kernels that dont have commit 8f2f3eb59dff (<4.2)
     skip_tests "ptrace03" # Only works for <2.6.26, above that the kernel allows to trace init
+
+    # Tests skipped depending on the architecture
+    if [ "$ARCHITECTURE" = "x86_64" ]; then
+        skip_tests "chown01_16 chown02_16 chown03_16 chown04_16 chown05_16"
+        skip_tests "fchown01_16 fchown02_16 fchown03_16 fchown04_16 fchown05_16"
+        skip_tests "getegid01_16 getegid02_16 geteuid01_16 geteuid02_16 getgid01_16 getgid03_16"
+        skip_tests "getgroups01_16 getgroups03_16"
+        skip_tests "getuid01_16 getuid03_16"
+        skip_tests "lchown01_16 lchown02_16 lchown03_16"
+        skip_tests "setfsgid01_16 setfsgid02_16 setfsgid03_16"
+        skip_tests "setfsuid01_16 setfsuid02_16 setfsuid03_16 setfsuid04_16"
+        skip_tests "setgid01_16 setgid02_16 setgid03_16"
+        skip_tests "setgroups01_16 setgroups02_16 setgroups03_16 setgroups04_16"
+        skip_tests "setregid01_16 setregid02_16 setregid03_16 setregid04_16"
+        skip_tests "setresgid01_16 setresgid02_16 setresgid03_16 setresgid04_16"
+        skip_tests "setresuid01_16 setresuid02_16 setresuid03_16 setresuid04_16 setresuid05_16"
+        skip_tests "setreuid01_16 setreuid02_16 setreuid03_16 setreuid04_16 setreuid05_16 setreuid06_16 setreuid07_16"
+        skip_tests "setuid01_16 setuid02_16 setuid03_16 setuid04_16"
+        skip_tests "msgrcv08" # does not work on 64 bit
+        skip_tests "readdir21" # This system call does not exist on x86-64 (man 2 readdir)
+    fi
+
+    if [ "$ARCHITECTURE" != "i386" ]; then
+        skip_tests "modify_ldt01 modify_ldt02 modify_ldt03" # Only work on i386
+    fi
 }
 
 function test_build {
-- 
2.7.4

[Fuego] [PATCH 14/18] LTP: skip tests depending on the target kernel config

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index d54b299..43d007d 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -93,6 +93,21 @@ function skip_tests {
     done
 }
 
+# $1: string with kernel config options separated by spaces
+# $2: string with LTP test names separated by spaces
+function skip_if_kconfig_differs {
+    set -f
+    local KCONFIGS=($1)
+    set +f
+    local TESTS="$2"
+
+    for cfg in ${KCONFIGS[@]}; do
+        if ! check_kconfig "$cfg"; then
+            skip_tests "${TESTS}"
+        fi
+    done
+}
+
 function test_pre_check {
     if [ -n "${FUNCTIONAL_LTP_HOMEDIR}" ] ; then
         # user or spec has specified a home directory for LTP.
@@ -163,6 +178,12 @@ function test_pre_check {
     skip_tests "inotify06" # will loop/crash kernels that dont have commit 8f2f3eb59dff (<4.2)
     skip_tests "ptrace03" # Only works for <2.6.26, above that the kernel allows to trace init
 
+    # Tests skipped depending on the configuration of the target kernel
+    skip_if_kconfig_differs "CONFIG_INOTIFY_USER=y" "inotify_init1_01 inotify_init1_02 inotify01 inotify02 inotify03 inotify04 inotify05 inotify06"
+    skip_if_kconfig_differs "CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y" "fanotify01 fanotify02 fanotify03 fanotify04 fanotify05 fanotify06"
+    skip_if_kconfig_differs "CONFIG_EXT4_FS=y CONFIG_EXT4DEV_COMPAT=y CONFIG_EXT4_FS_XATTR=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y" "ext4-nsec-timestamps"
+    skip_if_kconfig_differs "CONFIG_CHECKPOINT_RESTORE=y" "kcmp01 kcmp02 kcmp03"
+
     # Tests skipped depending on the architecture
     if [ "$ARCHITECTURE" = "x86_64" ]; then
         skip_tests "chown01_16 chown02_16 chown03_16 chown04_16 chown05_16"
-- 
2.7.4

[Fuego] [PATCH 15/18] LTP: skip tests that depend on unavailable commands

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 35 +++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 43d007d..2a6f7d7 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -93,6 +93,20 @@ function skip_tests {
     done
 }
 
+# $1: command/program to look for
+# $2: string with LTP test names separated by spaces
+function skip_if_command_unavailable {
+    local PROGRAM="$1"
+    local TESTS="$2"
+    export FOUND=""
+
+    is_on_target ${PROGRAM} FOUND /bin:/usr/bin:/usr/sbin:/usr/local/bin
+    if [ -z "${FOUND}" ]; then
+        echo "WARNING: ${PROGRAM} is not installed on the target."
+        skip_tests "${TESTS}"
+    fi
+}
+
 # $1: string with kernel config options separated by spaces
 # $2: string with LTP test names separated by spaces
 function skip_if_kconfig_differs {
@@ -178,6 +192,27 @@ function test_pre_check {
     skip_tests "inotify06" # will loop/crash kernels that dont have commit 8f2f3eb59dff (<4.2)
     skip_tests "ptrace03" # Only works for <2.6.26, above that the kernel allows to trace init
 
+    # Tests skipped depending on the availability of a command on the target
+    # FIXTHIS: only check the necessary ones
+    skip_if_command_unavailable bash "rwtest01 rwtest02 rwtest04 rwtest05 iogen01 fs_inod01 fs_di BindMounts"
+    skip_if_command_unavailable expect "su01"
+    skip_if_command_unavailable at "at_deny01 at_allow01"
+    skip_if_command_unavailable cron "cron cron02 cron_deny01 cron_allow01 cron_dirs_checks01"
+    skip_if_command_unavailable ar "ar"
+    skip_if_command_unavailable ld "ld01"
+    skip_if_command_unavailable ldd "ldd01"
+    skip_if_command_unavailable nm "nm01"
+    skip_if_command_unavailable file "file01"
+    skip_if_command_unavailable tar "tar01"
+    skip_if_command_unavailable logrotate "logrotate01"
+    skip_if_command_unavailable mkfs.ext3 "ext4-nsec-timestamps"
+    skip_if_command_unavailable mkfs.ext4 "ext4-nsec-timestamps"
+    skip_if_command_unavailable tune2fs "ext4-nsec-timestamps"
+    skip_if_command_unavailable mount "ext4-nsec-timestamps"
+    skip_if_command_unavailable umount "ext4-nsec-timestamps"
+    skip_if_command_unavailable touch "ext4-nsec-timestamps"
+    skip_if_command_unavailable quotacheck "quotactl01"
+
     # Tests skipped depending on the configuration of the target kernel
     skip_if_kconfig_differs "CONFIG_INOTIFY_USER=y" "inotify_init1_01 inotify_init1_02 inotify01 inotify02 inotify03 inotify04 inotify05 inotify06"
     skip_if_kconfig_differs "CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y" "fanotify01 fanotify02 fanotify03 fanotify04 fanotify05 fanotify06"
-- 
2.7.4

[Fuego] [PATCH 16/18] LTP: skip tests depending on the config.h generated by LTP

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 2a6f7d7..84e62d7 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -107,6 +107,25 @@ function skip_if_command_unavailable {
     fi
 }
 
+# $1: a HAVE_xxx option to grep from ltp's include/config.h (after ./configure)
+# $2: string with LTP test names separated by spaces
+# $3: optional message
+# [Note] can only be called after the configure step has finished
+function skip_if_config_unavailable {
+    local CONFIG="$1"
+    local TESTS="$2"
+    local MSG="$3"
+
+    if ! grep "#define $CONFIG" ${WORKSPACE}/${JOB_BUILD_DIR}/include/config.h; then
+        echo "WARNING: $CONFIG is not available."
+        skip_tests "${TESTS}"
+        if [ -n "$MSG" ]; then
+            echo "$MSG"
+        fi
+    fi
+
+}
+
 # $1: string with kernel config options separated by spaces
 # $2: string with LTP test names separated by spaces
 function skip_if_kconfig_differs {
@@ -254,6 +273,12 @@ function test_build {
             --with-open-posix-testsuite --with-realtime-testsuite --without-perl --without-python --target=$PREFIX --host=$PREFIX \
             --prefix=`pwd`/target_bin --build=`uname -m`-unknown-linux-gnu
 
+        # Skip tests depending on the results of configure (e.g.: SDK requirements)
+        skip_if_config_unavailable "HAVE_XFS_QUOTA" "quotactl02" "Try adding xfslibs-dev to your SDK"
+        skip_if_config_unavailable "HAVE_QUOTAV" "quotactl01 quotactl02" "Try adding libc6-dev to your SDK"
+        skip_if_config_unavailable "HAVE_NUMA" "get_mempolicy01 mbind01 migrate_pages01 migrate_pages02 move_pages01 move_pages02 move_pages03 move_pages04 move_pages05 move_pages06 move_pages07 move_pages08 move_pages09 move_pages10 move_pages11" "No NUMA support"
+        skip_if_config_unavailable "HAVE_MODIFY_LDT" "modify_ldt01 modify_ldt02 modify_ldt03" "Add modify_ldt support"
+
         # save build results in build.log
         set -o pipefail
         make CC="${CC}" 2>&1 | tee build.log
-- 
2.7.4

[Fuego] [PATCH 17/18] LTP: add note about typical LTP build failure on arm

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 84e62d7..263fc89 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -280,6 +280,10 @@ function test_build {
         skip_if_config_unavailable "HAVE_MODIFY_LDT" "modify_ldt01 modify_ldt02 modify_ldt03" "Add modify_ldt support"
 
         # save build results in build.log
+        # Typical build error:
+        #   undefined reference to `io_submit'
+        # Solution: install libaio for your arch
+        #   E.g: /fuego-ro/toolchains/install_cross_toolchain.sh armh
         set -o pipefail
         make CC="${CC}" 2>&1 | tee build.log
         set +o pipefail
-- 
2.7.4

[Fuego] [PATCH 18/18] LTP: remove trailing space

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index 263fc89..cdc643b 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -385,7 +385,7 @@ function test_run {
     else
         echo "Skipping LTP run"
 
-        # for a build-only test, put the build results into the testlog 
+        # for a build-only test, put the build results into the testlog
         # (via roundtrip to board - pretty wasteful, but oh well...)
         put build.log $BOARD_TESTDIR/fuego.$TESTDIR
         report "cat $BOARD_TESTDIR/fuego.$TESTDIR/build.log"
-- 
2.7.4

Re: [Fuego] [PATCH 01/18] need_check: correct errata

[Tim.Bird]

Applied to my master branch.

Thanks
 -- Tim


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, January 30, 2018 1:59 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 01/18] need_check: correct errata
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/need_check.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
> index 1fd6d6d..04c551b 100755
> --- a/engine/scripts/need_check.sh
> +++ b/engine/scripts/need_check.sh
> @@ -219,7 +219,7 @@ function get_full_kconfig {
>    return 1
>  }
> 
> -# #1 has a single kconfig entry to check
> +# $1 has a single kconfig entry to check
>  function check_one_kconfig {
>    local cfg=$1
> 
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 02/18] need_check: cache the kconfig file

[Bird, Timothy]

Nice addition.  Applied.

Thanks.
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, January 30, 2018 1:59 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 02/18] need_check: cache the kconfig file
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/need_check.sh | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
> index 04c551b..58f76c3 100755
> --- a/engine/scripts/need_check.sh
> +++ b/engine/scripts/need_check.sh
> @@ -265,8 +265,10 @@ function check_kconfig {
>    arg_array=($1)
>    set +f
> 
> -  get_full_kconfig $LOGDIR
>    kconf_filename=$LOGDIR/kconfig
> +  if [ ! -f $kconf_filename ]; then
> +    get_full_kconfig $LOGDIR
> +  fi
> 
>    for cfg in "${arg_array[@]}" ; do
>      echo "Checking cfg: [$cfg]"
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 03/18] need_check: no need for testing if the file exists

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> Subject: [Fuego] [PATCH 03/18] need_check: no need for testing if the file
> exists
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/need_check.sh | 14 +++++---------
>  1 file changed, 5 insertions(+), 9 deletions(-)
> 
> diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
> index 58f76c3..d85e89a 100755
> --- a/engine/scripts/need_check.sh
> +++ b/engine/scripts/need_check.sh
> @@ -173,22 +173,18 @@ function get_full_kconfig {
>    gfk_board_tmpfile=$(mktemp)
>    gfk_host_tmpfile=$(mktemp)
> 
> -  # try /proc/config.gz first
I'm not sure why you don't like my comments.  ;-)  But whatever.

> -  if cmd "test -f /proc/config.gz" ; then
> -    if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
> -      get $gfk_board_tmpfile $out_filename
> -      return 0
> -    fi
> -  else
> -    echo "/proc/config.gz not found"
> +  if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
> +    get $gfk_board_tmpfile $out_filename
> +    return 0
>    fi
> 
> -  # try /boot directory
>    cmd "uname -r >$gfk_board_tmpfile"
>    get $gfk_board_tmpfile $gfk_host_tmpfile
>    kver=$(cat $gfk_host_tmpfile)
I took the liberty of simplifying this sequence to
kver=$(cmd uname -r)

>    conf_filename="/boot/config-$kver"
> 
> +  echo "/proc/config.gz not found, trying $conf_filename"
> +
>    if cmd "test -f $conf_filename" ; then
>      get $conf_filename $out_filename
>      return 0
> --
> 2.7.4

Applied.  Thanks.
 -- Tim

Re: [Fuego] [PATCH 03/18] need_check: no need for testing if the file exists

[Daniel Sangorrin]

Hi Tim

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, January 31, 2018 1:34 PM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 03/18] need_check: no need for testing if the file exists
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> > Subject: [Fuego] [PATCH 03/18] need_check: no need for testing if the file
> > exists
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/scripts/need_check.sh | 14 +++++---------
> >  1 file changed, 5 insertions(+), 9 deletions(-)
> >
> > diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
> > index 58f76c3..d85e89a 100755
> > --- a/engine/scripts/need_check.sh
> > +++ b/engine/scripts/need_check.sh
> > @@ -173,22 +173,18 @@ function get_full_kconfig {
> >    gfk_board_tmpfile=$(mktemp)
> >    gfk_host_tmpfile=$(mktemp)
> >
> > -  # try /proc/config.gz first
> I'm not sure why you don't like my comments.  ;-)  But whatever.

Oops sorry about that!! Please insert it back, I think I got confused
with the echo message below.

> 
> > -  if cmd "test -f /proc/config.gz" ; then
> > -    if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
> > -      get $gfk_board_tmpfile $out_filename
> > -      return 0
> > -    fi
> > -  else
> > -    echo "/proc/config.gz not found"
> > +  if cmd "zcat /proc/config.gz >$gfk_board_tmpfile 2>/dev/null" ; then
> > +    get $gfk_board_tmpfile $out_filename
> > +    return 0
> >    fi
> >
> > -  # try /boot directory
> >    cmd "uname -r >$gfk_board_tmpfile"
> >    get $gfk_board_tmpfile $gfk_host_tmpfile
> >    kver=$(cat $gfk_host_tmpfile)
> I took the liberty of simplifying this sequence to
> kver=$(cmd uname -r)
> 
> >    conf_filename="/boot/config-$kver"
> >
> > +  echo "/proc/config.gz not found, trying $conf_filename"
> > +
> >    if cmd "test -f $conf_filename" ; then
> >      get $conf_filename $out_filename
> >      return 0
> > --
> > 2.7.4
> 
> Applied.  Thanks.
>  -- Tim
> 

Re: [Fuego] [PATCH 04/18] need_check: check_kconfig should return 1 on first failure

[Bird, Timothy]

Applied.

Thanks.
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, January 30, 2018 1:59 AM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 04/18] need_check: check_kconfig should return 1
> on first failure
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/need_check.sh | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/engine/scripts/need_check.sh b/engine/scripts/need_check.sh
> index d85e89a..aa96cbe 100755
> --- a/engine/scripts/need_check.sh
> +++ b/engine/scripts/need_check.sh
> @@ -268,7 +268,9 @@ function check_kconfig {
> 
>    for cfg in "${arg_array[@]}" ; do
>      echo "Checking cfg: [$cfg]"
> -    check_one_kconfig $cfg
> +    if ! check_one_kconfig $cfg; then
> +        return 1
> +    fi
>    done
>  }
> 
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 05/18] LTP: we need root permissions

[Bird, Timothy]

I think some LTP tests don't need root.  This is rather a big hammer,
and means that none of LTP will run, if the Fuego test user is
not root.

Do we know which ones will fail?  The LTP style guide
https://github.com/linux-test-project/ltp/wiki/StyleGuide
says for individual LTP tests to use tst_require_root()
if they need it, and to fall back to other behavior if
it's not available.

Please confirm that this global dependency check what you want
for LTP.
 -- Tim



> -----Original Message-----
> From: Daniel Sangorrin
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> b/engine/tests/Functional.LTP/fuego_test.sh
> index 2473cf1..1305421 100755
> --- a/engine/tests/Functional.LTP/fuego_test.sh
> +++ b/engine/tests/Functional.LTP/fuego_test.sh
> @@ -1,6 +1,8 @@
>  # FIXTHIS: use https://github.com/linux-test-project/ltp.git instead of a
> tarball
>  tarball=ltp-7eb77fbfd80.tar.bz2
> 
> +NEED_ROOT=1
> +
>  ALLTESTS="
>  admin_tools         fs_perms_simple       ltp-aiodio.part3  net_stress.appl
> quickhit
>  can                 fs_readonly           ltp-aiodio.part4  net_stress.broken_ip
> rpc_tests
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 06/18] LTP: patch runtests before running

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> 
> The reason is that we want to support use cases where the
> LTP is installed by the user.

OK - but why do you modify the patch, if you're going to replace it
with a 'sed' line?

If we're not using the patch, we should remove it from Fuego.

Not applied yet.
  -- Tim

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.LTP/fuego_test.sh        |  5 +++--
>  engine/tests/Functional.LTP/shorter-fork13.patch | 11 -----------
>  2 files changed, 3 insertions(+), 13 deletions(-)
>  delete mode 100644 engine/tests/Functional.LTP/shorter-fork13.patch
> 
> diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> b/engine/tests/Functional.LTP/fuego_test.sh
> index 1305421..8278e98 100755
> --- a/engine/tests/Functional.LTP/fuego_test.sh
> +++ b/engine/tests/Functional.LTP/fuego_test.sh
> @@ -117,8 +117,6 @@ function test_pre_check {
>  function test_build {
>      if [[ "$FUNCTIONAL_LTP_PHASES" == *build* ]] ; then
>          echo "Building LTP"
> -        patch -p1 -N -s < $TEST_HOME/shorter-fork13.patch
> -
>          # Build the LTP tests
>          make autotools
>          ./configure CC="${CC}" AR="${AR}" RANLIB="${RANLIB}"
> LDFLAGS="$LDFLAGS" SYSROOT="${SDKROOT}" \
> @@ -190,6 +188,9 @@ function test_run {
>      if [[ "$FUNCTIONAL_LTP_PHASES" == *run* ]] ; then
>          echo "Running LTP"
> 
> +        # we patch them here because of the different phases
> +        cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> $LTP_DESTDIR/runtest/syscalls"
> +
>          TESTS=""
>          PTSTESTS=""
>          RTTESTS=""
> diff --git a/engine/tests/Functional.LTP/shorter-fork13.patch
> b/engine/tests/Functional.LTP/shorter-fork13.patch
> deleted file mode 100644
> index 09e8365..0000000
> --- a/engine/tests/Functional.LTP/shorter-fork13.patch
> +++ /dev/null
> @@ -1,11 +0,0 @@
> ---- a/runtest/syscalls	2017-08-25 00:15:46.584034299 +0000
> -+++ b/runtest/syscalls	2017-08-25 00:15:33.880034153 +0000
> -@@ -295,7 +295,7 @@ fork08 fork08
> - fork09 fork09
> - fork10 fork10
> - fork11 fork11
> --fork13 fork13 -i 1000000
> -+fork13 fork13 -i 100000
> - fork14 fork14
> -
> - fpathconf01 fpathconf01
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> b/engine/tests/Functional.LTP/fuego_test.sh
> index 8278e98..ae85bac 100755
> --- a/engine/tests/Functional.LTP/fuego_test.sh
> +++ b/engine/tests/Functional.LTP/fuego_test.sh
> @@ -190,6 +190,8 @@ function test_run {
> 
>          # we patch them here because of the different phases
>          cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> $LTP_DESTDIR/runtest/syscalls"
> +        cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30
> diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
> +        cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/'
> $LTP_DESTDIR/runtest/syscalls"

Can we make this dependent on some spec ARG, like
FUNCTIONAL_LTP_SHORT_RUN or something?
(other name suggestions are welcome.)

I think in general, quite a few tests would benefit from
having a spec that allowed for a shorter version of the test
to run.

> 
>          TESTS=""
>          PTSTESTS=""
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] LTP prechecks

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin
> Hi Tim,
> 
> I send you a batch of patches that do various prechecks and
> autogenerate a skiplist when you run LTP on a target.
> The list is not exhausitive yet but I'd like to have a first
> review and hopefully have other members contribute with
> their experencies or knowledge.
> 
> [PATCH 01/18] need_check: correct errata
> [PATCH 02/18] need_check: cache the kconfig file
> [PATCH 03/18] need_check: no need for testing if the file exists
> [PATCH 04/18] need_check: check_kconfig should return 1 on first

The above were accepted.

> [PATCH 05/18] LTP: we need root permissions
This one I had a question on - see my email.

> [PATCH 06/18] LTP: patch runtests before running
> [PATCH 07/18] LTP: reduce the time required for two more tests
These two I also had questions on.  In principle I like the approach
of getting rid of patches against the LTP source.
But please see my email.

> [PATCH 08/18] LTP: put the skiplist on the log directory
> [PATCH 09/18] LTP: show the generated skiplist file on the jenkins
> [PATCH 10/18] LTP: skip a few problematic tests for now
> [PATCH 11/18] LTP: add tests to skip if tmp is mounted on tmpfs
> [PATCH 12/18] LTP: skip tests that depend on specific kernel commits
> [PATCH 13/18] LTP: skip tests depending on the architecture
> [PATCH 14/18] LTP: skip tests depending on the target kernel config
> [PATCH 15/18] LTP: skip tests that depend on unavailable commands
> [PATCH 16/18] LTP: skip tests depending on the config.h generated by
> [PATCH 17/18] LTP: add note about typical LTP build failure on arm
> [PATCH 18/18] LTP: remove trailing space

I added all these, and will push to my 'master' branch shortly.

I do have a few questions about whether putting some of these skips
into a spec file, that could be customized per board, would be better.

I think I need to play around with it for a while to see if I like this style
of runtime determination of skiplist.

Putting the skiplist in the list of files you can see from the interface is good.
Otherwise users might have a hard time figuring out what a test isn't showing
a status result for a particular run.

Thanks very much.
 -- Tim

Re: [Fuego] [PATCH 05/18] LTP: we need root permissions

[Daniel Sangorrin]

Hi Tim,

I agree with you that there are tests that do not require root permissions.
For example, inotify01 works fine:
$ ./testcases/bin/inotify01
inotify01 1 TPASS : ...

However, when we try to run the test with runltp the scripts..
$ ./runltp -f syscalls -s inotify01
Error: permission denied: opening /dev/kmsg
losetup: /dev/loop0: detach failed: Permission denied
FAIL

Actually, we can run the test without checking /dev/kmsg
by specifying the -Q option and then the test passes.

My point is that at the moment Fuego LTP test requires root.
But in the future we can modify the run script and maybe create
a spec for non-root tests only.

Thanks,
Daniel

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, January 31, 2018 1:54 PM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 05/18] LTP: we need root permissions
> 
> I think some LTP tests don't need root.  This is rather a big hammer,
> and means that none of LTP will run, if the Fuego test user is
> not root.
> 
> Do we know which ones will fail?  The LTP style guide
> https://github.com/linux-test-project/ltp/wiki/StyleGuide
> says for individual LTP tests to use tst_require_root()
> if they need it, and to fall back to other behavior if
> it's not available.
> 
> Please confirm that this global dependency check what you want
> for LTP.
>  -- Tim
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > b/engine/tests/Functional.LTP/fuego_test.sh
> > index 2473cf1..1305421 100755
> > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > @@ -1,6 +1,8 @@
> >  # FIXTHIS: use https://github.com/linux-test-project/ltp.git instead of a
> > tarball
> >  tarball=ltp-7eb77fbfd80.tar.bz2
> >
> > +NEED_ROOT=1
> > +
> >  ALLTESTS="
> >  admin_tools         fs_perms_simple       ltp-aiodio.part3  net_stress.appl
> > quickhit
> >  can                 fs_readonly           ltp-aiodio.part4  net_stress.broken_ip
> > rpc_tests
> > --
> > 2.7.4
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] LTP prechecks

[Tim.Bird]

> -----Original Message-----
> From: Bird, Timothy
> I added all these, and will push to my 'master' branch shortly.

OK  - I have pushed to my 'master' branch.  Can you please test it out?

I can start an LTP test here, but it's late, and I won't see the results
before I go to bed.  I don't like pushing something I haven't tested
yet, but the recent set of patches only affect backfire, IOZone and LTP.

Let me know if you see any problems.

BTW - I still haven't integrated the patch to add kernel headers for the
backfire test.  That's on my to-do list after the Dockerfile refactoring
that profusion is working on.  That work is on my 'next' branch, so it
shouldn't affect anything on master yet.
 -- Tim

Re: [Fuego] [PATCH 06/18] LTP: patch runtests before running

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, January 31, 2018 1:57 PM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 06/18] LTP: patch runtests before running
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > The reason is that we want to support use cases where the
> > LTP is installed by the user.
> 
> OK - but why do you modify the patch, if you're going to replace it
> with a 'sed' line?
> 
> If we're not using the patch, we should remove it from Fuego.

Mmm That is actually what I  did. I removed the patch from Fuego and
applied the change to runtests/syscalls. Did I miss something?

Thanks,
Daniel

> 
> Not applied yet.
>   -- Tim
> 
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.LTP/fuego_test.sh        |  5 +++--
> >  engine/tests/Functional.LTP/shorter-fork13.patch | 11 -----------
> >  2 files changed, 3 insertions(+), 13 deletions(-)
> >  delete mode 100644 engine/tests/Functional.LTP/shorter-fork13.patch
> >
> > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > b/engine/tests/Functional.LTP/fuego_test.sh
> > index 1305421..8278e98 100755
> > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > @@ -117,8 +117,6 @@ function test_pre_check {
> >  function test_build {
> >      if [[ "$FUNCTIONAL_LTP_PHASES" == *build* ]] ; then
> >          echo "Building LTP"
> > -        patch -p1 -N -s < $TEST_HOME/shorter-fork13.patch
> > -
> >          # Build the LTP tests
> >          make autotools
> >          ./configure CC="${CC}" AR="${AR}" RANLIB="${RANLIB}"
> > LDFLAGS="$LDFLAGS" SYSROOT="${SDKROOT}" \
> > @@ -190,6 +188,9 @@ function test_run {
> >      if [[ "$FUNCTIONAL_LTP_PHASES" == *run* ]] ; then
> >          echo "Running LTP"
> >
> > +        # we patch them here because of the different phases
> > +        cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> > $LTP_DESTDIR/runtest/syscalls"
> > +
> >          TESTS=""
> >          PTSTESTS=""
> >          RTTESTS=""
> > diff --git a/engine/tests/Functional.LTP/shorter-fork13.patch
> > b/engine/tests/Functional.LTP/shorter-fork13.patch
> > deleted file mode 100644
> > index 09e8365..0000000
> > --- a/engine/tests/Functional.LTP/shorter-fork13.patch
> > +++ /dev/null
> > @@ -1,11 +0,0 @@
> > ---- a/runtest/syscalls	2017-08-25 00:15:46.584034299 +0000
> > -+++ b/runtest/syscalls	2017-08-25 00:15:33.880034153 +0000
> > -@@ -295,7 +295,7 @@ fork08 fork08
> > - fork09 fork09
> > - fork10 fork10
> > - fork11 fork11
> > --fork13 fork13 -i 1000000
> > -+fork13 fork13 -i 100000
> > - fork14 fork14
> > -
> > - fpathconf01 fpathconf01
> > --
> > 2.7.4
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, January 31, 2018 2:00 PM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > b/engine/tests/Functional.LTP/fuego_test.sh
> > index 8278e98..ae85bac 100755
> > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > @@ -190,6 +190,8 @@ function test_run {
> >
> >          # we patch them here because of the different phases
> >          cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> > $LTP_DESTDIR/runtest/syscalls"
> > +        cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30
> > diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
> > +        cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/'
> > $LTP_DESTDIR/runtest/syscalls"
> 
> Can we make this dependent on some spec ARG, like
> FUNCTIONAL_LTP_SHORT_RUN or something?
> (other name suggestions are welcome.)
> 
> I think in general, quite a few tests would benefit from
> having a spec that allowed for a shorter version of the test
> to run.

Sure, good idea. 

> 
> >
> >          TESTS=""
> >          PTSTESTS=""
> > --
> > 2.7.4
> >
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 05/18] LTP: we need root permissions

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin on Tuesday, January 30, 2018 9:22 PM
>
> I agree with you that there are tests that do not require root permissions.
> For example, inotify01 works fine:
> $ ./testcases/bin/inotify01
> inotify01 1 TPASS : ...
> 
> However, when we try to run the test with runltp the scripts..
> $ ./runltp -f syscalls -s inotify01
> Error: permission denied: opening /dev/kmsg
> losetup: /dev/loop0: detach failed: Permission denied
> FAIL
> 
> Actually, we can run the test without checking /dev/kmsg
> by specifying the -Q option and then the test passes.
> 
> My point is that at the moment Fuego LTP test requires root.
> But in the future we can modify the run script and maybe create
> a spec for non-root tests only.

OK.  I'll apply this patch, but we may want to revisit this.

We may want to put together our own replacement for runltp.
Fuego doesn't currently use a lot of the features of runltp,
and we might be able to avoid some of the requirements
it has for root access.

Thanks,
 -- Tim


> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > Sent: Wednesday, January 31, 2018 1:54 PM
> > To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> > Subject: RE: [Fuego] [PATCH 05/18] LTP: we need root permissions
> >
> > I think some LTP tests don't need root.  This is rather a big hammer,
> > and means that none of LTP will run, if the Fuego test user is
> > not root.
> >
> > Do we know which ones will fail?  The LTP style guide
> > https://github.com/linux-test-project/ltp/wiki/StyleGuide
> > says for individual LTP tests to use tst_require_root()
> > if they need it, and to fall back to other behavior if
> > it's not available.
> >
> > Please confirm that this global dependency check what you want
> > for LTP.
> >  -- Tim
> >
> >
> >
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > >  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > > b/engine/tests/Functional.LTP/fuego_test.sh
> > > index 2473cf1..1305421 100755
> > > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > > @@ -1,6 +1,8 @@
> > >  # FIXTHIS: use https://github.com/linux-test-project/ltp.git instead of a
> > > tarball
> > >  tarball=ltp-7eb77fbfd80.tar.bz2
> > >
> > > +NEED_ROOT=1
> > > +
> > >  ALLTESTS="
> > >  admin_tools         fs_perms_simple       ltp-aiodio.part3  net_stress.appl
> > > quickhit
> > >  can                 fs_readonly           ltp-aiodio.part4  net_stress.broken_ip
> > > rpc_tests
> > > --
> > > 2.7.4
> > >
> > >
> > > _______________________________________________
> > > Fuego mailing list
> > > Fuego@lists.linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 03/18] need_check: no need for testing if the file exists

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > -----Original Message-----
> > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > > -  # try /proc/config.gz first
> > I'm not sure why you don't like my comments.  ;-)  But whatever.
> 
> Oops sorry about that!! Please insert it back, I think I got confused
> with the echo message below.

Well, the code it was describing is now short enough that I don't
think it's needed any more.  So, it's better to remove it anyway.

I was just giving you a hard time. :-)
 -- Tim

Re: [Fuego] [PATCH 06/18] LTP: patch runtests before running

[Bird, Timothy]

> -----Original Message-----
> From:  Daniel Sangorrin
> > -----Original Message-----
> > From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > >
> > > The reason is that we want to support use cases where the
> > > LTP is installed by the user.
> >
> > OK - but why do you modify the patch, if you're going to replace it
> > with a 'sed' line?
> >
> > If we're not using the patch, we should remove it from Fuego.
> 
> Mmm That is actually what I  did. I removed the patch from Fuego and
> applied the change to runtests/syscalls. Did I miss something?

No.  I misread the patch.  What you did was correct.

Applied.

Thanks.
 -- Tim

Re: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > -----Original Message-----
> > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > -----Original Message-----
> > > From: Daniel Sangorrin
> > >
> > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > ---
> > >  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > > b/engine/tests/Functional.LTP/fuego_test.sh
> > > index 8278e98..ae85bac 100755
> > > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > > @@ -190,6 +190,8 @@ function test_run {
> > >
> > >          # we patch them here because of the different phases
> > >          cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> > > $LTP_DESTDIR/runtest/syscalls"
> > > +        cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30
> > > diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
> > > +        cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/'
> > > $LTP_DESTDIR/runtest/syscalls"
> >
> > Can we make this dependent on some spec ARG, like
> > FUNCTIONAL_LTP_SHORT_RUN or something?
> > (other name suggestions are welcome.)
> >
> > I think in general, quite a few tests would benefit from
> > having a spec that allowed for a shorter version of the test
> > to run.
> 
> Sure, good idea.

OK.  In the mean time, I've applied this patch.

One issue with this is what happens if the sed line fails to find the match.
Does this end up causing the test to error out?  Having full lines like this is
a bit fragile.  If the arguments change upstream, this will break.

I haven't tested it, but we might want something like the following:
cmd "sed -i 's/^dio30 diotest6 \(.*\) -n 100 \(.*\)/dio30 diotest6 \1 -n 5 \2/' $LTP_DESTDIR/runtest/dio"

This preserves the other arguments, and only checks to change '-n 100' to '-n 5'.

 -- Tim

Re: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com [mailto:Tim.Bird@sony.com]
> Sent: Thursday, February 01, 2018 4:41 AM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 07/18] LTP: reduce the time required for two more tests
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
> > > -----Original Message-----
> > > From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> > > > -----Original Message-----
> > > > From: Daniel Sangorrin
> > > >
> > > > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > > > ---
> > > >  engine/tests/Functional.LTP/fuego_test.sh | 2 ++
> > > >  1 file changed, 2 insertions(+)
> > > >
> > > > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > > > b/engine/tests/Functional.LTP/fuego_test.sh
> > > > index 8278e98..ae85bac 100755
> > > > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > > > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > > > @@ -190,6 +190,8 @@ function test_run {
> > > >
> > > >          # we patch them here because of the different phases
> > > >          cmd "sed -i 's/^fork13 fork13 -i 1000000/fork13 fork13 -i 100000/'
> > > > $LTP_DESTDIR/runtest/syscalls"
> > > > +        cmd "sed -i 's/^dio30 diotest6 -b 65536 -n 100 -i 100 -o 1024000/dio30
> > > > diotest6 -b 65536 -n 5 -i 100 -o 1024000/' $LTP_DESTDIR/runtest/dio"
> > > > +        cmd "sed -i 's/^msgctl11 msgctl11/msgctl11 msgctl11 -n 5/'
> > > > $LTP_DESTDIR/runtest/syscalls"
> > >
> > > Can we make this dependent on some spec ARG, like
> > > FUNCTIONAL_LTP_SHORT_RUN or something?
> > > (other name suggestions are welcome.)
> > >
> > > I think in general, quite a few tests would benefit from
> > > having a spec that allowed for a shorter version of the test
> > > to run.
> >
> > Sure, good idea.
> 
> OK.  In the mean time, I've applied this patch.
> 
> One issue with this is what happens if the sed line fails to find the match.
> Does this end up causing the test to error out?  Having full lines like this is
> a bit fragile.  If the arguments change upstream, this will break.
> 
> I haven't tested it, but we might want something like the following:
> cmd "sed -i 's/^dio30 diotest6 \(.*\) -n 100 \(.*\)/dio30 diotest6 \1 -n 5 \2/' $LTP_DESTDIR/runtest/dio"
> 
> This preserves the other arguments, and only checks to change '-n 100' to '-n 5'.

You are right. I will fix that.

Thanks,
Daniel

> 
>  -- Tim

Re: [Fuego] LTP prechecks

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, January 31, 2018 2:16 PM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] LTP prechecks
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> > Hi Tim,
> >
> > I send you a batch of patches that do various prechecks and
> > autogenerate a skiplist when you run LTP on a target.
> > The list is not exhausitive yet but I'd like to have a first
> > review and hopefully have other members contribute with
> > their experencies or knowledge.
> >
> > [PATCH 01/18] need_check: correct errata
> > [PATCH 02/18] need_check: cache the kconfig file
> > [PATCH 03/18] need_check: no need for testing if the file exists
> > [PATCH 04/18] need_check: check_kconfig should return 1 on first
> 
> The above were accepted.
> 
> > [PATCH 05/18] LTP: we need root permissions
> This one I had a question on - see my email.
> 
> > [PATCH 06/18] LTP: patch runtests before running
> > [PATCH 07/18] LTP: reduce the time required for two more tests
> These two I also had questions on.  In principle I like the approach
> of getting rid of patches against the LTP source.
> But please see my email.
> 
> > [PATCH 08/18] LTP: put the skiplist on the log directory
> > [PATCH 09/18] LTP: show the generated skiplist file on the jenkins
> > [PATCH 10/18] LTP: skip a few problematic tests for now
> > [PATCH 11/18] LTP: add tests to skip if tmp is mounted on tmpfs
> > [PATCH 12/18] LTP: skip tests that depend on specific kernel commits
> > [PATCH 13/18] LTP: skip tests depending on the architecture
> > [PATCH 14/18] LTP: skip tests depending on the target kernel config
> > [PATCH 15/18] LTP: skip tests that depend on unavailable commands
> > [PATCH 16/18] LTP: skip tests depending on the config.h generated by
> > [PATCH 17/18] LTP: add note about typical LTP build failure on arm
> > [PATCH 18/18] LTP: remove trailing space
> 
> I added all these, and will push to my 'master' branch shortly.
> 
> I do have a few questions about whether putting some of these skips
> into a spec file, that could be customized per board, would be better.
> 
> I think I need to play around with it for a while to see if I like this style
> of runtime determination of skiplist.
> 
> Putting the skiplist in the list of files you can see from the interface is good.
> Otherwise users might have a hard time figuring out what a test isn't showing
> a status result for a particular run.

That's a great idea. I will add a new variable "skipfile" so that instead of having to
specify each test case one by one we can specify a file.
This would be useful to make sure that you are always skipping the same tests.

By the way, it seems that 
[PATCH 08/18] LTP: put the skiplist on the log directory
didn't make it into your repository.

Could you check that?

Thanks a lot!
Daniel

Re: [Fuego] LTP prechecks

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin [mailto:daniel.sangorrin@toshiba.co.jp]
...
> By the way, it seems that
> [PATCH 08/18] LTP: put the skiplist on the log directory
> didn't make it into your repository.
> 
> Could you check that?
You're right. I missed that one!

Applied and pushed.  Sorry about that.
 -- Tim