[Fuego] resending rebuild and kernel_build patches

[Fuego] resending rebuild and kernel_build patches

[Daniel Sangorrin]

Hi Tim,

I have taken into account the comments that you made on some
of my previous patches, and I am resending them again.

[PATCH 01/11] rebuild: add if_source_changed flag
[PATCH 02/11] testsuite version: record the testsuite version
[PATCH 03/11] kernel_build: check build target in pre_check
[PATCH 04/11] kernel_build: use tftpboot by default
[PATCH 05/11] kernel_build: mkdir deploy path if non existent
[PATCH 06/11] kernel_build: set local_dir and default path

Additionally, I send you a few other small patches that I wrote
recently.

[PATCH 07/11] make_pre_run_file: use spaces
[PATCH 08/11] Target_PostCleanup: output into run json
[PATCH 09/11] ltp_process: add support for SIGNALED
[PATCH 10/11] LTP: add small remark on how the rt list was obtained
[PATCH 11/11] LTP: use git instead of tarball by default

All of them are on my git repo.

Thanks,
Daniel

[Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Daniel Sangorrin]

if_source_changed allows rebuilding only when the test source
has changed which is the main use case. For that reason,
we set it as the default option.

FIXTHIS: consider conflicts when updating via git pull

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc                                 | 10 +++---
 engine/scripts/functions.sh                        | 40 +++++++++++++++++++---
 engine/tests/Functional.kernel_build/fuego_test.sh | 12 -------
 3 files changed, 42 insertions(+), 20 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 7de2b70..b675dea 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -96,13 +96,14 @@ where_help = \
 command_help = {
 "add-jobs": ("Adds jobs to Jenkins.",
     """Usage: ftc add-jobs -b <board>[,board2...] [-p <testplan> | -t <testcase> -s <testspec>]
-       [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
+       [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot <true|false>]
        [--precleanup <true|false>] [--postcleanup <true|false>]
   Example: ftc add-jobs -b docker -p testplan_docker
   Example: ftc add-jobs -b docker -t Benchmark.Dhrystone -k 5m --rebuild false
   board,testplan,testpec: use list-board/plans/specs to see the available ones.
   timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
   rebuild: if true rebuild the test source even if it was already built.
+  if_source_changed can be used to rebuild the test only when its source changes.
   reboot: if true reboot the board before the test.
   precleanup: if true clean the board's test folder before the test.
   postcleanup: if true clean the board's test folder after the test.
@@ -241,7 +242,7 @@ You can obtain a list of run_ids for runs on the local system with
 
 "run-test": ("Run a test on a board.",
     """Usage: ftc run-test -b <board> -t <test> [-s <spec>] [-p <phases>]
-    [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
+    [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot <true|false>]
     [--precleanup <true|false>] [--postcleanup <true|false>]
     [--dynamic-vars python_dict]
 Run the indicated test on the specified board.  Use the
@@ -250,6 +251,7 @@ indicated spec, if one is provided.
 board,spec: use list-board/specs to see the available ones.
 timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
 rebuild: if true rebuild the test source even if it was already built.
+if_source_changed can be used to rebuild the test only when its source changes.
 reboot: if true reboot the board before the test.
 precleanup: if true clean the board's test folder before the test.
 postcleanup: if true clean the board's test folder after the test.
@@ -652,7 +654,7 @@ class test_class:
         'timeout' : '30m',
         'spec'    : 'default',
         'reboot'  : 'false',
-        'rebuild' : 'false',
+        'rebuild' : 'if_source_changed',
         'precleanup'  : 'true',
         'postcleanup' : 'true'
     }
@@ -1489,7 +1491,7 @@ def do_add_jobs(conf, options):
             rebuild = options[options.index('--rebuild') + 1]
         except IndexError:
             error_out('Rebuild option not provided after --rebuild')
-        if rebuild not in ['true', 'false']:
+        if rebuild not in ['true', 'false', 'if_source_changed']:
             error_out("Invalid rebuild option '%s'" % rebuild)
         options.remove(rebuild)
         options.remove('--rebuild')
diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 952e9a4..05bdbc1 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -94,6 +94,9 @@ function untar {
         *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
     esac
     tar ${key}xf $TEST_HOME/$tarball --strip-components=1
+
+    # record md5sum for possible source code updates
+    md5sum $TEST_HOME/$tarball > fuego_tarball_src_md5sum
 }
 
 # Unpacks/clones the test source code into the current directory.
@@ -321,6 +324,11 @@ function build_error {
     abort_job "Build failed: $@"
 }
 
+function prepare_build_dir {
+    find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs -0 rm -rf
+    unpack || abort_job "Error while unpacking"
+}
+
 # process Rebuild flag, and unpack test sources if necessary.
 function pre_build {
     cd ${WORKSPACE}
@@ -341,11 +349,35 @@ function pre_build {
 
     lock_build_dir
 
-    if [ "$Rebuild" = "false" ] && [ -e fuego_test_successfully_built ]; then
-        echo "The test is already built"
+    if [ "$Rebuild" = "true" ]; then
+        prepare_build_dir
+    elif [ "$Rebuild" = "false" ]; then
+        if [ -e fuego_test_successfully_built ]; then
+            echo "The test was already successfully built"
+        else
+            prepare_build_dir
+        fi
+    elif [ "$Rebuild" = "if_source_changed" ]; then
+        if [ -e fuego_tarball_src_md5sum ]; then
+            if md5sum -c fuego_tarball_src_md5sum; then
+                echo "No updates to the source code tarball"
+            else
+                echo "The source code tarball got updated, rebuilding"
+                prepare_build_dir
+            fi
+        elif [ -d ".git" ]; then
+            echo "Trying to update the source code with git pull"
+            if git pull | grep "Already up-to-date"; then
+                echo "No source code updates"
+            else
+                echo "The source code was updated, rebuilding"
+                rm -f fuego_test_successfully_built
+            fi
+        else
+            prepare_build_dir
+        fi
     else
-        find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs -0 rm -rf
-        unpack || abort_job "Error while unpacking"
+        abort_job "Rebuild flag $Rebuild is not recognized."
     fi
 }
 
diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index 0b4ac3c..2d36055 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -18,21 +18,9 @@ function test_pre_check {
     if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
         is_on_sdk mkimage
     fi
-
-    # Force the kernel to be rebuilt
-    # That's the whole point of the test - don't let Fuego skip the build.
-    export Rebuild=True
 }
 
 function test_build {
-    # make sure we have the latest source code.
-    cd "$WORKSPACE/$JOB_BUILD_DIR"
-    if [ -d ".git" ]; then
-        git pull || echo "WARNING: git pull failed"
-    else
-        echo "WARNING: no git repository, assuming you used a tarball"
-    fi
-
     # Config
     if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
         FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
-- 
2.7.4

[Fuego] [PATCH 02/11] testsuite version: record the testsuite version

[Daniel Sangorrin]

For tarballs we use an md5sum hash value, and for git
we use the current commit ID.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh     | 11 ++++++++++-
 engine/scripts/parser/common.py |  4 ++--
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 05bdbc1..21a4b00 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -379,6 +379,15 @@ function pre_build {
     else
         abort_job "Rebuild flag $Rebuild is not recognized."
     fi
+
+    # Record the version of the test source
+    if [ -e fuego_tarball_src_md5sum ]; then
+        TESTSUITE_VERSION=$(cat fuego_tarball_src_md5sum | awk '{ print $1 }')
+    elif [ -d ".git" ]; then
+        TESTSUITE_VERSION=$(git log -1 --format="%H")
+    fi
+    export TESTSUITE_VERSION
+    echo "Testsuite version: $TESTSUITE_VERSION"
 }
 
 function build {
@@ -710,7 +719,7 @@ function make_pre_run_file {
         "test_plan": "$TESTPLAN",
         "test_spec": "$TESTSPEC",
         "timestamp": "$BUILD_TIMESTAMP",
-	"testsuite_version": "unknown",
+        "testsuite_version": "$TESTSUITE_VERSION",
 	"toolchain:": "$TOOLCHAIN",
         "workspace": "$WORKSPACE"
     },
diff --git a/engine/scripts/parser/common.py b/engine/scripts/parser/common.py
index c218190..b93a874 100644
--- a/engine/scripts/parser/common.py
+++ b/engine/scripts/parser/common.py
@@ -59,7 +59,7 @@ env_list = ['FUEGO_RW', 'FUEGO_RO', 'FUEGO_CORE', 'NODE_NAME', 'TESTDIR',
         'TESTSPEC', 'BUILD_NUMBER', 'BUILD_ID', 'BUILD_TIMESTAMP',
         'TOOLCHAIN', 'FWVER', 'LOGDIR', 'FUEGO_START_TIME', 'Reboot',
         'Rebuild', 'Target_PreCleanup', 'WORKSPACE', 'JOB_NAME',
-        'FUEGO_VERSION', 'FUEGO_CORE_VERSION'
+        'FUEGO_VERSION', 'FUEGO_CORE_VERSION', 'TESTSUITE_VERSION'
         ]
 
 # add certain environment variables to this module
@@ -456,7 +456,7 @@ def prepare_run_data(results, criteria_data):
         "metadata":{
             "fuego_version":FUEGO_VERSION,
             "fuego_core_version":FUEGO_CORE_VERSION,
-            "testsuite_version":"v1.1-805adb0", #FIXTHIS
+            "testsuite_version":TESTSUITE_VERSION,
             "host_name":FUEGO_HOST,
             "board":NODE_NAME,
             "compiled_on":"docker", #FIXTHIS
-- 
2.7.4

[Fuego] [PATCH 03/11] kernel_build: check build target in pre_check

[Daniel Sangorrin]

test_build could be skipped

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 23 ++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index 2d36055..687a6f4 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -15,6 +15,10 @@ function test_pre_check {
         assert_define PROGRAM_FLEX
     fi
 
+    if [ -z "$FUNCTIONAL_KERNEL_BUILD_TARGET" ]; then
+        FUNCTIONAL_KERNEL_BUILD_TARGET="bzImage"
+    fi
+
     if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
         is_on_sdk mkimage
     fi
@@ -36,10 +40,6 @@ function test_build {
 
     # Building
     echo "Building Kernel"
-    if [ -z "$FUNCTIONAL_KERNEL_BUILD_TARGET" ]; then
-        FUNCTIONAL_KERNEL_BUILD_TARGET="bzImage"
-    fi
-
     if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
         FUNCTIONAL_KERNEL_BUILD_PARAMS="-j$(nproc)"
     fi
@@ -58,6 +58,7 @@ function test_build {
         cd ..
         rm -rf built_modules
     fi
+    BUILD_EXECUTED=1
 }
 
 function test_deploy {
@@ -94,11 +95,17 @@ function test_deploy {
 }
 
 function test_processing {
-    echo "Processing kernel build log"
-    if [ -z "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" ]; then
-        log_compare "$TESTDIR" "1" "[ \t]*Kernel: arch/.* is ready" "p"
+    if [ -z "$BUILD_EXECUTED" ]; then
+        echo "test_build was not executed, so there is not log to parse"
+        RETURN_VALUE=0
+        return $RETURN_VALUE
     else
-        log_compare "$TESTDIR" "1" "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" "p"
+        echo "Processing kernel build log"
+        if [ -z "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" ]; then
+            log_compare "$TESTDIR" "1" "[ \t]*Kernel: arch/.* is ready" "p"
+        else
+            log_compare "$TESTDIR" "1" "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" "p"
+        fi
     fi
 }
 
-- 
2.7.4

[Fuego] [PATCH 04/11] kernel_build: use tftpboot by default

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 4 ++--
 engine/tests/Functional.kernel_build/test.yaml     | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index 687a6f4..fc3aadd 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -74,11 +74,11 @@ function test_deploy {
     fi
 
     if [ "$DEPLOY_METHOD" == "local_dir" ]; then
-        DEPLOY_PATH="${FUNCTIONAL_KERNEL_BUILD_DEPLOY_PATH:-/var/lib/tftp}"
+        DEPLOY_PATH="${FUNCTIONAL_KERNEL_BUILD_DEPLOY_PATH:-/var/lib/tftpboot}"
         echo "Deploying kernel to ${DEPLOY_PATH}"
         cp $WORKSPACE/$JOB_BUILD_DIR/arch/$ARCHITECTURE/boot/$FUNCTIONAL_KERNEL_BUILD_TARGET "${DEPLOY_PATH}"
         if [ -f $WORKSPACE/$JOB_BUILD_DIR/modules.tar.gz ]; then
-            cp $WORKSPACE/$JOB_BUILD_DIR/modules.tar.gz /var/lib/tftp/
+            cp $WORKSPACE/$JOB_BUILD_DIR/modules.tar.gz $DEPLOY_PATH
         fi
     fi
 
diff --git a/engine/tests/Functional.kernel_build/test.yaml b/engine/tests/Functional.kernel_build/test.yaml
index 02c29a6..c076bd7 100644
--- a/engine/tests/Functional.kernel_build/test.yaml
+++ b/engine/tests/Functional.kernel_build/test.yaml
@@ -37,7 +37,7 @@ params:
     - deploy_path:
         description: deploy location
         optional: yes
-        default: for local_dir: /var/lib/tftp, for board_dir: /boot
+        default: for local_dir: /var/lib/tftpboot, for board_dir: /boot
     - regex_p:
         description: regular expression to confirm a build was succesful
         example: "^  OBJCOPY arch/arm64/boot/Image"
-- 
2.7.4

[Fuego] [PATCH 05/11] kernel_build: mkdir deploy path if non existent

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/fuego_test.sh | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh b/engine/tests/Functional.kernel_build/fuego_test.sh
index fc3aadd..de7b834 100755
--- a/engine/tests/Functional.kernel_build/fuego_test.sh
+++ b/engine/tests/Functional.kernel_build/fuego_test.sh
@@ -76,6 +76,10 @@ function test_deploy {
     if [ "$DEPLOY_METHOD" == "local_dir" ]; then
         DEPLOY_PATH="${FUNCTIONAL_KERNEL_BUILD_DEPLOY_PATH:-/var/lib/tftpboot}"
         echo "Deploying kernel to ${DEPLOY_PATH}"
+        if [ ! -d $DEPLOY_PATH ]; then
+            echo "Deploy path does not exist, trying to make it"
+            mkdir -p $DEPLOY_PATH
+        fi
         cp $WORKSPACE/$JOB_BUILD_DIR/arch/$ARCHITECTURE/boot/$FUNCTIONAL_KERNEL_BUILD_TARGET "${DEPLOY_PATH}"
         if [ -f $WORKSPACE/$JOB_BUILD_DIR/modules.tar.gz ]; then
             cp $WORKSPACE/$JOB_BUILD_DIR/modules.tar.gz $DEPLOY_PATH
-- 
2.7.4

[Fuego] [PATCH 06/11] kernel_build: set local_dir and default path

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.kernel_build/spec.json | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/tests/Functional.kernel_build/spec.json b/engine/tests/Functional.kernel_build/spec.json
index 37d2aea..4c9e283 100644
--- a/engine/tests/Functional.kernel_build/spec.json
+++ b/engine/tests/Functional.kernel_build/spec.json
@@ -3,7 +3,9 @@
     "specs": {
         "default": {
             "gitrepo": "https://github.com/torvalds/linux.git",
-            "config": "tinyconfig"
+            "config": "tinyconfig",
+            "deploy_method": "local_dir",
+            "deploy_path": "/fuego-rw/boards/$NODE_NAME/"
         },
         "lts-414y": {
             "gitrepo": "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git",
-- 
2.7.4

[Fuego] [PATCH 07/11] make_pre_run_file: use spaces

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 21a4b00..7a6940a 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -705,10 +705,10 @@ function make_pre_run_file {
     "duration_ms": 0,
     "metadata" : {
         "board": "$NODE_NAME",
-	"build_number": "$BUILD_NUMBER",
-	"compiled_on": "docker",
-	"fuego_core_version": "$FUEGO_CORE_VERSION",
-	"fuego_version": "$FUEGO_VERSION",
+        "build_number": "$BUILD_NUMBER",
+        "compiled_on": "docker",
+        "fuego_core_version": "$FUEGO_CORE_VERSION",
+        "fuego_version": "$FUEGO_VERSION",
         "host_name": "$FUEGO_HOST",
         "keep_log": "true",
         "reboot": "$Reboot",
-- 
2.7.4

[Fuego] [PATCH 08/11] Target_PostCleanup: output into run json

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/parser/common.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/engine/scripts/parser/common.py b/engine/scripts/parser/common.py
index b93a874..9b858f1 100644
--- a/engine/scripts/parser/common.py
+++ b/engine/scripts/parser/common.py
@@ -59,7 +59,8 @@ env_list = ['FUEGO_RW', 'FUEGO_RO', 'FUEGO_CORE', 'NODE_NAME', 'TESTDIR',
         'TESTSPEC', 'BUILD_NUMBER', 'BUILD_ID', 'BUILD_TIMESTAMP',
         'TOOLCHAIN', 'FWVER', 'LOGDIR', 'FUEGO_START_TIME', 'Reboot',
         'Rebuild', 'Target_PreCleanup', 'WORKSPACE', 'JOB_NAME',
-        'FUEGO_VERSION', 'FUEGO_CORE_VERSION', 'TESTSUITE_VERSION'
+        'FUEGO_VERSION', 'FUEGO_CORE_VERSION', 'TESTSUITE_VERSION',
+        'Target_PostCleanup'
         ]
 
 # add certain environment variables to this module
@@ -472,7 +473,7 @@ def prepare_run_data(results, criteria_data):
             "reboot":Reboot,
             "rebuild":Rebuild,
             "target_precleanup":Target_PreCleanup,
-            "target_postcleanup":True, #FIXTHIS
+            "target_postcleanup":Target_PostCleanup,
             "workspace":WORKSPACE,
             "attachments":[
                 {
-- 
2.7.4

[Fuego] [PATCH 09/11] ltp_process: add support for SIGNALED

[Daniel Sangorrin]

Shown while testing with PTS.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/ltp_process.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/tests/Functional.LTP/ltp_process.py b/engine/tests/Functional.LTP/ltp_process.py
index dd0865b..a252665 100644
--- a/engine/tests/Functional.LTP/ltp_process.py
+++ b/engine/tests/Functional.LTP/ltp_process.py
@@ -289,7 +289,7 @@ for test_category in tests:
 # Posix Test Suite processing
 last_was_conformance = False
 set_pts_format = False
-fills = {'UNRESOLVED':brok_fill, 'FAILED':fail_fill, 'PASS':pass_fill, 'UNTESTED':conf_fill, 'UNSUPPORTED':info_fill}
+fills = {'UNRESOLVED':brok_fill, 'FAILED':fail_fill, 'PASS':pass_fill, 'UNTESTED':conf_fill, 'UNSUPPORTED':info_fill, 'SIGNALED':fail_fill}
 
 def pts_set_style(ws):
     for r in range(1, ws.get_highest_row()):
@@ -333,6 +333,8 @@ if os.path.exists('pts.log'):
                     result = "UNRESOLVED"
                 elif result.startswith("UNSUPPORTED"):
                     result = "UNSUPPORTED"
+                elif result.startswith("SIGNALED"):
+                    result = "SIGNALED"
                 last_was_conformance = True
                 ws.append([test_case, result])
                 continue
-- 
2.7.4

[Fuego] [PATCH 10/11] LTP: add small remark on how the rt list was obtained

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index ecbf758..a378573 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -24,6 +24,7 @@ fs_ext4             ltp-aio-stress.part1  net_stress.broken_ip  pty"
 
 ALLPTSTESTS="AIO MEM MSG SEM SIG THR TMR TPS"
 
+# This list can be obtained by doing ./testscripts/test_realtime.sh -t list
 ALLRTTESTS="
 perf/latency
 func/measurement
-- 
2.7.4

[Fuego] [PATCH 11/11] LTP: use git instead of tarball by default

[Daniel Sangorrin]

This change can be discussed, but personally I always
use the git version. Now that if_source_changed is supported
it is even more useful.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.LTP/fuego_test.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/engine/tests/Functional.LTP/fuego_test.sh b/engine/tests/Functional.LTP/fuego_test.sh
index a378573..6c5871e 100755
--- a/engine/tests/Functional.LTP/fuego_test.sh
+++ b/engine/tests/Functional.LTP/fuego_test.sh
@@ -1,5 +1,5 @@
-# gitrepo="https://github.com/linux-test-project/ltp.git"
-tarball=ltp-full-20180118.tar.bz2
+gitrepo="https://github.com/linux-test-project/ltp.git"
+#tarball=ltp-full-20180118.tar.bz2
 
 NEED_ROOT=1
 
-- 
2.7.4

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Daniel Sangorrin]

Note: there was one place where if_source_changed was not updated, it is updated now in my repository.

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org
> <fuego-bounces@lists.linuxfoundation.org> On Behalf Of Daniel Sangorrin
> Sent: Tuesday, September 25, 2018 3:59 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag
> 
> if_source_changed allows rebuilding only when the test source
> has changed which is the main use case. For that reason,
> we set it as the default option.
> 
> FIXTHIS: consider conflicts when updating via git pull
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc                                 | 10 +++---
>  engine/scripts/functions.sh                        | 40
> +++++++++++++++++++---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 12 -------
>  3 files changed, 42 insertions(+), 20 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 7de2b70..b675dea 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -96,13 +96,14 @@ where_help = \
>  command_help = {
>  "add-jobs": ("Adds jobs to Jenkins.",
>      """Usage: ftc add-jobs -b <board>[,board2...] [-p <testplan> | -t <testcase>
> -s <testspec>]
> -       [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> +       [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot
> <true|false>]
>         [--precleanup <true|false>] [--postcleanup <true|false>]
>    Example: ftc add-jobs -b docker -p testplan_docker
>    Example: ftc add-jobs -b docker -t Benchmark.Dhrystone -k 5m --rebuild false
>    board,testplan,testpec: use list-board/plans/specs to see the available ones.
>    timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
>    rebuild: if true rebuild the test source even if it was already built.
> +  if_source_changed can be used to rebuild the test only when its source changes.
>    reboot: if true reboot the board before the test.
>    precleanup: if true clean the board's test folder before the test.
>    postcleanup: if true clean the board's test folder after the test.
> @@ -241,7 +242,7 @@ You can obtain a list of run_ids for runs on the local system
> with
> 
>  "run-test": ("Run a test on a board.",
>      """Usage: ftc run-test -b <board> -t <test> [-s <spec>] [-p <phases>]
> -    [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> +    [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot
> <true|false>]
>      [--precleanup <true|false>] [--postcleanup <true|false>]
>      [--dynamic-vars python_dict]
>  Run the indicated test on the specified board.  Use the
> @@ -250,6 +251,7 @@ indicated spec, if one is provided.
>  board,spec: use list-board/specs to see the available ones.
>  timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
>  rebuild: if true rebuild the test source even if it was already built.
> +if_source_changed can be used to rebuild the test only when its source changes.
>  reboot: if true reboot the board before the test.
>  precleanup: if true clean the board's test folder before the test.
>  postcleanup: if true clean the board's test folder after the test.
> @@ -652,7 +654,7 @@ class test_class:
>          'timeout' : '30m',
>          'spec'    : 'default',
>          'reboot'  : 'false',
> -        'rebuild' : 'false',
> +        'rebuild' : 'if_source_changed',
>          'precleanup'  : 'true',
>          'postcleanup' : 'true'
>      }
> @@ -1489,7 +1491,7 @@ def do_add_jobs(conf, options):
>              rebuild = options[options.index('--rebuild') + 1]
>          except IndexError:
>              error_out('Rebuild option not provided after --rebuild')
> -        if rebuild not in ['true', 'false']:
> +        if rebuild not in ['true', 'false', 'if_source_changed']:
>              error_out("Invalid rebuild option '%s'" % rebuild)
>          options.remove(rebuild)
>          options.remove('--rebuild')
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 952e9a4..05bdbc1 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -94,6 +94,9 @@ function untar {
>          *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
>      esac
>      tar ${key}xf $TEST_HOME/$tarball --strip-components=1
> +
> +    # record md5sum for possible source code updates
> +    md5sum $TEST_HOME/$tarball > fuego_tarball_src_md5sum
>  }
> 
>  # Unpacks/clones the test source code into the current directory.
> @@ -321,6 +324,11 @@ function build_error {
>      abort_job "Build failed: $@"
>  }
> 
> +function prepare_build_dir {
> +    find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs -0
> rm -rf
> +    unpack || abort_job "Error while unpacking"
> +}
> +
>  # process Rebuild flag, and unpack test sources if necessary.
>  function pre_build {
>      cd ${WORKSPACE}
> @@ -341,11 +349,35 @@ function pre_build {
> 
>      lock_build_dir
> 
> -    if [ "$Rebuild" = "false" ] && [ -e fuego_test_successfully_built ]; then
> -        echo "The test is already built"
> +    if [ "$Rebuild" = "true" ]; then
> +        prepare_build_dir
> +    elif [ "$Rebuild" = "false" ]; then
> +        if [ -e fuego_test_successfully_built ]; then
> +            echo "The test was already successfully built"
> +        else
> +            prepare_build_dir
> +        fi
> +    elif [ "$Rebuild" = "if_source_changed" ]; then
> +        if [ -e fuego_tarball_src_md5sum ]; then
> +            if md5sum -c fuego_tarball_src_md5sum; then
> +                echo "No updates to the source code tarball"
> +            else
> +                echo "The source code tarball got updated, rebuilding"
> +                prepare_build_dir
> +            fi
> +        elif [ -d ".git" ]; then
> +            echo "Trying to update the source code with git pull"
> +            if git pull | grep "Already up-to-date"; then
> +                echo "No source code updates"
> +            else
> +                echo "The source code was updated, rebuilding"
> +                rm -f fuego_test_successfully_built
> +            fi
> +        else
> +            prepare_build_dir
> +        fi
>      else
> -        find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs
> -0 rm -rf
> -        unpack || abort_job "Error while unpacking"
> +        abort_job "Rebuild flag $Rebuild is not recognized."
>      fi
>  }
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index 0b4ac3c..2d36055 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -18,21 +18,9 @@ function test_pre_check {
>      if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
>          is_on_sdk mkimage
>      fi
> -
> -    # Force the kernel to be rebuilt
> -    # That's the whole point of the test - don't let Fuego skip the build.
> -    export Rebuild=True
>  }
> 
>  function test_build {
> -    # make sure we have the latest source code.
> -    cd "$WORKSPACE/$JOB_BUILD_DIR"
> -    if [ -d ".git" ]; then
> -        git pull || echo "WARNING: git pull failed"
> -    else
> -        echo "WARNING: no git repository, assuming you used a tarball"
> -    fi
> -
>      # Config
>      if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
>          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Tim.Bird]

Does the ftc run-test code depend on this, or can I take them out of order?
I'd like to apply that patch, but consider this one a bit longer.

I'd like to keep these issues separate, if possible, because while this
series includes things that are very valuable, I think there are issues
with a confusion between the software under test and test software,
for the Functional.kernel_build test.

I think I supported the idea to move the actual build steps from 
Functional.kernel_build's test_run() function to the test_build()
function.  But now I am not sure that was the correct decision.

Logically: the kernel build test should consist of:
build the kernel, get compiler report of errors as the testlog, parse those errors
there should not be a deploy step at all, as it is purely
a host-side operation.

Well - the deploy should consist of storing the build artifacts somewhere that
can be used for other tests that utilize this software.
I guess that's what the current  deploy step does, but it's not
quite as explicit about it as I'd like.

A separate boot test might consist of:
build the kernel, deploy the kernel, boot the DUT, get console logs from DUT as the testlog, parse for any errors
(also, detect if the DUT did not boot at all)

In this case the kernel is the test program (it executes on the board to perform the test, and
produces the log used to check for results).

The part to build the kernel might be: 'ftc test-run -b $BOARD -t Functional.kernel_build'.
I can imagine other tests (eg. a static kernel size checker) that would want a built kernel, but
would do different operations on it, rather than executing it.

The bottom line is this:  I know I recommended it, but I think this solves a
different problem for the kernel_build test than it solves for other tests.
And this difference is making me want to take a step back before I apply this
change.

I want to consider this more, before creating a new rebuild flag.
 -- Tim

> -----Original Message-----
> From: Daniel Sangorrin
> 
> if_source_changed allows rebuilding only when the test source
> has changed which is the main use case. For that reason,
> we set it as the default option.
> 
> FIXTHIS: consider conflicts when updating via git pull
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc                                 | 10 +++---
>  engine/scripts/functions.sh                        | 40 +++++++++++++++++++---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 12 -------
>  3 files changed, 42 insertions(+), 20 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 7de2b70..b675dea 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -96,13 +96,14 @@ where_help = \
>  command_help = {
>  "add-jobs": ("Adds jobs to Jenkins.",
>      """Usage: ftc add-jobs -b <board>[,board2...] [-p <testplan> | -t
> <testcase> -s <testspec>]
> -       [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> +       [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot
> <true|false>]
>         [--precleanup <true|false>] [--postcleanup <true|false>]
>    Example: ftc add-jobs -b docker -p testplan_docker
>    Example: ftc add-jobs -b docker -t Benchmark.Dhrystone -k 5m --rebuild
> false
>    board,testplan,testpec: use list-board/plans/specs to see the available
> ones.
>    timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
>    rebuild: if true rebuild the test source even if it was already built.
> +  if_source_changed can be used to rebuild the test only when its source
> changes.
>    reboot: if true reboot the board before the test.
>    precleanup: if true clean the board's test folder before the test.
>    postcleanup: if true clean the board's test folder after the test.
> @@ -241,7 +242,7 @@ You can obtain a list of run_ids for runs on the local
> system with
> 
>  "run-test": ("Run a test on a board.",
>      """Usage: ftc run-test -b <board> -t <test> [-s <spec>] [-p <phases>]
> -    [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> +    [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot
> <true|false>]
>      [--precleanup <true|false>] [--postcleanup <true|false>]
>      [--dynamic-vars python_dict]
>  Run the indicated test on the specified board.  Use the
> @@ -250,6 +251,7 @@ indicated spec, if one is provided.
>  board,spec: use list-board/specs to see the available ones.
>  timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
>  rebuild: if true rebuild the test source even if it was already built.
> +if_source_changed can be used to rebuild the test only when its source
> changes.
>  reboot: if true reboot the board before the test.
>  precleanup: if true clean the board's test folder before the test.
>  postcleanup: if true clean the board's test folder after the test.
> @@ -652,7 +654,7 @@ class test_class:
>          'timeout' : '30m',
>          'spec'    : 'default',
>          'reboot'  : 'false',
> -        'rebuild' : 'false',
> +        'rebuild' : 'if_source_changed',
>          'precleanup'  : 'true',
>          'postcleanup' : 'true'
>      }
> @@ -1489,7 +1491,7 @@ def do_add_jobs(conf, options):
>              rebuild = options[options.index('--rebuild') + 1]
>          except IndexError:
>              error_out('Rebuild option not provided after --rebuild')
> -        if rebuild not in ['true', 'false']:
> +        if rebuild not in ['true', 'false', 'if_source_changed']:
>              error_out("Invalid rebuild option '%s'" % rebuild)
>          options.remove(rebuild)
>          options.remove('--rebuild')
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 952e9a4..05bdbc1 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -94,6 +94,9 @@ function untar {
>          *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
>      esac
>      tar ${key}xf $TEST_HOME/$tarball --strip-components=1
> +
> +    # record md5sum for possible source code updates
> +    md5sum $TEST_HOME/$tarball > fuego_tarball_src_md5sum
>  }
> 
>  # Unpacks/clones the test source code into the current directory.
> @@ -321,6 +324,11 @@ function build_error {
>      abort_job "Build failed: $@"
>  }
> 
> +function prepare_build_dir {
> +    find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs
> -0 rm -rf
> +    unpack || abort_job "Error while unpacking"
> +}
> +
>  # process Rebuild flag, and unpack test sources if necessary.
>  function pre_build {
>      cd ${WORKSPACE}
> @@ -341,11 +349,35 @@ function pre_build {
> 
>      lock_build_dir
> 
> -    if [ "$Rebuild" = "false" ] && [ -e fuego_test_successfully_built ]; then
> -        echo "The test is already built"
> +    if [ "$Rebuild" = "true" ]; then
> +        prepare_build_dir
> +    elif [ "$Rebuild" = "false" ]; then
> +        if [ -e fuego_test_successfully_built ]; then
> +            echo "The test was already successfully built"
> +        else
> +            prepare_build_dir
> +        fi
> +    elif [ "$Rebuild" = "if_source_changed" ]; then
> +        if [ -e fuego_tarball_src_md5sum ]; then
> +            if md5sum -c fuego_tarball_src_md5sum; then
> +                echo "No updates to the source code tarball"
> +            else
> +                echo "The source code tarball got updated, rebuilding"
> +                prepare_build_dir
> +            fi
> +        elif [ -d ".git" ]; then
> +            echo "Trying to update the source code with git pull"
> +            if git pull | grep "Already up-to-date"; then
> +                echo "No source code updates"
> +            else
> +                echo "The source code was updated, rebuilding"
> +                rm -f fuego_test_successfully_built
> +            fi
> +        else
> +            prepare_build_dir
> +        fi
>      else
> -        find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 |
> xargs -0 rm -rf
> -        unpack || abort_job "Error while unpacking"
> +        abort_job "Rebuild flag $Rebuild is not recognized."
>      fi
>  }
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index 0b4ac3c..2d36055 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -18,21 +18,9 @@ function test_pre_check {
>      if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
>          is_on_sdk mkimage
>      fi
> -
> -    # Force the kernel to be rebuilt
> -    # That's the whole point of the test - don't let Fuego skip the build.
> -    export Rebuild=True
>  }
> 
>  function test_build {
> -    # make sure we have the latest source code.
> -    cd "$WORKSPACE/$JOB_BUILD_DIR"
> -    if [ -d ".git" ]; then
> -        git pull || echo "WARNING: git pull failed"
> -    else
> -        echo "WARNING: no git repository, assuming you used a tarball"
> -    fi
> -
>      # Config
>      if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
>          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=8ywuWmFMqK6gIy82Qwk2GhR4SAydFRy_Hm1rE1T6C0o&s=zVM
> w_UdedEmHCijaZOExVLun0-MxGge7Ey5eo0gkIeY&e=

Re: [Fuego] [PATCH 02/11] testsuite version: record the testsuite version

[Tim.Bird]

Looks good. applied.
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, September 24, 2018 11:59 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 02/11] testsuite version: record the testsuite
> version
> 
> For tarballs we use an md5sum hash value, and for git
> we use the current commit ID.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh     | 11 ++++++++++-
>  engine/scripts/parser/common.py |  4 ++--
>  2 files changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 05bdbc1..21a4b00 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -379,6 +379,15 @@ function pre_build {
>      else
>          abort_job "Rebuild flag $Rebuild is not recognized."
>      fi
> +
> +    # Record the version of the test source
> +    if [ -e fuego_tarball_src_md5sum ]; then
> +        TESTSUITE_VERSION=$(cat fuego_tarball_src_md5sum | awk '{ print $1
> }')
> +    elif [ -d ".git" ]; then
> +        TESTSUITE_VERSION=$(git log -1 --format="%H")
> +    fi
> +    export TESTSUITE_VERSION
> +    echo "Testsuite version: $TESTSUITE_VERSION"
>  }
> 
>  function build {
> @@ -710,7 +719,7 @@ function make_pre_run_file {
>          "test_plan": "$TESTPLAN",
>          "test_spec": "$TESTSPEC",
>          "timestamp": "$BUILD_TIMESTAMP",
> -	"testsuite_version": "unknown",
> +        "testsuite_version": "$TESTSUITE_VERSION",
>  	"toolchain:": "$TOOLCHAIN",
>          "workspace": "$WORKSPACE"
>      },
> diff --git a/engine/scripts/parser/common.py
> b/engine/scripts/parser/common.py
> index c218190..b93a874 100644
> --- a/engine/scripts/parser/common.py
> +++ b/engine/scripts/parser/common.py
> @@ -59,7 +59,7 @@ env_list = ['FUEGO_RW', 'FUEGO_RO', 'FUEGO_CORE',
> 'NODE_NAME', 'TESTDIR',
>          'TESTSPEC', 'BUILD_NUMBER', 'BUILD_ID', 'BUILD_TIMESTAMP',
>          'TOOLCHAIN', 'FWVER', 'LOGDIR', 'FUEGO_START_TIME', 'Reboot',
>          'Rebuild', 'Target_PreCleanup', 'WORKSPACE', 'JOB_NAME',
> -        'FUEGO_VERSION', 'FUEGO_CORE_VERSION'
> +        'FUEGO_VERSION', 'FUEGO_CORE_VERSION', 'TESTSUITE_VERSION'
>          ]
> 
>  # add certain environment variables to this module
> @@ -456,7 +456,7 @@ def prepare_run_data(results, criteria_data):
>          "metadata":{
>              "fuego_version":FUEGO_VERSION,
>              "fuego_core_version":FUEGO_CORE_VERSION,
> -            "testsuite_version":"v1.1-805adb0", #FIXTHIS
> +            "testsuite_version":TESTSUITE_VERSION,
>              "host_name":FUEGO_HOST,
>              "board":NODE_NAME,
>              "compiled_on":"docker", #FIXTHIS
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=xORNxB2czj6nk50V4iI20FjcfmxKIxk-rxhU5Zk288g&s=v-
> 3LXQeIQyxl2CZBZblqEGHFRRPsDTlKKL6sJhEQMFw&e=

Re: [Fuego] [PATCH 03/11] kernel_build: check build target in pre_check

[Tim.Bird]

I applied all the kernel_build patches (3-5).

They look good to me.
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, September 24, 2018 11:59 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 03/11] kernel_build: check build target in pre_check
> 
> test_build could be skipped
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/fuego_test.sh | 23 ++++++++++++++-
> -------
>  1 file changed, 15 insertions(+), 8 deletions(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> b/engine/tests/Functional.kernel_build/fuego_test.sh
> index 2d36055..687a6f4 100755
> --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> @@ -15,6 +15,10 @@ function test_pre_check {
>          assert_define PROGRAM_FLEX
>      fi
> 
> +    if [ -z "$FUNCTIONAL_KERNEL_BUILD_TARGET" ]; then
> +        FUNCTIONAL_KERNEL_BUILD_TARGET="bzImage"
> +    fi
> +
>      if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
>          is_on_sdk mkimage
>      fi
> @@ -36,10 +40,6 @@ function test_build {
> 
>      # Building
>      echo "Building Kernel"
> -    if [ -z "$FUNCTIONAL_KERNEL_BUILD_TARGET" ]; then
> -        FUNCTIONAL_KERNEL_BUILD_TARGET="bzImage"
> -    fi
> -
>      if [ -z "$FUNCTIONAL_KERNEL_BUILD_PARAMS" ]; then
>          FUNCTIONAL_KERNEL_BUILD_PARAMS="-j$(nproc)"
>      fi
> @@ -58,6 +58,7 @@ function test_build {
>          cd ..
>          rm -rf built_modules
>      fi
> +    BUILD_EXECUTED=1
>  }
> 
>  function test_deploy {
> @@ -94,11 +95,17 @@ function test_deploy {
>  }
> 
>  function test_processing {
> -    echo "Processing kernel build log"
> -    if [ -z "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" ]; then
> -        log_compare "$TESTDIR" "1" "[ \t]*Kernel: arch/.* is ready" "p"
> +    if [ -z "$BUILD_EXECUTED" ]; then
> +        echo "test_build was not executed, so there is not log to parse"
> +        RETURN_VALUE=0
> +        return $RETURN_VALUE
>      else
> -        log_compare "$TESTDIR" "1" "$FUNCTIONAL_KERNEL_BUILD_REGEX_P"
> "p"
> +        echo "Processing kernel build log"
> +        if [ -z "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" ]; then
> +            log_compare "$TESTDIR" "1" "[ \t]*Kernel: arch/.* is ready" "p"
> +        else
> +            log_compare "$TESTDIR" "1"
> "$FUNCTIONAL_KERNEL_BUILD_REGEX_P" "p"
> +        fi
>      fi
>  }
> 
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=Y_qa6uywDpUmCGXh43YMkpxwKrfMPiUgl4wdf5YwHkY&s=klMc
> zm1bqiJA7ebZjfUF0bKoEmPxznZL5NZo0kPriVk&e=

Re: [Fuego] [PATCH 06/11] kernel_build: set local_dir and default path

[Tim.Bird]

Also applied.  Thanks.
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, September 24, 2018 11:59 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 06/11] kernel_build: set local_dir and default path
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.kernel_build/spec.json | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/engine/tests/Functional.kernel_build/spec.json
> b/engine/tests/Functional.kernel_build/spec.json
> index 37d2aea..4c9e283 100644
> --- a/engine/tests/Functional.kernel_build/spec.json
> +++ b/engine/tests/Functional.kernel_build/spec.json
> @@ -3,7 +3,9 @@
>      "specs": {
>          "default": {
>              "gitrepo": "https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__github.com_torvalds_linux.git&d=DwICAg&c=fP4tf--
> 1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=R7EDJnDWvN7e0qrf-
> xNqcB8G1U00_qUfzUTHmwa2cG0&s=1MnEeqqNGpaHixc--
> D11Bbby2jfNGZaIToSOc23pWgk&e=",
> -            "config": "tinyconfig"
> +            "config": "tinyconfig",
> +            "deploy_method": "local_dir",
> +            "deploy_path": "/fuego-rw/boards/$NODE_NAME/"
>          },
>          "lts-414y": {
>              "gitrepo": "https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__git.kernel.org_pub_scm_linux_kernel_git_stable_linux-
> 2Dstable.git&d=DwICAg&c=fP4tf--1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=R7EDJnDWvN7e0qrf-
> xNqcB8G1U00_qUfzUTHmwa2cG0&s=ADUZ8g_0ch_Y-
> WyB3wgl9iDPeTpJatCqRgZ0-iDEJ6A&e=",
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=R7EDJnDWvN7e0qrf-
> xNqcB8G1U00_qUfzUTHmwa2cG0&s=8wF6RYxZ1QTOZbZgNLMZzkTl-9yO-
> FJ51ekKTJ-J-4w&e=

Re: [Fuego] [PATCH 07/11] make_pre_run_file: use spaces

[Tim.Bird]

These small fixup patches  (7-10) were also applied.

Thanks.
 -- Tim


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, September 24, 2018 11:59 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 07/11] make_pre_run_file: use spaces
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 21a4b00..7a6940a 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -705,10 +705,10 @@ function make_pre_run_file {
>      "duration_ms": 0,
>      "metadata" : {
>          "board": "$NODE_NAME",
> -	"build_number": "$BUILD_NUMBER",
> -	"compiled_on": "docker",
> -	"fuego_core_version": "$FUEGO_CORE_VERSION",
> -	"fuego_version": "$FUEGO_VERSION",
> +        "build_number": "$BUILD_NUMBER",
> +        "compiled_on": "docker",
> +        "fuego_core_version": "$FUEGO_CORE_VERSION",
> +        "fuego_version": "$FUEGO_VERSION",
>          "host_name": "$FUEGO_HOST",
>          "keep_log": "true",
>          "reboot": "$Reboot",
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=eqL3qHmy8uFi79RTamrE0H89PTYfgZy79k-
> Zq3K72sw&s=kfUQZtKVQDsY1ds-QCGeJLkQFzT_23rEgXeh9RYrSBM&e=

Re: [Fuego] [PATCH 11/11] LTP: use git instead of tarball by default

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> 
> This change can be discussed, but personally I always
> use the git version. Now that if_source_changed is supported
> it is even more useful.

I'm still uncomfortable with the source for a test changing automatically, and
possibly without the tester's knowledge.  I'm
worried that different versions of LTP will produce different results, and that will
cause confusion trying to compare results between sites.

(Note: We don't currently allow people to compare results between sites,
but that's an envisioned feature for the future.   Adding the testsuite_version
to the LTP file helps a lot in this regard.)

Let's discuss this one more, as there may be another mechanism that would
work for both of us (maybe a board variable or a site configuration variable).

This was not applied.
 -- Tim

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.LTP/fuego_test.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> b/engine/tests/Functional.LTP/fuego_test.sh
> index a378573..6c5871e 100755
> --- a/engine/tests/Functional.LTP/fuego_test.sh
> +++ b/engine/tests/Functional.LTP/fuego_test.sh
> @@ -1,5 +1,5 @@
> -# gitrepo="https://github.com/linux-test-project/ltp.git"
> -tarball=ltp-full-20180118.tar.bz2
> +gitrepo=" https://github.com/linux-test-project/ltp.git"
> +#tarball=ltp-full-20180118.tar.bz2
> 
>  NEED_ROOT=1
> 
> --
> 2.7.4
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> -1dS0biCFlB0saz0I0kjO5v7-
> GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> GzCRg&m=qvMHzm_M5MVRsAkMXJIw2hsZsQuFX2jDsnFZkB3zwlY&s=QKaH
> Q42v7ummrqryEOApGCZcfCUZ84SB_GBhGZAFvZA&e=

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Daniel Sangorrin]

Hi Tim,

> Does the ftc run-test code depend on this, or can I take them out of order?
> I'd like to apply that patch, but consider this one a bit longer.

No, it doesn't depend on this.

> I'd like to keep these issues separate, if possible, because while this
> series includes things that are very valuable, I think there are issues
> with a confusion between the software under test and test software,
> for the Functional.kernel_build test.
>
> I think I supported the idea to move the actual build steps from
> Functional.kernel_build's test_run() function to the test_build()
> function.  But now I am not sure that was the correct decision.

I understand what you say, but the if_source_changed flag is useful for any test, not just for kernel_build.
In addition, even if we move the "build test" back to test_run, the source code will still be fetched during the build phase.

The user can choose between cloning the entire kernel repository each time (Rebuild=True) or only pulling the latest changes (if_source_changed). Where we run the build test (make uImage etc) , either on test_build or on test_run, is a separate issue from how we are fetching the source code, which is what this patch is mainly about.

Yest another separate issue is whether we want to build from pristine state (do make mrproper) or reuse the cache of objects as much as possible.

> Logically: the kernel build test should consist of:
> build the kernel, get compiler report of errors as the testlog, parse those errors
> there should not be a deploy step at all, as it is purely
> a host-side operation.
> 
> Well - the deploy should consist of storing the build artifacts somewhere that
> can be used for other tests that utilize this software.

The results (the kernel and modules) are copied to a local directory. The user can choose to copy them to the log directory, or to /var/lib/tftpboot (after installing tftpd on Fuego's docker) for example.

The next step would be to create a Functional.kernel_deploy test to deploy the kernel to the target directly (using put), or indirectly through a server (ftp, html, tftp, LAVA..).

Finally, we could create Functional.reboot to reboot the DUT and check if the boot was succesful. This "check" can be many things: check it booted in less than 10 seconds, check we get the expected prompt, check we can connect to the board, check Functional.hello_world works, etc..

> I guess that's what the current  deploy step does, but it's not
> quite as explicit about it as I'd like.
Could you elaborate a bit on that?

> A separate boot test might consist of:
> build the kernel, deploy the kernel, boot the DUT, get console logs from DUT as the
> testlog, parse for any errors
> (also, detect if the DUT did not boot at all)

That could be done like this:
ftc run-test -b myboard -t Funtional.kernel_build -s myboard-tftpboot
[Opt] ftc run-test -b myboard -t Functional.kernel_deploy
ftc run-test -b myboard -t Functional.reboot -s myboard-console-prompt

The first part would build the kernel and copy it into /var/lib/tftpboot. Optionally, we can leave the deploy step to Functional.kernel_deploy.
Then, Functional.reboot would power cycle the board (or ssh reboot it) and check that it boots succesfully.
 
> In this case the kernel is the test program (it executes on the board to perform the
> test, and
> produces the log used to check for results).
> 
> The part to build the kernel might be: 'ftc test-run -b $BOARD -t
> Functional.kernel_build'.
> I can imagine other tests (eg. a static kernel size checker) that would want a built
> kernel, but
> would do different operations on it, rather than executing it.

That could be easily done:
ftc test-run -b $BOARD -t Functional.kernel_build -s logdir
ftc test-run -b $BOARD -t Functional.kernel_sizecheck -s latestbuildnumber

> The bottom line is this:  I know I recommended it, but I think this solves a
> different problem for the kernel_build test than it solves for other tests.
> And this difference is making me want to take a step back before I apply this
> change.
> 
> I want to consider this more, before creating a new rebuild flag.

I understand what you say, but the rebuild flag is more about what to do with the source code.
Do we want to clone the kernel again or just do a git pull?
The default is if_source_changed, but the user can set --rebuid true
If we keep a hardcoded "export Rebuild=true" then the user cannot decide.

Thanks,
Daniel


>  -- Tim
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > if_source_changed allows rebuilding only when the test source
> > has changed which is the main use case. For that reason,
> > we set it as the default option.
> >
> > FIXTHIS: consider conflicts when updating via git pull
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/scripts/ftc                                 | 10 +++---
> >  engine/scripts/functions.sh                        | 40
> +++++++++++++++++++---
> >  engine/tests/Functional.kernel_build/fuego_test.sh | 12 -------
> >  3 files changed, 42 insertions(+), 20 deletions(-)
> >
> > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > index 7de2b70..b675dea 100755
> > --- a/engine/scripts/ftc
> > +++ b/engine/scripts/ftc
> > @@ -96,13 +96,14 @@ where_help = \
> >  command_help = {
> >  "add-jobs": ("Adds jobs to Jenkins.",
> >      """Usage: ftc add-jobs -b <board>[,board2...] [-p <testplan> | -t
> > <testcase> -s <testspec>]
> > -       [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> > +       [-k <kill timeout>] [--rebuild <true|false|if_source_changed>]
> [--reboot
> > <true|false>]
> >         [--precleanup <true|false>] [--postcleanup <true|false>]
> >    Example: ftc add-jobs -b docker -p testplan_docker
> >    Example: ftc add-jobs -b docker -t Benchmark.Dhrystone -k 5m --rebuild
> > false
> >    board,testplan,testpec: use list-board/plans/specs to see the available
> > ones.
> >    timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
> >    rebuild: if true rebuild the test source even if it was already built.
> > +  if_source_changed can be used to rebuild the test only when its source
> > changes.
> >    reboot: if true reboot the board before the test.
> >    precleanup: if true clean the board's test folder before the test.
> >    postcleanup: if true clean the board's test folder after the test.
> > @@ -241,7 +242,7 @@ You can obtain a list of run_ids for runs on the local
> > system with
> >
> >  "run-test": ("Run a test on a board.",
> >      """Usage: ftc run-test -b <board> -t <test> [-s <spec>] [-p <phases>]
> > -    [-k <kill timeout>] [--rebuild <true|false>] [--reboot <true|false>]
> > +    [-k <kill timeout>] [--rebuild <true|false|if_source_changed>] [--reboot
> > <true|false>]
> >      [--precleanup <true|false>] [--postcleanup <true|false>]
> >      [--dynamic-vars python_dict]
> >  Run the indicated test on the specified board.  Use the
> > @@ -250,6 +251,7 @@ indicated spec, if one is provided.
> >  board,spec: use list-board/specs to see the available ones.
> >  timeout: integer with a suffix from 'smhd' (seconds, minutes, hours, days).
> >  rebuild: if true rebuild the test source even if it was already built.
> > +if_source_changed can be used to rebuild the test only when its source
> > changes.
> >  reboot: if true reboot the board before the test.
> >  precleanup: if true clean the board's test folder before the test.
> >  postcleanup: if true clean the board's test folder after the test.
> > @@ -652,7 +654,7 @@ class test_class:
> >          'timeout' : '30m',
> >          'spec'    : 'default',
> >          'reboot'  : 'false',
> > -        'rebuild' : 'false',
> > +        'rebuild' : 'if_source_changed',
> >          'precleanup'  : 'true',
> >          'postcleanup' : 'true'
> >      }
> > @@ -1489,7 +1491,7 @@ def do_add_jobs(conf, options):
> >              rebuild = options[options.index('--rebuild') + 1]
> >          except IndexError:
> >              error_out('Rebuild option not provided after --rebuild')
> > -        if rebuild not in ['true', 'false']:
> > +        if rebuild not in ['true', 'false', 'if_source_changed']:
> >              error_out("Invalid rebuild option '%s'" % rebuild)
> >          options.remove(rebuild)
> >          options.remove('--rebuild')
> > diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> > index 952e9a4..05bdbc1 100755
> > --- a/engine/scripts/functions.sh
> > +++ b/engine/scripts/functions.sh
> > @@ -94,6 +94,9 @@ function untar {
> >          *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
> >      esac
> >      tar ${key}xf $TEST_HOME/$tarball --strip-components=1
> > +
> > +    # record md5sum for possible source code updates
> > +    md5sum $TEST_HOME/$tarball > fuego_tarball_src_md5sum
> >  }
> >
> >  # Unpacks/clones the test source code into the current directory.
> > @@ -321,6 +324,11 @@ function build_error {
> >      abort_job "Build failed: $@"
> >  }
> >
> > +function prepare_build_dir {
> > +    find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 | xargs
> > -0 rm -rf
> > +    unpack || abort_job "Error while unpacking"
> > +}
> > +
> >  # process Rebuild flag, and unpack test sources if necessary.
> >  function pre_build {
> >      cd ${WORKSPACE}
> > @@ -341,11 +349,35 @@ function pre_build {
> >
> >      lock_build_dir
> >
> > -    if [ "$Rebuild" = "false" ] && [ -e fuego_test_successfully_built ]; then
> > -        echo "The test is already built"
> > +    if [ "$Rebuild" = "true" ]; then
> > +        prepare_build_dir
> > +    elif [ "$Rebuild" = "false" ]; then
> > +        if [ -e fuego_test_successfully_built ]; then
> > +            echo "The test was already successfully built"
> > +        else
> > +            prepare_build_dir
> > +        fi
> > +    elif [ "$Rebuild" = "if_source_changed" ]; then
> > +        if [ -e fuego_tarball_src_md5sum ]; then
> > +            if md5sum -c fuego_tarball_src_md5sum; then
> > +                echo "No updates to the source code tarball"
> > +            else
> > +                echo "The source code tarball got updated, rebuilding"
> > +                prepare_build_dir
> > +            fi
> > +        elif [ -d ".git" ]; then
> > +            echo "Trying to update the source code with git pull"
> > +            if git pull | grep "Already up-to-date"; then
> > +                echo "No source code updates"
> > +            else
> > +                echo "The source code was updated, rebuilding"
> > +                rm -f fuego_test_successfully_built
> > +            fi
> > +        else
> > +            prepare_build_dir
> > +        fi
> >      else
> > -        find . -maxdepth 1 -mindepth 1 ! -name "fuego.build.lock" -print0 |
> > xargs -0 rm -rf
> > -        unpack || abort_job "Error while unpacking"
> > +        abort_job "Rebuild flag $Rebuild is not recognized."
> >      fi
> >  }
> >
> > diff --git a/engine/tests/Functional.kernel_build/fuego_test.sh
> > b/engine/tests/Functional.kernel_build/fuego_test.sh
> > index 0b4ac3c..2d36055 100755
> > --- a/engine/tests/Functional.kernel_build/fuego_test.sh
> > +++ b/engine/tests/Functional.kernel_build/fuego_test.sh
> > @@ -18,21 +18,9 @@ function test_pre_check {
> >      if [[ "$FUNCTIONAL_KERNEL_BUILD_TARGET" == *"uImage"* ]]; then
> >          is_on_sdk mkimage
> >      fi
> > -
> > -    # Force the kernel to be rebuilt
> > -    # That's the whole point of the test - don't let Fuego skip the build.
> > -    export Rebuild=True
> >  }
> >
> >  function test_build {
> > -    # make sure we have the latest source code.
> > -    cd "$WORKSPACE/$JOB_BUILD_DIR"
> > -    if [ -d ".git" ]; then
> > -        git pull || echo "WARNING: git pull failed"
> > -    else
> > -        echo "WARNING: no git repository, assuming you used a tarball"
> > -    fi
> > -
> >      # Config
> >      if [ -z "$FUNCTIONAL_KERNEL_BUILD_CONFIG" ]; then
> >          FUNCTIONAL_KERNEL_BUILD_CONFIG="defconfig"
> > --
> > 2.7.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> > -1dS0biCFlB0saz0I0kjO5v7-
> > GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> > GzCRg&m=8ywuWmFMqK6gIy82Qwk2GhR4SAydFRy_Hm1rE1T6C0o&s=zVM
> > w_UdedEmHCijaZOExVLun0-MxGge7Ey5eo0gkIeY&e=

Re: [Fuego] [PATCH 11/11] LTP: use git instead of tarball by default

[Daniel Sangorrin]

Hi Tim,

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Friday, September 28, 2018 5:16 AM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 11/11] LTP: use git instead of tarball by default
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > This change can be discussed, but personally I always
> > use the git version. Now that if_source_changed is supported
> > it is even more useful.
> 
> I'm still uncomfortable with the source for a test changing automatically, and
> possibly without the tester's knowledge.  I'm
> worried that different versions of LTP will produce different results, and that will
> cause confusion trying to compare results between sites.

The user can just use --rebuild false to avoid any change. Or specify the git tag to use with gitref in the spec or dynamic variable.

> (Note: We don't currently allow people to compare results between sites,
> but that's an envisioned feature for the future.   Adding the testsuite_version
> to the LTP file helps a lot in this regard.)

I think the testsuite_version is enough. We could ask the user whether to compare only results using the same exact testsuite_version, or the riskier option of comparing results from different testsuite_versions.

> Let's discuss this one more, as there may be another mechanism that would
> work for both of us (maybe a board variable or a site configuration variable).
That sounds good.

Thanks,
Daniel

> This was not applied.
>  -- Tim
> 
> >
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> >  engine/tests/Functional.LTP/fuego_test.sh | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/engine/tests/Functional.LTP/fuego_test.sh
> > b/engine/tests/Functional.LTP/fuego_test.sh
> > index a378573..6c5871e 100755
> > --- a/engine/tests/Functional.LTP/fuego_test.sh
> > +++ b/engine/tests/Functional.LTP/fuego_test.sh
> > @@ -1,5 +1,5 @@
> > -# gitrepo="https://github.com/linux-test-project/ltp.git"
> > -tarball=ltp-full-20180118.tar.bz2
> > +gitrepo=" https://github.com/linux-test-project/ltp.git"
> > +#tarball=ltp-full-20180118.tar.bz2
> >
> >  NEED_ROOT=1
> >
> > --
> > 2.7.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://urldefense.proofpoint.com/v2/url?u=https-
> > 3A__lists.linuxfoundation.org_mailman_listinfo_fuego&d=DwICAg&c=fP4tf-
> > -1dS0biCFlB0saz0I0kjO5v7-
> > GLPtvShAo4cc&r=jjTc71ylyJg68rRxrFQuDFMMybIqPCnrHF85A-
> > GzCRg&m=qvMHzm_M5MVRsAkMXJIw2hsZsQuFX2jDsnFZkB3zwlY&s=QKaH
> > Q42v7ummrqryEOApGCZcfCUZ84SB_GBhGZAFvZA&e=

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin 
> > Does the ftc run-test code depend on this, or can I take them out of order?
> > I'd like to apply that patch, but consider this one a bit longer.
> 
> No, it doesn't depend on this.

OK - I applied it, but had some problems with logging, and ended up
reverting the conversion to the python logger module for 'ftc run-test'.
I found some other bugs with handling stdout and calling into the container,
that I also made fixes for.

My main reason for pulling out the python logging module conversion is
that that module is very complicated (ie hard to debug, IMO) and didn't
really match what we were using it for.  The logging data should be a direct
pass-through of what goes into the log, to the ftc stdout.  We shouldn't be
adding any additional fields or adding levels or the like.  Whatever main.sh
produces as stdout, ftc should also.  So I didn't want the extra complexity.

It's been a long debug session, so I'll respond to the other issues about
the if_source_changed change  tomorrow.  But the code with the
'Jenkins calls ftc run-test' is now applied and pushed.  Please give it a whirl.

And thanks very much for the work!  I've been wanting to convert over to
a command-line oriented approach for some time, and this is great stuff.
In particular, the separation of actions based on caller type is really good.

Regards,
 -- Tim

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Daniel Sangorrin]

Thanks a lot Tim!

Toshiba released a whitepaper about Fuego written by me this month where the main diagram shows Jenkins calling ftc run-test.
https://www.toshiba.co.jp/tech/review/2018/05/73_05pdf/r02.pdf

I need to upstream the code that makes Fuego and KernelCI/Squad work together soon. I am also using Ktest together with Fuego. It works very well.
Ktest does the building, rebooting, provisioning, bisecting... and Fuego does the testing. I hope I can explain how it works in some conference soon.

Regards,
Daniel


> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> Sent: Friday, September 28, 2018 3:36 PM
> To: daniel.sangorrin@toshiba.co.jp; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag
> 
> 
> 
> > -----Original Message-----
> > From: Daniel Sangorrin
> > > Does the ftc run-test code depend on this, or can I take them out of order?
> > > I'd like to apply that patch, but consider this one a bit longer.
> >
> > No, it doesn't depend on this.
> 
> OK - I applied it, but had some problems with logging, and ended up
> reverting the conversion to the python logger module for 'ftc run-test'.
> I found some other bugs with handling stdout and calling into the container,
> that I also made fixes for.
> 
> My main reason for pulling out the python logging module conversion is
> that that module is very complicated (ie hard to debug, IMO) and didn't
> really match what we were using it for.  The logging data should be a direct
> pass-through of what goes into the log, to the ftc stdout.  We shouldn't be
> adding any additional fields or adding levels or the like.  Whatever main.sh
> produces as stdout, ftc should also.  So I didn't want the extra complexity.
> 
> It's been a long debug session, so I'll respond to the other issues about
> the if_source_changed change  tomorrow.  But the code with the
> 'Jenkins calls ftc run-test' is now applied and pushed.  Please give it a whirl.
> 
> And thanks very much for the work!  I've been wanting to convert over to
> a command-line oriented approach for some time, and this is great stuff.
> In particular, the separation of actions based on caller type is really good.
> 
> Regards,
>  -- Tim

Re: [Fuego] [PATCH 02/11] testsuite version: record the testsuite version

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin
> 
> Hi Tim,
> 
> When you applied this something went wrong.
> function untar contains -c $TEST_HOME/$targetball
> instead of $TEST_HOME/$tarball

Ugh.  Sorry.  Fixed.
 -- Tim

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Tim.Bird]

> -----Original Message-----
> From: Daniel Sangorrin 
> 
> > It's been a long debug session, so I'll respond to the other issues about
> > the if_source_changed change  tomorrow.  But the code with the
> > 'Jenkins calls ftc run-test' is now applied and pushed.  Please give it a whirl.
> 
> I gave it a whirl and it worked fine. I had some problems with the criteria, I
> will investigate tomorrow.

I found the bug with the criteria handling.  It was due to the following not 
being equivalent for OrderedDicts:
ref_value = reference["value"]
ref_value = reference.get("value", None)

Go figure.  I introduced this when I refactored a patch in the
criteria handling code.  I think it's fixed (and has lots of debug
messages, if you should turn on debugging), but please give
it a try.

Thanks,
 -- Tim

Re: [Fuego] [PATCH 01/11] rebuild: add if_source_changed flag

[Daniel Sangorrin]

> -----Original Message-----
> From: Tim.Bird@sony.com <Tim.Bird@sony.com>
> > -----Original Message-----
> > From: Daniel Sangorrin
> >
> > > It's been a long debug session, so I'll respond to the other issues about
> > > the if_source_changed change  tomorrow.  But the code with the
> > > 'Jenkins calls ftc run-test' is now applied and pushed.  Please give it a whirl.
> >
> > I gave it a whirl and it worked fine. I had some problems with the criteria, I
> > will investigate tomorrow.
> 
> I found the bug with the criteria handling.  It was due to the following not
> being equivalent for OrderedDicts:
> ref_value = reference["value"]
> ref_value = reference.get("value", None)

Oh, good to know.

> Go figure.  I introduced this when I refactored a patch in the
> criteria handling code.  I think it's fixed (and has lots of debug
> messages, if you should turn on debugging), but please give
> it a try.

Thanks a lot. It seems to work now!

Daniel