[Fuego] [PATCH 1/4] run.json: add fwver and test_spec information

[Fuego] [PATCH 1/4] run.json: add fwver and test_spec information

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index c7e98bf..070c185 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -459,6 +459,7 @@ function make_run_file {
     "description": "unknown",
     "device": "$NODE_NAME",
     "duration": $DURATION,
+    "fwver": "$FWVER",
     "files": [
         "devlog.txt",
         "syslog.before.txt",
@@ -477,6 +478,7 @@ function make_run_file {
     "target_precleanup": "$Target_PreCleanup",
     "test_name": "$TESTDIR",
     "testplan": "$TESTPLAN",
+    "test_spec": "$TESTSPEC",
     "timestamp": "$BUILD_TIMESTAMP",
     "workspace": "$WORKSPACE"
 }
-- 
2.7.4

[Fuego] [PATCH 2/4] FUEGO_RESULT: should only be modified from the main script

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 4 +---
 engine/scripts/main.sh      | 1 -
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 070c185..cd77b77 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -572,8 +572,7 @@ function log_compare {
         current_count=`cat ${LOGDIR}/testlog.txt | grep -E "${3}" 2>&1 | wc -l`
         if [ "$4" = "p" ]; then
             if [ $current_count -ge $2 ] ; then
-                echo "log_compare: pattern $3 found $current_count times (expected greater or equal than $2)"
-                FUEGO_RESULT=0
+                echo "log_compare: pattern '$3' found $current_count times (expected greater or equal than $2)"
             else
                 echo "ERROR: log_compare: pattern $3 found $current_count times (expected greater or equal than $2)"
                 RETURN_VALUE=1
@@ -583,7 +582,6 @@ function log_compare {
         if [ "$4" = "n" ]; then
             if [ $current_count -le $2 ] ; then
                 echo "log_compare: pattern $3 found $current_count times (expected less or equal than $2)"
-                FUEGO_RESULT=0
             else
                 echo "ERROR: log_compare: pattern $3 found $current_count times (expected less or equal than $2)"
                 RETURN_VALUE=1
diff --git a/engine/scripts/main.sh b/engine/scripts/main.sh
index 06e4e5d..2a4589f 100644
--- a/engine/scripts/main.sh
+++ b/engine/scripts/main.sh
@@ -48,7 +48,6 @@ echo "##### doing fuego phase: run ########"
 call_if_present test_run
 
 echo "##### doing fuego phase: processing ########"
-FUEGO_RESULT=0
 processing
 export FUEGO_RESULT=$?
 
-- 
2.7.4

[Fuego] [PATCH 3/4] log_compare: add single quotation marks for clarity

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index cd77b77..99996c7 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -574,16 +574,16 @@ function log_compare {
             if [ $current_count -ge $2 ] ; then
                 echo "log_compare: pattern '$3' found $current_count times (expected greater or equal than $2)"
             else
-                echo "ERROR: log_compare: pattern $3 found $current_count times (expected greater or equal than $2)"
+                echo "ERROR: log_compare: pattern '$3' found $current_count times (expected greater or equal than $2)"
                 RETURN_VALUE=1
             fi
         fi
 
         if [ "$4" = "n" ]; then
             if [ $current_count -le $2 ] ; then
-                echo "log_compare: pattern $3 found $current_count times (expected less or equal than $2)"
+                echo "log_compare: pattern '$3' found $current_count times (expected less or equal than $2)"
             else
-                echo "ERROR: log_compare: pattern $3 found $current_count times (expected less or equal than $2)"
+                echo "ERROR: log_compare: pattern '$3' found $current_count times (expected less or equal than $2)"
                 RETURN_VALUE=1
             fi
         fi
-- 
2.7.4

[Fuego] [PATCH 4/4] phases: rework phases execution order after the test is run

[Daniel Sangorrin]

I have reworked the test phases to reduce the time that a
board needs to be connected (e.g.: required by boards shared
in a farm like AGL):

- fetch results: gets the testlogs and results from the board
- cleanup: mostly equivalent to the previous post_test
- teardown link: from here on the board shouldn't be access
- processing: process the result data fetched before

TODO: pre_test should be also reworked.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 39 ++++++++++++++++-----------------------
 engine/scripts/main.sh      | 15 +++++++++++----
 2 files changed, 27 insertions(+), 27 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 99996c7..6c1f63d 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -27,7 +27,7 @@
 function signal_handler {
   # if we got here, something went wrong.  Let's clean up and leave
   echo "in signal_handler"
-  post_test $TESTDIR
+  cleanup
   exit 130
 }
 
@@ -303,13 +303,9 @@ function pre_test {
     ov_rootfs_drop_caches
 }
 
-function processing {
-    # PWD: /fuego-rw/buildzone/board.spec.testtype.testcase-platform
-    local fuego_test_dir=${BOARD_TESTDIR}/fuego.$TESTDIR
+function fetch_results {
     local fuego_test_tmp=${FUEGO_TARGET_TMP:-/tmp}/fuego.$TESTDIR
-    local RETURN_VALUE=0
 
-    # fetch data for processing
     firmware
     get $BOARD_TESTDIR/fuego.$TESTDIR/$TESTDIR.log ${LOGDIR}/testlog.txt
 
@@ -321,6 +317,10 @@ function processing {
         touch ${LOGDIR}/syslog.before.txt
     fi
     get ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.after ${LOGDIR}/syslog.after.txt
+}
+
+function processing {
+    local RETURN_VALUE=0
 
     # process the fetched data
     call_if_present test_processing
@@ -372,6 +372,9 @@ function processing {
         ln -s "/var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log" $LOGDIR/consolelog.txt
     fi
 
+    # create run.json file
+    make_run_file $RETURN_VALUE
+
     return $RETURN_VALUE
 }
 
@@ -439,6 +442,7 @@ function call_if_present {
 
 # create the run.json file
 # this routine uses a whole lot of variables...
+# $1: result of the test
 function make_run_file {
     # calculate duration
     end_arr=( $(date +"%s %N") )
@@ -446,7 +450,7 @@ function make_run_file {
     FUEGO_END_TIME="${end_arr[0]}${end_arr[1]:0:3}"
     DURATION=$(( $FUEGO_END_TIME - $FUEGO_START_TIME ))
 
-    if [ "$FUEGO_RESULT" = "0" ] ; then
+    if [ "$1" = "0" ] ; then
         FUEGO_RESULT_STR="SUCCESS"
     else
         FUEGO_RESULT_STR="FAILURE"
@@ -485,15 +489,14 @@ function make_run_file {
 RUN_JSON_TEMPLATE
 }
 
-# capture SIGTERM during post_test
+# capture SIGTERM during cleanup
 # if Jenkins is trying to abort us, it sends SIGTERM multiple times
-# but we'd rather finish up post_test than stop
+# but we'd rather finish up cleanup than stop
 function post_term_handler {
-  echo "Received SIGTERM during post_test - ignoring it"
+  echo "Received SIGTERM during cleanup - ignoring it"
 }
 
-# $1 - $TESTDIR
-function post_test {
+function cleanup {
   # reset the signal handler to avoid an infinite loop
   trap post_term_handler SIGTERM
   trap - SIGHUP SIGALRM SIGINT ERR EXIT
@@ -512,17 +515,7 @@ function post_test {
   call_if_present test_cleanup
 
   # Remove work and log dirs
-  [ "$Target_PostCleanup" = "true" ] && target_cleanup $1 || true
-
-  # log test completion message.
-  echo "Test $1 is finished"
-
-  # Teardown communication link to target if board specifies
-  # a function to do so
-  [ -n "$TARGET_TEARDOWN_LINK" ] && $TARGET_TEARDOWN_LINK
-
-  # create run.json file
-  make_run_file
+  [ "$Target_PostCleanup" = "true" ] && target_cleanup $TESTDIR || true
 }
 
 function target_cleanup {
diff --git a/engine/scripts/main.sh b/engine/scripts/main.sh
index 2a4589f..5f72004 100644
--- a/engine/scripts/main.sh
+++ b/engine/scripts/main.sh
@@ -34,6 +34,7 @@ source $FUEGO_CORE/engine/scripts/functions.sh
 source $FUEGO_CORE/engine/tests/$TESTDIR/$TESTNAME.sh
 
 echo "##### doing fuego phase: pre_test ########"
+# FIXTHIS: try to move link connection to the deploy phase
 pre_test
 
 echo "##### doing fuego phase: build ########"
@@ -47,12 +48,18 @@ deploy
 echo "##### doing fuego phase: run ########"
 call_if_present test_run
 
+echo "##### doing fuego phase: fetch_results ########"
+fetch_results
+
+echo "##### doing fuego phase: cleanup ########"
+cleanup
+
+echo "##### doing fuego phase: teardown board link ########"
+[ -n "$TARGET_TEARDOWN_LINK" ] && $TARGET_TEARDOWN_LINK
+
 echo "##### doing fuego phase: processing ########"
 processing
-export FUEGO_RESULT=$?
-
-echo "##### doing fuego phase: post_test ########"
-post_test $TESTDIR
+FUEGO_RESULT=$?
 
 echo "Fuego: all test phases complete!"
 exit $FUEGO_RESULT
-- 
2.7.4

Re: [Fuego] [PATCH 4/4] phases: rework phases execution order after the test is run

[Bird, Timothy]

I like the concept of this one, but I'm leery of a few details.  Please see questions
and comment inline below.

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, April 24, 2017 11:15 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 4/4] phases: rework phases execution order after
> the test is run
> 
> I have reworked the test phases to reduce the time that a
> board needs to be connected (e.g.: required by boards shared
> in a farm like AGL):
> 
> - fetch results: gets the testlogs and results from the board
> - cleanup: mostly equivalent to the previous post_test
> - teardown link: from here on the board shouldn't be access
> - processing: process the result data fetched before
> 
> TODO: pre_test should be also reworked.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh | 39 ++++++++++++++++-----------------------
>  engine/scripts/main.sh      | 15 +++++++++++----
>  2 files changed, 27 insertions(+), 27 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 99996c7..6c1f63d 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -27,7 +27,7 @@
>  function signal_handler {
>    # if we got here, something went wrong.  Let's clean up and leave
>    echo "in signal_handler"
> -  post_test $TESTDIR
> +  cleanup

This means that if an error occurs, only the cleanup will occur, and none
of the other post-processing (fetching logs, processing, etc.)  In particular,
it means that no logs will be gathered and no run.json file created.
So, NAK on this change.

>    exit 130
>  }
> 
> @@ -303,13 +303,9 @@ function pre_test {
>      ov_rootfs_drop_caches
>  }
> 
> -function processing {
> -    # PWD: /fuego-rw/buildzone/board.spec.testtype.testcase-platform
> -    local fuego_test_dir=${BOARD_TESTDIR}/fuego.$TESTDIR
> +function fetch_results {
>      local fuego_test_tmp=${FUEGO_TARGET_TMP:-/tmp}/fuego.$TESTDIR
> -    local RETURN_VALUE=0
> 
> -    # fetch data for processing
>      firmware
>      get $BOARD_TESTDIR/fuego.$TESTDIR/$TESTDIR.log ${LOGDIR}/testlog.txt
> 
> @@ -321,6 +317,10 @@ function processing {
>          touch ${LOGDIR}/syslog.before.txt
>      fi
>      get
> ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.after
> ${LOGDIR}/syslog.after.txt
> +}
> +
> +function processing {
> +    local RETURN_VALUE=0
> 
>      # process the fetched data
>      call_if_present test_processing
> @@ -372,6 +372,9 @@ function processing {
>          ln -s
> "/var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log"
> $LOGDIR/consolelog.txt
>      fi
> 
> +    # create run.json file
> +    make_run_file $RETURN_VALUE
> +
>      return $RETURN_VALUE
>  }
> 
> @@ -439,6 +442,7 @@ function call_if_present {
> 
>  # create the run.json file
>  # this routine uses a whole lot of variables...
> +# $1: result of the test
>  function make_run_file {
>      # calculate duration
>      end_arr=( $(date +"%s %N") )
> @@ -446,7 +450,7 @@ function make_run_file {
>      FUEGO_END_TIME="${end_arr[0]}${end_arr[1]:0:3}"
>      DURATION=$(( $FUEGO_END_TIME - $FUEGO_START_TIME ))
> 
> -    if [ "$FUEGO_RESULT" = "0" ] ; then
> +    if [ "$1" = "0" ] ; then
>          FUEGO_RESULT_STR="SUCCESS"
>      else
>          FUEGO_RESULT_STR="FAILURE"
> @@ -485,15 +489,14 @@ function make_run_file {
>  RUN_JSON_TEMPLATE
>  }
> 
> -# capture SIGTERM during post_test
> +# capture SIGTERM during cleanup
>  # if Jenkins is trying to abort us, it sends SIGTERM multiple times
> -# but we'd rather finish up post_test than stop
> +# but we'd rather finish up cleanup than stop
>  function post_term_handler {
> -  echo "Received SIGTERM during post_test - ignoring it"
> +  echo "Received SIGTERM during cleanup - ignoring it"
>  }
> 
> -# $1 - $TESTDIR
> -function post_test {
> +function cleanup {
>    # reset the signal handler to avoid an infinite loop
>    trap post_term_handler SIGTERM
>    trap - SIGHUP SIGALRM SIGINT ERR EXIT
> @@ -512,17 +515,7 @@ function post_test {
>    call_if_present test_cleanup
> 
>    # Remove work and log dirs
> -  [ "$Target_PostCleanup" = "true" ] && target_cleanup $1 || true
> -
> -  # log test completion message.
> -  echo "Test $1 is finished"
> -
> -  # Teardown communication link to target if board specifies
> -  # a function to do so
> -  [ -n "$TARGET_TEARDOWN_LINK" ] && $TARGET_TEARDOWN_LINK
> -
> -  # create run.json file
> -  make_run_file
> +  [ "$Target_PostCleanup" = "true" ] && target_cleanup $TESTDIR || true
>  }
> 
>  function target_cleanup {
> diff --git a/engine/scripts/main.sh b/engine/scripts/main.sh
> index 2a4589f..5f72004 100644
> --- a/engine/scripts/main.sh
> +++ b/engine/scripts/main.sh
> @@ -34,6 +34,7 @@ source $FUEGO_CORE/engine/scripts/functions.sh
>  source $FUEGO_CORE/engine/tests/$TESTDIR/$TESTNAME.sh
> 
>  echo "##### doing fuego phase: pre_test ########"
> +# FIXTHIS: try to move link connection to the deploy phase
>  pre_test
> 
>  echo "##### doing fuego phase: build ########"
> @@ -47,12 +48,18 @@ deploy
>  echo "##### doing fuego phase: run ########"
>  call_if_present test_run
> 
> +echo "##### doing fuego phase: fetch_results ########"
Every time a phase name changes, I cry a little inside.  It means I have to
re-label diagrams in documentation I've created.  This is not a reason
not to change the phases, but I don't want too many.

Thee phases (and phase messages) are mainly
there to help with 1) debugging the core and the test scripts and
2) to help Fuego newbies mental model of the test architecture.

> +fetch_results
> +
> +echo "##### doing fuego phase: cleanup ########"
> +cleanup
> +
> +echo "##### doing fuego phase: teardown board link ########"
> +[ -n "$TARGET_TEARDOWN_LINK" ] && $TARGET_TEARDOWN_LINK
> +
I think this would be better as the last step of cleanup, instead of being
its own whole phase.  This should get replaced soon (in the 1.2 release)
with ov_transport_disconnect.

>  echo "##### doing fuego phase: processing ########"
>  processing
> -export FUEGO_RESULT=$?
> -
> -echo "##### doing fuego phase: post_test ########"
> -post_test $TESTDIR
> +FUEGO_RESULT=$?
> 
>  echo "Fuego: all test phases complete!"
>  exit $FUEGO_RESULT
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego