[Fuego] fuego-core patches (migration to testspecs and more)

[Fuego] fuego-core patches (migration to testspecs and more)

[Daniel Sangorrin]

Hi,

This bag of patches includes bug fixes and changes in functionality.

[PATCH 01/16] indentation: fix indentation for dhrystone parser
[PATCH 02/16] abort: fix the abort function
  -> without this fix failed jobs were looping until the timeout.
     Seems like a Jenkins' bug.

[PATCH 03/16] shell e flag: remove any e flag from fuego
  -> As Tim mentioned during the AGL web meeting the e flag is
     not the way to catch errors so I removed it.

[PATCH 04/16] parser: remove error message when no matches
[PATCH 05/16] benchmark: use the same pattern as in functional
[PATCH 06/16] board file: remove conf from the path to the board file
  -> fuego-ro/boards/ instead of fuego-ro/conf/boards

[PATCH 07/16] batch: remove batch_testplan reference
  -> I will implement an add-batch subcommand in the future and
     fix reports.sh. For now remove this dead code.

[PATCH 08/16] testplans: use testspecs instead of testplans
  -> This is the MOST IMPORTANT patch in this series. It removes
     the need to have a testplan or have your test in testplan_default.
     Actually testplans are only needed when adding jobs, or
     when creating a batch job. The rest can be done with just
     the test spec. This also allowed me to simplify ovgen.py
     a little bit more.

[PATCH 09/16] distrib: DISTRIB must be defined in the board file
  -> I didnt update all of the boards, because I want to completely
     rewrite the board files.

[PATCH 10/16] user_checks: add to commands so they execute under
   -> thanks for the tip

[PATCH 11/16] ftc:rm-jobs: remove limitation of 4 words
[PATCH 12/16] description setter: put log if the test fails
  -> Tim, you were right about this.

[PATCH 13/16] ftc: remove testplans and use testspecs
  -> Maybe i should have rebased this and squash it with 08/16 sorry

[PATCH 14/16] ftc test: remove distrib and update paths
[PATCH 15/16] fail_check_cases should not abort the job
  -> this might need more review

[PATCH 16/16] a bit of cleaning and style fixes

Thanks,
Daniel

[Fuego] [PATCH 01/16] indentation: fix indentation for dhrystone parser

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.Dhrystone/parser.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/engine/tests/Benchmark.Dhrystone/parser.py b/engine/tests/Benchmark.Dhrystone/parser.py
index 58d62ce..d98d2d5 100755
--- a/engine/tests/Benchmark.Dhrystone/parser.py
+++ b/engine/tests/Benchmark.Dhrystone/parser.py
@@ -16,9 +16,9 @@ cur_raw_values = cur_file.readlines()
 cur_file.close()
 
 for cur_item in cur_raw_values:
-	cur_match = re.match(cur_search_str, cur_item)
-	if cur_match:
-		cur_dict["Dhrystone"] = cur_match.group(3)
+    cur_match = re.match(cur_search_str, cur_item)
+    if cur_match:
+        cur_dict["Dhrystone"] = cur_match.group(3)
 
 sys.exit(plib.process_data(ref_section_pat, cur_dict, 's', 'FPS'))
 
-- 
2.7.4

[Fuego] [PATCH 02/16] abort: fix the abort function

[Daniel Sangorrin]

it seems that BUILD_URL is not being exported, maybe
a bug in Jenkins.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/common.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/engine/scripts/common.sh b/engine/scripts/common.sh
index 8e104cc..3d51fa4 100644
--- a/engine/scripts/common.sh
+++ b/engine/scripts/common.sh
@@ -40,8 +40,11 @@ function abort_job {
   echo -e "\n*** ABORTED ***\n"
   [ -n "$1" ] && echo -e "Fuego error reason: $1\n"
 
-  wget -qO- ${BUILD_URL}/stop > /dev/null
-  while true; do sleep 5; done
+  # TODO: why BUILD_URL is not availables?
+  wget -qO- http://localhost:8080/fuego/job/$JOB_NAME/$BUILD_NUMBER/stop > /dev/null
+  sleep 5
+  echo -e "Jenkins didn't stop the job so let's die by ourselves"
+  false
 }
 
 # check is variable is set and fail if otherwise
-- 
2.7.4

Re: [Fuego] [PATCH 02/16] abort: fix the abort function

[Bird, Timothy]

> -----Original Message-----
> From Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> it seems that BUILD_URL is not being exported, maybe
> a bug in Jenkins.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/common.sh | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/engine/scripts/common.sh b/engine/scripts/common.sh
> index 8e104cc..3d51fa4 100644
> --- a/engine/scripts/common.sh
> +++ b/engine/scripts/common.sh
> @@ -40,8 +40,11 @@ function abort_job {
>    echo -e "\n*** ABORTED ***\n"
>    [ -n "$1" ] && echo -e "Fuego error reason: $1\n"
> 
> -  wget -qO- ${BUILD_URL}/stop > /dev/null
> -  while true; do sleep 5; done
> +  # TODO: why BUILD_URL is not availables?
> +  wget -qO-
> http://localhost:8080/fuego/job/$JOB_NAME/$BUILD_NUMBER/stop >
> /dev/null
> +  sleep 5

Is this enough time?  I'd be more comfortable with 30 seconds.
I guess we know at this point that we're not doing any more processing.

I may revisit the timing here when I debug my trap handling for
the script system.

> +  echo -e "Jenkins didn't stop the job so let's die by ourselves"
> +  false
>  }
> 
>  # check is variable is set and fail if otherwise
> --
> 2.7.4

Acked-by: Tim Bird
 -- Tim

Re: [Fuego] [PATCH 02/16] abort: fix the abort function

[Bird, Timothy]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 02/16] abort: fix the abort function
> 
> it seems that BUILD_URL is not being exported, maybe
> a bug in Jenkins.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/common.sh | 7 +++++--
>  1 file changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/engine/scripts/common.sh b/engine/scripts/common.sh
> index 8e104cc..3d51fa4 100644
> --- a/engine/scripts/common.sh
> +++ b/engine/scripts/common.sh
> @@ -40,8 +40,11 @@ function abort_job {
>    echo -e "\n*** ABORTED ***\n"
>    [ -n "$1" ] && echo -e "Fuego error reason: $1\n"
> 
> -  wget -qO- ${BUILD_URL}/stop > /dev/null
> -  while true; do sleep 5; done
> +  # TODO: why BUILD_URL is not availables?

One more thing: Can we use FIXTHIS instead of TODO for comments
about things to work on in the future.  That's what I've
used in other places and it would be good to be consistent.

I will likely use some code in the fuego wiki to automatically scan
for these types of comments and present them in the wiki.
 -- Tim

[Fuego] [PATCH 03/16] shell e flag: remove any e flag from fuego

[Daniel Sangorrin]

This wasn't working well, and we should not need the e flag.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/benchmark.sh                                    | 1 -
 engine/scripts/functional.sh                                   | 3 ---
 engine/scripts/functions.sh                                    | 7 -------
 engine/tests/Functional.LTP/LTP.sh                             | 2 --
 engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh | 3 ---
 5 files changed, 16 deletions(-)

diff --git a/engine/scripts/benchmark.sh b/engine/scripts/benchmark.sh
index 1bb67ad..3f9e72c 100644
--- a/engine/scripts/benchmark.sh
+++ b/engine/scripts/benchmark.sh
@@ -24,7 +24,6 @@
 if [ -n "$FUEGO_DEBUG" ] ; then
 	set -x
 fi
-set -e
 
 source $FUEGO_CORE/engine/scripts/overlays.sh
 set_overlay_vars
diff --git a/engine/scripts/functional.sh b/engine/scripts/functional.sh
index 6a56235..f1e0207 100644
--- a/engine/scripts/functional.sh
+++ b/engine/scripts/functional.sh
@@ -24,7 +24,6 @@
 if [ -n "$FUEGO_DEBUG" ] ; then
 	set -x
 fi
-set -e
 
 source $FUEGO_CORE/engine/scripts/overlays.sh
 set_overlay_vars
@@ -51,10 +50,8 @@ get_testlog $TESTDIR
 
 echo "##### doing fuego phase: processing ########"
 FUEGO_RESULT=0
-set +e
 test_processing
 export FUEGO_RESULT=$?
-set -e
 
 echo "##### doing fuego phase: post_test ########"
 post_test $TESTDIR
diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index e9b0f84..b7f0f72 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -422,7 +422,6 @@ function post_test {
   dump_syslogs ${fuego_test_tmp} "after"
 
 # Get syslogs
-  set +e
   get ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.before ${LOGDIR}/syslog.before.txt
   if [ $? -ne 0 ] ; then
      echo "Fuego error: Can't read 'before' system log, possibly because /tmp was cleared on boot"
@@ -438,7 +437,6 @@ function post_test {
   if [ ! -e $LOGDIR/consolelog.txt ] ; then
      ln -s "/var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log" $LOGDIR/consolelog.txt
   fi
-  set -e
 
 # Remove work and log dirs
   [ "$Target_PostCleanup" = "true" ] && target_cleanup $1 || true
@@ -453,14 +451,12 @@ function post_test {
 
 # create functional result file
   # don't freak out if the parsing doesn't happen
-  set +e
   PYTHON_ARGS="-W ignore::DeprecationWarning -W ignore::UserWarning"
   if startswith $TESTDIR "Functional." ; then
       if [ -e "$TEST_HOME/parser.py" ] ; then
          run_python $PYTHON_ARGS "$TEST_HOME/parser.py" -t $TESTDIR -b $NODE_NAME -j $JOB_NAME -n $BUILD_NUMBER -s $BUILD_TIMESTAMP -r $FUEGO_RESULT
       fi
   fi
-  set -e
 }
 
 function target_cleanup {
@@ -482,8 +478,6 @@ function target_reboot {
   # for target come back up
   sleep 10
 
-  # set +e because cmd will fail during the boot process
-  set +e
   retries=0
   while [ $retries -le $1 ]
   do
@@ -492,7 +486,6 @@ function target_reboot {
     sleep 1
     retries=$((retries+1));
   done
-  set -e
   abort_job "FAIL: reboot retries exhausted\n"
 }
 
diff --git a/engine/tests/Functional.LTP/LTP.sh b/engine/tests/Functional.LTP/LTP.sh
index 379626c..1895ed8 100755
--- a/engine/tests/Functional.LTP/LTP.sh
+++ b/engine/tests/Functional.LTP/LTP.sh
@@ -111,9 +111,7 @@ function test_run {
     done
 
     # Let some of the tests fail, the information will be in the result xlsx file
-    set +e
     report "cd $BOARD_TESTDIR/fuego.$TESTDIR; TESTS=\"$TESTS\"; PTSTESTS=\"$PTSTESTS\"; RTTESTS=\"$RTTESTS\"; . ./ltp_target_run.sh"
-    set -e
 }
 
 function test_processing {
diff --git a/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh b/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
index 8d9dd5a..7c33340 100755
--- a/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
+++ b/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
@@ -37,8 +37,6 @@ function test_processing {
 function test_cleanup {
     PHASE_STR="${PHASE_STR}:test_cleanup"
 
-    set +e
-
     # last chance to put something into the log
     # but have to do it directly
     LOGFILE=${LOGDIR}/testlog.txt
@@ -71,7 +69,6 @@ function test_cleanup {
     rm $TMPFILE2
 
     echo "PHASE_STR=$PHASE_STR" >>$LOGFILE
-    set -e
 }
 
 . $FUEGO_CORE/engine/scripts/functional.sh
-- 
2.7.4

Re: [Fuego] [PATCH 03/16] shell e flag: remove any e flag from fuego

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> This wasn't working well, and we should not need the e flag.

Hmmm.  I'm worried that removing the 'set -e' will lead to error cascades
instead of clean exits in some cases.  For example, if something in test_build fails,
do we really want to continue?  We haven't added any error handling to compensate
for removing this auto-die functionality.

Right now, the test scripts are written with the assumption that any single
failure should stop the test.  I disagree with that assumption, but I feel
like if we change the basic assumption we should be at least looking at the
results from the test_xxx functions (written by the test author), to see
if we should continue or not.

> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/benchmark.sh                                    | 1 -
>  engine/scripts/functional.sh                                   | 3 ---
>  engine/scripts/functions.sh                                    | 7 -------
>  engine/tests/Functional.LTP/LTP.sh                             | 2 --
>  engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh | 3 ---
>  5 files changed, 16 deletions(-)
> 
> diff --git a/engine/scripts/benchmark.sh b/engine/scripts/benchmark.sh
> index 1bb67ad..3f9e72c 100644
> --- a/engine/scripts/benchmark.sh
> +++ b/engine/scripts/benchmark.sh
> @@ -24,7 +24,6 @@
>  if [ -n "$FUEGO_DEBUG" ] ; then
>  	set -x
>  fi
> -set -e
> 
>  source $FUEGO_CORE/engine/scripts/overlays.sh
>  set_overlay_vars
> diff --git a/engine/scripts/functional.sh b/engine/scripts/functional.sh
> index 6a56235..f1e0207 100644
> --- a/engine/scripts/functional.sh
> +++ b/engine/scripts/functional.sh
> @@ -24,7 +24,6 @@
>  if [ -n "$FUEGO_DEBUG" ] ; then
>  	set -x
>  fi
> -set -e
> 
>  source $FUEGO_CORE/engine/scripts/overlays.sh
>  set_overlay_vars
> @@ -51,10 +50,8 @@ get_testlog $TESTDIR
> 
>  echo "##### doing fuego phase: processing ########"
>  FUEGO_RESULT=0
> -set +e
>  test_processing
>  export FUEGO_RESULT=$?
> -set -e

This breaks the logic in test_processing to do short-circuit
execution of multiple log_compare processes.  I don't
know if that's a terrible thing or not, but it's a significant
change in program logic.
 
>  echo "##### doing fuego phase: post_test ########"
>  post_test $TESTDIR
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index e9b0f84..b7f0f72 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -422,7 +422,6 @@ function post_test {
>    dump_syslogs ${fuego_test_tmp} "after"
> 
>  # Get syslogs
> -  set +e
>    get
> ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.befor
> e ${LOGDIR}/syslog.before.txt
>    if [ $? -ne 0 ] ; then
>       echo "Fuego error: Can't read 'before' system log, possibly because /tmp
> was cleared on boot"
> @@ -438,7 +437,6 @@ function post_test {
>    if [ ! -e $LOGDIR/consolelog.txt ] ; then
>       ln -s "/var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log"
> $LOGDIR/consolelog.txt
>    fi
> -  set -e
> 
>  # Remove work and log dirs
>    [ "$Target_PostCleanup" = "true" ] && target_cleanup $1 || true
> @@ -453,14 +451,12 @@ function post_test {
> 
>  # create functional result file
>    # don't freak out if the parsing doesn't happen
> -  set +e
>    PYTHON_ARGS="-W ignore::DeprecationWarning -W ignore::UserWarning"
>    if startswith $TESTDIR "Functional." ; then
>        if [ -e "$TEST_HOME/parser.py" ] ; then
>           run_python $PYTHON_ARGS "$TEST_HOME/parser.py" -t $TESTDIR -b
> $NODE_NAME -j $JOB_NAME -n $BUILD_NUMBER -s $BUILD_TIMESTAMP -r
> $FUEGO_RESULT
>        fi
>    fi
> -  set -e
>  }
> 
>  function target_cleanup {
> @@ -482,8 +478,6 @@ function target_reboot {
>    # for target come back up
>    sleep 10
> 
> -  # set +e because cmd will fail during the boot process
> -  set +e
>    retries=0
>    while [ $retries -le $1 ]
>    do
> @@ -492,7 +486,6 @@ function target_reboot {
>      sleep 1
>      retries=$((retries+1));
>    done
> -  set -e
>    abort_job "FAIL: reboot retries exhausted\n"
>  }
> 
> diff --git a/engine/tests/Functional.LTP/LTP.sh
> b/engine/tests/Functional.LTP/LTP.sh
> index 379626c..1895ed8 100755
> --- a/engine/tests/Functional.LTP/LTP.sh
> +++ b/engine/tests/Functional.LTP/LTP.sh
> @@ -111,9 +111,7 @@ function test_run {
>      done
> 
>      # Let some of the tests fail, the information will be in the result xlsx file
> -    set +e
>      report "cd $BOARD_TESTDIR/fuego.$TESTDIR; TESTS=\"$TESTS\";
> PTSTESTS=\"$PTSTESTS\"; RTTESTS=\"$RTTESTS\"; . ./ltp_target_run.sh"
> -    set -e
>  }
> 
>  function test_processing {
> diff --git
> a/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
> b/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
> index 8d9dd5a..7c33340 100755
> --- a/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
> +++ b/engine/tests/Functional.fuego_test_phases/fuego_test_phases.sh
> @@ -37,8 +37,6 @@ function test_processing {
>  function test_cleanup {
>      PHASE_STR="${PHASE_STR}:test_cleanup"
> 
> -    set +e
> -
>      # last chance to put something into the log
>      # but have to do it directly
>      LOGFILE=${LOGDIR}/testlog.txt
> @@ -71,7 +69,6 @@ function test_cleanup {
>      rm $TMPFILE2
> 
>      echo "PHASE_STR=$PHASE_STR" >>$LOGFILE
> -    set -e
>  }
> 
>  . $FUEGO_CORE/engine/scripts/functional.sh
> --
> 2.7.4

I think I'm going to hold off applying this one, until we have more discussion
and possible examination of the effects of it with existing tests.
 -- Tim

[Fuego] [PATCH 04/16] parser: remove error message when no matches

[Daniel Sangorrin]

This is not supposed to be a parser error, it's rather
an error in the test which is already managed later

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/parser/common.py | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/engine/scripts/parser/common.py b/engine/scripts/parser/common.py
index d0bdde5..fd74827 100644
--- a/engine/scripts/parser/common.py
+++ b/engine/scripts/parser/common.py
@@ -98,9 +98,8 @@ def write_report_results(rep_data):
 
 
 def process_data(ref_section_pat, cur_dict, m, label):
-	if not cur_dict:
-		print "\nFuego error reason: could not parse test results in %s\n" % CUR_LOG
-		sys.exit(1)
+        if not cur_dict:
+                sys.exit(1)
 
         if custom_write_report == False:
                 write_report_results(cur_dict)
-- 
2.7.4

Re: [Fuego] [PATCH 04/16] parser: remove error message when no matches

[Bird, Timothy]

> -----Original Message-----
> From Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> This is not supposed to be a parser error, it's rather
> an error in the test which is already managed later

I don't see anywhere else in the parser where failure to find the
benchmark metrics is reported to the user.  This test is a bit crude.
A better one would be to check for all metrics that are supposed
to be there.  (That test would have to be test-specific, and the
way some parsers work is they don't even have the strings for
the metrics themselves - it comes from the regex applied to the
test log.  So maybe this idea is not feasible.)

Is there somewhere else in the system (besides the parser) that catches
this type of error and reports it to the user?  (I guess missing benchmark
data might do it.)
 -- Tim

 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/parser/common.py | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/engine/scripts/parser/common.py
> b/engine/scripts/parser/common.py
> index d0bdde5..fd74827 100644
> --- a/engine/scripts/parser/common.py
> +++ b/engine/scripts/parser/common.py
> @@ -98,9 +98,8 @@ def write_report_results(rep_data):
> 
> 
>  def process_data(ref_section_pat, cur_dict, m, label):
> -	if not cur_dict:
> -		print "\nFuego error reason: could not parse test results in
> %s\n" % CUR_LOG
> -		sys.exit(1)
> +        if not cur_dict:
> +                sys.exit(1)
> 
>          if custom_write_report == False:
>                  write_report_results(cur_dict)
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

[Fuego] [PATCH 05/16] benchmark: use the same pattern as in functional

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/benchmark.sh | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/scripts/benchmark.sh b/engine/scripts/benchmark.sh
index 3f9e72c..d666587 100644
--- a/engine/scripts/benchmark.sh
+++ b/engine/scripts/benchmark.sh
@@ -48,12 +48,14 @@ test_run
 
 echo "##### doing fuego phases: get_testlog AND processing ########"
 set_testres_file
+
+FUEGO_RESULT=0
 bench_processing
 export FUEGO_RESULT=$?
-
 check_create_logrun
 
 echo "##### doing fuego phase: post_test ########"
 post_test $TESTDIR
+echo "Fuego: all test phases complete!"
 return $FUEGO_RESULT
 
-- 
2.7.4

Re: [Fuego] [PATCH 05/16] benchmark: use the same pattern as in functional

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/benchmark.sh | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/engine/scripts/benchmark.sh b/engine/scripts/benchmark.sh
> index 3f9e72c..d666587 100644
> --- a/engine/scripts/benchmark.sh
> +++ b/engine/scripts/benchmark.sh
> @@ -48,12 +48,14 @@ test_run
> 
>  echo "##### doing fuego phases: get_testlog AND processing ########"
>  set_testres_file
> +
> +FUEGO_RESULT=0
>  bench_processing
>  export FUEGO_RESULT=$?

Good catch.  One reason I was a bit sloppy here was that I eventually planned
to merge functional.sh and benchmark.sh.  They are identical until the last
few steps.  But until that merge, these should have the same logic and output.

Thanks.
 Acked-by: Tim Bird
  -- Tim

[Fuego] [PATCH 06/16] board file: remove conf from the path to the board file

[Daniel Sangorrin]

Note: I also simplified the code a bit. Might need review

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc         |  9 +++++----
 engine/scripts/overlays.sh | 16 ++++------------
 2 files changed, 9 insertions(+), 16 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index c1aa151..be76427 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -75,7 +75,7 @@ use_statusouput = 1
 server = jenkins.Jenkins('http://localhost:8080/fuego')
 
 # keep configuration file in /fuego-ro/conf area
-config_dir = "/fuego-ro/conf"
+config_dir = "/fuego-ro"
 CONFIG_FILE = "ftc.conf"
 
 # This is the name of the environment variable
@@ -562,12 +562,13 @@ class run_class:
 # return a map of {'<target_name>': target_instance }
 def get_fuego_targets(conf):
     # scan board directory, and find the board names
-    bfile_list = os.listdir(conf.FUEGO_RO + '/conf/boards')
+    # TODO: use glob
+    bfile_list = os.listdir(conf.FUEGO_RO + '/boards')
     tmap = {}
 
     for filename in bfile_list:
         if filename.endswith(".board"):
-            board_path = conf.FUEGO_RO + '/conf/boards/' + filename
+            board_path = conf.FUEGO_RO + '/boards/' + filename
             name = filename[:-6]
             target = target_class(name, board_path, None, None)
             tmap[name] = target
@@ -1789,7 +1790,7 @@ def do_run_test(conf, test_name, target, options):
 
     # FIXTHIS - (!) in run_test, HARDCODE DISTRIB for now (Yuk)
     target.env_vars["DISTRIB"]="nosyslogd.dist"
-    target.env_vars["BOARD_OVERLAY"]="%s.board" % build_data.target_name
+
     for var_name in target.env_vars.keys():
         os.environ[var_name] = target.env_vars[var_name]
 
diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
index a833215..d717e0f 100644
--- a/engine/scripts/overlays.sh
+++ b/engine/scripts/overlays.sh
@@ -38,18 +38,10 @@ OF_TESTPLAN_ARGS=""
 OF_DEBUG_ARGS=""
 
 function set_overlay_vars() {
-    BOARD_OVERLAY="boards/$NODE_NAME.board"
-
-    echo "board overlay: $BOARD_OVERLAY"
-
-    if [ "$BOARD_OVERLAY" ] ; then
-        echo "using $BOARD_OVERLAY board overlay"
-        OF_BOARD_FILE="$FUEGO_RO/conf/$BOARD_OVERLAY"
-        if [ ! -f $OF_BOARD_FILE ] ; then
-            abort_job "$OF_BOARD_FILE does not exist"
-        fi
-    else
-        abort_job "BOARD_OVERLAY is not defined"
+    # By convention board configuration filenames should be the same as node names
+    OF_BOARD_FILE="$FUEGO_RO/boards/$NODE_NAME.board"
+    if [ ! -f $OF_BOARD_FILE ] ; then
+        abort_job "$OF_BOARD_FILE does not exist"
     fi
 
     # check for $DISTRIB and make file path to it
-- 
2.7.4

Re: [Fuego] [PATCH 06/16] board file: remove conf from the path to the board file

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
> Note: I also simplified the code a bit. Might need review
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc         |  9 +++++----
>  engine/scripts/overlays.sh | 16 ++++------------
>  2 files changed, 9 insertions(+), 16 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index c1aa151..be76427 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -75,7 +75,7 @@ use_statusouput = 1
>  server = jenkins.Jenkins('http://localhost:8080/fuego')
> 
>  # keep configuration file in /fuego-ro/conf area
> -config_dir = "/fuego-ro/conf"
> +config_dir = "/fuego-ro"
>  CONFIG_FILE = "ftc.conf"
This is OK.  I think we'll probably have only one configuration
file, so a whole directory is overkill.  The configuration file
might migrate to ~/.ftc.conf outside the container.
(Or not - I'm still thinking about this.)

It might end up in /fuego-rw, if I implement something
like 'ftc config' (similar to 'git config') to manage configuration
from the command line.  However, disallowing stuff in the
container from overwriting the system config has advantages.
I'm still thinking about this as well.

> 
>  # This is the name of the environment variable
> @@ -562,12 +562,13 @@ class run_class:
>  # return a map of {'<target_name>': target_instance }
>  def get_fuego_targets(conf):
>      # scan board directory, and find the board names
> -    bfile_list = os.listdir(conf.FUEGO_RO + '/conf/boards')
> +    # TODO: use glob
> +    bfile_list = os.listdir(conf.FUEGO_RO + '/boards')
>      tmap = {}
> 
>      for filename in bfile_list:
>          if filename.endswith(".board"):
> -            board_path = conf.FUEGO_RO + '/conf/boards/' + filename
> +            board_path = conf.FUEGO_RO + '/boards/' + filename
>              name = filename[:-6]
>              target = target_class(name, board_path, None, None)
>              tmap[name] = target
> @@ -1789,7 +1790,7 @@ def do_run_test(conf, test_name, target, options):
> 
>      # FIXTHIS - (!) in run_test, HARDCODE DISTRIB for now (Yuk)
>      target.env_vars["DISTRIB"]="nosyslogd.dist"
> -    target.env_vars["BOARD_OVERLAY"]="%s.board" %
> build_data.target_name
> +

This is a nice simplification.  Thanks.

>      for var_name in target.env_vars.keys():
>          os.environ[var_name] = target.env_vars[var_name]
> 
> diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
> index a833215..d717e0f 100644
> --- a/engine/scripts/overlays.sh
> +++ b/engine/scripts/overlays.sh
> @@ -38,18 +38,10 @@ OF_TESTPLAN_ARGS=""
>  OF_DEBUG_ARGS=""
> 
>  function set_overlay_vars() {
> -    BOARD_OVERLAY="boards/$NODE_NAME.board"
> -
> -    echo "board overlay: $BOARD_OVERLAY"
> -
> -    if [ "$BOARD_OVERLAY" ] ; then
> -        echo "using $BOARD_OVERLAY board overlay"
> -        OF_BOARD_FILE="$FUEGO_RO/conf/$BOARD_OVERLAY"
> -        if [ ! -f $OF_BOARD_FILE ] ; then
> -            abort_job "$OF_BOARD_FILE does not exist"
> -        fi
> -    else
> -        abort_job "BOARD_OVERLAY is not defined"
> +    # By convention board configuration filenames should be the same as
> node names
I agree with this.  I think using tools to manage the nodes (ftc add-node)
makes this almost foolproof (a user can't mess up the node name), and
it simplifies things nicely.  We have a few different rules like this,
that I will start documenting on the wiki.  (See http://bird.org/fuego/Fuego_naming_rules)
I may even write a test to validate that a test conforms to the rules.

> +    OF_BOARD_FILE="$FUEGO_RO/boards/$NODE_NAME.board"
> +    if [ ! -f $OF_BOARD_FILE ] ; then
> +        abort_job "$OF_BOARD_FILE does not exist"
>      fi
> 
>      # check for $DISTRIB and make file path to it
> --
> 2.7.4

Looks good. 

Acked-by: Tim Bird
 -- Tim

[Fuego] [PATCH 07/16] batch: remove batch_testplan reference

[Daniel Sangorrin]

This is not currently supported. A similar functionality
will need to be provided for reports.sh but probably with
a different interface.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/overlays.sh | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
index d717e0f..e1d2233 100644
--- a/engine/scripts/overlays.sh
+++ b/engine/scripts/overlays.sh
@@ -55,17 +55,7 @@ function set_overlay_vars() {
         abort_job "DISTRIB is not defined"
     fi
 
-    # prefer batch testplan over test-specific testplan
-    # TODO: fix batch testplans and reports
-    if [ "$BATCH_TESTPLAN" ] ; then
-        echo "using $BATCH_TESTPLAN batch testplan"
-        OF_TESTPLAN="$OF_ROOT/$BATCH_TESTPLAN"
-        if [ ! -f $OF_TESTPLAN ] ; then
-            abort_job "$OF_TESTPLAN does not exist"
-        fi
-        OF_TESTPLAN_ARGS="--testplan $OF_TESTPLAN"
-
-    elif [ "$TESTPLAN" ] ; then
+    if [ "$TESTPLAN" ] ; then
         echo "BATCH_TESTPLAN is not set, using $TESTPLAN.json testplan"
         OF_TESTPLAN="$FUEGO_CORE/engine/overlays/testplans/$TESTPLAN.json"
 
-- 
2.7.4

Re: [Fuego] [PATCH 07/16] batch: remove batch_testplan reference

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> This is not currently supported. A similar functionality
> will need to be provided for reports.sh but probably with
> a different interface.
Agreed.

These used to be set by the "Run SELECTED test on SELECTED targets" job.
If a user has Jenkins with dynamic variable support, this is actually a 
fairly nice mechanism.

What happens if you set TESTPLAN in a matrix job in Jenkins?
(Say you had something like: docker.default.batch, but inside
you set the environment variable TESTPLAN=quick (and assuming
the tests in the batch job has 'quick' specs)

Is this the direction you're heading?

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/overlays.sh | 12 +-----------
>  1 file changed, 1 insertion(+), 11 deletions(-)
> 
> diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
> index d717e0f..e1d2233 100644
> --- a/engine/scripts/overlays.sh
> +++ b/engine/scripts/overlays.sh
> @@ -55,17 +55,7 @@ function set_overlay_vars() {
>          abort_job "DISTRIB is not defined"
>      fi
> 
> -    # prefer batch testplan over test-specific testplan
> -    # TODO: fix batch testplans and reports
> -    if [ "$BATCH_TESTPLAN" ] ; then
> -        echo "using $BATCH_TESTPLAN batch testplan"
> -        OF_TESTPLAN="$OF_ROOT/$BATCH_TESTPLAN"
> -        if [ ! -f $OF_TESTPLAN ] ; then
> -            abort_job "$OF_TESTPLAN does not exist"
> -        fi
> -        OF_TESTPLAN_ARGS="--testplan $OF_TESTPLAN"
> -
> -    elif [ "$TESTPLAN" ] ; then
> +    if [ "$TESTPLAN" ] ; then
>          echo "BATCH_TESTPLAN is not set, using $TESTPLAN.json testplan"
> 
> OF_TESTPLAN="$FUEGO_CORE/engine/overlays/testplans/$TESTPLAN.json"
> 
> --
> 2.7.4
Looks good.

Acked-by: Tim Bird
 -- Tim

[Fuego] [PATCH 08/16] testplans: use testspecs instead of testplans internally

[Daniel Sangorrin]

This is quite a big change but necessary. Testplans are useful
to batch tests and define which spec or timeout we want for
each test. However, we also want to be able to run tests
individually even if they are not inside a testplan.

Until now one prolog.sh was generated for a testplan. This is not
a good idea for several reasons: collisions between different
test plans or the inability to have the same test with two
different specs in the same testplan.

This patch puts the "prolog.sh" generated each time a job is run
on the corresponding job's log folder. It also modifies the names
of the jobs which are now <board>.<testspec>.<testdir>

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh |   7 +-
 engine/scripts/overlays.sh  |  36 +++------
 engine/scripts/ovgen.py     | 191 +++++++++++---------------------------------
 3 files changed, 61 insertions(+), 173 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index b7f0f72..2483f61 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -261,9 +261,7 @@ function pre_test {
 
   cmd "true" || abort_job "Cannot connect to $DEVICE via $TRANSPORT"
 
-# Create the log directory for this test run
   export LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAMP}.${BUILD_NUMBER}"
-  mkdir -p $LOGDIR
 
   # make a link to the log dir that Jenkins can get to
   # (Jenkins might not have the BUILD_TIMESTAMP)
@@ -400,10 +398,11 @@ function post_test {
   # reset the signal handler to avoid an infinite loop
   trap - SIGTERM SIGHUP SIGALRM SIGINT
 
+  export LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAMP}.${BUILD_NUMBER}"
+
   # source generated prolog.sh file since post_test is called separately
-  source $FUEGO_RW/work/${NODE_NAME}_prolog.sh
+  source $LOGDIR/prolog.sh
   export SSHPASS=$PASSWORD
-  export LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAMP}.${BUILD_NUMBER}"
 
   # re-source params to set correct DEVICE, LOGIN, SSH vars
   source $FUEGO_CORE/engine/scripts/params.sh
diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
index e1d2233..cfecc73 100644
--- a/engine/scripts/overlays.sh
+++ b/engine/scripts/overlays.sh
@@ -25,15 +25,13 @@
 . $FUEGO_CORE/engine/scripts/common.sh
 
 assert_define "NODE_NAME"
+assert_define "TESTSPEC"
+assert_define "BUILD_NUMBER"
+assert_define "TESTDIR"
 
-OF_CLASSDIR_ARGS="--classdir $FUEGO_CORE/engine/overlays/base"
-OF_OUTPUT_FILE="$FUEGO_RW/work/${NODE_NAME}_prolog.sh"
-OF_OUTPUT_FILE_ARGS="--output $OF_OUTPUT_FILE"
+OF_CLASSDIRS="$FUEGO_CORE/engine/overlays/base"
 OF_OVGEN="$FUEGO_CORE/engine/scripts/ovgen.py"
 
-OF_DISTRIB_FILE=""
-OF_OVFILES_ARGS=""
-OF_TESTPLAN_ARGS=""
 #OF_DEBUG_ARGS="--debug 3"
 OF_DEBUG_ARGS=""
 
@@ -55,29 +53,19 @@ function set_overlay_vars() {
         abort_job "DISTRIB is not defined"
     fi
 
-    if [ "$TESTPLAN" ] ; then
-        echo "BATCH_TESTPLAN is not set, using $TESTPLAN.json testplan"
-        OF_TESTPLAN="$FUEGO_CORE/engine/overlays/testplans/$TESTPLAN.json"
+    # Create the log directory for this test run here so we can place the prolog.sh
+    export LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAMP}.${BUILD_NUMBER}"
+    mkdir -p $LOGDIR
 
-        if [ ! -f $OF_TESTPLAN ] ; then
-            if [ ! "$(basename $OF_TESTPLAN)" == "testplan_default.json" ] ; then
-                abort_job "$OF_TESTPLAN does not exist"
-            fi
-        fi
-        OF_TESTPLAN_ARGS="--testplan $OF_TESTPLAN"
-    fi
-
-    OF_OVFILES_ARGS="--ovfiles $OF_DISTRIB_FILE $OF_BOARD_FILE"
-
-    rm -f $OF_OUTPUT_FILE
+    rm -f $LOGDIR/prolog.sh
 
-    run_python $OF_OVGEN $OF_DEBUG_ARGS $OF_CLASSDIR_ARGS $OF_OVFILES_ARGS $OF_TESTPLAN_ARGS $OF_OUTPUT_FILE_ARGS || abort_job "Error while prolog.sh file generation"
+    run_python $OF_OVGEN $OF_DEBUG_ARGS --classdir $OF_CLASSDIRS --ovfiles $OF_DISTRIB_FILE $OF_BOARD_FILE --testdir $TESTDIR --testspec $TESTSPEC --output $LOGDIR/prolog.sh || abort_job "Error while prolog.sh file generation"
 
-    if [ ! -f "$OF_OUTPUT_FILE" ]
+    if [ ! -f "$LOGDIR/prolog.sh" ]
     then
-        abort_job "$OF_OUTPUT_FILE not found"
+        abort_job "$LOGDIR/prolog.sh not found"
     fi
 
-    source $OF_OUTPUT_FILE
+    source $LOGDIR/prolog.sh
 }
 
diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
index 6012453..10d1cf4 100755
--- a/engine/scripts/ovgen.py
+++ b/engine/scripts/ovgen.py
@@ -68,17 +68,15 @@ class OFLayer:
 
 class TestSpecs:
     def __init__(self):
-        self.testName=""
-        self.specList={}
-        self.method=None
-        self.tarball=None
+        self.name=""
+        self.variables={}
         self.fail_case=None
 
     def __repr__(self):
-        return "<TestSpecs name:\"%s\" specList:\"%s\" method:%s>\n" % (self.testName, self.specList, self.method)
+        return "<TestSpecs name:\"%s\" variables:\"%s\" fail_case:%s>\n" % (self.name, self.variables, self.fail_case)
 
     def __str__(self):
-        return "<TestSpecs name:\"%s\" specList:\"%s\" method:%s>\n" % (self.testName, self.specList, self.method)
+        return "<TestSpecs name:\"%s\" variables:\"%s\" fail_case:%s>\n" % (self.name, self.variables, self.fail_case)
 
 class OFParseException(Exception):
      def __init__(self, value):
@@ -377,7 +375,7 @@ def parseOverrideFile(overrideFile, layer, ofcls):
 
     return classes
 
-def generateProlog(outFilePath, ofcls, classes, tpFiles):
+def generateProlog(outFilePath, ofcls, classes, testdir, testspec):
     outfile = open(outFilePath, "w")
 
     for ofc in classes:
@@ -407,43 +405,34 @@ def generateProlog(outFilePath, ofcls, classes, tpFiles):
 
         file.write(outfile, "\n")
 
-    if tpFiles != None:
-        for tpf in tpFiles:
-            parseGenTestPlan(tpf, outfile, outFilePath)
-
-# generateSpec - generate shell output for all specs of specific test
-# Arguments:
-#     curTestSpecs:Dict  -  selected spec from testplan for current test
-#     testName: String -    name of test
-#     specName: String  -   name of test spec
-#     curSpecs: TestSpecs - all specs for current test
-#     fout - file output descriptor opened -> prolog.sh
-
-def generateSpec(curTestSpecs, testName, specName, curSpec, fout):
-    del curTestSpecs["name"]
-    for par in curTestSpecs:
-        if (par == "method" or par == "tarball" or par == "fail_case"):
-            continue
+    ts = parseSpec(testdir, testspec)
+    generateSpec(ts, outfile)
+
+# generateSpec - generate shell output for the spec
+#     ts:TestSpecs  -  parsed specs for our testdir
+#     testspec: String  -   name of test spec (e.g. 'default')
+#     outFilePath - file output descriptor opened -> prolog.sh
+def generateSpec(ts, fout):
+    debug_print("generating spec '%s' for '%s'" % (ts.variables['name'], ts.name))
 
-        varname = "%s_%s" % (testName, par)
+    for var in ts.variables:
+        if var == 'name':
+            continue
+        varname = "%s_%s" % (ts.name, var)
         varname = string.replace(varname, ".", "_").upper()
-        value = "%s" % (curTestSpecs[par])
+        value = "%s" % (ts.variables[var])
         outStr = '%s="%s"' % (varname, value)
-
-        debug_print (outStr, 3)
+        debug_print(outStr, 3)
         fout.write(outStr + "\n")
 
-    tNameUp = string.replace(testName, ".", "_").upper()
-    tNameUp = string.replace(tNameUp, "-", "_").upper()
-
-    #print "cts: %s\n" % (curTestSpecs)
-
-    if curSpec.fail_case:
-        fc_num = len(curSpec.fail_case)
+    if ts.fail_case:
+        tNameUp = string.replace(ts.name, ".", "_").upper()
+        tNameUp = string.replace(tNameUp, "-", "_").upper()
+        fc_num = len(ts.fail_case)
         outNum = "%s_FAIL_CASE_COUNT=%s" % (tNameUp, fc_num)
         fout.write(outNum + "\n")
 
-        for fmsg, num in zip(curSpec.fail_case, range(fc_num)):
+        for fmsg, num in zip(ts.fail_case, range(fc_num)):
             outPattern = "%s_FAIL_PATTERN_%s=\"%s\"" % (tNameUp, num, fmsg["fail_regexp"])
             outMessage = "%s_FAIL_MESSAGE_%s=\"%s\"" % (tNameUp, num, fmsg["fail_message"])
 
@@ -453,127 +442,46 @@ def generateSpec(curTestSpecs, testName, specName, curSpec, fout):
             fout.write(outPattern + "\n")
             fout.write(outMessage + "\n")
 
-    if "method" in curTestSpecs:
-        if (curTestSpecs["method"] == "tarball"):
-            outStr = "%s_METHOD=tarball\n%s_TARBALL_NAME=%s" % (tNameUp, tNameUp, curTestSpecs["tarball"])
-            fout.write(outStr + "\n")
-        else:
-            raise SpecException ("unkown method: %s" % (curTestSpecs["method"]))
-    elif curSpec.method:
-        if (curSpec.method == "tarball"):
-            outStr = ("%s_METHOD=tarball\n%s_TARBALL_NAME=%s" % (tNameUp, tNameUp, curSpec.tarball))
-            fout.write(outStr + "\n")
-        else:
-            raise SpecException ("unkown method: %s" % (curSpec.method))
-    else:
-	pass
-	# TRB 2016-09-09 - ignore missing test spec methods - this feature doesn't work or isn't used
-        #print("Can not find method for %s[%s]" % (testName,specName))
-        #outStr = ("%s_METHOD=\"none\"" % (tNameUp))
-        #fout.write(outStr + "\n")
-        # raise SpecException ("Can not find method for %s[%s]" % (testName,specName))
-
-
-def parseGenTestPlan(tpFilePath, fout, fname):
-    with open(tpFilePath) as f:
-        debug_print("parsing `%s' TP" % (tpFilePath), 1)
-        try:
-            jd = json.load(f)
-        except:
-            print "Error parsing testplan file %s" % tpFilePath
-            f.seek(0)
-            js = json.load(f)
-
-        name = jd["testPlanName"]
-        fout.write("#testplan: %s\n" % (name))
-        for t in jd["tests"]:
-            testName = t["testName"]
-            specName = t["spec"]
-
-            specs = {}
-            specs = parseSpecDir("/fuego-core/engine/tests/" + testName)
-
-            if testName not in specs:
-                raise SpecException("Cannot find test %s in spec list" % (testName))
-
-            curSpecs = specs[testName]
-
-            if specName not in curSpecs.specList:
-                raise SpecException("Cannot find spec %s in %s" % (specName, testName))
-
-            curSpecList = curSpecs.specList[specName]
-
-            debug_print("generating spec %s for `%s'" % (specName, testName))
-            generateSpec(curSpecList, testName, specName, curSpecs, fout)
-
-        fout.write("\n")
-
-def parseSpec(specFileName):
+def parseSpec(testdir, testspec):
+    # TODO: get fuego-core from env
+    specpath = '/fuego-core/engine/tests/%s/%s.spec' % (testdir, testdir)
     ts = TestSpecs()
-    ts.specList={}
 
-    debug_print("Parsing %s spec file" % (specFileName))
+    debug_print("Parsing %s spec file" % (specpath))
 
-    with open(specFileName) as f:
+    with open(specpath) as f:
         try:
             jd = json.load(f)
         except:
-            print "Error parsing spec file %s" % specFileName
-            f.seek(0)
-            jd = json.load(f)
+            raise Exception("Error parsing spec file %s" % specpath)
 
-        name = jd["testName"]
-        debug_print("parsing `%s' spec" % (name), 1)
-
-        if  "method" in jd:
-            if jd["method"] == "tarball":
-                ts.method = "tarball"
-                ts.tarball = jd["tarball"]
-            else:
-                raise SpecException("%s: Unknown method: %s", (name, ts.tarball))
+        ts.name = jd["testName"]
 
         if "fail_case" in jd:
             ts.fail_case = jd["fail_case"]
-            debug_print ("Found fail_case msgs for `%s' specs" % (name)
-)
-        for spec in jd["specs"]:
-            sn = spec["name"]
-            ts.testName = name
-            ts.specList[sn] = spec
-
-    return ts
+            debug_print ("Found fail_case msgs in '%s'" % specpath)
 
+        for spec in jd["specs"]:
+            if spec["name"] == testspec:
+                ts.variables = spec
 
-def parseSpecDir(specDir):
-    specFiles = glob.glob(specDir + "/*.spec")
-    tspList = {}
-
-    debug_print ("Found following %s spec files in %s" % (specFiles, specDir))
-
-    for sf in specFiles:
-        sp = parseSpec(sf)
-        tspList[sp.testName] = sp
-
-    debug_print ("parsed specs: %s" % (tspList))
-
-    return tspList
+    if not ts.variables:
+        raise Exception("Could not find %s in %s" % (testspec, specpath))
 
+    return ts
 
 def run(test_args=None):
     parser = argparse.ArgumentParser(description='Read OF class files, override files and generate prolog defining all variables and functions')
 
     parser.add_argument('--classdir', help='OF base class directory', required=True)
     parser.add_argument('--ovfiles', nargs='+', metavar='OVFILE', help='list of directories containing .override files', required=True)
-    parser.add_argument('--testplans', nargs='+', metavar='TESTPLAN', help='list of test plan files', required=False)
+    parser.add_argument('--testdir', help='e.g.: Benchmark.Dhrystone', required=True)
+    parser.add_argument('--testspec', nargs='+', metavar='TESTSPEC', help='testspec', required=True)
     parser.add_argument('--output', help='output file name', required=True)
     parser.add_argument('--debug', help='{1,2,3} debug level (default is no debugging)', type=int, required=False)
 
     args = parser.parse_args(args=test_args)
 
-    classdir = args.classdir
-    ovfiles = args.ovfiles
-    output = args.output
-
     if args.debug:
 
         if args.debug < 1 or args.debug > 3:
@@ -583,29 +491,22 @@ def run(test_args=None):
         global log_lvl
         log_lvl = args.debug
 
-    testPlans = {}
-    if args.testplans != None:
-        testPlans = args.testplans
-
-    debug_print ("Using =%s, ovfiles=%s" % (classdir, ovfiles), 1)
+    debug_print("Using classdir=%s, ovfiles=%s testdir=%s testspec=%s" % (args.classdir, args.ovfiles, args.testdir, args.testspec), 1)
 
     ofcls = {}
-    parseBaseDir(classdir, ofcls)
+    parseBaseDir(args.classdir, ofcls)
 
     layers = {}
 
     classes = []
-
-    for ovf in ovfiles:
+    for ovf in args.ovfiles:
         classes = classes + parseOverrideFile(ovf, layers, ofcls)
         debug_print ("parsed %s override\n------------\n" % (ovf))
 
-    generateProlog(output, ofcls, classes, testPlans)
-
+    generateProlog(args.output, ofcls, classes, args.testdir, args.testspec[0])
 
 def testrun():
-    test_args =  "--classdir /fuego-core/engine/overlays/base/ --ovfiles /fuego-ro/conf/boards/qemu-arm.board --output prolog.sh --testplans /fuego-core/engine/overlays/testplans/testplan_default.json --debug 2".split()
-    # test_args =  "--classdir overlays-new/base/ --ovfiles overlays-new/boards/minnow.board --output prolog.sh  --debug 2".split()
+    test_args =  "--debug 2 --classdir /fuego-core/engine/overlays/base/ --ovfiles /fuego-ro/boards/qemu-arm.board --testdir Benchmark.Dhrystone --testspec default --output prolog.sh".split()
     run(test_args)
 
 run()
-- 
2.7.4

Re: [Fuego] [PATCH 08/16] testplans: use testspecs instead of testplans internally

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
> 
> This is quite a big change but necessary. Testplans are useful
> to batch tests and define which spec or timeout we want for
> each test. However, we also want to be able to run tests
> individually even if they are not inside a testplan.
> 
> Until now one prolog.sh was generated for a testplan. This is not
> a good idea for several reasons: collisions between different
> test plans or the inability to have the same test with two
> different specs in the same testplan.
> 
> This patch puts the "prolog.sh" generated each time a job is run
> on the corresponding job's log folder. It also modifies the names
> of the jobs which are now <board>.<testspec>.<testdir>

Sounds good.

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh |   7 +-
>  engine/scripts/overlays.sh  |  36 +++------
>  engine/scripts/ovgen.py     | 191 +++++++++++---------------------------------
>  3 files changed, 61 insertions(+), 173 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index b7f0f72..2483f61 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -261,9 +261,7 @@ function pre_test {
> 
>    cmd "true" || abort_job "Cannot connect to $DEVICE via $TRANSPORT"
> 
> -# Create the log directory for this test run
>    export
> LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAM
> P}.${BUILD_NUMBER}"
> -  mkdir -p $LOGDIR
> 
>    # make a link to the log dir that Jenkins can get to
>    # (Jenkins might not have the BUILD_TIMESTAMP)
> @@ -400,10 +398,11 @@ function post_test {
>    # reset the signal handler to avoid an infinite loop
>    trap - SIGTERM SIGHUP SIGALRM SIGINT
> 
> +  export
> LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAM
> P}.${BUILD_NUMBER}"
> +
>    # source generated prolog.sh file since post_test is called separately
> -  source $FUEGO_RW/work/${NODE_NAME}_prolog.sh
> +  source $LOGDIR/prolog.sh
>    export SSHPASS=$PASSWORD
> -  export
> LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAM
> P}.${BUILD_NUMBER}"
> 
>    # re-source params to set correct DEVICE, LOGIN, SSH vars
>    source $FUEGO_CORE/engine/scripts/params.sh
> diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
> index e1d2233..cfecc73 100644
> --- a/engine/scripts/overlays.sh
> +++ b/engine/scripts/overlays.sh
> @@ -25,15 +25,13 @@
>  . $FUEGO_CORE/engine/scripts/common.sh
> 
>  assert_define "NODE_NAME"
> +assert_define "TESTSPEC"
> +assert_define "BUILD_NUMBER"
> +assert_define "TESTDIR"
> 
> -OF_CLASSDIR_ARGS="--classdir $FUEGO_CORE/engine/overlays/base"
> -OF_OUTPUT_FILE="$FUEGO_RW/work/${NODE_NAME}_prolog.sh"
> -OF_OUTPUT_FILE_ARGS="--output $OF_OUTPUT_FILE"
> +OF_CLASSDIRS="$FUEGO_CORE/engine/overlays/base"
>  OF_OVGEN="$FUEGO_CORE/engine/scripts/ovgen.py"
> 
> -OF_DISTRIB_FILE=""
> -OF_OVFILES_ARGS=""
> -OF_TESTPLAN_ARGS=""
>  #OF_DEBUG_ARGS="--debug 3"
>  OF_DEBUG_ARGS=""
> 
> @@ -55,29 +53,19 @@ function set_overlay_vars() {
>          abort_job "DISTRIB is not defined"
>      fi
> 
> -    if [ "$TESTPLAN" ] ; then
> -        echo "BATCH_TESTPLAN is not set, using $TESTPLAN.json testplan"
> -
> OF_TESTPLAN="$FUEGO_CORE/engine/overlays/testplans/$TESTPLAN.json"
> +    # Create the log directory for this test run here so we can place the
> prolog.sh
> +    export
> LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAM
> P}.${BUILD_NUMBER}"
> +    mkdir -p $LOGDIR
> 
> -        if [ ! -f $OF_TESTPLAN ] ; then
> -            if [ ! "$(basename $OF_TESTPLAN)" == "testplan_default.json" ] ; then
> -                abort_job "$OF_TESTPLAN does not exist"
> -            fi
> -        fi
> -        OF_TESTPLAN_ARGS="--testplan $OF_TESTPLAN"
> -    fi
> -
> -    OF_OVFILES_ARGS="--ovfiles $OF_DISTRIB_FILE $OF_BOARD_FILE"
> -
> -    rm -f $OF_OUTPUT_FILE
> +    rm -f $LOGDIR/prolog.sh
> 
> -    run_python $OF_OVGEN $OF_DEBUG_ARGS $OF_CLASSDIR_ARGS
> $OF_OVFILES_ARGS $OF_TESTPLAN_ARGS $OF_OUTPUT_FILE_ARGS ||
> abort_job "Error while prolog.sh file generation"
> +    run_python $OF_OVGEN $OF_DEBUG_ARGS --classdir $OF_CLASSDIRS --
> ovfiles $OF_DISTRIB_FILE $OF_BOARD_FILE --testdir $TESTDIR --testspec
> $TESTSPEC --output $LOGDIR/prolog.sh || abort_job "Error while prolog.sh
> file generation"
> 
> -    if [ ! -f "$OF_OUTPUT_FILE" ]
> +    if [ ! -f "$LOGDIR/prolog.sh" ]
>      then
> -        abort_job "$OF_OUTPUT_FILE not found"
> +        abort_job "$LOGDIR/prolog.sh not found"
>      fi
> 
> -    source $OF_OUTPUT_FILE
> +    source $LOGDIR/prolog.sh
>  }
> 
> diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
> index 6012453..10d1cf4 100755
> --- a/engine/scripts/ovgen.py
> +++ b/engine/scripts/ovgen.py
> @@ -68,17 +68,15 @@ class OFLayer:
> 
>  class TestSpecs:
>      def __init__(self):
> -        self.testName=""
> -        self.specList={}
> -        self.method=None
> -        self.tarball=None
> +        self.name=""
> +        self.variables={}
>          self.fail_case=None
> 
>      def __repr__(self):
> -        return "<TestSpecs name:\"%s\" specList:\"%s\" method:%s>\n" %
> (self.testName, self.specList, self.method)
> +        return "<TestSpecs name:\"%s\" variables:\"%s\" fail_case:%s>\n" %
> (self.name, self.variables, self.fail_case)
> 
>      def __str__(self):
> -        return "<TestSpecs name:\"%s\" specList:\"%s\" method:%s>\n" %
> (self.testName, self.specList, self.method)
> +        return "<TestSpecs name:\"%s\" variables:\"%s\" fail_case:%s>\n" %
> (self.name, self.variables, self.fail_case)
> 
>  class OFParseException(Exception):
>       def __init__(self, value):
> @@ -377,7 +375,7 @@ def parseOverrideFile(overrideFile, layer, ofcls):
> 
>      return classes
> 
> -def generateProlog(outFilePath, ofcls, classes, tpFiles):
> +def generateProlog(outFilePath, ofcls, classes, testdir, testspec):
>      outfile = open(outFilePath, "w")
> 
>      for ofc in classes:
> @@ -407,43 +405,34 @@ def generateProlog(outFilePath, ofcls, classes,
> tpFiles):
> 
>          file.write(outfile, "\n")
> 
> -    if tpFiles != None:
> -        for tpf in tpFiles:
> -            parseGenTestPlan(tpf, outfile, outFilePath)
> -
> -# generateSpec - generate shell output for all specs of specific test
> -# Arguments:
> -#     curTestSpecs:Dict  -  selected spec from testplan for current test
> -#     testName: String -    name of test
> -#     specName: String  -   name of test spec
> -#     curSpecs: TestSpecs - all specs for current test
> -#     fout - file output descriptor opened -> prolog.sh
> -
> -def generateSpec(curTestSpecs, testName, specName, curSpec, fout):
> -    del curTestSpecs["name"]
> -    for par in curTestSpecs:
> -        if (par == "method" or par == "tarball" or par == "fail_case"):
> -            continue
> +    ts = parseSpec(testdir, testspec)
> +    generateSpec(ts, outfile)
> +
> +# generateSpec - generate shell output for the spec
> +#     ts:TestSpecs  -  parsed specs for our testdir
> +#     testspec: String  -   name of test spec (e.g. 'default')
> +#     outFilePath - file output descriptor opened -> prolog.sh
> +def generateSpec(ts, fout):
> +    debug_print("generating spec '%s' for '%s'" % (ts.variables['name'],
> ts.name))
> 
> -        varname = "%s_%s" % (testName, par)
> +    for var in ts.variables:
> +        if var == 'name':
> +            continue
> +        varname = "%s_%s" % (ts.name, var)
>          varname = string.replace(varname, ".", "_").upper()
> -        value = "%s" % (curTestSpecs[par])
> +        value = "%s" % (ts.variables[var])
>          outStr = '%s="%s"' % (varname, value)
> -
> -        debug_print (outStr, 3)
> +        debug_print(outStr, 3)
>          fout.write(outStr + "\n")
> 
> -    tNameUp = string.replace(testName, ".", "_").upper()
> -    tNameUp = string.replace(tNameUp, "-", "_").upper()
> -
> -    #print "cts: %s\n" % (curTestSpecs)
> -
> -    if curSpec.fail_case:
> -        fc_num = len(curSpec.fail_case)
> +    if ts.fail_case:
> +        tNameUp = string.replace(ts.name, ".", "_").upper()
> +        tNameUp = string.replace(tNameUp, "-", "_").upper()
> +        fc_num = len(ts.fail_case)
>          outNum = "%s_FAIL_CASE_COUNT=%s" % (tNameUp, fc_num)
>          fout.write(outNum + "\n")
> 
> -        for fmsg, num in zip(curSpec.fail_case, range(fc_num)):
> +        for fmsg, num in zip(ts.fail_case, range(fc_num)):
>              outPattern = "%s_FAIL_PATTERN_%s=\"%s\"" % (tNameUp, num,
> fmsg["fail_regexp"])
>              outMessage = "%s_FAIL_MESSAGE_%s=\"%s\"" % (tNameUp, num,
> fmsg["fail_message"])
> 
> @@ -453,127 +442,46 @@ def generateSpec(curTestSpecs, testName,
> specName, curSpec, fout):
>              fout.write(outPattern + "\n")
>              fout.write(outMessage + "\n")
> 
> -    if "method" in curTestSpecs:
> -        if (curTestSpecs["method"] == "tarball"):
> -            outStr = "%s_METHOD=tarball\n%s_TARBALL_NAME=%s" %
> (tNameUp, tNameUp, curTestSpecs["tarball"])
> -            fout.write(outStr + "\n")
> -        else:
> -            raise SpecException ("unkown method: %s" %
> (curTestSpecs["method"]))
> -    elif curSpec.method:
> -        if (curSpec.method == "tarball"):
> -            outStr = ("%s_METHOD=tarball\n%s_TARBALL_NAME=%s" %
> (tNameUp, tNameUp, curSpec.tarball))
> -            fout.write(outStr + "\n")
> -        else:
> -            raise SpecException ("unkown method: %s" % (curSpec.method))
> -    else:
> -	pass
> -	# TRB 2016-09-09 - ignore missing test spec methods - this feature
> doesn't work or isn't used
> -        #print("Can not find method for %s[%s]" % (testName,specName))
> -        #outStr = ("%s_METHOD=\"none\"" % (tNameUp))
> -        #fout.write(outStr + "\n")
> -        # raise SpecException ("Can not find method for %s[%s]" %
> (testName,specName))
> -
> -
> -def parseGenTestPlan(tpFilePath, fout, fname):
> -    with open(tpFilePath) as f:
> -        debug_print("parsing `%s' TP" % (tpFilePath), 1)
> -        try:
> -            jd = json.load(f)
> -        except:
> -            print "Error parsing testplan file %s" % tpFilePath
> -            f.seek(0)
> -            js = json.load(f)
> -
> -        name = jd["testPlanName"]
> -        fout.write("#testplan: %s\n" % (name))
> -        for t in jd["tests"]:
> -            testName = t["testName"]
> -            specName = t["spec"]
> -
> -            specs = {}
> -            specs = parseSpecDir("/fuego-core/engine/tests/" + testName)
> -
> -            if testName not in specs:
> -                raise SpecException("Cannot find test %s in spec list" % (testName))
> -
> -            curSpecs = specs[testName]
> -
> -            if specName not in curSpecs.specList:
> -                raise SpecException("Cannot find spec %s in %s" % (specName,
> testName))
> -
> -            curSpecList = curSpecs.specList[specName]
> -
> -            debug_print("generating spec %s for `%s'" % (specName, testName))
> -            generateSpec(curSpecList, testName, specName, curSpecs, fout)
> -
> -        fout.write("\n")
> -
> -def parseSpec(specFileName):
> +def parseSpec(testdir, testspec):
> +    # TODO: get fuego-core from env
> +    specpath = '/fuego-core/engine/tests/%s/%s.spec' % (testdir, testdir)
>      ts = TestSpecs()
> -    ts.specList={}
> 
> -    debug_print("Parsing %s spec file" % (specFileName))
> +    debug_print("Parsing %s spec file" % (specpath))
> 
> -    with open(specFileName) as f:
> +    with open(specpath) as f:
>          try:
>              jd = json.load(f)
>          except:
> -            print "Error parsing spec file %s" % specFileName
> -            f.seek(0)
> -            jd = json.load(f)
> +            raise Exception("Error parsing spec file %s" % specpath)
> 
> -        name = jd["testName"]
> -        debug_print("parsing `%s' spec" % (name), 1)
> -
> -        if  "method" in jd:
> -            if jd["method"] == "tarball":
> -                ts.method = "tarball"
> -                ts.tarball = jd["tarball"]
> -            else:
> -                raise SpecException("%s: Unknown method: %s", (name, ts.tarball))
> +        ts.name = jd["testName"]
> 
>          if "fail_case" in jd:
>              ts.fail_case = jd["fail_case"]
> -            debug_print ("Found fail_case msgs for `%s' specs" % (name)
> -)
> -        for spec in jd["specs"]:
> -            sn = spec["name"]
> -            ts.testName = name
> -            ts.specList[sn] = spec
> -
> -    return ts
> +            debug_print ("Found fail_case msgs in '%s'" % specpath)
> 
> +        for spec in jd["specs"]:
> +            if spec["name"] == testspec:
> +                ts.variables = spec
> 
> -def parseSpecDir(specDir):
> -    specFiles = glob.glob(specDir + "/*.spec")
> -    tspList = {}
> -
> -    debug_print ("Found following %s spec files in %s" % (specFiles, specDir))
> -
> -    for sf in specFiles:
> -        sp = parseSpec(sf)
> -        tspList[sp.testName] = sp
> -
> -    debug_print ("parsed specs: %s" % (tspList))
> -
> -    return tspList
> +    if not ts.variables:
> +        raise Exception("Could not find %s in %s" % (testspec, specpath))
> 
> +    return ts
> 
>  def run(test_args=None):
>      parser = argparse.ArgumentParser(description='Read OF class files,
> override files and generate prolog defining all variables and functions')
> 
>      parser.add_argument('--classdir', help='OF base class directory',
> required=True)
>      parser.add_argument('--ovfiles', nargs='+', metavar='OVFILE', help='list of
> directories containing .override files', required=True)
> -    parser.add_argument('--testplans', nargs='+', metavar='TESTPLAN',
> help='list of test plan files', required=False)
> +    parser.add_argument('--testdir', help='e.g.: Benchmark.Dhrystone',
> required=True)
> +    parser.add_argument('--testspec', nargs='+', metavar='TESTSPEC',
> help='testspec', required=True)
>      parser.add_argument('--output', help='output file name', required=True)
>      parser.add_argument('--debug', help='{1,2,3} debug level (default is no
> debugging)', type=int, required=False)
> 
>      args = parser.parse_args(args=test_args)
> 
> -    classdir = args.classdir
> -    ovfiles = args.ovfiles
> -    output = args.output
> -
>      if args.debug:
> 
>          if args.debug < 1 or args.debug > 3:
> @@ -583,29 +491,22 @@ def run(test_args=None):
>          global log_lvl
>          log_lvl = args.debug
> 
> -    testPlans = {}
> -    if args.testplans != None:
> -        testPlans = args.testplans
> -
> -    debug_print ("Using =%s, ovfiles=%s" % (classdir, ovfiles), 1)
> +    debug_print("Using classdir=%s, ovfiles=%s testdir=%s testspec=%s" %
> (args.classdir, args.ovfiles, args.testdir, args.testspec), 1)
> 
>      ofcls = {}
> -    parseBaseDir(classdir, ofcls)
> +    parseBaseDir(args.classdir, ofcls)
> 
>      layers = {}
> 
>      classes = []
> -
> -    for ovf in ovfiles:
> +    for ovf in args.ovfiles:
>          classes = classes + parseOverrideFile(ovf, layers, ofcls)
>          debug_print ("parsed %s override\n------------\n" % (ovf))
> 
> -    generateProlog(output, ofcls, classes, testPlans)
> -
> +    generateProlog(args.output, ofcls, classes, args.testdir, args.testspec[0])
> 
>  def testrun():
> -    test_args =  "--classdir /fuego-core/engine/overlays/base/ --ovfiles
> /fuego-ro/conf/boards/qemu-arm.board --output prolog.sh --testplans
> /fuego-core/engine/overlays/testplans/testplan_default.json --debug
> 2".split()
> -    # test_args =  "--classdir overlays-new/base/ --ovfiles overlays-
> new/boards/minnow.board --output prolog.sh  --debug 2".split()
> +    test_args =  "--debug 2 --classdir /fuego-core/engine/overlays/base/ --
> ovfiles /fuego-ro/boards/qemu-arm.board --testdir Benchmark.Dhrystone --
> testspec default --output prolog.sh".split()
>      run(test_args)
> 
>  run()
> --

OK - good stuff.

Acked-by: Tim Bird
 -- Tim

[Fuego] [PATCH 09/16] distrib: DISTRIB must be defined in the board file

[Daniel Sangorrin]

If it isn't it defaults to nosyslogd.dist

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc         | 21 ++++-----------------
 engine/scripts/overlays.sh | 20 +++++++++++---------
 2 files changed, 15 insertions(+), 26 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index be76427..eb1eb3e 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -741,7 +741,7 @@ def get_includes(include_filename, conf):
     inc_vars = parse_shell_file(inc_path, conf)
     return inc_vars
 
-def create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks):
+def create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks):
     # flot only necessary for Benchmarks
     if str(test).split('.')[0] == 'Benchmark':
         flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
@@ -780,7 +780,6 @@ export Target_PreCleanup={precleanup}
 export Target_PostCleanup={postcleanup}
 export TESTDIR={testdir}
 export TESTNAME={testname}
-export DISTRIB="{distrib}"
 export TESTPLAN="{testplan}"
 timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
 </command>
@@ -798,7 +797,7 @@ timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
     <buildWrappers/>
 </project>
 """.format(board=board, reboot=str(reboot), rebuild=str(rebuild), precleanup=str(precleanup), postcleanup=str(postcleanup),
-        distrib=distrib, testdir=str(test), testname=str(test).split('.')[1],
+        testdir=str(test), testname=str(test).split('.')[1],
         testplan=testplan, timeout=timeout, flot_link=flot_link, html_links=html_links))
     fd.close()
 
@@ -929,14 +928,6 @@ def do_add_jobs(conf, options):
     else:
         raise Exception('No testplan or testcase supplied.')
 
-    if '-d' in options:
-        distrib = options[options.index('-d') + 1]
-        # TODO: distrib should be in the board's configuration
-        options.remove('-d')
-        options.remove(distrib)
-    else:
-        distrib = 'nosyslogd.dist'
-
     if test:
         timeout = '20m'
         reboot = 'false'
@@ -944,11 +935,11 @@ def do_add_jobs(conf, options):
         precleanup = 'true'
         postcleanup = 'true'
         jobextralinks = 'None'
-        create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+        create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
     else:
         alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks = parse_testplan(testplan)
         for test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks):
-            create_job(board, distrib, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+            create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
         create_batch_job(board, testplan, alltests)
     sys.exit(0)
 
@@ -1785,12 +1776,8 @@ def do_run_test(conf, test_name, target, options):
     os.environ["NODE_LABELS"] = build_data.target_name
     os.environ["Device"] = build_data.target_name
     os.environ["TESTDIR"] = build_data.testdir
-
     os.environ["WORKSPACE"] = build_data.workspace
 
-    # FIXTHIS - (!) in run_test, HARDCODE DISTRIB for now (Yuk)
-    target.env_vars["DISTRIB"]="nosyslogd.dist"
-
     for var_name in target.env_vars.keys():
         os.environ[var_name] = target.env_vars[var_name]
 
diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
index cfecc73..2565428 100644
--- a/engine/scripts/overlays.sh
+++ b/engine/scripts/overlays.sh
@@ -42,16 +42,18 @@ function set_overlay_vars() {
         abort_job "$OF_BOARD_FILE does not exist"
     fi
 
-    # check for $DISTRIB and make file path to it
-    if [ "$DISTRIB" ]; then
-        echo "using $DISTRIB overlay"
-        OF_DISTRIB_FILE="$FUEGO_CORE/engine/overlays/distribs/$DISTRIB"
-        if [ ! -f $OF_DISTRIB_FILE ] ; then
-            abort_job "$OF_DISTRIB_FILE does not exist"
-        fi
-    else
-        abort_job "DISTRIB is not defined"
+    # check for DISTRIB in the board configuration
+    DISTRIB_LINE=$(grep DISTRIB $OF_BOARD_FILE)
+    eval $DISTRIB_LINE
+    if [ ! "$DISTRIB" ]; then
+        # TODO: automatically discover the best option
+        DISTRIB="nosyslogd.dist"
     fi
+    OF_DISTRIB_FILE="$FUEGO_CORE/engine/overlays/distribs/$DISTRIB"
+    if [ ! -f $OF_DISTRIB_FILE ] ; then
+        abort_job "$OF_DISTRIB_FILE does not exist"
+    fi
+    echo "Using $DISTRIB overlay"
 
     # Create the log directory for this test run here so we can place the prolog.sh
     export LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAMP}.${BUILD_NUMBER}"
-- 
2.7.4

Re: [Fuego] [PATCH 09/16] distrib: DISTRIB must be defined in the board file

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> If it isn't it defaults to nosyslogd.dist
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc         | 21 ++++-----------------
>  engine/scripts/overlays.sh | 20 +++++++++++---------
>  2 files changed, 15 insertions(+), 26 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index be76427..eb1eb3e 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -741,7 +741,7 @@ def get_includes(include_filename, conf):
>      inc_vars = parse_shell_file(inc_path, conf)
>      return inc_vars
> 
> -def create_job(board, distrib, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks):
> +def create_job(board, testplan, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks):
>      # flot only necessary for Benchmarks
>      if str(test).split('.')[0] == 'Benchmark':
>          flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
> @@ -780,7 +780,6 @@ export Target_PreCleanup={precleanup}
>  export Target_PostCleanup={postcleanup}
>  export TESTDIR={testdir}
>  export TESTNAME={testname}
> -export DISTRIB="{distrib}"
>  export TESTPLAN="{testplan}"
>  timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
>  </command>
> @@ -798,7 +797,7 @@ timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
>      <buildWrappers/>
>  </project>
>  """.format(board=board, reboot=str(reboot), rebuild=str(rebuild),
> precleanup=str(precleanup), postcleanup=str(postcleanup),
> -        distrib=distrib, testdir=str(test), testname=str(test).split('.')[1],
> +        testdir=str(test), testname=str(test).split('.')[1],
>          testplan=testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links))
>      fd.close()
> 
> @@ -929,14 +928,6 @@ def do_add_jobs(conf, options):
>      else:
>          raise Exception('No testplan or testcase supplied.')
> 
> -    if '-d' in options:
> -        distrib = options[options.index('-d') + 1]
> -        # TODO: distrib should be in the board's configuration
> -        options.remove('-d')
> -        options.remove(distrib)
> -    else:
> -        distrib = 'nosyslogd.dist'
> -
>      if test:
>          timeout = '20m'
>          reboot = 'false'
> @@ -944,11 +935,11 @@ def do_add_jobs(conf, options):
>          precleanup = 'true'
>          postcleanup = 'true'
>          jobextralinks = 'None'
> -        create_job(board, distrib, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks)
> +        create_job(board, testplan, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks)
>      else:
>          alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks = parse_testplan(testplan)
>          for test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks):
> -            create_job(board, distrib, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks)
> +            create_job(board, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks)
>          create_batch_job(board, testplan, alltests)
>      sys.exit(0)
> 
> @@ -1785,12 +1776,8 @@ def do_run_test(conf, test_name, target,
> options):
>      os.environ["NODE_LABELS"] = build_data.target_name
>      os.environ["Device"] = build_data.target_name
>      os.environ["TESTDIR"] = build_data.testdir
> -
>      os.environ["WORKSPACE"] = build_data.workspace
> 
> -    # FIXTHIS - (!) in run_test, HARDCODE DISTRIB for now (Yuk)
> -    target.env_vars["DISTRIB"]="nosyslogd.dist"
> 
Hey!  Thanks for cleaning up my code!

-
>      for var_name in target.env_vars.keys():
>          os.environ[var_name] = target.env_vars[var_name]
> 
> diff --git a/engine/scripts/overlays.sh b/engine/scripts/overlays.sh
> index cfecc73..2565428 100644
> --- a/engine/scripts/overlays.sh
> +++ b/engine/scripts/overlays.sh
> @@ -42,16 +42,18 @@ function set_overlay_vars() {
>          abort_job "$OF_BOARD_FILE does not exist"
>      fi
> 
> -    # check for $DISTRIB and make file path to it
> -    if [ "$DISTRIB" ]; then
> -        echo "using $DISTRIB overlay"
> -        OF_DISTRIB_FILE="$FUEGO_CORE/engine/overlays/distribs/$DISTRIB"
> -        if [ ! -f $OF_DISTRIB_FILE ] ; then
> -            abort_job "$OF_DISTRIB_FILE does not exist"
> -        fi
> -    else
> -        abort_job "DISTRIB is not defined"
> +    # check for DISTRIB in the board configuration
> +    DISTRIB_LINE=$(grep DISTRIB $OF_BOARD_FILE)
> +    eval $DISTRIB_LINE

Ok - that's a bit dicey.  Hopefully no one ever puts:
"rm -rf / ; DISTRIB=foo" in their board file.

I think we should parse this instead of evaluate it.
I modified the patch to do this.

> +    if [ ! "$DISTRIB" ]; then
> +        # TODO: automatically discover the best option
> +        DISTRIB="nosyslogd.dist"
>      fi
> +    OF_DISTRIB_FILE="$FUEGO_CORE/engine/overlays/distribs/$DISTRIB"
> +    if [ ! -f $OF_DISTRIB_FILE ] ; then
> +        abort_job "$OF_DISTRIB_FILE does not exist"
> +    fi
> +    echo "Using $DISTRIB overlay"
> 
>      # Create the log directory for this test run here so we can place the
> prolog.sh
>      export
> LOGDIR="$FUEGO_RW/logs/$TESTDIR/${NODE_NAME}.${BUILD_TIMESTAM
> P}.${BUILD_NUMBER}"
> --
> 2.7.4

Acked-by: Tim Bird

Thanks.
 -- Tim

[Fuego] [PATCH 10/16] user_checks: add to commands so they execute under jenkins

[Daniel Sangorrin]

Instead of using sudo -u jenkins ftc xxx use the user_checks
allows to run the command as jenkins even if the caller
is running as root

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index eb1eb3e..e08c56a 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -2769,6 +2769,7 @@ def main():
 
     if command=="add-jobs":
         # adds Jenkins jobs
+        user_check()
         try:
             do_add_jobs(conf, options)
         except Exception as e:
@@ -2776,6 +2777,7 @@ def main():
 
     if command=="rm-jobs":
         # removes Jenkins jobs
+        user_check()
         try:
             do_rm_jobs(conf, options)
         except Exception as e:
@@ -2783,6 +2785,7 @@ def main():
 
     if command=="add-nodes":
         # adds Jenkins nodes
+        user_check()
         try:
             do_add_nodes(conf, options)
         except Exception as e:
@@ -2790,6 +2793,7 @@ def main():
 
     if command=="rm-nodes":
         # removes Jenkins nodes
+        user_check()
         try:
             do_rm_nodes(conf, options)
         except Exception as e:
@@ -2801,10 +2805,12 @@ def main():
 
     if command=="list-nodes":
         # shows jenkins nodes
+        user_check()
         do_list_nodes(conf)
 
     if command=="list-jobs":
         # shows jenkins jobs
+        user_check()
         do_list_jobs(conf)
 
     if command=="list-plans":
-- 
2.7.4

Re: [Fuego] [PATCH 10/16] user_checks: add to commands so they execute under jenkins

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> Instead of using sudo -u jenkins ftc xxx use the user_checks
> allows to run the command as jenkins even if the caller
> is running as root
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index eb1eb3e..e08c56a 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -2769,6 +2769,7 @@ def main():
> 
>      if command=="add-jobs":
>          # adds Jenkins jobs
> +        user_check()
>          try:
>              do_add_jobs(conf, options)
>          except Exception as e:
> @@ -2776,6 +2777,7 @@ def main():
> 
>      if command=="rm-jobs":
>          # removes Jenkins jobs
> +        user_check()
>          try:
>              do_rm_jobs(conf, options)
>          except Exception as e:
> @@ -2783,6 +2785,7 @@ def main():
> 
>      if command=="add-nodes":
>          # adds Jenkins nodes
> +        user_check()
>          try:
>              do_add_nodes(conf, options)
>          except Exception as e:
> @@ -2790,6 +2793,7 @@ def main():
> 
>      if command=="rm-nodes":
>          # removes Jenkins nodes
> +        user_check()
>          try:
>              do_rm_nodes(conf, options)
>          except Exception as e:
> @@ -2801,10 +2805,12 @@ def main():
> 
>      if command=="list-nodes":
>          # shows jenkins nodes
> +        user_check()
>          do_list_nodes(conf)
> 
>      if command=="list-jobs":
>          # shows jenkins jobs
> +        user_check()
>          do_list_jobs(conf)
> 
>      if command=="list-plans":
> --
> 2.7.4

Looks good.  I'm still wondering why these need to be run as 'jenkins'.
I have run them as root without problems.  Is this a permission thing
with Jenkins security configured a certain way?

Acked-by: Tim Bird
 -- Tim

[Fuego] [PATCH 11/16] ftc:rm-jobs: remove limitation of 4 words

[Daniel Sangorrin]

This is so that batch or other jobs can be removed too

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 2 --
 1 file changed, 2 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index e08c56a..6f7d635 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -952,8 +952,6 @@ def do_rm_jobs(conf, options):
     else:
         for opt in options:
             jobs = [job['name'] for job in server.get_jobs()]
-            if len(opt.split('.')) != 4:
-                raise Exception('%s does not have 4 parts.' % opt)
             pattern = '^' + opt
             pattern = pattern.replace('.', '\.')
             pattern = pattern.replace('*', '\w*')
-- 
2.7.4

Re: [Fuego] [PATCH 11/16] ftc:rm-jobs: remove limitation of 4 words

[Bird, Timothy]

> -----Original Message-----
> From Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 11/16] ftc:rm-jobs: remove limitation of 4 words
> 
> This is so that batch or other jobs can be removed too
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc | 2 --
>  1 file changed, 2 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index e08c56a..6f7d635 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -952,8 +952,6 @@ def do_rm_jobs(conf, options):
>      else:
>          for opt in options:
>              jobs = [job['name'] for job in server.get_jobs()]
> -            if len(opt.split('.')) != 4:
> -                raise Exception('%s does not have 4 parts.' % opt)
>              pattern = '^' + opt
>              pattern = pattern.replace('.', '\.')
>              pattern = pattern.replace('*', '\w*')
> --
> 2.7.4

Acked-by: Tim Bird
 -- Tim

[Fuego] [PATCH 12/16] description setter: put log if the test fails

[Daniel Sangorrin]

Before log was set by default, but as Tim noticed the
fail case was not considered.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 6f7d635..82a8c18 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -751,6 +751,7 @@ def create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, post
     # prepare links for descriptionsetter. Put a link to testlog.txt by default
     template_link = '&lt;a href=&quot;/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_ID}.${BUILD_ID}/%%s&quot;&gt;%%s&lt;/a&gt;' % test
     html_links = template_link % ('testlog.txt', 'log')
+    failed_links = html_links
     if jobextralinks != 'None':
         for key, value in jobextralinks.iteritems():
             html_links = html_links + ' ' + template_link % (value, key)
@@ -788,17 +789,18 @@ timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
     <publishers>
     {flot_link}
     <hudson.plugins.descriptionsetter.DescriptionSetterPublisher plugin="description-setter@1.10">
-        <regexp></regexp>
-        <regexpForFailed></regexpForFailed>
-        <description>{html_links}</description>
-        <setForMatrix>false</setForMatrix>
+      <regexp></regexp>
+      <regexpForFailed></regexpForFailed>
+      <description>{html_links}</description>
+      <descriptionForFailed>{failed_links}</descriptionForFailed>
+      <setForMatrix>false</setForMatrix>
     </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
     </publishers>
     <buildWrappers/>
 </project>
 """.format(board=board, reboot=str(reboot), rebuild=str(rebuild), precleanup=str(precleanup), postcleanup=str(postcleanup),
         testdir=str(test), testname=str(test).split('.')[1],
-        testplan=testplan, timeout=timeout, flot_link=flot_link, html_links=html_links))
+        testplan=testplan, timeout=timeout, flot_link=flot_link, html_links=html_links, failed_links=failed_links))
     fd.close()
 
     print("Creating job " + test)
-- 
2.7.4

Re: [Fuego] [PATCH 12/16] description setter: put log if the test fails

[Bird, Timothy]

Looks good.

Acked-by: Tim Bird
 -- Tim

> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, March 29, 2017 6:05 PM
>
> Before log was set by default, but as Tim noticed the
> fail case was not considered.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 6f7d635..82a8c18 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -751,6 +751,7 @@ def create_job(board, testplan, test, timeout, reboot,
> rebuild, precleanup, post
>      # prepare links for descriptionsetter. Put a link to testlog.txt by default
>      template_link = '&lt;a
> href=&quot;/fuego/userContent/fuego.logs/%s/${NODE_NAME}.${BUILD_I
> D}.${BUILD_ID}/%%s&quot;&gt;%%s&lt;/a&gt;' % test
>      html_links = template_link % ('testlog.txt', 'log')
> +    failed_links = html_links
>      if jobextralinks != 'None':
>          for key, value in jobextralinks.iteritems():
>              html_links = html_links + ' ' + template_link % (value, key)
> @@ -788,17 +789,18 @@ timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
>      <publishers>
>      {flot_link}
>      <hudson.plugins.descriptionsetter.DescriptionSetterPublisher
> plugin="description-setter@1.10">
> -        <regexp></regexp>
> -        <regexpForFailed></regexpForFailed>
> -        <description>{html_links}</description>
> -        <setForMatrix>false</setForMatrix>
> +      <regexp></regexp>
> +      <regexpForFailed></regexpForFailed>
> +      <description>{html_links}</description>
> +      <descriptionForFailed>{failed_links}</descriptionForFailed>
> +      <setForMatrix>false</setForMatrix>
>      </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
>      </publishers>
>      <buildWrappers/>
>  </project>
>  """.format(board=board, reboot=str(reboot), rebuild=str(rebuild),
> precleanup=str(precleanup), postcleanup=str(postcleanup),
>          testdir=str(test), testname=str(test).split('.')[1],
> -        testplan=testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links))
> +        testplan=testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links, failed_links=failed_links))
>      fd.close()
> 
>      print("Creating job " + test)
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

[Fuego] [PATCH 13/16] ftc: remove testplans and use testspecs

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc | 36 +++++++++++++++++++++---------------
 1 file changed, 21 insertions(+), 15 deletions(-)

diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 82a8c18..372c32a 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -85,7 +85,7 @@ TARGET_ENV_VAR="FTC_TARGET"
 # format for command_help mapping with: key=name, value=(summary, long description)
 command_help = {
 "add-jobs":("Adds jobs to Jenkins.",
-    """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase>] [-d <foo.dist>]
+    """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase> -s <testspec>]
   Example: sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
   Example: sudo -u jenkins ftc add-jobs -b docker -t Benchmark.Dhrystone
   Use list-plans to see the available test plans.
@@ -93,13 +93,13 @@ command_help = {
   This interface may change in the future."""),
 
 "rm-jobs":("Removes jobs from Jenkins.",
-    """Usage: ftc rm-jobs <target>.<testplan>.<testcase>
+    """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
   Use list-jobs to see the existing jobs. A wildcard can be used to
   specify which jobs to remove (just make sure you have 4 words):
     Example: sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
     Example: sudo -u jenkins ftc rm-jobs docker.*.F*.*stress
-  Multiple combinations of the <target>.<testplan>.<testcase> pattern
-  can be passed as well.
+  Multiple combinations of the <target>.<testspec>.<testtype>.<testcase>
+  pattern can be passed as well.
 
   If no option is provided all existing jobs will be removed."""),
 
@@ -741,7 +741,7 @@ def get_includes(include_filename, conf):
     inc_vars = parse_shell_file(inc_path, conf)
     return inc_vars
 
-def create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks):
+def create_job(board, testspec, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks):
     # flot only necessary for Benchmarks
     if str(test).split('.')[0] == 'Benchmark':
         flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
@@ -781,7 +781,7 @@ export Target_PreCleanup={precleanup}
 export Target_PostCleanup={postcleanup}
 export TESTDIR={testdir}
 export TESTNAME={testname}
-export TESTPLAN="{testplan}"
+export TESTSPEC="{testspec}"
 timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
 </command>
     </hudson.tasks.Shell>
@@ -800,13 +800,13 @@ timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
 </project>
 """.format(board=board, reboot=str(reboot), rebuild=str(rebuild), precleanup=str(precleanup), postcleanup=str(postcleanup),
         testdir=str(test), testname=str(test).split('.')[1],
-        testplan=testplan, timeout=timeout, flot_link=flot_link, html_links=html_links, failed_links=failed_links))
+        testspec=testspec, timeout=timeout, flot_link=flot_link, html_links=html_links, failed_links=failed_links))
     fd.close()
 
     print("Creating job " + test)
     try:
         subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/fuego create-job ' +
-            board + '.' + testplan + '.' + str(test) + ' < ' + tmp, shell=True)
+            board + '.' + testspec + '.' + str(test) + ' < ' + tmp, shell=True)
         subprocess.call('rm -f ' + tmp, shell=True)
     except Exception as e:
         print("Job already exists")
@@ -862,6 +862,7 @@ def create_batch_job(board, testplan, alltests):
 def parse_testplan(testplan):
     abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
     alltests = []
+    alltestspecs = []
     alltimeouts = []
     allextralinks = []
     allreboots = []
@@ -873,6 +874,7 @@ def parse_testplan(testplan):
         plan = json.load(f)
         for test in plan['tests']:
             alltests.append(test['testName'])
+            alltestspecs.append(test['spec'])
             if 'timeout' in test:
                 alltimeouts.append(test['timeout'])
             else:
@@ -897,7 +899,7 @@ def parse_testplan(testplan):
                 allextralinks.append(test['extralinks'])
             else:
                 allextralinks.append('None')
-    return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks
+    return alltests, alltestspecs, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks
 
 def do_add_jobs(conf, options):
     if '-b' in options:
@@ -925,8 +927,12 @@ def do_add_jobs(conf, options):
             raise Exception('Test %s not found.' % test)
         options.remove('-t')
         options.remove(test)
-        # TODO: use spec instead of testplan_default
-        testplan = 'testplan_default'
+        if '-s' in options:
+            spec = options[options.index('-s') + 1]
+            options.remove('-s')
+            options.remove(spec)
+        else:
+            spec = 'default'
     else:
         raise Exception('No testplan or testcase supplied.')
 
@@ -937,11 +943,11 @@ def do_add_jobs(conf, options):
         precleanup = 'true'
         postcleanup = 'true'
         jobextralinks = 'None'
-        create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+        create_job(board, spec, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
     else:
-        alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks = parse_testplan(testplan)
-        for test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks):
-            create_job(board, testplan, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
+        alltests, alltestspecs, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks = parse_testplan(testplan)
+        for test, spec, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks in zip(alltests, alltestspecs, alltimeouts, allreboots, allrebuilds, allprecleanups, allpostcleanups, allextralinks):
+            create_job(board, spec, test, timeout, reboot, rebuild, precleanup, postcleanup, jobextralinks)
         create_batch_job(board, testplan, alltests)
     sys.exit(0)
 
-- 
2.7.4

Re: [Fuego] [PATCH 13/16] ftc: remove testplans and use testspecs

[Bird, Timothy]

OK.

Acked-by: Tim Bird
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 13/16] ftc: remove testplans and use testspecs
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc | 36 +++++++++++++++++++++---------------
>  1 file changed, 21 insertions(+), 15 deletions(-)
> 
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 82a8c18..372c32a 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -85,7 +85,7 @@ TARGET_ENV_VAR="FTC_TARGET"
>  # format for command_help mapping with: key=name, value=(summary,
> long description)
>  command_help = {
>  "add-jobs":("Adds jobs to Jenkins.",
> -    """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase>] [-d
> <foo.dist>]
> +    """Usage: ftc add-jobs -b <target> [-p <testplan> | -t <testcase> -s
> <testspec>]
>    Example: sudo -u jenkins ftc add-jobs -b docker -p testplan_docker
>    Example: sudo -u jenkins ftc add-jobs -b docker -t Benchmark.Dhrystone
>    Use list-plans to see the available test plans.
> @@ -93,13 +93,13 @@ command_help = {
>    This interface may change in the future."""),
> 
>  "rm-jobs":("Removes jobs from Jenkins.",
> -    """Usage: ftc rm-jobs <target>.<testplan>.<testcase>
> +    """Usage: ftc rm-jobs <target>.<testspec>.<testtype>.<testcase>
>    Use list-jobs to see the existing jobs. A wildcard can be used to
>    specify which jobs to remove (just make sure you have 4 words):
>      Example: sudo -u jenkins ftc rm-jobs docker.testplan_docker.*.*
>      Example: sudo -u jenkins ftc rm-jobs docker.*.F*.*stress
> -  Multiple combinations of the <target>.<testplan>.<testcase> pattern
> -  can be passed as well.
> +  Multiple combinations of the <target>.<testspec>.<testtype>.<testcase>
> +  pattern can be passed as well.
> 
>    If no option is provided all existing jobs will be removed."""),
> 
> @@ -741,7 +741,7 @@ def get_includes(include_filename, conf):
>      inc_vars = parse_shell_file(inc_path, conf)
>      return inc_vars
> 
> -def create_job(board, testplan, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks):
> +def create_job(board, testspec, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks):
>      # flot only necessary for Benchmarks
>      if str(test).split('.')[0] == 'Benchmark':
>          flot_link = '<flotile.FlotPublisher plugin="flot@1.0-SNAPSHOT"/>'
> @@ -781,7 +781,7 @@ export Target_PreCleanup={precleanup}
>  export Target_PostCleanup={postcleanup}
>  export TESTDIR={testdir}
>  export TESTNAME={testname}
> -export TESTPLAN="{testplan}"
> +export TESTSPEC="{testspec}"
>  timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{TESTNAME}}.sh
>  </command>
>      </hudson.tasks.Shell>
> @@ -800,13 +800,13 @@ timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
>  </project>
>  """.format(board=board, reboot=str(reboot), rebuild=str(rebuild),
> precleanup=str(precleanup), postcleanup=str(postcleanup),
>          testdir=str(test), testname=str(test).split('.')[1],
> -        testplan=testplan, timeout=timeout, flot_link=flot_link,
> html_links=html_links, failed_links=failed_links))
> +        testspec=testspec, timeout=timeout, flot_link=flot_link,
> html_links=html_links, failed_links=failed_links))
>      fd.close()
> 
>      print("Creating job " + test)
>      try:
>          subprocess.call('java -jar /var/cache/jenkins/war/WEB-INF/jenkins-
> cli.jar -s http://localhost:8080/fuego create-job ' +
> -            board + '.' + testplan + '.' + str(test) + ' < ' + tmp, shell=True)
> +            board + '.' + testspec + '.' + str(test) + ' < ' + tmp, shell=True)
>          subprocess.call('rm -f ' + tmp, shell=True)
>      except Exception as e:
>          print("Job already exists")
> @@ -862,6 +862,7 @@ def create_batch_job(board, testplan, alltests):
>  def parse_testplan(testplan):
>      abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
>      alltests = []
> +    alltestspecs = []
>      alltimeouts = []
>      allextralinks = []
>      allreboots = []
> @@ -873,6 +874,7 @@ def parse_testplan(testplan):
>          plan = json.load(f)
>          for test in plan['tests']:
>              alltests.append(test['testName'])
> +            alltestspecs.append(test['spec'])
>              if 'timeout' in test:
>                  alltimeouts.append(test['timeout'])
>              else:
> @@ -897,7 +899,7 @@ def parse_testplan(testplan):
>                  allextralinks.append(test['extralinks'])
>              else:
>                  allextralinks.append('None')
> -    return alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks
> +    return alltests, alltestspecs, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks
> 
>  def do_add_jobs(conf, options):
>      if '-b' in options:
> @@ -925,8 +927,12 @@ def do_add_jobs(conf, options):
>              raise Exception('Test %s not found.' % test)
>          options.remove('-t')
>          options.remove(test)
> -        # TODO: use spec instead of testplan_default
> -        testplan = 'testplan_default'
> +        if '-s' in options:
> +            spec = options[options.index('-s') + 1]
> +            options.remove('-s')
> +            options.remove(spec)
> +        else:
> +            spec = 'default'
>      else:
>          raise Exception('No testplan or testcase supplied.')
> 
> @@ -937,11 +943,11 @@ def do_add_jobs(conf, options):
>          precleanup = 'true'
>          postcleanup = 'true'
>          jobextralinks = 'None'
> -        create_job(board, testplan, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks)
> +        create_job(board, spec, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks)
>      else:
> -        alltests, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks = parse_testplan(testplan)
> -        for test, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks in zip(alltests, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks):
> -            create_job(board, testplan, test, timeout, reboot, rebuild,
> precleanup, postcleanup, jobextralinks)
> +        alltests, alltestspecs, alltimeouts, allreboots, allrebuilds, allprecleanups,
> allpostcleanups, allextralinks = parse_testplan(testplan)
> +        for test, spec, timeout, reboot, rebuild, precleanup, postcleanup,
> jobextralinks in zip(alltests, alltestspecs, alltimeouts, allreboots, allrebuilds,
> allprecleanups, allpostcleanups, allextralinks):
> +            create_job(board, spec, test, timeout, reboot, rebuild, precleanup,
> postcleanup, jobextralinks)
>          create_batch_job(board, testplan, alltests)
>      sys.exit(0)
> 
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

[Fuego] [PATCH 14/16] ftc test: remove distrib and update paths

[Daniel Sangorrin]

Note: this is untested and probably won't work but
it's closer to working :)

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/ftc-unit-test.sh | 21 ++-------------------
 1 file changed, 2 insertions(+), 19 deletions(-)

diff --git a/engine/scripts/ftc-unit-test.sh b/engine/scripts/ftc-unit-test.sh
index b7d910a..911888e 100755
--- a/engine/scripts/ftc-unit-test.sh
+++ b/engine/scripts/ftc-unit-test.sh
@@ -2,7 +2,7 @@
 #
 # this script performs a unit test of ftc
 #
-TEST_BOARD_FILE=/userdata/conf/boards/ftc-test.board
+TEST_BOARD_FILE=/fuego-ro/boards/ftc-test.board
 PATCH_FILE=add-ftc-test-node.patch
 
 #set -x
@@ -100,26 +100,9 @@ cat <<'EOFP' > $PATCH_FILE
 +      <mode>NORMAL</mode>
 +      <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
 +      <launcher class="hudson.slaves.CommandLauncher">
-+        <agentCommand>java -jar /home/jenkins/slave.jar</agentCommand>
++        <agentCommand>java -jar /fuego-core/engine/slave.jar</agentCommand>
 +      </launcher>
 +      <label></label>
-+      <nodeProperties>
-+        <hudson.slaves.EnvironmentVariablesNodeProperty>
-+          <envVars serialization="custom">
-+            <unserializable-parents/>
-+            <tree-map>
-+              <default>
-+                <comparator class="hudson.util.CaseInsensitiveComparator"/>
-+              </default>
-+              <int>2</int>
-+              <string>BOARD_OVERLAY</string>
-+              <string>boards/ftc-test.board</string>
-+              <string>DISTRIB</string>
-+              <string>distribs/nologger.dist</string>
-+            </tree-map>
-+          </envVars>
-+        </hudson.slaves.EnvironmentVariablesNodeProperty>
-+      </nodeProperties>
 +      <userId>anonymous</userId>
 +    </slave>
    </slaves>
-- 
2.7.4

Re: [Fuego] [PATCH 14/16] ftc test: remove distrib and update paths

[Bird, Timothy]

Thanks. I applied the patch.

The whole approach of ftc-unit-test.sh needs to be revisited.
Jenkins nodes are not just entries in Jenkins' config.xml, and since
the board directory is read-only, a different approach will be needed
to create the ftc-test board.

Do you think I should just put it there, and leave it there, for testing
purposes?  It won't show up as a board unless a Fuego user does an
add-node with it, and then it will be available to do things like:
add-node, rm-node, get, set and other tests with.

Let me know what you think.

For now,
Acked-by: Tim Bird
 -- Tim
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 14/16] ftc test: remove distrib and update paths
> 
> Note: this is untested and probably won't work but
> it's closer to working :)
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/ftc-unit-test.sh | 21 ++-------------------
>  1 file changed, 2 insertions(+), 19 deletions(-)
> 
> diff --git a/engine/scripts/ftc-unit-test.sh b/engine/scripts/ftc-unit-test.sh
> index b7d910a..911888e 100755
> --- a/engine/scripts/ftc-unit-test.sh
> +++ b/engine/scripts/ftc-unit-test.sh
> @@ -2,7 +2,7 @@
>  #
>  # this script performs a unit test of ftc
>  #
> -TEST_BOARD_FILE=/userdata/conf/boards/ftc-test.board
> +TEST_BOARD_FILE=/fuego-ro/boards/ftc-test.board
>  PATCH_FILE=add-ftc-test-node.patch
> 
>  #set -x
> @@ -100,26 +100,9 @@ cat <<'EOFP' > $PATCH_FILE
>  +      <mode>NORMAL</mode>
>  +      <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
>  +      <launcher class="hudson.slaves.CommandLauncher">
> -+        <agentCommand>java -jar
> /home/jenkins/slave.jar</agentCommand>
> ++        <agentCommand>java -jar /fuego-
> core/engine/slave.jar</agentCommand>
>  +      </launcher>
>  +      <label></label>
> -+      <nodeProperties>
> -+        <hudson.slaves.EnvironmentVariablesNodeProperty>
> -+          <envVars serialization="custom">
> -+            <unserializable-parents/>
> -+            <tree-map>
> -+              <default>
> -+                <comparator class="hudson.util.CaseInsensitiveComparator"/>
> -+              </default>
> -+              <int>2</int>
> -+              <string>BOARD_OVERLAY</string>
> -+              <string>boards/ftc-test.board</string>
> -+              <string>DISTRIB</string>
> -+              <string>distribs/nologger.dist</string>
> -+            </tree-map>
> -+          </envVars>
> -+        </hudson.slaves.EnvironmentVariablesNodeProperty>
> -+      </nodeProperties>
>  +      <userId>anonymous</userId>
>  +    </slave>
>     </slaves>
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

[Fuego] [PATCH 15/16] fail_check_cases should not abort the job

[Daniel Sangorrin]

failure is something normal and should be kept in a variable
to return it at the end to jenkins. If we just abort then
post_test will not have a chance to return $FUEGO_RESULT

TODO: probably we should change $FUEGO_RESULT though

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 2483f61..4379271 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -331,7 +331,6 @@ function bench_processing {
 }
 
 # search in test log for {!JOB_NAME}_FAIL_PATTERN_n fail cases and abort with message {!JOB_NAME}_FAIL_MESSAGE_n if found
-# args: $1 - path to test log
 function fail_check_cases () {
     testlog="${LOGDIR}/testlog.txt"
     slog_prefix="${LOGDIR}/syslog"
@@ -364,8 +363,7 @@ function fail_check_cases () {
 
             if diff -ua ${slog_prefix}.before ${slog_prefix}.after | grep -vEf "$FUEGO_CORE/engine/scripts/syslog.ignore" | grep -E -e $fptemplate;
             then
-                echo "Located failing message in syslog diff"
-                abort_job "Detected fail message in syslog diff: $fpmessage"
+                echo "Detected fail message in syslog diff: $fpmessage"
             else
                 continue
             fi
@@ -373,8 +371,7 @@ function fail_check_cases () {
 
         if grep -e "$fptemplate" $testlog ;
         then
-            echo "Located failing message in $1"
-            abort_job "Detected fail message: $fpmessage"
+            echo "Detected fail message in $testlog: $fpmessage"
         fi
     done
 }
@@ -444,8 +441,10 @@ function post_test {
   ov_logger "Test $1 is finished"
 
 # Syslog comparison
+  # TODO: should affect FUEGO_RESULT
   syslog_cmp
 
+  # TODO: should affect FUEGO_RESULT
   fail_check_cases  || true
 
 # create functional result file
-- 
2.7.4

Re: [Fuego] [PATCH 15/16] fail_check_cases should not abort the job

[Bird, Timothy]

OK.  Thanks.

I agree that the error should be reflected into FUEGO_RESULT somehow.

This is a bit of a weird test, looking in the syslog for oopses and such.  This is
probably a nice safety mechanism for kernel and driver testing, to detect
when some operation caused an oops, so it's worth keeping them around.

Acked-by: Tim Bird
 -- Tim


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 15/16] fail_check_cases should not abort the job
> 
> failure is something normal and should be kept in a variable
> to return it at the end to jenkins. If we just abort then
> post_test will not have a chance to return $FUEGO_RESULT
> 
> TODO: probably we should change $FUEGO_RESULT though
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 2483f61..4379271 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -331,7 +331,6 @@ function bench_processing {
>  }
> 
>  # search in test log for {!JOB_NAME}_FAIL_PATTERN_n fail cases and abort
> with message {!JOB_NAME}_FAIL_MESSAGE_n if found
> -# args: $1 - path to test log
>  function fail_check_cases () {
>      testlog="${LOGDIR}/testlog.txt"
>      slog_prefix="${LOGDIR}/syslog"
> @@ -364,8 +363,7 @@ function fail_check_cases () {
> 
>              if diff -ua ${slog_prefix}.before ${slog_prefix}.after | grep -vEf
> "$FUEGO_CORE/engine/scripts/syslog.ignore" | grep -E -e $fptemplate;
>              then
> -                echo "Located failing message in syslog diff"
> -                abort_job "Detected fail message in syslog diff: $fpmessage"
> +                echo "Detected fail message in syslog diff: $fpmessage"
>              else
>                  continue
>              fi
> @@ -373,8 +371,7 @@ function fail_check_cases () {
> 
>          if grep -e "$fptemplate" $testlog ;
>          then
> -            echo "Located failing message in $1"
> -            abort_job "Detected fail message: $fpmessage"
> +            echo "Detected fail message in $testlog: $fpmessage"
>          fi
>      done
>  }
> @@ -444,8 +441,10 @@ function post_test {
>    ov_logger "Test $1 is finished"
> 
>  # Syslog comparison
> +  # TODO: should affect FUEGO_RESULT
>    syslog_cmp
> 
> +  # TODO: should affect FUEGO_RESULT
>    fail_check_cases  || true
> 
>  # create functional result file
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

[Fuego] [PATCH 16/16] a bit of cleaning and style fixes

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh |  2 +-
 engine/scripts/ovgen.py     | 17 ++---------------
 2 files changed, 3 insertions(+), 16 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index 4379271..e499ab4 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -500,7 +500,7 @@ function log_compare {
 	  return $FUEGO_RESULT
   fi
 
-  cd ${LOGDIR} 
+  cd ${LOGDIR}
   LOGFILE="testlog.txt"
   PARSED_LOGFILE="testlog.${4}.txt"
 
diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
index 10d1cf4..4b25d86 100755
--- a/engine/scripts/ovgen.py
+++ b/engine/scripts/ovgen.py
@@ -37,7 +37,6 @@ OFVAR_DESCRIPTION="DESCRIPTION"
 LVAR_NAME="NAME"
 LVAR_DESCRIPTION="DESCRIPTION"
 
-
 class OFClass:
     def __init__(self):
         self.name=""
@@ -65,7 +64,6 @@ class OFLayer:
     def __str__(self):
         return "<OFLayer name:\"%s\" descr:\"%s\" vars:%s>\n" % (self.name, self.description, self.vars)
 
-
 class TestSpecs:
     def __init__(self):
         self.name=""
@@ -108,7 +106,6 @@ class SpecException(Exception):
      def __str__(self):
          return repr(self.value)
 
-
 def debug_print(string, lev=1):
     if lev <= log_lvl:
         print "log: " + string
@@ -130,7 +127,6 @@ def parseOFVars(line, ofc):
 
     return True
 
-
 # parse variables definitions
 def parseVars(line, ofc):
     m = re.search("(\w+)=\"(.*)\"", line)
@@ -164,7 +160,6 @@ def parseFunctionBody(name, f):
 
     return funcbody
 
-
 def parseFunction(line, f):
     m = re.search("\s?function (\w+).*{", line)
     if m==None:
@@ -209,7 +204,6 @@ def parseBaseFile(baseFilePath, ofc):
         elif baseParseFunction(line, f, ofc):
             debug_print("function found", 3)
 
-
 # parse all base files in dir
 def parseBaseDir(baseDirPath, ofcls):
     debug_print ("\n------------\nparsing " + baseDirPath + " fuegoclass dir ...\n")
@@ -221,13 +215,12 @@ def parseBaseDir(baseDirPath, ofcls):
         debug_print ("parsed %s class\n------------\n" % (ofc.name))
         ofcls[ofc.name]=(ofc)
 
-
 def parseInherit(line, ofcls):
     m = re.search("inherit \"(.+)\"", line)
-    if m==None: return None
+    if m == None:
+        return None
     clname = m.group(1)
 
-
     if clname not in ofcls:
         raise OFClassNotFoundException("No such class: %s" % (clname))
 
@@ -259,7 +252,6 @@ def parseLayerVarOverride(line, layer, inhclass):
 
     return True
 
-
 def parseLayerFuncOverride(line, layer, inhclass, f):
     m = re.search("override-func (\w+)(.*)", line)
     if m==None: return False
@@ -322,7 +314,6 @@ def parseLayerCapList(line, layer, inhclass):
 
     return True
 
-
 def parseOverrideFile(overrideFile, layer, ofcls):
     debug_print ("\n-----------\nparsing %s override ...\n" % (overrideFile))
     f = open(overrideFile)
@@ -472,7 +463,6 @@ def parseSpec(testdir, testspec):
 
 def run(test_args=None):
     parser = argparse.ArgumentParser(description='Read OF class files, override files and generate prolog defining all variables and functions')
-
     parser.add_argument('--classdir', help='OF base class directory', required=True)
     parser.add_argument('--ovfiles', nargs='+', metavar='OVFILE', help='list of directories containing .override files', required=True)
     parser.add_argument('--testdir', help='e.g.: Benchmark.Dhrystone', required=True)
@@ -483,11 +473,9 @@ def run(test_args=None):
     args = parser.parse_args(args=test_args)
 
     if args.debug:
-
         if args.debug < 1 or args.debug > 3:
             print "Error: wrong debug lvl: %s" % (args.debug)
             sys.exit
-
         global log_lvl
         log_lvl = args.debug
 
@@ -497,7 +485,6 @@ def run(test_args=None):
     parseBaseDir(args.classdir, ofcls)
 
     layers = {}
-
     classes = []
     for ovf in args.ovfiles:
         classes = classes + parseOverrideFile(ovf, layers, ofcls)
-- 
2.7.4

Re: [Fuego] [PATCH 16/16] a bit of cleaning and style fixes

[Bird, Timothy]

OK.

Acked-by: Tim Bird
 -- Tim

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Wednesday, March 29, 2017 6:05 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 16/16] a bit of cleaning and style fixes
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh |  2 +-
>  engine/scripts/ovgen.py     | 17 ++---------------
>  2 files changed, 3 insertions(+), 16 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index 4379271..e499ab4 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -500,7 +500,7 @@ function log_compare {
>  	  return $FUEGO_RESULT
>    fi
> 
> -  cd ${LOGDIR}
> +  cd ${LOGDIR}
>    LOGFILE="testlog.txt"
>    PARSED_LOGFILE="testlog.${4}.txt"
> 
> diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
> index 10d1cf4..4b25d86 100755
> --- a/engine/scripts/ovgen.py
> +++ b/engine/scripts/ovgen.py
> @@ -37,7 +37,6 @@ OFVAR_DESCRIPTION="DESCRIPTION"
>  LVAR_NAME="NAME"
>  LVAR_DESCRIPTION="DESCRIPTION"
> 
> -
>  class OFClass:
>      def __init__(self):
>          self.name=""
> @@ -65,7 +64,6 @@ class OFLayer:
>      def __str__(self):
>          return "<OFLayer name:\"%s\" descr:\"%s\" vars:%s>\n" % (self.name,
> self.description, self.vars)
> 
> -
>  class TestSpecs:
>      def __init__(self):
>          self.name=""
> @@ -108,7 +106,6 @@ class SpecException(Exception):
>       def __str__(self):
>           return repr(self.value)
> 
> -
>  def debug_print(string, lev=1):
>      if lev <= log_lvl:
>          print "log: " + string
> @@ -130,7 +127,6 @@ def parseOFVars(line, ofc):
> 
>      return True
> 
> -
>  # parse variables definitions
>  def parseVars(line, ofc):
>      m = re.search("(\w+)=\"(.*)\"", line)
> @@ -164,7 +160,6 @@ def parseFunctionBody(name, f):
> 
>      return funcbody
> 
> -
>  def parseFunction(line, f):
>      m = re.search("\s?function (\w+).*{", line)
>      if m==None:
> @@ -209,7 +204,6 @@ def parseBaseFile(baseFilePath, ofc):
>          elif baseParseFunction(line, f, ofc):
>              debug_print("function found", 3)
> 
> -
>  # parse all base files in dir
>  def parseBaseDir(baseDirPath, ofcls):
>      debug_print ("\n------------\nparsing " + baseDirPath + " fuegoclass dir
> ...\n")
> @@ -221,13 +215,12 @@ def parseBaseDir(baseDirPath, ofcls):
>          debug_print ("parsed %s class\n------------\n" % (ofc.name))
>          ofcls[ofc.name]=(ofc)
> 
> -
>  def parseInherit(line, ofcls):
>      m = re.search("inherit \"(.+)\"", line)
> -    if m==None: return None
> +    if m == None:
> +        return None
>      clname = m.group(1)
> 
> -
>      if clname not in ofcls:
>          raise OFClassNotFoundException("No such class: %s" % (clname))
> 
> @@ -259,7 +252,6 @@ def parseLayerVarOverride(line, layer, inhclass):
> 
>      return True
> 
> -
>  def parseLayerFuncOverride(line, layer, inhclass, f):
>      m = re.search("override-func (\w+)(.*)", line)
>      if m==None: return False
> @@ -322,7 +314,6 @@ def parseLayerCapList(line, layer, inhclass):
> 
>      return True
> 
> -
>  def parseOverrideFile(overrideFile, layer, ofcls):
>      debug_print ("\n-----------\nparsing %s override ...\n" % (overrideFile))
>      f = open(overrideFile)
> @@ -472,7 +463,6 @@ def parseSpec(testdir, testspec):
> 
>  def run(test_args=None):
>      parser = argparse.ArgumentParser(description='Read OF class files,
> override files and generate prolog defining all variables and functions')
> -
>      parser.add_argument('--classdir', help='OF base class directory',
> required=True)
>      parser.add_argument('--ovfiles', nargs='+', metavar='OVFILE', help='list of
> directories containing .override files', required=True)
>      parser.add_argument('--testdir', help='e.g.: Benchmark.Dhrystone',
> required=True)
> @@ -483,11 +473,9 @@ def run(test_args=None):
>      args = parser.parse_args(args=test_args)
> 
>      if args.debug:
> -
>          if args.debug < 1 or args.debug > 3:
>              print "Error: wrong debug lvl: %s" % (args.debug)
>              sys.exit
> -
>          global log_lvl
>          log_lvl = args.debug
> 
> @@ -497,7 +485,6 @@ def run(test_args=None):
>      parseBaseDir(args.classdir, ofcls)
> 
>      layers = {}
> -
>      classes = []
>      for ovf in args.ovfiles:
>          classes = classes + parseOverrideFile(ovf, layers, ofcls)
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego