[Fuego] rt-test related fixes

[Fuego] rt-test related fixes

[Daniel Sangorrin]

Hi Tim,

I have updated the cyclictest tarball to use the branch unstable/devel/v1.1.1
which seems to work fine on the Beablebone black (no negative latencies). Note that other branches
can be used by just specifying them on the spec file.

I also upgraded the version for other rt-tests that were in Fuego (except 
hackbench): signaltest and pi_tests. Note that signaltest will probably
cross the 1ms latency threshold and fail unless you change the kernel
configuration or patch it with real-time support.

There are actually more test cases inside rt-tests. At first, I thought about 
putting them all together. But then I thought that it would be nice to be
able to choose the background load for cyclictest from any of the tests 
currently available in Fuego (e.g. a kernel build). That would require 
support for running multiple jobs concurrently on the same board.

Regards
Daniel

[PATCH 1/9] cyclictest: update tarball version (not sent to mailing list)
[PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
[PATCH 3/9] cyclictest: update twothreads spec
[PATCH 4/9] functions: fix preference order when unpacking or cloning
[PATCH 5/9] cyclictest: convert tabs to spaces
[PATCH 6/9] cyclictest: add assert_define BENCHMARK_CYCLICTEST_PARAMS
[PATCH 7/9] cyclictest: add NEED_ROOT for now
[PATCH 8/9] signaltest: update to new framework
[PATCH 9/9] pi_tests: add patch for old toolchains

[Fuego] [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh b/engine/tests/Benchmark.cyclictest/fuego_test.sh
index a32eb6e..9576bab 100755
--- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
+++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
@@ -1,4 +1,4 @@
-tarball=rt-tests-1.0.tar.gz
+tarball=rt-tests-v1.1.1.tar.gz
 
 function test_build {
 	patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-kernels.patch
-- 
2.7.4

[Fuego] [PATCH 3/9] cyclictest: update twothreads spec

[Daniel Sangorrin]

This test is supposed to run 2 threads on the same core so we
define the affinity parameter. Also we specify the priority
of the threads and add the mlock flag.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.cyclictest/spec.json | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/engine/tests/Benchmark.cyclictest/spec.json b/engine/tests/Benchmark.cyclictest/spec.json
index a0a3a00..c963026 100644
--- a/engine/tests/Benchmark.cyclictest/spec.json
+++ b/engine/tests/Benchmark.cyclictest/spec.json
@@ -5,7 +5,9 @@
             "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q"
         },
         "twothreads": {
-            "PARAMS": "-t 2 -l 10000 -q"
+            "gitrepo": "https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git",
+            "gitref": "unstable/devel/v1.1.1",
+            "PARAMS": "-a 0 -m -t 2 -p 60 -l 10000 -q"
         }
     }
 }
-- 
2.7.4

[Fuego] [PATCH 4/9] functions: fix preference order when unpacking or cloning

[Daniel Sangorrin]

In the previous implementation the preference was
  spec tarball > fuego_test tarball > spec gitrepo > fuego_test gitrepo

This patch changes it to
  spec gitrepo > spec tarball > fuego_test gitrepo > fuego_test tarball

Therefore, spec.json gitrepo/tarball variables take precedence over
fuego_test.sh variables. And if, for some reason, gitrepo and
tarball variables are defined at the same time, then gitrepo will
take precedence. This case is not supposed to happen though.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/scripts/functions.sh | 137 ++++++++++++++++++++++++++------------------
 1 file changed, 82 insertions(+), 55 deletions(-)

diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
index da3415b..fe27b0c 100755
--- a/engine/scripts/functions.sh
+++ b/engine/scripts/functions.sh
@@ -38,72 +38,99 @@ trap signal_handler SIGTERM SIGHUP SIGALRM SIGINT EXIT
 # cause ERR trap setting to be visible inside functions
 set -o errtrace
 
-# Unpacks/clones the test source code into the current directory.
-#
-# Tarball or git variables can be in the spec or directly in fuego_test.sh
-# Variables in the spec have preference.
-#
-# Tarball variables
-#   - tarball: tarball file name (e.g. bc-script.tar.gz)
-# Git variables
-#   - gitrepo: git repository (e.g. https://github.com/torvalds/linux.git)
-#   - gitref: git branch, tag or commit id (optional, default: master)
-function unpack {
-    # Tarballs
-    upName=`echo "${TESTDIR^^}"| tr '.' '_'`
-    spec_tarball="${upName}_TARBALL"
-
-    if [ ! -z "${!spec_tarball}" ]; then
-        tarball=${!spec_tarball}
+# Clones a git repository into the current folder
+# $1 (gitrepo): git repository URL
+# $2 (gitref): branch, tag or commit id (optional, default: master)
+# FIXTHIS: add commit id information to the json output
+function git_clone {
+    local gitrepo=${1}
+    local gitref=${2}
+
+    is_empty "$gitrepo"
+
+    if [ -z "$gitref" ]; then
+        gitref="master"
     fi
 
-    if [ ! -z ${tarball+x} ]; then
-        echo "Unpacking $tarball"
-        case ${tarball/*./} in
-            gz|tgz) key=z ;;
-            bz2) key=j ;;
-            tar) key= ;;
-            *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
-        esac
-        tar ${key}xf $TEST_HOME/$tarball --strip-components=1
-        return
+    local is_namedref=$(git ls-remote $gitrepo $gitref)
+
+    if [ "$is_namedref" = "" ]; then
+        echo "Clone repository $gitrepo."
+        git clone -n $gitrepo fuego_git_repo
+    else
+        echo "Clone repository $gitrepo."
+        git clone -n --depth=1 --branch=$gitref $gitrepo fuego_git_repo
     fi
 
-    # GIT repositories
-    spec_gitrepo="${upName}_GITREPO"
-    spec_gitref="${upName}_GITREF"
+    # equivalent to tarball's --strip-components=1
+    mv fuego_git_repo/.git .git
+    rm -rf fuego_git_repo
 
-    if [ ! -z "${!spec_gitrepo}" ]; then
-        gitrepo=${!spec_gitrepo}
-    fi
+    echo "Checkout branch/tag/commit id $gitref."
+    git checkout $gitref
+}
 
-    if [ ! -z "${!spec_gitref}" ]; then
-        gitref=${!spec_gitref}
-    fi
+# Untars a tarball in the current folder
+# $1 (tarball): file to untar
+function untar {
+    local tarball=${1}
 
-    # FIXTHIS: add commit id information to the json output
-    if [ ! -z ${gitrepo+x} ]; then
-        if [ -z ${gitref+x} ]; then
-            gitref="master"
-        fi
+    is_empty "$tarball"
 
-        is_namedref=$(git ls-remote $gitrepo $gitref)
+    echo "Unpacking $tarball"
+    case ${tarball/*./} in
+        gz|tgz) key=z ;;
+        bz2) key=j ;;
+        tar) key= ;;
+        *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
+    esac
+    tar ${key}xf $TEST_HOME/$tarball --strip-components=1
+}
 
-        if [ "$is_namedref" = "" ]; then
-            echo "Clone repository $gitrepo."
-            git clone -n $gitrepo fuego_git_repo
+# Unpacks/clones the test source code into the current directory.
+#
+# The following tarball and git variables can be specified in the test's
+# spec.json or fuego_test.sh.
+#   - Tarball variables
+#     - tarball: tarball file name (e.g. bc-script.tar.gz)
+#   - Git variables
+#     - gitrepo: git repository (e.g. https://github.com/torvalds/linux.git)
+#     - gitref: git branch, tag or commit id (optional, default: master)
+#
+# tarball and git variables follow the next preference rule:
+#   spec gitrepo > spec tarball > fuego_test gitrepo > fuego_test tarball
+function unpack {
+    # prepare variables
+    upName=`echo "${TESTDIR^^}"| tr '.' '_'`
+    spec_gitrepo="${upName}_GITREPO"
+    spec_gitref="${upName}_GITREF"
+    spec_tarball="${upName}_TARBALL"
+    # 1) spec gitrepo
+    if [ ! -z "${!spec_gitrepo}" ]; then
+        gitrepo=${!spec_gitrepo}
+        if [ ! -z "${!spec_gitref}" ]; then
+            gitref=${!spec_gitref}
         else
-            echo "Clone repository $gitrepo."
-            git clone -n --depth=1 --branch=$gitref $gitrepo fuego_git_repo
+            # gitref could have been defined by fuego_test.sh
+            gitref="master"
         fi
-
-        # equivalent to tarball's --strip-components=1
-        mv fuego_git_repo/.git .git
-        rm -rf fuego_git_repo
-
-        echo "Checkout branch/tag/commit id $gitref."
-        git checkout $gitref
-
+        git_clone "$gitrepo" "$gitref"
+        return
+    fi
+    # 2) spec tarball
+    if [ ! -z "${!spec_tarball}" ]; then
+        tarball=${!spec_tarball}
+        untar "$tarball"
+        return
+    fi
+    # 3) fuego_test gitrepo
+    if [ ! -z "$gitrepo" ]; then
+        git_clone "$gitrepo" "$gitref"
+        return
+    fi
+    # 4) fuego_test tarball
+    if [ ! -z "$tarball" ]; then
+        untar "$tarball"
         return
     fi
 
-- 
2.7.4

[Fuego] [PATCH 5/9] cyclictest: convert tabs to spaces

[Daniel Sangorrin]

TODO: we should define clearly what style to use for shell
scripts and apply it uniformly to all the of them.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.cyclictest/fuego_test.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh b/engine/tests/Benchmark.cyclictest/fuego_test.sh
index 9576bab..eec07bb 100755
--- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
+++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
@@ -1,16 +1,16 @@
 tarball=rt-tests-v1.1.1.tar.gz
 
 function test_build {
-	patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-kernels.patch
-	make NUMA=0 cyclictest
+    patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-kernels.patch
+    make NUMA=0 cyclictest
 }
 
 function test_deploy {
-	put cyclictest  $BOARD_TESTDIR/fuego.$TESTDIR/
+    put cyclictest  $BOARD_TESTDIR/fuego.$TESTDIR/
 }
 
 function test_run {
-	report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest $BENCHMARK_CYCLICTEST_PARAMS"
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest $BENCHMARK_CYCLICTEST_PARAMS"
 }
 
 
-- 
2.7.4

[Fuego] [PATCH 6/9] cyclictest: add assert_define BENCHMARK_CYCLICTEST_PARAMS

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh b/engine/tests/Benchmark.cyclictest/fuego_test.sh
index eec07bb..5f6584b 100755
--- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
+++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
@@ -10,6 +10,8 @@ function test_deploy {
 }
 
 function test_run {
+    assert_define BENCHMARK_CYCLICTEST_PARAMS
+
     report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest $BENCHMARK_CYCLICTEST_PARAMS"
 }
 
-- 
2.7.4

[Fuego] [PATCH 7/9] cyclictest: add NEED_ROOT for now

[Daniel Sangorrin]

cyclitest does not necessarily need root. You can
add the corresponding 'rtprio' settings to
/etc/security/limits.conf for your user.
However, we have no clean way to check that at the moment.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh b/engine/tests/Benchmark.cyclictest/fuego_test.sh
index 5f6584b..48c0173 100755
--- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
+++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
@@ -1,5 +1,7 @@
 tarball=rt-tests-v1.1.1.tar.gz
 
+NEED_ROOT=1
+
 function test_build {
     patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-kernels.patch
     make NUMA=0 cyclictest
-- 
2.7.4

[Fuego] [PATCH 8/9] signaltest: update to new framework

[Daniel Sangorrin]

signaltest, like cyclictest, one testcase inside the rt-tests
package. The output format is the same as cyclictest so I just
copied the parser and criteria files.

TODO: there are more rt-tests (e.g. Functional.pi_tests) inside
Fuego that need to be updated.

TODO: some of the rt-tests are supposed to run in parallel.
For example, cyclictest may run with hackbench in background.
Unfortunately, Fuego currently executes tests on the board
in a sequential manner. We need to make sure that Fuego can
execute multiple jobs on the same board in parallel or at least
provide functionality to "disturb" a test with some background
load.

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 .../tests/Benchmark.signaltest/chart_config.json   |   4 ++-
 engine/tests/Benchmark.signaltest/criteria.json    |  26 +++++++++++++++
 engine/tests/Benchmark.signaltest/fuego_test.sh    |  12 ++++---
 engine/tests/Benchmark.signaltest/parser.py        |  35 ++++++++++-----------
 engine/tests/Benchmark.signaltest/reference.log    |  17 ----------
 .../tests/Benchmark.signaltest/signaltest.tar.gz   | Bin 3710 -> 0 bytes
 engine/tests/Benchmark.signaltest/spec.json        |   3 +-
 7 files changed, 53 insertions(+), 44 deletions(-)
 create mode 100644 engine/tests/Benchmark.signaltest/criteria.json
 delete mode 100644 engine/tests/Benchmark.signaltest/reference.log
 delete mode 100644 engine/tests/Benchmark.signaltest/signaltest.tar.gz

diff --git a/engine/tests/Benchmark.signaltest/chart_config.json b/engine/tests/Benchmark.signaltest/chart_config.json
index f8dcb3b..cdaf6a2 100644
--- a/engine/tests/Benchmark.signaltest/chart_config.json
+++ b/engine/tests/Benchmark.signaltest/chart_config.json
@@ -1,3 +1,5 @@
 {
-        "signaltest":["Thread0", "Thread1"]
+	"chart_type": "measure_plot",
+	"measures": ["default.latencies.max_latency",
+        "default.latencies.avg_latency"]
 }
diff --git a/engine/tests/Benchmark.signaltest/criteria.json b/engine/tests/Benchmark.signaltest/criteria.json
new file mode 100644
index 0000000..f137d55
--- /dev/null
+++ b/engine/tests/Benchmark.signaltest/criteria.json
@@ -0,0 +1,26 @@
+{
+    "schema_version":"1.0",
+    "criteria":[
+        {
+            "tguid":"default.latencies.max_latency",
+            "reference":{
+                "value":1000,
+                "operator":"le"
+            }
+        },
+        {
+            "tguid":"default.latencies.min_latency",
+            "reference":{
+                "value":1000,
+                "operator":"le"
+            }
+        },
+        {
+            "tguid":"default.latencies.avg_latency",
+            "reference":{
+                "value":1000,
+                "operator":"le"
+            }
+        }
+    ]
+}
diff --git a/engine/tests/Benchmark.signaltest/fuego_test.sh b/engine/tests/Benchmark.signaltest/fuego_test.sh
index af62c0c..84b5063 100755
--- a/engine/tests/Benchmark.signaltest/fuego_test.sh
+++ b/engine/tests/Benchmark.signaltest/fuego_test.sh
@@ -1,19 +1,21 @@
-tarball=signaltest.tar.gz
+gitrepo="https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git"
+gitref="unstable/devel/v1.1.1"
 
 NEED_ROOT=1
 
 function test_build {
-  make CC="$CC" LD="$LD" LDFLAGS="$LDFLAGS" CFLAGS="$CFLAGS"
+    patch -p1 -N -s < $TEST_HOME/../Benchmark.cyclictest/0001-Add-scheduling-policies-for-old-kernels.patch
+    make NUMA=0 signaltest
 }
 
 function test_deploy {
-	put signaltest  $BOARD_TESTDIR/fuego.$TESTDIR/
+    put signaltest  $BOARD_TESTDIR/fuego.$TESTDIR/
 }
 
 function test_run {
-	assert_define BENCHMARK_SIGNALTEST_LOOPS
+    assert_define BENCHMARK_SIGNALTEST_PARAMS
 
-	report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./signaltest  -l $BENCHMARK_SIGNALTEST_LOOPS -q"  
+    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./signaltest $BENCHMARK_SIGNALTEST_PARAMS"
 }
 
 
diff --git a/engine/tests/Benchmark.signaltest/parser.py b/engine/tests/Benchmark.signaltest/parser.py
index 44425ea..3f4f05d 100755
--- a/engine/tests/Benchmark.signaltest/parser.py
+++ b/engine/tests/Benchmark.signaltest/parser.py
@@ -1,26 +1,23 @@
 #!/usr/bin/python
-# See common.py for description of command-line arguments
-
 import os, re, sys
 sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
 import common as plib
 
-ref_section_pat = "^\[[\w\d_ .]+.[gle]{2}\]"
-cur_search_pat = re.compile("^T:([\s\d]+)(.*)P:(.*)C:(.*)Min:([\s\d]+)Act:([\s\d]+)Avg:([\s\d]+)Max:([\s\d]+)",re.MULTILINE)
-
-res_dict = {}
-cur_dict = {}
-
-pat_result = plib.parse(cur_search_pat)
-if pat_result:
-	cur_dict["Thread0.Min"] = '%d' % int(pat_result[0][4])
-	cur_dict["Thread0.Act"] = '%d' % int(pat_result[0][5])
-	cur_dict["Thread0.Avg"] = '%d' % int(pat_result[0][6])
-	cur_dict["Thread0.Max"] = '%d' % int(pat_result[0][7])
-	cur_dict["Thread1.Min"] = '%d' % int(pat_result[1][4])
-	cur_dict["Thread1.Act"] = '%d' % int(pat_result[1][5])
-	cur_dict["Thread1.Avg"] = '%d' % int(pat_result[1][6])
-	cur_dict["Thread1.Max"] = '%d' % int(pat_result[1][7])
+regex_string = "^T:.*Min:\s+(\d+).*Avg:\s+(\d+) Max:\s+(\d+)"
+measurements = {}
+matches = plib.parse_log(regex_string)
 
+if matches:
+	min_latencies = []
+	avg_latencies = []
+	max_latencies = []
+	for thread in matches:
+		min_latencies.append(float(thread[0]))
+		avg_latencies.append(float(thread[1]))
+		max_latencies.append(float(thread[2]))
+	measurements['default.latencies'] = [
+		{"name": "max_latency", "measure" : max(max_latencies)},
+		{"name": "min_latency", "measure" : min(min_latencies)},
+		{"name": "avg_latency", "measure" : sum(avg_latencies)/len(avg_latencies)}]
 
-sys.exit(plib.process_data(ref_section_pat, cur_dict, 'm', 'usec'))
+sys.exit(plib.process(measurements))
diff --git a/engine/tests/Benchmark.signaltest/reference.log b/engine/tests/Benchmark.signaltest/reference.log
deleted file mode 100644
index 83a959d..0000000
--- a/engine/tests/Benchmark.signaltest/reference.log
+++ /dev/null
@@ -1,17 +0,0 @@
-#sdfdsf
-[Thread0.Min|le]
-10000000
-[Thread0.Act|le]
-10000000
-[Thread0.Avg|le]
-10000000
-[Thread0.Max|le]
-10000000
-[Thread1.Min|le]
-10000000
-[Thread1.Act|le]
-10000000
-[Thread1.Avg|le]
-10000000
-[Thread1.Max|le]
-10000000
diff --git a/engine/tests/Benchmark.signaltest/signaltest.tar.gz b/engine/tests/Benchmark.signaltest/signaltest.tar.gz
deleted file mode 100644
index 2c15b87f03a88b2d2818994223be08837d89e343..0000000000000000000000000000000000000000
GIT binary patch
literal 0
HcmV?d00001

literal 3710
zcmV-^4uSC>iwFqm`FK(Q19NF-ZeeV6Wpi{cbYXG;?HX%y+cq*^rC+gj&o1Riwj{?+
zDo4(xu9JGskBsf~rsw83GzCelDN-RpKic|!`|U0O5|m^onQ41-cf@pJaIsh{7W-Dy
zU=(?w6sg>J@<)v}j^|rj^tbbTt8M=^<4LF6-Q4bMZfz30v$_3z>j`ZAVH{Z_OQn|p
zJn_Rg8;pfFcmVHH>;I)AE#CjgyAs176u<i!^uXKCo;~>f-RGO;{kz?*O?>~Y&CTs6
z(Ei;ySup<3-+ylpUiING&fe+C*_pd{a`f)taKAC~{rdUq)3^KQ(1$`r>TeETzdb`x
z>+fC|LhGbkKRo!B0pV1R6O<0l5?Qa~a?jeh?$>L0#$E$YVj4L4C$wVxRL9T)k!sZI
zeki=CSFa@#Xbmldwe=X*2VyFs0nXR{RsT<m?tfxf_HRGe?|R1L_Kzmtc|K$R+uiQ7
z%Krb1v9VT%H8?%j#tKQCMFW`xQy@ivG#<(uFA<1Dcn(@{7lS#RkK>7#!rM>;w~<KT
zr5uH~e+`2ulSv!}w`i!lde8YdNMV}9qr{s4PKSvQIo%z&i!<=O2of;}Qat!FlL7(>
zUNqQ<6CyB)2f^@;V8|MtCC37pOC~AAL;iSs{2ty4WCQtsmtBT|4~Ky-qEx^SA_?L&
zA~Ib-hNrZgj>P~j?|APU;^s_q18?Z|o($q>M{&PVuYWbbs2Cxs{nN99lVkVn{Pf`X
zEiC^4?e)!NTn{2Y%mxBp4*f`m>*HMuG!inN$}&uQ$}&)ie+DK8VQ@JI3*uQ&5=5gi
zOa>EChGY?nmk+go^kBCe3zyQnbYq(Mvlk*TT8x~B%DbsZ7l1V<xg1}gp1%IkJv;cv
zzH{3ECk=iP@sOq)py&8;(f-iDSg^4M??>TX6HqH)9S6OLA%wD(!N6$%9^+%caa{NK
z)K$<%ao+p0{XO^H{;7+9YqgGYI{E7HqE$kmvyad@z@>m`e1m$2+SY|7n}Z)-AAbA(
z4c`U#Zg0ML#@o(hlKB#33^MVebclPW9%{cptxXwB8mz)4QcE>D16;<0U+c9Xk`N4b
z)MScktfxv-@FR<8imDvO(Fi8qt?S=m@Jw;3Op@LU*680<)40rrlk{q*e!pWgJd6_{
zEYya;Pv==<I={{yoFBRfO#5(2aTrAT)VnoP&zCqI#AuiTyz7yHuU&g##&os4xQ351
zD9(X&G4(+%U5Xq=Qly*m-F#pnjx#xkZ;&Aih_B^~FqojH=|j7;G~~58a2yV-g<>Tn
z?YNGE0mw9plZj&liN=slg%4{oZPZbeH)Bk=!I5ceS6;j5v3%I=mz1(W60g|^tun$+
zv5c!vpW^p@{eG4i^(J2Eqc0uKu_HT8n1gkhL*7W6=#}sfUe&fJ3JMO2qJcEl6${V;
z4e4q~!|Z+9FJnjZ=ma8PK&FV>5+SoB(hS^_wvsQ`!AiZh@zgBuHJ!!qilabmQhb?3
zrbf$nDgG!-C-pH%XbhfDX)PF~Lb_~&II#@P2<=j!5s4;vL$vir3{f4}+O>^x8e`<T
zGb?2A@CwfMzTf}OeRJ^U1bX`8<ox^nQ{xOUyiC#LSpOI@43d<{cTgW|!4P#T*J`Co
z=q;s~ILpEx$IyXYKsO)z8|166L=uT`gLk;-QC)wUrOWhspS_Esdy+*FMx+HB+YfKV
z`$p3Br0XYMI{valvs5H5M#0fYU>dPtZ|gHRyFv2t%h<ja$z`01&)qad@l2+uLZ}c6
zZ@F;S4OZsa)I}}(0<oH-C@_Sc6p??YwvtjPf7ZaS&k*v&$FxYgBJ$8jvv|MYZLD4B
zm&hVKiOwn@oi!1h2%<?5yLV_;&Z@GnhB7&CFtC74Z8&V^8iP*8bo9zOJ9z7Udw8<<
zw<hH1$L|jh`Gkp>C^~G;O=vew!l57yrpD8CZT@J3Ngzu$X>ic5NuFe(NF1c4%$E*U
z)OL!?+ChmN^kTy#m<cHSGAy16GC1(=@C4gy{}Q85Uu$&uK}DDf!sm32Sm_vHW6|PY
zxtfl%N0M=mPL5B`PmT}vkX7Zbm{lE#mP};S&SJ*DMez-tXa@z8l%%W{bP4>p@dE3`
zc+!BE&{p?~F$an>$s0b!B70J$a_IYOHT3=mLn;Qal%PCH`8y?Ie9%ycsq(%c&^GU&
zv#mC9VFOgzr0WdX(A+NZ)1p4EuELU|UQ8t~unO(l_L~MXoiS;LGepA<Rvi-jEJ;M9
zWTdEm>Pjd?em9fFTm_iCYE_BNO;DBJpPhE{qt33VCd)6(92S>gpn9^Ji2xK4YBlx5
z_^F?+=Esr5v|rE%>4=Hyd2+zgYeKFY^EBv0-t`g`+vt~+FEL+;o2!RG-ZjYHt&Qy6
zTF459X&|QxDl2U0Rr6jZ=HYu;85@3?V=TkcBGFqoIa*+7vzFH8TjzFF7?xfzHlpa=
z7kvA2?k-A|l-2XetZpg+x5S{b-HRFZvv5D;lx;di)=vVM1@-%26XfGAQN6B|dq5NH
z)k(L~n?U+ws!S+Z#l(vA>LE|qe$WG2>B%SEAoUD6MbQZ!d1+4Ys2+Joxd3FTHxih?
z4v8nK$yG(dSuH1agCL!T-km!Z;gnN7&ha>r7&^;EI}29QpQ7dFOa)I*X>i}uQ9VSe
z*FX(>`%5h0@%VK2r>KO|y5v+Cpw(ijvO@J4lv0<Hfl3>}k0%okVK-wDfey#0s)Sso
zIfRGCFpGQ^gQDm#DdYhiAu3*QEgEG?Asn8ZygNhg!Z@C$ef32n$|jeppyibwU_I+)
zq3pLE<Qp|5ogp=acc%v@bWu`-zWU&iD+Ur92ct2`i&Izi+GR2y^{mzUm<8xMi@^^8
zaTMNx2Wf_~Lf4H~6}N#b(~zhtM@&QOQ(t}Cx6&)cy}D%@*Et_LearbklbB`_lr<6q
zDH?h<TN;<BJ@{S#N^^S<_59v7M16@l3UUd4pED}B?(;#Y<EZRNlvI*#?2w<iFBPmt
z&ZY7bX}S8YXiweJn%xhu!1}@^7(d32w!GNK=QrxgrK8sVS?ilBRM#w15eHr}@|)m~
zv7}nV`SryIwn<tNNfIa8e~>txogJO`(91IsH;4w})`YH&wKX`=#A$=-rKy;w8PVy9
z+RZhtVdsMUCHwwgmkZ-WCy)rdE>;OHvX@OTaH|Zl3144+ecx0Yh6Xu`=h22a8XM1}
zB*Prnyx*hf(;S_j<N*dg=2-geAEhaCY*VT%rh1*@kvn|=>6JlJ-)*zdrWkl{l0jdk
z_V^MmCyI{J3etxrEMNA9VQ<>|D0|n-Ow4{Zj2pHp<n@YAv8>f8MX;I<TaK^v$oJ5Y
z@fCaJMvf=rz(JyTd}9al%?dk2Y&&2+jy*+eD-u46`w?-Km!n#Ic2(kP*?J!(c8%!9
zRV^a-3fGN&*{k=e4B&LZ@QqnG{PtUMk6rlY8<K0WU+=K_dkNizI1FfT5?RDza7ylr
zy#*!%m=mkYu!z-Ef4LGuL7hSN8hFc#?$-7P)6Kw4c;fxsq~;-J_t_;KJf!R)o{Gp>
zRz)XnN>OJ}X;0kK?@o>ne?*ns-rgqlLE4DT-5?@%O{r~nYm1gJPfrCxX-?-NYO6zj
z1G9cga#G)mh7RvoKBP`L=!0Nou+IPCiL1=sD)YCB3=%Pt41!5#yOfu4LfnUKnj;EI
zv-5(4X0;DHa}vV@WQ;D@(dCtrmrgtvZcrT*f;bQLxti?n*|eH=I9Sjed$3ofuD4`%
z=B2Ar_9f&@C*3BwI`6G?2jHx14I1!{=X<@C=V1oRgk{3$Q4kSy0Iz-7TiK$?^+-*Q
zyxZQ&i;$9urnN<1BX1})iz3}dvoqjkQLmUOrcMuG!z*jSAve`<@&aX}lvzWM+X;Dn
zV~h`|hG22%`np7ashppa+_miT0vim<f&zhmKqwapJGS6gUJOWXSuVV?$O<A&K1$a%
zGi!N`!%bzCSp?Bsu;*y6ZF;szYJRoA%%^Ya+PU6np`?$rO*OHA0-nrE2+}@`HzK26
zk`ka1K_sx$z`7>MBKr9+iDS8>6D^f#wenucOg>`;Ev{io4(tm%R_gLI%_?}jTczp`
z!Z|oTZyIl)Y3P;n{nI0R2`|CDZ2C1U_H74Vw7TYn-Y_nUIw9niivCPZ+y7@1B>aLN
ze^~_iadq|=aGD1JIzx*r0-ioCd`ij5Ff$jyhjpEu=WJO&!{uzTwKno{f2q7zSN)nT
zq@wJ`O9(}o5|X#*LW>-(ZkT91Zea#H1%T_GzLlzfg-5qdxvrrNZK3?pY9SJqT*0uQ
zqJP`yYpt>9mlBoI?|u1Ra?!ZBs;_I>5WJhaGWr+J(gJs2S_@XF4}-6?UINaR@g2LB
z<>S9nIICr3lP21~&7jUJrj7kAZa}i13B%HH2>v`3Qr*afe532_DlI5cZ<~e=D^0P~
zD0j{t=YW-kxv;g?wDz<|zX+ao3{xfX&Y-nzq@sB<-#jcZEr;`vSlMZZ_`W9D%Cjv@
zf|^R=)-Ov&ASMT;Rw>7O({h~6$imngAoYbAS#35aTO~&FtgKeb%Mh8koJq~B7?&31
zHn(S*e62=dtVLGD3fKO?b&K&Vs=#o{eKaFZ4cWD=jo-A8k8bSEbceO(nQRTy?P5o{
zhIv4&_7#6>Z*E?!3|`~aHkGl|RP;khX!q5dO$`l0TZW-LLnG3rF6r)ki8EeoNV~eJ
zKd+CL`RiC!NM;c!lQwEXKgXD6Eb=3TS1)8W_sju4b6^^__cO;@L4ba*oP{eH=PL~*
zW~AEFP>PcZ*wYQ0gk%4G_5UPRqZ-wyMm4HYjcQb*8r7&qHL6jKYE+{d)u=`_s!@$<
cRHGWzs75uaQH^R;<If%c26d@S;s8(p0Jbtg`Tzg`

diff --git a/engine/tests/Benchmark.signaltest/spec.json b/engine/tests/Benchmark.signaltest/spec.json
index 4cddb88..01b0f88 100644
--- a/engine/tests/Benchmark.signaltest/spec.json
+++ b/engine/tests/Benchmark.signaltest/spec.json
@@ -2,8 +2,7 @@
     "testName": "Benchmark.signaltest",
     "specs": {
         "default": {
-            "LOOPS":"10000"
+            "PARAMS": "-t 2 -m -l 10000 -p 60 -q"
         }
     }
 }
-
-- 
2.7.4

[Fuego] [PATCH 9/9] pi_tests: add patch for old toolchains

[Daniel Sangorrin]

Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
 engine/tests/Functional.pi_tests/fuego_test.sh | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/engine/tests/Functional.pi_tests/fuego_test.sh b/engine/tests/Functional.pi_tests/fuego_test.sh
index 57874e9..712a9a1 100755
--- a/engine/tests/Functional.pi_tests/fuego_test.sh
+++ b/engine/tests/Functional.pi_tests/fuego_test.sh
@@ -1,7 +1,9 @@
-gitrepo=https://github.com/clrkwllms/rt-tests.git
+gitrepo="https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git"
+gitref="unstable/devel/v1.1.1"
 
 function test_build {
-    make CC="$CC" LD="$LD" NUMA=0 pi_stress
+    patch -p1 -N -s < $TEST_HOME/../Benchmark.cyclictest/0001-Add-scheduling-policies-for-old-kernels.patch
+    make NUMA=0 pi_stress
 }
 
 function test_deploy {
-- 
2.7.4

Re: [Fuego] rt-test related fixes

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> I have updated the cyclictest tarball to use the branch unstable/devel/v1.1.1
> which seems to work fine on the Beablebone black (no negative latencies).

Thanks!

> Note that other branches
> can be used by just specifying them on the spec file.

This is directly related to the other discussion we were having about
per_job_build.  Uses of this variable are now starting to pile up.
When I made the variable, I thought that only a few tests would
use it, but if it becomes a regular practice to put alternate source 
repositories into the spec file, then it's use will become more common.
I'm not sure that using multiple sources for the same test is something
that's a good idea in general, but it appears to have its uses and possibly
they are more common than I anticipated.

I think the per_job_build variable is confusing enough that maybe I should look at
handling code sharing automatically.  That is, try to detect when source
sharing is allowable and when not.  Maybe a hash on the tarball or gitrepo
would work for this, to uniquely identify the source used in the
build directory.

>
> I also upgraded the version for other rt-tests that were in Fuego (except
> hackbench): signaltest and pi_tests. Note that signaltest will probably
> cross the 1ms latency threshold and fail unless you change the kernel
> configuration or patch it with real-time support.
OK.  Also, a board without real-time support should have a different
pass criteria, for starters.

> 
> There are actually more test cases inside rt-tests. At first, I thought about
> putting them all together. But then I thought that it would be nice to be
> able to choose the background load for cyclictest from any of the tests
> currently available in Fuego (e.g. a kernel build). That would require
> support for running multiple jobs concurrently on the same board.

Supporting background load is something that's been on my long-range
radar for some time.  Zero-day has support for this, but I'm pretty sure
they are not executing full tests as their background load.  Thinking about this,
I'm not sure this would be very hard.  It might only require a flag to ftc
indicating to 1) run a test without checking the Jenkins status, and 2) without
processing the results, and 3) keeping it in a loop while the test is executing.

Maybe we could do something like this, in the spec:
  "default": {
            "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q",
            "host-load-command": "ftc run-test -b $BOARD -t Functional.mystress --as-load"
        },

The reason to name it "host-load-command" is to distinguish it from "load-command", which
could be something that already exists on the target:
"load-command": "/usr/local/bin/dohell"
(dohell is a test Ingo Molnar wrote for his own RT-Preempt testing.)

> Regards
> Daniel
> 
> [PATCH 1/9] cyclictest: update tarball version (not sent to mailing list)

Where does the tarball come from?  Manually created from the git repo? Or is there an
official tarball somewhere?  If we are creating it manually, it would be good to add
a comment to fuego_test.sh about the steps used to create it.  In general, Fuego
tests currently don't do a good job of documenting where the test program
source comes from. With the introduction of gitrepo, this improves dramatically. 
But I've thought for a while that in the case of tarballs, there should be some
comment or something that indicates the source of the tarball, so future
generations can update them more easily.

I'll review the other patches individually.

Thanks!
 -- Tim

> [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
> [PATCH 3/9] cyclictest: update twothreads spec
> [PATCH 4/9] functions: fix preference order when unpacking or cloning
> [PATCH 5/9] cyclictest: convert tabs to spaces
> [PATCH 6/9] cyclictest: add assert_define
> BENCHMARK_CYCLICTEST_PARAMS
> [PATCH 7/9] cyclictest: add NEED_ROOT for now
> [PATCH 8/9] signaltest: update to new framework
> [PATCH 9/9] pi_tests: add patch for old toolchains

Re: [Fuego] rt-test related fixes

[Daniel Sangorrin]

> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, November 08, 2017 10:51 AM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] rt-test related fixes
> 
> > -----Original Message-----
> > From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> > I have updated the cyclictest tarball to use the branch unstable/devel/v1.1.1
> > which seems to work fine on the Beablebone black (no negative latencies).
> 
> Thanks!
> 
> > Note that other branches
> > can be used by just specifying them on the spec file.
> 
> This is directly related to the other discussion we were having about
> per_job_build.  Uses of this variable are now starting to pile up.
> When I made the variable, I thought that only a few tests would
> use it, but if it becomes a regular practice to put alternate source
> repositories into the spec file, then it's use will become more common.
> I'm not sure that using multiple sources for the same test is something
> that's a good idea in general, but it appears to have its uses and possibly
> they are more common than I anticipated.

I think there will be more use cases.

For example, kselftests (which for some reason didn't get into your branch) will
probably require having different sources depending on the kernel you want to
test.

Another use case is to use source code that works for relatively old toolchains when
the latest source code requires a more recent compiler.

> I think the per_job_build variable is confusing enough that maybe I should look at
> handling code sharing automatically.  That is, try to detect when source
> sharing is allowable and when not.  Maybe a hash on the tarball or gitrepo
> would work for this, to uniquely identify the source used in the
> build directory.

That's a great idea.

> > I also upgraded the version for other rt-tests that were in Fuego (except
> > hackbench): signaltest and pi_tests. Note that signaltest will probably
> > cross the 1ms latency threshold and fail unless you change the kernel
> > configuration or patch it with real-time support.
> OK.  Also, a board without real-time support should have a different
> pass criteria, for starters.
> 
> >
> > There are actually more test cases inside rt-tests. At first, I thought about
> > putting them all together. But then I thought that it would be nice to be
> > able to choose the background load for cyclictest from any of the tests
> > currently available in Fuego (e.g. a kernel build). That would require
> > support for running multiple jobs concurrently on the same board.
> 
> Supporting background load is something that's been on my long-range
> radar for some time.  Zero-day has support for this, but I'm pretty sure
> they are not executing full tests as their background load.  Thinking about this,
> I'm not sure this would be very hard.  It might only require a flag to ftc
> indicating to 1) run a test without checking the Jenkins status, and 2) without
> processing the results, and 3) keeping it in a loop while the test is executing.
> 
> Maybe we could do something like this, in the spec:
>   "default": {
>             "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q",
>             "host-load-command": "ftc run-test -b $BOARD -t Functional.mystress --as-load"
>         },
> 
> The reason to name it "host-load-command" is to distinguish it from "load-command", which
> could be something that already exists on the target:
> "load-command": "/usr/local/bin/dohell"
> (dohell is a test Ingo Molnar wrote for his own RT-Preempt testing.)

Sounds good. I have to confess that I haven't used ftc run-test yet.

> 
> > Regards
> > Daniel
> >
> > [PATCH 1/9] cyclictest: update tarball version (not sent to mailing list)
> 
> Where does the tarball come from?  Manually created from the git repo? Or is there an
> official tarball somewhere?  If we are creating it manually, it would be good to add
> a comment to fuego_test.sh about the steps used to create it.  In general, Fuego
> tests currently don't do a good job of documenting where the test program
> source comes from. With the introduction of gitrepo, this improves dramatically.
> But I've thought for a while that in the case of tarballs, there should be some
> comment or something that indicates the source of the tarball, so future
> generations can update them more easily.

Sorry, the origin of the tarball is written in the commit log. It comes from the unstable
development v1.1.1 branch (please see the commit id in the commit log).
By the way, all tests belonging to rt-tests (Benchmark.hackbench, Benchmakr.signaltest,
Benchmark.cyclictest, Functional.pi_tests etc..) will be sharing the same source code.
Maybe I should specify a common build folder and just build all binaries at once?

Thanks,
Daniel

> 
> > [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
> > [PATCH 3/9] cyclictest: update twothreads spec
> > [PATCH 4/9] functions: fix preference order when unpacking or cloning
> > [PATCH 5/9] cyclictest: convert tabs to spaces
> > [PATCH 6/9] cyclictest: add assert_define
> > BENCHMARK_CYCLICTEST_PARAMS
> > [PATCH 7/9] cyclictest: add NEED_ROOT for now
> > [PATCH 8/9] signaltest: update to new framework
> > [PATCH 9/9] pi_tests: add patch for old toolchains
> 

Re: [Fuego] [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1

[Bird, Timothy]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, November 06, 2017 11:21 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 2/9] cyclictest: update fuego_test to use v1.1.1
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> index a32eb6e..9576bab 100755
> --- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> +++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> @@ -1,4 +1,4 @@
> -tarball=rt-tests-1.0.tar.gz
> +tarball=rt-tests-v1.1.1.tar.gz
> 
>  function test_build {
>  	patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-
> kernels.patch
> --

Looks good.  applied.
 -- Tim

Re: [Fuego] [PATCH 3/9] cyclictest: update twothreads spec

[Bird, Timothy]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, November 06, 2017 11:21 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 3/9] cyclictest: update twothreads spec
> 
> This test is supposed to run 2 threads on the same core so we
> define the affinity parameter. Also we specify the priority
> of the threads and add the mlock flag.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.cyclictest/spec.json | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/engine/tests/Benchmark.cyclictest/spec.json
> b/engine/tests/Benchmark.cyclictest/spec.json
> index a0a3a00..c963026 100644
> --- a/engine/tests/Benchmark.cyclictest/spec.json
> +++ b/engine/tests/Benchmark.cyclictest/spec.json
> @@ -5,7 +5,9 @@
>              "PARAMS": "-S -p 60 -m -D 20 -i 1000 -q"
>          },
>          "twothreads": {
> -            "PARAMS": "-t 2 -l 10000 -q"
> +            "gitrepo": "https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git",
> +            "gitref": "unstable/devel/v1.1.1",
> +            "PARAMS": "-a 0 -m -t 2 -p 60 -l 10000 -q"
>          }
>      }
>  }
> --
> 2.7.4

Looks good. Applied.

Despite our ongoing discussion about PER_JOB_BUILD, that's the mechanism
we have now to prevent re-use of a build directory, when different
specs result in different source for the same test. I plan to add the variable
to the twothreads for this test - after I finish your patch series.
 -- Tim

Re: [Fuego] [PATCH 5/9] cyclictest: convert tabs to spaces

[Bird, Timothy]

> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, November 06, 2017 11:21 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 5/9] cyclictest: convert tabs to spaces
> 
> TODO: we should define clearly what style to use for shell
> scripts and apply it uniformly to all the of them.

Very much agreed.

We should start a document that will live in the documentation directory.
Ideally we would enforce the rules programmatically, prior to code check-in.
The Linux kernel has checkpatch.pl, which is probably too much for us, but
we might write a few simple scripts that check the status of code, and tell
us where we stand.

Here are a few rules I'm going to toss out there for discussion.
Let me know if you agree or disagree:
 * no tabs in shell code
 * indentation = 4 spaces
 * no tabs in python code
 * indentation = 4 spaces
 * no trailing white space on a line

I'm sure we'll think of more, but that's a start.

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.cyclictest/fuego_test.sh | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> index 9576bab..eec07bb 100755
> --- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> +++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> @@ -1,16 +1,16 @@
>  tarball=rt-tests-v1.1.1.tar.gz
> 
>  function test_build {
> -	patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-
> kernels.patch
> -	make NUMA=0 cyclictest
> +    patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-
> kernels.patch
> +    make NUMA=0 cyclictest
>  }
> 
>  function test_deploy {
> -	put cyclictest  $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put cyclictest  $BOARD_TESTDIR/fuego.$TESTDIR/
>  }
> 
>  function test_run {
> -	report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest
> $BENCHMARK_CYCLICTEST_PARAMS"
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest
> $BENCHMARK_CYCLICTEST_PARAMS"
>  }
> 
> 
> --
> 2.7.4

This patch is OK, and is applied.
 -- Tim

Re: [Fuego] [PATCH 6/9] cyclictest: add assert_define BENCHMARK_CYCLICTEST_PARAMS

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> index eec07bb..5f6584b 100755
> --- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> +++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> @@ -10,6 +10,8 @@ function test_deploy {
>  }
> 
>  function test_run {
> +    assert_define BENCHMARK_CYCLICTEST_PARAMS
> +
This should go in test_pre_check.  I'm going to apply this but with
it moved there.  Let me know if you object.

>      report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./cyclictest
> $BENCHMARK_CYCLICTEST_PARAMS"
>  }
> 
> --
> 2.7.4

Thanks!
 -- Tim

Re: [Fuego] [PATCH 7/9] cyclictest: add NEED_ROOT for now

[Bird, Timothy]

applied - thanks.
 -- Tim


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, November 06, 2017 11:21 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 7/9] cyclictest: add NEED_ROOT for now
> 
> cyclitest does not necessarily need root. You can
> add the corresponding 'rtprio' settings to
> /etc/security/limits.conf for your user.
> However, we have no clean way to check that at the moment.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Benchmark.cyclictest/fuego_test.sh | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> index 5f6584b..48c0173 100755
> --- a/engine/tests/Benchmark.cyclictest/fuego_test.sh
> +++ b/engine/tests/Benchmark.cyclictest/fuego_test.sh
> @@ -1,5 +1,7 @@
>  tarball=rt-tests-v1.1.1.tar.gz
> 
> +NEED_ROOT=1
> +
>  function test_build {
>      patch -p1 -N -s < $TEST_HOME/0001-Add-scheduling-policies-for-old-
> kernels.patch
>      make NUMA=0 cyclictest
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 8/9] signaltest: update to new framework

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> signaltest, like cyclictest, one testcase inside the rt-tests
> package. The output format is the same as cyclictest so I just
> copied the parser and criteria files.

OK. Everything looks OK.  I haven't had time to test this on my 
machines, but I applied and pushed to master.  Please test
when you get a chance.

> 
> TODO: there are more rt-tests (e.g. Functional.pi_tests) inside
> Fuego that need to be updated.
Agreed.  Should I make an issue for this?

> 
> TODO: some of the rt-tests are supposed to run in parallel.
> For example, cyclictest may run with hackbench in background.
> Unfortunately, Fuego currently executes tests on the board
> in a sequential manner. We need to make sure that Fuego can
> execute multiple jobs on the same board in parallel or at least
> provide functionality to "disturb" a test with some background
> load.
Agreed.  Should I make an issue for this also?
  -- Tim

> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  .../tests/Benchmark.signaltest/chart_config.json   |   4 ++-
>  engine/tests/Benchmark.signaltest/criteria.json    |  26 +++++++++++++++
>  engine/tests/Benchmark.signaltest/fuego_test.sh    |  12 ++++---
>  engine/tests/Benchmark.signaltest/parser.py        |  35 ++++++++++----------
> -
>  engine/tests/Benchmark.signaltest/reference.log    |  17 ----------
>  .../tests/Benchmark.signaltest/signaltest.tar.gz   | Bin 3710 -> 0 bytes
>  engine/tests/Benchmark.signaltest/spec.json        |   3 +-
>  7 files changed, 53 insertions(+), 44 deletions(-)
>  create mode 100644 engine/tests/Benchmark.signaltest/criteria.json
>  delete mode 100644 engine/tests/Benchmark.signaltest/reference.log
>  delete mode 100644 engine/tests/Benchmark.signaltest/signaltest.tar.gz
> 
> diff --git a/engine/tests/Benchmark.signaltest/chart_config.json
> b/engine/tests/Benchmark.signaltest/chart_config.json
> index f8dcb3b..cdaf6a2 100644
> --- a/engine/tests/Benchmark.signaltest/chart_config.json
> +++ b/engine/tests/Benchmark.signaltest/chart_config.json
> @@ -1,3 +1,5 @@
>  {
> -        "signaltest":["Thread0", "Thread1"]
> +	"chart_type": "measure_plot",
> +	"measures": ["default.latencies.max_latency",
> +        "default.latencies.avg_latency"]
>  }
> diff --git a/engine/tests/Benchmark.signaltest/criteria.json
> b/engine/tests/Benchmark.signaltest/criteria.json
> new file mode 100644
> index 0000000..f137d55
> --- /dev/null
> +++ b/engine/tests/Benchmark.signaltest/criteria.json
> @@ -0,0 +1,26 @@
> +{
> +    "schema_version":"1.0",
> +    "criteria":[
> +        {
> +            "tguid":"default.latencies.max_latency",
> +            "reference":{
> +                "value":1000,
> +                "operator":"le"
> +            }
> +        },
> +        {
> +            "tguid":"default.latencies.min_latency",
> +            "reference":{
> +                "value":1000,
> +                "operator":"le"
> +            }
> +        },
> +        {
> +            "tguid":"default.latencies.avg_latency",
> +            "reference":{
> +                "value":1000,
> +                "operator":"le"
> +            }
> +        }
> +    ]
> +}
> diff --git a/engine/tests/Benchmark.signaltest/fuego_test.sh
> b/engine/tests/Benchmark.signaltest/fuego_test.sh
> index af62c0c..84b5063 100755
> --- a/engine/tests/Benchmark.signaltest/fuego_test.sh
> +++ b/engine/tests/Benchmark.signaltest/fuego_test.sh
> @@ -1,19 +1,21 @@
> -tarball=signaltest.tar.gz
> +gitrepo="https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git"
> +gitref="unstable/devel/v1.1.1"
> 
>  NEED_ROOT=1
> 
>  function test_build {
> -  make CC="$CC" LD="$LD" LDFLAGS="$LDFLAGS" CFLAGS="$CFLAGS"
> +    patch -p1 -N -s < $TEST_HOME/../Benchmark.cyclictest/0001-Add-
> scheduling-policies-for-old-kernels.patch
> +    make NUMA=0 signaltest
>  }
> 
>  function test_deploy {
> -	put signaltest  $BOARD_TESTDIR/fuego.$TESTDIR/
> +    put signaltest  $BOARD_TESTDIR/fuego.$TESTDIR/
>  }
> 
>  function test_run {
> -	assert_define BENCHMARK_SIGNALTEST_LOOPS
> +    assert_define BENCHMARK_SIGNALTEST_PARAMS
> 
> -	report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./signaltest  -l
> $BENCHMARK_SIGNALTEST_LOOPS -q"
> +    report "cd $BOARD_TESTDIR/fuego.$TESTDIR; ./signaltest
> $BENCHMARK_SIGNALTEST_PARAMS"
>  }
> 
> 
> diff --git a/engine/tests/Benchmark.signaltest/parser.py
> b/engine/tests/Benchmark.signaltest/parser.py
> index 44425ea..3f4f05d 100755
> --- a/engine/tests/Benchmark.signaltest/parser.py
> +++ b/engine/tests/Benchmark.signaltest/parser.py
> @@ -1,26 +1,23 @@
>  #!/usr/bin/python
> -# See common.py for description of command-line arguments
> -
>  import os, re, sys
>  sys.path.insert(0, os.environ['FUEGO_CORE'] + '/engine/scripts/parser')
>  import common as plib
> 
> -ref_section_pat = "^\[[\w\d_ .]+.[gle]{2}\]"
> -cur_search_pat =
> re.compile("^T:([\s\d]+)(.*)P:(.*)C:(.*)Min:([\s\d]+)Act:([\s\d]+)Avg:([\s\d]
> +)Max:([\s\d]+)",re.MULTILINE)
> -
> -res_dict = {}
> -cur_dict = {}
> -
> -pat_result = plib.parse(cur_search_pat)
> -if pat_result:
> -	cur_dict["Thread0.Min"] = '%d' % int(pat_result[0][4])
> -	cur_dict["Thread0.Act"] = '%d' % int(pat_result[0][5])
> -	cur_dict["Thread0.Avg"] = '%d' % int(pat_result[0][6])
> -	cur_dict["Thread0.Max"] = '%d' % int(pat_result[0][7])
> -	cur_dict["Thread1.Min"] = '%d' % int(pat_result[1][4])
> -	cur_dict["Thread1.Act"] = '%d' % int(pat_result[1][5])
> -	cur_dict["Thread1.Avg"] = '%d' % int(pat_result[1][6])
> -	cur_dict["Thread1.Max"] = '%d' % int(pat_result[1][7])
> +regex_string = "^T:.*Min:\s+(\d+).*Avg:\s+(\d+) Max:\s+(\d+)"
> +measurements = {}
> +matches = plib.parse_log(regex_string)
> 
> +if matches:
> +	min_latencies = []
> +	avg_latencies = []
> +	max_latencies = []
> +	for thread in matches:
> +		min_latencies.append(float(thread[0]))
> +		avg_latencies.append(float(thread[1]))
> +		max_latencies.append(float(thread[2]))
> +	measurements['default.latencies'] = [
> +		{"name": "max_latency", "measure" : max(max_latencies)},
> +		{"name": "min_latency", "measure" : min(min_latencies)},
> +		{"name": "avg_latency", "measure" :
> sum(avg_latencies)/len(avg_latencies)}]
> 
> -sys.exit(plib.process_data(ref_section_pat, cur_dict, 'm', 'usec'))
> +sys.exit(plib.process(measurements))
> diff --git a/engine/tests/Benchmark.signaltest/reference.log
> b/engine/tests/Benchmark.signaltest/reference.log
> deleted file mode 100644
> index 83a959d..0000000
> --- a/engine/tests/Benchmark.signaltest/reference.log
> +++ /dev/null
> @@ -1,17 +0,0 @@
> -#sdfdsf
> -[Thread0.Min|le]
> -10000000
> -[Thread0.Act|le]
> -10000000
> -[Thread0.Avg|le]
> -10000000
> -[Thread0.Max|le]
> -10000000
> -[Thread1.Min|le]
> -10000000
> -[Thread1.Act|le]
> -10000000
> -[Thread1.Avg|le]
> -10000000
> -[Thread1.Max|le]
> -10000000
> diff --git a/engine/tests/Benchmark.signaltest/signaltest.tar.gz
> b/engine/tests/Benchmark.signaltest/signaltest.tar.gz
> deleted file mode 100644
> index
> 2c15b87f03a88b2d2818994223be08837d89e343..0000000000000000000000000
> 000000000000000
> GIT binary patch
> literal 0
> HcmV?d00001
> 
> literal 3710
> zcmV-^4uSC>iwFqm`FK(Q19NF-
> ZeeV6Wpi{cbYXG;?HX%y+cq*^rC+gj&o1Riwj{?+
> zDo4(xu9JGskBsf~rsw83GzCelDN-RpKic|!`|U0O5|m^onQ41-cf@pJaIsh{7W-
> Dy
> zU=(?w6sg>J@<)v}j^|rj^tbbTt8M=^<4LF6-
> Q4bMZfz30v$_3z>j`ZAVH{Z_OQn|p
> zJn_Rg8;pfFcmVHH>;I)AE#CjgyAs176u<i!^uXKCo;~>f-RGO;{kz?*O?>~Y&CTs6
> z(Ei;ySup<3-+ylpUiING&fe+C*_pd{a`f)taKAC~{rdUq)3^KQ(1$`r>TeETzdb`x
> z>+fC|LhGbkKRo!B0pV1R6O<0l5?Qa~a?jeh?$>L0#$E$YVj4L4C$wVxRL9T)k!sZ
> I
> zeki=CSFa@#Xbmldwe=X*2VyFs0nXR{RsT<m?tfxf_HRGe?|R1L_Kzmtc|K$R+
> uiQ7
> z%Krb1v9VT%H8?%j#tKQCMFW`xQy@ivG#<(uFA<1Dcn(@{7lS#RkK>7#!rM>;
> w~<KT
> zr5uH~e+`2ulSv!}w`i!lde8YdNMV}9qr{s4PKSvQIo%z&i!<=O2of;}Qat!FlL7(>
> zUNqQ<6CyB)2f^@;V8|MtCC37pOC~AAL;iSs{2ty4WCQtsmtBT|4~Ky-
> qEx^SA_?L&
> zA~Ib-
> hNrZgj>P~j?|APU;^s_q18?Z|o($q>M{&PVuYWbbs2Cxs{nN99lVkVn{Pf`X
> zEiC^4?e)!NTn{2Y%mxBp4*f`m>*HMuG!inN$}&uQ$}&)ie+DK8VQ@JI3*uQ&
> 5=5gi
> zOa>EChGY?nmk+go^kBCe3zyQnbYq(Mvlk*TT8x~B%DbsZ7l1V<xg1}gp1%IkJ
> v;cv
> zzH{3ECk=iP@sOq)py&8;(f-
> iDSg^4M??>TX6HqH)9S6OLA%wD(!N6$%9^+%caa{NK
> z)K$<%ao+p0{XO^H{;7+9YqgGYI{E7HqE$kmvyad@z@>m`e1m$2+SY|7n}Z)-
> AAbA(
> z4c`U#Zg0ML#@o(hlKB#33^MVebclPW9%{cptxXwB8mz)4QcE>D16;<0U+c9X
> k`N4b
> z)MScktfxv-
> @FR<8imDvO(Fi8qt?S=m@Jw;3Op@LU*680<)40rrlk{q*e!pWgJd6_{
> zEYya;Pv==<I={{yoFBRfO#5(2aTrAT)VnoP&zCqI#AuiTyz7yHuU&g##&os4xQ3
> 51
> zD9(X&G4(+%U5Xq=Qly*m-F#pnjx#xkZ;&Aih_B^~FqojH=|j7;G~~58a2yV-
> g<>Tn
> z?YNGE0mw9plZj&liN=slg%4{oZPZbeH)Bk=!I5ceS6;j5v3%I=mz1(W60g|^tun$
> +
> zv5c!vpW^p@{eG4i^(J2Eqc0uKu_HT8n1gkhL*7W6=#}sfUe&fJ3JMO2qJcEl6${
> V;
> z4e4q~!|Z+9FJnjZ=ma8PK&FV>5+SoB(hS^_wvsQ`!AiZh@zgBuHJ!!qilabmQhb
> ?3
> zrbf$nDgG!-C-pH%XbhfDX)PF~Lb_~&II#@P2<=j!5s4;vL$vir3{f4}+O>^x8e`<T
> zGb?2A@CwfMzTf}OeRJ^U1bX`8<ox^nQ{xOUyiC#LSpOI@43d<{cTgW|!4P#T
> *J`Co
> z=q;s~ILpEx$IyXYKsO)z8|166L=uT`gLk;-
> QC)wUrOWhspS_Esdy+*FMx+HB+YfKV
> z`$p3Br0XYMI{valvs5H5M#0fYU>dPtZ|gHRyFv2t%h<ja$z`01&)qad@l2+uLZ}c6
> zZ@F;S4OZsa)I}}(0<oH-
> C@_Sc6p??YwvtjPf7ZaS&k*v&$FxYgBJ$8jvv|MYZLD4B
> zm&hVKiOwn@oi!1h2%<?5yLV_;&Z@GnhB7&CFtC74Z8&V^8iP*8bo9zOJ9z7
> Udw8<<
> zw<hH1$L|jh`Gkp>C^~G;O=vew!l57yrpD8CZT@J3Ngzu$X>ic5NuFe(NF1c4%$
> E*U
> z)OL!?+ChmN^kTy#m<cHSGAy16GC1(=@C4gy{}Q85Uu$&uK}DDf!sm32Sm_v
> HW6|PY
> zxtfl%N0M=mPL5B`PmT}vkX7Zbm{lE#mP};S&SJ*DMez-
> tXa@z8l%%W{bP4>p@dE3`
> zc+!BE&{p?~F$an>$s0b!B70J$a_IYOHT3=mLn;Qal%PCH`8y?Ie9%ycsq(%c&^G
> U&
> zv#mC9VFOgzr0WdX(A+NZ)1p4EuELU|UQ8t~unO(l_L~MXoiS;LGepA<Rvi-
> jEJ;M9
> zWTdEm>Pjd?em9fFTm_iCYE_BNO;DBJpPhE{qt33VCd)6(92S>gpn9^Ji2xK4YBl
> x5
> z_^F?+=Esr5v|rE%>4=Hyd2+zgYeKFY^EBv0-t`g`+vt~+FEL+;o2!RG-ZjYHt&Qy6
> zTF459X&|QxDl2U0Rr6jZ=HYu;85@3?V=TkcBGFqoIa*+7vzFH8TjzFF7?xfzHlpa
> =
> z7kvA2?k-A|l-2XetZpg+x5S{b-HRFZvv5D;lx;di)=vVM1@-
> %26XfGAQN6B|dq5NH
> z)k(L~n?U+ws!S+Z#l(vA>LE|qe$WG2>B%SEAoUD6MbQZ!d1+4Ys2+Joxd3FTH
> xih?
> z4v8nK$yG(dSuH1agCL!T-
> km!Z;gnN7&ha>r7&^;EI}29QpQ7dFOa)I*X>i}uQ9VSe
> z*FX(>`%5h0@%VK2r>KO|y5v+Cpw(ijvO@J4lv0<Hfl3>}k0%okVK-
> wDfey#0s)Sso
> zIfRGCFpGQ^gQDm#DdYhiAu3*QEgEG?Asn8ZygNhg!Z@C$ef32n$|jeppyibw
> U_I+)
> zq3pLE<Qp|5ogp=acc%v@bWu`-
> zWU&iD+Ur92ct2`i&Izi+GR2y^{mzUm<8xMi@^^8
> zaTMNx2Wf_~Lf4H~6}N#b(~zhtM@&QOQ(t}Cx6&)cy}D%@*Et_LearbklbB`_l
> r<6q
> zDH?h<TN;<BJ@{S#N^^S<_59v7M16@l3UUd4pED}B?(;#Y<EZRNlvI*#?2w<iF
> BPmt
> z&ZY7bX}S8YXiweJn%xhu!1}@^7(d32w!GNK=QrxgrK8sVS?ilBRM#w15eHr}@
> |)m~
> zv7}nV`SryIwn<tNNfIa8e~>txogJO`(91IsH;4w})`YH&wKX`=#A$=-rKy;w8PVy9
> z+RZhtVdsMUCHwwgmkZ-
> WCy)rdE>;OHvX@OTaH|Zl3144+ecx0Yh6Xu`=h22a8XM1}
> zB*Prnyx*hf(;S_j<N*dg=2-geAEhaCY*VT%rh1*@kvn|=>6JlJ-)*zdrWkl{l0jdk
> z_V^MmCyI{J3etxrEMNA9VQ<>|D0|n-
> Ow4{Zj2pHp<n@YAv8>f8MX;I<TaK^v$oJ5Y
> z@fCaJMvf=rz(JyTd}9al%?dk2Y&&2+jy*+eD-u46`w?-Km!n#Ic2(kP*?J!(c8%!9
> zRV^a-
> 3fGN&*{k=e4B&LZ@QqnG{PtUMk6rlY8<K0WU+=K_dkNizI1FfT5?RDza7ylr
> zy#*!%m=mkYu!z-Ef4LGuL7hSN8hFc#?$-7P)6Kw4c;fxsq~;-J_t_;KJf!R)o{Gp>
> zRz)XnN>OJ}X;0kK?@o>ne?*ns-rgqlLE4DT-5?@%O{r~nYm1gJPfrCxX-?-
> NYO6zj
> z1G9cga#G)mh7RvoKBP`L=!0Nou+IPCiL1=sD)YCB3=%Pt41!5#yOfu4LfnUKnj;E
> I
> zv-
> 5(4X0;DHa}vV@WQ;D@(dCtrmrgtvZcrT*f;bQLxti?n*|eH=I9Sjed$3ofuD4`%
> z=B2Ar_9f&@C*3BwI`6G?2jHx14I1!{=X<@C=V1oRgk{3$Q4kSy0Iz-7TiK$?^+-
> *Q
> zyxZQ&i;$9urnN<1BX1})iz3}dvoqjkQLmUOrcMuG!z*jSAve`<@&aX}lvzWM+X
> ;Dn
> zV~h`|hG22%`np7ashppa+_miT0vim<f&zhmKqwapJGS6gUJOWXSuVV?$O<A
> &K1$a%
> zGi!N`!%bzCSp?Bsu;*y6ZF;szYJRoA%%^Ya+PU6np`?$rO*OHA0-
> nrE2+}@`HzK26
> zk`ka1K_sx$z`7>MBKr9+iDS8>6D^f#wenucOg>`;Ev{io4(tm%R_gLI%_?}jTczp`
> z!Z|oTZyIl)Y3P;n{nI0R2`|CDZ2C1U_H74Vw7TYn-
> Y_nUIw9niivCPZ+y7@1B>aLN
> ze^~_iadq|=aGD1JIzx*r0-ioCd`ij5Ff$jyhjpEu=WJO&!{uzTwKno{f2q7zSN)nT
> zq@wJ`O9(}o5|X#*LW>-(ZkT91Zea#H1%T_GzLlzfg-
> 5qdxvrrNZK3?pY9SJqT*0uQ
> zqJP`yYpt>9mlBoI?|u1Ra?!ZBs;_I>5WJhaGWr+J(gJs2S_@XF4}-
> 6?UINaR@g2LB
> z<>S9nIICr3lP21~&7jUJrj7kAZa}i13B%HH2>v`3Qr*afe532_DlI5cZ<~e=D^0P~
> zD0j{t=YW-kxv;g?wDz<|zX+ao3{xfX&Y-nzq@sB<-
> #jcZEr;`vSlMZZ_`W9D%Cjv@
> zf|^R=)-
> Ov&ASMT;Rw>7O({h~6$imngAoYbAS#35aTO~&FtgKeb%Mh8koJq~B7?&31
> zHn(S*e62=dtVLGD3fKO?b&K&Vs=#o{eKaFZ4cWD=jo-
> A8k8bSEbceO(nQRTy?P5o{
> zhIv4&_7#6>Z*E?!3|`~aHkGl|RP;khX!q5dO$`l0TZW-LLnG3rF6r)ki8EeoNV~eJ
> zKd+CL`RiC!NM;c!lQwEXKgXD6Eb=3TS1)8W_sju4b6^^__cO;@L4ba*oP{eH=P
> L~*
> zW~AEFP>PcZ*wYQ0gk%4G_5UPRqZ-
> wyMm4HYjcQb*8r7&qHL6jKYE+{d)u=`_s!@$<
> cRHGWzs75uaQH^R;<If%c26d@S;s8(p0Jbtg`Tzg`
> 
> diff --git a/engine/tests/Benchmark.signaltest/spec.json
> b/engine/tests/Benchmark.signaltest/spec.json
> index 4cddb88..01b0f88 100644
> --- a/engine/tests/Benchmark.signaltest/spec.json
> +++ b/engine/tests/Benchmark.signaltest/spec.json
> @@ -2,8 +2,7 @@
>      "testName": "Benchmark.signaltest",
>      "specs": {
>          "default": {
> -            "LOOPS":"10000"
> +            "PARAMS": "-t 2 -m -l 10000 -p 60 -q"
>          }
>      }
>  }
> -
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 9/9] pi_tests: add patch for old toolchains

[Bird, Timothy]

> -----Original Message-----
> From: Daniel Sangorrin on Monday, November 06, 2017 11:21 PM
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/tests/Functional.pi_tests/fuego_test.sh | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/engine/tests/Functional.pi_tests/fuego_test.sh
> b/engine/tests/Functional.pi_tests/fuego_test.sh
> index 57874e9..712a9a1 100755
> --- a/engine/tests/Functional.pi_tests/fuego_test.sh
> +++ b/engine/tests/Functional.pi_tests/fuego_test.sh
> @@ -1,7 +1,9 @@
> -gitrepo=https://github.com/clrkwllms/rt-tests.git
> +gitrepo="https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git"
> +gitref="unstable/devel/v1.1.1"
> 
>  function test_build {
> -    make CC="$CC" LD="$LD" NUMA=0 pi_stress
> +    patch -p1 -N -s < $TEST_HOME/../Benchmark.cyclictest/0001-Add-
> scheduling-policies-for-old-kernels.patch
Hmmm.  I don't like that too much.  I don't think a test should
reach into another test's home directory.  This could lead to some hard-to-track
dependencies.  (e.g. someone updates the source for cyclictest and
removes the patch, not knowing another test is using it).  This is
also a problem for test packaging.

The simplest alternative is to copy the patch into the pi_tests home
directory.  But then it has the problem that updates to the patch
might not get propagated to other copies.  This is also bad.

cyclictest is one sub-test under rt-tests, as is pi_tests.  If we copy from
anywhere, it should probably be from some hierarchically higher test,
and not from a peer within the same test "group".

I think if we're sharing source, then maybe patches for shared source should
go into directory specifically for this.  This is all related to the commonAPI
stuff that Wenlong just proposed, as well as the shared build directory
that was added in 1.2.

I need to think about this more.

... Oops - I just noticed this is in Benchmark.signaltest/fuego_test.sh also.

I think I'll apply this as is, but I'd like to see it changed to something else, when
the dust settles on our preferred method of sharing source code (both tarballs
and patches).  I suspect we'll end up with a directory like:
fuego-core/engine/tests/source/rt-tests
where I think something like this should live (and move netperf and OpenSSL
code under tests/source as well).
 -- Tim

> +    make NUMA=0 pi_stress
>  }
> 
>  function test_deploy {
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego

Re: [Fuego] [PATCH 4/9] functions: fix preference order when unpacking or cloning

[Bird, Timothy]

This one's going to take me a bit more time to review and test.
I don't want to commit something in the core without more
thorough testing than I have time for right now.
I'll try to get this in tomorrow (or give feedback on it.)
 -- Tim


> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org [mailto:fuego-
> bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Monday, November 06, 2017 11:21 PM
> To: fuego@lists.linuxfoundation.org
> Subject: [Fuego] [PATCH 4/9] functions: fix preference order when
> unpacking or cloning
> 
> In the previous implementation the preference was
>   spec tarball > fuego_test tarball > spec gitrepo > fuego_test gitrepo
> 
> This patch changes it to
>   spec gitrepo > spec tarball > fuego_test gitrepo > fuego_test tarball
> 
> Therefore, spec.json gitrepo/tarball variables take precedence over
> fuego_test.sh variables. And if, for some reason, gitrepo and
> tarball variables are defined at the same time, then gitrepo will
> take precedence. This case is not supposed to happen though.
> 
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
>  engine/scripts/functions.sh | 137 ++++++++++++++++++++++++++----------
> --------
>  1 file changed, 82 insertions(+), 55 deletions(-)
> 
> diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> index da3415b..fe27b0c 100755
> --- a/engine/scripts/functions.sh
> +++ b/engine/scripts/functions.sh
> @@ -38,72 +38,99 @@ trap signal_handler SIGTERM SIGHUP SIGALRM
> SIGINT EXIT
>  # cause ERR trap setting to be visible inside functions
>  set -o errtrace
> 
> -# Unpacks/clones the test source code into the current directory.
> -#
> -# Tarball or git variables can be in the spec or directly in fuego_test.sh
> -# Variables in the spec have preference.
> -#
> -# Tarball variables
> -#   - tarball: tarball file name (e.g. bc-script.tar.gz)
> -# Git variables
> -#   - gitrepo: git repository (e.g. https://github.com/torvalds/linux.git)
> -#   - gitref: git branch, tag or commit id (optional, default: master)
> -function unpack {
> -    # Tarballs
> -    upName=`echo "${TESTDIR^^}"| tr '.' '_'`
> -    spec_tarball="${upName}_TARBALL"
> -
> -    if [ ! -z "${!spec_tarball}" ]; then
> -        tarball=${!spec_tarball}
> +# Clones a git repository into the current folder
> +# $1 (gitrepo): git repository URL
> +# $2 (gitref): branch, tag or commit id (optional, default: master)
> +# FIXTHIS: add commit id information to the json output
> +function git_clone {
> +    local gitrepo=${1}
> +    local gitref=${2}
> +
> +    is_empty "$gitrepo"
> +
> +    if [ -z "$gitref" ]; then
> +        gitref="master"
>      fi
> 
> -    if [ ! -z ${tarball+x} ]; then
> -        echo "Unpacking $tarball"
> -        case ${tarball/*./} in
> -            gz|tgz) key=z ;;
> -            bz2) key=j ;;
> -            tar) key= ;;
> -            *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
> -        esac
> -        tar ${key}xf $TEST_HOME/$tarball --strip-components=1
> -        return
> +    local is_namedref=$(git ls-remote $gitrepo $gitref)
> +
> +    if [ "$is_namedref" = "" ]; then
> +        echo "Clone repository $gitrepo."
> +        git clone -n $gitrepo fuego_git_repo
> +    else
> +        echo "Clone repository $gitrepo."
> +        git clone -n --depth=1 --branch=$gitref $gitrepo fuego_git_repo
>      fi
> 
> -    # GIT repositories
> -    spec_gitrepo="${upName}_GITREPO"
> -    spec_gitref="${upName}_GITREF"
> +    # equivalent to tarball's --strip-components=1
> +    mv fuego_git_repo/.git .git
> +    rm -rf fuego_git_repo
> 
> -    if [ ! -z "${!spec_gitrepo}" ]; then
> -        gitrepo=${!spec_gitrepo}
> -    fi
> +    echo "Checkout branch/tag/commit id $gitref."
> +    git checkout $gitref
> +}
> 
> -    if [ ! -z "${!spec_gitref}" ]; then
> -        gitref=${!spec_gitref}
> -    fi
> +# Untars a tarball in the current folder
> +# $1 (tarball): file to untar
> +function untar {
> +    local tarball=${1}
> 
> -    # FIXTHIS: add commit id information to the json output
> -    if [ ! -z ${gitrepo+x} ]; then
> -        if [ -z ${gitref+x} ]; then
> -            gitref="master"
> -        fi
> +    is_empty "$tarball"
> 
> -        is_namedref=$(git ls-remote $gitrepo $gitref)
> +    echo "Unpacking $tarball"
> +    case ${tarball/*./} in
> +        gz|tgz) key=z ;;
> +        bz2) key=j ;;
> +        tar) key= ;;
> +        *) echo "Unknown $tarball file format. Not unpacking."; return 1;;
> +    esac
> +    tar ${key}xf $TEST_HOME/$tarball --strip-components=1
> +}
> 
> -        if [ "$is_namedref" = "" ]; then
> -            echo "Clone repository $gitrepo."
> -            git clone -n $gitrepo fuego_git_repo
> +# Unpacks/clones the test source code into the current directory.
> +#
> +# The following tarball and git variables can be specified in the test's
> +# spec.json or fuego_test.sh.
> +#   - Tarball variables
> +#     - tarball: tarball file name (e.g. bc-script.tar.gz)
> +#   - Git variables
> +#     - gitrepo: git repository (e.g. https://github.com/torvalds/linux.git)
> +#     - gitref: git branch, tag or commit id (optional, default: master)
> +#
> +# tarball and git variables follow the next preference rule:
> +#   spec gitrepo > spec tarball > fuego_test gitrepo > fuego_test tarball
> +function unpack {
> +    # prepare variables
> +    upName=`echo "${TESTDIR^^}"| tr '.' '_'`
> +    spec_gitrepo="${upName}_GITREPO"
> +    spec_gitref="${upName}_GITREF"
> +    spec_tarball="${upName}_TARBALL"
> +    # 1) spec gitrepo
> +    if [ ! -z "${!spec_gitrepo}" ]; then
> +        gitrepo=${!spec_gitrepo}
> +        if [ ! -z "${!spec_gitref}" ]; then
> +            gitref=${!spec_gitref}
>          else
> -            echo "Clone repository $gitrepo."
> -            git clone -n --depth=1 --branch=$gitref $gitrepo fuego_git_repo
> +            # gitref could have been defined by fuego_test.sh
> +            gitref="master"
>          fi
> -
> -        # equivalent to tarball's --strip-components=1
> -        mv fuego_git_repo/.git .git
> -        rm -rf fuego_git_repo
> -
> -        echo "Checkout branch/tag/commit id $gitref."
> -        git checkout $gitref
> -
> +        git_clone "$gitrepo" "$gitref"
> +        return
> +    fi
> +    # 2) spec tarball
> +    if [ ! -z "${!spec_tarball}" ]; then
> +        tarball=${!spec_tarball}
> +        untar "$tarball"
> +        return
> +    fi
> +    # 3) fuego_test gitrepo
> +    if [ ! -z "$gitrepo" ]; then
> +        git_clone "$gitrepo" "$gitref"
> +        return
> +    fi
> +    # 4) fuego_test tarball
> +    if [ ! -z "$tarball" ]; then
> +        untar "$tarball"
>          return
>      fi
> 
> --
> 2.7.4
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego