* [Fuego] Changes to the specs and separate LTP build and run
@ 2017-04-19 8:32 Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH] cross: add libaio dependency for building LTP Daniel Sangorrin
` (3 more replies)
0 siblings, 4 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-19 8:32 UTC (permalink / raw)
To: fuego
This series of patches contain quite a few changes:
- Changes to the specs and testplans.
+ The format of the test specs is changed. Now
spec names are keys, so we can access them
directly.
+ A board's testplan can now override default
parameters (timeout, cleanup etc) by defining
a single variable (this one is for you Jan-Simon ;))
- Changes to LTP
+ I have added support for building and running LTP tests
separately. The instructions are written in the
corresponding commit but I put them also here for
your convenience (this one is for you Dhinakar ;))
fuego patches:
[PATCH] cross: add libaio dependency for building LTP
fuego-core patches:
[PATCH 1/3] specs: put links in the spec and don't use the name key
[PATCH 2/3] defaults: handle defaults in a more efficient way
[PATCH 3/3] LTP: support separate phases for build and run
Pre-requisites for LTP
- Make sure no other fuego or jenkins is running on the machine
- Install Fuego from scratch out of my 'next' branch
+ remove previous images if necessary
- Start fuego's docker and install an armhf toolchain:
# /fuego-ro/toolchains/install_cross_toolchain.sh armhf
LTP instructions:
1) ftc add-jobs -b docker -t Functional.LTP -s buildonly
2) Execute the job on Jenkins and wait for it to finish
3) If the build is succesful a tar.gz link will appear. This
links to ltp.tar.gz which contains the LTP binaries.
Download them and uncompress them into your target's /opt
folder. For example:
$ cd /media/user/rootfs/opt/ <-- target's SD Card
# tar zxvf <path to>/ltp.tar.gz
4) ftc add-jobs -b zynq -t Functional.LTP -s runonly
5) Execute the job on Jenkins and wait for it to finish.
6) If the execution is succesful you should get a link
to the results.
Note: depending on your target you will need to modify the
toolchain architecture and Functional.LTP.spec.
Cheers,
Daniel
^ permalink raw reply [flat|nested] 7+ messages in thread
* [Fuego] [PATCH] cross: add libaio dependency for building LTP
2017-04-19 8:32 [Fuego] Changes to the specs and separate LTP build and run Daniel Sangorrin
@ 2017-04-19 8:32 ` Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key Daniel Sangorrin
` (2 subsequent siblings)
3 siblings, 0 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-19 8:32 UTC (permalink / raw)
To: fuego
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
fuego-ro/toolchains/install_cross_toolchain.sh | 3 +++
1 file changed, 3 insertions(+)
diff --git a/fuego-ro/toolchains/install_cross_toolchain.sh b/fuego-ro/toolchains/install_cross_toolchain.sh
index eb74f31..dc95fbd 100755
--- a/fuego-ro/toolchains/install_cross_toolchain.sh
+++ b/fuego-ro/toolchains/install_cross_toolchain.sh
@@ -12,3 +12,6 @@ dpkg --add-architecture $ARCH
curl http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | sudo apt-key add -
DEBIAN_FRONTEND=noninteractive apt-get update
DEBIAN_FRONTEND=noninteractive apt-get -yV install crossbuild-essential-$ARCH
+
+# libaio is needed for LTP
+DEBIAN_FRONTEND=noninteractive apt-get -yV install libaio-dev:$ARCH
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key
2017-04-19 8:32 [Fuego] Changes to the specs and separate LTP build and run Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH] cross: add libaio dependency for building LTP Daniel Sangorrin
@ 2017-04-19 8:32 ` Daniel Sangorrin
2017-04-26 0:43 ` Bird, Timothy
2017-04-19 8:32 ` [Fuego] [PATCH 2/3] defaults: handle defaults in a more efficient way Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 3/3] LTP: support separate phases for build and run Daniel Sangorrin
3 siblings, 1 reply; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-19 8:32 UTC (permalink / raw)
To: fuego
This is quite a big change. Until now links that were
displayed by the "descriptionsetter" plugin were defined
in the testplans. That doesn't make sense because those links
depend on the test spec not on the board's testplan. For
that reason I put all of the links inside the spec files.
Additionally, I have modified the spec file format. Before
specs where just an array of dicts so to find a specific
spec you would have to traverse the whole array and compare
the 'name' property. Now, the spec names are the keys so
we can quickly access their attributes.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/scripts/ftc | 40 ++++++++++++------
engine/scripts/ovgen.py | 20 ++++-----
.../Benchmark.Dhrystone/Benchmark.Dhrystone.spec | 23 +++++-----
.../tests/Benchmark.GLMark/Benchmark.GLMark.spec | 15 ++++---
.../tests/Benchmark.IOzone/Benchmark.IOzone.spec | 26 ++++--------
.../Benchmark.Interbench/Benchmark.Interbench.spec | 27 ++++--------
engine/tests/Benchmark.Java/Benchmark.Java.spec | 15 ++++---
.../tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec | 14 +++----
.../tests/Benchmark.Stream/Benchmark.Stream.spec | 14 +++----
.../Benchmark.Whetstone/Benchmark.Whetstone.spec | 16 +++----
engine/tests/Benchmark.aim7/Benchmark.aim7.spec | 10 ++---
.../Benchmark.blobsallad/Benchmark.blobsallad.spec | 17 ++++----
.../tests/Benchmark.bonnie/Benchmark.bonnie.spec | 22 ++++------
.../Benchmark.cyclictest/Benchmark.cyclictest.spec | 11 +++--
.../tests/Benchmark.dbench/Benchmark.dbench.spec | 23 ++++------
.../tests/Benchmark.ebizzy/Benchmark.ebizzy.spec | 12 +++---
engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec | 23 ++++------
engine/tests/Benchmark.fio/Benchmark.fio.spec | 23 ++++------
.../tests/Benchmark.fs_mark/Benchmark.fs_mark.spec | 10 ++---
.../Benchmark.fuego_check_plots.spec | 15 +++----
.../tests/Benchmark.gtkperf/Benchmark.gtkperf.spec | 17 ++++----
.../Benchmark.hackbench/Benchmark.hackbench.spec | 12 +++---
.../tests/Benchmark.himeno/Benchmark.himeno.spec | 15 ++++---
engine/tests/Benchmark.iperf/Benchmark.iperf.spec | 12 +++---
.../tests/Benchmark.linpack/Benchmark.linpack.spec | 15 ++++---
.../Benchmark.lmbench2/Benchmark.lmbench2.spec | 15 ++++---
.../Benchmark.nbench-byte.spec | 10 ++---
.../Benchmark.nbench_byte.spec | 15 ++++---
.../tests/Benchmark.netperf/Benchmark.netperf.spec | 17 ++++----
.../tests/Benchmark.netpipe/Benchmark.netpipe.spec | 12 +++---
.../tests/Benchmark.reboot/Benchmark.reboot.spec | 15 ++++---
.../Benchmark.signaltest/Benchmark.signaltest.spec | 14 +++----
.../Benchmark.sysbench/Benchmark.sysbench.spec | 10 ++---
.../Benchmark.tiobench/Benchmark.tiobench.spec | 19 ++++-----
.../tests/Benchmark.x11perf/Benchmark.x11perf.spec | 12 +++---
engine/tests/Functional.LTP/Functional.LTP.spec | 49 +++++++++++-----------
.../Functional.OpenSSL/Functional.OpenSSL.spec | 10 ++---
.../Functional.aiostress/Functional.aiostress.spec | 20 ++++-----
.../Functional.arch_timer.spec | 10 ++---
engine/tests/Functional.bc/Functional.bc.spec | 18 ++++----
.../tests/Functional.boost/Functional.boost.spec | 11 +++--
.../tests/Functional.bsdiff/Functional.bsdiff.spec | 10 ++---
.../tests/Functional.bzip2/Functional.bzip2.spec | 11 +++--
engine/tests/Functional.cmt/Functional.cmt.spec | 10 ++---
.../Functional.commonAPI_C++.spec | 10 ++---
.../Functional.commonAPI_Dbus.spec | 10 ++---
.../Functional.commonAPI_SomeIp.spec | 10 ++---
.../Functional.crashme/Functional.crashme.spec | 10 ++---
.../tests/Functional.croco/Functional.croco.spec | 10 ++---
engine/tests/Functional.curl/Functional.curl.spec | 11 +++--
.../tests/Functional.expat/Functional.expat.spec | 11 +++--
.../Functional.fixesproto.spec | 10 ++---
.../Functional.fontconfig.spec | 11 +++--
.../tests/Functional.fsfuzz/Functional.fsfuzz.spec | 10 ++---
.../Functional.ft2demos/Functional.ft2demos.spec | 11 +++--
.../Functional.fuego_abort.spec | 11 +++--
.../Functional.fuego_board_check.spec | 11 +++--
.../Functional.fuego_test_phases.spec | 11 +++--
.../Functional.fuego_transport.spec | 11 +++--
engine/tests/Functional.fuse/Functional.fuse.spec | 10 ++---
.../tests/Functional.giflib/Functional.giflib.spec | 10 ++---
engine/tests/Functional.glib/Functional.glib.spec | 11 +++--
.../tests/Functional.glib2/Functional.glib2.spec | 10 ++---
.../tests/Functional.glibc/Functional.glibc.spec | 10 ++---
.../Functional.hciattach/Functional.hciattach.spec | 10 ++---
.../Functional.hello_world.spec | 16 ++++---
.../Functional.imagemagick.spec | 10 ++---
.../Functional.iptables/Functional.iptables.spec | 10 ++---
.../Functional.iputils/Functional.iputils.spec | 10 ++---
.../Functional.ipv6connect.spec | 11 +++--
engine/tests/Functional.jpeg/Functional.jpeg.spec | 11 +++--
.../Functional.kernel_build.spec | 16 +++----
engine/tests/Functional.kmod/Functional.kmod.spec | 10 ++---
.../tests/Functional.libogg/Functional.libogg.spec | 10 ++---
.../Functional.libpcap/Functional.libpcap.spec | 10 ++---
.../Functional.librsvg/Functional.librsvg.spec | 10 ++---
.../Functional.libspeex/Functional.libspeex.spec | 10 ++---
.../tests/Functional.libtar/Functional.libtar.spec | 10 ++---
.../Functional.libwebsocket.spec | 10 ++---
.../Functional.linus_stress.spec | 10 ++---
engine/tests/Functional.lwip/Functional.lwip.spec | 10 ++---
.../Functional.mesa-demos.spec | 10 ++---
engine/tests/Functional.neon/Functional.neon.spec | 10 ++---
.../Functional.net-tools/Functional.net-tools.spec | 10 ++---
.../Functional.netperf/Functional.netperf.spec | 13 +++---
.../Functional.pi_tests/Functional.pi_tests.spec | 10 ++---
.../tests/Functional.pixman/Functional.pixman.spec | 10 ++---
engine/tests/Functional.pppd/Functional.pppd.spec | 10 ++---
.../Functional.protobuf/Functional.protobuf.spec | 10 ++---
.../Functional.rmaptest/Functional.rmaptest.spec | 10 ++---
.../tests/Functional.scifab/Functional.scifab.spec | 10 ++---
.../Functional.scrashme/Functional.scrashme.spec | 22 +++++-----
.../tests/Functional.sdhi_0/Functional.sdhi_0.spec | 10 ++---
.../tests/Functional.stress/Functional.stress.spec | 10 ++---
.../Functional.synctest/Functional.synctest.spec | 23 +++++-----
.../tests/Functional.thrift/Functional.thrift.spec | 10 ++---
engine/tests/Functional.tiff/Functional.tiff.spec | 10 ++---
.../Functional.vsomeip/Functional.vsomeip.spec | 10 ++---
.../Functional.xorg-macros.spec | 10 ++---
engine/tests/Functional.zlib/Functional.zlib.spec | 11 +++--
100 files changed, 633 insertions(+), 714 deletions(-)
diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 939043b..98cd7f6 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -788,14 +788,29 @@ def create_job(board, test):
else:
flot_link = ''
- # prepare links for descriptionsetter. Put a link to testlog.txt by default
+ # prepare links for the descriptionsetter plugin
+ test_spec_path = '/fuego-core/engine/tests/%s/%s.spec' % (test.name, test.name)
template_link = '<a href="/fuego/userContent/fuego.logs/%s/%s.%s.${BUILD_NUMBER}.${BUILD_ID}/%%s">%%s</a>' % (test.name, board, test.spec)
-
- html_links = template_link % ('testlog.txt', 'log')
- failed_links = html_links
- if test.extralinks:
- for key, value in test.extralinks.iteritems():
- html_links = html_links + ' ' + template_link % (str(value), str(key))
+ with open(test_spec_path, "r") as f:
+ test_specs_json = json.load(f)
+ test_spec_json = test_specs_json['specs'][test.spec]
+
+ success_links = ''
+ if 'success_links' in test_specs_json:
+ for key, value in test_specs_json['success_links'].iteritems():
+ success_links = success_links + ' ' + template_link % (str(value), str(key))
+ if 'extra_success_links' in test_spec_json:
+ for key, value in test_spec_json['extra_success_links'].iteritems():
+ success_links = success_links + ' ' + template_link % (str(value), str(key))
+
+ fail_links = ''
+ if 'fail_links' in test_specs_json:
+ for key, value in test_specs_json['fail_links'].iteritems():
+ fail_links = fail_links + ' ' + template_link % (str(value), str(key))
+
+ if 'extra_fail_links' in test_spec_json:
+ for key, value in test_spec_json['extra_fail_links'].iteritems():
+ fail_links = fail_links + ' ' + template_link % (str(value), str(key))
tmp = "/tmp/fuego_tmp_job"
fd = open(tmp, "w+")
@@ -833,8 +848,8 @@ timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
<hudson.plugins.descriptionsetter.DescriptionSetterPublisher plugin="description-setter@1.10">
<regexp></regexp>
<regexpForFailed></regexpForFailed>
- <description>{html_links}</description>
- <descriptionForFailed>{failed_links}</descriptionForFailed>
+ <description>{success_links}</description>
+ <descriptionForFailed>{fail_links}</descriptionForFailed>
<setForMatrix>false</setForMatrix>
</hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
</publishers>
@@ -844,7 +859,7 @@ timeout --signal=9 {timeout} /bin/bash $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
precleanup=test.precleanup, postcleanup=test.postcleanup,
testdir=test.name, testname=test.base_name,
testspec=test.spec, timeout=test.timeout,
- flot_link=flot_link, html_links=html_links, failed_links=failed_links))
+ flot_link=flot_link, success_links=success_links, fail_links=fail_links))
fd.close()
job_name=board+"."+test.spec+"."+test.name
@@ -960,7 +975,7 @@ def do_add_jobs(conf, options):
spec = options[options.index('-s') + 1]
except IndexError:
error_out('Testspec not provided after -s.')
- specnames = [s['name'] for s in get_specs(conf, test_name)]
+ specnames = get_specs(conf, test_name)
if spec not in specnames:
error_out('Unknown spec %s' % spec)
options.remove('-s')
@@ -1225,8 +1240,7 @@ def do_list_specs(conf, options):
else:
error_out('No testcase name supplied.')
- specs = get_specs(conf, testcase)
- specnames = [spec['name'] for spec in specs]
+ specnames = get_specs(conf, testcase).keys()
specnames.sort()
indent = show_list_title("Testspecs for %s:" % testcase)
diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
index f3f77e3..b331243 100755
--- a/engine/scripts/ovgen.py
+++ b/engine/scripts/ovgen.py
@@ -404,11 +404,9 @@ def generateProlog(outFilePath, ofcls, classes, testdir, testspec):
# testspec: String - name of test spec (e.g. 'default')
# outFilePath - file output descriptor opened -> prolog.sh
def generateSpec(ts, fout):
- debug_print("generating spec '%s' for '%s'" % (ts.variables['name'], ts.name))
+ debug_print("generating spec %s" % (ts.name))
for var in ts.variables:
- if var == 'name':
- continue
varname = "%s_%s" % (ts.name, var)
varname = string.replace(varname, ".", "_").upper()
value = "%s" % (ts.variables[var])
@@ -446,17 +444,15 @@ def parseSpec(testdir, testspec):
except:
raise Exception("Error parsing spec file %s" % specpath)
- ts.name = jd["testName"]
+ ts.name = jd["testName"]
- if "fail_case" in jd:
- ts.fail_case = jd["fail_case"]
- debug_print ("Found fail_case msgs in '%s'" % specpath)
+ if "fail_case" in jd:
+ ts.fail_case = jd["fail_case"]
+ debug_print ("Found fail_case msgs in '%s'" % specpath)
- for spec in jd["specs"]:
- if spec["name"] == testspec:
- ts.variables = spec
-
- if not ts.variables:
+ if testspec in jd["specs"]:
+ ts.variables = jd["specs"][testspec]
+ else:
raise Exception("Could not find %s in %s" % (testspec, specpath))
return ts
diff --git a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
index 794ca68..7ceee2a 100644
--- a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
+++ b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
@@ -1,25 +1,22 @@
- {
+{
"testName": "Benchmark.Dhrystone",
"fail_case": [
{
"fail_regexp": "Measured time too small to obtain meaningful results",
"fail_message": "Measured time too small to obtain meaningful results. Please increase LOOPS parameter in Dhrystone test spec."
- }
- ],
- "specs":
- [
- {
- "name":"default",
+ }
+ ],
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"LOOPS":"10000000"
},
- {
- "name":"100M",
+ "100M": {
"LOOPS":"100000000"
},
- {
- "name":"500M",
+ "500M": {
"LOOPS":"500000000"
}
- ]
+ }
}
-
diff --git a/engine/tests/Benchmark.GLMark/Benchmark.GLMark.spec b/engine/tests/Benchmark.GLMark/Benchmark.GLMark.spec
index d94129d..f23548f 100644
--- a/engine/tests/Benchmark.GLMark/Benchmark.GLMark.spec
+++ b/engine/tests/Benchmark.GLMark/Benchmark.GLMark.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.GLMark",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.IOzone/Benchmark.IOzone.spec b/engine/tests/Benchmark.IOzone/Benchmark.IOzone.spec
index 13ba89b..6799d5a 100644
--- a/engine/tests/Benchmark.IOzone/Benchmark.IOzone.spec
+++ b/engine/tests/Benchmark.IOzone/Benchmark.IOzone.spec
@@ -1,45 +1,37 @@
{
"testName": "Benchmark.IOzone",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png", "xlsx": "results.xlsx"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"FILE_SIZE":"2M",
"TESTS":"'-i 0 -i 1 -i 2 -i 6 -i 7 -i 8 -i 9'"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"FILE_SIZE":"2M",
"TESTS":"'-i 0 -i 1 -i 2 -i 6 -i 7 -i 8 -i 9'"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"FILE_SIZE":"2M",
"TESTS":"'-i 0 -i 1 -i 2 -i 6 -i 7 -i 8 -i 9'"
},
- {
- "name":"nopread",
+ "nopread": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"FILE_SIZE":"2M",
"TESTS":"'-i 0 -i 1 -i 2 -i 6 -i 7 -i 8'"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"FILE_SIZE":"2M",
"TESTS":"'-i 0 -i 1 -i 2 -i 6 -i 7 -i 8 -i 9'"
}
- ]
+ }
}
-
-
-
-
diff --git a/engine/tests/Benchmark.Interbench/Benchmark.Interbench.spec b/engine/tests/Benchmark.Interbench/Benchmark.Interbench.spec
index e3627bb..d9379b6 100644
--- a/engine/tests/Benchmark.Interbench/Benchmark.Interbench.spec
+++ b/engine/tests/Benchmark.Interbench/Benchmark.Interbench.spec
@@ -1,20 +1,11 @@
- {
+{
"testName": "Benchmark.Interbench",
- "specs":
- [
- {
- "name":"default"
- },
- {
- "name":"sata"
- },
- {
- "name":"usb"
- },
- {
- "name":"mmc"
- }
-
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {},
+ "sata": {},
+ "usb": {},
+ "mmc": {}
+ }
}
-
diff --git a/engine/tests/Benchmark.Java/Benchmark.Java.spec b/engine/tests/Benchmark.Java/Benchmark.Java.spec
index fdad1c9..f14f3d5 100644
--- a/engine/tests/Benchmark.Java/Benchmark.Java.spec
+++ b/engine/tests/Benchmark.Java/Benchmark.Java.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.Java",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec b/engine/tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec
index 7c00496..a835477 100644
--- a/engine/tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec
+++ b/engine/tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec
@@ -1,10 +1,8 @@
- {
+{
"testName": "Benchmark.OpenSSL",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
diff --git a/engine/tests/Benchmark.Stream/Benchmark.Stream.spec b/engine/tests/Benchmark.Stream/Benchmark.Stream.spec
index 2b88075..f4ba3df 100644
--- a/engine/tests/Benchmark.Stream/Benchmark.Stream.spec
+++ b/engine/tests/Benchmark.Stream/Benchmark.Stream.spec
@@ -1,10 +1,8 @@
- {
+{
"testName": "Benchmark.Stream",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
diff --git a/engine/tests/Benchmark.Whetstone/Benchmark.Whetstone.spec b/engine/tests/Benchmark.Whetstone/Benchmark.Whetstone.spec
index b755e31..84b0533 100644
--- a/engine/tests/Benchmark.Whetstone/Benchmark.Whetstone.spec
+++ b/engine/tests/Benchmark.Whetstone/Benchmark.Whetstone.spec
@@ -1,11 +1,11 @@
- {
+{
"testName": "Benchmark.Whetstone",
- "specs":
- [
- {
- "name":"default",
- "LOOPS":"100000"
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
+ "LOOPS": "100000"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.aim7/Benchmark.aim7.spec b/engine/tests/Benchmark.aim7/Benchmark.aim7.spec
index 4ed9154..1418b8c 100644
--- a/engine/tests/Benchmark.aim7/Benchmark.aim7.spec
+++ b/engine/tests/Benchmark.aim7/Benchmark.aim7.spec
@@ -1,8 +1,8 @@
{
"testName": "Benchmark.aim7",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Benchmark.blobsallad/Benchmark.blobsallad.spec b/engine/tests/Benchmark.blobsallad/Benchmark.blobsallad.spec
index a12b7e8..8f8d34b 100644
--- a/engine/tests/Benchmark.blobsallad/Benchmark.blobsallad.spec
+++ b/engine/tests/Benchmark.blobsallad/Benchmark.blobsallad.spec
@@ -1,16 +1,15 @@
- {
+{
"testName": "Benchmark.blobsallad",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
"fail_case": [
{
"fail_regexp": "xrandr: command not found",
"fail_message": "target system does not have xrandr"
}
- ],
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ ],
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.bonnie/Benchmark.bonnie.spec b/engine/tests/Benchmark.bonnie/Benchmark.bonnie.spec
index 426f94b..fb26016 100644
--- a/engine/tests/Benchmark.bonnie/Benchmark.bonnie.spec
+++ b/engine/tests/Benchmark.bonnie/Benchmark.bonnie.spec
@@ -1,47 +1,43 @@
{
"testName": "Benchmark.bonnie",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"SIZE":"32",
"RAM":"0",
"ROOT":"true"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"SIZE":"32",
"RAM":"0",
"ROOT":"true"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"SIZE":"32",
"RAM":"0",
"ROOT":"true"
},
- {
- "name":"noroot",
+ "noroot": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"SIZE":"32",
"RAM":"0",
"ROOT":"false"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"SIZE":"32",
"RAM":"0",
"ROOT":"true"
}
- ]
+ }
}
diff --git a/engine/tests/Benchmark.cyclictest/Benchmark.cyclictest.spec b/engine/tests/Benchmark.cyclictest/Benchmark.cyclictest.spec
index 617c1ac..b69f9ee 100644
--- a/engine/tests/Benchmark.cyclictest/Benchmark.cyclictest.spec
+++ b/engine/tests/Benchmark.cyclictest/Benchmark.cyclictest.spec
@@ -1,11 +1,10 @@
{
"testName": "Benchmark.cyclictest",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"LOOPS":"10000"
}
- ]
+ }
}
-
diff --git a/engine/tests/Benchmark.dbench/Benchmark.dbench.spec b/engine/tests/Benchmark.dbench/Benchmark.dbench.spec
index f7700ab..b3ce167 100644
--- a/engine/tests/Benchmark.dbench/Benchmark.dbench.spec
+++ b/engine/tests/Benchmark.dbench/Benchmark.dbench.spec
@@ -1,42 +1,37 @@
{
"testName": "Benchmark.dbench",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"TIMELIMIT":"10",
"NPROCS":"2"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"TIMELIMIT":"10",
"NPROCS":"2"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"TIMELIMIT":"10",
"NPROCS":"2"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"/a",
"TIMELIMIT":"10",
"NPROCS":"2"
},
- {
- "name":"testdir",
+ "testdir": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"TIMELIMIT":"10",
"NPROCS":"2"
}
- ]
+ }
}
-
diff --git a/engine/tests/Benchmark.ebizzy/Benchmark.ebizzy.spec b/engine/tests/Benchmark.ebizzy/Benchmark.ebizzy.spec
index 062fc1f..df7c299 100644
--- a/engine/tests/Benchmark.ebizzy/Benchmark.ebizzy.spec
+++ b/engine/tests/Benchmark.ebizzy/Benchmark.ebizzy.spec
@@ -1,14 +1,14 @@
{
"testName": "Benchmark.ebizzy",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"CHUNKS":"1000",
"CHUNK_SIZE":"51200",
"TIME":"100",
"THREADS":"100"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec b/engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec
index acff523..2656260 100644
--- a/engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec
+++ b/engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec
@@ -1,30 +1,23 @@
{
"testName": "Benchmark.ffsb",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work"
}
- ]
+ }
}
-
-
-
-
diff --git a/engine/tests/Benchmark.fio/Benchmark.fio.spec b/engine/tests/Benchmark.fio/Benchmark.fio.spec
index 947015d..c7cdd08 100644
--- a/engine/tests/Benchmark.fio/Benchmark.fio.spec
+++ b/engine/tests/Benchmark.fio/Benchmark.fio.spec
@@ -1,35 +1,28 @@
{
"testName": "Benchmark.fio",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"TIMEOUT":"10"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"TIMEOUT":"10"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"TIMEOUT":"10"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"TIMEOUT":"10"
}
- ]
+ }
}
-
-
-
-
diff --git a/engine/tests/Benchmark.fs_mark/Benchmark.fs_mark.spec b/engine/tests/Benchmark.fs_mark/Benchmark.fs_mark.spec
index 7a6623b..9960b76 100644
--- a/engine/tests/Benchmark.fs_mark/Benchmark.fs_mark.spec
+++ b/engine/tests/Benchmark.fs_mark/Benchmark.fs_mark.spec
@@ -1,8 +1,8 @@
{
"testName": "Benchmark.fs_mark",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Benchmark.fuego_check_plots/Benchmark.fuego_check_plots.spec b/engine/tests/Benchmark.fuego_check_plots/Benchmark.fuego_check_plots.spec
index 5ae2190..ba4265f 100644
--- a/engine/tests/Benchmark.fuego_check_plots/Benchmark.fuego_check_plots.spec
+++ b/engine/tests/Benchmark.fuego_check_plots/Benchmark.fuego_check_plots.spec
@@ -1,13 +1,8 @@
{
"testName": "Benchmark.fuego_check_plots",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
-
-
-
diff --git a/engine/tests/Benchmark.gtkperf/Benchmark.gtkperf.spec b/engine/tests/Benchmark.gtkperf/Benchmark.gtkperf.spec
index 9ec9d5f..c27b286 100644
--- a/engine/tests/Benchmark.gtkperf/Benchmark.gtkperf.spec
+++ b/engine/tests/Benchmark.gtkperf/Benchmark.gtkperf.spec
@@ -1,16 +1,15 @@
- {
+{
"testName": "Benchmark.gtkperf",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
"fail_case": [
{
"fail_regexp": "xrandr: command not found",
"fail_message": "target system does not have xrandr"
}
- ],
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ ],
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.hackbench/Benchmark.hackbench.spec b/engine/tests/Benchmark.hackbench/Benchmark.hackbench.spec
index f915796..e982ea4 100644
--- a/engine/tests/Benchmark.hackbench/Benchmark.hackbench.spec
+++ b/engine/tests/Benchmark.hackbench/Benchmark.hackbench.spec
@@ -1,11 +1,11 @@
{
"testName": "Benchmark.hackbench",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"GPOUPS":"20"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.himeno/Benchmark.himeno.spec b/engine/tests/Benchmark.himeno/Benchmark.himeno.spec
index 4cb58d6..3d973a0 100644
--- a/engine/tests/Benchmark.himeno/Benchmark.himeno.spec
+++ b/engine/tests/Benchmark.himeno/Benchmark.himeno.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.himeno",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.iperf/Benchmark.iperf.spec b/engine/tests/Benchmark.iperf/Benchmark.iperf.spec
index e4429f4..417393e 100644
--- a/engine/tests/Benchmark.iperf/Benchmark.iperf.spec
+++ b/engine/tests/Benchmark.iperf/Benchmark.iperf.spec
@@ -1,11 +1,11 @@
{
"testName": "Benchmark.iperf",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"SRV":"default"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.linpack/Benchmark.linpack.spec b/engine/tests/Benchmark.linpack/Benchmark.linpack.spec
index 9e1ca86..e1e61da 100644
--- a/engine/tests/Benchmark.linpack/Benchmark.linpack.spec
+++ b/engine/tests/Benchmark.linpack/Benchmark.linpack.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.linpack",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.lmbench2/Benchmark.lmbench2.spec b/engine/tests/Benchmark.lmbench2/Benchmark.lmbench2.spec
index f1055c4..3f8ffe5 100644
--- a/engine/tests/Benchmark.lmbench2/Benchmark.lmbench2.spec
+++ b/engine/tests/Benchmark.lmbench2/Benchmark.lmbench2.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.lmbench2",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.nbench-byte/Benchmark.nbench-byte.spec b/engine/tests/Benchmark.nbench-byte/Benchmark.nbench-byte.spec
index 319b42d..fd31900 100644
--- a/engine/tests/Benchmark.nbench-byte/Benchmark.nbench-byte.spec
+++ b/engine/tests/Benchmark.nbench-byte/Benchmark.nbench-byte.spec
@@ -1,8 +1,8 @@
{
"testName": "Benchmark.nbench-byte",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Benchmark.nbench_byte/Benchmark.nbench_byte.spec b/engine/tests/Benchmark.nbench_byte/Benchmark.nbench_byte.spec
index 2dab856..5be7564 100644
--- a/engine/tests/Benchmark.nbench_byte/Benchmark.nbench_byte.spec
+++ b/engine/tests/Benchmark.nbench_byte/Benchmark.nbench_byte.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.nbench_byte",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.netperf/Benchmark.netperf.spec b/engine/tests/Benchmark.netperf/Benchmark.netperf.spec
index eac8fcd..afef247 100644
--- a/engine/tests/Benchmark.netperf/Benchmark.netperf.spec
+++ b/engine/tests/Benchmark.netperf/Benchmark.netperf.spec
@@ -1,14 +1,13 @@
{
"testName": "Benchmark.netperf",
- "specs":
- [
- {
- "name":"default",
- "SRV":"default"
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
+ "SRV": "default"
},
- {
- "name":"docker",
- "SRV":"docker"
+ "docker": {
+ "SRV": "docker"
}
- ]
+ }
}
diff --git a/engine/tests/Benchmark.netpipe/Benchmark.netpipe.spec b/engine/tests/Benchmark.netpipe/Benchmark.netpipe.spec
index 0ed6717..0ca0bed 100644
--- a/engine/tests/Benchmark.netpipe/Benchmark.netpipe.spec
+++ b/engine/tests/Benchmark.netpipe/Benchmark.netpipe.spec
@@ -1,12 +1,12 @@
{
"testName": "Benchmark.netpipe",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"SRV":"default",
"PERT":"2"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.reboot/Benchmark.reboot.spec b/engine/tests/Benchmark.reboot/Benchmark.reboot.spec
index c67f4a5..35a0631 100644
--- a/engine/tests/Benchmark.reboot/Benchmark.reboot.spec
+++ b/engine/tests/Benchmark.reboot/Benchmark.reboot.spec
@@ -1,10 +1,9 @@
- {
+{
"testName": "Benchmark.reboot",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
-
+
diff --git a/engine/tests/Benchmark.signaltest/Benchmark.signaltest.spec b/engine/tests/Benchmark.signaltest/Benchmark.signaltest.spec
index 0b2f09d..708537e 100644
--- a/engine/tests/Benchmark.signaltest/Benchmark.signaltest.spec
+++ b/engine/tests/Benchmark.signaltest/Benchmark.signaltest.spec
@@ -1,11 +1,11 @@
- {
+{
"testName": "Benchmark.signaltest",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"LOOPS":"10000"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Benchmark.sysbench/Benchmark.sysbench.spec b/engine/tests/Benchmark.sysbench/Benchmark.sysbench.spec
index ffe4534..cae3202 100644
--- a/engine/tests/Benchmark.sysbench/Benchmark.sysbench.spec
+++ b/engine/tests/Benchmark.sysbench/Benchmark.sysbench.spec
@@ -1,8 +1,8 @@
{
"testName": "Benchmark.sysbench",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Benchmark.tiobench/Benchmark.tiobench.spec b/engine/tests/Benchmark.tiobench/Benchmark.tiobench.spec
index e38a7d7..9faea9b 100644
--- a/engine/tests/Benchmark.tiobench/Benchmark.tiobench.spec
+++ b/engine/tests/Benchmark.tiobench/Benchmark.tiobench.spec
@@ -1,36 +1,33 @@
{
"testName": "Benchmark.tiobench",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"THREADS":"4",
"SIZE":"10"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"THREADS":"4",
"SIZE":"10"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"THREADS":"4",
"SIZE":"10"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"THREADS":"4",
"SIZE":"10"
}
- ]
+ }
}
diff --git a/engine/tests/Benchmark.x11perf/Benchmark.x11perf.spec b/engine/tests/Benchmark.x11perf/Benchmark.x11perf.spec
index 602bc2d..abeffd2 100644
--- a/engine/tests/Benchmark.x11perf/Benchmark.x11perf.spec
+++ b/engine/tests/Benchmark.x11perf/Benchmark.x11perf.spec
@@ -1,11 +1,11 @@
{
"testName": "Benchmark.x11perf",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt", "plot": "plot.png"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"TIME":"1"
}
- ]
+ }
}
-
+
diff --git a/engine/tests/Functional.LTP/Functional.LTP.spec b/engine/tests/Functional.LTP/Functional.LTP.spec
index 1c1a090..06c037e 100644
--- a/engine/tests/Functional.LTP/Functional.LTP.spec
+++ b/engine/tests/Functional.LTP/Functional.LTP.spec
@@ -1,34 +1,35 @@
{
"testName": "Functional.LTP",
- "specs":
- [
- {
- "name": "default",
- "tests": "syscalls SEM"
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
+ "tests": "syscalls SEM",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "selection",
- "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS"
+ "selection": {
+ "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "selectionwithrt",
- "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter"
+ "selectionwithrt": {
+ "tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "ltplite",
- "tests": "ltplite"
+ "ltplite": {
+ "tests": "ltplite",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "ptsonly",
- "tests": "AIO MEM MSG SEM SIG THR TMR TPS"
+ "ptsonly": {
+ "tests": "AIO MEM MSG SEM SIG THR TMR TPS",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "smoketest",
- "tests": "fs_bind"
+ "smoketest": {
+ "tests": "fs_bind",
+ "extra_success_links": {"xlsx": "results.xlsx"}
},
- {
- "name": "rtonly",
- "tests": "func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter"
+ "rtonly": {
+ "tests": "func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter",
+ "extra_success_links": {"xlsx": "results.xlsx"}
}
- ]
+ }
}
diff --git a/engine/tests/Functional.OpenSSL/Functional.OpenSSL.spec b/engine/tests/Functional.OpenSSL/Functional.OpenSSL.spec
index d3e4df0..0f535c8 100644
--- a/engine/tests/Functional.OpenSSL/Functional.OpenSSL.spec
+++ b/engine/tests/Functional.OpenSSL/Functional.OpenSSL.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.OpenSSL",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.aiostress/Functional.aiostress.spec b/engine/tests/Functional.aiostress/Functional.aiostress.spec
index d41f4a0..7eb37f9 100644
--- a/engine/tests/Functional.aiostress/Functional.aiostress.spec
+++ b/engine/tests/Functional.aiostress/Functional.aiostress.spec
@@ -1,31 +1,27 @@
{
"testName": "Functional.aiostress",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"SIZE":"20M"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"SIZE":"20M"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"SIZE":"20M"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"SIZE":"20M"
}
- ]
+ }
}
-
diff --git a/engine/tests/Functional.arch_timer/Functional.arch_timer.spec b/engine/tests/Functional.arch_timer/Functional.arch_timer.spec
index 3d28603..6206476 100644
--- a/engine/tests/Functional.arch_timer/Functional.arch_timer.spec
+++ b/engine/tests/Functional.arch_timer/Functional.arch_timer.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.arch_timer",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.bc/Functional.bc.spec b/engine/tests/Functional.bc/Functional.bc.spec
index 9b22ea6..913fc3b 100644
--- a/engine/tests/Functional.bc/Functional.bc.spec
+++ b/engine/tests/Functional.bc/Functional.bc.spec
@@ -10,24 +10,22 @@
"fail_message": "Bug or Oops detected in system log",
"use_syslog": "1"
}
- ],
- "specs":
- [
- {
- "name":"bc-mult",
+ ],
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "bc-mult": {
"EXPR":"2*2",
"RESULT": "4"
},
- {
- "name":"bc-add",
+ "bc-add": {
"EXPR":"3+3",
"RESULT":"6"
},
- {
- "name":"default",
+ "default": {
"EXPR":"3+3",
"RESULT":"6"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.boost/Functional.boost.spec b/engine/tests/Functional.boost/Functional.boost.spec
index b25b352..f6b77b9 100644
--- a/engine/tests/Functional.boost/Functional.boost.spec
+++ b/engine/tests/Functional.boost/Functional.boost.spec
@@ -1,10 +1,9 @@
{
"testName": "Functional.boost",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.bsdiff/Functional.bsdiff.spec b/engine/tests/Functional.bsdiff/Functional.bsdiff.spec
index 538beee..443160d 100644
--- a/engine/tests/Functional.bsdiff/Functional.bsdiff.spec
+++ b/engine/tests/Functional.bsdiff/Functional.bsdiff.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.bsdiff",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.bzip2/Functional.bzip2.spec b/engine/tests/Functional.bzip2/Functional.bzip2.spec
index 9350805..eaf1645 100644
--- a/engine/tests/Functional.bzip2/Functional.bzip2.spec
+++ b/engine/tests/Functional.bzip2/Functional.bzip2.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.bzip2",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.cmt/Functional.cmt.spec b/engine/tests/Functional.cmt/Functional.cmt.spec
index d88d487..ea54fdb 100644
--- a/engine/tests/Functional.cmt/Functional.cmt.spec
+++ b/engine/tests/Functional.cmt/Functional.cmt.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.cmt",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.commonAPI_C++/Functional.commonAPI_C++.spec b/engine/tests/Functional.commonAPI_C++/Functional.commonAPI_C++.spec
index 401ce13..a31132c 100644
--- a/engine/tests/Functional.commonAPI_C++/Functional.commonAPI_C++.spec
+++ b/engine/tests/Functional.commonAPI_C++/Functional.commonAPI_C++.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.commonAPI_C++",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.commonAPI_Dbus/Functional.commonAPI_Dbus.spec b/engine/tests/Functional.commonAPI_Dbus/Functional.commonAPI_Dbus.spec
index 699e591..9c0d366 100644
--- a/engine/tests/Functional.commonAPI_Dbus/Functional.commonAPI_Dbus.spec
+++ b/engine/tests/Functional.commonAPI_Dbus/Functional.commonAPI_Dbus.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.commonAPI_Dbus",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.commonAPI_SomeIp/Functional.commonAPI_SomeIp.spec b/engine/tests/Functional.commonAPI_SomeIp/Functional.commonAPI_SomeIp.spec
index 4ca9398..73d8ba5 100644
--- a/engine/tests/Functional.commonAPI_SomeIp/Functional.commonAPI_SomeIp.spec
+++ b/engine/tests/Functional.commonAPI_SomeIp/Functional.commonAPI_SomeIp.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.commonAPI_SomeIp",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.crashme/Functional.crashme.spec b/engine/tests/Functional.crashme/Functional.crashme.spec
index 1793be3..e6978d3 100644
--- a/engine/tests/Functional.crashme/Functional.crashme.spec
+++ b/engine/tests/Functional.crashme/Functional.crashme.spec
@@ -1,14 +1,14 @@
{
"testName": "Functional.crashme",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"NBYTES":"1000",
"INC":"1000",
"SRAND":"2",
"NTRYS":"100",
"NSUB":"3000"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.croco/Functional.croco.spec b/engine/tests/Functional.croco/Functional.croco.spec
index bd36f60..9e63072 100644
--- a/engine/tests/Functional.croco/Functional.croco.spec
+++ b/engine/tests/Functional.croco/Functional.croco.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.croco",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.curl/Functional.curl.spec b/engine/tests/Functional.curl/Functional.curl.spec
index ee8160a..5016c98 100644
--- a/engine/tests/Functional.curl/Functional.curl.spec
+++ b/engine/tests/Functional.curl/Functional.curl.spec
@@ -1,10 +1,9 @@
{
"testName": "Functional.curl",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.expat/Functional.expat.spec b/engine/tests/Functional.expat/Functional.expat.spec
index 2edd1b6..7804c78 100644
--- a/engine/tests/Functional.expat/Functional.expat.spec
+++ b/engine/tests/Functional.expat/Functional.expat.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.expat",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fixesproto/Functional.fixesproto.spec b/engine/tests/Functional.fixesproto/Functional.fixesproto.spec
index 2f9ea91..9b78b18 100644
--- a/engine/tests/Functional.fixesproto/Functional.fixesproto.spec
+++ b/engine/tests/Functional.fixesproto/Functional.fixesproto.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.fixesproto",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fontconfig/Functional.fontconfig.spec b/engine/tests/Functional.fontconfig/Functional.fontconfig.spec
index 1237d46..dae7012 100644
--- a/engine/tests/Functional.fontconfig/Functional.fontconfig.spec
+++ b/engine/tests/Functional.fontconfig/Functional.fontconfig.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.fontconfig",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fsfuzz/Functional.fsfuzz.spec b/engine/tests/Functional.fsfuzz/Functional.fsfuzz.spec
index 1fb7446..abcba48 100644
--- a/engine/tests/Functional.fsfuzz/Functional.fsfuzz.spec
+++ b/engine/tests/Functional.fsfuzz/Functional.fsfuzz.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.fsfuzz",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.ft2demos/Functional.ft2demos.spec b/engine/tests/Functional.ft2demos/Functional.ft2demos.spec
index edce877..5577939 100644
--- a/engine/tests/Functional.ft2demos/Functional.ft2demos.spec
+++ b/engine/tests/Functional.ft2demos/Functional.ft2demos.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.ft2demos",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fuego_abort/Functional.fuego_abort.spec b/engine/tests/Functional.fuego_abort/Functional.fuego_abort.spec
index 47c6eda..bb9f075 100644
--- a/engine/tests/Functional.fuego_abort/Functional.fuego_abort.spec
+++ b/engine/tests/Functional.fuego_abort/Functional.fuego_abort.spec
@@ -1,10 +1,9 @@
{
"testName": "Functional.fuego_test_phases",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fuego_board_check/Functional.fuego_board_check.spec b/engine/tests/Functional.fuego_board_check/Functional.fuego_board_check.spec
index 05ff8e6..1e8fa72 100644
--- a/engine/tests/Functional.fuego_board_check/Functional.fuego_board_check.spec
+++ b/engine/tests/Functional.fuego_board_check/Functional.fuego_board_check.spec
@@ -1,11 +1,10 @@
{
"testName": "Functional.fuego_board_check",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fuego_test_phases/Functional.fuego_test_phases.spec b/engine/tests/Functional.fuego_test_phases/Functional.fuego_test_phases.spec
index 47c6eda..bb9f075 100644
--- a/engine/tests/Functional.fuego_test_phases/Functional.fuego_test_phases.spec
+++ b/engine/tests/Functional.fuego_test_phases/Functional.fuego_test_phases.spec
@@ -1,10 +1,9 @@
{
"testName": "Functional.fuego_test_phases",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fuego_transport/Functional.fuego_transport.spec b/engine/tests/Functional.fuego_transport/Functional.fuego_transport.spec
index a4fae24..9618c54 100644
--- a/engine/tests/Functional.fuego_transport/Functional.fuego_transport.spec
+++ b/engine/tests/Functional.fuego_transport/Functional.fuego_transport.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.fuego_transport",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.fuse/Functional.fuse.spec b/engine/tests/Functional.fuse/Functional.fuse.spec
index 11a0ffc..67b4d53 100644
--- a/engine/tests/Functional.fuse/Functional.fuse.spec
+++ b/engine/tests/Functional.fuse/Functional.fuse.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.fuse",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.giflib/Functional.giflib.spec b/engine/tests/Functional.giflib/Functional.giflib.spec
index 8143cf4..15187da 100644
--- a/engine/tests/Functional.giflib/Functional.giflib.spec
+++ b/engine/tests/Functional.giflib/Functional.giflib.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.giflib",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.glib/Functional.glib.spec b/engine/tests/Functional.glib/Functional.glib.spec
index 1266660..d70fec0 100644
--- a/engine/tests/Functional.glib/Functional.glib.spec
+++ b/engine/tests/Functional.glib/Functional.glib.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.glib",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.glib2/Functional.glib2.spec b/engine/tests/Functional.glib2/Functional.glib2.spec
index 15bd79f..2629f04 100644
--- a/engine/tests/Functional.glib2/Functional.glib2.spec
+++ b/engine/tests/Functional.glib2/Functional.glib2.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.glib2",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.glibc/Functional.glibc.spec b/engine/tests/Functional.glibc/Functional.glibc.spec
index f687253..9fac55a 100644
--- a/engine/tests/Functional.glibc/Functional.glibc.spec
+++ b/engine/tests/Functional.glibc/Functional.glibc.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.glibc",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.hciattach/Functional.hciattach.spec b/engine/tests/Functional.hciattach/Functional.hciattach.spec
index 536da9b..ec4eb25 100644
--- a/engine/tests/Functional.hciattach/Functional.hciattach.spec
+++ b/engine/tests/Functional.hciattach/Functional.hciattach.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.hciattach",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.hello_world/Functional.hello_world.spec b/engine/tests/Functional.hello_world/Functional.hello_world.spec
index 4e1bea5..8ca7e67 100644
--- a/engine/tests/Functional.hello_world/Functional.hello_world.spec
+++ b/engine/tests/Functional.hello_world/Functional.hello_world.spec
@@ -1,19 +1,17 @@
{
"testName": "Functional.hello_world",
- "specs":
- [
- {
- "name":"hello-fail",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "hello-fail": {
"ARG":"-f"
},
- {
- "name":"hello-random",
+ "hello-random": {
"ARG":"-r"
},
- {
- "name":"default",
+ "default": {
"ARG":""
}
- ]
+ }
}
diff --git a/engine/tests/Functional.imagemagick/Functional.imagemagick.spec b/engine/tests/Functional.imagemagick/Functional.imagemagick.spec
index 0c9babc..667b9e7 100644
--- a/engine/tests/Functional.imagemagick/Functional.imagemagick.spec
+++ b/engine/tests/Functional.imagemagick/Functional.imagemagick.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.imagemagick",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.iptables/Functional.iptables.spec b/engine/tests/Functional.iptables/Functional.iptables.spec
index 101d26b..37a758c 100644
--- a/engine/tests/Functional.iptables/Functional.iptables.spec
+++ b/engine/tests/Functional.iptables/Functional.iptables.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.iptables",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.iputils/Functional.iputils.spec b/engine/tests/Functional.iputils/Functional.iputils.spec
index 2dc83c7..a347cc6 100644
--- a/engine/tests/Functional.iputils/Functional.iputils.spec
+++ b/engine/tests/Functional.iputils/Functional.iputils.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.iputils",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.ipv6connect/Functional.ipv6connect.spec b/engine/tests/Functional.ipv6connect/Functional.ipv6connect.spec
index 0c425b7..61f7bf4 100644
--- a/engine/tests/Functional.ipv6connect/Functional.ipv6connect.spec
+++ b/engine/tests/Functional.ipv6connect/Functional.ipv6connect.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.ipv6connect",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.jpeg/Functional.jpeg.spec b/engine/tests/Functional.jpeg/Functional.jpeg.spec
index a32bc0e..87598c9 100644
--- a/engine/tests/Functional.jpeg/Functional.jpeg.spec
+++ b/engine/tests/Functional.jpeg/Functional.jpeg.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.jpeg",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.kernel_build/Functional.kernel_build.spec b/engine/tests/Functional.kernel_build/Functional.kernel_build.spec
index 39914a8..291979c 100644
--- a/engine/tests/Functional.kernel_build/Functional.kernel_build.spec
+++ b/engine/tests/Functional.kernel_build/Functional.kernel_build.spec
@@ -1,18 +1,20 @@
{
"testName": "Functional.kernel_build",
- "specs":
- [
- {
- "name": "cip-x86",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
+ "repo": "https://github.com/torvalds/linux.git"
+ },
+ "cip-x86": {
"repo": "https://github.com/cip-project/linux-cip.git"
},
- {
- "name": "cip-arm64",
+ "cip-arm64": {
"repo": "https://github.com/cip-project/linux-cip.git",
"platform": "aarch64",
"arch": "arm64",
"params": "-j8 Image dtbs modules",
"regex_p": "^ OBJCOPY arch/arm64/boot/Image"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.kmod/Functional.kmod.spec b/engine/tests/Functional.kmod/Functional.kmod.spec
index 33df4b1..648dcbe 100644
--- a/engine/tests/Functional.kmod/Functional.kmod.spec
+++ b/engine/tests/Functional.kmod/Functional.kmod.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.kmod",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.libogg/Functional.libogg.spec b/engine/tests/Functional.libogg/Functional.libogg.spec
index 8b5b2b8..a9badbc 100644
--- a/engine/tests/Functional.libogg/Functional.libogg.spec
+++ b/engine/tests/Functional.libogg/Functional.libogg.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.libogg",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.libpcap/Functional.libpcap.spec b/engine/tests/Functional.libpcap/Functional.libpcap.spec
index 3130092..c3b1ad8 100644
--- a/engine/tests/Functional.libpcap/Functional.libpcap.spec
+++ b/engine/tests/Functional.libpcap/Functional.libpcap.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.libpcap",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.librsvg/Functional.librsvg.spec b/engine/tests/Functional.librsvg/Functional.librsvg.spec
index 0a15abe..2190d74 100644
--- a/engine/tests/Functional.librsvg/Functional.librsvg.spec
+++ b/engine/tests/Functional.librsvg/Functional.librsvg.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.librsvg",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.libspeex/Functional.libspeex.spec b/engine/tests/Functional.libspeex/Functional.libspeex.spec
index a8b066c..c08cf3e 100644
--- a/engine/tests/Functional.libspeex/Functional.libspeex.spec
+++ b/engine/tests/Functional.libspeex/Functional.libspeex.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.libspeex",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.libtar/Functional.libtar.spec b/engine/tests/Functional.libtar/Functional.libtar.spec
index 1af026b..1c7c1f9 100644
--- a/engine/tests/Functional.libtar/Functional.libtar.spec
+++ b/engine/tests/Functional.libtar/Functional.libtar.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.libtar",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.libwebsocket/Functional.libwebsocket.spec b/engine/tests/Functional.libwebsocket/Functional.libwebsocket.spec
index f1e6178..28e1070 100644
--- a/engine/tests/Functional.libwebsocket/Functional.libwebsocket.spec
+++ b/engine/tests/Functional.libwebsocket/Functional.libwebsocket.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.libwebsocket",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.linus_stress/Functional.linus_stress.spec b/engine/tests/Functional.linus_stress/Functional.linus_stress.spec
index 8894892..0554900 100644
--- a/engine/tests/Functional.linus_stress/Functional.linus_stress.spec
+++ b/engine/tests/Functional.linus_stress/Functional.linus_stress.spec
@@ -1,11 +1,11 @@
{
"testName": "Functional.linus_stress",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"DIRTY_BG":"2",
"DIRTY":"4"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.lwip/Functional.lwip.spec b/engine/tests/Functional.lwip/Functional.lwip.spec
index bb9f9b3..b2d8f04 100644
--- a/engine/tests/Functional.lwip/Functional.lwip.spec
+++ b/engine/tests/Functional.lwip/Functional.lwip.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.lwip",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.mesa-demos/Functional.mesa-demos.spec b/engine/tests/Functional.mesa-demos/Functional.mesa-demos.spec
index 0712e09..2265e37 100644
--- a/engine/tests/Functional.mesa-demos/Functional.mesa-demos.spec
+++ b/engine/tests/Functional.mesa-demos/Functional.mesa-demos.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.mesa-demos",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.neon/Functional.neon.spec b/engine/tests/Functional.neon/Functional.neon.spec
index 348b913..6bcc08c 100644
--- a/engine/tests/Functional.neon/Functional.neon.spec
+++ b/engine/tests/Functional.neon/Functional.neon.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.neon",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.net-tools/Functional.net-tools.spec b/engine/tests/Functional.net-tools/Functional.net-tools.spec
index f702fa6..fed0b86 100644
--- a/engine/tests/Functional.net-tools/Functional.net-tools.spec
+++ b/engine/tests/Functional.net-tools/Functional.net-tools.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.net-tools",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.netperf/Functional.netperf.spec b/engine/tests/Functional.netperf/Functional.netperf.spec
index 2838d34..12313c8 100644
--- a/engine/tests/Functional.netperf/Functional.netperf.spec
+++ b/engine/tests/Functional.netperf/Functional.netperf.spec
@@ -1,14 +1,13 @@
{
"testName": "Functional.netperf",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"SRV":"default"
},
- {
- "name":"docker",
+ "docker": {
"SRV":"docker"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.pi_tests/Functional.pi_tests.spec b/engine/tests/Functional.pi_tests/Functional.pi_tests.spec
index 2971959..df10f17 100644
--- a/engine/tests/Functional.pi_tests/Functional.pi_tests.spec
+++ b/engine/tests/Functional.pi_tests/Functional.pi_tests.spec
@@ -1,12 +1,12 @@
{
"testName": "Functional.pi_tests",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"INVERSIONS":"100",
"GROUPS":"1",
"TIMEOUT":"10"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.pixman/Functional.pixman.spec b/engine/tests/Functional.pixman/Functional.pixman.spec
index 1cd16f8..e98c266 100644
--- a/engine/tests/Functional.pixman/Functional.pixman.spec
+++ b/engine/tests/Functional.pixman/Functional.pixman.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.pixman",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.pppd/Functional.pppd.spec b/engine/tests/Functional.pppd/Functional.pppd.spec
index d1cbc93..ca7f833 100644
--- a/engine/tests/Functional.pppd/Functional.pppd.spec
+++ b/engine/tests/Functional.pppd/Functional.pppd.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.pppd",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.protobuf/Functional.protobuf.spec b/engine/tests/Functional.protobuf/Functional.protobuf.spec
index a04b726..6e682c4 100644
--- a/engine/tests/Functional.protobuf/Functional.protobuf.spec
+++ b/engine/tests/Functional.protobuf/Functional.protobuf.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.protobuf",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.rmaptest/Functional.rmaptest.spec b/engine/tests/Functional.rmaptest/Functional.rmaptest.spec
index d19b99f..0ee7980 100644
--- a/engine/tests/Functional.rmaptest/Functional.rmaptest.spec
+++ b/engine/tests/Functional.rmaptest/Functional.rmaptest.spec
@@ -1,14 +1,14 @@
{
"testName": "Functional.rmaptest",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"ITERATIONS":"100",
"VMA":"100",
"VMA_SIZE":"100",
"TASKS":"100",
"VMAS_FOR_PROCESS":"10"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.scifab/Functional.scifab.spec b/engine/tests/Functional.scifab/Functional.scifab.spec
index 0ae5344..467441e 100644
--- a/engine/tests/Functional.scifab/Functional.scifab.spec
+++ b/engine/tests/Functional.scifab/Functional.scifab.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.scifab",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.scrashme/Functional.scrashme.spec b/engine/tests/Functional.scrashme/Functional.scrashme.spec
index 5065506..689598d 100644
--- a/engine/tests/Functional.scrashme/Functional.scrashme.spec
+++ b/engine/tests/Functional.scrashme/Functional.scrashme.spec
@@ -1,21 +1,19 @@
{
"testName": "Functional.scrashme",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"NUM":"100",
- "MODE":"random"
+ "MODE":"random"
},
- {
- "name":"rotate",
+ "rotate": {
"NUM":"100",
- "MODE":"rotate"
+ "MODE":"rotate"
},
- {
- "name":"capcheck",
+ "capcheck": {
"NUM":"100",
- "MODE":"capcheck"
+ "MODE":"capcheck"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.sdhi_0/Functional.sdhi_0.spec b/engine/tests/Functional.sdhi_0/Functional.sdhi_0.spec
index c0f134e..729205a 100644
--- a/engine/tests/Functional.sdhi_0/Functional.sdhi_0.spec
+++ b/engine/tests/Functional.sdhi_0/Functional.sdhi_0.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.sdhi_0",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.stress/Functional.stress.spec b/engine/tests/Functional.stress/Functional.stress.spec
index 5a40f31..da91cee 100644
--- a/engine/tests/Functional.stress/Functional.stress.spec
+++ b/engine/tests/Functional.stress/Functional.stress.spec
@@ -1,9 +1,9 @@
{
"testName": "Functional.stress",
- "specs":
- [
- {
- "name":"default",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {
"SYNC":"2",
"SQRT":"2",
"HDD":"2",
@@ -12,5 +12,5 @@
"VM_BYTES":"128M",
"TIMEOUT":"60"
}
- ]
+ }
}
diff --git a/engine/tests/Functional.synctest/Functional.synctest.spec b/engine/tests/Functional.synctest/Functional.synctest.spec
index 4d920b9..d7056e1 100644
--- a/engine/tests/Functional.synctest/Functional.synctest.spec
+++ b/engine/tests/Functional.synctest/Functional.synctest.spec
@@ -1,38 +1,35 @@
{
"testName": "Functional.synctest",
- "specs":
- [
- {
- "name":"sata",
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "sata": {
"MOUNT_BLOCKDEV":"$SATA_DEV",
"MOUNT_POINT":"$SATA_MP",
"LEN":"10",
"LOOP":"10"
},
- {
- "name":"mmc",
+ "mmc": {
"MOUNT_BLOCKDEV":"$MMC_DEV",
"MOUNT_POINT":"$MMC_MP",
"LEN":"10",
"LOOP":"10"
},
- {
- "name":"usb",
+ "usb": {
"MOUNT_BLOCKDEV":"$USB_DEV",
"MOUNT_POINT":"$USB_MP",
"LEN":"10",
"LOOP":"10"
},
- {
- "name":"default",
+ "default": {
"MOUNT_BLOCKDEV":"ROOT",
"MOUNT_POINT":"$BOARD_TESTDIR/work",
"LEN":"10",
"LOOP":"10"
}
- ]
+ }
}
-
-
+
+
diff --git a/engine/tests/Functional.thrift/Functional.thrift.spec b/engine/tests/Functional.thrift/Functional.thrift.spec
index 860aea2..3df6d07 100644
--- a/engine/tests/Functional.thrift/Functional.thrift.spec
+++ b/engine/tests/Functional.thrift/Functional.thrift.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.thrift",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.tiff/Functional.tiff.spec b/engine/tests/Functional.tiff/Functional.tiff.spec
index e7d3736..68c8d6b 100644
--- a/engine/tests/Functional.tiff/Functional.tiff.spec
+++ b/engine/tests/Functional.tiff/Functional.tiff.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.tiff",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.vsomeip/Functional.vsomeip.spec b/engine/tests/Functional.vsomeip/Functional.vsomeip.spec
index 61022d2..7f1fabf 100644
--- a/engine/tests/Functional.vsomeip/Functional.vsomeip.spec
+++ b/engine/tests/Functional.vsomeip/Functional.vsomeip.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.vsomeip",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.xorg-macros/Functional.xorg-macros.spec b/engine/tests/Functional.xorg-macros/Functional.xorg-macros.spec
index e1b83f7..036c438 100644
--- a/engine/tests/Functional.xorg-macros/Functional.xorg-macros.spec
+++ b/engine/tests/Functional.xorg-macros/Functional.xorg-macros.spec
@@ -1,8 +1,8 @@
{
"testName": "Functional.xorg-macros",
- "specs": [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
diff --git a/engine/tests/Functional.zlib/Functional.zlib.spec b/engine/tests/Functional.zlib/Functional.zlib.spec
index be3ed34..37308e9 100644
--- a/engine/tests/Functional.zlib/Functional.zlib.spec
+++ b/engine/tests/Functional.zlib/Functional.zlib.spec
@@ -1,9 +1,8 @@
{
"testName": "Functional.zlib",
- "specs":
- [
- {
- "name":"default"
- }
- ]
+ "success_links": {"log": "testlog.txt"},
+ "fail_links": {"log": "testlog.txt"},
+ "specs": {
+ "default": {}
+ }
}
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Fuego] [PATCH 2/3] defaults: handle defaults in a more efficient way
2017-04-19 8:32 [Fuego] Changes to the specs and separate LTP build and run Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH] cross: add libaio dependency for building LTP Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key Daniel Sangorrin
@ 2017-04-19 8:32 ` Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 3/3] LTP: support separate phases for build and run Daniel Sangorrin
3 siblings, 0 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-19 8:32 UTC (permalink / raw)
To: fuego
This patch allows overwritting the build defaults (timeout,
rebuild, etc..) in the board's testplan. The benefits are
slimmer testplans.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/overlays/testplans/testplan_default.json | 634 ++++++------------------
engine/overlays/testplans/testplan_docker.json | 93 ++--
engine/scripts/ftc | 59 ++-
3 files changed, 228 insertions(+), 558 deletions(-)
diff --git a/engine/overlays/testplans/testplan_default.json b/engine/overlays/testplans/testplan_default.json
index a31ad6e..693c122 100644
--- a/engine/overlays/testplans/testplan_default.json
+++ b/engine/overlays/testplans/testplan_default.json
@@ -1,486 +1,162 @@
{
"testPlanName": "default",
+ "default_timeout": "40m",
+ "default_spec": "default",
+ "default_reboot" : "false",
+ "default_rebuild" : "true",
+ "default_precleanup" : "true",
+ "default_postcleanup" : "true",
"tests": [
{
- "testName": "Benchmark.bfoo",
- "spec": "default",
- "timeout": "60m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.Dhrystone",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.dbench",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.blobsallad",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.gtkperf",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.glmark",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.ebizzy",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.java",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.hackbench",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.himeno",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.Interbench",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.iperf",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.netperf",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.netpipe",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.openssl",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.stream",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.Whetstone",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.x11perf",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.signaltest",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.reboot",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.linpack",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.nbench_byte",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.cyclictest",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Functional.bc",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.bonnie",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.ffsb",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.fio",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.IOzone",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png", "xlsx": "results.xlsx"}
- },
- {
- "testName": "Benchmark.lmbench2",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Benchmark.tiobench",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"plot": "plot.png"}
- },
- {
- "testName": "Functional.LTP",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true",
- "extralinks": {"xlsx": "results.xlsx"}
- },
- {
- "testName": "Functional.OpenSSL",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.aiostress",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.bzip2",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.crashme",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.expat",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.fontconfig",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.ft2demos",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.glib",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.ipv6connect",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.jpeg",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.linus_stress",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.pi_tests",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.rmaptest",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.netperf",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.scrashme",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.stress",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.synctest",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.zlib",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
- },
- {
- "testName": "Functional.hello_world",
- "spec": "default",
- "timeout": "100m",
- "reboot" : "false",
- "rebuild" : "true",
- "precleanup" : "true",
- "postcleanup" : "true"
+ "testName": "Benchmark.Dhrystone"
+ },
+ {
+ "testName": "Benchmark.dbench"
+ },
+ {
+ "testName": "Benchmark.blobsallad"
+ },
+ {
+ "testName": "Benchmark.gtkperf"
+ },
+ {
+ "testName": "Benchmark.GLMark"
+ },
+ {
+ "testName": "Benchmark.ebizzy"
+ },
+ {
+ "testName": "Benchmark.Java"
+ },
+ {
+ "testName": "Benchmark.hackbench"
+ },
+ {
+ "testName": "Benchmark.himeno"
+ },
+ {
+ "testName": "Benchmark.Interbench"
+ },
+ {
+ "testName": "Benchmark.iperf"
+ },
+ {
+ "testName": "Benchmark.netperf"
+ },
+ {
+ "testName": "Benchmark.netpipe"
+ },
+ {
+ "testName": "Benchmark.OpenSSL"
+ },
+ {
+ "testName": "Benchmark.Stream"
+ },
+ {
+ "testName": "Benchmark.Whetstone"
+ },
+ {
+ "testName": "Benchmark.x11perf"
+ },
+ {
+ "testName": "Benchmark.signaltest"
+ },
+ {
+ "testName": "Benchmark.reboot"
+ },
+ {
+ "testName": "Benchmark.linpack"
+ },
+ {
+ "testName": "Benchmark.nbench_byte"
+ },
+ {
+ "testName": "Benchmark.cyclictest"
+ },
+ {
+ "testName": "Functional.bc"
+ },
+ {
+ "testName": "Benchmark.bonnie"
+ },
+ {
+ "testName": "Benchmark.ffsb"
+ },
+ {
+ "testName": "Benchmark.fio"
+ },
+ {
+ "testName": "Benchmark.IOzone"
+ },
+ {
+ "testName": "Benchmark.lmbench2"
+ },
+ {
+ "testName": "Benchmark.tiobench"
+ },
+ {
+ "testName": "Functional.kernel_build",
+ "timeout": "100m"
+ },
+ {
+ "testName": "Functional.LTP"
+ },
+ {
+ "testName": "Functional.OpenSSL"
+ },
+ {
+ "testName": "Functional.aiostress"
+ },
+ {
+ "testName": "Functional.bzip2"
+ },
+ {
+ "testName": "Functional.crashme"
+ },
+ {
+ "testName": "Functional.expat"
+ },
+ {
+ "testName": "Functional.fontconfig"
+ },
+ {
+ "testName": "Functional.ft2demos"
+ },
+ {
+ "testName": "Functional.glib"
+ },
+ {
+ "testName": "Functional.ipv6connect"
+ },
+ {
+ "testName": "Functional.jpeg"
+ },
+ {
+ "testName": "Functional.linus_stress"
+ },
+ {
+ "testName": "Functional.pi_tests"
+ },
+ {
+ "testName": "Functional.rmaptest"
+ },
+ {
+ "testName": "Functional.netperf"
+ },
+ {
+ "testName": "Functional.scrashme"
+ },
+ {
+ "testName": "Functional.stress"
+ },
+ {
+ "testName": "Functional.synctest"
+ },
+ {
+ "testName": "Functional.zlib"
+ },
+ {
+ "testName": "Functional.hello_world"
}
]
}
diff --git a/engine/overlays/testplans/testplan_docker.json b/engine/overlays/testplans/testplan_docker.json
index cb6fc3d..f8db7d5 100644
--- a/engine/overlays/testplans/testplan_docker.json
+++ b/engine/overlays/testplans/testplan_docker.json
@@ -1,127 +1,92 @@
{
"testPlanName": "docker",
+ "default_timeout": "3m",
+ "default_spec": "default",
"tests": [
{
"testName": "Benchmark.Dhrystone",
- "spec": "100M",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "spec": "100M"
},
{
"testName": "Benchmark.dbench",
- "spec": "testdir",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "spec": "testdir"
},
{
- "testName": "Benchmark.hackbench",
- "spec": "default",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "testName": "Benchmark.hackbench"
},
{
- "testName": "Benchmark.Whetstone",
- "spec": "default",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "testName": "Benchmark.Whetstone"
},
{
- "testName": "Benchmark.linpack",
- "spec": "default",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "testName": "Benchmark.linpack"
},
{
"testName": "Benchmark.bonnie",
- "spec": "noroot",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "spec": "noroot"
},
{
- "testName": "Benchmark.ffsb",
- "spec": "default",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "testName": "Benchmark.ffsb"
},
{
- "testName": "Benchmark.fio",
- "spec": "default",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png"}
+ "testName": "Benchmark.fio"
},
{
"testName": "Benchmark.IOzone",
- "spec": "nopread",
- "timeout": "3m",
- "extralinks": {"plot": "plot.png", "xlsx": "results.xlsx"}
+ "spec": "nopread"
},
{
"testName": "Benchmark.tiobench",
- "spec": "default",
- "timeout": "10m",
- "extralinks": {"plot": "plot.png"}
+ "timeout": "10m"
},
{
"testName": "Benchmark.netperf",
"spec": "docker",
- "timeout": "10m",
- "extralinks": {"plot": "plot.png"}
+ "timeout": "10m"
+ },
+ {
+ "testName": "Benchmark.OpenSSL",
+ "timeout": "30m"
+ },
+ {
+ "testName": "Functional.kernel_build",
+ "timeout": "100m"
},
{
"testName": "Functional.LTP",
- "spec": "default",
- "timeout": "100m",
- "extralinks": {"xlsx": "results.xlsx"}
+ "timeout": "100m"
},
{
- "testName": "Functional.aiostress",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.aiostress"
},
{
- "testName": "Functional.bzip2",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.bzip2"
},
{
"testName": "Functional.glib",
- "spec": "default",
"timeout": "20m"
},
{
- "testName": "Functional.ipv6connect",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.ipv6connect"
},
{
"testName": "Functional.linus_stress",
- "spec": "default",
"timeout": "20m"
},
{
"testName": "Functional.rmaptest",
- "spec": "default",
"timeout": "15m"
},
{
- "testName": "Functional.stress",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.stress"
},
{
- "testName": "Functional.bc",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.bc"
},
{
- "testName": "Functional.zlib",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.zlib"
},
{
- "testName": "Functional.hello_world",
- "spec": "default",
- "timeout": "3m"
+ "testName": "Functional.hello_world"
},
{
"testName": "Functional.netperf",
diff --git a/engine/scripts/ftc b/engine/scripts/ftc
index 98cd7f6..c534688 100755
--- a/engine/scripts/ftc
+++ b/engine/scripts/ftc
@@ -431,17 +431,16 @@ class board_class:
# FIXTHIS - board_class should have methods to read board and dist file
class plantest_class:
- def __init__(self, test_dict):
+ def __init__(self, test_dict, defaults):
self.name = str(test_dict["testName"])
self.test_type = self.name.split(".")[0]
self.base_name = self.name.split(".")[1]
- self.spec = str(test_dict["spec"])
- self.timeout = str(test_dict.get("timeout", "20m"))
- self.reboot = str(test_dict.get("reboot", "false"))
- self.rebuild = str(test_dict.get("rebuild", "true"))
- self.precleanup = str(test_dict.get("precleanup", "true"))
- self.postcleanup = str(test_dict.get("postcleanup", "true"))
- self.extralinks = test_dict.get("extralinks", [])
+ self.spec = str(test_dict.get("spec", defaults['spec']))
+ self.timeout = str(test_dict.get("timeout", defaults['timeout']))
+ self.reboot = str(test_dict.get("reboot", defaults['reboot']))
+ self.rebuild = str(test_dict.get("rebuild", defaults['rebuild']))
+ self.precleanup = str(test_dict.get("precleanup", defaults['precleanup']))
+ self.postcleanup = str(test_dict.get("postcleanup", defaults['postcleanup']))
class run_class:
def __init__(self, conf, timestamp, test_name, host, target, rundir, num):
@@ -922,16 +921,37 @@ def create_batch_job(board, testplan, plan_tests):
sys.exit(1)
# returns a list of plantest_class instances
-def parse_testplan(testplan):
+def parse_testplan(testplan, defaults):
abspath = '/fuego-core/engine/overlays/testplans/' + testplan + '.json'
plan_tests = []
with open(abspath, "r") as f:
plan = json.load(f)
- for test_dict in plan['tests']:
- # test_dict is a dictionary with values for the test
- test = plantest_class(test_dict)
- plan_tests.append(test)
+
+ if 'default_timeout' in plan:
+ defaults['timeout'] = plan['default_timeout']
+
+ if 'default_spec' in plan:
+ defaults['spec'] = plan['default_spec']
+
+ if 'default_reboot' in plan:
+ defaults['reboot'] = plan['default_reboot']
+
+ if 'default_rebuild' in plan:
+ defaults['rebuild'] = plan['default_rebuild']
+
+ if 'default_precleanup' in plan:
+ defaults['precleanup'] = plan['default_precleanup']
+
+ if 'default_postcleanup' in plan:
+ defaults['postcleanup'] = plan['default_postcleanup']
+
+
+ for test_dict in plan['tests']:
+ # test_dict is a dictionary with values for the test
+ test = plantest_class(test_dict, defaults)
+ plan_tests.append(test)
+
return plan_tests
def do_add_jobs(conf, options):
@@ -985,14 +1005,23 @@ def do_add_jobs(conf, options):
else:
raise Exception('No testplan or testcase supplied.')
+ defaults = {
+ 'timeout' : '30m',
+ 'spec' : 'default',
+ 'reboot' : 'false',
+ 'rebuild' : 'true',
+ 'precleanup' : 'true',
+ 'postcleanup' : 'true'
+ }
+
if test_name:
# FIXTHIS - we could parse more parameters for the job here, from the ftc command line
# use all defaults, except for the spec
tp_dict = {"testName":test_name, "spec":spec}
- test = plantest_class(tp_dict)
+ test = plantest_class(tp_dict, defaults)
create_job(board, test)
else:
- plan_tests = parse_testplan(testplan)
+ plan_tests = parse_testplan(testplan, defaults)
for test in plan_tests:
create_job(board, test)
create_batch_job(board, testplan, plan_tests)
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Fuego] [PATCH 3/3] LTP: support separate phases for build and run
2017-04-19 8:32 [Fuego] Changes to the specs and separate LTP build and run Daniel Sangorrin
` (2 preceding siblings ...)
2017-04-19 8:32 ` [Fuego] [PATCH 2/3] defaults: handle defaults in a more efficient way Daniel Sangorrin
@ 2017-04-19 8:32 ` Daniel Sangorrin
3 siblings, 0 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-19 8:32 UTC (permalink / raw)
To: fuego
LTP can be too big for sending its binaries over serial port.
This patch introduces support for building and running separately.
The steps to follow are:
1) ftc add-jobs -b docker -t Functional.LTP -s buildonly
2) Execute the job on Jenkins and wait for it to finish
3) If the build is succesful a tar.gz link will appear. This
links to ltp.tar.gz which contains the LTP binaries.
Download them and uncompress them into your target's /opt
folder. For example:
$ cd /media/user/rootfs/opt/ <-- target's SD Card
# tar zxvf <path to>/ltp.tar.gz
4) ftc add-jobs -b zynq -t Functional.LTP -s runonly
5) Execute the job on Jenkins and wait for it to finish.
6) If the execution is succesful you should get a link
to the results.
Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
---
engine/tests/Functional.LTP/Functional.LTP.spec | 12 ++
engine/tests/Functional.LTP/LTP.sh | 178 +++++++++++++++---------
2 files changed, 125 insertions(+), 65 deletions(-)
diff --git a/engine/tests/Functional.LTP/Functional.LTP.spec b/engine/tests/Functional.LTP/Functional.LTP.spec
index 06c037e..3d1825d 100644
--- a/engine/tests/Functional.LTP/Functional.LTP.spec
+++ b/engine/tests/Functional.LTP/Functional.LTP.spec
@@ -11,6 +11,18 @@
"tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS",
"extra_success_links": {"xlsx": "results.xlsx"}
},
+ "buildonly": {
+ "tests": "ltplite",
+ "buildonly": "true",
+ "platform": "qemu-armv7hf",
+ "extra_success_links": {"tar.gz": "ltp.tar.gz"}
+ },
+ "runonly": {
+ "tests": "ltplite",
+ "runonly": "true",
+ "runfolder": "/opt/ltp",
+ "extra_success_links": {"xlsx": "results.xlsx"}
+ },
"selectionwithrt": {
"tests": "syscalls fs pipes sched timers dio mm ipc pty AIO MSG SEM SIG THR TMR TPS func/hrtimer-prio func/pi-tests func/rt-migrate func/sched_latency func/sched_jitter",
"extra_success_links": {"xlsx": "results.xlsx"}
diff --git a/engine/tests/Functional.LTP/LTP.sh b/engine/tests/Functional.LTP/LTP.sh
index 7f7376c..6905ec0 100755
--- a/engine/tests/Functional.LTP/LTP.sh
+++ b/engine/tests/Functional.LTP/LTP.sh
@@ -42,6 +42,24 @@ func/sched_jitter
"
function test_pre_check {
+ if [ -z ${FUNCTIONAL_LTP_RUNONLY+x} ]; then
+ FUNCTIONAL_LTP_RUNONLY="false"
+ fi
+
+ if [ -z ${FUNCTIONAL_LTP_BUILDONLY+x} ]; then
+ FUNCTIONAL_LTP_BUILDONLY="false"
+ else
+ # buildonly and runonly are mutually exclusive
+ FUNCTIONAL_LTP_RUNONLY="false"
+
+ # we build in docker but use the board's toolchain
+ assert_define FUNCTIONAL_LTP_PLATFORM
+ OLD_PLATFORM=$PLATFORM
+ PLATFORM=$FUNCTIONAL_LTP_PLATFORM
+ source $FUEGO_RO/toolchains/tools.sh
+ PLATFORM=$OLD_PLATFORM
+ fi
+
assert_define SDKROOT
assert_define CC
assert_define AR
@@ -51,84 +69,114 @@ function test_pre_check {
}
function test_build {
- echo "Building LTP"
-
- # patch the Posix test suite script for easier log processing
- patch -p1 -N -s < $TEST_HOME/0001-pts-merge-all-logfile-files-into-a-pts.log-for-proce.patch
-
- # Build the LTP tests
- make autotools
- ./configure CC="${CC}" AR="${AR}" RANLIB="${RANLIB}" LDFLAGS="$LDFLAGS" SYSROOT="${SDKROOT}" \
- --with-open-posix-testsuite --with-realtime-testsuite --without-perl --without-python --target=$PREFIX --host=$PREFIX \
- --prefix=`pwd`/target_bin --build=`uname -m`-unknown-linux-gnu
- make CC="${CC}"
- make install
-
- cp --parents testcases/realtime/scripts/setenv.sh `pwd`/target_bin/
- cp $TEST_HOME/ltp_target_run.sh `pwd`/target_bin/
-
- touch test_suite_ready
+ if [ "$FUNCTIONAL_LTP_RUNONLY" = "false" ]; then
+ echo "Building LTP"
+ # patch the Posix test suite script for easier log processing
+ patch -p1 -N -s < $TEST_HOME/0001-pts-merge-all-logfile-files-into-a-pts.log-for-proce.patch
+
+ # Build the LTP tests
+ make autotools
+ ./configure CC="${CC}" AR="${AR}" RANLIB="${RANLIB}" LDFLAGS="$LDFLAGS" SYSROOT="${SDKROOT}" \
+ --with-open-posix-testsuite --with-realtime-testsuite --without-perl --without-python --target=$PREFIX --host=$PREFIX \
+ --prefix=`pwd`/target_bin --build=`uname -m`-unknown-linux-gnu
+ make CC="${CC}" > build.log 2>&1 || true
+
+ if [ "$FUNCTIONAL_LTP_BUILDONLY" = "true" ]; then
+ report "cat build.log"
+ fi
+
+ make install
+
+ cp --parents testcases/realtime/scripts/setenv.sh `pwd`/target_bin/
+ cp $TEST_HOME/ltp_target_run.sh `pwd`/target_bin/
+
+ touch test_suite_ready
+ else
+ echo "Skipping LTP build"
+ fi
}
function test_deploy {
- echo "Deploying LTP"
-
- # the syscalls group occupies by far the largest space so remove it if unneeded
- cp_syscalls=$(echo $FUNCTIONAL_LTP_TESTS | awk '{print match($0,"syscalls")}')
- if [ $cp_syscalls -eq 0 ]; then
- echo "Removing syscalls binaries"
- awk '/^[^#]/ { print "rm -f target_bin/testcases/bin/" $2 }' target_bin/runtest/syscalls | sh
+ if [ "$FUNCTIONAL_LTP_RUNONLY" = "false" ]; then
+ echo "Deploying LTP"
+
+ # the syscalls group occupies by far the largest space so remove it if unneeded
+ cp_syscalls=$(echo $FUNCTIONAL_LTP_TESTS | awk '{print match($0,"syscalls")}')
+ if [ $cp_syscalls -eq 0 ]; then
+ echo "Removing syscalls binaries"
+ awk '/^[^#]/ { print "rm -f target_bin/testcases/bin/" $2 }' target_bin/runtest/syscalls | sh
+ fi
+
+ if [ "$FUNCTIONAL_LTP_BUILDONLY" = "true" ]; then
+ echo "Creating LTP binary tarball"
+ mv target_bin ltp
+ tar zcvpf ltp.tar.gz ltp/
+ mv ltp.tar.gz ${LOGDIR}/
+ else
+ put target_bin/* $BOARD_TESTDIR/fuego.$TESTDIR/
+ fi
+ else
+ echo "Skipping LTP deploy"
fi
-
- put target_bin/* $BOARD_TESTDIR/fuego.$TESTDIR/
}
function test_run {
- echo "Running LTP"
-
- TESTS=""
- PTSTESTS=""
- RTTESTS=""
-
- for a in $FUNCTIONAL_LTP_TESTS; do
- for b in $ALLTESTS; do
- if [ "$a" == "$b" ]; then
- TESTS+="$a "
- fi
+ if [ "$FUNCTIONAL_LTP_BUILDONLY" = "false" ]; then
+ echo "Running LTP"
+
+ TESTS=""
+ PTSTESTS=""
+ RTTESTS=""
+
+ for a in $FUNCTIONAL_LTP_TESTS; do
+ for b in $ALLTESTS; do
+ if [ "$a" == "$b" ]; then
+ TESTS+="$a "
+ fi
+ done
+
+ for b in $ALLPTSTESTS; do
+ if [ "$a" == "$b" ]; then
+ PTSTESTS+="$a "
+ fi
+ done
+
+ for b in $ALLRTTESTS; do
+ if [ "$a" == "$b" ]; then
+ RTTESTS+="$a "
+ fi
+ done
done
- for b in $ALLPTSTESTS; do
- if [ "$a" == "$b" ]; then
- PTSTESTS+="$a "
- fi
- done
+ if [ -z ${FUNCTIONAL_LTP_RUNFOLDER+x} ]; then
+ FUNCTIONAL_LTP_RUNFOLDER="$BOARD_TESTDIR/fuego.$TESTDIR"
+ fi
- for b in $ALLRTTESTS; do
- if [ "$a" == "$b" ]; then
- RTTESTS+="$a "
- fi
- done
- done
-
- # Let some of the tests fail, the information will be in the result xlsx file
- set +e
- report "cd $BOARD_TESTDIR/fuego.$TESTDIR; TESTS=\"$TESTS\"; PTSTESTS=\"$PTSTESTS\"; RTTESTS=\"$RTTESTS\"; . ./ltp_target_run.sh"
- set -e
+ # Let some of the tests fail, the information will be in the result xlsx file
+ report "cd $FUNCTIONAL_LTP_RUNFOLDER; TESTS=\"$TESTS\"; PTSTESTS=\"$PTSTESTS\"; RTTESTS=\"$RTTESTS\"; . ./ltp_target_run.sh"
+ else
+ echo "Skipping LTP run"
+ fi
}
function test_processing {
- echo "Processing LTP"
-
- get $BOARD_TESTDIR/fuego.$TESTDIR/result .
- cd result/
- # Restore the path so that python works properly
- export PATH=$ORIG_PATH
- cp $TEST_HOME/ltp_process.py .
- python ltp_process.py
- [ -e results.xlsx ] && cp results.xlsx ${LOGDIR}/results.xlsx
- [ -e rt.log ] && cp rt.log $FUEGO_RW/logs/$JOB_NAME/
- cd ..
- rm -rf result/
+ if [ "$FUNCTIONAL_LTP_BUILDONLY" = "false" ]; then
+ echo "Processing LTP results"
+
+ get $FUNCTIONAL_LTP_RUNFOLDER/result .
+ cd result/
+ # Restore the path so that python works properly
+ export PATH=$ORIG_PATH
+ cp $TEST_HOME/ltp_process.py .
+ python ltp_process.py
+ [ -e results.xlsx ] && cp results.xlsx ${LOGDIR}/results.xlsx
+ [ -e rt.log ] && cp rt.log $FUEGO_RW/logs/$JOB_NAME/
+ cd ..
+ rm -rf result/
+ else
+ echo "Processing LTP build log"
+ log_compare "$TESTDIR" "1836" "compile PASSED$" "p"
+ fi
}
. $FUEGO_CORE/engine/scripts/functional.sh
--
2.7.4
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key
2017-04-19 8:32 ` [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key Daniel Sangorrin
@ 2017-04-26 0:43 ` Bird, Timothy
2017-04-26 0:48 ` Daniel Sangorrin
0 siblings, 1 reply; 7+ messages in thread
From: Bird, Timothy @ 2017-04-26 0:43 UTC (permalink / raw)
To: Daniel Sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin on Wednesday, April 19, 2017 1:32 AM
>
> This is quite a big change. Until now links that were
> displayed by the "descriptionsetter" plugin were defined
> in the testplans. That doesn't make sense because those links
> depend on the test spec not on the board's testplan. For
> that reason I put all of the links inside the spec files.
I agree that the links should be based on the spec, which might
define different values for links to test materials based on
the variables passed to the test. However, the changes
just duplicate the same strings into all the spec files. This
is a sign that there should have been a calculated default,
with empty values specifying to use that default.
> Additionally, I have modified the spec file format. Before
> specs where just an array of dicts so to find a specific
> spec you would have to traverse the whole array and compare
> the 'name' property. Now, the spec names are the keys so
> we can quickly access their attributes.
This change is OK.
> Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> ---
> engine/scripts/ftc | 40 ++++++++++++------
> engine/scripts/ovgen.py | 20 ++++-----
> .../Benchmark.Dhrystone/Benchmark.Dhrystone.spec | 23 +++++-----
> .../tests/Benchmark.GLMark/Benchmark.GLMark.spec | 15 ++++---
> .../tests/Benchmark.IOzone/Benchmark.IOzone.spec | 26 ++++--------
> .../Benchmark.Interbench/Benchmark.Interbench.spec | 27 ++++--------
> engine/tests/Benchmark.Java/Benchmark.Java.spec | 15 ++++---
> .../tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec | 14 +++----
> .../tests/Benchmark.Stream/Benchmark.Stream.spec | 14 +++----
> .../Benchmark.Whetstone/Benchmark.Whetstone.spec | 16 +++----
> engine/tests/Benchmark.aim7/Benchmark.aim7.spec | 10 ++---
> .../Benchmark.blobsallad/Benchmark.blobsallad.spec | 17 ++++----
> .../tests/Benchmark.bonnie/Benchmark.bonnie.spec | 22 ++++------
> .../Benchmark.cyclictest/Benchmark.cyclictest.spec | 11 +++--
> .../tests/Benchmark.dbench/Benchmark.dbench.spec | 23 ++++------
> .../tests/Benchmark.ebizzy/Benchmark.ebizzy.spec | 12 +++---
> engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec | 23 ++++------
> engine/tests/Benchmark.fio/Benchmark.fio.spec | 23 ++++------
> .../tests/Benchmark.fs_mark/Benchmark.fs_mark.spec | 10 ++---
> .../Benchmark.fuego_check_plots.spec | 15 +++----
> .../tests/Benchmark.gtkperf/Benchmark.gtkperf.spec | 17 ++++----
> .../Benchmark.hackbench/Benchmark.hackbench.spec | 12 +++---
> .../tests/Benchmark.himeno/Benchmark.himeno.spec | 15 ++++---
> engine/tests/Benchmark.iperf/Benchmark.iperf.spec | 12 +++---
> .../tests/Benchmark.linpack/Benchmark.linpack.spec | 15 ++++---
> .../Benchmark.lmbench2/Benchmark.lmbench2.spec | 15 ++++---
> .../Benchmark.nbench-byte.spec | 10 ++---
> .../Benchmark.nbench_byte.spec | 15 ++++---
> .../tests/Benchmark.netperf/Benchmark.netperf.spec | 17 ++++----
> .../tests/Benchmark.netpipe/Benchmark.netpipe.spec | 12 +++---
> .../tests/Benchmark.reboot/Benchmark.reboot.spec | 15 ++++---
> .../Benchmark.signaltest/Benchmark.signaltest.spec | 14 +++----
> .../Benchmark.sysbench/Benchmark.sysbench.spec | 10 ++---
> .../Benchmark.tiobench/Benchmark.tiobench.spec | 19 ++++-----
> .../tests/Benchmark.x11perf/Benchmark.x11perf.spec | 12 +++---
> engine/tests/Functional.LTP/Functional.LTP.spec | 49 +++++++++++--------
> ---
> .../Functional.OpenSSL/Functional.OpenSSL.spec | 10 ++---
> .../Functional.aiostress/Functional.aiostress.spec | 20 ++++-----
> .../Functional.arch_timer.spec | 10 ++---
> engine/tests/Functional.bc/Functional.bc.spec | 18 ++++----
> .../tests/Functional.boost/Functional.boost.spec | 11 +++--
> .../tests/Functional.bsdiff/Functional.bsdiff.spec | 10 ++---
> .../tests/Functional.bzip2/Functional.bzip2.spec | 11 +++--
> engine/tests/Functional.cmt/Functional.cmt.spec | 10 ++---
> .../Functional.commonAPI_C++.spec | 10 ++---
> .../Functional.commonAPI_Dbus.spec | 10 ++---
> .../Functional.commonAPI_SomeIp.spec | 10 ++---
> .../Functional.crashme/Functional.crashme.spec | 10 ++---
> .../tests/Functional.croco/Functional.croco.spec | 10 ++---
> engine/tests/Functional.curl/Functional.curl.spec | 11 +++--
> .../tests/Functional.expat/Functional.expat.spec | 11 +++--
> .../Functional.fixesproto.spec | 10 ++---
> .../Functional.fontconfig.spec | 11 +++--
> .../tests/Functional.fsfuzz/Functional.fsfuzz.spec | 10 ++---
> .../Functional.ft2demos/Functional.ft2demos.spec | 11 +++--
> .../Functional.fuego_abort.spec | 11 +++--
> .../Functional.fuego_board_check.spec | 11 +++--
> .../Functional.fuego_test_phases.spec | 11 +++--
> .../Functional.fuego_transport.spec | 11 +++--
> engine/tests/Functional.fuse/Functional.fuse.spec | 10 ++---
> .../tests/Functional.giflib/Functional.giflib.spec | 10 ++---
> engine/tests/Functional.glib/Functional.glib.spec | 11 +++--
> .../tests/Functional.glib2/Functional.glib2.spec | 10 ++---
> .../tests/Functional.glibc/Functional.glibc.spec | 10 ++---
> .../Functional.hciattach/Functional.hciattach.spec | 10 ++---
> .../Functional.hello_world.spec | 16 ++++---
> .../Functional.imagemagick.spec | 10 ++---
> .../Functional.iptables/Functional.iptables.spec | 10 ++---
> .../Functional.iputils/Functional.iputils.spec | 10 ++---
> .../Functional.ipv6connect.spec | 11 +++--
> engine/tests/Functional.jpeg/Functional.jpeg.spec | 11 +++--
> .../Functional.kernel_build.spec | 16 +++----
> engine/tests/Functional.kmod/Functional.kmod.spec | 10 ++---
> .../tests/Functional.libogg/Functional.libogg.spec | 10 ++---
> .../Functional.libpcap/Functional.libpcap.spec | 10 ++---
> .../Functional.librsvg/Functional.librsvg.spec | 10 ++---
> .../Functional.libspeex/Functional.libspeex.spec | 10 ++---
> .../tests/Functional.libtar/Functional.libtar.spec | 10 ++---
> .../Functional.libwebsocket.spec | 10 ++---
> .../Functional.linus_stress.spec | 10 ++---
> engine/tests/Functional.lwip/Functional.lwip.spec | 10 ++---
> .../Functional.mesa-demos.spec | 10 ++---
> engine/tests/Functional.neon/Functional.neon.spec | 10 ++---
> .../Functional.net-tools/Functional.net-tools.spec | 10 ++---
> .../Functional.netperf/Functional.netperf.spec | 13 +++---
> .../Functional.pi_tests/Functional.pi_tests.spec | 10 ++---
> .../tests/Functional.pixman/Functional.pixman.spec | 10 ++---
> engine/tests/Functional.pppd/Functional.pppd.spec | 10 ++---
> .../Functional.protobuf/Functional.protobuf.spec | 10 ++---
> .../Functional.rmaptest/Functional.rmaptest.spec | 10 ++---
> .../tests/Functional.scifab/Functional.scifab.spec | 10 ++---
> .../Functional.scrashme/Functional.scrashme.spec | 22 +++++-----
> .../tests/Functional.sdhi_0/Functional.sdhi_0.spec | 10 ++---
> .../tests/Functional.stress/Functional.stress.spec | 10 ++---
> .../Functional.synctest/Functional.synctest.spec | 23 +++++-----
> .../tests/Functional.thrift/Functional.thrift.spec | 10 ++---
> engine/tests/Functional.tiff/Functional.tiff.spec | 10 ++---
> .../Functional.vsomeip/Functional.vsomeip.spec | 10 ++---
> .../Functional.xorg-macros.spec | 10 ++---
> engine/tests/Functional.zlib/Functional.zlib.spec | 11 +++--
> 100 files changed, 633 insertions(+), 714 deletions(-)
>
> diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> index 939043b..98cd7f6 100755
> --- a/engine/scripts/ftc
> +++ b/engine/scripts/ftc
> @@ -788,14 +788,29 @@ def create_job(board, test):
> else:
> flot_link = ''
>
> - # prepare links for descriptionsetter. Put a link to testlog.txt by default
> + # prepare links for the descriptionsetter plugin
> + test_spec_path = '/fuego-core/engine/tests/%s/%s.spec' % (test.name,
> test.name)
> template_link = '<a
> href="/fuego/userContent/fuego.logs/%s/%s.%s.${BUILD_NUMBER}.
> ${BUILD_ID}/%%s">%%s</a>' % (test.name, board, test.spec)
> -
> - html_links = template_link % ('testlog.txt', 'log')
> - failed_links = html_links
> - if test.extralinks:
> - for key, value in test.extralinks.iteritems():
> - html_links = html_links + ' ' + template_link % (str(value), str(key))
> + with open(test_spec_path, "r") as f:
> + test_specs_json = json.load(f)
> + test_spec_json = test_specs_json['specs'][test.spec]
> +
> + success_links = ''
> + if 'success_links' in test_specs_json:
> + for key, value in test_specs_json['success_links'].iteritems():
> + success_links = success_links + ' ' + template_link % (str(value),
> str(key))
Can't we have an else here and fabricate a default value for the links
(depending on whether the test is a Functional or Benchmark test)?
> + if 'extra_success_links' in test_spec_json:
> + for key, value in test_spec_json['extra_success_links'].iteritems():
> + success_links = success_links + ' ' + template_link % (str(value),
> str(key))
> +
> + fail_links = ''
> + if 'fail_links' in test_specs_json:
> + for key, value in test_specs_json['fail_links'].iteritems():
> + fail_links = fail_links + ' ' + template_link % (str(value), str(key))
Same here - let's have a default link. It's currently the same in every spec file.
I'd rather have a default, and override it, than declare the same strings
in every single spec file.
> +
> + if 'extra_fail_links' in test_spec_json:
> + for key, value in test_spec_json['extra_fail_links'].iteritems():
> + fail_links = fail_links + ' ' + template_link % (str(value), str(key))
>
> tmp = "/tmp/fuego_tmp_job"
> fd = open(tmp, "w+")
> @@ -833,8 +848,8 @@ timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
> <hudson.plugins.descriptionsetter.DescriptionSetterPublisher
> plugin="description-setter@1.10">
> <regexp></regexp>
> <regexpForFailed></regexpForFailed>
> - <description>{html_links}</description>
> - <descriptionForFailed>{failed_links}</descriptionForFailed>
> + <description>{success_links}</description>
> + <descriptionForFailed>{fail_links}</descriptionForFailed>
> <setForMatrix>false</setForMatrix>
> </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
> </publishers>
> @@ -844,7 +859,7 @@ timeout --signal=9 {timeout} /bin/bash
> $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
> precleanup=test.precleanup, postcleanup=test.postcleanup,
> testdir=test.name, testname=test.base_name,
> testspec=test.spec, timeout=test.timeout,
> - flot_link=flot_link, html_links=html_links, failed_links=failed_links))
> + flot_link=flot_link, success_links=success_links, fail_links=fail_links))
> fd.close()
>
> job_name=board+"."+test.spec+"."+test.name
> @@ -960,7 +975,7 @@ def do_add_jobs(conf, options):
> spec = options[options.index('-s') + 1]
> except IndexError:
> error_out('Testspec not provided after -s.')
> - specnames = [s['name'] for s in get_specs(conf, test_name)]
> + specnames = get_specs(conf, test_name)
> if spec not in specnames:
> error_out('Unknown spec %s' % spec)
> options.remove('-s')
> @@ -1225,8 +1240,7 @@ def do_list_specs(conf, options):
> else:
> error_out('No testcase name supplied.')
>
> - specs = get_specs(conf, testcase)
> - specnames = [spec['name'] for spec in specs]
> + specnames = get_specs(conf, testcase).keys()
> specnames.sort()
>
> indent = show_list_title("Testspecs for %s:" % testcase)
> diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
> index f3f77e3..b331243 100755
> --- a/engine/scripts/ovgen.py
> +++ b/engine/scripts/ovgen.py
> @@ -404,11 +404,9 @@ def generateProlog(outFilePath, ofcls, classes,
> testdir, testspec):
> # testspec: String - name of test spec (e.g. 'default')
> # outFilePath - file output descriptor opened -> prolog.sh
> def generateSpec(ts, fout):
> - debug_print("generating spec '%s' for '%s'" % (ts.variables['name'],
> ts.name))
> + debug_print("generating spec %s" % (ts.name))
>
> for var in ts.variables:
> - if var == 'name':
> - continue
> varname = "%s_%s" % (ts.name, var)
> varname = string.replace(varname, ".", "_").upper()
> value = "%s" % (ts.variables[var])
> @@ -446,17 +444,15 @@ def parseSpec(testdir, testspec):
> except:
> raise Exception("Error parsing spec file %s" % specpath)
>
> - ts.name = jd["testName"]
> + ts.name = jd["testName"]
>
> - if "fail_case" in jd:
> - ts.fail_case = jd["fail_case"]
> - debug_print ("Found fail_case msgs in '%s'" % specpath)
> + if "fail_case" in jd:
> + ts.fail_case = jd["fail_case"]
> + debug_print ("Found fail_case msgs in '%s'" % specpath)
>
> - for spec in jd["specs"]:
> - if spec["name"] == testspec:
> - ts.variables = spec
> -
> - if not ts.variables:
> + if testspec in jd["specs"]:
> + ts.variables = jd["specs"][testspec]
> + else:
> raise Exception("Could not find %s in %s" % (testspec, specpath))
>
> return ts
> diff --git a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> index 794ca68..7ceee2a 100644
> --- a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> +++ b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> @@ -1,25 +1,22 @@
> - {
> +{
> "testName": "Benchmark.Dhrystone",
> "fail_case": [
> {
> "fail_regexp": "Measured time too small to obtain meaningful results",
> "fail_message": "Measured time too small to obtain meaningful
> results. Please increase LOOPS parameter in Dhrystone test spec."
> - }
> - ],
> - "specs":
> - [
> - {
> - "name":"default",
> + }
> + ],
> + "success_links": {"log": "testlog.txt", "plot": "plot.png"},
> + "fail_links": {"log": "testlog.txt"},
Let's remove all these, unless they are different from the default.
> + "specs": {
> + "default": {
> "LOOPS":"10000000"
> },
> - {
> - "name":"100M",
> + "100M": {
> "LOOPS":"100000000"
> },
> - {
> - "name":"500M",
> + "500M": {
> "LOOPS":"500000000"
> }
> - ]
> + }
> }
> -
Thanks. Can you please clean up as directed and re-send?
-- Tim
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key
2017-04-26 0:43 ` Bird, Timothy
@ 2017-04-26 0:48 ` Daniel Sangorrin
0 siblings, 0 replies; 7+ messages in thread
From: Daniel Sangorrin @ 2017-04-26 0:48 UTC (permalink / raw)
To: 'Bird, Timothy', fuego
> -----Original Message-----
> From: Bird, Timothy [mailto:Tim.Bird@sony.com]
> Sent: Wednesday, April 26, 2017 9:44 AM
> To: Daniel Sangorrin; fuego@lists.linuxfoundation.org
> Subject: RE: [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key
>
> > -----Original Message-----
> > From: Daniel Sangorrin on Wednesday, April 19, 2017 1:32 AM
> >
> > This is quite a big change. Until now links that were
> > displayed by the "descriptionsetter" plugin were defined
> > in the testplans. That doesn't make sense because those links
> > depend on the test spec not on the board's testplan. For
> > that reason I put all of the links inside the spec files.
>
> I agree that the links should be based on the spec, which might
> define different values for links to test materials based on
> the variables passed to the test. However, the changes
> just duplicate the same strings into all the spec files. This
> is a sign that there should have been a calculated default,
> with empty values specifying to use that default.
OK, I will add testlog.txt and plot.png as defaults, and allow
overwritting them from the spec when necessary.
>
> > Additionally, I have modified the spec file format. Before
> > specs where just an array of dicts so to find a specific
> > spec you would have to traverse the whole array and compare
> > the 'name' property. Now, the spec names are the keys so
> > we can quickly access their attributes.
> This change is OK.
>
> > Signed-off-by: Daniel Sangorrin <daniel.sangorrin@toshiba.co.jp>
> > ---
> > engine/scripts/ftc | 40 ++++++++++++------
> > engine/scripts/ovgen.py | 20 ++++-----
> > .../Benchmark.Dhrystone/Benchmark.Dhrystone.spec | 23 +++++-----
> > .../tests/Benchmark.GLMark/Benchmark.GLMark.spec | 15 ++++---
> > .../tests/Benchmark.IOzone/Benchmark.IOzone.spec | 26 ++++--------
> > .../Benchmark.Interbench/Benchmark.Interbench.spec | 27 ++++--------
> > engine/tests/Benchmark.Java/Benchmark.Java.spec | 15 ++++---
> > .../tests/Benchmark.OpenSSL/Benchmark.OpenSSL.spec | 14 +++----
> > .../tests/Benchmark.Stream/Benchmark.Stream.spec | 14 +++----
> > .../Benchmark.Whetstone/Benchmark.Whetstone.spec | 16 +++----
> > engine/tests/Benchmark.aim7/Benchmark.aim7.spec | 10 ++---
> > .../Benchmark.blobsallad/Benchmark.blobsallad.spec | 17 ++++----
> > .../tests/Benchmark.bonnie/Benchmark.bonnie.spec | 22 ++++------
> > .../Benchmark.cyclictest/Benchmark.cyclictest.spec | 11 +++--
> > .../tests/Benchmark.dbench/Benchmark.dbench.spec | 23 ++++------
> > .../tests/Benchmark.ebizzy/Benchmark.ebizzy.spec | 12 +++---
> > engine/tests/Benchmark.ffsb/Benchmark.ffsb.spec | 23 ++++------
> > engine/tests/Benchmark.fio/Benchmark.fio.spec | 23 ++++------
> > .../tests/Benchmark.fs_mark/Benchmark.fs_mark.spec | 10 ++---
> > .../Benchmark.fuego_check_plots.spec | 15 +++----
> > .../tests/Benchmark.gtkperf/Benchmark.gtkperf.spec | 17 ++++----
> > .../Benchmark.hackbench/Benchmark.hackbench.spec | 12 +++---
> > .../tests/Benchmark.himeno/Benchmark.himeno.spec | 15 ++++---
> > engine/tests/Benchmark.iperf/Benchmark.iperf.spec | 12 +++---
> > .../tests/Benchmark.linpack/Benchmark.linpack.spec | 15 ++++---
> > .../Benchmark.lmbench2/Benchmark.lmbench2.spec | 15 ++++---
> > .../Benchmark.nbench-byte.spec | 10 ++---
> > .../Benchmark.nbench_byte.spec | 15 ++++---
> > .../tests/Benchmark.netperf/Benchmark.netperf.spec | 17 ++++----
> > .../tests/Benchmark.netpipe/Benchmark.netpipe.spec | 12 +++---
> > .../tests/Benchmark.reboot/Benchmark.reboot.spec | 15 ++++---
> > .../Benchmark.signaltest/Benchmark.signaltest.spec | 14 +++----
> > .../Benchmark.sysbench/Benchmark.sysbench.spec | 10 ++---
> > .../Benchmark.tiobench/Benchmark.tiobench.spec | 19 ++++-----
> > .../tests/Benchmark.x11perf/Benchmark.x11perf.spec | 12 +++---
> > engine/tests/Functional.LTP/Functional.LTP.spec | 49 +++++++++++--------
> > ---
> > .../Functional.OpenSSL/Functional.OpenSSL.spec | 10 ++---
> > .../Functional.aiostress/Functional.aiostress.spec | 20 ++++-----
> > .../Functional.arch_timer.spec | 10 ++---
> > engine/tests/Functional.bc/Functional.bc.spec | 18 ++++----
> > .../tests/Functional.boost/Functional.boost.spec | 11 +++--
> > .../tests/Functional.bsdiff/Functional.bsdiff.spec | 10 ++---
> > .../tests/Functional.bzip2/Functional.bzip2.spec | 11 +++--
> > engine/tests/Functional.cmt/Functional.cmt.spec | 10 ++---
> > .../Functional.commonAPI_C++.spec | 10 ++---
> > .../Functional.commonAPI_Dbus.spec | 10 ++---
> > .../Functional.commonAPI_SomeIp.spec | 10 ++---
> > .../Functional.crashme/Functional.crashme.spec | 10 ++---
> > .../tests/Functional.croco/Functional.croco.spec | 10 ++---
> > engine/tests/Functional.curl/Functional.curl.spec | 11 +++--
> > .../tests/Functional.expat/Functional.expat.spec | 11 +++--
> > .../Functional.fixesproto.spec | 10 ++---
> > .../Functional.fontconfig.spec | 11 +++--
> > .../tests/Functional.fsfuzz/Functional.fsfuzz.spec | 10 ++---
> > .../Functional.ft2demos/Functional.ft2demos.spec | 11 +++--
> > .../Functional.fuego_abort.spec | 11 +++--
> > .../Functional.fuego_board_check.spec | 11 +++--
> > .../Functional.fuego_test_phases.spec | 11 +++--
> > .../Functional.fuego_transport.spec | 11 +++--
> > engine/tests/Functional.fuse/Functional.fuse.spec | 10 ++---
> > .../tests/Functional.giflib/Functional.giflib.spec | 10 ++---
> > engine/tests/Functional.glib/Functional.glib.spec | 11 +++--
> > .../tests/Functional.glib2/Functional.glib2.spec | 10 ++---
> > .../tests/Functional.glibc/Functional.glibc.spec | 10 ++---
> > .../Functional.hciattach/Functional.hciattach.spec | 10 ++---
> > .../Functional.hello_world.spec | 16 ++++---
> > .../Functional.imagemagick.spec | 10 ++---
> > .../Functional.iptables/Functional.iptables.spec | 10 ++---
> > .../Functional.iputils/Functional.iputils.spec | 10 ++---
> > .../Functional.ipv6connect.spec | 11 +++--
> > engine/tests/Functional.jpeg/Functional.jpeg.spec | 11 +++--
> > .../Functional.kernel_build.spec | 16 +++----
> > engine/tests/Functional.kmod/Functional.kmod.spec | 10 ++---
> > .../tests/Functional.libogg/Functional.libogg.spec | 10 ++---
> > .../Functional.libpcap/Functional.libpcap.spec | 10 ++---
> > .../Functional.librsvg/Functional.librsvg.spec | 10 ++---
> > .../Functional.libspeex/Functional.libspeex.spec | 10 ++---
> > .../tests/Functional.libtar/Functional.libtar.spec | 10 ++---
> > .../Functional.libwebsocket.spec | 10 ++---
> > .../Functional.linus_stress.spec | 10 ++---
> > engine/tests/Functional.lwip/Functional.lwip.spec | 10 ++---
> > .../Functional.mesa-demos.spec | 10 ++---
> > engine/tests/Functional.neon/Functional.neon.spec | 10 ++---
> > .../Functional.net-tools/Functional.net-tools.spec | 10 ++---
> > .../Functional.netperf/Functional.netperf.spec | 13 +++---
> > .../Functional.pi_tests/Functional.pi_tests.spec | 10 ++---
> > .../tests/Functional.pixman/Functional.pixman.spec | 10 ++---
> > engine/tests/Functional.pppd/Functional.pppd.spec | 10 ++---
> > .../Functional.protobuf/Functional.protobuf.spec | 10 ++---
> > .../Functional.rmaptest/Functional.rmaptest.spec | 10 ++---
> > .../tests/Functional.scifab/Functional.scifab.spec | 10 ++---
> > .../Functional.scrashme/Functional.scrashme.spec | 22 +++++-----
> > .../tests/Functional.sdhi_0/Functional.sdhi_0.spec | 10 ++---
> > .../tests/Functional.stress/Functional.stress.spec | 10 ++---
> > .../Functional.synctest/Functional.synctest.spec | 23 +++++-----
> > .../tests/Functional.thrift/Functional.thrift.spec | 10 ++---
> > engine/tests/Functional.tiff/Functional.tiff.spec | 10 ++---
> > .../Functional.vsomeip/Functional.vsomeip.spec | 10 ++---
> > .../Functional.xorg-macros.spec | 10 ++---
> > engine/tests/Functional.zlib/Functional.zlib.spec | 11 +++--
> > 100 files changed, 633 insertions(+), 714 deletions(-)
> >
> > diff --git a/engine/scripts/ftc b/engine/scripts/ftc
> > index 939043b..98cd7f6 100755
> > --- a/engine/scripts/ftc
> > +++ b/engine/scripts/ftc
> > @@ -788,14 +788,29 @@ def create_job(board, test):
> > else:
> > flot_link = ''
> >
> > - # prepare links for descriptionsetter. Put a link to testlog.txt by default
> > + # prepare links for the descriptionsetter plugin
> > + test_spec_path = '/fuego-core/engine/tests/%s/%s.spec' % (test.name,
> > test.name)
> > template_link = '<a
> > href="/fuego/userContent/fuego.logs/%s/%s.%s.${BUILD_NUMBER}.
> > ${BUILD_ID}/%%s">%%s</a>' % (test.name, board, test.spec)
> > -
> > - html_links = template_link % ('testlog.txt', 'log')
> > - failed_links = html_links
> > - if test.extralinks:
> > - for key, value in test.extralinks.iteritems():
> > - html_links = html_links + ' ' + template_link % (str(value), str(key))
> > + with open(test_spec_path, "r") as f:
> > + test_specs_json = json.load(f)
> > + test_spec_json = test_specs_json['specs'][test.spec]
> > +
> > + success_links = ''
> > + if 'success_links' in test_specs_json:
> > + for key, value in test_specs_json['success_links'].iteritems():
> > + success_links = success_links + ' ' + template_link % (str(value),
> > str(key))
> Can't we have an else here and fabricate a default value for the links
> (depending on whether the test is a Functional or Benchmark test)?
>
> > + if 'extra_success_links' in test_spec_json:
> > + for key, value in test_spec_json['extra_success_links'].iteritems():
> > + success_links = success_links + ' ' + template_link % (str(value),
> > str(key))
> > +
> > + fail_links = ''
> > + if 'fail_links' in test_specs_json:
> > + for key, value in test_specs_json['fail_links'].iteritems():
> > + fail_links = fail_links + ' ' + template_link % (str(value), str(key))
> Same here - let's have a default link. It's currently the same in every spec file.
>
> I'd rather have a default, and override it, than declare the same strings
> in every single spec file.
>
> > +
> > + if 'extra_fail_links' in test_spec_json:
> > + for key, value in test_spec_json['extra_fail_links'].iteritems():
> > + fail_links = fail_links + ' ' + template_link % (str(value), str(key))
> >
> > tmp = "/tmp/fuego_tmp_job"
> > fd = open(tmp, "w+")
> > @@ -833,8 +848,8 @@ timeout --signal=9 {timeout} /bin/bash
> > $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
> > <hudson.plugins.descriptionsetter.DescriptionSetterPublisher
> > plugin="description-setter@1.10">
> > <regexp></regexp>
> > <regexpForFailed></regexpForFailed>
> > - <description>{html_links}</description>
> > - <descriptionForFailed>{failed_links}</descriptionForFailed>
> > + <description>{success_links}</description>
> > + <descriptionForFailed>{fail_links}</descriptionForFailed>
> > <setForMatrix>false</setForMatrix>
> > </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
> > </publishers>
> > @@ -844,7 +859,7 @@ timeout --signal=9 {timeout} /bin/bash
> > $FUEGO_CORE/engine/tests/${{TESTDIR}}/${{
> > precleanup=test.precleanup, postcleanup=test.postcleanup,
> > testdir=test.name, testname=test.base_name,
> > testspec=test.spec, timeout=test.timeout,
> > - flot_link=flot_link, html_links=html_links, failed_links=failed_links))
> > + flot_link=flot_link, success_links=success_links, fail_links=fail_links))
> > fd.close()
> >
> > job_name=board+"."+test.spec+"."+test.name
> > @@ -960,7 +975,7 @@ def do_add_jobs(conf, options):
> > spec = options[options.index('-s') + 1]
> > except IndexError:
> > error_out('Testspec not provided after -s.')
> > - specnames = [s['name'] for s in get_specs(conf, test_name)]
> > + specnames = get_specs(conf, test_name)
> > if spec not in specnames:
> > error_out('Unknown spec %s' % spec)
> > options.remove('-s')
> > @@ -1225,8 +1240,7 @@ def do_list_specs(conf, options):
> > else:
> > error_out('No testcase name supplied.')
> >
> > - specs = get_specs(conf, testcase)
> > - specnames = [spec['name'] for spec in specs]
> > + specnames = get_specs(conf, testcase).keys()
> > specnames.sort()
> >
> > indent = show_list_title("Testspecs for %s:" % testcase)
> > diff --git a/engine/scripts/ovgen.py b/engine/scripts/ovgen.py
> > index f3f77e3..b331243 100755
> > --- a/engine/scripts/ovgen.py
> > +++ b/engine/scripts/ovgen.py
> > @@ -404,11 +404,9 @@ def generateProlog(outFilePath, ofcls, classes,
> > testdir, testspec):
> > # testspec: String - name of test spec (e.g. 'default')
> > # outFilePath - file output descriptor opened -> prolog.sh
> > def generateSpec(ts, fout):
> > - debug_print("generating spec '%s' for '%s'" % (ts.variables['name'],
> > ts.name))
> > + debug_print("generating spec %s" % (ts.name))
> >
> > for var in ts.variables:
> > - if var == 'name':
> > - continue
> > varname = "%s_%s" % (ts.name, var)
> > varname = string.replace(varname, ".", "_").upper()
> > value = "%s" % (ts.variables[var])
> > @@ -446,17 +444,15 @@ def parseSpec(testdir, testspec):
> > except:
> > raise Exception("Error parsing spec file %s" % specpath)
> >
> > - ts.name = jd["testName"]
> > + ts.name = jd["testName"]
> >
> > - if "fail_case" in jd:
> > - ts.fail_case = jd["fail_case"]
> > - debug_print ("Found fail_case msgs in '%s'" % specpath)
> > + if "fail_case" in jd:
> > + ts.fail_case = jd["fail_case"]
> > + debug_print ("Found fail_case msgs in '%s'" % specpath)
> >
> > - for spec in jd["specs"]:
> > - if spec["name"] == testspec:
> > - ts.variables = spec
> > -
> > - if not ts.variables:
> > + if testspec in jd["specs"]:
> > + ts.variables = jd["specs"][testspec]
> > + else:
> > raise Exception("Could not find %s in %s" % (testspec, specpath))
> >
> > return ts
> > diff --git a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> > b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> > index 794ca68..7ceee2a 100644
> > --- a/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> > +++ b/engine/tests/Benchmark.Dhrystone/Benchmark.Dhrystone.spec
> > @@ -1,25 +1,22 @@
> > - {
> > +{
> > "testName": "Benchmark.Dhrystone",
> > "fail_case": [
> > {
> > "fail_regexp": "Measured time too small to obtain meaningful results",
> > "fail_message": "Measured time too small to obtain meaningful
> > results. Please increase LOOPS parameter in Dhrystone test spec."
> > - }
> > - ],
> > - "specs":
> > - [
> > - {
> > - "name":"default",
> > + }
> > + ],
> > + "success_links": {"log": "testlog.txt", "plot": "plot.png"},
> > + "fail_links": {"log": "testlog.txt"},
> Let's remove all these, unless they are different from the default.
> > + "specs": {
> > + "default": {
> > "LOOPS":"10000000"
> > },
> > - {
> > - "name":"100M",
> > + "100M": {
> > "LOOPS":"100000000"
> > },
> > - {
> > - "name":"500M",
> > + "500M": {
> > "LOOPS":"500000000"
> > }
> > - ]
> > + }
> > }
> > -
>
> Thanks. Can you please clean up as directed and re-send?
>
> -- Tim
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2017-04-26 0:48 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-19 8:32 [Fuego] Changes to the specs and separate LTP build and run Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH] cross: add libaio dependency for building LTP Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 1/3] specs: put links in the spec and don't use the name key Daniel Sangorrin
2017-04-26 0:43 ` Bird, Timothy
2017-04-26 0:48 ` Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 2/3] defaults: handle defaults in a more efficient way Daniel Sangorrin
2017-04-19 8:32 ` [Fuego] [PATCH 3/3] LTP: support separate phases for build and run Daniel Sangorrin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.