[Fuego] [Fuego 1.2.1] reference.json file is missing for Benchmark.signaltest

[Fuego] [Fuego 1.2.1] reference.json file is missing for Benchmark.signaltest

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 8332 bytes --]

[-- Attachment #2: Type: image/gif, Size: 13402 bytes --]

Re: [Fuego] [Fuego 1.2.1] reference.json file is missing for Benchmark.signaltest

[Bird, Timothy]

Thanks for the report.  Comments inline below.

> -----Original Message-----
> From: Dhinakar Kalyanasundaram on Thursday, November 30, 2017 11:50 PM
> 
> While executing Benchmark.signaltest I get the 'No reference.json available'
> error.
> 
> Please find the execution log pasted below for reference.
> 
> 
> 
> Started by user anonymous <http://107.108.83.19:8080/storm/user/null>
> Building remotely on TRAV-Sanity-Test
> <http://107.108.83.19:8080/storm/computer/TRAV-Sanity-Test>  in
> workspace /fuego-rw/buildzone
> [buildzone] $ /bin/sh -xe /tmp/hudson8078951296844586781.sh
> + export Reboot=false
> + Reboot=false
> + export Rebuild=false
> + Rebuild=false
> + export Target_PreCleanup=true
> + Target_PreCleanup=true
> + export Target_PostCleanup=true
> + Target_PostCleanup=true
> + export TESTDIR=Benchmark.signaltest
> + TESTDIR=Benchmark.signaltest
> + export TESTSPEC=default
> + TESTSPEC=default
> + timeout --signal=9 6m /bin/bash /fuego-core/engine/scripts/main.sh
> Using nosyslogd.dist overlay
> ##### doing fuego phase: pre_test ########
> 
> 
> Logged in as user root
> OK:    have root privileges
> Firmware revision: 4.9.0+
> 
>  00:00:18 up 0 min,  1 user,  load average: 0.00, 0.00, 0.00
> 
>               total        used        free      shared  buff/cache   available
> Mem:        2342088       53704     2265868         204       22516     2243232
> Swap:             0           0           0
> 
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/root       7.2G  4.1G  2.7G  61% /
> devtmpfs        337M     0  337M   0% /dev
> tmpfs           1.2G     0  1.2G   0% /dev/shm
> tmpfs           1.2G   48K  1.2G   1% /tmp
> tmpfs           1.2G  156K  1.2G   1% /run
> 
> /dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered)
> devtmpfs on /dev type devtmpfs
> (rw,relatime,size=344208k,nr_inodes=86052,mode=755)
> proc on /proc type proc (rw,relatime)
> devpts on /dev/pts type devpts
> (rw,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /dev/shm type tmpfs (rw,relatime,mode=777)
> tmpfs on /tmp type tmpfs (rw,relatime)
> tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
> sysfs on /sys type sysfs (rw,relatime)
> 
> BusyBox v1.26.2 (2017-05-05 12:57:46 IST) multi-call binary.
> 
> Usage: grep [-HhnlLoqvsriwFE] [-m N] [-A/B/C N] PATTERN/-e PATTERN.../-f
> FILE [FILE]...
> 
> Search for PATTERN in FILEs (or stdin)
> 
> 	-H	Add 'filename:' prefix
> 	-h	Do not add 'filename:' prefix
> 	-n	Add 'line_no:' prefix
> 	-l	Show only names of files that match
> 	-L	Show only names of files that don't match
> 	-c	Show only count of matching lines
> 	-o	Show only the matching part of line
> 	-q	Quiet. Return 0 if PATTERN is found, 1 otherwise
> 	-v	Select non-matching lines
> 	-s	Suppress open and read errors
> 	-r	Recurse
> 	-i	Ignore case
> 	-w	Match whole words only
> 	-x	Match whole lines only
> 	-F	PATTERN is a literal (not regexp)
> 	-E	PATTERN is an extended regexp
> 	-m N	Match up to N times per file
> 	-A N	Print N lines of trailing context
> 	-B N	Print N lines of leading context
> 	-C N	Same as '-A N -B N'
> 	-e PTRN	Pattern to match
> 	-f FILE	Read pattern from file

OK - this is not what you reported, but this output tells me that something
is messed up in Fuego's use of grep here.   Can you turn on FUEGO_DEBUG,
and send the console log for this again?

> [   20.705434] sh (1468): drop_caches: 3
> 
> ##### doing fuego phase: build ########
> 
> The test is already built
> Fuego test_build duration=0 seconds
> ##### doing fuego phase: deploy ########
> ##### doing fuego phase: pre_deploy ########
> ##### doing fuego phase: test_deploy ########
> ##### doing fuego phase: post_deploy ########
> ##### doing fuego phase: run ########
> WARNING: test log file parameter empty, so will use default
> T: 0 ( 1478) P:60 C:  10000 Min:     12 Act:   15 Avg:   15 Max:      33
> T: 1 ( 1479) P:60 C:  10000 Min:     13 Act:   15 Avg:  640 Max:   10045
> 
> ##### doing fuego phase: post_test ########
> Teardown board link
> ##### doing fuego phase: processing ########
> Going to check 1 fail cases for TRAV-Sanity-Test.default.Benchmark.signaltest
> T: 0 ( 1478) P:60 C:  10000 Min:     12 Act:   15 Avg:   15 Max:      33
> T: 1 ( 1479) P:60 C:  10000 Min:     13 Act:   15 Avg:  640 Max:   10045
> Detected fail message in /fuego-rw/logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/testlog.txt:
> ERROR: fail_check_cases returned an error
Hmmm.  I'm not sure what's going on here.

Can you turn on FUEGO_DEBUG (by editing the job configuration in Jenkins and
uncommenting the line with FUEGO_DEBUG), and send the console log?


> running python with
> PATH=/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/gam
> es
> Parsing /fuego-rw/logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/testlog.txt with regex: ^T:.*Min:\s+(\d+).*Avg:\s+(\d+)
> Max:\s+(\d+)
> matches: [('12', '15', '33'), ('13', '640', '10045')]
> No reference.json available
I think this is just a warning.  This file is not required for test execution
or log processing.  In general, the parser (which is where this and the lines
below are coming from) are a bit too verbose, IMHO.

I think this is something that should be cleaned up, and I'll look at it
(probably next week).

> ref={'test_sets': [{'test_cases': [{'measurements': [{'name': 'max_latency',
> 'measure': 10045.0}, {'name': 'min_latency', 'measure': 12.0}, {'name':
> 'avg_latency', 'measure': 327.5}], 'name': 'latencies'}], 'name': 'default'}]}
> Applying criterion {'max_fail': 0, 'tguid': 'default.latencies'}
> Applying criterion {'max_fail': 0, 'tguid': 'default'}
> Applying criterion {'max_fail': 0, 'tguid': 'Benchmark.signaltest'}
> Writing run data to /fuego-rw/logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/run.json
> reference.json not available
> WARNING: Unable to parse /fuego-rw/logs/Benchmark.signaltest/TRAV-
> Sanity-Test.default.2.2/run.json
> WARNING: Unable to parse /fuego-rw/logs/Benchmark.signaltest/TRAV-
> Sanity-Test.default.3.3/run.json
> WARNING: Unable to parse /fuego-rw/logs/Benchmark.signaltest/TRAV-
> System-Test.default.1.1/run.json
> Writing merged results to  /fuego-rw/logs/Benchmark.signaltest/results.json
> Writing flat results to /fuego-
> rw/logs/Benchmark.signaltest/flat_plot_data.txt
> Writing chart data to /fuego-
> rw/logs/Benchmark.signaltest/flot_chart_data.json
> ERROR: results did not satisfy the threshold

I believe this is the actual cause of the final test result (FAILURE).

The max latency of over 10,000 reported in the test log is greater
than the threshold  specified in the criteria.json file (1000 milliseconds).

I assume you have not created a board-specific criteria.json file?

> Fuego: all test phases complete!
> Build step 'Execute shell' marked build as failure
> [description-setter] Description set:  <a
> href="/storm/userContent/fuego.logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/testlog.txt">testlog</a> <a
> href="/storm/userContent/fuego.logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/run.json">run.json</a> <a
> href="/storm/userContent/fuego.logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/consolelog.txt">fuegolog</a> <a
> href="/storm/userContent/fuego.logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/devlog.txt">devlog</a> <a
> href="/storm/userContent/fuego.logs/Benchmark.signaltest/TRAV-Sanity-
> Test.default.4.4/prolog.sh">prolog.sh</a>
> Finished: FAILURE

Thanks.
  -- Tim

Re: [Fuego] [Fuego 1.2.1] reference.json file is missing for Benchmark.signaltest

[Dhinakar Kalyanasundaram]

[-- Attachment #1: Type: text/html, Size: 60514 bytes --]

[-- Attachment #2: 201602111742151_N3WZA6X7.png --]
[-- Type: image/png, Size: 33527 bytes --]