* [PATCH v2] libevent: add granularity to ptest log
@ 2019-09-06 18:51 Trevor Gamblin
2019-09-06 19:02 ` Trevor Gamblin
2019-09-07 12:11 ` Richard Purdie
0 siblings, 2 replies; 4+ messages in thread
From: Trevor Gamblin @ 2019-09-06 18:51 UTC (permalink / raw)
To: openembedded-core
From: Trevor Gamblin <trevor.gamblin@windriver.com>
The libevent ptest used to report only a global pass or a fail result.
Count individual PASS, FAIL, SKIP results. The SKIP results now
include tests that are disabled in the libevent code.
libevent's ptest output did not comply with the automake-style output
"result: testname", and reported a FAIL status at the end of the test
run if any of the libevent tests failed. This patch makes the log
consistent with the automake style:
PASS: http/cancel_by_host_no_ns
PASS: http/cancel_inactive_server
PASS: http/cancel_by_host_no_ns_inactive_server
SKIPPED: http/cancel_by_host_server_timeout
SKIPPED: http/cancel_server_timeout
and provides a summary as follows:
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
---
.../libevent/libevent/run-ptest | 34 ++++++++++++-------
.../libevent/libevent_2.1.11.bb | 3 ++
2 files changed, 25 insertions(+), 12 deletions(-)
diff --git a/meta/recipes-support/libevent/libevent/run-ptest b/meta/recipes-support/libevent/libevent/run-ptest
index 0241851c70..080806dea9 100644
--- a/meta/recipes-support/libevent/libevent/run-ptest
+++ b/meta/recipes-support/libevent/libevent/run-ptest
@@ -1,18 +1,28 @@
#!/bin/sh
-fail=0
+# run-ptest - 'ptest' test infrastructure shell script that
+# wraps the libevent test scripts
+#
+# Trevor Gamblin <trevor.gamblin@windriver.com>
+###############################################################
+LIBEVENTLIB=@libdir@/libevent
+LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
+
+cd ${LIBEVENTLIB}/ptest
+
for test in ./test/*
do
- $test
- if [ $? -ne 0 ]
- then
- fail=1
- fi
+ $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
done
-if [ $fail -eq 0 ]
-then
- echo "PASS: libevent"
-else
- echo "FAIL: libevent"
-fi
+passed=`grep PASS ${LOG}|wc -l`
+failed=`grep FAIL ${LOG}|wc -l`
+skipped=`grep -E SKIP ${LOG}|wc -l`
+all=$((passed + failed + skipped))
+
+( echo "=== Test Summary ==="
+ echo "TOTAL: ${all}"
+ echo "PASSED: ${passed}"
+ echo "FAILED: ${failed}"
+ echo "SKIPPED: ${skipped}"
+) | tee -a ${LOG}
diff --git a/meta/recipes-support/libevent/libevent_2.1.11.bb b/meta/recipes-support/libevent/libevent_2.1.11.bb
index 1e18f0ab2c..f005ab8bda 100644
--- a/meta/recipes-support/libevent/libevent_2.1.11.bb
+++ b/meta/recipes-support/libevent/libevent_2.1.11.bb
@@ -43,4 +43,7 @@ do_install_ptest() {
do
install -m 0755 $file ${D}${PTEST_PATH}/test
done
+
+ # handle multilib
+ sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
}
--
2.21.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH v2] libevent: add granularity to ptest log
2019-09-06 18:51 [PATCH v2] libevent: add granularity to ptest log Trevor Gamblin
@ 2019-09-06 19:02 ` Trevor Gamblin
2019-09-06 19:21 ` Trevor Gamblin
2019-09-07 12:11 ` Richard Purdie
1 sibling, 1 reply; 4+ messages in thread
From: Trevor Gamblin @ 2019-09-06 19:02 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 7042 bytes --]
On 9/6/19 2:51 PM, Trevor Gamblin wrote:
> From: Trevor Gamblin <trevor.gamblin@windriver.com>
>
> The libevent ptest used to report only a global pass or a fail result.
> Count individual PASS, FAIL, SKIP results. The SKIP results now
> include tests that are disabled in the libevent code.
>
> libevent's ptest output did not comply with the automake-style output
> "result: testname", and reported a FAIL status at the end of the test
> run if any of the libevent tests failed. This patch makes the log
> consistent with the automake style:
>
> PASS: http/cancel_by_host_no_ns
> PASS: http/cancel_inactive_server
> PASS: http/cancel_by_host_no_ns_inactive_server
> SKIPPED: http/cancel_by_host_server_timeout
> SKIPPED: http/cancel_server_timeout
>
> and provides a summary as follows:
>
> === Test Summary ===
> TOTAL: 316
> PASSED: 300
> FAILED: 0
> SKIPPED: 16
> DURATION: 87
> END: /usr/lib/libevent/ptest
>
> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
> ---
> .../libevent/libevent/run-ptest | 34 ++++++++++++-------
> .../libevent/libevent_2.1.11.bb | 3 ++
> 2 files changed, 25 insertions(+), 12 deletions(-)
>
> diff --git a/meta/recipes-support/libevent/libevent/run-ptest b/meta/recipes-support/libevent/libevent/run-ptest
> index 0241851c70..080806dea9 100644
> --- a/meta/recipes-support/libevent/libevent/run-ptest
> +++ b/meta/recipes-support/libevent/libevent/run-ptest
> @@ -1,18 +1,28 @@
> #!/bin/sh
>
> -fail=0
> +# run-ptest - 'ptest' test infrastructure shell script that
> +# wraps the libevent test scripts
> +#
> +# Trevor Gamblin <trevor.gamblin@windriver.com>
> +###############################################################
> +LIBEVENTLIB=@libdir@/libevent
> +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
> +
> +cd ${LIBEVENTLIB}/ptest
> +
> for test in ./test/*
> do
> - $test
> - if [ $? -ne 0 ]
> - then
> - fail=1
> - fi
> + $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
> done
>
> -if [ $fail -eq 0 ]
> -then
> - echo "PASS: libevent"
> -else
> - echo "FAIL: libevent"
> -fi
> +passed=`grep PASS ${LOG}|wc -l`
> +failed=`grep FAIL ${LOG}|wc -l`
> +skipped=`grep -E SKIP ${LOG}|wc -l`
> +all=$((passed + failed + skipped))
> +
> +( echo "=== Test Summary ==="
> + echo "TOTAL: ${all}"
> + echo "PASSED: ${passed}"
> + echo "FAILED: ${failed}"
> + echo "SKIPPED: ${skipped}"
> +) | tee -a ${LOG}
> diff --git a/meta/recipes-support/libevent/libevent_2.1.11.bb b/meta/recipes-support/libevent/libevent_2.1.11.bb
> index 1e18f0ab2c..f005ab8bda 100644
> --- a/meta/recipes-support/libevent/libevent_2.1.11.bb
> +++ b/meta/recipes-support/libevent/libevent_2.1.11.bb
> @@ -43,4 +43,7 @@ do_install_ptest() {
> do
> install -m 0755 $file ${D}${PTEST_PATH}/test
> done
> +
> + # handle multilib
> + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
> }
ptest logs are currently still including some performance test outputs
in addition to the PASS/FAIL stuff we are concerned with. That being
said, I figured it'd be better to get a working patch in to use on a
test run before the weekend. Example of the performance output below:||
|||write callback. should only see this once||
||timeout fired, time to end test||
||usec used=80, usec passed=1501872, cpu usage=0.01%||
||closed_cb: detected socket close with success||
||=====expected||
||Now= 1567793437.757017||
||Inserted:||
|| 0x405940 [fd 3] Write||
|| 0x4059d0 [fd 4] Read Persist Timeout=T+1||
|| 0x405a60 [fd 5] Write Persist Timeout=T+1||
|| 0x405af0 [fd 6] Read Timeout=T+2.5||
|| 0x405b80 [fd -1] Timeout=T+1||
|| 0x405c10 [fd -1] Timeout=T+1||
|| 0x405ca0 [fd -1] Timeout=T+1||
|| 0x405d30 [fd -1] Persist Timeout=T+2.5||
|| 0x405dc0 [fd -1] Persist Timeout=T+1||
|| 0x405e50 [fd -1] Persist Timeout=T+2.5||
|| 0x406000 [sig 2] Signal Persist||
||Active:||
|| 0x405ee0 [fd -1, priority=0] Read active||
|| 0x405f70 [fd -1, priority=0] Read Write Timeout active||
|| 0x4059d0 [fd 4, priority=0] Read active||
||======received||
||Inserted events:||
|| 0x405940 [fd 3] Write||
|| 0x4059d0 [fd 4] Read Persist Timeout=1567793438.756966||
|| 0x405a60 [fd 5] Write Persist Timeout=1567793438.756966||
|| 0x405af0 [fd 6] Read Timeout=1567793440.256966||
|| 0x405308 [fd 8] Read Persist Internal||
|| 0x406000 [sig 2] Signal Persist||
|| 0x405800 [fd -1] Internal Timeout=1567793438.756966||
|| 0x4058b0 [fd -1] Internal Timeout=1567793440.256966||
|| 0x405ca0 [fd -1] Timeout=1567793438.756966||
|| 0x405e50 [fd -1] Persist Timeout=1567793440.256966||
|| 0x405b80 [fd -1] Timeout=1567793438.756966||
|| 0x405c10 [fd -1] Timeout=1567793438.756966||
|| 0x405dc0 [fd -1] Persist Timeout=1567793438.756966||
|| 0x405d30 [fd -1] Persist Timeout=1567793440.256966||
||Active events:||
|| 0x4059d0 [fd 4, priority=0] Read active||
|| 0x405ee0 [fd -1, priority=0] Read active||
|| 0x405f70 [fd -1, priority=0] Read Write Timeout active||
||read_cb: read 12||
||read_cb: read 0 - means EOF||
||waiting for 30 conns||
||Average group read bucket level: 0.000000||
||Average group write bucket level: 0.000000||
||1: 683014.200000 per second||
||2: 4389273.600000 per second||
||3: 683014.200000 per second||
||4: 687929.400000 per second||
||5: 683014.200000 per second||
||6: 592950.200000 per second||
||7: 683014.200000 per second||
||8: 687929.400000 per second||
||9: 1433587.200000 per second||
||10: 17691353.400000 per second||
||11: 1433590.400000 per second||
||12: 17667606.200000 per second||
||13: 1202576.000000 per second||
||14: 592950.200000 per second||
||15: 592950.200000 per second||
||16: 17667606.200000 per second||
||17: 1199100.600000 per second||
||18: 16940233.600000 per second||
||19: 1402460.800000 per second||
||20: 17667606.200000 per second||
||21: 17670063.800000 per second||
||22: 592950.200000 per second||
||23: 592950.200000 per second||
||24: 17667606.200000 per second||
||25: 17670063.800000 per second||
||26: 17667606.200000 per second||
||27: 17699551.800000 per second||
||28: 17697907.000000 per second||
||29: 17699551.800000 per second||
||30: 1216486.400000 per second||
|| total: 230756497.800000 per second||
|| average: 7691883.260000 per second||
|| stddev: 8132709.287741 per second||
||209005, 20000||
||write_cb: write 12||
||write_cb: write -1|
At least some of the libevent test scripts output only this content
(i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look into
dropping those from the ptest unless there's a reason we'd want to leave
them in.
||||||
[-- Attachment #2: Type: text/html, Size: 10287 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] libevent: add granularity to ptest log
2019-09-06 19:02 ` Trevor Gamblin
@ 2019-09-06 19:21 ` Trevor Gamblin
0 siblings, 0 replies; 4+ messages in thread
From: Trevor Gamblin @ 2019-09-06 19:21 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 7503 bytes --]
On 9/6/19 3:02 PM, Trevor Gamblin wrote:
>
> On 9/6/19 2:51 PM, Trevor Gamblin wrote:
>
>> From: Trevor Gamblin<trevor.gamblin@windriver.com>
>>
>> The libevent ptest used to report only a global pass or a fail result.
>> Count individual PASS, FAIL, SKIP results. The SKIP results now
>> include tests that are disabled in the libevent code.
>>
>> libevent's ptest output did not comply with the automake-style output
>> "result: testname", and reported a FAIL status at the end of the test
>> run if any of the libevent tests failed. This patch makes the log
>> consistent with the automake style:
>>
>> PASS: http/cancel_by_host_no_ns
>> PASS: http/cancel_inactive_server
>> PASS: http/cancel_by_host_no_ns_inactive_server
>> SKIPPED: http/cancel_by_host_server_timeout
>> SKIPPED: http/cancel_server_timeout
>>
>> and provides a summary as follows:
>>
>> === Test Summary ===
>> TOTAL: 316
>> PASSED: 300
>> FAILED: 0
>> SKIPPED: 16
>> DURATION: 87
>> END: /usr/lib/libevent/ptest
>>
>> Signed-off-by: Trevor Gamblin<trevor.gamblin@windriver.com>
>> ---
>> .../libevent/libevent/run-ptest | 34 ++++++++++++-------
>> .../libevent/libevent_2.1.11.bb | 3 ++
>> 2 files changed, 25 insertions(+), 12 deletions(-)
>>
>> diff --git a/meta/recipes-support/libevent/libevent/run-ptest b/meta/recipes-support/libevent/libevent/run-ptest
>> index 0241851c70..080806dea9 100644
>> --- a/meta/recipes-support/libevent/libevent/run-ptest
>> +++ b/meta/recipes-support/libevent/libevent/run-ptest
>> @@ -1,18 +1,28 @@
>> #!/bin/sh
>>
>> -fail=0
>> +# run-ptest - 'ptest' test infrastructure shell script that
>> +# wraps the libevent test scripts
>> +#
>> +# Trevor Gamblin<trevor.gamblin@windriver.com>
>> +###############################################################
>> +LIBEVENTLIB=@libdir@/libevent
>> +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
>> +
>> +cd ${LIBEVENTLIB}/ptest
>> +
>> for test in ./test/*
>> do
>> - $test
>> - if [ $? -ne 0 ]
>> - then
>> - fail=1
>> - fi
>> + $test 2>&1| sed -e '/OK/ s/^/PASS: / ; /FAILED/ s/^/FAIL: / ; /SKIPPED/ s/^/SKIP: / ; /DISABLED/ s/^/SKIP: /' | cut -f1,2 -d ':' | tee -a ${LOG}
>> done
>>
>> -if [ $fail -eq 0 ]
>> -then
>> - echo "PASS: libevent"
>> -else
>> - echo "FAIL: libevent"
>> -fi
>> +passed=`grep PASS ${LOG}|wc -l`
>> +failed=`grep FAIL ${LOG}|wc -l`
>> +skipped=`grep -E SKIP ${LOG}|wc -l`
>> +all=$((passed + failed + skipped))
>> +
>> +( echo "=== Test Summary ==="
>> + echo "TOTAL: ${all}"
>> + echo "PASSED: ${passed}"
>> + echo "FAILED: ${failed}"
>> + echo "SKIPPED: ${skipped}"
>> +) | tee -a ${LOG}
>> diff --git a/meta/recipes-support/libevent/libevent_2.1.11.bb b/meta/recipes-support/libevent/libevent_2.1.11.bb
>> index 1e18f0ab2c..f005ab8bda 100644
>> --- a/meta/recipes-support/libevent/libevent_2.1.11.bb
>> +++ b/meta/recipes-support/libevent/libevent_2.1.11.bb
>> @@ -43,4 +43,7 @@ do_install_ptest() {
>> do
>> install -m 0755 $file ${D}${PTEST_PATH}/test
>> done
>> +
>> + # handle multilib
>> + sed -i s:@libdir@:${libdir}:g ${D}${PTEST_PATH}/run-ptest
>> }
>
> ptest logs are currently still including some performance test outputs
> in addition to the PASS/FAIL stuff we are concerned with. That being
> said, I figured it'd be better to get a working patch in to use on a
> test run before the weekend. Example of the performance output below:||
>
> |||write callback. should only see this once||
> ||timeout fired, time to end test||
> ||usec used=80, usec passed=1501872, cpu usage=0.01%||
> ||closed_cb: detected socket close with success||
> ||=====expected||
> ||Now= 1567793437.757017||
> ||Inserted:||
> || 0x405940 [fd 3] Write||
> || 0x4059d0 [fd 4] Read Persist Timeout=T+1||
> || 0x405a60 [fd 5] Write Persist Timeout=T+1||
> || 0x405af0 [fd 6] Read Timeout=T+2.5||
> || 0x405b80 [fd -1] Timeout=T+1||
> || 0x405c10 [fd -1] Timeout=T+1||
> || 0x405ca0 [fd -1] Timeout=T+1||
> || 0x405d30 [fd -1] Persist Timeout=T+2.5||
> || 0x405dc0 [fd -1] Persist Timeout=T+1||
> || 0x405e50 [fd -1] Persist Timeout=T+2.5||
> || 0x406000 [sig 2] Signal Persist||
> ||Active:||
> || 0x405ee0 [fd -1, priority=0] Read active||
> || 0x405f70 [fd -1, priority=0] Read Write Timeout active||
> || 0x4059d0 [fd 4, priority=0] Read active||
> ||======received||
> ||Inserted events:||
> || 0x405940 [fd 3] Write||
> || 0x4059d0 [fd 4] Read Persist Timeout=1567793438.756966||
> || 0x405a60 [fd 5] Write Persist Timeout=1567793438.756966||
> || 0x405af0 [fd 6] Read Timeout=1567793440.256966||
> || 0x405308 [fd 8] Read Persist Internal||
> || 0x406000 [sig 2] Signal Persist||
> || 0x405800 [fd -1] Internal Timeout=1567793438.756966||
> || 0x4058b0 [fd -1] Internal Timeout=1567793440.256966||
> || 0x405ca0 [fd -1] Timeout=1567793438.756966||
> || 0x405e50 [fd -1] Persist Timeout=1567793440.256966||
> || 0x405b80 [fd -1] Timeout=1567793438.756966||
> || 0x405c10 [fd -1] Timeout=1567793438.756966||
> || 0x405dc0 [fd -1] Persist Timeout=1567793438.756966||
> || 0x405d30 [fd -1] Persist Timeout=1567793440.256966||
> ||Active events:||
> || 0x4059d0 [fd 4, priority=0] Read active||
> || 0x405ee0 [fd -1, priority=0] Read active||
> || 0x405f70 [fd -1, priority=0] Read Write Timeout active||
> ||read_cb: read 12||
> ||read_cb: read 0 - means EOF||
> ||waiting for 30 conns||
> ||Average group read bucket level: 0.000000||
> ||Average group write bucket level: 0.000000||
> ||1: 683014.200000 per second||
> ||2: 4389273.600000 per second||
> ||3: 683014.200000 per second||
> ||4: 687929.400000 per second||
> ||5: 683014.200000 per second||
> ||6: 592950.200000 per second||
> ||7: 683014.200000 per second||
> ||8: 687929.400000 per second||
> ||9: 1433587.200000 per second||
> ||10: 17691353.400000 per second||
> ||11: 1433590.400000 per second||
> ||12: 17667606.200000 per second||
> ||13: 1202576.000000 per second||
> ||14: 592950.200000 per second||
> ||15: 592950.200000 per second||
> ||16: 17667606.200000 per second||
> ||17: 1199100.600000 per second||
> ||18: 16940233.600000 per second||
> ||19: 1402460.800000 per second||
> ||20: 17667606.200000 per second||
> ||21: 17670063.800000 per second||
> ||22: 592950.200000 per second||
> ||23: 592950.200000 per second||
> ||24: 17667606.200000 per second||
> ||25: 17670063.800000 per second||
> ||26: 17667606.200000 per second||
> ||27: 17699551.800000 per second||
> ||28: 17697907.000000 per second||
> ||29: 17699551.800000 per second||
> ||30: 1216486.400000 per second||
> || total: 230756497.800000 per second||
> || average: 7691883.260000 per second||
> || stddev: 8132709.287741 per second||
> ||209005, 20000||
> ||write_cb: write 12||
> ||write_cb: write -1|
>
> At least some of the libevent test scripts output only this content
> (i.e. they don't run any of the PASS/FAIL/SKIP tests), so I'll look
> into dropping those from the ptest unless there's a reason we'd want
> to leave them in.
> ||||||
>
>
Just determined that only the libevent "regress" test is involved in the
PASS/FAIL tests of interest. I'll submit a v3 on Monday.
[-- Attachment #2: Type: text/html, Size: 11069 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [PATCH v2] libevent: add granularity to ptest log
2019-09-06 18:51 [PATCH v2] libevent: add granularity to ptest log Trevor Gamblin
2019-09-06 19:02 ` Trevor Gamblin
@ 2019-09-07 12:11 ` Richard Purdie
1 sibling, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2019-09-07 12:11 UTC (permalink / raw)
To: Trevor Gamblin, openembedded-core
On Fri, 2019-09-06 at 14:51 -0400, Trevor Gamblin wrote:
> From: Trevor Gamblin <trevor.gamblin@windriver.com>
>
> The libevent ptest used to report only a global pass or a fail
> result.
> Count individual PASS, FAIL, SKIP results. The SKIP results now
> include tests that are disabled in the libevent code.
>
> libevent's ptest output did not comply with the automake-style output
> "result: testname", and reported a FAIL status at the end of the test
> run if any of the libevent tests failed. This patch makes the log
> consistent with the automake style:
>
> PASS: http/cancel_by_host_no_ns
> PASS: http/cancel_inactive_server
> PASS: http/cancel_by_host_no_ns_inactive_server
> SKIPPED: http/cancel_by_host_server_timeout
> SKIPPED: http/cancel_server_timeout
>
> and provides a summary as follows:
>
> === Test Summary ===
> TOTAL: 316
> PASSED: 300
> FAILED: 0
> SKIPPED: 16
> DURATION: 87
> END: /usr/lib/libevent/ptest
>
> Signed-off-by: Trevor Gamblin <trevor.gamblin@windriver.com>
Thanks for this, it is a definite improvement on what was there before.
Results from the autobuilder run:
https://autobuilder.yocto.io/pub/non-release/20190907-4/testresults/testresult-report.txt
Cheers,
Richard
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-09-07 12:11 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-06 18:51 [PATCH v2] libevent: add granularity to ptest log Trevor Gamblin
2019-09-06 19:02 ` Trevor Gamblin
2019-09-06 19:21 ` Trevor Gamblin
2019-09-07 12:11 ` Richard Purdie
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.