* Re: [Fuego] [PATCH] core: add log_this function
2018-05-22 2:21 ` Daniel Sangorrin
@ 2018-05-22 2:31 ` Daniel Sangorrin
2018-05-22 23:20 ` Tim.Bird
1 sibling, 0 replies; 4+ messages in thread
From: Daniel Sangorrin @ 2018-05-22 2:31 UTC (permalink / raw)
To: Tim.Bird, fuego
> -----Original Message-----
> From: fuego-bounces@lists.linuxfoundation.org
> [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of Daniel Sangorrin
> Sent: Tuesday, May 22, 2018 11:21 AM
> To: Tim.Bird@sony.com; fuego@lists.linuxfoundation.org
> Subject: Re: [Fuego] [PATCH] core: add log_this function
>
> Hi Tim,
>
> I noticed that you are appending the log to testlog. It would be nice to add a
> separator such as:
> Fuego: board A testlog
> ...
> Fuego: host testlog
> ....x
>
> Also I was thinking that from a systems perspective, it would be nice to be able to
> rename the log files.
>
> For example, suppose that we have a test that requires 4 boards (A, B, C, D) and the
> test is coordinated by board E (it could be "docker" or the host as well).
>
> Initially, we would run ftc run-test on board E (the coordinator). Then board-E's
> fuego_test.sh would execute
> ftc run-test -b boardname --log boardname.log -t ...
> on boards [A..D].
> # it would be nice to have a new category (like Benchmark and Functional) for
> monitoring tests that just check the disk usage or network status. Something like
> MONITORING.free (check memory usage) etc.
>
> Finally, during the post processing phase board E can merge the logs into one (using
> a separator) and give it to the parser.
Also, board-E should merge the JSON results for each subtest (e.g. each ftc run-test executed).
Probably the best idea would be to use the global result of the test (instead of the results of each test set or test case) and add it the board-E's JSON file as a test case result. And maybe add a link to the detailed JSON file produced by each board.
>
> # Of course Board E should be able to use Fuego core functions for switching on the
> boards, waiting until all of them are ready etc.
>
> Thanks,
> Daniel
>
>
>
>
>
>
>
>
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org
> > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of
> > Tim.Bird@sony.com
> > Sent: Tuesday, May 22, 2018 3:40 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH] core: add log_this function
> >
> > Hey Fuego-ans,
> >
> > Here is a patch that I applied to fuego-core last week. I've been doing
> > some thinking about some longstanding issues with tests that have a
> > host-side component to their data gathering. Based on this, and recent
> > discussions on the list, I implemented a new "log_this" function.
> > It's like the "report" function, but for a host-side command.
> >
> > I believe this will be a new, important architectural feature of Fuego.
> >
> > This is part of a broader effort to expand the scope of Fuego testing, from just
> > target-side testing, to more system-wide testing. It's clear that for some types of
> > hardware testing, additional off-DUT frameworks will need to be accessed,
> > and in some cases controlled. This new function "log_this" is the start of
> > support for logging the access to such non-DUT frameworks
> > (facilities, devices, harnesses, resources, etc.)
> >
> > I'm also thinking about what's needed to provide for generalized control
> > of such things. This is a tricky subject, due to the incredible fragmentation
> > there is in board control hardware, secondary resource control, and associated
> > driving software.
> > However, I'm considering implementing some kind of generic resource
> > reservation and management system (over the long run - this is not the highest
> > priority at the moment).
> >
> > In any event, here's the patch for this little bit, which is actually pretty simple...
> > --------------
> > Some tests need to get information and data from host-side
> > operations, that needs to be reported and analyzed by Fuego.
> >
> > The log_this function captures the output of commands executed
> > on the host, and puts it (ultimately) into the test log for a run.
> > Any command executed with "log_this" is saved during test execution,
> > and placed in the final testlog.txt, after any
> > board-side log data (from report and report_append) commands.
> >
> > There are several tests (especially Fuego self-tests) that could
> > use this feature, to avoid an awkward sequence of push-to-target,
> > and report-cat, to get log data from the host into the testlog.
> >
> > Signed-off-by: Tim Bird <tim.bird@sony.com>
> > ---
> > engine/scripts/functions.sh | 19 +++++++++++++++++++
> > 1 file changed, 19 insertions(+)
> >
> > diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> > index 0b293db..8fabd85 100755
> > --- a/engine/scripts/functions.sh
> > +++ b/engine/scripts/functions.sh
> > @@ -226,6 +226,21 @@ function report_append {
> > return ${RESULT}
> > }
> >
> > +# $1 - local shell command
> > +function log_this {
> > + is_empty $1
> > +
> > + RETCODE=/tmp/$$-${RANDOM}
> > + touch $RETCODE
> > +
> > + { $1; echo $? > $RETCODE ; } 2>&1 | tee -a ${LOGDIR}/hostlog.txt
> > +
> > + RESULT=$(cat $RETCODE)
> > + rm -f $RETCODE
> > + export REPORT_RETURN_VALUE=${RESULT}
> > + return ${RESULT}
> > +}
> > +
> > function dump_syslogs {
> > # 1 - tmp dir, 2 - before/after
> >
> > @@ -466,6 +481,10 @@ function fetch_results {
> > get $BOARD_TESTDIR/fuego.$TESTDIR/$TESTDIR.log
> ${LOGDIR}/testlog.txt
> > || \
> > echo "INFO: the test did not produce a test log on the target" | tee
> > ${LOGDIR}/testlog.txt
> >
> > + if [ -f ${LOGDIR}/hostlog.txt ] ; then
> > + cat ${LOGDIR}/hostlog.txt >> ${LOGDIR}/testlog.txt
> > + fi
> > +
> > # Get syslogs
> > dump_syslogs ${fuego_test_tmp} "after"
> > get
> > ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.before
> > ${LOGDIR}/syslog.before.txt
> > --
> > 2.1.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
>
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [Fuego] [PATCH] core: add log_this function
2018-05-22 2:21 ` Daniel Sangorrin
2018-05-22 2:31 ` Daniel Sangorrin
@ 2018-05-22 23:20 ` Tim.Bird
1 sibling, 0 replies; 4+ messages in thread
From: Tim.Bird @ 2018-05-22 23:20 UTC (permalink / raw)
To: daniel.sangorrin, fuego
> -----Original Message-----
> From: Daniel Sangorrin
> Hi Tim,
>
> I noticed that you are appending the log to testlog. It would be nice to add a
> separator such as:
> Fuego: board A testlog
> ...
> Fuego: host testlog
> ....x
>
> Also I was thinking that from a systems perspective, it would be nice to be
> able to rename the log files.
>
> For example, suppose that we have a test that requires 4 boards (A, B, C, D)
> and the test is coordinated by board E (it could be "docker" or the host as
> well).
>
> Initially, we would run ftc run-test on board E (the coordinator). Then board-
> E's fuego_test.sh would execute
> ftc run-test -b boardname --log boardname.log -t ...
> on boards [A..D].
If the log is coming from another board, and is created from a Fuego-style
test, then the rename can happen during the post-processing phase
of the "master" test, as you indicate below. It should be possible to just
take the sub-test's testlog.txt, and incorporate it into the master testlog.
(with a suitable header, as you've indicated).
> # it would be nice to have a new category (like Benchmark and Functional)
> for monitoring tests that just check the disk usage or network status.
> Something like MONITORING.free (check memory usage) etc.
I agree. Monitoring could happen on the DUT, on a separate node,
or on the host (possibly watching a serial port or watching the output
of some other piece of hardware connected to both the DUT and the
host - like a temperature monitor or a power monitor.) It might be nice
to standardize the output (timestamp, event value), and provide
charting and criteria processing for a monitor's output.
Some monitors I can think of:
Monitor.serial-console-for-panics
Monitor.free-memory
Monitor.disk-IO-load
Monitor.board-power-draw
I'd also like to add an "Action" type job, for things like:
Action.make_report_for_last_batch_run
Action.pre-build-all-test-binaries
Action.provision-board-with-latest-LTS
(those are wordy, but you get the idea)
The point is to share common operations among Fuego users.
> Finally, during the post processing phase board E can merge the logs into one
> (using a separator) and give it to the parser.
>
> # Of course Board E should be able to use Fuego core functions for switching
> on the boards, waiting until all of them are ready etc.
These are good suggestions. I nearly started add a resource reservation
system to ftc, but I started thinking it should be a separate
standalone command, and it might make sense to integrate it with
a board control system. Reserving a board is only one type of
resource, but there could be others. We should talk more about this
in June.
-- Tim
> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org
> > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of
> > Tim.Bird@sony.com
> > Sent: Tuesday, May 22, 2018 3:40 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH] core: add log_this function
> >
> > Hey Fuego-ans,
> >
> > Here is a patch that I applied to fuego-core last week. I've been doing
> > some thinking about some longstanding issues with tests that have a
> > host-side component to their data gathering. Based on this, and recent
> > discussions on the list, I implemented a new "log_this" function.
> > It's like the "report" function, but for a host-side command.
> >
> > I believe this will be a new, important architectural feature of Fuego.
> >
> > This is part of a broader effort to expand the scope of Fuego testing, from
> just
> > target-side testing, to more system-wide testing. It's clear that for some
> types of
> > hardware testing, additional off-DUT frameworks will need to be accessed,
> > and in some cases controlled. This new function "log_this" is the start of
> > support for logging the access to such non-DUT frameworks
> > (facilities, devices, harnesses, resources, etc.)
> >
> > I'm also thinking about what's needed to provide for generalized control
> > of such things. This is a tricky subject, due to the incredible fragmentation
> > there is in board control hardware, secondary resource control, and
> associated
> > driving software.
> > However, I'm considering implementing some kind of generic resource
> > reservation and management system (over the long run - this is not the
> highest
> > priority at the moment).
> >
> > In any event, here's the patch for this little bit, which is actually pretty
> simple...
> > --------------
> > Some tests need to get information and data from host-side
> > operations, that needs to be reported and analyzed by Fuego.
> >
> > The log_this function captures the output of commands executed
> > on the host, and puts it (ultimately) into the test log for a run.
> > Any command executed with "log_this" is saved during test execution,
> > and placed in the final testlog.txt, after any
> > board-side log data (from report and report_append) commands.
> >
> > There are several tests (especially Fuego self-tests) that could
> > use this feature, to avoid an awkward sequence of push-to-target,
> > and report-cat, to get log data from the host into the testlog.
> >
> > Signed-off-by: Tim Bird <tim.bird@sony.com>
> > ---
> > engine/scripts/functions.sh | 19 +++++++++++++++++++
> > 1 file changed, 19 insertions(+)
> >
> > diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> > index 0b293db..8fabd85 100755
> > --- a/engine/scripts/functions.sh
> > +++ b/engine/scripts/functions.sh
> > @@ -226,6 +226,21 @@ function report_append {
> > return ${RESULT}
> > }
> >
> > +# $1 - local shell command
> > +function log_this {
> > + is_empty $1
> > +
> > + RETCODE=/tmp/$$-${RANDOM}
> > + touch $RETCODE
> > +
> > + { $1; echo $? > $RETCODE ; } 2>&1 | tee -a ${LOGDIR}/hostlog.txt
> > +
> > + RESULT=$(cat $RETCODE)
> > + rm -f $RETCODE
> > + export REPORT_RETURN_VALUE=${RESULT}
> > + return ${RESULT}
> > +}
> > +
> > function dump_syslogs {
> > # 1 - tmp dir, 2 - before/after
> >
> > @@ -466,6 +481,10 @@ function fetch_results {
> > get $BOARD_TESTDIR/fuego.$TESTDIR/$TESTDIR.log
> ${LOGDIR}/testlog.txt
> > || \
> > echo "INFO: the test did not produce a test log on the target" | tee
> > ${LOGDIR}/testlog.txt
> >
> > + if [ -f ${LOGDIR}/hostlog.txt ] ; then
> > + cat ${LOGDIR}/hostlog.txt >> ${LOGDIR}/testlog.txt
> > + fi
> > +
> > # Get syslogs
> > dump_syslogs ${fuego_test_tmp} "after"
> > get
> >
> ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.befor
> e
> > ${LOGDIR}/syslog.before.txt
> > --
> > 2.1.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
>
>
>
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego
^ permalink raw reply [flat|nested] 4+ messages in thread