Re: [Fuego] [PATCH] core: add log_this function

[<Tim.Bird@sony.com>]

> -----Original Message-----
> From: Daniel Sangorrin
> Hi Tim,
> 
> I noticed that you are appending the log to testlog. It would be nice to add a
> separator such as:
> Fuego: board A testlog
> ...
> Fuego: host testlog
> ....x
> 
> Also I was thinking that from a systems perspective, it would be nice to be
> able to rename the log files.
> 
> For example, suppose that we have a test that requires 4 boards (A, B, C, D)
> and the test is coordinated by board E (it could be "docker" or the host as
> well).
> 
> Initially, we would run ftc run-test on board E (the coordinator). Then board-
> E's fuego_test.sh would execute
> ftc run-test -b boardname --log boardname.log -t ...
> on boards [A..D].

If the log is coming from another board, and is created from a Fuego-style
test, then the rename can happen during the post-processing phase
of the "master" test, as you indicate below.  It should be possible to just
take the sub-test's testlog.txt, and incorporate it into the master testlog.
(with a suitable header, as you've indicated).

> # it would be nice to have a new category (like Benchmark and Functional)
> for monitoring tests that just check the disk usage or network status.
> Something like MONITORING.free (check memory usage) etc.

I agree.  Monitoring could happen on the DUT, on a separate node,
or on the host (possibly watching a serial port or watching the output
of some other piece of hardware connected to both the DUT and the
host - like a temperature monitor or a power monitor.)  It might be nice
to standardize the output (timestamp, event value), and provide
charting and criteria processing for a monitor's output.
Some monitors I can think of:
Monitor.serial-console-for-panics
Monitor.free-memory
Monitor.disk-IO-load
Monitor.board-power-draw


I'd also like to add an "Action" type job, for things like:
Action.make_report_for_last_batch_run
Action.pre-build-all-test-binaries
Action.provision-board-with-latest-LTS

(those are wordy, but you get the idea)

The point is to share common operations among Fuego users.
 
> Finally, during the post processing phase board E can merge the logs into one
> (using a separator) and give it to the parser.
> 
> # Of course Board E should be able to use Fuego core functions for switching
> on the boards, waiting until all of them are ready etc.

These are good suggestions.  I nearly started add a resource reservation
system to ftc, but I started thinking it should be a separate
standalone command, and it might make sense to integrate it with
a board control system.  Reserving a board is only one type of
resource, but there could be others.  We should talk more about this
in June.
 -- Tim

> > -----Original Message-----
> > From: fuego-bounces@lists.linuxfoundation.org
> > [mailto:fuego-bounces@lists.linuxfoundation.org] On Behalf Of
> > Tim.Bird@sony.com
> > Sent: Tuesday, May 22, 2018 3:40 AM
> > To: fuego@lists.linuxfoundation.org
> > Subject: [Fuego] [PATCH] core: add log_this function
> >
> > Hey Fuego-ans,
> >
> > Here is a patch that I applied to fuego-core last week.  I've been doing
> > some thinking about some longstanding issues with tests that have a
> > host-side component to their data gathering. Based on this, and recent
> > discussions on the list, I implemented a new "log_this" function.
> > It's like the "report" function, but for a host-side command.
> >
> > I believe this will be a new, important architectural feature of Fuego.
> >
> > This is part of a broader effort to expand the scope of Fuego testing, from
> just
> > target-side testing, to more system-wide testing.  It's clear that for some
> types of
> > hardware testing, additional off-DUT frameworks will need to be accessed,
> > and in some cases controlled.  This new function "log_this" is the start of
> > support for logging the access to such non-DUT frameworks
> > (facilities, devices, harnesses, resources, etc.)
> >
> > I'm also thinking about what's needed to provide for generalized control
> > of such things.  This is a tricky subject, due to the incredible fragmentation
> > there is in board control hardware, secondary resource control, and
> associated
> > driving software.
> > However, I'm considering implementing some kind of generic resource
> > reservation and management system (over the long run - this is not the
> highest
> > priority at the moment).
> >
> > In any event, here's the patch for this little bit, which is actually pretty
> simple...
> > --------------
> > Some tests need to get information and data from host-side
> > operations, that needs to be reported and analyzed by Fuego.
> >
> > The log_this function captures the output of commands executed
> > on the host, and puts it (ultimately) into the test log for a run.
> > Any command executed with "log_this" is saved during test execution,
> > and placed in the final testlog.txt, after any
> > board-side log data (from report and report_append) commands.
> >
> > There are several tests (especially Fuego self-tests) that could
> > use this feature, to avoid an awkward sequence of push-to-target,
> > and report-cat, to get log data from the host into the testlog.
> >
> > Signed-off-by: Tim Bird <tim.bird@sony.com>
> > ---
> >  engine/scripts/functions.sh | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> >
> > diff --git a/engine/scripts/functions.sh b/engine/scripts/functions.sh
> > index 0b293db..8fabd85 100755
> > --- a/engine/scripts/functions.sh
> > +++ b/engine/scripts/functions.sh
> > @@ -226,6 +226,21 @@ function report_append {
> >    return ${RESULT}
> >  }
> >
> > +# $1 - local shell command
> > +function log_this {
> > +  is_empty $1
> > +
> > +  RETCODE=/tmp/$$-${RANDOM}
> > +  touch $RETCODE
> > +
> > +  { $1; echo $? > $RETCODE ; } 2>&1 | tee -a ${LOGDIR}/hostlog.txt
> > +
> > +  RESULT=$(cat $RETCODE)
> > +  rm -f $RETCODE
> > +  export REPORT_RETURN_VALUE=${RESULT}
> > +  return ${RESULT}
> > +}
> > +
> >  function dump_syslogs {
> >  # 1 - tmp dir, 2 - before/after
> >
> > @@ -466,6 +481,10 @@ function fetch_results {
> >      get $BOARD_TESTDIR/fuego.$TESTDIR/$TESTDIR.log
> ${LOGDIR}/testlog.txt
> > || \
> >          echo "INFO: the test did not produce a test log on the target" | tee
> > ${LOGDIR}/testlog.txt
> >
> > +    if [ -f ${LOGDIR}/hostlog.txt ] ; then
> > +        cat ${LOGDIR}/hostlog.txt >> ${LOGDIR}/testlog.txt
> > +    fi
> > +
> >      # Get syslogs
> >      dump_syslogs ${fuego_test_tmp} "after"
> >      get
> >
> ${fuego_test_tmp}/${NODE_NAME}.${BUILD_ID}.${BUILD_NUMBER}.befor
> e
> > ${LOGDIR}/syslog.before.txt
> > --
> > 2.1.4
> >
> > _______________________________________________
> > Fuego mailing list
> > Fuego@lists.linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/fuego
> 
> 
> 
> _______________________________________________
> Fuego mailing list
> Fuego@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/fuego