Re: [PATCH 2/4] Add use of RCU for qemu_logfile.

[Robert Foley <robert.foley@linaro.org>]

On Thu, 7 Nov 2019 at 11:24, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Robert Foley <robert.foley@linaro.org> writes:
<snip>
> > diff --git a/include/qemu/log.h b/include/qemu/log.h
> > index a91105b2ad..975de18e23 100644
> > --- a/include/qemu/log.h
> > +++ b/include/qemu/log.h
> > @@ -3,9 +3,17 @@
> >
> >  /* A small part of this API is split into its own header */
> >  #include "qemu/log-for-trace.h"
> > +#include "qemu/rcu.h"
> > +
> > +struct QemuLogFile {
> > +    struct rcu_head rcu;
> > +    FILE *fd;
> > +};
> > +typedef struct QemuLogFile QemuLogFile;
>
> If you are declaring a typedef then do it in one:
>
>   typedef struct QemuLogFile { ...
>
> We only really use the second form for opaque structs where the handle
> is passed around but the contents hidden from the rest of QEMU.
>

OK, makes sense.  Will correct it.  Thanks for explaining.

> >
> >  /* Private global variable, don't use */
> > -extern FILE *qemu_logfile;
> > +extern QemuLogFile *qemu_logfile;
> > +
>
> Do we need multiple

Not 100% sure on the meaning here.  Are you asking if we need to
extern this?  We have a few other cases outside of the log module
where this is getting used so the extern is needed here.

> >
> >  /*
> >   * The new API:
> > @@ -25,7 +33,17 @@ static inline bool qemu_log_enabled(void)
> >   */
> >  static inline bool qemu_log_separate(void)
> >  {
> > -    return qemu_logfile != NULL && qemu_logfile != stderr;
> > +    QemuLogFile *logfile;
> > +
> > +    if (qemu_log_enabled()) {
> > +        rcu_read_lock();
> > +        logfile = atomic_rcu_read(&qemu_logfile);
> > +        if (logfile && logfile->fd != stderr) {
> > +            return true;
> > +        }
> > +        rcu_read_unlock();
> > +    }
> > +    return false;
>
> This is unbalanced as you'll have incremented the reader count. Also
> qemu_log_enabled() is also synonymous with logfile existing so you could
> fold that into:
>
>   bool res = false;
>
>   rcu_read_lock();
>   *logfile = atomic_rcu_read(&qemu_logfile);
>   if (logfile && logfile->fd != stderr) {
>      res = true;
>   }
>   rcu_read_unlock();
>   return res;
>
> There is technically a race there as the answer may become invalid after
> qemu_log_separate() returns. However given the users I don't see what
> could fail afterwards.
>

I agree, will make these changes.



> >  }
> >
> >  #define CPU_LOG_TB_OUT_ASM (1 << 0)
> > @@ -55,12 +73,23 @@ static inline bool qemu_log_separate(void)
> >
> >  static inline void qemu_log_lock(void)
> >  {
> > -    qemu_flockfile(qemu_logfile);
> > +    QemuLogFile *logfile;
> > +    rcu_read_lock();
> > +    logfile = atomic_rcu_read(&qemu_logfile);
> > +    if (logfile) {
> > +        qemu_flockfile(logfile->fd);
> > +    }
> > +    rcu_read_unlock();
> >  }
>
> static inline FILE *fd qemu_log_lock(void)
> {
       QemuLogFile *logfile;
>     rcu_read_lock();
       logfile = atomic_rcu_read(&qemu_logfile);
       if (logfile) {
>         qemu_flockfile(logfile->fd);
           return logfile->fd;
>     } else {
>         rcu_read_unlock();
>         return NULL;
>     }
> }
>
> static inline qemu_log_unlock(FILE *fd)
> {
>     if (fd) {
>         qemu_funlockfile(fd);
>         rcu_read_unlock();
>     }
> }
>
> While the rcu_read_lock() is in progress then we won't ever finish with
> the fd - which we don't want to do until the file locking is finished.

Agree with the adjustments you made to qemu_log_lock().  I updated the
code above with a few tweaks for the QemuLogFile type returned by
atomic_rcu_read().  Does this look OK or is there any other adjustment
we should make here?

The intent here was for qemu_log_lock() to hold the rcu_read_lock()
until after we can qemu_funlockfile(fd).  The idea being that we want
to prevent the fclose(fd) from happening by holding the
rcu_read_lock() across the qemu_log_lock() until qemu_log_unlock().
So we have the qemu_funlockfile(fd) first, and then once we're done
with the fd, it is safe to rcu_read_unlock().


> <snip>
<snip more>
> > diff --git a/util/log.c b/util/log.c
<snip>
> > @@ -65,6 +70,8 @@ static bool log_uses_own_buffers;
> >  /* enable or disable low levels log */
> >  void qemu_set_log(int log_flags)
> >  {
> > +    QemuLogFile *logfile;
> > +
> >      qemu_loglevel = log_flags;
> >  #ifdef CONFIG_TRACE_LOG
> >      qemu_loglevel |= LOG_TRACE;
> > @@ -77,43 +84,51 @@ void qemu_set_log(int log_flags)
> >      qemu_mutex_lock(&qemu_logfile_mutex);
> >      if (!qemu_logfile &&
> >          (is_daemonized() ? logfilename != NULL : qemu_loglevel)) {
> > +        logfile = g_new0(QemuLogFile, 1);
> >          if (logfilename) {
>
> You can assume logfilename exists as glib memory allocations would
> abort() otherwise.
This is good to know about the glib memory allocations as it relates
to the logfile above, since we did not add any check for null on
allocation.

We are thinking that logfilename might be NULL if we either never set
it with a call to qemu_set_log_filename(), or (with our intended new
fix) if we took an error actually opening the file in
qemu_set_log_filename(), and went down the error path.

> > -            qemu_logfile = fopen(logfilename, log_append ? "a" : "w");
> > -            if (!qemu_logfile) {
> > +            logfile->fd = fopen(logfilename, log_append ? "a" : "w");
> > +            if (!logfile->fd) {
> > +                g_free(logfile);
> >                  perror(logfilename);
> >                  _exit(1);
<snip>
> > +void qemu_logfile_free(QemuLogFile *logfile)
>
> This can be static as it's internal to log.c

Absolutely, makes sense.  Will change it.
> > +{
> > +    if (logfile) {
>
> g_assert(logfile) instead of the if?

I agree, the assert is cleaner.  Will change it.

> > +        if (logfile->fd != stderr) {
> > +            fclose(logfile->fd);
> > +        }
> > +        g_free(logfile);
> > +    }
> >  }
> >
> >  /* Close the log file */
> >  void qemu_log_close(void)
> >  {
> > -    if (qemu_logfile) {
> > -        if (qemu_logfile != stderr) {
> > -            fclose(qemu_logfile);
> > -        }
> > -        qemu_logfile = NULL;
> > +    QemuLogFile *logfile;
> > +
> > +    qemu_mutex_lock(&qemu_logfile_mutex);
> > +    logfile = qemu_logfile;
> > +
> > +    if (logfile) {
> > +        atomic_rcu_set(&qemu_logfile, NULL);
> > +        qemu_mutex_unlock(&qemu_logfile_mutex);
>
> I think you can move the both the unlocks out of the if/else and drop
> the else.

Agreed.  Originally we were thinking to hold the lock until after the
call_rcu, but I agree with you, it seems safe to move it.

> > +        call_rcu(logfile, qemu_logfile_free, rcu);
> > +    } else {
> > +        qemu_mutex_unlock(&qemu_logfile_mutex);
> >      }
> >  }
<snip>
> > diff --git a/tcg/tcg.c b/tcg/tcg.c
> > index 5475d49ed1..220eaac7c7 100644
> > --- a/tcg/tcg.c
> > +++ b/tcg/tcg.c
> > @@ -2114,9 +2114,17 @@ static void tcg_dump_ops(TCGContext *s, bool have_prefs)
> >          }
> >
> >          if (have_prefs || op->life) {
> > -            for (; col < 40; ++col) {
> > -                putc(' ', qemu_logfile);
> > +
> > +            QemuLogFile *logfile;
> > +
> > +            rcu_read_lock();
> > +            logfile = atomic_rcu_read(&qemu_logfile);
>
> This can probably be a qemu_log_lock() instead given interleaving output
> is going to be confusing.

The function calling tcg_dump_ops(), tcg_gen_code(), already has the
qemu_log_lock() held.

We were thinking about another type of cleanup, but it applies to this
case as well, since this case is using the qemu_logfile.   We were
thinking that at some point (another patch) it might be good to
cleanup the use of the qemu_logfile global so that it is only
referenced from inside the log module.    We noticed a comment in
log.h calling it a private global, which we thought might imply that
it was an intent to keep it private, and that this cleanup might be a
good direction to move in.

Thanks,
-Rob Foley

> > +            if (logfile) {
> > +                for (; col < 40; ++col) {
> > +                    putc(' ', logfile->fd);
> > +                }
> >              }
> > +            rcu_read_unlock();
>
> and qemu_log_unlock();
>
> This cleanup could even be in a separate patch.
>
> >          }
> >
> >          if (op->life) {
>
>
> --
> Alex Bennée