XSA-180 follow-up: repurpose xenconsoled for logging

XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

Hi all

During the discussion of XSA-180 Ian came up with the idea that we
repurpose xenconsoled to handle logging. I've done some research (and
some coding as well!).

In my reply to George the other day:

> I just read the code of virtlogd and xenconsoled.
> 
> I think xenconsoled is missing at least things.
> 
> From a higher level:
> 
> 1. Abstraction of rotating file.
> 2. Abstraction of client.
> 3. IPC interface to libxl -- presumably we need to create a socket.
> 

I've done #1 and port existing code to use that -- would be useful in
general.

#2 is not too hard because xenconsoled already has concept of a domain.
I suspect some refactoring will be fine.

#3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
pretty sure we *don't* want that for xenconsoled. So I spent some time
this morning and write up a draft for a xenstore based protocol. See
below.

Also there is an implication here: we put xenconsoled in a really
critical path. If for some reason it doesn't work all guests are
blocked. Do we really want to do this?

Wei.


XXX DRAFT DRAFT DRAFT XXX

Per domain logging via xenconsoled
==================================

As of Xen release XXX, xenconsoled is repurposed to handle logging for
QEMU. Libxenlight will arrange xenconsoled to create and handle the
log file. It's possible to expose API(s) so that the user of
libxenlight can leverage such ability, but it is not currently done.

Xenstore path and nodes
-----------------------

Libxenlight communicates with xenconsoled via a simple xenstore based
protocol.  All information for a specific domain is stored under
/libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).

Several xenstore nodes are needed (placed under logging/$LOGFILEID).

  pipe: the absolute path of the logging pipe
  file: the absolute path of the file to write to
  limit: the maximum length of the file before it gets rotated
  copies: the number of copies to keep
  state: xenconsoled writes "ready" to this node to indicate readiness

Xenconsoled will sanitise both pipe and file fields. Pipe has to be
placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
(XXX doesn't seem to be configurable at the moment, should introduce
XEN_LOG_DIR?).

Libxenlight and xenconsoled interaction
---------------------------------------

Initiate logging
----------------

1. Libxenlight:
  1. Generates a unique log file id $LOGFILEID
  2. Creates a pipe $PIPE
  3. Writes parameter to xenstore
  4. Wait for readiness indication
2. Xenconsoled
  1. Watch global logging and per domain logging xenstore paths
  2. Gets notified, read parameters from xenstore
  3. Sanitise parameters
  4. Create log files
  5. Connect to the pipe provided
  6. Write "ready" to xenstore state node
3. Libxenlight:
  1. Detects ready state from xenconsoled
  2. Open the pipe and return relevant handles to user

In case of xenconsoled failure, libxenlight will time out and bail.

Clean up
--------

When doing per domain logging, libxenlight will remove all domain
specific xenstore paths when a guest is gone. Xenconsoled use that to
do clean up.

Libxenlight is responsible for deleting the pipe.

Global logging
--------------

Since we don't plan to provide new APIs now, we don't support global
logging because that would require us to provide a cleanup API that
libxenlight users can call.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

On 01/06/16 15:00, Wei Liu wrote:
> Hi all
> 
> During the discussion of XSA-180 Ian came up with the idea that we
> repurpose xenconsoled to handle logging. I've done some research (and
> some coding as well!).
> 
> In my reply to George the other day:
> 
>> I just read the code of virtlogd and xenconsoled.
>>
>> I think xenconsoled is missing at least things.
>>
>> From a higher level:
>>
>> 1. Abstraction of rotating file.
>> 2. Abstraction of client.
>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>
> 
> I've done #1 and port existing code to use that -- would be useful in
> general.
> 
> #2 is not too hard because xenconsoled already has concept of a domain.
> I suspect some refactoring will be fine.
> 
> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> pretty sure we *don't* want that for xenconsoled. So I spent some time
> this morning and write up a draft for a xenstore based protocol. See
> below.
> 
> Also there is an implication here: we put xenconsoled in a really
> critical path. If for some reason it doesn't work all guests are
> blocked. Do we really want to do this?
> 
> Wei.
> 
> 
> XXX DRAFT DRAFT DRAFT XXX
> 
> Per domain logging via xenconsoled
> ==================================
> 
> As of Xen release XXX, xenconsoled is repurposed to handle logging for
> QEMU. Libxenlight will arrange xenconsoled to create and handle the
> log file. It's possible to expose API(s) so that the user of
> libxenlight can leverage such ability, but it is not currently done.
> 
> Xenstore path and nodes
> -----------------------
> 
> Libxenlight communicates with xenconsoled via a simple xenstore based
> protocol.  All information for a specific domain is stored under
> /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).
> 
> Several xenstore nodes are needed (placed under logging/$LOGFILEID).
> 
>   pipe: the absolute path of the logging pipe
>   file: the absolute path of the file to write to
>   limit: the maximum length of the file before it gets rotated
>   copies: the number of copies to keep
>   state: xenconsoled writes "ready" to this node to indicate readiness
> 
> Xenconsoled will sanitise both pipe and file fields. Pipe has to be
> placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
> (XXX doesn't seem to be configurable at the moment, should introduce
> XEN_LOG_DIR?).
> 
> Libxenlight and xenconsoled interaction
> ---------------------------------------
> 
> Initiate logging
> ----------------
> 
> 1. Libxenlight:
>   1. Generates a unique log file id $LOGFILEID
>   2. Creates a pipe $PIPE
>   3. Writes parameter to xenstore
>   4. Wait for readiness indication
> 2. Xenconsoled
>   1. Watch global logging and per domain logging xenstore paths
>   2. Gets notified, read parameters from xenstore
>   3. Sanitise parameters
>   4. Create log files
>   5. Connect to the pipe provided
>   6. Write "ready" to xenstore state node
> 3. Libxenlight:
>   1. Detects ready state from xenconsoled
>   2. Open the pipe and return relevant handles to user
> 
> In case of xenconsoled failure, libxenlight will time out and bail.

...and in the case of domain logging (i.e., qemu), it will either pipe
it to /dev/null instead, or fail creation of the domain (whichever is
the most sensible option given the requested configuration)?

This seems OK to me, but I don't feel like  I have enough experience
with xenstore protocols to know where the traps are.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Fri, Jun 03, 2016 at 11:57:14AM +0100, George Dunlap wrote:
> On 01/06/16 15:00, Wei Liu wrote:
> > Hi all
> > 
> > During the discussion of XSA-180 Ian came up with the idea that we
> > repurpose xenconsoled to handle logging. I've done some research (and
> > some coding as well!).
> > 
> > In my reply to George the other day:
> > 
> >> I just read the code of virtlogd and xenconsoled.
> >>
> >> I think xenconsoled is missing at least things.
> >>
> >> From a higher level:
> >>
> >> 1. Abstraction of rotating file.
> >> 2. Abstraction of client.
> >> 3. IPC interface to libxl -- presumably we need to create a socket.
> >>
> > 
> > I've done #1 and port existing code to use that -- would be useful in
> > general.
> > 
> > #2 is not too hard because xenconsoled already has concept of a domain.
> > I suspect some refactoring will be fine.
> > 
> > #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> > pretty sure we *don't* want that for xenconsoled. So I spent some time
> > this morning and write up a draft for a xenstore based protocol. See
> > below.
> > 
> > Also there is an implication here: we put xenconsoled in a really
> > critical path. If for some reason it doesn't work all guests are
> > blocked. Do we really want to do this?
> > 
> > Wei.
> > 
> > 
> > XXX DRAFT DRAFT DRAFT XXX
> > 
> > Per domain logging via xenconsoled
> > ==================================
> > 
> > As of Xen release XXX, xenconsoled is repurposed to handle logging for
> > QEMU. Libxenlight will arrange xenconsoled to create and handle the
> > log file. It's possible to expose API(s) so that the user of
> > libxenlight can leverage such ability, but it is not currently done.
> > 
> > Xenstore path and nodes
> > -----------------------
> > 
> > Libxenlight communicates with xenconsoled via a simple xenstore based
> > protocol.  All information for a specific domain is stored under
> > /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).
> > 
> > Several xenstore nodes are needed (placed under logging/$LOGFILEID).
> > 
> >   pipe: the absolute path of the logging pipe
> >   file: the absolute path of the file to write to
> >   limit: the maximum length of the file before it gets rotated
> >   copies: the number of copies to keep
> >   state: xenconsoled writes "ready" to this node to indicate readiness
> > 
> > Xenconsoled will sanitise both pipe and file fields. Pipe has to be
> > placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
> > (XXX doesn't seem to be configurable at the moment, should introduce
> > XEN_LOG_DIR?).
> > 
> > Libxenlight and xenconsoled interaction
> > ---------------------------------------
> > 
> > Initiate logging
> > ----------------
> > 
> > 1. Libxenlight:
> >   1. Generates a unique log file id $LOGFILEID
> >   2. Creates a pipe $PIPE
> >   3. Writes parameter to xenstore
> >   4. Wait for readiness indication
> > 2. Xenconsoled
> >   1. Watch global logging and per domain logging xenstore paths
> >   2. Gets notified, read parameters from xenstore
> >   3. Sanitise parameters
> >   4. Create log files
> >   5. Connect to the pipe provided
> >   6. Write "ready" to xenstore state node
> > 3. Libxenlight:
> >   1. Detects ready state from xenconsoled
> >   2. Open the pipe and return relevant handles to user
> > 
> > In case of xenconsoled failure, libxenlight will time out and bail.
> 
> ...and in the case of domain logging (i.e., qemu), it will either pipe
> it to /dev/null instead, or fail creation of the domain (whichever is
> the most sensible option given the requested configuration)?
> 

Obtaining a logfile fd is a step prior to QEMU creation.  The creation
will fail because a critical component in the system is not available.
Another strategy is to warn but continue.

Wei.

> This seems OK to me, but I don't feel like  I have enough experience
> with xenstore protocols to know where the traps are.
> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

On 03/06/16 14:30, Wei Liu wrote:
> On Fri, Jun 03, 2016 at 11:57:14AM +0100, George Dunlap wrote:
>> On 01/06/16 15:00, Wei Liu wrote:
>>> Hi all
>>>
>>> During the discussion of XSA-180 Ian came up with the idea that we
>>> repurpose xenconsoled to handle logging. I've done some research (and
>>> some coding as well!).
>>>
>>> In my reply to George the other day:
>>>
>>>> I just read the code of virtlogd and xenconsoled.
>>>>
>>>> I think xenconsoled is missing at least things.
>>>>
>>>> From a higher level:
>>>>
>>>> 1. Abstraction of rotating file.
>>>> 2. Abstraction of client.
>>>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>>>
>>>
>>> I've done #1 and port existing code to use that -- would be useful in
>>> general.
>>>
>>> #2 is not too hard because xenconsoled already has concept of a domain.
>>> I suspect some refactoring will be fine.
>>>
>>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
>>> pretty sure we *don't* want that for xenconsoled. So I spent some time
>>> this morning and write up a draft for a xenstore based protocol. See
>>> below.
>>>
>>> Also there is an implication here: we put xenconsoled in a really
>>> critical path. If for some reason it doesn't work all guests are
>>> blocked. Do we really want to do this?
>>>
>>> Wei.
>>>
>>>
>>> XXX DRAFT DRAFT DRAFT XXX
>>>
>>> Per domain logging via xenconsoled
>>> ==================================
>>>
>>> As of Xen release XXX, xenconsoled is repurposed to handle logging for
>>> QEMU. Libxenlight will arrange xenconsoled to create and handle the
>>> log file. It's possible to expose API(s) so that the user of
>>> libxenlight can leverage such ability, but it is not currently done.
>>>
>>> Xenstore path and nodes
>>> -----------------------
>>>
>>> Libxenlight communicates with xenconsoled via a simple xenstore based
>>> protocol.  All information for a specific domain is stored under
>>> /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).
>>>
>>> Several xenstore nodes are needed (placed under logging/$LOGFILEID).
>>>
>>>   pipe: the absolute path of the logging pipe
>>>   file: the absolute path of the file to write to
>>>   limit: the maximum length of the file before it gets rotated
>>>   copies: the number of copies to keep
>>>   state: xenconsoled writes "ready" to this node to indicate readiness
>>>
>>> Xenconsoled will sanitise both pipe and file fields. Pipe has to be
>>> placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
>>> (XXX doesn't seem to be configurable at the moment, should introduce
>>> XEN_LOG_DIR?).
>>>
>>> Libxenlight and xenconsoled interaction
>>> ---------------------------------------
>>>
>>> Initiate logging
>>> ----------------
>>>
>>> 1. Libxenlight:
>>>   1. Generates a unique log file id $LOGFILEID
>>>   2. Creates a pipe $PIPE
>>>   3. Writes parameter to xenstore
>>>   4. Wait for readiness indication
>>> 2. Xenconsoled
>>>   1. Watch global logging and per domain logging xenstore paths
>>>   2. Gets notified, read parameters from xenstore
>>>   3. Sanitise parameters
>>>   4. Create log files
>>>   5. Connect to the pipe provided
>>>   6. Write "ready" to xenstore state node
>>> 3. Libxenlight:
>>>   1. Detects ready state from xenconsoled
>>>   2. Open the pipe and return relevant handles to user
>>>
>>> In case of xenconsoled failure, libxenlight will time out and bail.
>>
>> ...and in the case of domain logging (i.e., qemu), it will either pipe
>> it to /dev/null instead, or fail creation of the domain (whichever is
>> the most sensible option given the requested configuration)?
>>
> 
> Obtaining a logfile fd is a step prior to QEMU creation.  The creation
> will fail because a critical component in the system is not available.
> Another strategy is to warn but continue.

Right, this is what I meant.  If the user has explicitly asked for qemu
to be logged somewhere, and that fails, then we should probably fail
with an error.  But if the user hasn't asked for anything specific, and
we're just doing the default "log to a file", then continuing the domain
creation with a warning might be a better option.

 -George



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Fri, Jun 03, 2016 at 03:10:32PM +0100, George Dunlap wrote:
> On 03/06/16 14:30, Wei Liu wrote:
> > On Fri, Jun 03, 2016 at 11:57:14AM +0100, George Dunlap wrote:
> >> On 01/06/16 15:00, Wei Liu wrote:
> >>> Hi all
> >>>
> >>> During the discussion of XSA-180 Ian came up with the idea that we
> >>> repurpose xenconsoled to handle logging. I've done some research (and
> >>> some coding as well!).
> >>>
> >>> In my reply to George the other day:
> >>>
> >>>> I just read the code of virtlogd and xenconsoled.
> >>>>
> >>>> I think xenconsoled is missing at least things.
> >>>>
> >>>> From a higher level:
> >>>>
> >>>> 1. Abstraction of rotating file.
> >>>> 2. Abstraction of client.
> >>>> 3. IPC interface to libxl -- presumably we need to create a socket.
> >>>>
> >>>
> >>> I've done #1 and port existing code to use that -- would be useful in
> >>> general.
> >>>
> >>> #2 is not too hard because xenconsoled already has concept of a domain.
> >>> I suspect some refactoring will be fine.
> >>>
> >>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> >>> pretty sure we *don't* want that for xenconsoled. So I spent some time
> >>> this morning and write up a draft for a xenstore based protocol. See
> >>> below.
> >>>
> >>> Also there is an implication here: we put xenconsoled in a really
> >>> critical path. If for some reason it doesn't work all guests are
> >>> blocked. Do we really want to do this?
> >>>
> >>> Wei.
> >>>
> >>>
> >>> XXX DRAFT DRAFT DRAFT XXX
> >>>
> >>> Per domain logging via xenconsoled
> >>> ==================================
> >>>
> >>> As of Xen release XXX, xenconsoled is repurposed to handle logging for
> >>> QEMU. Libxenlight will arrange xenconsoled to create and handle the
> >>> log file. It's possible to expose API(s) so that the user of
> >>> libxenlight can leverage such ability, but it is not currently done.
> >>>
> >>> Xenstore path and nodes
> >>> -----------------------
> >>>
> >>> Libxenlight communicates with xenconsoled via a simple xenstore based
> >>> protocol.  All information for a specific domain is stored under
> >>> /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).
> >>>
> >>> Several xenstore nodes are needed (placed under logging/$LOGFILEID).
> >>>
> >>>   pipe: the absolute path of the logging pipe
> >>>   file: the absolute path of the file to write to
> >>>   limit: the maximum length of the file before it gets rotated
> >>>   copies: the number of copies to keep
> >>>   state: xenconsoled writes "ready" to this node to indicate readiness
> >>>
> >>> Xenconsoled will sanitise both pipe and file fields. Pipe has to be
> >>> placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
> >>> (XXX doesn't seem to be configurable at the moment, should introduce
> >>> XEN_LOG_DIR?).
> >>>
> >>> Libxenlight and xenconsoled interaction
> >>> ---------------------------------------
> >>>
> >>> Initiate logging
> >>> ----------------
> >>>
> >>> 1. Libxenlight:
> >>>   1. Generates a unique log file id $LOGFILEID
> >>>   2. Creates a pipe $PIPE
> >>>   3. Writes parameter to xenstore
> >>>   4. Wait for readiness indication
> >>> 2. Xenconsoled
> >>>   1. Watch global logging and per domain logging xenstore paths
> >>>   2. Gets notified, read parameters from xenstore
> >>>   3. Sanitise parameters
> >>>   4. Create log files
> >>>   5. Connect to the pipe provided
> >>>   6. Write "ready" to xenstore state node
> >>> 3. Libxenlight:
> >>>   1. Detects ready state from xenconsoled
> >>>   2. Open the pipe and return relevant handles to user
> >>>
> >>> In case of xenconsoled failure, libxenlight will time out and bail.
> >>
> >> ...and in the case of domain logging (i.e., qemu), it will either pipe
> >> it to /dev/null instead, or fail creation of the domain (whichever is
> >> the most sensible option given the requested configuration)?
> >>
> > 
> > Obtaining a logfile fd is a step prior to QEMU creation.  The creation
> > will fail because a critical component in the system is not available.
> > Another strategy is to warn but continue.
> 
> Right, this is what I meant.  If the user has explicitly asked for qemu
> to be logged somewhere, and that fails, then we should probably fail
> with an error.  But if the user hasn't asked for anything specific, and
> we're just doing the default "log to a file", then continuing the domain
> creation with a warning might be a better option.
> 

That's an option, but that requires changing user visible interface
which I explicitly avoided in this document.

Wei.

>  -George
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Ian Jackson]

Wei Liu writes ("XSA-180 follow-up: repurpose xenconsoled for logging"):
> XXX DRAFT DRAFT DRAFT XXX

Thanks!

> Per domain logging via xenconsoled
> ==================================
> 
> As of Xen release XXX, xenconsoled is repurposed to handle logging for
> QEMU. Libxenlight will arrange xenconsoled to create and handle the
> log file. It's possible to expose API(s) so that the user of
> libxenlight can leverage such ability, but it is not currently done.
> 
> Xenstore path and nodes
> -----------------------
> 
> Libxenlight communicates with xenconsoled via a simple xenstore based
> protocol.  All information for a specific domain is stored under
> /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).

Are these IDs short strings or what ?

> Several xenstore nodes are needed (placed under logging/$LOGFILEID).
> 
>   pipe: the absolute path of the logging pipe
>   file: the absolute path of the file to write to
>   limit: the maximum length of the file before it gets rotated
>   copies: the number of copies to keep
>   state: xenconsoled writes "ready" to this node to indicate readiness

The use of named pipes for rendezvous is a little unusual, but I think
given that xenconsoled is primarily driven by xenstore it is probably
appropriate.

> Xenconsoled will sanitise both pipe and file fields. Pipe has to be
> placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
> (XXX doesn't seem to be configurable at the moment, should introduce
> XEN_LOG_DIR?).

Yes.

NB the pipe has to be mode 600.  Also, opening a fifo has to be done
with care.  For example, the following dance is needed get a
write-only fd without risking blocking:

   fdrw = open("/var/run/xen/DOMAIN.qemu.fifo", O_RDWR);
   fdwo = open("/var/run/xen/DOMAIN.qemu.fifo", O_WRONLY);
   close(fdwr);

But I think qemu should be provided with a O_RDWR fd.  This is because
otherwise, if xenconsoled crashes or is restarted or something, qemu
would get a SIGPIPE (or EPIPE) from writes to its stderr.

It is better for a qemu in this situation to block waiting for a new
xenconsoled to appear.

> 1. Libxenlight:
>   1. Generates a unique log file id $LOGFILEID
>   2. Creates a pipe $PIPE
>   3. Writes parameter to xenstore
>   4. Wait for readiness indication
> 2. Xenconsoled
>   1. Watch global logging and per domain logging xenstore paths
>   2. Gets notified, read parameters from xenstore
>   3. Sanitise parameters
>   4. Create log files
>   5. Connect to the pipe provided
>   6. Write "ready" to xenstore state node

Is it actually necessary to synchronise ?  If xenconsoled is slow, the
approach above will simply block qemu's writes once the pipe buffer is
full, until xenconsoled catches up.

If xenconsoled is missing the lack of synchronisation will result in
qemu blocking, perhaps tripping the domain startup qemu readiness
timeout.

> 3. Libxenlight:
>   1. Detects ready state from xenconsoled
>   2. Open the pipe and return relevant handles to user
> 
> In case of xenconsoled failure, libxenlight will time out and bail.

Would it be possible for a user to connect to xenconsoled and get a
copy of the qemu output with a suitable `xl console' rune ?

> Clean up
> --------
> 
> When doing per domain logging, libxenlight will remove all domain
> specific xenstore paths when a guest is gone. Xenconsoled use that to
> do clean up.
> 
> Libxenlight is responsible for deleting the pipe.

The logfiles will presumably remain.

> Global logging
> --------------
> 
> Since we don't plan to provide new APIs now, we don't support global
> logging because that would require us to provide a cleanup API that
> libxenlight users can call.

I don't understand what you mean by `global logging'.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Andrew Cooper]

On 01/06/16 15:00, Wei Liu wrote:
> Hi all
>
> During the discussion of XSA-180 Ian came up with the idea that we
> repurpose xenconsoled to handle logging. I've done some research (and
> some coding as well!).
>
> In my reply to George the other day:
>
>> I just read the code of virtlogd and xenconsoled.
>>
>> I think xenconsoled is missing at least things.
>>
>> From a higher level:
>>
>> 1. Abstraction of rotating file.
>> 2. Abstraction of client.
>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>
> I've done #1 and port existing code to use that -- would be useful in
> general.
>
> #2 is not too hard because xenconsoled already has concept of a domain.
> I suspect some refactoring will be fine.
>
> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> pretty sure we *don't* want that for xenconsoled. So I spent some time
> this morning and write up a draft for a xenstore based protocol. See
> below.
>
> Also there is an implication here: we put xenconsoled in a really
> critical path. If for some reason it doesn't work all guests are
> blocked. Do we really want to do this?

Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
now done.

Architecturally, this is a bad idea, and -1 from me.

First of all, you are making the assumption that xenconsoled and all
qemus are in the same domain.  This is not necessarily the case,
although I suppose that `xl console` does depend on xl and xenconsoled
being in the same domain.

At the moment, if xenconsoled crashes, the worst that happens is that
you cant interact with guest consoles, and the guests will notice that
their console rings aren't being drained.  This is a safe, albeit
suboptimal, situation which allows guests to continue running unimpeeded.

If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
As you identify, there is a security implication here, where a guest
which can crash xenconsoled can also DoS all other HVM domains in the
system.

From the scalability side, XenServer supports running 1000 windows VMs
on a single host (subject to sufficient RAM), and xenstored is a big
monlithic single point of failure.  We would like to avoid adding a second.


Managing logging is a hard problem.  However, managing logging is a
system level problem, and needs to fall to the distro package maintainer
to integrate suitably with the distro's expectations, and to use
logrotation/other as applicable.  Introducing custom, project-specific
configuration makes it harder to integrate sensibly, not easier.

I think the best solution here is to manage expectations, not to follow
the kneejerk reaction and hack up a solution.

If someone builds Xen from source and finds later that their disks are
full up with logs, then tough.  If someone installs Xen from a distro,
they should reasonably expect logging, and log management, to be
consistent with other packages from the same distro.

Perhaps providing a default logrotate config file would make things
easier for distro packagers (after all, we provide default
sysvinit/systemd configuration files), and perhaps a written statement
of what gets logged where under which circumstances would make it easier
for people who don't use logrotate, but I think that we shouldn't go any
further than that.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

On 03/06/16 18:38, Andrew Cooper wrote:
> On 01/06/16 15:00, Wei Liu wrote:
>> Hi all
>>
>> During the discussion of XSA-180 Ian came up with the idea that we
>> repurpose xenconsoled to handle logging. I've done some research (and
>> some coding as well!).
>>
>> In my reply to George the other day:
>>
>>> I just read the code of virtlogd and xenconsoled.
>>>
>>> I think xenconsoled is missing at least things.
>>>
>>> From a higher level:
>>>
>>> 1. Abstraction of rotating file.
>>> 2. Abstraction of client.
>>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>>
>> I've done #1 and port existing code to use that -- would be useful in
>> general.
>>
>> #2 is not too hard because xenconsoled already has concept of a domain.
>> I suspect some refactoring will be fine.
>>
>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
>> pretty sure we *don't* want that for xenconsoled. So I spent some time
>> this morning and write up a draft for a xenstore based protocol. See
>> below.
>>
>> Also there is an implication here: we put xenconsoled in a really
>> critical path. If for some reason it doesn't work all guests are
>> blocked. Do we really want to do this?
> 
> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
> now done.
> 
> Architecturally, this is a bad idea, and -1 from me.
> 
> First of all, you are making the assumption that xenconsoled and all
> qemus are in the same domain.  This is not necessarily the case,
> although I suppose that `xl console` does depend on xl and xenconsoled
> being in the same domain.
> 
> At the moment, if xenconsoled crashes, the worst that happens is that
> you cant interact with guest consoles, and the guests will notice that
> their console rings aren't being drained.  This is a safe, albeit
> suboptimal, situation which allows guests to continue running unimpeeded.
> 
> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
> As you identify, there is a security implication here, where a guest
> which can crash xenconsoled can also DoS all other HVM domains in the
> system.
> 
> From the scalability side, XenServer supports running 1000 windows VMs
> on a single host (subject to sufficient RAM), and xenstored is a big
> monlithic single point of failure.  We would like to avoid adding a second.

qemu stubdoms is a red herring; that's a separate codepath which will be
modified appropriately for its use case.   The potential for qemu to
block if the daemon dies, as well as scalability concerns, are good
points which should be considered; and we certainly need to allow other
ways of dealing with logging.

But...

> If someone builds Xen from source and finds later that their disks are
> full up with logs, then tough.

...I cannot express how wrong-headed I think this statement is; and...

> If someone installs Xen from a distro,
> they should reasonably expect logging, and log management, to be
> consistent with other packages from the same distro.

...the degree to which this is true depends on the degree to which
distributions are actually equipped to deal with these sorts of things.

I agree that if logging using existing logging systems were workable,
that would be a better solution.  Wei did actually spend a decent chunk
of time looking at other options, including logrotate and journald
before settling on using xenconsoled.

A lot of this brainstorming and discussion happened off-list because
XSA-180 was still embargoed, so I can understand why it looks like this
came out of nowhere.  It would probably be good for Wei to report here
what he found and why he decided to propose this solution instead.

FWIW, the libvirt project has exactly the same problem, and they did the
analog of what Wei is proposing -- they added a new daemon, virtlogd, to
handle all the console and debug log rotation in a fashion resistant to
DoSing.  Without reading their discussion, it's reasonable to assume
that using system logging was at least considered using system-level
logging before deciding to write their own code.

We already have a daemon to do logging of consoles; it just doesn't have
any of the logrotate features that are needed to make it robust against
DoS.  There's no sense in having log rotation code in two places, so
upgrading xenconsoled to do what virtlogd is doing makes more sense than
say, either writing our own, or stealing virtlogd.

> I think the best solution here is to manage expectations, not to follow
> the kneejerk reaction and hack up a solution.

Hopefully you can see now that this is not a knee-jerk solution. :-)
Thank you for making sure other options is properly discussed in public.

Tying off a few random technical threads:

What does xapi do at the moment?

Re the interface, it seems like having the option to log to {nothing,
file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
for instance could take the pipe and pipe it to virtlogd (and xapi could
do something similar if it wanted).

Regarding qemu grinding to a halt if xenconsoled dies: That may be
something worth considering; but at least on Linux you can set pipes as
O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
blocking.  We'd have to make sure that the error didn't cause any other
failures (ignoring it and letting the write drop is probably the best
thing), but that shouldn't be terribly difficult to do.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Andrew Cooper]

On 06/06/16 11:12, George Dunlap wrote:
> On 03/06/16 18:38, Andrew Cooper wrote:
>> On 01/06/16 15:00, Wei Liu wrote:
>>> Hi all
>>>
>>> During the discussion of XSA-180 Ian came up with the idea that we
>>> repurpose xenconsoled to handle logging. I've done some research (and
>>> some coding as well!).
>>>
>>> In my reply to George the other day:
>>>
>>>> I just read the code of virtlogd and xenconsoled.
>>>>
>>>> I think xenconsoled is missing at least things.
>>>>
>>>> From a higher level:
>>>>
>>>> 1. Abstraction of rotating file.
>>>> 2. Abstraction of client.
>>>> 3. IPC interface to libxl -- presumably we need to create a socket.
>>>>
>>> I've done #1 and port existing code to use that -- would be useful in
>>> general.
>>>
>>> #2 is not too hard because xenconsoled already has concept of a domain.
>>> I suspect some refactoring will be fine.
>>>
>>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
>>> pretty sure we *don't* want that for xenconsoled. So I spent some time
>>> this morning and write up a draft for a xenstore based protocol. See
>>> below.
>>>
>>> Also there is an implication here: we put xenconsoled in a really
>>> critical path. If for some reason it doesn't work all guests are
>>> blocked. Do we really want to do this?
>> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
>> now done.
>>
>> Architecturally, this is a bad idea, and -1 from me.
>>
>> First of all, you are making the assumption that xenconsoled and all
>> qemus are in the same domain.  This is not necessarily the case,
>> although I suppose that `xl console` does depend on xl and xenconsoled
>> being in the same domain.
>>
>> At the moment, if xenconsoled crashes, the worst that happens is that
>> you cant interact with guest consoles, and the guests will notice that
>> their console rings aren't being drained.  This is a safe, albeit
>> suboptimal, situation which allows guests to continue running unimpeeded.
>>
>> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
>> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
>> As you identify, there is a security implication here, where a guest
>> which can crash xenconsoled can also DoS all other HVM domains in the
>> system.
>>
>> From the scalability side, XenServer supports running 1000 windows VMs
>> on a single host (subject to sufficient RAM), and xenstored is a big
>> monlithic single point of failure.  We would like to avoid adding a second.
> qemu stubdoms is a red herring; that's a separate codepath which will be
> modified appropriately for its use case.

Most certainly not (and I didn't specify qemu stubdoms).  What happens
if someone wishes to run xenconsoled in a separate non-stub domain? 
What happens if someone has separate non-stub domains for running qemus?

All of these are currently viable options with XSM.

> The potential for qemu to
> block if the daemon dies, as well as scalability concerns, are good
> points which should be considered; and we certainly need to allow other
> ways of dealing with logging.
>
> But...
>
>> If someone builds Xen from source and finds later that their disks are
>> full up with logs, then tough.
> ...I cannot express how wrong-headed I think this statement is; and...

That is merely your opinion.  You know mine, and I stand by it.

>
>> If someone installs Xen from a distro,
>> they should reasonably expect logging, and log management, to be
>> consistent with other packages from the same distro.
> ...the degree to which this is true depends on the degree to which
> distributions are actually equipped to deal with these sorts of things.

A distro which isn't equipped to deal with these things is of no
practical use in the real world.

Any distro we care about supporting is equipped to deal with logs.

>
> I agree that if logging using existing logging systems were workable,
> that would be a better solution.  Wei did actually spend a decent chunk
> of time looking at other options, including logrotate and journald
> before settling on using xenconsoled.
>
> A lot of this brainstorming and discussion happened off-list because
> XSA-180 was still embargoed, so I can understand why it looks like this
> came out of nowhere.  It would probably be good for Wei to report here
> what he found and why he decided to propose this solution instead.

Please do.  Until there is an understanding of why the standard
mechanisms are not suitable, it is premature and naive to re-invent a wheel.

>
> FWIW, the libvirt project has exactly the same problem, and they did the
> analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> handle all the console and debug log rotation in a fashion resistant to
> DoSing.  Without reading their discussion, it's reasonable to assume
> that using system logging was at least considered using system-level
> logging before deciding to write their own code.

Just because a project followed this path, doesn't mean it is a sensible
thing to do.

>
> We already have a daemon to do logging of consoles; it just doesn't have
> any of the logrotate features that are needed to make it robust against
> DoS.  There's no sense in having log rotation code in two places, so
> upgrading xenconsoled to do what virtlogd is doing makes more sense than
> say, either writing our own, or stealing virtlogd.

We have a daemon for connecting local pty's to guest consoles, which has
a disabled-by-default option to log consoles to files.

>
>> I think the best solution here is to manage expectations, not to follow
>> the kneejerk reaction and hack up a solution.
> Hopefully you can see now that this is not a knee-jerk solution. :-)
> Thank you for making sure other options is properly discussed in public.
>
> Tying off a few random technical threads:
>
> What does xapi do at the moment?

Plain syslog and logrotate.  No frills.

> Re the interface, it seems like having the option to log to {nothing,
> file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
> for instance could take the pipe and pipe it to virtlogd (and xapi could
> do something similar if it wanted).

My main concern is that this is re-inventing a wheel which noone will use.

>
> Regarding qemu grinding to a halt if xenconsoled dies: That may be
> something worth considering; but at least on Linux you can set pipes as
> O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
> blocking.  We'd have to make sure that the error didn't cause any other
> failures (ignoring it and letting the write drop is probably the best
> thing), but that shouldn't be terribly difficult to do.

stderr really shouldn't be in a position where it gets -EAGAINs in a
common case.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Mon, Jun 06, 2016 at 02:03:45PM +0100, Andrew Cooper wrote:
> On 06/06/16 11:12, George Dunlap wrote:
> > On 03/06/16 18:38, Andrew Cooper wrote:
> >> On 01/06/16 15:00, Wei Liu wrote:
> >>> Hi all
> >>>
> >>> During the discussion of XSA-180 Ian came up with the idea that we
> >>> repurpose xenconsoled to handle logging. I've done some research (and
> >>> some coding as well!).
> >>>
> >>> In my reply to George the other day:
> >>>
> >>>> I just read the code of virtlogd and xenconsoled.
> >>>>
> >>>> I think xenconsoled is missing at least things.
> >>>>
> >>>> From a higher level:
> >>>>
> >>>> 1. Abstraction of rotating file.
> >>>> 2. Abstraction of client.
> >>>> 3. IPC interface to libxl -- presumably we need to create a socket.
> >>>>
> >>> I've done #1 and port existing code to use that -- would be useful in
> >>> general.
> >>>
> >>> #2 is not too hard because xenconsoled already has concept of a domain.
> >>> I suspect some refactoring will be fine.
> >>>
> >>> #3 requires a bit more thinking. Virtlogd has an RPC interface -- I'm
> >>> pretty sure we *don't* want that for xenconsoled. So I spent some time
> >>> this morning and write up a draft for a xenstore based protocol. See
> >>> below.
> >>>
> >>> Also there is an implication here: we put xenconsoled in a really
> >>> critical path. If for some reason it doesn't work all guests are
> >>> blocked. Do we really want to do this?
> >> Sorry to be late to this thread, but I think all my Xen 4.7 tasks are
> >> now done.
> >>
> >> Architecturally, this is a bad idea, and -1 from me.
> >>
> >> First of all, you are making the assumption that xenconsoled and all
> >> qemus are in the same domain.  This is not necessarily the case,
> >> although I suppose that `xl console` does depend on xl and xenconsoled
> >> being in the same domain.
> >>
> >> At the moment, if xenconsoled crashes, the worst that happens is that
> >> you cant interact with guest consoles, and the guests will notice that
> >> their console rings aren't being drained.  This is a safe, albeit
> >> suboptimal, situation which allows guests to continue running unimpeeded.
> >>
> >> If qemu is connected via xenconsoled, and xenconsoled crashes, qemu will
> >> block.  Once qemu blocks, the VM is for all intents and purposes, dead. 
> >> As you identify, there is a security implication here, where a guest
> >> which can crash xenconsoled can also DoS all other HVM domains in the
> >> system.
> >>
> >> From the scalability side, XenServer supports running 1000 windows VMs
> >> on a single host (subject to sufficient RAM), and xenstored is a big
> >> monlithic single point of failure.  We would like to avoid adding a second.
> > qemu stubdoms is a red herring; that's a separate codepath which will be
> > modified appropriately for its use case.
> 
> Most certainly not (and I didn't specify qemu stubdoms).  What happens
> if someone wishes to run xenconsoled in a separate non-stub domain? 
> What happens if someone has separate non-stub domains for running qemus?
> 
> All of these are currently viable options with XSM.
> 

The scalability issue is true. TBH I'm also a bit skeptical to the
scalability issue. But in the end I think it wouldn't be any different
if you want to use syslog or any other daemon. The issue is going to be
the there if you opt to use any daemon to handle log.

There is a difference between requiring QEMU and the logging daemon to
live in the same domain v.s.  requiring the toolstack that sets up
logging for QEMU knows how to do it. That is, the whatever toolstack to
spawn QEMU in the other domain will know how to handle logging anyway.
Running QEMU and xenconsoled in the same domain won't be a hard
requirement. It is fair to expect xenconsoled and the toolstack live in
the same domain (TOOLSTACK_DOMAIN), so it would be fair to expect
xenconsoled to be available when libxl tries to start QEMU.

I wouldn't expect xenconsoled to crash easily.

> > The potential for qemu to
> > block if the daemon dies, as well as scalability concerns, are good
> > points which should be considered; and we certainly need to allow other
> > ways of dealing with logging.
> >
> > But...
> >
> >> If someone builds Xen from source and finds later that their disks are
> >> full up with logs, then tough.
> > ...I cannot express how wrong-headed I think this statement is; and...
> 
> That is merely your opinion.  You know mine, and I stand by it.
> 

I don't think these two views are fundamentally in conflict with each
other. To provide a system level solution, we need to provide something
to be integrated with the system. There is none so far.

> >
> >> If someone installs Xen from a distro,
> >> they should reasonably expect logging, and log management, to be
> >> consistent with other packages from the same distro.
> > ...the degree to which this is true depends on the degree to which
> > distributions are actually equipped to deal with these sorts of things.
> 
> A distro which isn't equipped to deal with these things is of no
> practical use in the real world.
> 
> Any distro we care about supporting is equipped to deal with logs.
> 
> >
> > I agree that if logging using existing logging systems were workable,
> > that would be a better solution.  Wei did actually spend a decent chunk
> > of time looking at other options, including logrotate and journald
> > before settling on using xenconsoled.
> >
> > A lot of this brainstorming and discussion happened off-list because
> > XSA-180 was still embargoed, so I can understand why it looks like this
> > came out of nowhere.  It would probably be good for Wei to report here
> > what he found and why he decided to propose this solution instead.
> 
> Please do.  Until there is an understanding of why the standard
> mechanisms are not suitable, it is premature and naive to re-invent a wheel.
> 

With syslog and logrotate you will still end up filling up your disk.
Logrotate can't actively rotate log files.

You can't just tap syslog to QEMU at the moment unless you use the
script I sent to XSA-180 security@ discussion. That's still a hacked up
solution.

I actually don't mind having syslog deal with those, but we need to
provide some not-so-hacked-up way for doing it.

> >
> > FWIW, the libvirt project has exactly the same problem, and they did the
> > analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> > handle all the console and debug log rotation in a fashion resistant to
> > DoSing.  Without reading their discussion, it's reasonable to assume
> > that using system logging was at least considered using system-level
> > logging before deciding to write their own code.
> 
> Just because a project followed this path, doesn't mean it is a sensible
> thing to do.
> 
> >
> > We already have a daemon to do logging of consoles; it just doesn't have
> > any of the logrotate features that are needed to make it robust against
> > DoS.  There's no sense in having log rotation code in two places, so
> > upgrading xenconsoled to do what virtlogd is doing makes more sense than
> > say, either writing our own, or stealing virtlogd.
> 
> We have a daemon for connecting local pty's to guest consoles, which has
> a disabled-by-default option to log consoles to files.
> 
> >
> >> I think the best solution here is to manage expectations, not to follow
> >> the kneejerk reaction and hack up a solution.
> > Hopefully you can see now that this is not a knee-jerk solution. :-)
> > Thank you for making sure other options is properly discussed in public.
> >
> > Tying off a few random technical threads:
> >
> > What does xapi do at the moment?
> 
> Plain syslog and logrotate.  No frills.

How does XAPI redirect QEMU's stderr to syslog?

If it isn't too complex I would like to add that to libxl regardless of
whether we want to use xenconsoled or not.

> 
> > Re the interface, it seems like having the option to log to {nothing,
> > file, pipe, xenconsoled} should cover the reasonble usecases: libvirt
> > for instance could take the pipe and pipe it to virtlogd (and xapi could
> > do something similar if it wanted).
> 
> My main concern is that this is re-inventing a wheel which noone will use.
> 

It's going to be used by a default installation of open source Xen.

But, I would like to devise a mechanism that system administrator can
opt out.

> >
> > Regarding qemu grinding to a halt if xenconsoled dies: That may be
> > something worth considering; but at least on Linux you can set pipes as
> > O_NONBLOCK and you'll get EAGAIN when writing to a full pipe instead of
> > blocking.  We'd have to make sure that the error didn't cause any other
> > failures (ignoring it and letting the write drop is probably the best
> > thing), but that shouldn't be terribly difficult to do.
> 
> stderr really shouldn't be in a position where it gets -EAGAINs in a
> common case.
> 

I agree.

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Fri, Jun 03, 2016 at 05:57:29PM +0100, Ian Jackson wrote:
> Wei Liu writes ("XSA-180 follow-up: repurpose xenconsoled for logging"):
> > XXX DRAFT DRAFT DRAFT XXX
> 
> Thanks!
> 
> > Per domain logging via xenconsoled
> > ==================================
> > 
> > As of Xen release XXX, xenconsoled is repurposed to handle logging for
> > QEMU. Libxenlight will arrange xenconsoled to create and handle the
> > log file. It's possible to expose API(s) so that the user of
> > libxenlight can leverage such ability, but it is not currently done.
> > 
> > Xenstore path and nodes
> > -----------------------
> > 
> > Libxenlight communicates with xenconsoled via a simple xenstore based
> > protocol.  All information for a specific domain is stored under
> > /libxl/$DOMID/logging. Each log file has its own unique id ($LOGFILEID).
> 
> Are these IDs short strings or what ?
> 

Numeric ID.

> > Several xenstore nodes are needed (placed under logging/$LOGFILEID).
> > 
> >   pipe: the absolute path of the logging pipe
> >   file: the absolute path of the file to write to
> >   limit: the maximum length of the file before it gets rotated
> >   copies: the number of copies to keep
> >   state: xenconsoled writes "ready" to this node to indicate readiness
> 
> The use of named pipes for rendezvous is a little unusual, but I think
> given that xenconsoled is primarily driven by xenstore it is probably
> appropriate.
> 
> > Xenconsoled will sanitise both pipe and file fields. Pipe has to be
> > placed under XEN_RUN_DIR. File has to be placed under /var/log/xen
> > (XXX doesn't seem to be configurable at the moment, should introduce
> > XEN_LOG_DIR?).
> 
> Yes.
> 
> NB the pipe has to be mode 600.  Also, opening a fifo has to be done
> with care.  For example, the following dance is needed get a
> write-only fd without risking blocking:
> 
>    fdrw = open("/var/run/xen/DOMAIN.qemu.fifo", O_RDWR);
>    fdwo = open("/var/run/xen/DOMAIN.qemu.fifo", O_WRONLY);
>    close(fdwr);
> 

Heh, good trick.

> But I think qemu should be provided with a O_RDWR fd.  This is because
> otherwise, if xenconsoled crashes or is restarted or something, qemu
> would get a SIGPIPE (or EPIPE) from writes to its stderr.
> 
> It is better for a qemu in this situation to block waiting for a new
> xenconsoled to appear.
> 

So there are two issues:
1. xenconsoled crashes
2. xenconsoled gets bogged down, not able to process things fast enough

Either issue will render the client (QEMU for now) blocked.

I think we need to think this through and set the expectation straight
here.

> > 1. Libxenlight:
> >   1. Generates a unique log file id $LOGFILEID
> >   2. Creates a pipe $PIPE
> >   3. Writes parameter to xenstore
> >   4. Wait for readiness indication
> > 2. Xenconsoled
> >   1. Watch global logging and per domain logging xenstore paths
> >   2. Gets notified, read parameters from xenstore
> >   3. Sanitise parameters
> >   4. Create log files
> >   5. Connect to the pipe provided
> >   6. Write "ready" to xenstore state node
> 
> Is it actually necessary to synchronise ?  If xenconsoled is slow, the
> approach above will simply block qemu's writes once the pipe buffer is
> full, until xenconsoled catches up.
> 
> If xenconsoled is missing the lack of synchronisation will result in
> qemu blocking, perhaps tripping the domain startup qemu readiness
> timeout.
> 

I will think about this later. This is related to the issues I mentioned
above.

> > 3. Libxenlight:
> >   1. Detects ready state from xenconsoled
> >   2. Open the pipe and return relevant handles to user
> > 
> > In case of xenconsoled failure, libxenlight will time out and bail.
> 
> Would it be possible for a user to connect to xenconsoled and get a
> copy of the qemu output with a suitable `xl console' rune ?
> 

An xl console extension to attach a particular logging stream that is
handled by xenconsoled is doable, I think. It's yet another fd anyway.

But for the time being I don't think this is essential because user can
use existing system utilities (tail --follow=name for example)  to
achieve the same functionality.

> > Clean up
> > --------
> > 
> > When doing per domain logging, libxenlight will remove all domain
> > specific xenstore paths when a guest is gone. Xenconsoled use that to
> > do clean up.
> > 
> > Libxenlight is responsible for deleting the pipe.
> 
> The logfiles will presumably remain.
> 
> > Global logging
> > --------------
> > 
> > Since we don't plan to provide new APIs now, we don't support global
> > logging because that would require us to provide a cleanup API that
> > libxenlight users can call.
> 
> I don't understand what you mean by `global logging'.
> 

Right, that's a very confusing term. Let me explain.

So with all the machinery in place, we can handle not only guest logs
but logs from all sorts of xen components. Say, xenstored wants to write
to a log file, no problem, just call libxl_create_logfile and you now
get a fd you can write to, and then it will call libxl_close_logfile to
close the log file.  But I don't want to go into exporting this facility
to libxl users yet.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Doug Goldstein]

[-- Attachment #1.1.1: Type: text/plain, Size: 1693 bytes --]

On 6/6/16 5:12 AM, George Dunlap wrote:
> On 03/06/16 18:38, Andrew Cooper wrote:
>> On 01/06/16 15:00, Wei Liu wrote:
>>> Hi all
>>>

<snip>

> FWIW, the libvirt project has exactly the same problem, and they did the
> analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> handle all the console and debug log rotation in a fashion resistant to
> DoSing.  Without reading their discussion, it's reasonable to assume
> that using system logging was at least considered using system-level
> logging before deciding to write their own code.

If I recall they use RPCs and the logs are generated as a best effort to
not block QEMU.

> 
> We already have a daemon to do logging of consoles; it just doesn't have
> any of the logrotate features that are needed to make it robust against
> DoS.  There's no sense in having log rotation code in two places, so
> upgrading xenconsoled to do what virtlogd is doing makes more sense than
> say, either writing our own, or stealing virtlogd.

What if we made xl / libxl really good at the limited scope of things it
should be good at and left the other bits to others. At this point it
seems like yet another feature that xl / libxl is gaining that matches
what libvirt does. Maybe an approach is something you appear to suggest
and just point people to virtlogd and ask the libvirt guys if they would
make it a separate package. Honestly it seems like xl could slim down
from a feature set perspective and focus on improving libxl / libvirt
interaction. That's something that the Xen community has been interested
in to better support OpenStack anyway.

Just my 2 cents.

-- 
Doug Goldstein


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 959 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On 06/06/16 16:48, Wei Liu wrote:
>> A distro which isn't equipped to deal with these things is of no
>> practical use in the real world.
>>
>> Any distro we care about supporting is equipped to deal with logs.
>>
>>>
>>> I agree that if logging using existing logging systems were workable,
>>> that would be a better solution.  Wei did actually spend a decent chunk
>>> of time looking at other options, including logrotate and journald
>>> before settling on using xenconsoled.
>>>
>>> A lot of this brainstorming and discussion happened off-list because
>>> XSA-180 was still embargoed, so I can understand why it looks like this
>>> came out of nowhere.  It would probably be good for Wei to report here
>>> what he found and why he decided to propose this solution instead.
>>
>> Please do.  Until there is an understanding of why the standard
>> mechanisms are not suitable, it is premature and naive to re-invent a wheel.
>>
> 
> With syslog and logrotate you will still end up filling up your disk.
> Logrotate can't actively rotate log files.

FWIW CentOS 6 and 7 (which use rsyslogd and systemd-journald
respectively) seem to have rate-limiting stuff enabled by default; the
attached program causes a lot of CPU utilization, but no disk resource
exhaustion.

> You can't just tap syslog to QEMU at the moment unless you use the
> script I sent to XSA-180 security@ discussion. That's still a hacked up
> solution.
> 
> I actually don't mind having syslog deal with those, but we need to
> provide some not-so-hacked-up way for doing it.

On Linux, it looks like you can create a socket and "connect" to /dev/log.

 -George


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: syslog-flood.c --]
[-- Type: text/x-csrc; name="syslog-flood.c", Size: 372 bytes --]

#include <syslog.h>
#include <time.h>

int main(int argc, char * argv[]) {
    int rc;
    
    struct timespec tv = { .tv_nsec = 5000 };
    
    openlog("DoS Attempt", 0, LOG_DAEMON);

    while(!(rc = nanosleep(&tv, NULL))) {
        syslog(LOG_ERR, "This is a nasty attempt to DoS syslog\n");
    }

    if(rc) {
        perror("nanosleep");
    }
}

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Tue, Jun 07, 2016 at 10:57:14AM +0100, George Dunlap wrote:
> On 06/06/16 16:48, Wei Liu wrote:
> >> A distro which isn't equipped to deal with these things is of no
> >> practical use in the real world.
> >>
> >> Any distro we care about supporting is equipped to deal with logs.
> >>
> >>>
> >>> I agree that if logging using existing logging systems were workable,
> >>> that would be a better solution.  Wei did actually spend a decent chunk
> >>> of time looking at other options, including logrotate and journald
> >>> before settling on using xenconsoled.
> >>>
> >>> A lot of this brainstorming and discussion happened off-list because
> >>> XSA-180 was still embargoed, so I can understand why it looks like this
> >>> came out of nowhere.  It would probably be good for Wei to report here
> >>> what he found and why he decided to propose this solution instead.
> >>
> >> Please do.  Until there is an understanding of why the standard
> >> mechanisms are not suitable, it is premature and naive to re-invent a wheel.
> >>
> > 
> > With syslog and logrotate you will still end up filling up your disk.
> > Logrotate can't actively rotate log files.
> 
> FWIW CentOS 6 and 7 (which use rsyslogd and systemd-journald
> respectively) seem to have rate-limiting stuff enabled by default; the
> attached program causes a lot of CPU utilization, but no disk resource
> exhaustion.
> 

Not rate-limited on Debian Jessie installation.  We can make
recommendation that system administrators should rate-limit syslog.

But CPU utilisation is also bad, just a different kind of "bad" from
disk exhaustion.

> > You can't just tap syslog to QEMU at the moment unless you use the
> > script I sent to XSA-180 security@ discussion. That's still a hacked up
> > solution.
> > 
> > I actually don't mind having syslog deal with those, but we need to
> > provide some not-so-hacked-up way for doing it.
> 
> On Linux, it looks like you can create a socket and "connect" to /dev/log.
> 

That's the standard interface to syslog. I believe that's what openlog()
and syslog() use.  We can open a socket and write to that, but you won't
get what you want without providing the correct formated message.

Try with:
  sudo socat /dev/log STDIN

Wei.

>  -George
> 

> #include <syslog.h>
> #include <time.h>
> 
> int main(int argc, char * argv[]) {
>     int rc;
>     
>     struct timespec tv = { .tv_nsec = 5000 };
>     
>     openlog("DoS Attempt", 0, LOG_DAEMON);
> 
>     while(!(rc = nanosleep(&tv, NULL))) {
>         syslog(LOG_ERR, "This is a nasty attempt to DoS syslog\n");
>     }
> 
>     if(rc) {
>         perror("nanosleep");
>     }
> }


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

On Mon, Jun 06, 2016 at 03:47:37PM -0500, Doug Goldstein wrote:
> On 6/6/16 5:12 AM, George Dunlap wrote:
> > On 03/06/16 18:38, Andrew Cooper wrote:
> >> On 01/06/16 15:00, Wei Liu wrote:
> >>> Hi all
> >>>
> 
> <snip>
> 
> > FWIW, the libvirt project has exactly the same problem, and they did the
> > analog of what Wei is proposing -- they added a new daemon, virtlogd, to
> > handle all the console and debug log rotation in a fashion resistant to
> > DoSing.  Without reading their discussion, it's reasonable to assume
> > that using system logging was at least considered using system-level
> > logging before deciding to write their own code.
> 
> If I recall they use RPCs and the logs are generated as a best effort to
> not block QEMU.
> 

Does that mean it's more or less equivalent to O_NONBLOCK?

Is that configurable?  We might actually want to block in some cases.

> > 
> > We already have a daemon to do logging of consoles; it just doesn't have
> > any of the logrotate features that are needed to make it robust against
> > DoS.  There's no sense in having log rotation code in two places, so
> > upgrading xenconsoled to do what virtlogd is doing makes more sense than
> > say, either writing our own, or stealing virtlogd.
> 
> What if we made xl / libxl really good at the limited scope of things it
> should be good at and left the other bits to others. At this point it
> seems like yet another feature that xl / libxl is gaining that matches
> what libvirt does. Maybe an approach is something you appear to suggest
> and just point people to virtlogd and ask the libvirt guys if they would
> make it a separate package. Honestly it seems like xl could slim down
> from a feature set perspective and focus on improving libxl / libvirt
> interaction. That's something that the Xen community has been interested
> in to better support OpenStack anyway.
> 

To clarify: xenconsoled is not part of xl / libxl. I think it you're
talking about xen toolstack in general.

I think we can ask libvirt maintainers: 1. if it is possible to make
virtlogd a separate package, 2. if they can maintain a stable interface.

Then we can think about how to make sensible suggestions and provide a
way for sysamdins to configure that.

No matter what solution we end up with, the work to integrate that with
xl / libxl is unavoidable.

Wei.

> Just my 2 cents.
> 
> -- 
> Doug Goldstein
> 




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Wei Liu]

Here is what we have gathered so far:

1. Virtlogd is not the right answer to XSA-180 because it is inherently
   designed to work within libvirt.
2. Syslog is not suitable because it doesn't provide a fd for QEMU to
   write to.
3. Logrotate is not suitable because it only rotates periodically.
4. Syslog + logrotate combo is not suitable because (see above).

We can, however, just make recommendation that system administrators can
easily set up and call it a day. There are suggestions that we can
recommend conserver or sympathy, but I haven't seen a concrete viable
proposal yet. What I hope is that we can have a document in tree in the
end.

Another way is to invent our own "virtlogd" -- it could be a new daemon,
it could be xenconsoled. The major concern is that we're adding a
critical component to the system and it may not scale well. We can make
a compromise by using non-blocking fd to make the new component less
critical and doesn't hinder scalability.

Another way is to alter libxl API and ask the application to pass in a
fd for logging. The major concern is that this is not suitable in the
context of a security issue.

My ultimate goal is to remove the custom patch we have in QEMU tree so
that we don't create a problem for distro maintainers.  So I'm fine with
any solution really.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Juergen Gross]

On 21/06/16 16:46, Wei Liu wrote:
> Here is what we have gathered so far:
> 
> 1. Virtlogd is not the right answer to XSA-180 because it is inherently
>    designed to work within libvirt.
> 2. Syslog is not suitable because it doesn't provide a fd for QEMU to
>    write to.
> 3. Logrotate is not suitable because it only rotates periodically.

Huh? Excerpt from `man logrotate`:

Each log file may be handled daily, weekly, monthly, or when it grows
too large.

So why is logrotate not suitable?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Ian Jackson]

Wei Liu writes ("Re: XSA-180 follow-up: repurpose xenconsoled for logging"):
> Here is what we have gathered so far:
...
> We can, however, just make recommendation that system administrators can
> easily set up and call it a day. There are suggestions that we can
> recommend conserver or sympathy, but I haven't seen a concrete viable
> proposal yet. What I hope is that we can have a document in tree in the
> end.

sympathy would need some extra engineering to become suitable.  It's
also not widely adopted.  (Not even in Debian, yet.  Sorry about that,
but in my defence it's not my project...)

> Another way is to invent our own "virtlogd" -- it could be a new daemon,
> it could be xenconsoled. The major concern is that we're adding a
> critical component to the system and it may not scale well. We can make
> a compromise by using non-blocking fd to make the new component less
> critical and doesn't hinder scalability.

I think this is probably the best answer.  We already have most of
this in the form of xenconsoled.

> Another way is to alter libxl API and ask the application to pass in a
> fd for logging. The major concern is that this is not suitable in the
> context of a security issue.

Any solution needs to work for xl as well as other users of libxl.  So
this is not a description of a solution option; rather it is a
proposal to move the functionality/glue/problem/whatever out of libxl
into xl.

IMO it would be best to come up with a solution that is suitable for
all users of libxl, and put it in libxl.  If this is not possible then
whatever we implement could go in libxlu.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Ian Jackson]

Juergen Gross writes ("Re: [Xen-devel] XSA-180 follow-up: repurpose xenconsoled for logging"):
> Huh? Excerpt from `man logrotate`:
> 
> Each log file may be handled daily, weekly, monthly, or when it grows
> too large.
> 
> So why is logrotate not suitable?

logrotate runs "off-path", ie it checks the logfile size, when it is
run.  It's not designed to be run frequently.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

On 21/06/16 16:11, Ian Jackson wrote:
> Wei Liu writes ("Re: XSA-180 follow-up: repurpose xenconsoled for logging"):
>> Here is what we have gathered so far:
> ...
>> We can, however, just make recommendation that system administrators can
>> easily set up and call it a day. There are suggestions that we can
>> recommend conserver or sympathy, but I haven't seen a concrete viable
>> proposal yet. What I hope is that we can have a document in tree in the
>> end.
> 
> sympathy would need some extra engineering to become suitable.  It's
> also not widely adopted.  (Not even in Debian, yet.  Sorry about that,
> but in my defence it's not my project...)
> 
>> Another way is to invent our own "virtlogd" -- it could be a new daemon,
>> it could be xenconsoled. The major concern is that we're adding a
>> critical component to the system and it may not scale well. We can make
>> a compromise by using non-blocking fd to make the new component less
>> critical and doesn't hinder scalability.
> 
> I think this is probably the best answer.  We already have most of
> this in the form of xenconsoled.
> 
>> Another way is to alter libxl API and ask the application to pass in a
>> fd for logging. The major concern is that this is not suitable in the
>> context of a security issue.
> 
> Any solution needs to work for xl as well as other users of libxl.  So
> this is not a description of a solution option; rather it is a
> proposal to move the functionality/glue/problem/whatever out of libxl
> into xl.

...or libvirt, or xapi (should it ever be ported to libxl).

I think that having the option to pass an fd in would be useful and will
probably be wanted at some point; I think libvirt for instance should
probably be modified to use such an interface once it's available to
connect qemu to virtlogd.

> IMO it would be best to come up with a solution that is suitable for
> all users of libxl, and put it in libxl.  If this is not possible then
> whatever we implement could go in libxlu.

I wouldn't object to the functionality being in libxl, but it seems to
me like libxlu might be a better fit.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Ian Jackson]

George Dunlap writes ("Re: XSA-180 follow-up: repurpose xenconsoled for logging"):
> I think that having the option to pass an fd in would be useful and will
> probably be wanted at some point; I think libvirt for instance should
> probably be modified to use such an interface once it's available to
> connect qemu to virtlogd.

Quite possibly, yes.

> On 21/06/16 16:11, Ian Jackson wrote:
> > IMO it would be best to come up with a solution that is suitable for
> > all users of libxl, and put it in libxl.  If this is not possible then
> > whatever we implement could go in libxlu.
> 
> I wouldn't object to the functionality being in libxl, but it seems to
> me like libxlu might be a better fit.

My point was that I think we should decide first what our proposed
solution will look like, and _then_ whether it should be in libxlu or
libxl.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[George Dunlap]

On 21/06/16 17:04, Ian Jackson wrote:
> My point was that I think we should decide first what our proposed
> solution will look like, and _then_ whether it should be in libxlu or
> libxl.

Indeed.

I agree that direct consumers of xl are an important "market segment"
that we need to support.  I'd support adapting xenconsoled.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: XSA-180 follow-up: repurpose xenconsoled for logging

[Jim Fehlig]

On 06/21/2016 09:53 AM, George Dunlap wrote:
> On 21/06/16 16:11, Ian Jackson wrote:
>> Wei Liu writes ("Re: XSA-180 follow-up: repurpose xenconsoled for logging"):
>>> Here is what we have gathered so far:
>> ...
>>> We can, however, just make recommendation that system administrators can
>>> easily set up and call it a day. There are suggestions that we can
>>> recommend conserver or sympathy, but I haven't seen a concrete viable
>>> proposal yet. What I hope is that we can have a document in tree in the
>>> end.
>> sympathy would need some extra engineering to become suitable.  It's
>> also not widely adopted.  (Not even in Debian, yet.  Sorry about that,
>> but in my defence it's not my project...)
>>
>>> Another way is to invent our own "virtlogd" -- it could be a new daemon,
>>> it could be xenconsoled. The major concern is that we're adding a
>>> critical component to the system and it may not scale well. We can make
>>> a compromise by using non-blocking fd to make the new component less
>>> critical and doesn't hinder scalability.
>> I think this is probably the best answer.  We already have most of
>> this in the form of xenconsoled.
>>
>>> Another way is to alter libxl API and ask the application to pass in a
>>> fd for logging. The major concern is that this is not suitable in the
>>> context of a security issue.
>> Any solution needs to work for xl as well as other users of libxl.  So
>> this is not a description of a solution option; rather it is a
>> proposal to move the functionality/glue/problem/whatever out of libxl
>> into xl.
> ...or libvirt, or xapi (should it ever be ported to libxl).
>
> I think that having the option to pass an fd in would be useful and will
> probably be wanted at some point; I think libvirt for instance should
> probably be modified to use such an interface once it's available to
> connect qemu to virtlogd.

It would be easy enough to do since the qemu driver is already doing this.

Regards,
Jim


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel