All of lore.kernel.org
 help / color / mirror / Atom feed
* SOL Logging
@ 2018-08-24 21:02 Tanous, Ed
  2018-08-25  0:32 ` Sai Dasari
                   ` (3 more replies)
  0 siblings, 4 replies; 15+ messages in thread
From: Tanous, Ed @ 2018-08-24 21:02 UTC (permalink / raw)
  To: OpenBMC Maillist

[-- Attachment #1: Type: text/plain, Size: 1612 bytes --]

I would like to start a discussion around the feature of SOL logging, and how to best implement it.

There is a proposal here https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/ that proposes dumping the SOL console text buffer from Javascript into a file, and presenting it to the user as a download.  On its face, it seems to work as designed.  This has some pretty severe limitations, in that you can only dump the console log from your session, and your session is essentially destroyed when you switch pages in the webui, or hit refresh.  I think this is overly limiting, and of the production BMC stacks that I know of, none of them implement SOL logging in this way.

Instead, other BMCs implement it as a circular buffer inside the BMC itself, which allows SOL to log data all the time, not just while the user is logged in.  I think architecting it this way would be much more useful for admins consuming OpenBMC, and make us more useful as a solution.  Doing it this way also can make the SOL log available to IPMI and Redfish, as well as a file download, which improves the usability quite a bit.  Most places I see the SOL logging requested is for audit type purposes, which the javascript based version can't do.

I'm looking for feedback from the community here.  As written, I don't think the javascript console export is a useful feature given its limitations.  Do others agree?  Am I off base?  Are there use models where having a log of only the current session is useful?  Should we document the limitations so we can respond to bugs in the future?

Thanks,

-Ed

[-- Attachment #2: Type: text/html, Size: 3837 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-24 21:02 SOL Logging Tanous, Ed
@ 2018-08-25  0:32 ` Sai Dasari
  2018-08-27 12:48 ` Andrew Geissler
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 15+ messages in thread
From: Sai Dasari @ 2018-08-25  0:32 UTC (permalink / raw)
  To: Tanous, Ed, OpenBMC Maillist

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

>Instead, other BMCs implement it as a circular buffer inside the BMC itself, which allows SOL to log data all the time, not just while the user is logged in.  I think architecting it this way would be much more useful for >admins consuming OpenBMC, and make us more useful as a solution.  Doing it this way also can make the SOL log available to IPMI and Redfish, as well as a file download, which improves the usability quite a bit.  Most >places I see the SOL logging requested is for audit type purposes, which the javascript based version can’t do.

+1 for this design proposal of having a circular buffer inside BMC that captures the Host’s console all the time. Couple of thoughts on this: 1) make sure these console logs are available to privileged user only (e.g. root) 2) consider the case of preserving the logs (or periodically offload from BMC) to not lose them due to BMC reboots.

>I’m looking for feedback from the community here.  As written, I don’t think the javascript console export is a useful feature given its limitations.  Do others agree?  Am I off base?  Are there use models where having a log >of only the current session is useful?  Should we document the limitations so we can respond to bugs in the future?

One possible usecase (not certain though) this feature of keeping log of only current session is for security reason where the current user’s console session log is available to only that user.


[-- Attachment #2: Type: text/html, Size: 3500 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-24 21:02 SOL Logging Tanous, Ed
  2018-08-25  0:32 ` Sai Dasari
@ 2018-08-27 12:48 ` Andrew Geissler
  2018-08-27 14:21   ` Rebecca Broekhuis
  2018-08-27 17:00 ` Jeremy Kerr
  2018-08-29 11:37 ` Alexander Amelkin
  3 siblings, 1 reply; 15+ messages in thread
From: Andrew Geissler @ 2018-08-27 12:48 UTC (permalink / raw)
  To: ed.tanous; +Cc: OpenBMC Maillist

On Fri, Aug 24, 2018 at 4:03 PM Tanous, Ed <ed.tanous@intel.com> wrote:
>
> I would like to start a discussion around the feature of SOL logging, and how to best implement it.
>
>
>
> There is a proposal here https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/ that proposes dumping the SOL console text buffer from Javascript into a file, and presenting it to the user as a download.  On its face, it seems to work as designed.  This has some pretty severe limitations, in that you can only dump the console log from your session, and your session is essentially destroyed when you switch pages in the webui, or hit refresh.  I think this is overly limiting, and of the production BMC stacks that I know of, none of them implement SOL logging in this way.
>
>
>
> Instead, other BMCs implement it as a circular buffer inside the BMC itself, which allows SOL to log data all the time, not just while the user is logged in.  I think architecting it this way would be much more useful for admins consuming OpenBMC, and make us more useful as a solution.  Doing it this way also can make the SOL log available to IPMI and Redfish, as well as a file download, which improves the usability quite a bit.  Most places I see the SOL logging requested is for audit type purposes, which the javascript based version can’t do.

We use https://github.com/openbmc/obmc-console for this currently.  It
creates a file at /var/log/obmc-console.log and even has a
customizable size for the file to wrap at. I guess the advantage of
having it in the GUI is the user could ensure a full dump of the
console (vs. just grabbing the existing obmc-console.log which may
have wrapped). Being able to view the SOL output has it happens in the
GUI, then having an option to export it sounds kind of useful.  Also
be nice to be able to retrieve and view the existing obmc-console.log
from the system in case you missed it in the GUI.  Maybe that latter
part is enough though, reduces GUI complexity and testing. Something
good to run by our usability/design team Gunnar/Rebecca?

>
>
> I’m looking for feedback from the community here.  As written, I don’t think the javascript console export is a useful feature given its limitations.  Do others agree?  Am I off base?  Are there use models where having a log of only the current session is useful?  Should we document the limitations so we can respond to bugs in the future?
>
>
>
> Thanks,
>
>
>
> -Ed

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-27 12:48 ` Andrew Geissler
@ 2018-08-27 14:21   ` Rebecca Broekhuis
  0 siblings, 0 replies; 15+ messages in thread
From: Rebecca Broekhuis @ 2018-08-27 14:21 UTC (permalink / raw)
  To: geissonator; +Cc: ed.tanous, openbmc

[-- Attachment #1: Type: text/plain, Size: 3839 bytes --]

I agree with what you're saying about the javascript implementation of this
being limited. Besides the session being destroyed upon leaving the page, I
am also seeing that we are losing console messages in the front end when
the websocket closes for a moment (which it is consistently doing during
power on). We lose those messages sent during the time it takes to re-open
the connection. They aren't displayed in the terminal emulator and
therefore aren't exported to the text file when the user clicks 'export'. I
could see that as another reason to implement the ability to download the
existing obmc-console.log, since the pure javascript way of exporting to a
text file could be an incomplete log in some cases.

There might be a another way to address missing messages as a result of the
websocket closing, but from my current understanding it seems like another
reason having the javascript export exclusively would be an unreliable
option.

I can also see how having both downloads available could be confusing to
the user -- I'd be happy to run this by the design team to see if they have
input.

-- Becca

On Mon, Aug 27, 2018 at 7:49 AM Andrew Geissler <geissonator@gmail.com>
wrote:

> On Fri, Aug 24, 2018 at 4:03 PM Tanous, Ed <ed.tanous@intel.com> wrote:
> >
> > I would like to start a discussion around the feature of SOL logging,
> and how to best implement it.
> >
> >
> >
> > There is a proposal here
> https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/
> that proposes dumping the SOL console text buffer from Javascript into a
> file, and presenting it to the user as a download.  On its face, it seems
> to work as designed.  This has some pretty severe limitations, in that you
> can only dump the console log from your session, and your session is
> essentially destroyed when you switch pages in the webui, or hit refresh.
> I think this is overly limiting, and of the production BMC stacks that I
> know of, none of them implement SOL logging in this way.
> >
> >
> >
> > Instead, other BMCs implement it as a circular buffer inside the BMC
> itself, which allows SOL to log data all the time, not just while the user
> is logged in.  I think architecting it this way would be much more useful
> for admins consuming OpenBMC, and make us more useful as a solution.  Doing
> it this way also can make the SOL log available to IPMI and Redfish, as
> well as a file download, which improves the usability quite a bit.  Most
> places I see the SOL logging requested is for audit type purposes, which
> the javascript based version can’t do.
>
> We use https://github.com/openbmc/obmc-console for this currently.  It
> creates a file at /var/log/obmc-console.log and even has a
> customizable size for the file to wrap at. I guess the advantage of
> having it in the GUI is the user could ensure a full dump of the
> console (vs. just grabbing the existing obmc-console.log which may
> have wrapped). Being able to view the SOL output has it happens in the
> GUI, then having an option to export it sounds kind of useful.  Also
> be nice to be able to retrieve and view the existing obmc-console.log
> from the system in case you missed it in the GUI.  Maybe that latter
> part is enough though, reduces GUI complexity and testing. Something
> good to run by our usability/design team Gunnar/Rebecca?
>
> >
> >
> > I’m looking for feedback from the community here.  As written, I don’t
> think the javascript console export is a useful feature given its
> limitations.  Do others agree?  Am I off base?  Are there use models where
> having a log of only the current session is useful?  Should we document the
> limitations so we can respond to bugs in the future?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > -Ed
>

[-- Attachment #2: Type: text/html, Size: 4490 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-24 21:02 SOL Logging Tanous, Ed
  2018-08-25  0:32 ` Sai Dasari
  2018-08-27 12:48 ` Andrew Geissler
@ 2018-08-27 17:00 ` Jeremy Kerr
  2018-08-28 21:16   ` Rebecca Broekhuis
  2018-08-29 11:37 ` Alexander Amelkin
  3 siblings, 1 reply; 15+ messages in thread
From: Jeremy Kerr @ 2018-08-27 17:00 UTC (permalink / raw)
  To: Tanous, Ed, OpenBMC Maillist

Hi Ed,

> Instead, other BMCs implement it as a circular buffer inside the BMC 
> itself, which allows SOL to log data all the time, not just while the 
> user is logged in.

The obmc-console infrastructure does exactly that; the circular buffer 
is available under /var/log/obmc-console.log (by default), and the 
maximum size is configurable.

We may want to augment that with some other bits, like:

  - making this file available over the web UI (including
    transparently pre-populating the scrollback buffer of the
    console interface);

    and/or

  - archiving the log on certain events (like specific host state
    changes; checkstop on OpenPOWER implementeations)

If there's anything that I can change in obmc-console to make things 
work with your plans, let me know.

Cheers,


Jeremy

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-27 17:00 ` Jeremy Kerr
@ 2018-08-28 21:16   ` Rebecca Broekhuis
  2018-08-28 23:00     ` Tanous, Ed
  2018-08-29  2:13     ` Jeremy Kerr
  0 siblings, 2 replies; 15+ messages in thread
From: Rebecca Broekhuis @ 2018-08-28 21:16 UTC (permalink / raw)
  To: jk; +Cc: ed.tanous, openbmc

[-- Attachment #1: Type: text/plain, Size: 1357 bytes --]

Hi Jeremy,

You mentioned the possibility of pre-populating the scrollback buffer of
the console interface. I am wondering if you have any more information about
how difficult this would be to implement. Our concern when thinking about
doing this was how complex it might get to fetch the file and populate the
console
while at the same time receiving messages by websocket without missing any
messages. Any insight on this?

Thanks!
Becca

On Mon, Aug 27, 2018 at 12:00 PM Jeremy Kerr <jk@ozlabs.org> wrote:

> Hi Ed,
>
> > Instead, other BMCs implement it as a circular buffer inside the BMC
> > itself, which allows SOL to log data all the time, not just while the
> > user is logged in.
>
> The obmc-console infrastructure does exactly that; the circular buffer
> is available under /var/log/obmc-console.log (by default), and the
> maximum size is configurable.
>
> We may want to augment that with some other bits, like:
>
>   - making this file available over the web UI (including
>     transparently pre-populating the scrollback buffer of the
>     console interface);
>
>     and/or
>
>   - archiving the log on certain events (like specific host state
>     changes; checkstop on OpenPOWER implementeations)
>
> If there's anything that I can change in obmc-console to make things
> work with your plans, let me know.
>
> Cheers,
>
>
> Jeremy
>

[-- Attachment #2: Type: text/html, Size: 1794 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* RE: SOL Logging
  2018-08-28 21:16   ` Rebecca Broekhuis
@ 2018-08-28 23:00     ` Tanous, Ed
  2018-08-29  2:13     ` Jeremy Kerr
  1 sibling, 0 replies; 15+ messages in thread
From: Tanous, Ed @ 2018-08-28 23:00 UTC (permalink / raw)
  To: Rebecca Broekhuis, jk; +Cc: openbmc

> You mentioned the possibility of pre-populating the scrollback buffer of the console interface. I am wondering if you have any more information about
 > how difficult this would be to implement. Our concern when thinking about doing this was how complex it might get to fetch the file and populate the console
> while at the same time receiving messages by websocket without missing any messages. Any insight on this?

I don't think the hard part is reading in the file and playing it back.  I think the hard part is that the characters that changed the screen state may have already scrolled past the end of the buffer, in which case, you've lost the data that you need to properly paint the screen.  I would likely be better than nothing, but might not be enough?

With that said, isn't this exactly the use case that screen or tmux have?  Is there code we could steal and drop into obmc-console from one of those that tracks the screen state, and on the connection of a new client, blasts the screen state down so that it matches exactly?

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-28 21:16   ` Rebecca Broekhuis
  2018-08-28 23:00     ` Tanous, Ed
@ 2018-08-29  2:13     ` Jeremy Kerr
  1 sibling, 0 replies; 15+ messages in thread
From: Jeremy Kerr @ 2018-08-29  2:13 UTC (permalink / raw)
  To: Rebecca Broekhuis; +Cc: ed.tanous, openbmc

Hi Rebecca,

> You mentioned the possibility of pre-populating the scrollback buffer of 
> the console interface. I am wondering if you have any more information about
> how difficult this would be to implement.

It's easy to do, but difficult to do *correctly*.

A trivial implementation would just:
  1) serve the backlog at a specific static URL
  2) when the client established the live console connection, prepend
     the data from (1) into the session

However, getting the synchronisation between those two (as independent
channels of data) is difficult, and getting that wrong means either
missing or duplicated console data. We'd also need to handle the case
where the backlog has been rotated during the connection.

The logical fix for that would be to combine both channels into the one
stream (and further up the stack): when any client session connects to
the obmc-console server, the server replays the backlog before
outputting live data from the host. Because it's implemented
console-server-side, it would affect both web UI endpoint and the ssh
console endpoint.

The downside of that is the large amount of backlog to be transferred at
the start of the console session, which is typically when you need the
current data the most. In most cases, this could just be stale data from
a previous boot. Because of this, I think it'll be a net *loss* in
usability.

The alternative is just to make the backlog available as a separate
view, and not try to prepopulate the live console.

The console infrastructure really needs to be simple and correct; we use
it in low-level debugging, and complications in those situations can
cause major headaches. Because of this, I've tried to do no alteration
of the console data, keep the things simple, and treat the console
channel as opaque between host and client terminal.

Cheers,


Jeremy

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-24 21:02 SOL Logging Tanous, Ed
                   ` (2 preceding siblings ...)
  2018-08-27 17:00 ` Jeremy Kerr
@ 2018-08-29 11:37 ` Alexander Amelkin
  2018-08-29 20:08   ` Rebecca Broekhuis
  3 siblings, 1 reply; 15+ messages in thread
From: Alexander Amelkin @ 2018-08-29 11:37 UTC (permalink / raw)
  To: openbmc, Artem Senichev


[-- Attachment #1.1.1: Type: text/plain, Size: 2032 bytes --]


25.08.2018 00:02, Tanous, Ed wrote:
>
> I would like to start a discussion around the feature of SOL logging, and how to best implement it.
>
>  
>
> There is a proposal here https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/ that proposes dumping the SOL console text buffer from Javascript into a file, and presenting it to the user as a download.  On its face, it seems to work as designed.  This has some pretty severe limitations, in that you can only dump the console log from your session, and your session is essentially destroyed when you switch pages in the webui, or hit refresh.  I think this is overly limiting, and of the production BMC stacks that I know of, none of them implement SOL logging in this way.
>
>  
>
> Instead, other BMCs implement it as a circular buffer inside the BMC itself, which allows SOL to log data all the time, not just while the user is logged in.  I think architecting it this way would be much more useful for admins consuming OpenBMC, and make us more useful as a solution.  Doing it this way also can make the SOL log available to IPMI and Redfish, as well as a file download, which improves the usability quite a bit.  Most places I see the SOL logging requested is for audit type purposes, which the javascript based version can’t do.
>
>  
>
> I’m looking for feedback from the community here.  As written, I don’t think the javascript console export is a useful feature given its limitations.  Do others agree?  Am I off base?  Are there use models where having a log of only the current session is useful?  Should we document the limitations so we can respond to bugs in the future?
>
>  
>
That's exactly what we have implemented in our BMC for YADRO VESNIN.
We have a circular buffer that constantly logs host console output and also preserves the first several pages of each host boot. That way we always have the last host boot "dmesg" as well as the very last few pages before the host dies (or the logs were gathered).

Alexander.

[-- Attachment #1.1.2: Type: text/html, Size: 4491 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-29 11:37 ` Alexander Amelkin
@ 2018-08-29 20:08   ` Rebecca Broekhuis
  2018-09-05  0:07     ` Emily Shaffer
  0 siblings, 1 reply; 15+ messages in thread
From: Rebecca Broekhuis @ 2018-08-29 20:08 UTC (permalink / raw)
  To: a.amelkin; +Cc: openbmc, a.senichev

[-- Attachment #1: Type: text/plain, Size: 3047 bytes --]

>The downside of that is the large amount of backlog to be transferred at
>the start of the console session, which is typically when you need the
>current data the most. In most cases, this could just be stale data from
>a previous boot. Because of this, I think it'll be a net *loss* in
>usability.

I was looking at how AMI and supermicro's console works in the UI, and when
the console session is started
it is pre populated with just 30 or so log messages, nothing more. Would it
be possible to combine both channels, replaying only the N most recent log
statements?
I may be misunderstanding what is possible here.

I am also wondering if an implementation like this:
 a) is adequate
 b) would be a net gain in usability (considering the amount of backlog to
be transferred at the start of the session would be smaller)

Becca

On Wed, Aug 29, 2018 at 6:37 AM Alexander Amelkin <a.amelkin@yadro.com>
wrote:

>
> 25.08.2018 00:02, Tanous, Ed wrote:
>
> I would like to start a discussion around the feature of SOL logging, and
> how to best implement it.
>
>
>
> There is a proposal here
> https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/
> that proposes dumping the SOL console text buffer from Javascript into a
> file, and presenting it to the user as a download.  On its face, it seems
> to work as designed.  This has some pretty severe limitations, in that you
> can only dump the console log from your session, and your session is
> essentially destroyed when you switch pages in the webui, or hit refresh.
> I think this is overly limiting, and of the production BMC stacks that I
> know of, none of them implement SOL logging in this way.
>
>
>
> Instead, other BMCs implement it as a circular buffer inside the BMC
> itself, which allows SOL to log data all the time, not just while the user
> is logged in.  I think architecting it this way would be much more useful
> for admins consuming OpenBMC, and make us more useful as a solution.  Doing
> it this way also can make the SOL log available to IPMI and Redfish, as
> well as a file download, which improves the usability quite a bit.  Most
> places I see the SOL logging requested is for audit type purposes, which
> the javascript based version can’t do.
>
>
>
> I’m looking for feedback from the community here.  As written, I don’t
> think the javascript console export is a useful feature given its
> limitations.  Do others agree?  Am I off base?  Are there use models where
> having a log of only the current session is useful?  Should we document the
> limitations so we can respond to bugs in the future?
>
>
>
> That's exactly what we have implemented in our BMC for YADRO VESNIN.
> We have a circular buffer that constantly logs host console output and
> also preserves the first several pages of each host boot. That way we
> always have the last host boot "dmesg" as well as the very last few pages
> before the host dies (or the logs were gathered).
>
> Alexander.
>

[-- Attachment #2: Type: text/html, Size: 4431 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-08-29 20:08   ` Rebecca Broekhuis
@ 2018-09-05  0:07     ` Emily Shaffer
  2018-09-05  0:30       ` Ed Tanous
  0 siblings, 1 reply; 15+ messages in thread
From: Emily Shaffer @ 2018-09-05  0:07 UTC (permalink / raw)
  To: Rebecca Broekhuis; +Cc: a.amelkin, openbmc, a.senichev

[-- Attachment #1: Type: text/plain, Size: 3917 bytes --]

I wonder if there's a different and more useful way we can provide the
console scrollback. Thinking about this, it's not just a GUI issue, right -
we deliver our console over TCP to a client elsewhere and would have the
same dedupe issues if we needed to start providing backlog.  Speaking with
almost no experience implementing server-side sockets, is it possible for
us to keep that ring buffer in the console socket as well?

I'm also interested in how you all are getting the console over to webui.
Is it delivered via Redfish? (This is totally different from what we are
doing, so apologies if that's a fundamental question, but I'm certainly
keen to know now :) )

Emily

On Wed, Aug 29, 2018 at 1:09 PM Rebecca Broekhuis <beccabroek@gmail.com>
wrote:

>
> >The downside of that is the large amount of backlog to be transferred at
> >the start of the console session, which is typically when you need the
> >current data the most. In most cases, this could just be stale data from
> >a previous boot. Because of this, I think it'll be a net *loss* in
> >usability.
>
> I was looking at how AMI and supermicro's console works in the UI, and
> when the console session is started
> it is pre populated with just 30 or so log messages, nothing more. Would
> it be possible to combine both channels, replaying only the N most recent
> log statements?
> I may be misunderstanding what is possible here.
>
> I am also wondering if an implementation like this:
>  a) is adequate
>  b) would be a net gain in usability (considering the amount of backlog to
> be transferred at the start of the session would be smaller)
>
> Becca
>
> On Wed, Aug 29, 2018 at 6:37 AM Alexander Amelkin <a.amelkin@yadro.com>
> wrote:
>
>>
>> 25.08.2018 00:02, Tanous, Ed wrote:
>>
>> I would like to start a discussion around the feature of SOL logging, and
>> how to best implement it.
>>
>>
>>
>> There is a proposal here
>> https://gerrit.openbmc-project.xyz/#/c/openbmc/phosphor-webui/+/12063/
>> that proposes dumping the SOL console text buffer from Javascript into a
>> file, and presenting it to the user as a download.  On its face, it seems
>> to work as designed.  This has some pretty severe limitations, in that you
>> can only dump the console log from your session, and your session is
>> essentially destroyed when you switch pages in the webui, or hit refresh.
>> I think this is overly limiting, and of the production BMC stacks that I
>> know of, none of them implement SOL logging in this way.
>>
>>
>>
>> Instead, other BMCs implement it as a circular buffer inside the BMC
>> itself, which allows SOL to log data all the time, not just while the user
>> is logged in.  I think architecting it this way would be much more useful
>> for admins consuming OpenBMC, and make us more useful as a solution.  Doing
>> it this way also can make the SOL log available to IPMI and Redfish, as
>> well as a file download, which improves the usability quite a bit.  Most
>> places I see the SOL logging requested is for audit type purposes, which
>> the javascript based version can’t do.
>>
>>
>>
>> I’m looking for feedback from the community here.  As written, I don’t
>> think the javascript console export is a useful feature given its
>> limitations.  Do others agree?  Am I off base?  Are there use models where
>> having a log of only the current session is useful?  Should we document the
>> limitations so we can respond to bugs in the future?
>>
>>
>>
>> That's exactly what we have implemented in our BMC for YADRO VESNIN.
>> We have a circular buffer that constantly logs host console output and
>> also preserves the first several pages of each host boot. That way we
>> always have the last host boot "dmesg" as well as the very last few pages
>> before the host dies (or the logs were gathered).
>>
>> Alexander.
>>
>

[-- Attachment #2: Type: text/html, Size: 5582 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-09-05  0:07     ` Emily Shaffer
@ 2018-09-05  0:30       ` Ed Tanous
  2018-09-05 18:23         ` Rebecca Broekhuis
                           ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Ed Tanous @ 2018-09-05  0:30 UTC (permalink / raw)
  To: openbmc

On 09/04/2018 05:07 PM, Emily Shaffer wrote:
> I wonder if there's a different and more useful way we can provide the 
> console scrollback. Thinking about this, it's not just a GUI issue, 
> right - we deliver our console over TCP to a client elsewhere and would 
> have the same dedupe issues if we needed to start providing backlog.  

Correct, this isn't _just_ a webui issue.  It happens on all the 
available interfaces (SSH, HTML5, and obmc-console-client)

> Speaking with almost no experience implementing server-side sockets, is 
> it possible for us to keep that ring buffer in the console socket as well?

obmc-console already keeps a ring-buffer style log of all the previous 
characters.  The easiest to implement answer, which is probably good 
enough for most cases, would be to simply make obmc-console barf up the 
saved console log ring buffer when any new client connects.  This will 
have problems if your control characters have run off the end of the 
buffer, and the console would appear to quickly replay your last X 
characters worth of interactions, which isn't desired.

I could also imagine a case where this isn't the desired behavior if 
you're using automation to interact with SSH/SOL, so it might need to be 
selectable by the client or have some kind of reasonable workaround.

I'm fairly certain there's some buffering routines we can steal from 
screen and/or tmux that would just track the screen state, and blast the 
minimum character sets to each client on reconnect, but I haven't looked 
into it too deeply.

> 
> I'm also interested in how you all are getting the console over to 
> webui.  Is it delivered via Redfish? (This is totally different from 
> what we are doing, so apologies if that's a fundamental question, but 
> I'm certainly keen to know now :) )
> 

Today the console is being dumped to a dumb websocket that just mirrors, 
encodes, and encrypts the traffic to the browser using websocket frames. 
  On the browser side, we pipe that console into hterm, which turns the 
raw characters into a reasonably correct console.

https://chromium.googlesource.com/apps/libapps/+/master/hterm

At some point there was talk of moving the html5 stuff over to xterm, to 
support more encoding styles, but I don't think anyone has started that 
work.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-09-05  0:30       ` Ed Tanous
@ 2018-09-05 18:23         ` Rebecca Broekhuis
  2018-09-10  6:52         ` Stewart Smith
  2018-09-14  2:58         ` Jeremy Kerr
  2 siblings, 0 replies; 15+ messages in thread
From: Rebecca Broekhuis @ 2018-09-05 18:23 UTC (permalink / raw)
  To: ed.tanous; +Cc: openbmc

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

>At some point there was talk of moving the html5 stuff over to xterm, to
>support more encoding styles, but I don't think anyone has started that
>work.

I've started and am currently working on switching from hterm to xterm at
the moment, as an update.

https://github.com/xtermjs/xterm.js/


On Tue, Sep 4, 2018 at 7:29 PM Ed Tanous <ed.tanous@intel.com> wrote:

> On 09/04/2018 05:07 PM, Emily Shaffer wrote:
> > I wonder if there's a different and more useful way we can provide the
> > console scrollback. Thinking about this, it's not just a GUI issue,
> > right - we deliver our console over TCP to a client elsewhere and would
> > have the same dedupe issues if we needed to start providing backlog.
>
> Correct, this isn't _just_ a webui issue.  It happens on all the
> available interfaces (SSH, HTML5, and obmc-console-client)
>
> > Speaking with almost no experience implementing server-side sockets, is
> > it possible for us to keep that ring buffer in the console socket as
> well?
>
> obmc-console already keeps a ring-buffer style log of all the previous
> characters.  The easiest to implement answer, which is probably good
> enough for most cases, would be to simply make obmc-console barf up the
> saved console log ring buffer when any new client connects.  This will
> have problems if your control characters have run off the end of the
> buffer, and the console would appear to quickly replay your last X
> characters worth of interactions, which isn't desired.
>
> I could also imagine a case where this isn't the desired behavior if
> you're using automation to interact with SSH/SOL, so it might need to be
> selectable by the client or have some kind of reasonable workaround.
>
> I'm fairly certain there's some buffering routines we can steal from
> screen and/or tmux that would just track the screen state, and blast the
> minimum character sets to each client on reconnect, but I haven't looked
> into it too deeply.
>
> >
> > I'm also interested in how you all are getting the console over to
> > webui.  Is it delivered via Redfish? (This is totally different from
> > what we are doing, so apologies if that's a fundamental question, but
> > I'm certainly keen to know now :) )
> >
>
> Today the console is being dumped to a dumb websocket that just mirrors,
> encodes, and encrypts the traffic to the browser using websocket frames.
>   On the browser side, we pipe that console into hterm, which turns the
> raw characters into a reasonably correct console.
>
> https://chromium.googlesource.com/apps/libapps/+/master/hterm
>
> At some point there was talk of moving the html5 stuff over to xterm, to
> support more encoding styles, but I don't think anyone has started that
> work.
>

[-- Attachment #2: Type: text/html, Size: 3538 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-09-05  0:30       ` Ed Tanous
  2018-09-05 18:23         ` Rebecca Broekhuis
@ 2018-09-10  6:52         ` Stewart Smith
  2018-09-14  2:58         ` Jeremy Kerr
  2 siblings, 0 replies; 15+ messages in thread
From: Stewart Smith @ 2018-09-10  6:52 UTC (permalink / raw)
  To: Ed Tanous, openbmc

Ed Tanous <ed.tanous@intel.com> writes:
> On 09/04/2018 05:07 PM, Emily Shaffer wrote:
>> I wonder if there's a different and more useful way we can provide the 
>> console scrollback. Thinking about this, it's not just a GUI issue, 
>> right - we deliver our console over TCP to a client elsewhere and would 
>> have the same dedupe issues if we needed to start providing backlog.  
>
> Correct, this isn't _just_ a webui issue.  It happens on all the 
> available interfaces (SSH, HTML5, and obmc-console-client)
>
>> Speaking with almost no experience implementing server-side sockets, is 
>> it possible for us to keep that ring buffer in the console socket as well?
>
> obmc-console already keeps a ring-buffer style log of all the previous 
> characters.  The easiest to implement answer, which is probably good 
> enough for most cases, would be to simply make obmc-console barf up the 
> saved console log ring buffer when any new client connects.  This will 
> have problems if your control characters have run off the end of the 
> buffer, and the console would appear to quickly replay your last X 
> characters worth of interactions, which isn't desired.
>
> I could also imagine a case where this isn't the desired behavior if 
> you're using automation to interact with SSH/SOL, so it might need to be 
> selectable by the client or have some kind of reasonable workaround.

Yeah, if we got some random amount of previous console text over IPMI
SoL or SSH host console, I can imagine that all my CI scripts would
pretty spectacularly blow up.

For the Web UI though, I can see that keeping something like a tmux
session would be good though.
-- 
Stewart Smith
OPAL Architect, IBM.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: SOL Logging
  2018-09-05  0:30       ` Ed Tanous
  2018-09-05 18:23         ` Rebecca Broekhuis
  2018-09-10  6:52         ` Stewart Smith
@ 2018-09-14  2:58         ` Jeremy Kerr
  2 siblings, 0 replies; 15+ messages in thread
From: Jeremy Kerr @ 2018-09-14  2:58 UTC (permalink / raw)
  To: Ed Tanous, openbmc

Hi Ed,

> obmc-console already keeps a ring-buffer style log of all the previous 
> characters.  The easiest to implement answer, which is probably good 
> enough for most cases, would be to simply make obmc-console barf up the 
> saved console log ring buffer when any new client connects.

Yep, exactly. However, you're exactly right in that:

> I could also imagine a case where this isn't the desired behavior if 
> you're using automation to interact with SSH/SOL, so it might need to be 
> selectable by the client or have some kind of reasonable workaround.

This is definitely the main issue for me here (as well as the control 
characters). Having potentially days-old console data received on 
connect would be somewhat confusing for most users; automation or otherwise.

Cheers,


Jeremy

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2018-09-14  2:58 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-24 21:02 SOL Logging Tanous, Ed
2018-08-25  0:32 ` Sai Dasari
2018-08-27 12:48 ` Andrew Geissler
2018-08-27 14:21   ` Rebecca Broekhuis
2018-08-27 17:00 ` Jeremy Kerr
2018-08-28 21:16   ` Rebecca Broekhuis
2018-08-28 23:00     ` Tanous, Ed
2018-08-29  2:13     ` Jeremy Kerr
2018-08-29 11:37 ` Alexander Amelkin
2018-08-29 20:08   ` Rebecca Broekhuis
2018-09-05  0:07     ` Emily Shaffer
2018-09-05  0:30       ` Ed Tanous
2018-09-05 18:23         ` Rebecca Broekhuis
2018-09-10  6:52         ` Stewart Smith
2018-09-14  2:58         ` Jeremy Kerr

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.