Re: [RFC] Add Alternative Log Buffer Support for printk Messages

[Carl-Daniel Hailfinger <c-d.hailfinger.devel.2006@gmx.net>]

On 31.01.2009 10:59, Wolfgang Denk wrote:
> Dear Grant & Andrew,
>
> In message <C5A8B8CD.143B5%gerickson@nuovations.com> you wrote:
>   
>>> We have an external log buffer, into which stuff was written by (say)
>>> the boot prom code.
>>>       
>
> Correct. Typically we use this in U-Boot to pass Power-On Self Test
> results to applications running under Linux, using the standard syslog
> tools.
>
> Another are of use is to save the log buffer over a reset (crash)  of
> the  system,  so  that  it can be read either rom the boot loader or,
> after a reboot, from Linux.
>   

That part is not yet implemented in coreboot, but it certainly is useful.


>>> We want that to come out on the console during bootup.
>>>       
>
> Not necessarily. The main function is to feed it into the syslog
> processing.
>   

Indeed.


>>> So what we do is to divert the normal printk log buffer so that it
>>> starts writing at the tail of that existing external buffer.  We emit
>>> the entire buffer some time during bootup.  We henceforth continue to
>>> use that external buffer in the normal manner. If so, why do it this way?
>>>       
>
> Do you have a better approach how to share  the  log  buffer  between
> Linux  and  a boot loader in such a way that the content will survive
> crahses and reboots?
>
>
>   
>> I believe this to be a useful feature for the embedded system space;
>> however, I am not wedded in any way to the current implementation and am
>> happy to evolve it as needed to meet the needs of key stakeholders.
>>     
>
> Indeed we use this heavily in embedded systems, mainly to process POST
> results under Linux (for example, to disable certain functions when
> the POST results indicate hardware issues, etc.).
>   

For coreboot, the feature is used to avoid long delays for serial output
and to make the coreboot POST logs available on machines where you don't
want to or can't store the POST logs. Coreboot POST logs at maximum
verbosity are roughly 61kB large on my machine and may be bigger on
others. Default verbosity still has a few dozen lines in the log.


>>> Why not just copy the external buffer's contents into the normal buffer during
>>> bootup?  ie: printk("%s", external_buffer).
>>>       
>
> For two reasons:
>
> 1) Additional copy operation (ok, that's cheap).
>   

Indeed. It probably won't take more than a few microseconds if the
external buffer is already in DRAM.


> 2) We would lose the capability that the buffer content even survives 
>    a reset / crash / reboot.
>
>    This is a feature can be very useful in embedded systems where
>    there is (at least normally) nobody looking at any console
>    messages; being able to see the kernel's crash messages after a
>    reboot in the normal syslog messages can be really helpful.
>   

This benefit is a bit debatable on x86 due to the lack of non-volatile
fast memory with decent sizes. (Yes, you can store the log in the flash
ROM, but it will wear out.)
Coreboot can try to preserve RAM contents, but depending on the
particular hardware all bets are off.


>>> I think a quite bad part of this feature is that it renders the
>>> kernel's log_buf_len boot parameter (and its Kconfig defult)
>>> ineffective.  That's a pretty useful feature - people often set it to
>>> 1MB or more during problem diagnosis, and the new unalterable 16kbytes
>>> is very very small.
>>>       
>
> Yes, we are aware of this, but it's a compromise. Basicly it depends
> on hardware capabilities. In most cases, it is sufficient to keep the
> syslog buffer in RAM. But if you really have to guarantee that the
> content of the buffer remains stable, this cannot be used because
> during a processor reset there will be usually a too long pause until
> the memory controller gets re-initialized and RAM refresh continues -
> we might lose data in this time. In most cases it just works fine,
> but it's out of spec. In such situations, when such a guarantee is
> needed, we can use any SRAM that may be available on the board, or we
> can use some on-chip-memory (OCM, basicly also SRAM) that might be on
> the processor.
>
> The OCM on the processors in question is 16 kB, hence the limitation
> there.
>   

I'd like to have the buffer location at least configurable to use that
feature on regular desktop and server x86 machines.


> But note that the code is general enough to support the other options
> (other, bigger SRAM, or arbitrarily sized buffer in RAM), too.
>   

Good.


>> Agreed; however, I'm not sure how often this technique is employed in
>> embedded systems versus desktop systems.
>>     
>
> I haven't seen this ever used in desktop systems yet. And I doubt if
> there is any x86 with a boot loader that supports somehting like this.
>   

Coreboot v3 (free x86 firmware, formerly known as LinuxBIOS) does this.


> But there is for example a pretty large number of mobile cranes all
> over the world which are using this feature :-)
>
>   
>> I'd be happy to keep the existing log buffer and perform a prepend from the
>> boot buffer to it rather than an append of the kernel buffer to the boot
>> buffer. Thoughts?
>>     
>
> Please don't - this would implement only the way to pass the buffer
> from the boot loader too Linux, but not the way back.
>   

It would be great if the location of that buffer could be retrieved
either via ACPI or e820 tables on x86. Hardcoded locations do not make
sense for x86 because coreboot tries to keep memory as unfragmented as
possible and stores the buffer at the top of memory (well, you can
hardcode the buffer location in coreboot but that is just a hack).


Unless I misread the code, it assumes no wraparound has occured in the
firmware log buffer. With full verbosity in coreboot, such wraparounds
can happen.

The coreboot log buffer struct is really simple:
struct printk_buffer {
        int len;
        int readoffset;
        int writeoffset;
        char buffer[];
};

The buffer is implemented as a ring buffer with variable size. The size
is determined from a few runtime variables for minimum RAM waste, so
hardcoding it to some fixed value would be suboptimal.

Can we coordinate to make sure both U-Boot and coreboot can use the feature?


Regards,
Carl-Daniel

-- 
http://www.hailfinger.org/