[SPECIFICATION RFC v3] The firmware and bootloader log specification

[SPECIFICATION RFC v3] The firmware and bootloader log specification

[Alec Brown]

Hi everyone,

I've been working on improving the specification for the firmware and bootloader
log that Daniel Kiper has proposed and take into account most of the suggestions
that were made in these threads [1], [2].

The goal is to allow various boot components to pass logs to the running OS and
then to the user space for processing and analysis. These logs should be generic
enough so that it can work in multiple environments and be human readable.

It has yet to be decided where to put the final version of this specification.
It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
or be standalone, such as a part of OASIS Standards.

Below is how the layout of these logs would store their data.

bf_log_header:
               +-------------------+
u32            | version           |
u32            | size              |
u8[64]         | producer          |
u8[64]         | log_format        |
u64            | flags             |
u64            | next_bflh_addr    |
u64            | log_addr          |
u32            | log_size          |
               +-------------------+

bf_log_buffer:
               +-------------------+
u32            | version           |
u32            | size              |
u8[64]         | producer          |
u32            | next_msg_off      |
bf_log_msg[l]  | msgs              |
               +-------------------+

bf_log_msg:
               +-------------------+
u32            | size              |
u64            | ts_nsec           |
u32            | level             |
u32            | facility          |
u32            | msg_off           |
u8[n]          | type              |
u8[m]          | msg               |
               +-------------------+

Where l is the number of msgs, n is the size of type, and m is the size of the
msg.

The bf_log_header structure forms a linked list. Each bf_log_header element in
the linked list points to the individual log buffer and the next bf_log_header
element in the linked list. The first element in the linked list points to the
last boot component in the boot chain. The last element points to the starting
boot component in the boot chain. The log buffers which contain the log
messages are produced by the various boot components, typically from the
firmware to the bootloader. The log message is stored in a log format that is
compatible with the boot component that produced it.

The fields in bf_log_header structure:
  - version: the firmware and bootloader log header version number, 1 for now,
  - size: the size of the bf_log_header to allow for backward compatibility if 
    other fields are added,
  - producer: the producer/firmware/bootloader/... entity, NUL terminated
    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
  - log_format: the format used to record the log messages, NUL terminated
    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
    logs in various formats if needed,
  - flags: bit field used to store information about the log state, if bit 0 has
    been set it means the log was truncated,
  - next_bflh_addr: the physical address of the next bf_log_header structure,
    none if zero,
  - log_addr: the physical address of where the log buffer is stored,
  - log_size: the total size of the log buffer.

The bf_log_buffer is used to store log messages from the firmware and
bootloader. This format for storing messages is called the bf log format. The
bf_log_buffer contains the header information of the bf log format with the log
messages being stored in an array of bf_log_msg messages.

The fields in bf_log_buffer structure:
  - version: the firmware and bootloader log version number, 1 for now,
  - size: the total allocated space for the bf_log_buffer including the log
    messages stored in msgs,
  - producer: the producer/firmware/bootloader/... entity, NUL terminated
    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
    as the field in bf_log_header,
  - next_msg_off: the byte offset from the beginning of the allocated space for
    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
  - msgs: the array of log messages stored in the bf_log_msg structures.

The fields in bf_log_msg structure:
  - size: the total size of the bf_log_msg entry,
  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
    using this log format defines the meaning of 0,
  - level: similar to the syslog meaning; used to differentiate normal log
    messages from debug log messages, but the exact interpretation depends on
    the producer,
  - facility: similar to the syslog meaning; used to differentiate the sources
    of the log messages, but the exact interpretation depends on the producer,
  - msg_off: the byte offset which the msg field starts in bf_log_msg,
  - type: the log message type; similar to facility but NUL terminated string
    instead of integer, but the exact interpretation depends on the producer,
  - msg: the log message, NUL terminated string.

In bf_log_msg, the producers are free to use or ignore any of the level,
facility, and type fields. If level or facility are ignored, they should be set
to 0. If type is ignored, it should be set to an empty NUL terminated string.

Since it doesn't seem possible to have each boot component using the same log
format, we added a log_format and log_phys_addr fields to give flexibility in
how logs are stored. An example of a different log format that can be used is
the cbmem_console log format used by coreboot:

cbmem_console:
               +-------------------+
u32            | size              |
u32            | cursor            |
u8[m]          | body              |
               +-------------------+

There is still the outstanding issue of how the logs will be sent to the OS. If
UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
are used, it becomes a lot trickier to solve this issue.

Any suggestions are much appreciated and will be taken into consideration.

I will be presenting this work at the LPC System Boot and Security
Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
join if you want to discuss the design. The schedule for the System Boot and
Security Micro-conference can be found here [3].

Thanks!
Alec Brown

[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

[SPECIFICATION RFC v3] The firmware and bootloader log specification

[Alec Brown]

Hi everyone,

I've been working on improving the specification for the firmware and bootloader
log that Daniel Kiper has proposed and take into account most of the suggestions
that were made in these threads [1], [2].

The goal is to allow various boot components to pass logs to the running OS and
then to the user space for processing and analysis. These logs should be generic
enough so that it can work in multiple environments and be human readable.

It has yet to be decided where to put the final version of this specification.
It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
or be standalone, such as a part of OASIS Standards.

Below is how the layout of these logs would store their data.

bf_log_header:
               +-------------------+
u32            | version           |
u32            | size              |
u8[64]         | producer          |
u8[64]         | log_format        |
u64            | flags             |
u64            | next_bflh_addr    |
u64            | log_addr          |
u32            | log_size          |
               +-------------------+

bf_log_buffer:
               +-------------------+
u32            | version           |
u32            | size              |
u8[64]         | producer          |
u32            | next_msg_off      |
bf_log_msg[l]  | msgs              |
               +-------------------+

bf_log_msg:
               +-------------------+
u32            | size              |
u64            | ts_nsec           |
u32            | level             |
u32            | facility          |
u32            | msg_off           |
u8[n]          | type              |
u8[m]          | msg               |
               +-------------------+

Where l is the number of msgs, n is the size of type, and m is the size of the
msg.

The bf_log_header structure forms a linked list. Each bf_log_header element in
the linked list points to the individual log buffer and the next bf_log_header
element in the linked list. The first element in the linked list points to the
last boot component in the boot chain. The last element points to the starting
boot component in the boot chain. The log buffers which contain the log
messages are produced by the various boot components, typically from the
firmware to the bootloader. The log message is stored in a log format that is
compatible with the boot component that produced it.

The fields in bf_log_header structure:
  - version: the firmware and bootloader log header version number, 1 for now,
  - size: the size of the bf_log_header to allow for backward compatibility if 
    other fields are added,
  - producer: the producer/firmware/bootloader/... entity, NUL terminated
    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
  - log_format: the format used to record the log messages, NUL terminated
    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
    logs in various formats if needed,
  - flags: bit field used to store information about the log state, if bit 0 has
    been set it means the log was truncated,
  - next_bflh_addr: the physical address of the next bf_log_header structure,
    none if zero,
  - log_addr: the physical address of where the log buffer is stored,
  - log_size: the total size of the log buffer.

The bf_log_buffer is used to store log messages from the firmware and
bootloader. This format for storing messages is called the bf log format. The
bf_log_buffer contains the header information of the bf log format with the log
messages being stored in an array of bf_log_msg messages.

The fields in bf_log_buffer structure:
  - version: the firmware and bootloader log version number, 1 for now,
  - size: the total allocated space for the bf_log_buffer including the log
    messages stored in msgs,
  - producer: the producer/firmware/bootloader/... entity, NUL terminated
    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
    as the field in bf_log_header,
  - next_msg_off: the byte offset from the beginning of the allocated space for
    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
  - msgs: the array of log messages stored in the bf_log_msg structures.

The fields in bf_log_msg structure:
  - size: the total size of the bf_log_msg entry,
  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
    using this log format defines the meaning of 0,
  - level: similar to the syslog meaning; used to differentiate normal log
    messages from debug log messages, but the exact interpretation depends on
    the producer,
  - facility: similar to the syslog meaning; used to differentiate the sources
    of the log messages, but the exact interpretation depends on the producer,
  - msg_off: the byte offset which the msg field starts in bf_log_msg,
  - type: the log message type; similar to facility but NUL terminated string
    instead of integer, but the exact interpretation depends on the producer,
  - msg: the log message, NUL terminated string.

In bf_log_msg, the producers are free to use or ignore any of the level,
facility, and type fields. If level or facility are ignored, they should be set
to 0. If type is ignored, it should be set to an empty NUL terminated string.

Since it doesn't seem possible to have each boot component using the same log
format, we added a log_format and log_phys_addr fields to give flexibility in
how logs are stored. An example of a different log format that can be used is
the cbmem_console log format used by coreboot:

cbmem_console:
               +-------------------+
u32            | size              |
u32            | cursor            |
u8[m]          | body              |
               +-------------------+

There is still the outstanding issue of how the logs will be sent to the OS. If
UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
are used, it becomes a lot trickier to solve this issue.

Any suggestions are much appreciated and will be taken into consideration.

I will be presenting this work at the LPC System Boot and Security
Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
join if you want to discuss the design. The schedule for the System Boot and
Security Micro-conference can be found here [3].

Thanks!
Alec Brown

[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Heinrich Schuchardt]

Am 18. September 2021 18:04:13 MESZ schrieb Alec Brown <alec.r.brown@oracle.com>:
>Hi everyone,
>
>I've been working on improving the specification for the firmware and bootloader
>log that Daniel Kiper has proposed and take into account most of the suggestions
>that were made in these threads [1], [2].
>
>The goal is to allow various boot components to pass logs to the running OS and
>then to the user space for processing and analysis. These logs should be generic
>enough so that it can work in multiple environments and be human readable.

Hello Alec,

in your mail it remains unclear which information you want to put into the log and why it is needed. I would prefer the motivation and content to be clarified before defining any interface structures.

We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol). Why do we need to start from scratch?

Best regards

Heinrich 

>
>It has yet to be decided where to put the final version of this specification.
>It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
>or be standalone, such as a part of OASIS Standards.
>
>Below is how the layout of these logs would store their data.
>
>bf_log_header:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u8[64]         | log_format        |
>u64            | flags             |
>u64            | next_bflh_addr    |
>u64            | log_addr          |
>u32            | log_size          |
>               +-------------------+
>
>bf_log_buffer:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u32            | next_msg_off      |
>bf_log_msg[l]  | msgs              |
>               +-------------------+
>
>bf_log_msg:
>               +-------------------+
>u32            | size              |
>u64            | ts_nsec           |
>u32            | level             |
>u32            | facility          |
>u32            | msg_off           |
>u8[n]          | type              |
>u8[m]          | msg               |
>               +-------------------+
>
>Where l is the number of msgs, n is the size of type, and m is the size of the
>msg.
>
>The bf_log_header structure forms a linked list. Each bf_log_header element in
>the linked list points to the individual log buffer and the next bf_log_header
>element in the linked list. The first element in the linked list points to the
>last boot component in the boot chain. The last element points to the starting
>boot component in the boot chain. The log buffers which contain the log
>messages are produced by the various boot components, typically from the
>firmware to the bootloader. The log message is stored in a log format that is
>compatible with the boot component that produced it.
>
>The fields in bf_log_header structure:
>  - version: the firmware and bootloader log header version number, 1 for now,
>  - size: the size of the bf_log_header to allow for backward compatibility if 
>    other fields are added,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
>  - log_format: the format used to record the log messages, NUL terminated
>    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
>    logs in various formats if needed,
>  - flags: bit field used to store information about the log state, if bit 0 has
>    been set it means the log was truncated,
>  - next_bflh_addr: the physical address of the next bf_log_header structure,
>    none if zero,
>  - log_addr: the physical address of where the log buffer is stored,
>  - log_size: the total size of the log buffer.
>
>The bf_log_buffer is used to store log messages from the firmware and
>bootloader. This format for storing messages is called the bf log format. The
>bf_log_buffer contains the header information of the bf log format with the log
>messages being stored in an array of bf_log_msg messages.
>
>The fields in bf_log_buffer structure:
>  - version: the firmware and bootloader log version number, 1 for now,
>  - size: the total allocated space for the bf_log_buffer including the log
>    messages stored in msgs,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
>    as the field in bf_log_header,
>  - next_msg_off: the byte offset from the beginning of the allocated space for
>    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
>  - msgs: the array of log messages stored in the bf_log_msg structures.
>
>The fields in bf_log_msg structure:
>  - size: the total size of the bf_log_msg entry,
>  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
>    using this log format defines the meaning of 0,
>  - level: similar to the syslog meaning; used to differentiate normal log
>    messages from debug log messages, but the exact interpretation depends on
>    the producer,
>  - facility: similar to the syslog meaning; used to differentiate the sources
>    of the log messages, but the exact interpretation depends on the producer,
>  - msg_off: the byte offset which the msg field starts in bf_log_msg,
>  - type: the log message type; similar to facility but NUL terminated string
>    instead of integer, but the exact interpretation depends on the producer,
>  - msg: the log message, NUL terminated string.
>
>In bf_log_msg, the producers are free to use or ignore any of the level,
>facility, and type fields. If level or facility are ignored, they should be set
>to 0. If type is ignored, it should be set to an empty NUL terminated string.
>
>Since it doesn't seem possible to have each boot component using the same log
>format, we added a log_format and log_phys_addr fields to give flexibility in
>how logs are stored. An example of a different log format that can be used is
>the cbmem_console log format used by coreboot:
>
>cbmem_console:
>               +-------------------+
>u32            | size              |
>u32            | cursor            |
>u8[m]          | body              |
>               +-------------------+
>
>There is still the outstanding issue of how the logs will be sent to the OS. If
>UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
>use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
>are used, it becomes a lot trickier to solve this issue.
>
>Any suggestions are much appreciated and will be taken into consideration.
>
>I will be presenting this work at the LPC System Boot and Security
>Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
>join if you want to discuss the design. The schedule for the System Boot and
>Security Micro-conference can be found here [3].
>
>Thanks!
>Alec Brown
>
>[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
>[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
>[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Heinrich Schuchardt]

Am 18. September 2021 18:04:13 MESZ schrieb Alec Brown <alec.r.brown@oracle.com>:
>Hi everyone,
>
>I've been working on improving the specification for the firmware and bootloader
>log that Daniel Kiper has proposed and take into account most of the suggestions
>that were made in these threads [1], [2].
>
>The goal is to allow various boot components to pass logs to the running OS and
>then to the user space for processing and analysis. These logs should be generic
>enough so that it can work in multiple environments and be human readable.

Hello Alec,

in your mail it remains unclear which information you want to put into the log and why it is needed. I would prefer the motivation and content to be clarified before defining any interface structures.

We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol). Why do we need to start from scratch?

Best regards

Heinrich 

>
>It has yet to be decided where to put the final version of this specification.
>It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
>or be standalone, such as a part of OASIS Standards.
>
>Below is how the layout of these logs would store their data.
>
>bf_log_header:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u8[64]         | log_format        |
>u64            | flags             |
>u64            | next_bflh_addr    |
>u64            | log_addr          |
>u32            | log_size          |
>               +-------------------+
>
>bf_log_buffer:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u32            | next_msg_off      |
>bf_log_msg[l]  | msgs              |
>               +-------------------+
>
>bf_log_msg:
>               +-------------------+
>u32            | size              |
>u64            | ts_nsec           |
>u32            | level             |
>u32            | facility          |
>u32            | msg_off           |
>u8[n]          | type              |
>u8[m]          | msg               |
>               +-------------------+
>
>Where l is the number of msgs, n is the size of type, and m is the size of the
>msg.
>
>The bf_log_header structure forms a linked list. Each bf_log_header element in
>the linked list points to the individual log buffer and the next bf_log_header
>element in the linked list. The first element in the linked list points to the
>last boot component in the boot chain. The last element points to the starting
>boot component in the boot chain. The log buffers which contain the log
>messages are produced by the various boot components, typically from the
>firmware to the bootloader. The log message is stored in a log format that is
>compatible with the boot component that produced it.
>
>The fields in bf_log_header structure:
>  - version: the firmware and bootloader log header version number, 1 for now,
>  - size: the size of the bf_log_header to allow for backward compatibility if 
>    other fields are added,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
>  - log_format: the format used to record the log messages, NUL terminated
>    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
>    logs in various formats if needed,
>  - flags: bit field used to store information about the log state, if bit 0 has
>    been set it means the log was truncated,
>  - next_bflh_addr: the physical address of the next bf_log_header structure,
>    none if zero,
>  - log_addr: the physical address of where the log buffer is stored,
>  - log_size: the total size of the log buffer.
>
>The bf_log_buffer is used to store log messages from the firmware and
>bootloader. This format for storing messages is called the bf log format. The
>bf_log_buffer contains the header information of the bf log format with the log
>messages being stored in an array of bf_log_msg messages.
>
>The fields in bf_log_buffer structure:
>  - version: the firmware and bootloader log version number, 1 for now,
>  - size: the total allocated space for the bf_log_buffer including the log
>    messages stored in msgs,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
>    as the field in bf_log_header,
>  - next_msg_off: the byte offset from the beginning of the allocated space for
>    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
>  - msgs: the array of log messages stored in the bf_log_msg structures.
>
>The fields in bf_log_msg structure:
>  - size: the total size of the bf_log_msg entry,
>  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
>    using this log format defines the meaning of 0,
>  - level: similar to the syslog meaning; used to differentiate normal log
>    messages from debug log messages, but the exact interpretation depends on
>    the producer,
>  - facility: similar to the syslog meaning; used to differentiate the sources
>    of the log messages, but the exact interpretation depends on the producer,
>  - msg_off: the byte offset which the msg field starts in bf_log_msg,
>  - type: the log message type; similar to facility but NUL terminated string
>    instead of integer, but the exact interpretation depends on the producer,
>  - msg: the log message, NUL terminated string.
>
>In bf_log_msg, the producers are free to use or ignore any of the level,
>facility, and type fields. If level or facility are ignored, they should be set
>to 0. If type is ignored, it should be set to an empty NUL terminated string.
>
>Since it doesn't seem possible to have each boot component using the same log
>format, we added a log_format and log_phys_addr fields to give flexibility in
>how logs are stored. An example of a different log format that can be used is
>the cbmem_console log format used by coreboot:
>
>cbmem_console:
>               +-------------------+
>u32            | size              |
>u32            | cursor            |
>u8[m]          | body              |
>               +-------------------+
>
>There is still the outstanding issue of how the logs will be sent to the OS. If
>UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
>use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
>are used, it becomes a lot trickier to solve this issue.
>
>Any suggestions are much appreciated and will be taken into consideration.
>
>I will be presenting this work at the LPC System Boot and Security
>Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
>join if you want to discuss the design. The schedule for the System Boot and
>Security Micro-conference can be found here [3].
>
>Thanks!
>Alec Brown
>
>[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
>[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
>[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Heinrich Schuchardt]

Am 18. September 2021 18:04:13 MESZ schrieb Alec Brown <alec.r.brown@oracle.com>:
>Hi everyone,
>
>I've been working on improving the specification for the firmware and bootloader
>log that Daniel Kiper has proposed and take into account most of the suggestions
>that were made in these threads [1], [2].
>
>The goal is to allow various boot components to pass logs to the running OS and
>then to the user space for processing and analysis. These logs should be generic
>enough so that it can work in multiple environments and be human readable.

Hello Alec,

in your mail it remains unclear which information you want to put into the log and why it is needed. I would prefer the motivation and content to be clarified before defining any interface structures.

We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol). Why do we need to start from scratch?

Best regards

Heinrich 

>
>It has yet to be decided where to put the final version of this specification.
>It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
>or be standalone, such as a part of OASIS Standards.
>
>Below is how the layout of these logs would store their data.
>
>bf_log_header:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u8[64]         | log_format        |
>u64            | flags             |
>u64            | next_bflh_addr    |
>u64            | log_addr          |
>u32            | log_size          |
>               +-------------------+
>
>bf_log_buffer:
>               +-------------------+
>u32            | version           |
>u32            | size              |
>u8[64]         | producer          |
>u32            | next_msg_off      |
>bf_log_msg[l]  | msgs              |
>               +-------------------+
>
>bf_log_msg:
>               +-------------------+
>u32            | size              |
>u64            | ts_nsec           |
>u32            | level             |
>u32            | facility          |
>u32            | msg_off           |
>u8[n]          | type              |
>u8[m]          | msg               |
>               +-------------------+
>
>Where l is the number of msgs, n is the size of type, and m is the size of the
>msg.
>
>The bf_log_header structure forms a linked list. Each bf_log_header element in
>the linked list points to the individual log buffer and the next bf_log_header
>element in the linked list. The first element in the linked list points to the
>last boot component in the boot chain. The last element points to the starting
>boot component in the boot chain. The log buffers which contain the log
>messages are produced by the various boot components, typically from the
>firmware to the bootloader. The log message is stored in a log format that is
>compatible with the boot component that produced it.
>
>The fields in bf_log_header structure:
>  - version: the firmware and bootloader log header version number, 1 for now,
>  - size: the size of the bf_log_header to allow for backward compatibility if 
>    other fields are added,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
>  - log_format: the format used to record the log messages, NUL terminated
>    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
>    logs in various formats if needed,
>  - flags: bit field used to store information about the log state, if bit 0 has
>    been set it means the log was truncated,
>  - next_bflh_addr: the physical address of the next bf_log_header structure,
>    none if zero,
>  - log_addr: the physical address of where the log buffer is stored,
>  - log_size: the total size of the log buffer.
>
>The bf_log_buffer is used to store log messages from the firmware and
>bootloader. This format for storing messages is called the bf log format. The
>bf_log_buffer contains the header information of the bf log format with the log
>messages being stored in an array of bf_log_msg messages.
>
>The fields in bf_log_buffer structure:
>  - version: the firmware and bootloader log version number, 1 for now,
>  - size: the total allocated space for the bf_log_buffer including the log
>    messages stored in msgs,
>  - producer: the producer/firmware/bootloader/... entity, NUL terminated
>    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
>    as the field in bf_log_header,
>  - next_msg_off: the byte offset from the beginning of the allocated space for
>    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
>  - msgs: the array of log messages stored in the bf_log_msg structures.
>
>The fields in bf_log_msg structure:
>  - size: the total size of the bf_log_msg entry,
>  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
>    using this log format defines the meaning of 0,
>  - level: similar to the syslog meaning; used to differentiate normal log
>    messages from debug log messages, but the exact interpretation depends on
>    the producer,
>  - facility: similar to the syslog meaning; used to differentiate the sources
>    of the log messages, but the exact interpretation depends on the producer,
>  - msg_off: the byte offset which the msg field starts in bf_log_msg,
>  - type: the log message type; similar to facility but NUL terminated string
>    instead of integer, but the exact interpretation depends on the producer,
>  - msg: the log message, NUL terminated string.
>
>In bf_log_msg, the producers are free to use or ignore any of the level,
>facility, and type fields. If level or facility are ignored, they should be set
>to 0. If type is ignored, it should be set to an empty NUL terminated string.
>
>Since it doesn't seem possible to have each boot component using the same log
>format, we added a log_format and log_phys_addr fields to give flexibility in
>how logs are stored. An example of a different log format that can be used is
>the cbmem_console log format used by coreboot:
>
>cbmem_console:
>               +-------------------+
>u32            | size              |
>u32            | cursor            |
>u8[m]          | body              |
>               +-------------------+
>
>There is still the outstanding issue of how the logs will be sent to the OS. If
>UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
>use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
>are used, it becomes a lot trickier to solve this issue.
>
>Any suggestions are much appreciated and will be taken into consideration.
>
>I will be presenting this work at the LPC System Boot and Security
>Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
>join if you want to discuss the design. The schedule for the System Boot and
>Security Micro-conference can be found here [3].
>
>Thanks!
>Alec Brown
>
>[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
>[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
>[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Peter Stuge]

Alec Brown wrote:
> Below is how the layout of these logs would store their data.
> 
> bf_log_header:
>                +-------------------+
> u32            | version           |
> u32            | size              |
> u8[64]         | producer          |
> u8[64]         | log_format        |
> u64            | flags             |
> u64            | next_bflh_addr    |
> u64            | log_addr          |
> u32            | log_size          |
>                +-------------------+

I suggest to include a .magic at least in bf_log_header and an
.xor_checksum or .crc32 only in bf_log_header.

.magic doubles as endianess indicator when the structures are
stored on movable media. (Pick an asymmetric magic bit pattern!)

I suggest renaming .next_bflh_addr to .next_log_header and .log_addr
to .log_buffer_addr.

I suggest to remove .size and .log_size:

The rationale for .size is "to allow for backward compatibility" but
that seems redundant thanks to .version.

.log_size can be calculated from the subordinate data and is thus
mostly an unneccessary source of potential inconsistency.


> bf_log_buffer:
>                +-------------------+
> u32            | version           |
> u32            | size              |
> u8[64]         | producer          |
> u32            | next_msg_off      |
> bf_log_msg[l]  | msgs              |
>                +-------------------+

I suggest replacing .size and .next_msg_off with .messages containing l:

.size can then be calculated from .messages; again, reliably avoiding
inconsistency between .size and .next_msg_off.

Allocated size doesn't seem useful if writers must anyway maintain state
containing the starting address. If writers must be allowed to be completely
stateless then maybe at least rename .size to .allocated_size and see below
for discovery.

Having .messages also eliminates the need for an end-of-messages marker
when the allocated space is not yet filled.


> bf_log_msg:
>                +-------------------+
> u32            | size              |
> u64            | ts_nsec           |
> u32            | level             |
> u32            | facility          |
> u32            | msg_off           |
> u8[n]          | type              |
> u8[m]          | msg               |
>                +-------------------+

It seems inconsistent that log_header.size and log_msg.size cover only
the respective struct itself while log_buffer.size also covers all
subordinate messages. Skipping all .size in this version fixes that.

And log_msg.size is not very useful since both .type and .msg have variable
length; it's not possible to access .msg without scanning .type. Please at
a minimum add .type_size but better yet replace .size with .type_size and
.msg_size.


> There is still the outstanding issue of how the logs will be sent to the OS. If
> UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
> use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
> are used, it becomes a lot trickier to solve this issue.
> 
> Any suggestions are much appreciated and will be taken into consideration.

Having bf_log_header.magic and some bf_log_header.$checksum, a strict rule
for bf_log_header start address granularity and a strict maximum offset
for the first header from top and/or bottom of memory allows to quickly
discover a log in memory without explicit handover.


> LPC System Boot and Security Micro-conference on the 22nd of September
> at 7:50 AM PDT (14:50 UTC).

Have fun! :)


Heinrich Schuchardt wrote:
> We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol).
> Why do we need to start from scratch?

That's a good question. I guess noone wants to settle for a standard
from somewhere else. ;)

I wouldn't mind if log_msg was a syslog transport, but I can understand
if that's rejected because syslog messages require a lot of parsing for
presentation while Alec's proposal seems focused on efficiency and simplicity.

It's also nice to be able to strictly mandate UTF-8 for all fields.
(RFC 5424 allows MSG to be anything.)


Kind regards

//Peter

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Peter Stuge]

Alec Brown wrote:
> Below is how the layout of these logs would store their data.
> 
> bf_log_header:
>                +-------------------+
> u32            | version           |
> u32            | size              |
> u8[64]         | producer          |
> u8[64]         | log_format        |
> u64            | flags             |
> u64            | next_bflh_addr    |
> u64            | log_addr          |
> u32            | log_size          |
>                +-------------------+

I suggest to include a .magic at least in bf_log_header and an
.xor_checksum or .crc32 only in bf_log_header.

.magic doubles as endianess indicator when the structures are
stored on movable media. (Pick an asymmetric magic bit pattern!)

I suggest renaming .next_bflh_addr to .next_log_header and .log_addr
to .log_buffer_addr.

I suggest to remove .size and .log_size:

The rationale for .size is "to allow for backward compatibility" but
that seems redundant thanks to .version.

.log_size can be calculated from the subordinate data and is thus
mostly an unneccessary source of potential inconsistency.


> bf_log_buffer:
>                +-------------------+
> u32            | version           |
> u32            | size              |
> u8[64]         | producer          |
> u32            | next_msg_off      |
> bf_log_msg[l]  | msgs              |
>                +-------------------+

I suggest replacing .size and .next_msg_off with .messages containing l:

.size can then be calculated from .messages; again, reliably avoiding
inconsistency between .size and .next_msg_off.

Allocated size doesn't seem useful if writers must anyway maintain state
containing the starting address. If writers must be allowed to be completely
stateless then maybe at least rename .size to .allocated_size and see below
for discovery.

Having .messages also eliminates the need for an end-of-messages marker
when the allocated space is not yet filled.


> bf_log_msg:
>                +-------------------+
> u32            | size              |
> u64            | ts_nsec           |
> u32            | level             |
> u32            | facility          |
> u32            | msg_off           |
> u8[n]          | type              |
> u8[m]          | msg               |
>                +-------------------+

It seems inconsistent that log_header.size and log_msg.size cover only
the respective struct itself while log_buffer.size also covers all
subordinate messages. Skipping all .size in this version fixes that.

And log_msg.size is not very useful since both .type and .msg have variable
length; it's not possible to access .msg without scanning .type. Please at
a minimum add .type_size but better yet replace .size with .type_size and
.msg_size.


> There is still the outstanding issue of how the logs will be sent to the OS. If
> UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
> use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
> are used, it becomes a lot trickier to solve this issue.
> 
> Any suggestions are much appreciated and will be taken into consideration.

Having bf_log_header.magic and some bf_log_header.$checksum, a strict rule
for bf_log_header start address granularity and a strict maximum offset
for the first header from top and/or bottom of memory allows to quickly
discover a log in memory without explicit handover.


> LPC System Boot and Security Micro-conference on the 22nd of September
> at 7:50 AM PDT (14:50 UTC).

Have fun! :)


Heinrich Schuchardt wrote:
> We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol).
> Why do we need to start from scratch?

That's a good question. I guess noone wants to settle for a standard
from somewhere else. ;)

I wouldn't mind if log_msg was a syslog transport, but I can understand
if that's rejected because syslog messages require a lot of parsing for
presentation while Alec's proposal seems focused on efficiency and simplicity.

It's also nice to be able to strictly mandate UTF-8 for all fields.
(RFC 5424 allows MSG to be anything.)


Kind regards

//Peter

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Julius Werner]

> Since it doesn't seem possible to have each boot component using the same log
> format, we added a log_format and log_phys_addr fields to give flexibility in
> how logs are stored. An example of a different log format that can be used is
> the cbmem_console log format used by coreboot:

I am not exactly sure how you expect this interoperability you seem to
be suggesting here to work. Are you saying that your bf_log_header can
sometimes point to the bf_log_buffer structure you define, and
sometimes to a coreboot CBMEM console buffer? But that's a completely
different format that requires a different reader implementation, how
is that supposed to work? If this proposal is just "we define a
wrapper structure that points to everyone's custom firmware log
implementation", then I don't really see the point (the only benefit
still left then might be discovery of the log buffer, but that's the
part you leave open in your design while all those other
implementations already have working discovery mechanisms of their own
anyway).

For the other structures you have defined, the same feedback that I
think was already mentioned on the last iteration of this thread still
applies: it seems incredibly bloated for a simple firmware logging
mechanism. You have a whooping 24+n bytes of overhead *per line* which
probably comes out to somewhere between 30-50% of total space wasted
on overhead for the average log buffer. I guess there are just
fundamentally different opinions on how featureful a firmware log
mechanism needs to be so we're probably not gonna find a format that
makes everyone happy here, but at least for the coreboot project I see
little reason for us to implement something like this when we already
have a well-working existing solution with tooling and wideranged
support.

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Julius Werner]

> Since it doesn't seem possible to have each boot component using the same log
> format, we added a log_format and log_phys_addr fields to give flexibility in
> how logs are stored. An example of a different log format that can be used is
> the cbmem_console log format used by coreboot:

I am not exactly sure how you expect this interoperability you seem to
be suggesting here to work. Are you saying that your bf_log_header can
sometimes point to the bf_log_buffer structure you define, and
sometimes to a coreboot CBMEM console buffer? But that's a completely
different format that requires a different reader implementation, how
is that supposed to work? If this proposal is just "we define a
wrapper structure that points to everyone's custom firmware log
implementation", then I don't really see the point (the only benefit
still left then might be discovery of the log buffer, but that's the
part you leave open in your design while all those other
implementations already have working discovery mechanisms of their own
anyway).

For the other structures you have defined, the same feedback that I
think was already mentioned on the last iteration of this thread still
applies: it seems incredibly bloated for a simple firmware logging
mechanism. You have a whooping 24+n bytes of overhead *per line* which
probably comes out to somewhere between 30-50% of total space wasted
on overhead for the average log buffer. I guess there are just
fundamentally different opinions on how featureful a firmware log
mechanism needs to be so we're probably not gonna find a format that
makes everyone happy here, but at least for the coreboot project I see
little reason for us to implement something like this when we already
have a well-working existing solution with tooling and wideranged
support.

Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Alec Brown]

On Sun, Sep 19, 2021 at 12:53:35AM +0200, Heinrich Schuchardt wrote:
> 
> 
> Am 18. September 2021 18:04:13 MESZ schrieb Alec Brown <alec.r.brown@oracle.com>:
> >Hi everyone,
> >
> >I've been working on improving the specification for the firmware and bootloader
> >log that Daniel Kiper has proposed and take into account most of the suggestions
> >that were made in these threads [1], [2].
> >
> >The goal is to allow various boot components to pass logs to the running OS and
> >then to the user space for processing and analysis. These logs should be generic
> >enough so that it can work in multiple environments and be human readable.
> 
> Hello Alec,
> 
> in your mail it remains unclear which information you want to put into the log and why it is needed. I would prefer the motivation and content to be clarified before defining any interface structures.
> 
> We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol). Why do we need to start from scratch?
> 
> Best regards
> 
> Heinrich 

Hi Heinrich,

The motivation behind developing these logs was to allow TrenchBoot to be able
to view how the platform was setup during boot. We intend for our specification
to target the Bootloader and Firmware and collect logs from them, but not TPM
event logs, which is what the EFI_TCG2_PROTOCOL collects. We plan on using our
specification for the GRUB Bootloader since it will be more efficient and
flexible to use than the syslog protocol. However, if other boot components
want to use a different logging format, our bf_log_header allows them to do so.

Alec Brown

> 
> >
> >It has yet to be decided where to put the final version of this specification.
> >It should be merged into an existing specification, e.g. UEFI, ACPI, Multiboot2,
> >or be standalone, such as a part of OASIS Standards.
> >
> >Below is how the layout of these logs would store their data.
> >
> >bf_log_header:
> >               +-------------------+
> >u32            | version           |
> >u32            | size              |
> >u8[64]         | producer          |
> >u8[64]         | log_format        |
> >u64            | flags             |
> >u64            | next_bflh_addr    |
> >u64            | log_addr          |
> >u32            | log_size          |
> >               +-------------------+
> >
> >bf_log_buffer:
> >               +-------------------+
> >u32            | version           |
> >u32            | size              |
> >u8[64]         | producer          |
> >u32            | next_msg_off      |
> >bf_log_msg[l]  | msgs              |
> >               +-------------------+
> >
> >bf_log_msg:
> >               +-------------------+
> >u32            | size              |
> >u64            | ts_nsec           |
> >u32            | level             |
> >u32            | facility          |
> >u32            | msg_off           |
> >u8[n]          | type              |
> >u8[m]          | msg               |
> >               +-------------------+
> >
> >Where l is the number of msgs, n is the size of type, and m is the size of the
> >msg.
> >
> >The bf_log_header structure forms a linked list. Each bf_log_header element in
> >the linked list points to the individual log buffer and the next bf_log_header
> >element in the linked list. The first element in the linked list points to the
> >last boot component in the boot chain. The last element points to the starting
> >boot component in the boot chain. The log buffers which contain the log
> >messages are produced by the various boot components, typically from the
> >firmware to the bootloader. The log message is stored in a log format that is
> >compatible with the boot component that produced it.
> >
> >The fields in bf_log_header structure:
> >  - version: the firmware and bootloader log header version number, 1 for now,
> >  - size: the size of the bf_log_header to allow for backward compatibility if 
> >    other fields are added,
> >  - producer: the producer/firmware/bootloader/... entity, NUL terminated
> >    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage,
> >  - log_format: the format used to record the log messages, NUL terminated
> >    string, e.g. bf_log_msg, cbmem_cons, etc.; various producers may generate
> >    logs in various formats if needed,
> >  - flags: bit field used to store information about the log state, if bit 0 has
> >    been set it means the log was truncated,
> >  - next_bflh_addr: the physical address of the next bf_log_header structure,
> >    none if zero,
> >  - log_addr: the physical address of where the log buffer is stored,
> >  - log_size: the total size of the log buffer.
> >
> >The bf_log_buffer is used to store log messages from the firmware and
> >bootloader. This format for storing messages is called the bf log format. The
> >bf_log_buffer contains the header information of the bf log format with the log
> >messages being stored in an array of bf_log_msg messages.
> >
> >The fields in bf_log_buffer structure:
> >  - version: the firmware and bootloader log version number, 1 for now,
> >  - size: the total allocated space for the bf_log_buffer including the log
> >    messages stored in msgs,
> >  - producer: the producer/firmware/bootloader/... entity, NUL terminated
> >    string, e.g. GRUB, Coreboot; the length allows for ASCII UUID storage; same
> >    as the field in bf_log_header,
> >  - next_msg_off: the byte offset from the beginning of the allocated space for
> >    bf_log_buffer to the next byte after the last bf_log_msg in msgs,
> >  - msgs: the array of log messages stored in the bf_log_msg structures.
> >
> >The fields in bf_log_msg structure:
> >  - size: the total size of the bf_log_msg entry,
> >  - ts_nsec: the timestamp in nanoseconds starting from 0 (zero); the producer
> >    using this log format defines the meaning of 0,
> >  - level: similar to the syslog meaning; used to differentiate normal log
> >    messages from debug log messages, but the exact interpretation depends on
> >    the producer,
> >  - facility: similar to the syslog meaning; used to differentiate the sources
> >    of the log messages, but the exact interpretation depends on the producer,
> >  - msg_off: the byte offset which the msg field starts in bf_log_msg,
> >  - type: the log message type; similar to facility but NUL terminated string
> >    instead of integer, but the exact interpretation depends on the producer,
> >  - msg: the log message, NUL terminated string.
> >
> >In bf_log_msg, the producers are free to use or ignore any of the level,
> >facility, and type fields. If level or facility are ignored, they should be set
> >to 0. If type is ignored, it should be set to an empty NUL terminated string.
> >
> >Since it doesn't seem possible to have each boot component using the same log
> >format, we added a log_format and log_phys_addr fields to give flexibility in
> >how logs are stored. An example of a different log format that can be used is
> >the cbmem_console log format used by coreboot:
> >
> >cbmem_console:
> >               +-------------------+
> >u32            | size              |
> >u32            | cursor            |
> >u8[m]          | body              |
> >               +-------------------+
> >
> >There is still the outstanding issue of how the logs will be sent to the OS. If
> >UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
> >use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
> >are used, it becomes a lot trickier to solve this issue.
> >
> >Any suggestions are much appreciated and will be taken into consideration.
> >
> >I will be presenting this work at the LPC System Boot and Security
> >Micro-conference on the 22nd of September at 7:50 AM PDT (14:50 UTC). Come and
> >join if you want to discuss the design. The schedule for the System Boot and
> >Security Micro-conference can be found here [3].
> >
> >Thanks!
> >Alec Brown
> >
> >[1] https://lists.gnu.org/archive/html/grub-devel/2020-11/msg00100.html
> >[2] https://lists.gnu.org/archive/html/grub-devel/2020-12/msg00021.html
> >[3] https://linuxplumbersconf.org/event/11/sessions/116/#20210922

Re: [External] : Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Alec Brown]

On Tue, Sep 21, 2021 at 03:40:20PM +0000, Peter Stuge wrote:
> Alec Brown wrote:
> > Below is how the layout of these logs would store their data.
> > 
> > bf_log_header:
> >                +-------------------+
> > u32            | version           |
> > u32            | size              |
> > u8[64]         | producer          |
> > u8[64]         | log_format        |
> > u64            | flags             |
> > u64            | next_bflh_addr    |
> > u64            | log_addr          |
> > u32            | log_size          |
> >                +-------------------+
> 
> I suggest to include a .magic at least in bf_log_header and an
> .xor_checksum or .crc32 only in bf_log_header.
> 
> .magic doubles as endianess indicator when the structures are
> stored on movable media. (Pick an asymmetric magic bit pattern!)

This is something we will need to think about.

>  
> I suggest renaming .next_bflh_addr to .next_log_header and .log_addr
> to .log_buffer_addr.
> 
> I suggest to remove .size and .log_size:
> 
> The rationale for .size is "to allow for backward compatibility" but
> that seems redundant thanks to .version.
> 
> .log_size can be calculated from the subordinate data and is thus
> mostly an unneccessary source of potential inconsistency.

Looking back, I agree with removing .size since .version accomplishes the same
task. I'm not so sure on removing .log_size as it can be very convenient, and
.log_size won't need to be calculated every time someone wants to know the size
of the logs generated from the boot component.

> 
> 
> > bf_log_buffer:
> >                +-------------------+
> > u32            | version           |
> > u32            | size              |
> > u8[64]         | producer          |
> > u32            | next_msg_off      |
> > bf_log_msg[l]  | msgs              |
> >                +-------------------+
> 
> I suggest replacing .size and .next_msg_off with .messages containing l:
> 
> .size can then be calculated from .messages; again, reliably avoiding
> inconsistency between .size and .next_msg_off.
> 
> Allocated size doesn't seem useful if writers must anyway maintain state
> containing the starting address. If writers must be allowed to be completely
> stateless then maybe at least rename .size to .allocated_size and see below
> for discovery.
> 
> Having .messages also eliminates the need for an end-of-messages marker
> when the allocated space is not yet filled.
> 

After some thinking, it makes sense to replace the meaning of .size with the
meaning of .next_msg_off and removing .next_msg_off from the structure. This 
wouldn't be useful to store for the readers of the log.

> 
> > bf_log_msg:
> >                +-------------------+
> > u32            | size              |
> > u64            | ts_nsec           |
> > u32            | level             |
> > u32            | facility          |
> > u32            | msg_off           |
> > u8[n]          | type              |
> > u8[m]          | msg               |
> >                +-------------------+
> 
> It seems inconsistent that log_header.size and log_msg.size cover only
> the respective struct itself while log_buffer.size also covers all
> subordinate messages. Skipping all .size in this version fixes that.
> 
> And log_msg.size is not very useful since both .type and .msg have variable
> length; it's not possible to access .msg without scanning .type. Please at
> a minimum add .type_size but better yet replace .size with .type_size and
> .msg_size.
> 

You bring up some good points about the names for the fields and that they need
to be more consistent. By removing .size in bf_log_header, this should make it
more consistent.

> 
> > There is still the outstanding issue of how the logs will be sent to the OS. If
> > UEFI is used, we can use config tables. If ACPI or Device Tree is used, we can
> > use bf_log_header.next_bflh_addr to present the logs. If none of these platforms
> > are used, it becomes a lot trickier to solve this issue.
> > 
> > Any suggestions are much appreciated and will be taken into consideration.
> 
> Having bf_log_header.magic and some bf_log_header.$checksum, a strict rule
> for bf_log_header start address granularity and a strict maximum offset
> for the first header from top and/or bottom of memory allows to quickly
> discover a log in memory without explicit handover.
> 

This is something we'll have to think about some more. We aren't convinced about
using .magic for log discovery and are looking for a more explicit way of doing
this.

> 
> > LPC System Boot and Security Micro-conference on the 22nd of September
> > at 7:50 AM PDT (14:50 UTC).
> 
> Have fun! :)
> 
> 
> Heinrich Schuchardt wrote:
> > We already the EFI_TCG2_PROTOCOL and RFC 5424 (The syslog protocol).
> > Why do we need to start from scratch?
> 
> That's a good question. I guess noone wants to settle for a standard
> from somewhere else. ;)
> 
> I wouldn't mind if log_msg was a syslog transport, but I can understand
> if that's rejected because syslog messages require a lot of parsing for
> presentation while Alec's proposal seems focused on efficiency and simplicity.
> 
> It's also nice to be able to strictly mandate UTF-8 for all fields.
> (RFC 5424 allows MSG to be anything.)
> 
> 
> Kind regards
> 
> //Peter

Re: [External] : Re: [SPECIFICATION RFC v3] The firmware and bootloader log specification

[Alec Brown]

On Tue, Sep 21, 2021 at 03:07:25PM -0700, Julius Werner wrote:
> > Since it doesn't seem possible to have each boot component using the same log
> > format, we added a log_format and log_phys_addr fields to give flexibility in
> > how logs are stored. An example of a different log format that can be used is
> > the cbmem_console log format used by coreboot:
> 
> I am not exactly sure how you expect this interoperability you seem to
> be suggesting here to work. Are you saying that your bf_log_header can
> sometimes point to the bf_log_buffer structure you define, and
> sometimes to a coreboot CBMEM console buffer? But that's a completely
> different format that requires a different reader implementation, how
> is that supposed to work? If this proposal is just "we define a
> wrapper structure that points to everyone's custom firmware log
> implementation", then I don't really see the point (the only benefit
> still left then might be discovery of the log buffer, but that's the
> part you leave open in your design while all those other
> implementations already have working discovery mechanisms of their own
> anyway).
>

Depending on the preferred logging format by the boot component, bf_log_header
can point to our bf_log_buffer, a Coreboot CBMEM console buffer, or some other
preferred logging format. We are looking at ways to pass all logs through one
mechanism to make it simpler to discover all logs along the boot chain. From our
understanding, the CBMEM console is passed to the OS via a CBMEM table stored in
lower memory in a forwarding entry and is discovered by a user space tool
looking through /dev/mem. Is this correct? We aren't entirely sure how other
implementations of log buffers accomplish this, but we think that an explicit
mechanism can be beneficial. This is something we'll need to expand upon in the
specification.
 
> For the other structures you have defined, the same feedback that I
> think was already mentioned on the last iteration of this thread still
> applies: it seems incredibly bloated for a simple firmware logging
> mechanism. You have a whooping 24+n bytes of overhead *per line* which
> probably comes out to somewhere between 30-50% of total space wasted
> on overhead for the average log buffer. I guess there are just
> fundamentally different opinions on how featureful a firmware log
> mechanism needs to be so we're probably not gonna find a format that
> makes everyone happy here, but at least for the coreboot project I see
> little reason for us to implement something like this when we already
> have a well-working existing solution with tooling and wideranged
> support.

Since, the logging format we are proposing won't be the best for each boot
component environment, we are giving the option for boot components to use other
logging formats that best suits their environment.