Saving off warnings to a separate file with Python structured logging

Saving off warnings to a separate file with Python structured logging

[Joshua Watt]

It has been asked how to configure bitbake to save off WARNING messages (and higher) to a separate file when running bitbake. With python structured logging, this can be done relatively easily without having to modify any code.

First, create the structured config file (we'll call it warn.json for this example). This file should contain:

{

     "version": 1,

     "formatters": {

         "warnlogFormatter": {

             "()": "bb.msg.BBLogFormatter",

             "format": "%(levelname)s: %(message)s"

         }

     },

     "handlers": {

         "warnlog": {

             "class": "logging.FileHandler",

             "formatter": "warnlogFormatter",

             "level": "WARNING",

             "filename": "warn.log"

         }

     },

     "loggers": {

         "BitBake": {

             "handlers": ["warnlog"]

         }

     },

     "@disable_existing_loggers": false

}

After this, you need to point bitbake at this configuration file by adding to the path to it in BB_LOGCONFIG, for example:

BB_LOGCONFIG = "warn.json"


After doing this, bitbake should write all WARNING and higher messages to the file warn.log.

For more information on Python structured logging, see https://docs.python.org/3/howto/logging-cookbook.html.

Note that bitbake implements several helper classes for structured logging in lib/bb/msg.py

Re: [bitbake-devel] Saving off warnings to a separate file with Python structured logging

[Quentin Schulz]

Hi Joshua,

This seems to be too valuable information to be lost in the mailing list 
archives. Any chance of having this somewhere in the bitbake/yocto docs?

Cc'ing Michael and the docs mailing list on this.

Thanks for the tip!

Cheers,
Quentin

On 1/26/23 22:27, Joshua Watt wrote:
> It has been asked how to configure bitbake to save off WARNING messages 
> (and higher) to a separate file when running bitbake. With python 
> structured logging, this can be done relatively easily without having to 
> modify any code.
> 
> First, create the structured config file (we'll call it warn.json for 
> this example). This file should contain:
> 
> {
> 
>      "version": 1,
> 
>      "formatters": {
> 
>          "warnlogFormatter": {
> 
>              "()": "bb.msg.BBLogFormatter",
> 
>              "format": "%(levelname)s: %(message)s"
> 
>          }
> 
>      },
> 
>      "handlers": {
> 
>          "warnlog": {
> 
>              "class": "logging.FileHandler",
> 
>              "formatter": "warnlogFormatter",
> 
>              "level": "WARNING",
> 
>              "filename": "warn.log"
> 
>          }
> 
>      },
> 
>      "loggers": {
> 
>          "BitBake": {
> 
>              "handlers": ["warnlog"]
> 
>          }
> 
>      },
> 
>      "@disable_existing_loggers": false
> 
> }
> 
> After this, you need to point bitbake at this configuration file by 
> adding to the path to it in BB_LOGCONFIG, for example:
> 
> BB_LOGCONFIG = "warn.json"
> 
> 
> After doing this, bitbake should write all WARNING and higher messages 
> to the file warn.log.
> 
> For more information on Python structured logging, see 
> https://urldefense.com/v3/__https://docs.python.org/3/howto/logging-cookbook.html__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTnPSRsiUg$ .
> 
> Note that bitbake implements several helper classes for structured 
> logging in lib/bb/msg.py
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14355): https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/message/14355__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTlK7G8NjQ$
> Mute This Topic: https://urldefense.com/v3/__https://lists.openembedded.org/mt/96553269/6293953__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTl2pVuArw$
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://urldefense.com/v3/__https://lists.openembedded.org/g/bitbake-devel/unsub__;!!OOPJP91ZZw!iQrFYknjFg5zSdxaIH5idDius4IExGVkAMWKZo1EEHAl1ggvC5n3_HMrHn3tOmbSpVNIPPLepZqB53Daq9uen6TNsTne92m1xw$  [quentin.schulz@theobroma-systems.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [docs] [bitbake-devel] Saving off warnings to a separate file with Python structured logging

[Richard Purdie]

On Fri, 2023-01-27 at 14:24 +0100, Quentin Schulz via
lists.yoctoproject.org wrote:
> Hi Joshua,
> 
> This seems to be too valuable information to be lost in the mailing list 
> archives. Any chance of having this somewhere in the bitbake/yocto docs?
> 
> Cc'ing Michael and the docs mailing list on this.

The context was that we discussed this in triage yesterday and said
making this public was the first step but then we'd document it in the
manuals. Michael was on that call so knows about it! :)

Cheers,

Richard

Re: [docs] [bitbake-devel] Saving off warnings to a separate file with Python structured logging

[Michael Opdenacker]

On 27.01.23 at 14:26, Richard Purdie wrote:
> On Fri, 2023-01-27 at 14:24 +0100, Quentin Schulz via
> lists.yoctoproject.org wrote:
>> Hi Joshua,
>>
>> This seems to be too valuable information to be lost in the mailing list
>> archives. Any chance of having this somewhere in the bitbake/yocto docs?
>>
>> Cc'ing Michael and the docs mailing list on this.
> The context was that we discussed this in triage yesterday and said
> making this public was the first step but then we'd document it in the
> manuals. Michael was on that call so knows about it! :)

Actually, there is already documentation about BB_LOGCONFIG in 
https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging. 
I'll propose an update taking Joshua's guidelines into account. Let me 
know if you had something else in mind, and if you see obsolete details 
on this page.

The provided JSON file still seems to work, but it serves a different 
purpose.

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [bitbake-devel] Saving off warnings to a separate file with Python structured logging

[Richard Purdie]

On Fri, 2023-01-27 at 18:12 +0100, Michael Opdenacker wrote:
> On 27.01.23 at 14:26, Richard Purdie wrote:
> > On Fri, 2023-01-27 at 14:24 +0100, Quentin Schulz via
> > lists.yoctoproject.org wrote:
> > > Hi Joshua,
> > > 
> > > This seems to be too valuable information to be lost in the mailing list
> > > archives. Any chance of having this somewhere in the bitbake/yocto docs?
> > > 
> > > Cc'ing Michael and the docs mailing list on this.
> > The context was that we discussed this in triage yesterday and said
> > making this public was the first step but then we'd document it in the
> > manuals. Michael was on that call so knows about it! :)
> 
> Actually, there is already documentation about BB_LOGCONFIG in 
> https://docs.yoctoproject.org/bitbake/dev/bitbake-user-manual/bitbake-user-manual-execution.html#logging. 
> I'll propose an update taking Joshua's guidelines into account. Let me 
> know if you had something else in mind, and if you see obsolete details 
> on this page.
> 
> The provided JSON file still seems to work, but it serves a different 
> purpose.

Thanks, I'd forgotten we'd added that example so that makes sense. I
think we should add the warnings one as well, the link to the python
docs also seems useful and perhaps making it clearer how to use
BB_LOGCONFIG?

Cheers,

Richard

[PATCH] bitbake-user-manual: show how use BB_LOGCONFIG to log warnings

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
Suggested-by: Joshua Watt <JPEWhacker@gmail.com>
---
 .../bitbake-user-manual-execution.rst         | 45 +++++++++++++++++--
 1 file changed, 41 insertions(+), 4 deletions(-)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-execution.rst b/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
index f6ebf7ba..4fa3ca4b 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-execution.rst
@@ -656,7 +656,7 @@ builds are when execute, bitbake also supports user defined
 configuration of the `Python
 logging <https://docs.python.org/3/library/logging.html>`__ facilities
 through the :term:`BB_LOGCONFIG` variable. This
-variable defines a json or yaml `logging
+variable defines a JSON or YAML `logging
 configuration <https://docs.python.org/3/library/logging.config.html>`__
 that will be intelligently merged into the default configuration. The
 logging configuration is merged using the following rules:
@@ -690,9 +690,9 @@ logging configuration is merged using the following rules:
    adds a filter called ``BitBake.defaultFilter``, both filters will be
    applied to the logger
 
-As an example, consider the following user logging configuration file
-which logs all Hash Equivalence related messages of VERBOSE or higher to
-a file called ``hashequiv.log`` ::
+As a first example, you can create a ``hashequiv.json`` user logging
+configuration file to log all Hash Equivalence related messages of ``VERBOSE``
+or higher priority to a file called ``hashequiv.log``::
 
    {
        "version": 1,
@@ -721,3 +721,40 @@ a file called ``hashequiv.log`` ::
            }
        }
    }
+
+Then set the :term:`BB_LOGCONFIG` variable in ``conf/local.conf``::
+
+   BB_LOGCONFIG = "hashequiv.json"
+
+Another example is this ``warn.json`` file to log all ``WARNING`` and
+higher priority messages to a ``warn.log`` file::
+
+  {
+      "version": 1,
+      "formatters": {
+          "warnlogFormatter": {
+              "()": "bb.msg.BBLogFormatter",
+              "format": "%(levelname)s: %(message)s"
+          }
+      },
+
+      "handlers": {
+          "warnlog": {
+              "class": "logging.FileHandler",
+              "formatter": "warnlogFormatter",
+              "level": "WARNING",
+              "filename": "warn.log"
+          }
+      },
+
+      "loggers": {
+          "BitBake": {
+              "handlers": ["warnlog"]
+          }
+      },
+
+      "@disable_existing_loggers": false
+  }
+
+Note that BitBake's helper classes for structured logging are implemented in
+``lib/bb/msg.py``.
-- 
2.37.2