* [bitbake-devel][PATCH 00/15] Add support for python structured logging
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds support for users to provide configuration for Python structured
logging in bitbake. This gives a high degree of flexibility to the way
that logs are capture and displayed, so users can generate very targeted
logging if desired. The initial use case for this is to maintain the
high level of hash equivalence logging on the Yocto autobuilders, while
reducing the noise for most other users.
I've added documentation updates for the new BB_LOGCONFIG variable, but
it probably needs some grooming.
Joshua Watt (15):
bitbake: lib/bb/msg.py: Convert default domains to a dictionary
bitbake: knotty: Handle logging messages with specific logger
bitbake: lib/bb/msg.py: Use log level instead of debug count
bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
bitbake: knotty: Add commented logging_tree code
bitbake: lib/bb/msg.py: Add filter utilities
bitbake: lib/bb/msg.py: Remove unused filters
bitbake: lib/bb/msg.py: Add helper to set logging config
bitbake: knotty: Remove dependency on format variable
bitbake: knotty: Setup logs with config helper
bitbake: worker: Remove unnecessary addDefaultLogFilter
bitbake: Log hash equivalence with a different logger
bitbake: Add autobuilder logging configuration
bitbake: Lower hash equivalence logging
bitbake: doc: Add documentation for BB_LOGCONFIG
bitbake/bin/bitbake-worker | 3 +-
bitbake/contrib/autobuilderlog.json | 27 +++
.../bitbake-user-manual-execution.xml | 97 ++++++++++
.../bitbake-user-manual-ref-variables.xml | 11 ++
bitbake/lib/bb/__init__.py | 7 +-
bitbake/lib/bb/msg.py | 172 +++++++++++++----
bitbake/lib/bb/runqueue.py | 13 +-
bitbake/lib/bb/siggen.py | 17 +-
bitbake/lib/bb/tinfoil.py | 4 +-
bitbake/lib/bb/ui/knotty.py | 173 ++++++++++++------
10 files changed, 419 insertions(+), 105 deletions(-)
create mode 100644 bitbake/contrib/autobuilderlog.json
--
2.17.1
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH 01/15] bitbake: lib/bb/msg.py: Convert default domains to a dictionary
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Converts the default domain variable to a dictionary where the keys are
the logging domains and the values are the logging level (instead of the
debug count). This makes it easier to deal with the logging domains and
the awkward conversion from a list to a dictionary only needs to be done
once when logging is initialized. Finally, other code has been written
that already assumes this variable is a dictionary, see:
f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/__init__.py | 5 +++--
bitbake/lib/bb/msg.py | 17 +++++++----------
2 files changed, 10 insertions(+), 12 deletions(-)
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index 88641e280b..acd4af13a8 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -43,12 +43,13 @@ class BBLogger(Logger):
Logger.__init__(self, name)
def bbdebug(self, level, msg, *args, **kwargs):
+ loglevel = logging.DEBUG - level + 1
if not bb.event.worker_pid:
- if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]):
+ if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
return
if level > (bb.msg.loggerDefaultDebugLevel):
return
- return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
+ return self.log(loglevel, msg, *args, **kwargs)
def plain(self, msg, *args, **kwargs):
return self.log(logging.INFO + 1, msg, *args, **kwargs)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index 33c0e2fa19..d1b0e929d4 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -138,7 +138,7 @@ class BBLogFilterStdOut(BBLogFilter):
loggerDefaultDebugLevel = 0
loggerDefaultVerbose = False
loggerVerboseLogs = False
-loggerDefaultDomains = []
+loggerDefaultDomains = {}
def init_msgconfig(verbose, debug, debug_domains=None):
"""
@@ -148,15 +148,16 @@ def init_msgconfig(verbose, debug, debug_domains=None):
bb.msg.loggerDefaultVerbose = verbose
if verbose:
bb.msg.loggerVerboseLogs = True
+
+ bb.msg.loggerDefaultDomains = {}
if debug_domains:
- bb.msg.loggerDefaultDomains = debug_domains
- else:
- bb.msg.loggerDefaultDomains = []
+ for (domainarg, iterator) in groupby(debug_domains):
+ dlevel = len(tuple(iterator))
+ bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
def constructLogOptions():
debug = loggerDefaultDebugLevel
verbose = loggerDefaultVerbose
- domains = loggerDefaultDomains
if debug:
level = BBLogFormatter.DEBUG - debug + 1
@@ -165,11 +166,7 @@ def constructLogOptions():
else:
level = BBLogFormatter.NOTE
- debug_domains = {}
- for (domainarg, iterator) in groupby(domains):
- dlevel = len(tuple(iterator))
- debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
- return level, debug_domains
+ return level, loggerDefaultDomains
def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 01/15] bitbake: lib/bb/msg.py: Convert default domains to a dictionary
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Converts the default domain variable to a dictionary where the keys are
the logging domains and the values are the logging level (instead of the
debug count). This makes it easier to deal with the logging domains and
the awkward conversion from a list to a dictionary only needs to be done
once when logging is initialized. Finally, other code has been written
that already assumes this variable is a dictionary, see:
f04cd93109 ("bitbake: lib/bb: Optimise out debug messages from cooker")
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/__init__.py | 5 +++--
bitbake/lib/bb/msg.py | 17 +++++++----------
2 files changed, 10 insertions(+), 12 deletions(-)
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index 88641e280b..acd4af13a8 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -43,12 +43,13 @@ class BBLogger(Logger):
Logger.__init__(self, name)
def bbdebug(self, level, msg, *args, **kwargs):
+ loglevel = logging.DEBUG - level + 1
if not bb.event.worker_pid:
- if self.name in bb.msg.loggerDefaultDomains and level > (bb.msg.loggerDefaultDomains[self.name]):
+ if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
return
if level > (bb.msg.loggerDefaultDebugLevel):
return
- return self.log(logging.DEBUG - level + 1, msg, *args, **kwargs)
+ return self.log(loglevel, msg, *args, **kwargs)
def plain(self, msg, *args, **kwargs):
return self.log(logging.INFO + 1, msg, *args, **kwargs)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index 33c0e2fa19..d1b0e929d4 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -138,7 +138,7 @@ class BBLogFilterStdOut(BBLogFilter):
loggerDefaultDebugLevel = 0
loggerDefaultVerbose = False
loggerVerboseLogs = False
-loggerDefaultDomains = []
+loggerDefaultDomains = {}
def init_msgconfig(verbose, debug, debug_domains=None):
"""
@@ -148,15 +148,16 @@ def init_msgconfig(verbose, debug, debug_domains=None):
bb.msg.loggerDefaultVerbose = verbose
if verbose:
bb.msg.loggerVerboseLogs = True
+
+ bb.msg.loggerDefaultDomains = {}
if debug_domains:
- bb.msg.loggerDefaultDomains = debug_domains
- else:
- bb.msg.loggerDefaultDomains = []
+ for (domainarg, iterator) in groupby(debug_domains):
+ dlevel = len(tuple(iterator))
+ bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
def constructLogOptions():
debug = loggerDefaultDebugLevel
verbose = loggerDefaultVerbose
- domains = loggerDefaultDomains
if debug:
level = BBLogFormatter.DEBUG - debug + 1
@@ -165,11 +166,7 @@ def constructLogOptions():
else:
level = BBLogFormatter.NOTE
- debug_domains = {}
- for (domainarg, iterator) in groupby(domains):
- dlevel = len(tuple(iterator))
- debug_domains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
- return level, debug_domains
+ return level, loggerDefaultDomains
def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 02/15] bitbake: knotty: Handle logging messages with specific logger
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index cbb289b05f..a6a92b920f 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -522,7 +522,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn'):
event.msg = event.fn + ': ' + event.msg
- logger.handle(event)
+ logging.getLogger(event.name).handle(event)
continue
if isinstance(event, bb.build.TaskFailedSilent):
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 02/15] bitbake: knotty: Handle logging messages with specific logger
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Handles the log messages from the bitbake server with the specific
logger that the event originated from. This allows hierarchical logging
configurations to work as expected.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index cbb289b05f..a6a92b920f 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -522,7 +522,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn'):
event.msg = event.fn + ': ' + event.msg
- logger.handle(event)
+ logging.getLogger(event.name).handle(event)
continue
if isinstance(event, bb.build.TaskFailedSilent):
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 03/15] bitbake: lib/bb/msg.py: Use log level instead of debug count
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/bin/bitbake-worker | 2 +-
bitbake/lib/bb/__init__.py | 2 +-
bitbake/lib/bb/msg.py | 22 +++++++++-------------
bitbake/lib/bb/runqueue.py | 2 +-
4 files changed, 12 insertions(+), 16 deletions(-)
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index 1e641e81c2..fc28a34d8a 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -414,7 +414,7 @@ class BitbakeWorker(object):
def handle_workerdata(self, data):
self.workerdata = pickle.loads(data)
- bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"]
+ bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"]
bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"]
bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"]
bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index acd4af13a8..dc5e91e29a 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -47,7 +47,7 @@ class BBLogger(Logger):
if not bb.event.worker_pid:
if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
return
- if level > (bb.msg.loggerDefaultDebugLevel):
+ if loglevel > bb.msg.loggerDefaultLogLevel:
return
return self.log(loglevel, msg, *args, **kwargs)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index d1b0e929d4..ea6a9543f7 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -135,7 +135,7 @@ class BBLogFilterStdOut(BBLogFilter):
# Message control functions
#
-loggerDefaultDebugLevel = 0
+loggerDefaultLogLevel = BBLogFormatter.NOTE
loggerDefaultVerbose = False
loggerVerboseLogs = False
loggerDefaultDomains = {}
@@ -144,11 +144,17 @@ def init_msgconfig(verbose, debug, debug_domains=None):
"""
Set default verbosity and debug levels config the logger
"""
- bb.msg.loggerDefaultDebugLevel = debug
bb.msg.loggerDefaultVerbose = verbose
if verbose:
bb.msg.loggerVerboseLogs = True
+ if debug:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.DEBUG - debug + 1
+ elif verbose:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.VERBOSE
+ else:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.NOTE
+
bb.msg.loggerDefaultDomains = {}
if debug_domains:
for (domainarg, iterator) in groupby(debug_domains):
@@ -156,17 +162,7 @@ def init_msgconfig(verbose, debug, debug_domains=None):
bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
def constructLogOptions():
- debug = loggerDefaultDebugLevel
- verbose = loggerDefaultVerbose
-
- if debug:
- level = BBLogFormatter.DEBUG - debug + 1
- elif verbose:
- level = BBLogFormatter.VERBOSE
- else:
- level = BBLogFormatter.NOTE
-
- return level, loggerDefaultDomains
+ return loggerDefaultLogLevel, loggerDefaultDomains
def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 32966b4f7c..4106fa4bc4 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1254,7 +1254,7 @@ class RunQueue:
"fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
"fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv,
"sigdata" : bb.parse.siggen.get_taskdata(),
- "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
+ "logdefaultlevel" : bb.msg.loggerDefaultLogLevel,
"logdefaultverbose" : bb.msg.loggerDefaultVerbose,
"logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
"logdefaultdomain" : bb.msg.loggerDefaultDomains,
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 03/15] bitbake: lib/bb/msg.py: Use log level instead of debug count
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Passes around the actual logging level as the default log level variable
instead of the debug count. This makes it easier to deal with logging
levels since the conversion from debug count and verbose flag only has
to occur once when logging is initialized and after that actual log
levels can be used
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/bin/bitbake-worker | 2 +-
bitbake/lib/bb/__init__.py | 2 +-
bitbake/lib/bb/msg.py | 22 +++++++++-------------
bitbake/lib/bb/runqueue.py | 2 +-
4 files changed, 12 insertions(+), 16 deletions(-)
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index 1e641e81c2..fc28a34d8a 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -414,7 +414,7 @@ class BitbakeWorker(object):
def handle_workerdata(self, data):
self.workerdata = pickle.loads(data)
- bb.msg.loggerDefaultDebugLevel = self.workerdata["logdefaultdebug"]
+ bb.msg.loggerDefaultLogLevel = self.workerdata["logdefaultlevel"]
bb.msg.loggerDefaultVerbose = self.workerdata["logdefaultverbose"]
bb.msg.loggerVerboseLogs = self.workerdata["logdefaultverboselogs"]
bb.msg.loggerDefaultDomains = self.workerdata["logdefaultdomain"]
diff --git a/bitbake/lib/bb/__init__.py b/bitbake/lib/bb/__init__.py
index acd4af13a8..dc5e91e29a 100644
--- a/bitbake/lib/bb/__init__.py
+++ b/bitbake/lib/bb/__init__.py
@@ -47,7 +47,7 @@ class BBLogger(Logger):
if not bb.event.worker_pid:
if self.name in bb.msg.loggerDefaultDomains and loglevel > (bb.msg.loggerDefaultDomains[self.name]):
return
- if level > (bb.msg.loggerDefaultDebugLevel):
+ if loglevel > bb.msg.loggerDefaultLogLevel:
return
return self.log(loglevel, msg, *args, **kwargs)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index d1b0e929d4..ea6a9543f7 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -135,7 +135,7 @@ class BBLogFilterStdOut(BBLogFilter):
# Message control functions
#
-loggerDefaultDebugLevel = 0
+loggerDefaultLogLevel = BBLogFormatter.NOTE
loggerDefaultVerbose = False
loggerVerboseLogs = False
loggerDefaultDomains = {}
@@ -144,11 +144,17 @@ def init_msgconfig(verbose, debug, debug_domains=None):
"""
Set default verbosity and debug levels config the logger
"""
- bb.msg.loggerDefaultDebugLevel = debug
bb.msg.loggerDefaultVerbose = verbose
if verbose:
bb.msg.loggerVerboseLogs = True
+ if debug:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.DEBUG - debug + 1
+ elif verbose:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.VERBOSE
+ else:
+ bb.msg.loggerDefaultLogLevel = BBLogFormatter.NOTE
+
bb.msg.loggerDefaultDomains = {}
if debug_domains:
for (domainarg, iterator) in groupby(debug_domains):
@@ -156,17 +162,7 @@ def init_msgconfig(verbose, debug, debug_domains=None):
bb.msg.loggerDefaultDomains["BitBake.%s" % domainarg] = logging.DEBUG - dlevel + 1
def constructLogOptions():
- debug = loggerDefaultDebugLevel
- verbose = loggerDefaultVerbose
-
- if debug:
- level = BBLogFormatter.DEBUG - debug + 1
- elif verbose:
- level = BBLogFormatter.VERBOSE
- else:
- level = BBLogFormatter.NOTE
-
- return level, loggerDefaultDomains
+ return loggerDefaultLogLevel, loggerDefaultDomains
def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
level, debug_domains = constructLogOptions()
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 32966b4f7c..4106fa4bc4 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1254,7 +1254,7 @@ class RunQueue:
"fakerootdirs" : self.rqdata.dataCaches[mc].fakerootdirs,
"fakerootnoenv" : self.rqdata.dataCaches[mc].fakerootnoenv,
"sigdata" : bb.parse.siggen.get_taskdata(),
- "logdefaultdebug" : bb.msg.loggerDefaultDebugLevel,
+ "logdefaultlevel" : bb.msg.loggerDefaultLogLevel,
"logdefaultverbose" : bb.msg.loggerDefaultVerbose,
"logdefaultverboselogs" : bb.msg.loggerVerboseLogs,
"logdefaultdomain" : bb.msg.loggerDefaultDomains,
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 04/15] bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds a __repr__ function for BBLogFormatter. This allows it to get a
human readable string when printed using the logging_tree module
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index ea6a9543f7..c70fd80806 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -98,6 +98,9 @@ class BBLogFormatter(logging.Formatter):
def enable_color(self):
self.color_enabled = True
+ def __repr__(self):
+ return "%s fmt='%s' color=%s" % (self.__class__.__name__, self._fmt, "True" if self.color_enabled else "False")
+
class BBLogFilter(object):
def __init__(self, handler, level, debug_domains):
self.stdlevel = level
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 04/15] bitbake: lib/bb/msg.py: Add repr for BBLogFormatter
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds a __repr__ function for BBLogFormatter. This allows it to get a
human readable string when printed using the logging_tree module
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index ea6a9543f7..c70fd80806 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -98,6 +98,9 @@ class BBLogFormatter(logging.Formatter):
def enable_color(self):
self.color_enabled = True
+ def __repr__(self):
+ return "%s fmt='%s' color=%s" % (self.__class__.__name__, self._fmt, "True" if self.color_enabled else "False")
+
class BBLogFilter(object):
def __init__(self, handler, level, debug_domains):
self.stdlevel = level
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 05/15] bitbake: knotty: Add commented logging_tree code
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index a6a92b920f..aac12cd479 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -430,6 +430,11 @@ def main(server, eventHandler, params, tf = TerminalFilter):
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
+ # The logging_tree module is *extremely* helpful in debugging logging
+ # domains. Uncomment here to dump the logging tree when bitbake starts
+ #import logging_tree
+ #logging_tree.printout()
+
universe = False
if not params.observe_only:
params.updateFromServer(server)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 05/15] bitbake: knotty: Add commented logging_tree code
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds a comment section that can be easily uncommented to enable dumping
the logging tree. This module is extremely useful for debugging issued
with logging configuration
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index a6a92b920f..aac12cd479 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -430,6 +430,11 @@ def main(server, eventHandler, params, tf = TerminalFilter):
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
+ # The logging_tree module is *extremely* helpful in debugging logging
+ # domains. Uncomment here to dump the logging tree when bitbake starts
+ #import logging_tree
+ #logging_tree.printout()
+
universe = False
if not params.observe_only:
params.updateFromServer(server)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 06/15] bitbake: lib/bb/msg.py: Add filter utilities
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds generic filter utilities that can be used to filter when the log
level is greater than or equal to a defined value, or below a defined
value, as well as a generic function to translate a string to a logging
level (or bitbake logging level)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index c70fd80806..8561826a6e 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -135,6 +135,28 @@ class BBLogFilterStdOut(BBLogFilter):
return True
return False
+class LogFilterGEQLevel(logging.Filter):
+ def __init__(self, level):
+ self.strlevel = str(level)
+ self.level = stringToLevel(level)
+
+ def __repr__(self):
+ return "%s level >= %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+ def filter(self, record):
+ return (record.levelno >= self.level)
+
+class LogFilterLTLevel(logging.Filter):
+ def __init__(self, level):
+ self.strlevel = str(level)
+ self.level = stringToLevel(level)
+
+ def __repr__(self):
+ return "%s level < %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+ def filter(self, record):
+ return (record.levelno < self.level)
+
# Message control functions
#
@@ -175,6 +197,19 @@ def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
cls(handler, level, debug_domains)
+def stringToLevel(level):
+ try:
+ return int(level)
+ except ValueError:
+ pass
+
+ try:
+ return getattr(logging, level)
+ except AttributeError:
+ pass
+
+ return getattr(BBLogFormatter, level)
+
#
# Message handling functions
#
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 06/15] bitbake: lib/bb/msg.py: Add filter utilities
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds generic filter utilities that can be used to filter when the log
level is greater than or equal to a defined value, or below a defined
value, as well as a generic function to translate a string to a logging
level (or bitbake logging level)
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index c70fd80806..8561826a6e 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -135,6 +135,28 @@ class BBLogFilterStdOut(BBLogFilter):
return True
return False
+class LogFilterGEQLevel(logging.Filter):
+ def __init__(self, level):
+ self.strlevel = str(level)
+ self.level = stringToLevel(level)
+
+ def __repr__(self):
+ return "%s level >= %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+ def filter(self, record):
+ return (record.levelno >= self.level)
+
+class LogFilterLTLevel(logging.Filter):
+ def __init__(self, level):
+ self.strlevel = str(level)
+ self.level = stringToLevel(level)
+
+ def __repr__(self):
+ return "%s level < %s (%d)" % (self.__class__.__name__, self.strlevel, self.level)
+
+ def filter(self, record):
+ return (record.levelno < self.level)
+
# Message control functions
#
@@ -175,6 +197,19 @@ def addDefaultlogFilter(handler, cls = BBLogFilter, forcelevel=None):
cls(handler, level, debug_domains)
+def stringToLevel(level):
+ try:
+ return int(level)
+ except ValueError:
+ pass
+
+ try:
+ return getattr(logging, level)
+ except AttributeError:
+ pass
+
+ return getattr(BBLogFormatter, level)
+
#
# Message handling functions
#
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 07/15] bitbake: lib/bb/msg.py: Remove unused filters
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Now that the filter are described using the python logging structure,
these classes are no longer needed.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 16 ----------------
1 file changed, 16 deletions(-)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index 8561826a6e..cab079e333 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -119,22 +119,6 @@ class BBLogFilter(object):
return True
return False
-class BBLogFilterStdErr(BBLogFilter):
- def filter(self, record):
- if not BBLogFilter.filter(self, record):
- return False
- if record.levelno >= logging.ERROR:
- return True
- return False
-
-class BBLogFilterStdOut(BBLogFilter):
- def filter(self, record):
- if not BBLogFilter.filter(self, record):
- return False
- if record.levelno < logging.ERROR:
- return True
- return False
-
class LogFilterGEQLevel(logging.Filter):
def __init__(self, level):
self.strlevel = str(level)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 07/15] bitbake: lib/bb/msg.py: Remove unused filters
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Now that the filter are described using the python logging structure,
these classes are no longer needed.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 16 ----------------
1 file changed, 16 deletions(-)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index 8561826a6e..cab079e333 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -119,22 +119,6 @@ class BBLogFilter(object):
return True
return False
-class BBLogFilterStdErr(BBLogFilter):
- def filter(self, record):
- if not BBLogFilter.filter(self, record):
- return False
- if record.levelno >= logging.ERROR:
- return True
- return False
-
-class BBLogFilterStdOut(BBLogFilter):
- def filter(self, record):
- if not BBLogFilter.filter(self, record):
- return False
- if record.levelno < logging.ERROR:
- return True
- return False
-
class LogFilterGEQLevel(logging.Filter):
def __init__(self, level):
self.strlevel = str(level)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 08/15] bitbake: lib/bb/msg.py: Add helper to set logging config
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds a helper function to setup the structure logging information in
bitbake. The helper function takes a default configuration and an
optional path to a user config file that can be merged into the default.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 93 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 93 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index cab079e333..6259af037f 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -13,6 +13,7 @@ Message handling infrastructure for bitbake
import sys
import copy
import logging
+import logging.config
from itertools import groupby
import bb
import bb.event
@@ -227,3 +228,95 @@ def has_console_handler(logger):
if handler.stream in [sys.stderr, sys.stdout]:
return True
return False
+
+def setLoggingConfig(defaultconfig, userconfigfile=None):
+ logconfig = copy.deepcopy(defaultconfig)
+
+ if userconfigfile:
+ with open(userconfigfile, 'r') as f:
+ if userconfigfile.endswith('.yml') or userconfigfile.endswith('.yaml'):
+ import yaml
+ userconfig = yaml.load(f)
+ elif userconfigfile.endswith('.json') or userconfigfile.endswith('.cfg'):
+ import json
+ userconfig = json.load(f)
+ else:
+ raise BaseException("Unrecognized file format: %s" % userconfigfile)
+
+ if userconfig.get('bitbake_merge', True):
+ # Merge config with the default config
+ if userconfig.get('version') != logconfig['version']:
+ raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version')))
+
+ # Set some defaults to make merging easier
+ userconfig.setdefault("loggers", {})
+
+ # If a handler, formatter, or filter is defined in the user
+ # config, it will replace an existing one in the default config
+ for k in ("handlers", "formatters", "filters"):
+ logconfig.setdefault(k, {}).update(userconfig.get(k, {}))
+
+ seen_loggers = set()
+ for name, l in logconfig["loggers"].items():
+ # If the merge option is set, merge the handlers and
+ # filters. Otherwise, if it is False, this logger won't get
+ # add to the set of seen loggers and will replace the
+ # existing one
+ if l.get('bitbake_merge', True):
+ ulogger = userconfig["loggers"].setdefault(name, {})
+ ulogger.setdefault("handlers", [])
+ ulogger.setdefault("filters", [])
+
+ # Merge lists
+ l.setdefault("handlers", []).extend(ulogger["handlers"])
+ l.setdefault("filters", []).extend(ulogger["filters"])
+
+ # Replace other properties if present
+ if "level" in ulogger:
+ l["level"] = ulogger["level"]
+
+ if "propagate" in ulogger:
+ l["propagate"] = ulogger["propagate"]
+
+ seen_loggers.add(name)
+
+ # Add all loggers present in the user config, but not any that
+ # have already been processed
+ for name in set(userconfig["loggers"].keys()) - seen_loggers:
+ logconfig["loggers"][name] = userconfig["loggers"][name]
+
+ else:
+ # Replace the entire default config
+ logconfig = userconfig
+
+ # Convert all level parameters to integers in case users want to use the
+ # bitbake defined level names
+ for h in logconfig["handlers"].values():
+ if "level" in h:
+ h["level"] = bb.msg.stringToLevel(h["level"])
+
+ for l in logconfig["loggers"].values():
+ if "level" in l:
+ l["level"] = bb.msg.stringToLevel(l["level"])
+
+ logging.config.dictConfig(logconfig)
+
+ # The user may have specified logging domains they want at a higher debug
+ # level than the standard.
+ for name, l in logconfig["loggers"].items():
+ if not name.startswith("BitBake."):
+ continue
+
+ if not "level" in l:
+ continue
+
+ curlevel = bb.msg.loggerDefaultDomains.get(name)
+ # Note: level parameter should already be a int because of conversion
+ # above
+ newlevel = int(l["level"])
+ if curlevel is None or newlevel < curlevel:
+ bb.msg.loggerDefaultDomains[name] = newlevel
+
+ # TODO: I don't think that setting the global log level should be necessary
+ #if newlevel < bb.msg.loggerDefaultLogLevel:
+ # bb.msg.loggerDefaultLogLevel = newlevel
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 08/15] bitbake: lib/bb/msg.py: Add helper to set logging config
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds a helper function to setup the structure logging information in
bitbake. The helper function takes a default configuration and an
optional path to a user config file that can be merged into the default.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/msg.py | 93 +++++++++++++++++++++++++++++++++++++++++++
1 file changed, 93 insertions(+)
diff --git a/bitbake/lib/bb/msg.py b/bitbake/lib/bb/msg.py
index cab079e333..6259af037f 100644
--- a/bitbake/lib/bb/msg.py
+++ b/bitbake/lib/bb/msg.py
@@ -13,6 +13,7 @@ Message handling infrastructure for bitbake
import sys
import copy
import logging
+import logging.config
from itertools import groupby
import bb
import bb.event
@@ -227,3 +228,95 @@ def has_console_handler(logger):
if handler.stream in [sys.stderr, sys.stdout]:
return True
return False
+
+def setLoggingConfig(defaultconfig, userconfigfile=None):
+ logconfig = copy.deepcopy(defaultconfig)
+
+ if userconfigfile:
+ with open(userconfigfile, 'r') as f:
+ if userconfigfile.endswith('.yml') or userconfigfile.endswith('.yaml'):
+ import yaml
+ userconfig = yaml.load(f)
+ elif userconfigfile.endswith('.json') or userconfigfile.endswith('.cfg'):
+ import json
+ userconfig = json.load(f)
+ else:
+ raise BaseException("Unrecognized file format: %s" % userconfigfile)
+
+ if userconfig.get('bitbake_merge', True):
+ # Merge config with the default config
+ if userconfig.get('version') != logconfig['version']:
+ raise BaseException("Bad user configuration version. Expected %r, got %r" % (logconfig['version'], userconfig.get('version')))
+
+ # Set some defaults to make merging easier
+ userconfig.setdefault("loggers", {})
+
+ # If a handler, formatter, or filter is defined in the user
+ # config, it will replace an existing one in the default config
+ for k in ("handlers", "formatters", "filters"):
+ logconfig.setdefault(k, {}).update(userconfig.get(k, {}))
+
+ seen_loggers = set()
+ for name, l in logconfig["loggers"].items():
+ # If the merge option is set, merge the handlers and
+ # filters. Otherwise, if it is False, this logger won't get
+ # add to the set of seen loggers and will replace the
+ # existing one
+ if l.get('bitbake_merge', True):
+ ulogger = userconfig["loggers"].setdefault(name, {})
+ ulogger.setdefault("handlers", [])
+ ulogger.setdefault("filters", [])
+
+ # Merge lists
+ l.setdefault("handlers", []).extend(ulogger["handlers"])
+ l.setdefault("filters", []).extend(ulogger["filters"])
+
+ # Replace other properties if present
+ if "level" in ulogger:
+ l["level"] = ulogger["level"]
+
+ if "propagate" in ulogger:
+ l["propagate"] = ulogger["propagate"]
+
+ seen_loggers.add(name)
+
+ # Add all loggers present in the user config, but not any that
+ # have already been processed
+ for name in set(userconfig["loggers"].keys()) - seen_loggers:
+ logconfig["loggers"][name] = userconfig["loggers"][name]
+
+ else:
+ # Replace the entire default config
+ logconfig = userconfig
+
+ # Convert all level parameters to integers in case users want to use the
+ # bitbake defined level names
+ for h in logconfig["handlers"].values():
+ if "level" in h:
+ h["level"] = bb.msg.stringToLevel(h["level"])
+
+ for l in logconfig["loggers"].values():
+ if "level" in l:
+ l["level"] = bb.msg.stringToLevel(l["level"])
+
+ logging.config.dictConfig(logconfig)
+
+ # The user may have specified logging domains they want at a higher debug
+ # level than the standard.
+ for name, l in logconfig["loggers"].items():
+ if not name.startswith("BitBake."):
+ continue
+
+ if not "level" in l:
+ continue
+
+ curlevel = bb.msg.loggerDefaultDomains.get(name)
+ # Note: level parameter should already be a int because of conversion
+ # above
+ newlevel = int(l["level"])
+ if curlevel is None or newlevel < curlevel:
+ bb.msg.loggerDefaultDomains[name] = newlevel
+
+ # TODO: I don't think that setting the global log level should be necessary
+ #if newlevel < bb.msg.loggerDefaultLogLevel:
+ # bb.msg.loggerDefaultLogLevel = newlevel
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 09/15] bitbake: knotty: Remove dependency on format variable
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/tinfoil.py | 4 +---
bitbake/lib/bb/ui/knotty.py | 32 ++++++++++++++++++++------------
2 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
index 9560eb5b49..5c5be456e2 100644
--- a/bitbake/lib/bb/tinfoil.py
+++ b/bitbake/lib/bb/tinfoil.py
@@ -735,11 +735,9 @@ class Tinfoil:
console = handler
elif handler.stream == sys.stderr:
errconsole = handler
- format_str = "%(levelname)s: %(message)s"
- format = bb.msg.BBLogFormatter(format_str)
helper.shutdown = 0
parseprogress = None
- termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet)
+ termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, quiet=self.quiet)
try:
while True:
try:
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index aac12cd479..d5dce7172a 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -109,12 +109,11 @@ def pluralise(singular, plural, qty):
class InteractConsoleLogFilter(logging.Filter):
- def __init__(self, tf, format):
+ def __init__(self, tf):
self.tf = tf
- self.format = format
def filter(self, record):
- if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
return False
self.tf.clearFooter()
return True
@@ -150,7 +149,7 @@ class TerminalFilter(object):
cr = (25, 80)
return cr
- def __init__(self, main, helper, console, errconsole, format, quiet):
+ def __init__(self, main, helper, console, errconsole, quiet):
self.main = main
self.helper = helper
self.cuu = None
@@ -180,7 +179,16 @@ class TerminalFilter(object):
termios.tcsetattr(fd, termios.TCSADRAIN, new)
curses.setupterm()
if curses.tigetnum("colors") > 2:
- format.enable_color()
+ if console:
+ try:
+ console.formatter.enable_color()
+ except AttributeError:
+ pass
+ if errconsole:
+ try:
+ errconsole.formatter.enable_color()
+ except AttributeError:
+ pass
self.ed = curses.tigetstr("ed")
if self.ed:
self.cuu = curses.tigetstr("cuu")
@@ -197,9 +205,9 @@ class TerminalFilter(object):
bb.note("Unable to use interactive mode for this terminal, using fallback")
return
if console:
- console.addFilter(InteractConsoleLogFilter(self, format))
+ console.addFilter(InteractConsoleLogFilter(self))
if errconsole:
- errconsole.addFilter(InteractConsoleLogFilter(self, format))
+ errconsole.addFilter(InteractConsoleLogFilter(self))
self.main_progress = None
@@ -469,7 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
printinterval = 5000
lastprint = time.time()
- termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
+ termfilter = tf(main, helper, console, errconsole, params.options.quiet)
atexit.register(termfilter.finish)
while True:
@@ -508,21 +516,21 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, logging.LogRecord):
lastprint = time.time()
printinterval = 5000
- if event.levelno >= format.ERROR:
+ if event.levelno >= bb.msg.BBLogFormatter.ERROR:
errors = errors + 1
return_value = 1
- elif event.levelno == format.WARNING:
+ elif event.levelno == bb.msg.BBLogFormatter.WARNING:
warnings = warnings + 1
if event.taskpid != 0:
# For "normal" logging conditions, don't show note logs from tasks
# but do show them if the user has changed the default log level to
# include verbose/debug messages
- if event.levelno <= format.NOTE and (event.levelno < llevel or (event.levelno == format.NOTE and llevel != format.VERBOSE)):
+ if event.levelno <= bb.msg.BBLogFormatter.NOTE and (event.levelno < llevel or (event.levelno == bb.msg.BBLogFormatter.NOTE and llevel != bb.msg.BBLogFormatter.VERBOSE)):
continue
# Prefix task messages with recipe/task
- if event.taskpid in helper.pidmap and event.levelno != format.PLAIN:
+ if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN:
taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]]
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn'):
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 09/15] bitbake: knotty: Remove dependency on format variable
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Passing around the log formatter variable was unnecessary since the log
levels of interest can be accesses as class members of
bb.msg.BBLogFormatter. Switching to do this will make using the
structured python logging much easier, since it can be difficult to
extract out the formatter for a specific handler.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/tinfoil.py | 4 +---
bitbake/lib/bb/ui/knotty.py | 32 ++++++++++++++++++++------------
2 files changed, 21 insertions(+), 15 deletions(-)
diff --git a/bitbake/lib/bb/tinfoil.py b/bitbake/lib/bb/tinfoil.py
index 9560eb5b49..5c5be456e2 100644
--- a/bitbake/lib/bb/tinfoil.py
+++ b/bitbake/lib/bb/tinfoil.py
@@ -735,11 +735,9 @@ class Tinfoil:
console = handler
elif handler.stream == sys.stderr:
errconsole = handler
- format_str = "%(levelname)s: %(message)s"
- format = bb.msg.BBLogFormatter(format_str)
helper.shutdown = 0
parseprogress = None
- termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, format, quiet=self.quiet)
+ termfilter = bb.ui.knotty.TerminalFilter(helper, helper, console, errconsole, quiet=self.quiet)
try:
while True:
try:
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index aac12cd479..d5dce7172a 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -109,12 +109,11 @@ def pluralise(singular, plural, qty):
class InteractConsoleLogFilter(logging.Filter):
- def __init__(self, tf, format):
+ def __init__(self, tf):
self.tf = tf
- self.format = format
def filter(self, record):
- if record.levelno == self.format.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
+ if record.levelno == bb.msg.BBLogFormatter.NOTE and (record.msg.startswith("Running") or record.msg.startswith("recipe ")):
return False
self.tf.clearFooter()
return True
@@ -150,7 +149,7 @@ class TerminalFilter(object):
cr = (25, 80)
return cr
- def __init__(self, main, helper, console, errconsole, format, quiet):
+ def __init__(self, main, helper, console, errconsole, quiet):
self.main = main
self.helper = helper
self.cuu = None
@@ -180,7 +179,16 @@ class TerminalFilter(object):
termios.tcsetattr(fd, termios.TCSADRAIN, new)
curses.setupterm()
if curses.tigetnum("colors") > 2:
- format.enable_color()
+ if console:
+ try:
+ console.formatter.enable_color()
+ except AttributeError:
+ pass
+ if errconsole:
+ try:
+ errconsole.formatter.enable_color()
+ except AttributeError:
+ pass
self.ed = curses.tigetstr("ed")
if self.ed:
self.cuu = curses.tigetstr("cuu")
@@ -197,9 +205,9 @@ class TerminalFilter(object):
bb.note("Unable to use interactive mode for this terminal, using fallback")
return
if console:
- console.addFilter(InteractConsoleLogFilter(self, format))
+ console.addFilter(InteractConsoleLogFilter(self))
if errconsole:
- errconsole.addFilter(InteractConsoleLogFilter(self, format))
+ errconsole.addFilter(InteractConsoleLogFilter(self))
self.main_progress = None
@@ -469,7 +477,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
printinterval = 5000
lastprint = time.time()
- termfilter = tf(main, helper, console, errconsole, format, params.options.quiet)
+ termfilter = tf(main, helper, console, errconsole, params.options.quiet)
atexit.register(termfilter.finish)
while True:
@@ -508,21 +516,21 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, logging.LogRecord):
lastprint = time.time()
printinterval = 5000
- if event.levelno >= format.ERROR:
+ if event.levelno >= bb.msg.BBLogFormatter.ERROR:
errors = errors + 1
return_value = 1
- elif event.levelno == format.WARNING:
+ elif event.levelno == bb.msg.BBLogFormatter.WARNING:
warnings = warnings + 1
if event.taskpid != 0:
# For "normal" logging conditions, don't show note logs from tasks
# but do show them if the user has changed the default log level to
# include verbose/debug messages
- if event.levelno <= format.NOTE and (event.levelno < llevel or (event.levelno == format.NOTE and llevel != format.VERBOSE)):
+ if event.levelno <= bb.msg.BBLogFormatter.NOTE and (event.levelno < llevel or (event.levelno == bb.msg.BBLogFormatter.NOTE and llevel != bb.msg.BBLogFormatter.VERBOSE)):
continue
# Prefix task messages with recipe/task
- if event.taskpid in helper.pidmap and event.levelno != format.PLAIN:
+ if event.taskpid in helper.pidmap and event.levelno != bb.msg.BBLogFormatter.PLAIN:
taskinfo = helper.running_tasks[helper.pidmap[event.taskpid]]
event.msg = taskinfo['title'] + ': ' + event.msg
if hasattr(event, 'fn'):
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 10/15] bitbake: knotty: Setup logs with config helper
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 134 ++++++++++++++++++++++++++----------
1 file changed, 96 insertions(+), 38 deletions(-)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index d5dce7172a..386f27876d 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -370,7 +370,11 @@ def _log_settings_from_server(server, observe_only):
if error:
logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
raise BaseException(error)
- return includelogs, loglines, consolelogfile
+ logconfigfile, error = server.runCommand([cmd, "BB_LOGCONFIG"])
+ if error:
+ logger.error("Unable to get the value of BB_LOGCONFIG variable: %s" % error)
+ raise BaseException(error)
+ return includelogs, loglines, consolelogfile, logconfigfile
_evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.LogRecord",
"bb.build.TaskFailed", "bb.build.TaskBase", "bb.event.ParseStarted",
@@ -387,7 +391,87 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if not params.observe_only:
params.updateToServer(server, os.environ.copy())
- includelogs, loglines, consolelogfile = _log_settings_from_server(server, params.observe_only)
+ includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only)
+
+ loglevel, _ = bb.msg.constructLogOptions()
+
+ if params.options.quiet == 0:
+ console_loglevel = loglevel
+ elif params.options.quiet > 2:
+ console_loglevel = bb.msg.BBLogFormatter.ERROR
+ else:
+ console_loglevel = bb.msg.BBLogFormatter.WARNING
+
+ logconfig = {
+ "version": 1,
+ "handlers": {
+ "BitBake.console": {
+ "class": "logging.StreamHandler",
+ "formatter": "BitBake.consoleFormatter",
+ "level": console_loglevel,
+ "stream": "ext://sys.stdout",
+ "filters": ["BitBake.stdoutFilter"],
+ },
+ "BitBake.errconsole": {
+ "class": "logging.StreamHandler",
+ "formatter": "BitBake.consoleFormatter",
+ "level": loglevel,
+ "stream": "ext://sys.stderr",
+ "filters": ["BitBake.stderrFilter"],
+ },
+ },
+ "formatters": {
+ # This format instance will get color output enabled by the
+ # terminal
+ "BitBake.consoleFormatter" : {
+ "()": "bb.msg.BBLogFormatter",
+ "format": "%(levelname)s: %(message)s"
+ },
+ # The file log requires a separate instance so that it doesn't get
+ # color enabled
+ "BitBake.logfileFormatter": {
+ "()": "bb.msg.BBLogFormatter",
+ "format": "%(levelname)s: %(message)s"
+ }
+ },
+ "filters": {
+ "BitBake.stdoutFilter": {
+ "()": "bb.msg.LogFilterLTLevel",
+ "level": "ERROR"
+ },
+ "BitBake.stderrFilter": {
+ "()": "bb.msg.LogFilterGEQLevel",
+ "level": "ERROR"
+ }
+ },
+ "loggers": {
+ "BitBake": {
+ "level": loglevel,
+ "handlers": ["BitBake.console", "BitBake.errconsole"],
+ }
+ },
+ "disable_existing_loggers": False
+ }
+
+ # Enable the console log file if enabled
+ if consolelogfile and not params.options.show_environment and not params.options.show_versions:
+ logconfig["handlers"]["BitBake.consolelog"] ={
+ "class": "logging.FileHandler",
+ "formatter": "BitBake.logfileFormatter",
+ "level": "INFO",
+ "filename": consolelogfile,
+ }
+ logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog")
+
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+ bb.utils.remove(loglink)
+ try:
+ os.symlink(os.path.basename(consolelogfile), loglink)
+ except OSError:
+ pass
+
+ bb.msg.setLoggingConfig(logconfig, logconfigfile)
if sys.stdin.isatty() and sys.stdout.isatty():
log_exec_tty = True
@@ -396,23 +480,16 @@ def main(server, eventHandler, params, tf = TerminalFilter):
helper = uihelper.BBUIHelper()
- console = logging.StreamHandler(sys.stdout)
- errconsole = logging.StreamHandler(sys.stderr)
- format_str = "%(levelname)s: %(message)s"
- format = bb.msg.BBLogFormatter(format_str)
- if params.options.quiet == 0:
- forcelevel = None
- elif params.options.quiet > 2:
- forcelevel = bb.msg.BBLogFormatter.ERROR
- else:
- forcelevel = bb.msg.BBLogFormatter.WARNING
- bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel)
- bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
- console.setFormatter(format)
- errconsole.setFormatter(format)
- if not bb.msg.has_console_handler(logger):
- logger.addHandler(console)
- logger.addHandler(errconsole)
+ # Look for the specially designated handlers which need to be passed to the
+ # terminal handler
+ console = None
+ errconsole = None
+ for h in logger.handlers:
+ name = getattr(h, '_name', None)
+ if name == 'BitBake.console':
+ console = h
+ elif name == 'BitBake.errconsole':
+ errconsole = h
bb.utils.set_process_name("KnottyUI")
@@ -420,21 +497,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
server.terminateServer()
return
- consolelog = None
- if consolelogfile and not params.options.show_environment and not params.options.show_versions:
- bb.utils.mkdirhier(os.path.dirname(consolelogfile))
- conlogformat = bb.msg.BBLogFormatter(format_str)
- consolelog = logging.FileHandler(consolelogfile)
- bb.msg.addDefaultlogFilter(consolelog)
- consolelog.setFormatter(conlogformat)
- logger.addHandler(consolelog)
- loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
- bb.utils.remove(loglink)
- try:
- os.symlink(os.path.basename(consolelogfile), loglink)
- except OSError:
- pass
-
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
@@ -761,8 +823,4 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if e.errno == errno.EPIPE:
pass
- if consolelog:
- logger.removeHandler(consolelog)
- consolelog.close()
-
return return_value
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 10/15] bitbake: knotty: Setup logs with config helper
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Sets up logging in knotty to use python's structured logging config and
the bb.msg.setLoggingConfig() helper to setup logging. This allows the
user to specify additional logging mechanism in a config file with
BB_LOGCONFIG
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/ui/knotty.py | 134 ++++++++++++++++++++++++++----------
1 file changed, 96 insertions(+), 38 deletions(-)
diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index d5dce7172a..386f27876d 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -370,7 +370,11 @@ def _log_settings_from_server(server, observe_only):
if error:
logger.error("Unable to get the value of BB_CONSOLELOG variable: %s" % error)
raise BaseException(error)
- return includelogs, loglines, consolelogfile
+ logconfigfile, error = server.runCommand([cmd, "BB_LOGCONFIG"])
+ if error:
+ logger.error("Unable to get the value of BB_LOGCONFIG variable: %s" % error)
+ raise BaseException(error)
+ return includelogs, loglines, consolelogfile, logconfigfile
_evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.LogRecord",
"bb.build.TaskFailed", "bb.build.TaskBase", "bb.event.ParseStarted",
@@ -387,7 +391,87 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if not params.observe_only:
params.updateToServer(server, os.environ.copy())
- includelogs, loglines, consolelogfile = _log_settings_from_server(server, params.observe_only)
+ includelogs, loglines, consolelogfile, logconfigfile = _log_settings_from_server(server, params.observe_only)
+
+ loglevel, _ = bb.msg.constructLogOptions()
+
+ if params.options.quiet == 0:
+ console_loglevel = loglevel
+ elif params.options.quiet > 2:
+ console_loglevel = bb.msg.BBLogFormatter.ERROR
+ else:
+ console_loglevel = bb.msg.BBLogFormatter.WARNING
+
+ logconfig = {
+ "version": 1,
+ "handlers": {
+ "BitBake.console": {
+ "class": "logging.StreamHandler",
+ "formatter": "BitBake.consoleFormatter",
+ "level": console_loglevel,
+ "stream": "ext://sys.stdout",
+ "filters": ["BitBake.stdoutFilter"],
+ },
+ "BitBake.errconsole": {
+ "class": "logging.StreamHandler",
+ "formatter": "BitBake.consoleFormatter",
+ "level": loglevel,
+ "stream": "ext://sys.stderr",
+ "filters": ["BitBake.stderrFilter"],
+ },
+ },
+ "formatters": {
+ # This format instance will get color output enabled by the
+ # terminal
+ "BitBake.consoleFormatter" : {
+ "()": "bb.msg.BBLogFormatter",
+ "format": "%(levelname)s: %(message)s"
+ },
+ # The file log requires a separate instance so that it doesn't get
+ # color enabled
+ "BitBake.logfileFormatter": {
+ "()": "bb.msg.BBLogFormatter",
+ "format": "%(levelname)s: %(message)s"
+ }
+ },
+ "filters": {
+ "BitBake.stdoutFilter": {
+ "()": "bb.msg.LogFilterLTLevel",
+ "level": "ERROR"
+ },
+ "BitBake.stderrFilter": {
+ "()": "bb.msg.LogFilterGEQLevel",
+ "level": "ERROR"
+ }
+ },
+ "loggers": {
+ "BitBake": {
+ "level": loglevel,
+ "handlers": ["BitBake.console", "BitBake.errconsole"],
+ }
+ },
+ "disable_existing_loggers": False
+ }
+
+ # Enable the console log file if enabled
+ if consolelogfile and not params.options.show_environment and not params.options.show_versions:
+ logconfig["handlers"]["BitBake.consolelog"] ={
+ "class": "logging.FileHandler",
+ "formatter": "BitBake.logfileFormatter",
+ "level": "INFO",
+ "filename": consolelogfile,
+ }
+ logconfig["loggers"]["BitBake"]["handlers"].append("BitBake.consolelog")
+
+ bb.utils.mkdirhier(os.path.dirname(consolelogfile))
+ loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
+ bb.utils.remove(loglink)
+ try:
+ os.symlink(os.path.basename(consolelogfile), loglink)
+ except OSError:
+ pass
+
+ bb.msg.setLoggingConfig(logconfig, logconfigfile)
if sys.stdin.isatty() and sys.stdout.isatty():
log_exec_tty = True
@@ -396,23 +480,16 @@ def main(server, eventHandler, params, tf = TerminalFilter):
helper = uihelper.BBUIHelper()
- console = logging.StreamHandler(sys.stdout)
- errconsole = logging.StreamHandler(sys.stderr)
- format_str = "%(levelname)s: %(message)s"
- format = bb.msg.BBLogFormatter(format_str)
- if params.options.quiet == 0:
- forcelevel = None
- elif params.options.quiet > 2:
- forcelevel = bb.msg.BBLogFormatter.ERROR
- else:
- forcelevel = bb.msg.BBLogFormatter.WARNING
- bb.msg.addDefaultlogFilter(console, bb.msg.BBLogFilterStdOut, forcelevel)
- bb.msg.addDefaultlogFilter(errconsole, bb.msg.BBLogFilterStdErr)
- console.setFormatter(format)
- errconsole.setFormatter(format)
- if not bb.msg.has_console_handler(logger):
- logger.addHandler(console)
- logger.addHandler(errconsole)
+ # Look for the specially designated handlers which need to be passed to the
+ # terminal handler
+ console = None
+ errconsole = None
+ for h in logger.handlers:
+ name = getattr(h, '_name', None)
+ if name == 'BitBake.console':
+ console = h
+ elif name == 'BitBake.errconsole':
+ errconsole = h
bb.utils.set_process_name("KnottyUI")
@@ -420,21 +497,6 @@ def main(server, eventHandler, params, tf = TerminalFilter):
server.terminateServer()
return
- consolelog = None
- if consolelogfile and not params.options.show_environment and not params.options.show_versions:
- bb.utils.mkdirhier(os.path.dirname(consolelogfile))
- conlogformat = bb.msg.BBLogFormatter(format_str)
- consolelog = logging.FileHandler(consolelogfile)
- bb.msg.addDefaultlogFilter(consolelog)
- consolelog.setFormatter(conlogformat)
- logger.addHandler(consolelog)
- loglink = os.path.join(os.path.dirname(consolelogfile), 'console-latest.log')
- bb.utils.remove(loglink)
- try:
- os.symlink(os.path.basename(consolelogfile), loglink)
- except OSError:
- pass
-
llevel, debug_domains = bb.msg.constructLogOptions()
server.runCommand(["setEventMask", server.getEventHandle(), llevel, debug_domains, _evt_list])
@@ -761,8 +823,4 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if e.errno == errno.EPIPE:
pass
- if consolelog:
- logger.removeHandler(consolelog)
- consolelog.close()
-
return return_value
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 11/15] bitbake: worker: Remove unnecessary addDefaultLogFilter
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adding the default log filter here is unnecessary because there are no
defined logging domains when it is called, which means it does no actual
filtering.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/bin/bitbake-worker | 1 -
1 file changed, 1 deletion(-)
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index fc28a34d8a..97cc0fd60f 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -65,7 +65,6 @@ if 0:
format_str = "%(levelname)s: %(message)s"
conlogformat = bb.msg.BBLogFormatter(format_str)
consolelog = logging.FileHandler(logfilename)
- bb.msg.addDefaultlogFilter(consolelog)
consolelog.setFormatter(conlogformat)
logger.addHandler(consolelog)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 11/15] bitbake: worker: Remove unnecessary addDefaultLogFilter
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adding the default log filter here is unnecessary because there are no
defined logging domains when it is called, which means it does no actual
filtering.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/bin/bitbake-worker | 1 -
1 file changed, 1 deletion(-)
diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index fc28a34d8a..97cc0fd60f 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -65,7 +65,6 @@ if 0:
format_str = "%(levelname)s: %(message)s"
conlogformat = bb.msg.BBLogFormatter(format_str)
consolelog = logging.FileHandler(logfilename)
- bb.msg.addDefaultlogFilter(consolelog)
consolelog.setFormatter(conlogformat)
logger.addHandler(consolelog)
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 12/15] bitbake: Log hash equivalence with a different logger
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/runqueue.py | 11 ++++++-----
bitbake/lib/bb/siggen.py | 17 +++++++++--------
2 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 4106fa4bc4..e84890b8b7 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -27,6 +27,7 @@ import pprint
bblogger = logging.getLogger("BitBake")
logger = logging.getLogger("BitBake.RunQueue")
+hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv")
__find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
@@ -2263,7 +2264,7 @@ class RunQueueExecute:
self.updated_taskhash_queue.remove((tid, unihash))
if unihash != self.rqdata.runtaskentries[tid].unihash:
- logger.info("Task %s unihash changed to %s" % (tid, unihash))
+ hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
self.rqdata.runtaskentries[tid].unihash = unihash
bb.parse.siggen.set_unihash(tid, unihash)
toprocess.add(tid)
@@ -2308,7 +2309,7 @@ class RunQueueExecute:
elif tid in self.scenequeue_covered or tid in self.sq_live:
# Already ran this setscene task or it running. Report the new taskhash
bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
- logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+ hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
remapped = True
if not remapped:
@@ -2327,7 +2328,7 @@ class RunQueueExecute:
for mc in self.rq.fakeworker:
self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
- logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
+ hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
for tid in changed:
if tid not in self.rqdata.runq_setscene_tids:
@@ -2346,7 +2347,7 @@ class RunQueueExecute:
# Check no tasks this covers are running
for dep in self.sqdata.sq_covered_tasks[tid]:
if dep in self.runq_running and dep not in self.runq_complete:
- logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
+ hashequiv_logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
valid = False
break
if not valid:
@@ -2409,7 +2410,7 @@ class RunQueueExecute:
for (tid, harddepfail, origvalid) in update_tasks:
if tid in self.sqdata.valid and not origvalid:
- logger.info("Setscene task %s became valid" % tid)
+ hashequiv_logger.info("Setscene task %s became valid" % tid)
if harddepfail:
self.sq_task_failoutright(tid)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index c2d0c736cf..0357b544b5 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -16,6 +16,7 @@ from bb import runqueue
import hashserv
logger = logging.getLogger('BitBake.SigGen')
+hashequiv_logger = logging.getLogger('BitBake.SigGen.HashEquiv')
def init(d):
siggens = [obj for obj in globals().values()
@@ -502,9 +503,9 @@ class SignatureGeneratorUniHashMixIn(object):
# A unique hash equal to the taskhash is not very interesting,
# so it is reported it at debug level 2. If they differ, that
# is much more interesting, so it is reported at debug level 1
- bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
+ hashequiv_logger.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
else:
- bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
+ hashequiv_logger.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
@@ -578,12 +579,12 @@ class SignatureGeneratorUniHashMixIn(object):
new_unihash = data['unihash']
if new_unihash != unihash:
- bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
+ hashequiv_logger.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
self.set_unihash(tid, new_unihash)
d.setVar('BB_UNIHASH', new_unihash)
else:
- bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
+ hashequiv_logger.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
finally:
@@ -606,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
method = method + self.extramethod[tid]
data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
- bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+ hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
if data is None:
bb.warn("Server unable to handle unihash report")
@@ -615,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
finalunihash = data['unihash']
if finalunihash == current_unihash:
- bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+ hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
elif finalunihash == wanted_unihash:
- bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+ hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
self.set_unihash(tid, finalunihash)
return True
else:
# TODO: What to do here?
- bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+ hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 12/15] bitbake: Log hash equivalence with a different logger
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Switches the hash equivalence logging to use a different logger so that
it can be easily filtered out with python's structured logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/runqueue.py | 11 ++++++-----
bitbake/lib/bb/siggen.py | 17 +++++++++--------
2 files changed, 15 insertions(+), 13 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 4106fa4bc4..e84890b8b7 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -27,6 +27,7 @@ import pprint
bblogger = logging.getLogger("BitBake")
logger = logging.getLogger("BitBake.RunQueue")
+hashequiv_logger = logging.getLogger("BitBake.RunQueue.HashEquiv")
__find_sha256__ = re.compile( r'(?i)(?<![a-z0-9])[a-f0-9]{64}(?![a-z0-9])' )
@@ -2263,7 +2264,7 @@ class RunQueueExecute:
self.updated_taskhash_queue.remove((tid, unihash))
if unihash != self.rqdata.runtaskentries[tid].unihash:
- logger.info("Task %s unihash changed to %s" % (tid, unihash))
+ hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
self.rqdata.runtaskentries[tid].unihash = unihash
bb.parse.siggen.set_unihash(tid, unihash)
toprocess.add(tid)
@@ -2308,7 +2309,7 @@ class RunQueueExecute:
elif tid in self.scenequeue_covered or tid in self.sq_live:
# Already ran this setscene task or it running. Report the new taskhash
bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
- logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+ hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
remapped = True
if not remapped:
@@ -2327,7 +2328,7 @@ class RunQueueExecute:
for mc in self.rq.fakeworker:
self.rq.fakeworker[mc].process.stdin.write(b"<newtaskhashes>" + pickle.dumps(bb.parse.siggen.get_taskhashes()) + b"</newtaskhashes>")
- logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
+ hashequiv_logger.debug(1, pprint.pformat("Tasks changed:\n%s" % (changed)))
for tid in changed:
if tid not in self.rqdata.runq_setscene_tids:
@@ -2346,7 +2347,7 @@ class RunQueueExecute:
# Check no tasks this covers are running
for dep in self.sqdata.sq_covered_tasks[tid]:
if dep in self.runq_running and dep not in self.runq_complete:
- logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
+ hashequiv_logger.debug(2, "Task %s is running which blocks setscene for %s from running" % (dep, tid))
valid = False
break
if not valid:
@@ -2409,7 +2410,7 @@ class RunQueueExecute:
for (tid, harddepfail, origvalid) in update_tasks:
if tid in self.sqdata.valid and not origvalid:
- logger.info("Setscene task %s became valid" % tid)
+ hashequiv_logger.info("Setscene task %s became valid" % tid)
if harddepfail:
self.sq_task_failoutright(tid)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index c2d0c736cf..0357b544b5 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -16,6 +16,7 @@ from bb import runqueue
import hashserv
logger = logging.getLogger('BitBake.SigGen')
+hashequiv_logger = logging.getLogger('BitBake.SigGen.HashEquiv')
def init(d):
siggens = [obj for obj in globals().values()
@@ -502,9 +503,9 @@ class SignatureGeneratorUniHashMixIn(object):
# A unique hash equal to the taskhash is not very interesting,
# so it is reported it at debug level 2. If they differ, that
# is much more interesting, so it is reported at debug level 1
- bb.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
+ hashequiv_logger.debug((1, 2)[unihash == taskhash], 'Found unihash %s in place of %s for %s from %s' % (unihash, taskhash, tid, self.server))
else:
- bb.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
+ hashequiv_logger.debug(2, 'No reported unihash for %s:%s from %s' % (tid, taskhash, self.server))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
@@ -578,12 +579,12 @@ class SignatureGeneratorUniHashMixIn(object):
new_unihash = data['unihash']
if new_unihash != unihash:
- bb.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
+ hashequiv_logger.debug(1, 'Task %s unihash changed %s -> %s by server %s' % (taskhash, unihash, new_unihash, self.server))
bb.event.fire(bb.runqueue.taskUniHashUpdate(fn + ':do_' + task, new_unihash), d)
self.set_unihash(tid, new_unihash)
d.setVar('BB_UNIHASH', new_unihash)
else:
- bb.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
+ hashequiv_logger.debug(1, 'Reported task %s as unihash %s to %s' % (taskhash, unihash, self.server))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
finally:
@@ -606,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
method = method + self.extramethod[tid]
data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
- bb.note('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+ hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
if data is None:
bb.warn("Server unable to handle unihash report")
@@ -615,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
finalunihash = data['unihash']
if finalunihash == current_unihash:
- bb.note('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+ hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
elif finalunihash == wanted_unihash:
- bb.note('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+ hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
self.set_unihash(tid, finalunihash)
return True
else:
# TODO: What to do here?
- bb.note('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+ hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 13/15] bitbake: Add autobuilder logging configuration
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds a configuration file that the autobuilder can use to capture
interesting logging domains above the ones that show up for normal users
on stdout/stderr.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/contrib/autobuilderlog.json | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
create mode 100644 bitbake/contrib/autobuilderlog.json
diff --git a/bitbake/contrib/autobuilderlog.json b/bitbake/contrib/autobuilderlog.json
new file mode 100644
index 0000000000..103a1141f6
--- /dev/null
+++ b/bitbake/contrib/autobuilderlog.json
@@ -0,0 +1,27 @@
+{
+ "version": 1,
+ "handlers": {
+ "autobuilderlog": {
+ "class": "logging.FileHandler",
+ "formatter": "logfileFormatter",
+ "level": "DEBUG",
+ "filename": "autobuilder.log",
+ "mode": "w"
+ }
+ },
+ "formatters": {
+ "logfileFormatter": {
+ "format": "%(name)s: %(levelname)s: %(message)s"
+ }
+ },
+ "loggers": {
+ "BitBake.SigGen.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ },
+ "BitBake.RunQueue.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ }
+ }
+}
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 13/15] bitbake: Add autobuilder logging configuration
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds a configuration file that the autobuilder can use to capture
interesting logging domains above the ones that show up for normal users
on stdout/stderr.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/contrib/autobuilderlog.json | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
create mode 100644 bitbake/contrib/autobuilderlog.json
diff --git a/bitbake/contrib/autobuilderlog.json b/bitbake/contrib/autobuilderlog.json
new file mode 100644
index 0000000000..103a1141f6
--- /dev/null
+++ b/bitbake/contrib/autobuilderlog.json
@@ -0,0 +1,27 @@
+{
+ "version": 1,
+ "handlers": {
+ "autobuilderlog": {
+ "class": "logging.FileHandler",
+ "formatter": "logfileFormatter",
+ "level": "DEBUG",
+ "filename": "autobuilder.log",
+ "mode": "w"
+ }
+ },
+ "formatters": {
+ "logfileFormatter": {
+ "format": "%(name)s: %(levelname)s: %(message)s"
+ }
+ },
+ "loggers": {
+ "BitBake.SigGen.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ },
+ "BitBake.RunQueue.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ }
+ }
+}
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 14/15] bitbake: Lower hash equivalence logging
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/runqueue.py | 6 +++---
bitbake/lib/bb/siggen.py | 8 ++++----
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index e84890b8b7..cef9b0fbbf 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -2264,7 +2264,7 @@ class RunQueueExecute:
self.updated_taskhash_queue.remove((tid, unihash))
if unihash != self.rqdata.runtaskentries[tid].unihash:
- hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
+ hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash))
self.rqdata.runtaskentries[tid].unihash = unihash
bb.parse.siggen.set_unihash(tid, unihash)
toprocess.add(tid)
@@ -2309,7 +2309,7 @@ class RunQueueExecute:
elif tid in self.scenequeue_covered or tid in self.sq_live:
# Already ran this setscene task or it running. Report the new taskhash
bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
- hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+ hashequiv_logger.verbose("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
remapped = True
if not remapped:
@@ -2410,7 +2410,7 @@ class RunQueueExecute:
for (tid, harddepfail, origvalid) in update_tasks:
if tid in self.sqdata.valid and not origvalid:
- hashequiv_logger.info("Setscene task %s became valid" % tid)
+ hashequiv_logger.verbose("Setscene task %s became valid" % tid)
if harddepfail:
self.sq_task_failoutright(tid)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 0357b544b5..8bfc45235d 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -607,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
method = method + self.extramethod[tid]
data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
- hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+ hashequiv_logger.verbose('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
if data is None:
bb.warn("Server unable to handle unihash report")
@@ -616,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
finalunihash = data['unihash']
if finalunihash == current_unihash:
- hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash))
elif finalunihash == wanted_unihash:
- hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
self.set_unihash(tid, finalunihash)
return True
else:
# TODO: What to do here?
- hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 14/15] bitbake: Lower hash equivalence logging
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Lowers the level at which hash equivalence messages are logged so as to
not annoy the majority of users. The autobuilder can use a custom
logging configuration to log these to a file for debugging (see
contrib/autobuilderlog.json)
[YOCTO #13813]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
bitbake/lib/bb/runqueue.py | 6 +++---
bitbake/lib/bb/siggen.py | 8 ++++----
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index e84890b8b7..cef9b0fbbf 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -2264,7 +2264,7 @@ class RunQueueExecute:
self.updated_taskhash_queue.remove((tid, unihash))
if unihash != self.rqdata.runtaskentries[tid].unihash:
- hashequiv_logger.info("Task %s unihash changed to %s" % (tid, unihash))
+ hashequiv_logger.verbose("Task %s unihash changed to %s" % (tid, unihash))
self.rqdata.runtaskentries[tid].unihash = unihash
bb.parse.siggen.set_unihash(tid, unihash)
toprocess.add(tid)
@@ -2309,7 +2309,7 @@ class RunQueueExecute:
elif tid in self.scenequeue_covered or tid in self.sq_live:
# Already ran this setscene task or it running. Report the new taskhash
bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
- hashequiv_logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
+ hashequiv_logger.verbose("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
remapped = True
if not remapped:
@@ -2410,7 +2410,7 @@ class RunQueueExecute:
for (tid, harddepfail, origvalid) in update_tasks:
if tid in self.sqdata.valid and not origvalid:
- hashequiv_logger.info("Setscene task %s became valid" % tid)
+ hashequiv_logger.verbose("Setscene task %s became valid" % tid)
if harddepfail:
self.sq_task_failoutright(tid)
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 0357b544b5..8bfc45235d 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -607,7 +607,7 @@ class SignatureGeneratorUniHashMixIn(object):
method = method + self.extramethod[tid]
data = self.client().report_unihash_equiv(taskhash, method, wanted_unihash, extra_data)
- hashequiv_logger.info('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
+ hashequiv_logger.verbose('Reported task %s as unihash %s to %s (%s)' % (tid, wanted_unihash, self.server, str(data)))
if data is None:
bb.warn("Server unable to handle unihash report")
@@ -616,14 +616,14 @@ class SignatureGeneratorUniHashMixIn(object):
finalunihash = data['unihash']
if finalunihash == current_unihash:
- hashequiv_logger.info('Task %s unihash %s unchanged by server' % (tid, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash %s unchanged by server' % (tid, finalunihash))
elif finalunihash == wanted_unihash:
- hashequiv_logger.info('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash changed %s -> %s as wanted' % (tid, current_unihash, finalunihash))
self.set_unihash(tid, finalunihash)
return True
else:
# TODO: What to do here?
- hashequiv_logger.info('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
+ hashequiv_logger.verbose('Task %s unihash reported as unwanted hash %s' % (tid, finalunihash))
except hashserv.client.HashConnectionError as e:
bb.warn('Error contacting Hash Equivalence Server %s: %s' % (self.server, str(e)))
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
@ 2020-03-09 16:33 ` Joshua Watt
-1 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs
Adds documentation describing how to use the BB_LOGCONFIG variable to
enable custom logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
.../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
.../bitbake-user-manual-ref-variables.xml | 11 +++
2 files changed, 108 insertions(+)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
index 8f606676b4..3b31f748cc 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
@@ -929,4 +929,101 @@
section.
</para>
</section>
+
+ <section id="logging">
+ <title>Logging</title>
+ <para>
+ In addition to the standard command line option to control how
+ verbose builds are when execute, bitbake also supports user defined
+ configuration of the
+ <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+ facilities through the
+ <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+ variable. This variable defines a json or yaml
+ <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+ that will be intelligently merged into the default configuration.
+ The logging configuration is merged using the following rules:
+ <itemizedlist>
+ <listitem><para>
+ The user defined configuration will completely replace the default
+ configuration if top level key
+ <filename>bitbake_merge</filename> is set to the value
+ <filename>False</filename>. In this case, all other rules
+ are ignored.
+ </para></listitem>
+ <listitem><para>
+ The user configuration must have a top level
+ <filename>version</filename> which must match the value of
+ the default configuration.
+ </para></listitem>
+ <listitem><para>
+ Any keys defined in the <filename>handlers</filename>,
+ <filename>formatters</filename>, or <filename>filters</filename>,
+ will be merged into the same section in the default
+ configuration, with the user specified keys taking
+ replacing a default one if there is a conflict. In
+ practice, this means that if both the default configuration
+ and user configuration specify a handler named
+ <filename>myhandler</filename>, the user defined one will
+ replace the default. To prevent the user from inadvertently
+ replacing a default handler, formatter, or filter, all of
+ the default ones are named with a prefix of
+ "<filename>BitBake.</filename>"
+ </para></listitem>
+ <listitem><para>
+ If a logger is defined by the user with the key
+ <filename>bitbake_merge</filename> set to
+ <filename>False</filename>, that logger will be completely
+ replaced by user configuration. In this case, no other
+ rules will apply to that logger.
+ </listitem></para>
+ <listitem><para>
+ All user defined <filename>filter</filename> and
+ <filename>handlers</filename> properties for a given logger
+ will be merged with corresponding properties from the
+ default logger. For example, if the user configuration adds
+ a filter called <filename>myFilter</filename> to the
+ <filename>BitBake.SigGen</filename>, and the default
+ configuration adds a filter called
+ <filename>BitBake.defaultFilter</filename>, both filters
+ will be applied to the logger
+ </listitem></para>
+ </itemizedlist>
+ </para>
+
+ <para>
+ As an example, consider the following user logging configuration
+ file which logs all Hash Equivalence related messages of VERBOSE or
+ higher to a file called <filename>hashequiv.log</filename>
+ <literallayout class='monospaced'>
+ {
+ "version": 1,
+ "handlers": {
+ "autobuilderlog": {
+ "class": "logging.FileHandler",
+ "formatter": "logfileFormatter",
+ "level": "DEBUG",
+ "filename": "hashequiv.log",
+ "mode": "w"
+ }
+ },
+ "formatters": {
+ "logfileFormatter": {
+ "format": "%(name)s: %(levelname)s: %(message)s"
+ }
+ },
+ "loggers": {
+ "BitBake.SigGen.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ },
+ "BitBake.RunQueue.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ }
+ }
+ }
+ </literallayout>
+ </para>
+ </section>
</chapter>
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
index bae01d90c0..c4bd1f2584 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
@@ -539,6 +539,17 @@
</glossdef>
</glossentry>
+ <glossentry id='var-bb-BB_LOGCONFIG'><glossterm>BB_LOGCONFIG</glossterm>
+ <glossdef>
+ <para>
+ Specifies the name of a config file that contains the user
+ logging configuration. See
+ <link linkend="logging">Logging</link> for additional
+ information
+ </para>
+ </glossdef>
+ </glossentry>
+
<glossentry id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
<glossdef>
<para>
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* [bitbake-devel][PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
@ 2020-03-09 16:33 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-09 16:33 UTC (permalink / raw)
To: bitbake-devel; +Cc: docs, Joshua Watt
Adds documentation describing how to use the BB_LOGCONFIG variable to
enable custom logging.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
.../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
.../bitbake-user-manual-ref-variables.xml | 11 +++
2 files changed, 108 insertions(+)
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
index 8f606676b4..3b31f748cc 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
@@ -929,4 +929,101 @@
section.
</para>
</section>
+
+ <section id="logging">
+ <title>Logging</title>
+ <para>
+ In addition to the standard command line option to control how
+ verbose builds are when execute, bitbake also supports user defined
+ configuration of the
+ <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
+ facilities through the
+ <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
+ variable. This variable defines a json or yaml
+ <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
+ that will be intelligently merged into the default configuration.
+ The logging configuration is merged using the following rules:
+ <itemizedlist>
+ <listitem><para>
+ The user defined configuration will completely replace the default
+ configuration if top level key
+ <filename>bitbake_merge</filename> is set to the value
+ <filename>False</filename>. In this case, all other rules
+ are ignored.
+ </para></listitem>
+ <listitem><para>
+ The user configuration must have a top level
+ <filename>version</filename> which must match the value of
+ the default configuration.
+ </para></listitem>
+ <listitem><para>
+ Any keys defined in the <filename>handlers</filename>,
+ <filename>formatters</filename>, or <filename>filters</filename>,
+ will be merged into the same section in the default
+ configuration, with the user specified keys taking
+ replacing a default one if there is a conflict. In
+ practice, this means that if both the default configuration
+ and user configuration specify a handler named
+ <filename>myhandler</filename>, the user defined one will
+ replace the default. To prevent the user from inadvertently
+ replacing a default handler, formatter, or filter, all of
+ the default ones are named with a prefix of
+ "<filename>BitBake.</filename>"
+ </para></listitem>
+ <listitem><para>
+ If a logger is defined by the user with the key
+ <filename>bitbake_merge</filename> set to
+ <filename>False</filename>, that logger will be completely
+ replaced by user configuration. In this case, no other
+ rules will apply to that logger.
+ </listitem></para>
+ <listitem><para>
+ All user defined <filename>filter</filename> and
+ <filename>handlers</filename> properties for a given logger
+ will be merged with corresponding properties from the
+ default logger. For example, if the user configuration adds
+ a filter called <filename>myFilter</filename> to the
+ <filename>BitBake.SigGen</filename>, and the default
+ configuration adds a filter called
+ <filename>BitBake.defaultFilter</filename>, both filters
+ will be applied to the logger
+ </listitem></para>
+ </itemizedlist>
+ </para>
+
+ <para>
+ As an example, consider the following user logging configuration
+ file which logs all Hash Equivalence related messages of VERBOSE or
+ higher to a file called <filename>hashequiv.log</filename>
+ <literallayout class='monospaced'>
+ {
+ "version": 1,
+ "handlers": {
+ "autobuilderlog": {
+ "class": "logging.FileHandler",
+ "formatter": "logfileFormatter",
+ "level": "DEBUG",
+ "filename": "hashequiv.log",
+ "mode": "w"
+ }
+ },
+ "formatters": {
+ "logfileFormatter": {
+ "format": "%(name)s: %(levelname)s: %(message)s"
+ }
+ },
+ "loggers": {
+ "BitBake.SigGen.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ },
+ "BitBake.RunQueue.HashEquiv": {
+ "level": "VERBOSE",
+ "handlers": ["autobuilderlog"]
+ }
+ }
+ }
+ </literallayout>
+ </para>
+ </section>
</chapter>
diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
index bae01d90c0..c4bd1f2584 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
@@ -539,6 +539,17 @@
</glossdef>
</glossentry>
+ <glossentry id='var-bb-BB_LOGCONFIG'><glossterm>BB_LOGCONFIG</glossterm>
+ <glossdef>
+ <para>
+ Specifies the name of a config file that contains the user
+ logging configuration. See
+ <link linkend="logging">Logging</link> for additional
+ information
+ </para>
+ </glossdef>
+ </glossentry>
+
<glossentry id='var-bb-BB_LOGFMT'><glossterm>BB_LOGFMT</glossterm>
<glossdef>
<para>
--
2.17.1
^ permalink raw reply related [flat|nested] 35+ messages in thread* Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
2020-03-09 16:33 ` [bitbake-devel][PATCH " Joshua Watt
(?)
@ 2020-03-12 23:50 ` Rich Persaud
2020-03-13 1:42 ` [docs] [bitbake-devel][PATCH " Joshua Watt
-1 siblings, 1 reply; 35+ messages in thread
From: Rich Persaud @ 2020-03-12 23:50 UTC (permalink / raw)
To: docs, Joshua Watt; +Cc: bitbake-devel
On Mar 9, 2020, at 12:34, Joshua Watt <jpewhacker@gmail.com> wrote:
>
> Adds documentation describing how to use the BB_LOGCONFIG variable to
> enable custom logging.
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
> .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
> .../bitbake-user-manual-ref-variables.xml | 11 +++
> 2 files changed, 108 insertions(+)
>
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> index 8f606676b4..3b31f748cc 100644
> --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> @@ -929,4 +929,101 @@
> section.
> </para>
> </section>
> +
> + <section id="logging">
> + <title>Logging</title>
> + <para>
> + In addition to the standard command line option to control how
> + verbose builds are when execute, bitbake also supports user defined
> + configuration of the
> + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
> + facilities through the
> + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
> + variable. This variable defines a json or yaml
> + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
> + that will be intelligently merged into the default configuration.
> + The logging configuration is merged using the following rules:
> + <itemizedlist>
> + <listitem><para>
> + The user defined configuration will completely replace the default
> + configuration if top level key
> + <filename>bitbake_merge</filename> is set to the value
> + <filename>False</filename>. In this case, all other rules
> + are ignored.
> + </para></listitem>
> + <listitem><para>
> + The user configuration must have a top level
> + <filename>version</filename> which must match the value of
> + the default configuration.
> + </para></listitem>
> + <listitem><para>
> + Any keys defined in the <filename>handlers</filename>,
> + <filename>formatters</filename>, or <filename>filters</filename>,
> + will be merged into the same section in the default
> + configuration, with the user specified keys taking
> + replacing a default one if there is a conflict. In
> + practice, this means that if both the default configuration
> + and user configuration specify a handler named
> + <filename>myhandler</filename>, the user defined one will
> + replace the default. To prevent the user from inadvertently
> + replacing a default handler, formatter, or filter, all of
> + the default ones are named with a prefix of
> + "<filename>BitBake.</filename>"
> + </para></listitem>
> + <listitem><para>
> + If a logger is defined by the user with the key
> + <filename>bitbake_merge</filename> set to
> + <filename>False</filename>, that logger will be completely
> + replaced by user configuration. In this case, no other
> + rules will apply to that logger.
> + </listitem></para>
> + <listitem><para>
> + All user defined <filename>filter</filename> and
> + <filename>handlers</filename> properties for a given logger
> + will be merged with corresponding properties from the
> + default logger. For example, if the user configuration adds
> + a filter called <filename>myFilter</filename> to the
> + <filename>BitBake.SigGen</filename>, and the default
> + configuration adds a filter called
> + <filename>BitBake.defaultFilter</filename>, both filters
> + will be applied to the logger
> + </listitem></para>
> + </itemizedlist>
> + </para>
> +
> + <para>
> + As an example, consider the following user logging configuration
> + file which logs all Hash Equivalence related messages of VERBOSE or
> + higher to a file called <filename>hashequiv.log</filename>
> + <literallayout class='monospaced'>
> + {
> + "version": 1,
> + "handlers": {
> + "autobuilderlog": {
> + "class": "logging.FileHandler",
> + "formatter": "logfileFormatter",
> + "level": "DEBUG",
> + "filename": "hashequiv.log",
> + "mode": "w"
> + }
> + },
> + "formatters": {
> + "logfileFormatter": {
> + "format": "%(name)s: %(levelname)s: %(message)s"
What's the syntax for including a timestamp in the log message? Or is that implicit?
Rich
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [docs] [PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
2020-03-12 23:50 ` [docs] [PATCH " Rich Persaud
@ 2020-03-13 1:42 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-13 1:42 UTC (permalink / raw)
To: Rich Persaud; +Cc: docs, bitbake-devel
On Thu, Mar 12, 2020 at 6:50 PM Rich Persaud <rp@stacktrust.org> wrote:
>
> On Mar 9, 2020, at 12:34, Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > Adds documentation describing how to use the BB_LOGCONFIG variable to
> > enable custom logging.
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> > .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
> > .../bitbake-user-manual-ref-variables.xml | 11 +++
> > 2 files changed, 108 insertions(+)
> >
> > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > index 8f606676b4..3b31f748cc 100644
> > --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > @@ -929,4 +929,101 @@
> > section.
> > </para>
> > </section>
> > +
> > + <section id="logging">
> > + <title>Logging</title>
> > + <para>
> > + In addition to the standard command line option to control how
> > + verbose builds are when execute, bitbake also supports user defined
> > + configuration of the
> > + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
> > + facilities through the
> > + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
> > + variable. This variable defines a json or yaml
> > + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
> > + that will be intelligently merged into the default configuration.
> > + The logging configuration is merged using the following rules:
> > + <itemizedlist>
> > + <listitem><para>
> > + The user defined configuration will completely replace the default
> > + configuration if top level key
> > + <filename>bitbake_merge</filename> is set to the value
> > + <filename>False</filename>. In this case, all other rules
> > + are ignored.
> > + </para></listitem>
> > + <listitem><para>
> > + The user configuration must have a top level
> > + <filename>version</filename> which must match the value of
> > + the default configuration.
> > + </para></listitem>
> > + <listitem><para>
> > + Any keys defined in the <filename>handlers</filename>,
> > + <filename>formatters</filename>, or <filename>filters</filename>,
> > + will be merged into the same section in the default
> > + configuration, with the user specified keys taking
> > + replacing a default one if there is a conflict. In
> > + practice, this means that if both the default configuration
> > + and user configuration specify a handler named
> > + <filename>myhandler</filename>, the user defined one will
> > + replace the default. To prevent the user from inadvertently
> > + replacing a default handler, formatter, or filter, all of
> > + the default ones are named with a prefix of
> > + "<filename>BitBake.</filename>"
> > + </para></listitem>
> > + <listitem><para>
> > + If a logger is defined by the user with the key
> > + <filename>bitbake_merge</filename> set to
> > + <filename>False</filename>, that logger will be completely
> > + replaced by user configuration. In this case, no other
> > + rules will apply to that logger.
> > + </listitem></para>
> > + <listitem><para>
> > + All user defined <filename>filter</filename> and
> > + <filename>handlers</filename> properties for a given logger
> > + will be merged with corresponding properties from the
> > + default logger. For example, if the user configuration adds
> > + a filter called <filename>myFilter</filename> to the
> > + <filename>BitBake.SigGen</filename>, and the default
> > + configuration adds a filter called
> > + <filename>BitBake.defaultFilter</filename>, both filters
> > + will be applied to the logger
> > + </listitem></para>
> > + </itemizedlist>
> > + </para>
> > +
> > + <para>
> > + As an example, consider the following user logging configuration
> > + file which logs all Hash Equivalence related messages of VERBOSE or
> > + higher to a file called <filename>hashequiv.log</filename>
> > + <literallayout class='monospaced'>
> > + {
> > + "version": 1,
> > + "handlers": {
> > + "autobuilderlog": {
> > + "class": "logging.FileHandler",
> > + "formatter": "logfileFormatter",
> > + "level": "DEBUG",
> > + "filename": "hashequiv.log",
> > + "mode": "w"
> > + }
> > + },
> > + "formatters": {
> > + "logfileFormatter": {
> > + "format": "%(name)s: %(levelname)s: %(message)s"
>
> What's the syntax for including a timestamp in the log message? Or is that implicit?
It's not implicit, but you can add it easily with %(asctime)s. There
are also several other options:
https://docs.python.org/3/library/logging.html#logrecord-attributes
>
> Rich
^ permalink raw reply [flat|nested] 35+ messages in thread* Re: [docs] [bitbake-devel][PATCH 15/15] bitbake-user-manual: Add documentation for BB_LOGCONFIG
@ 2020-03-13 1:42 ` Joshua Watt
0 siblings, 0 replies; 35+ messages in thread
From: Joshua Watt @ 2020-03-13 1:42 UTC (permalink / raw)
To: Rich Persaud; +Cc: docs, bitbake-devel
On Thu, Mar 12, 2020 at 6:50 PM Rich Persaud <rp@stacktrust.org> wrote:
>
> On Mar 9, 2020, at 12:34, Joshua Watt <jpewhacker@gmail.com> wrote:
> >
> > Adds documentation describing how to use the BB_LOGCONFIG variable to
> > enable custom logging.
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> > .../bitbake-user-manual-execution.xml | 97 +++++++++++++++++++
> > .../bitbake-user-manual-ref-variables.xml | 11 +++
> > 2 files changed, 108 insertions(+)
> >
> > diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > index 8f606676b4..3b31f748cc 100644
> > --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-execution.xml
> > @@ -929,4 +929,101 @@
> > section.
> > </para>
> > </section>
> > +
> > + <section id="logging">
> > + <title>Logging</title>
> > + <para>
> > + In addition to the standard command line option to control how
> > + verbose builds are when execute, bitbake also supports user defined
> > + configuration of the
> > + <ulink url='https://docs.python.org/3/library/logging.html'>Python logging</ulink>
> > + facilities through the
> > + <link linkend="var-bb-BB_LOGCONFIG"><filename>BB_LOGCONFIG</filename></link>
> > + variable. This variable defines a json or yaml
> > + <ulink url='https://docs.python.org/3/library/logging.config.html'>logging configuration</ulink>
> > + that will be intelligently merged into the default configuration.
> > + The logging configuration is merged using the following rules:
> > + <itemizedlist>
> > + <listitem><para>
> > + The user defined configuration will completely replace the default
> > + configuration if top level key
> > + <filename>bitbake_merge</filename> is set to the value
> > + <filename>False</filename>. In this case, all other rules
> > + are ignored.
> > + </para></listitem>
> > + <listitem><para>
> > + The user configuration must have a top level
> > + <filename>version</filename> which must match the value of
> > + the default configuration.
> > + </para></listitem>
> > + <listitem><para>
> > + Any keys defined in the <filename>handlers</filename>,
> > + <filename>formatters</filename>, or <filename>filters</filename>,
> > + will be merged into the same section in the default
> > + configuration, with the user specified keys taking
> > + replacing a default one if there is a conflict. In
> > + practice, this means that if both the default configuration
> > + and user configuration specify a handler named
> > + <filename>myhandler</filename>, the user defined one will
> > + replace the default. To prevent the user from inadvertently
> > + replacing a default handler, formatter, or filter, all of
> > + the default ones are named with a prefix of
> > + "<filename>BitBake.</filename>"
> > + </para></listitem>
> > + <listitem><para>
> > + If a logger is defined by the user with the key
> > + <filename>bitbake_merge</filename> set to
> > + <filename>False</filename>, that logger will be completely
> > + replaced by user configuration. In this case, no other
> > + rules will apply to that logger.
> > + </listitem></para>
> > + <listitem><para>
> > + All user defined <filename>filter</filename> and
> > + <filename>handlers</filename> properties for a given logger
> > + will be merged with corresponding properties from the
> > + default logger. For example, if the user configuration adds
> > + a filter called <filename>myFilter</filename> to the
> > + <filename>BitBake.SigGen</filename>, and the default
> > + configuration adds a filter called
> > + <filename>BitBake.defaultFilter</filename>, both filters
> > + will be applied to the logger
> > + </listitem></para>
> > + </itemizedlist>
> > + </para>
> > +
> > + <para>
> > + As an example, consider the following user logging configuration
> > + file which logs all Hash Equivalence related messages of VERBOSE or
> > + higher to a file called <filename>hashequiv.log</filename>
> > + <literallayout class='monospaced'>
> > + {
> > + "version": 1,
> > + "handlers": {
> > + "autobuilderlog": {
> > + "class": "logging.FileHandler",
> > + "formatter": "logfileFormatter",
> > + "level": "DEBUG",
> > + "filename": "hashequiv.log",
> > + "mode": "w"
> > + }
> > + },
> > + "formatters": {
> > + "logfileFormatter": {
> > + "format": "%(name)s: %(levelname)s: %(message)s"
>
> What's the syntax for including a timestamp in the log message? Or is that implicit?
It's not implicit, but you can add it easily with %(asctime)s. There
are also several other options:
https://docs.python.org/3/library/logging.html#logrecord-attributes
>
> Rich
^ permalink raw reply [flat|nested] 35+ messages in thread