* [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target
@ 2021-02-07 11:46 Tomasz Dziendzielski
2021-02-09 15:50 ` [bitbake-devel] " Richard Purdie
0 siblings, 1 reply; 4+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-07 11:46 UTC (permalink / raw)
To: bitbake-devel; +Cc: Tomasz Dziendzielski
When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.
This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.
See [YOCTO #13071] for detailed bug information.
Add mc target name as a prefix to event handler name so there
won't be two different handlers with the same name.
Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
--- v4 add mc prefix to _bb_diskmonitor event handler to fix build, both build and test_stoptask_behavior should be working now
---
lib/bb/cookerdata.py | 5 ++++-
lib/bb/event.py | 2 ++
lib/bb/parse/ast.py | 7 ++++++-
lib/bb/runqueue.py | 15 +++++++++++++--
4 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index c39b5681..02995c8f 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -424,12 +424,15 @@ class CookerDataBuilder(object):
# Nomally we only register event handlers at the end of parsing .bb files
# We register any handlers we've found so far here...
for var in data.getVar('__BBHANDLERS', False) or []:
+ nvar=var
+ if mc:
+ var = nvar[len(mc):]
handlerfn = data.getVarFlag(var, "filename", False)
if not handlerfn:
parselog.critical("Undefined event handler function '%s'" % var)
raise bb.BBHandledException()
handlerln = int(data.getVarFlag(var, "lineno", False))
- bb.event.register(var, data.getVar(var, False), (data.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
+ bb.event.register(nvar, data.getVar(var, False), (data.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
data.setVar('BBINCLUDED',bb.parse.get_file_depends(data))
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 694b4705..afef7dbe 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -118,6 +118,8 @@ def fire_class_handlers(event, d):
if _eventfilter:
if not _eventfilter(name, handler, event, d):
continue
+ if d and name not in (d.getVar("__BBHANDLERS") or []):
+ continue
execute_handler(name, handler, event, d)
ui_queue = []
diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index 0714296a..b38978d3 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -261,6 +261,8 @@ class BBHandlerNode(AstNode):
def eval(self, data):
bbhands = data.getVar('__BBHANDLERS', False) or []
for h in self.hs:
+ if data.getVar('BB_CURRENT_MC', False):
+ h = data.getVar('BB_CURRENT_MC', False) + h
bbhands.append(h)
data.setVarFlag(h, "handler", 1)
data.setVar('__BBHANDLERS', bbhands)
@@ -331,11 +333,14 @@ def finalize(fn, d, variant = None):
try:
for var in d.getVar('__BBHANDLERS', False) or []:
# try to add the handler
+ nvar=var
+ if d.getVar('BB_CURRENT_MC', False):
+ var = nvar[len(d.getVar('BB_CURRENT_MC', False)):]
handlerfn = d.getVarFlag(var, "filename", False)
if not handlerfn:
bb.fatal("Undefined event handler function '%s'" % var)
handlerln = int(d.getVarFlag(var, "lineno", False))
- bb.event.register(var, d.getVar(var, False), (d.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
+ bb.event.register(nvar, d.getVar(var, False), (d.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
bb.event.fire(bb.event.RecipePreFinalise(fn), d)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 28bdadb4..ce14e111 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1464,7 +1464,14 @@ class RunQueue:
bb.event.fire(bb.event.DepTreeGenerated(depgraph), self.cooker.data)
if not self.dm_event_handler_registered:
- res = bb.event.register(self.dm_event_handler_name,
+ bbhands = self.cfgData.getVar('__BBHANDLERS', False) or []
+ nvar = self.dm_event_handler_name
+ if self.cfgData.getVar('BB_CURRENT_MC', False):
+ nvar = self.cfgData.getVar('BB_CURRENT_MC', False) + nvar
+
+ bbhands.append(nvar)
+ self.cfgData.setVar('__BBHANDLERS', bbhands)
+ res = bb.event.register(nvar,
lambda x: self.dm.check(self) if self.state in [runQueueRunning, runQueueCleanUp] else False,
('bb.event.HeartbeatEvent',))
self.dm_event_handler_registered = True
@@ -1505,7 +1512,11 @@ class RunQueue:
build_done = self.state is runQueueComplete or self.state is runQueueFailed
if build_done and self.dm_event_handler_registered:
- bb.event.remove(self.dm_event_handler_name, None)
+ nvar = self.dm_event_handler_name
+ if self.cfgData.getVar('BB_CURRENT_MC', False):
+ nvar = self.cfgData.getVar('BB_CURRENT_MC', False) + nvar
+
+ bb.event.remove(nvar, None)
self.dm_event_handler_registered = False
if build_done and self.rqexe:
--
2.30.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [bitbake-devel] [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target
2021-02-07 11:46 [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target Tomasz Dziendzielski
@ 2021-02-09 15:50 ` Richard Purdie
2021-02-09 15:56 ` Tomasz Dziendzielski
[not found] ` <16621F570CFF73B9.16853@lists.openembedded.org>
0 siblings, 2 replies; 4+ messages in thread
From: Richard Purdie @ 2021-02-09 15:50 UTC (permalink / raw)
To: Tomasz Dziendzielski, bitbake-devel
On Sun, 2021-02-07 at 12:46 +0100, Tomasz Dziendzielski wrote:
> When multiconfig is used bitbake might try to run events that don't
> exist for specific mc target. In cooker.py we pass
> `self.databuilder.mcdata[mc]` data that contains names of events'
> handlers per mc target, but fire_class_handlers uses global _handlers
> variable that is created during parsing of all the targets.
>
> This leads to a problem where bitbake runs event handler that don't
> exist for a target or even overrides them - if multiple targets use
> event handler with the same name but different code then only one
> version will be executed for all targets.
>
> See [YOCTO #13071] for detailed bug information.
>
> Add mc target name as a prefix to event handler name so there
> won't be two different handlers with the same name.
>
> Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
>
> --- v4 add mc prefix to _bb_diskmonitor event handler to fix build, both build and test_stoptask_behavior should be working now
> ---
> lib/bb/cookerdata.py | 5 ++++-
> lib/bb/event.py | 2 ++
> lib/bb/parse/ast.py | 7 ++++++-
> lib/bb/runqueue.py | 15 +++++++++++++--
> 4 files changed, 25 insertions(+), 4 deletions(-)
This still isn't right I'm afraid:
https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/3057
I did also wonder if we should be changing each of the event register
callers, or doing something inside the event register function instead?
Cheers,
Richard
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [bitbake-devel] [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target
2021-02-09 15:50 ` [bitbake-devel] " Richard Purdie
@ 2021-02-09 15:56 ` Tomasz Dziendzielski
[not found] ` <16621F570CFF73B9.16853@lists.openembedded.org>
1 sibling, 0 replies; 4+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-09 15:56 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 1972 bytes --]
>I did also wonder if we should be changing each of the event register
>callers, or doing something inside the event register function instead?
That should be possible. I'll work on that. Thanks and sorry for the
problem.
Best regards,
Tomasz Dziendzielski
wt., 9 lut 2021 o 16:50 Richard Purdie <richard.purdie@linuxfoundation.org>
napisał(a):
> On Sun, 2021-02-07 at 12:46 +0100, Tomasz Dziendzielski wrote:
> > When multiconfig is used bitbake might try to run events that don't
> > exist for specific mc target. In cooker.py we pass
> > `self.databuilder.mcdata[mc]` data that contains names of events'
> > handlers per mc target, but fire_class_handlers uses global _handlers
> > variable that is created during parsing of all the targets.
> >
> > This leads to a problem where bitbake runs event handler that don't
> > exist for a target or even overrides them - if multiple targets use
> > event handler with the same name but different code then only one
> > version will be executed for all targets.
> >
> > See [YOCTO #13071] for detailed bug information.
> >
> > Add mc target name as a prefix to event handler name so there
> > won't be two different handlers with the same name.
> >
> > Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
> >
> > --- v4 add mc prefix to _bb_diskmonitor event handler to fix build, both
> build and test_stoptask_behavior should be working now
> > ---
> > lib/bb/cookerdata.py | 5 ++++-
> > lib/bb/event.py | 2 ++
> > lib/bb/parse/ast.py | 7 ++++++-
> > lib/bb/runqueue.py | 15 +++++++++++++--
> > 4 files changed, 25 insertions(+), 4 deletions(-)
>
> This still isn't right I'm afraid:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/39/builds/3057
>
> I did also wonder if we should be changing each of the event register
> callers, or doing something inside the event register function instead?
>
> Cheers,
>
> Richard
>
>
[-- Attachment #2: Type: text/html, Size: 2720 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [bitbake-devel] [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target
[not found] ` <16621F570CFF73B9.16853@lists.openembedded.org>
@ 2021-02-10 11:15 ` Tomasz Dziendzielski
0 siblings, 0 replies; 4+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-10 11:15 UTC (permalink / raw)
To: Tomasz Dziendzielski; +Cc: Richard Purdie, bitbake-devel
[-- Attachment #1: Type: text/plain, Size: 975 bytes --]
>>I did also wonder if we should be changing each of the event register
>>callers, or doing something inside the event register function instead?
Thanks again for your suggestion, it was the right way to go that solved
the inconsistency with bb_diskmonitor event handler (and any other that
would be registered manually, without addhandler statement). I prepared the
change and it does not trigger any errors. I tested it with:
$ bitbake-selftest
$ oe-selftest -r multiconfig,
$ oe-selftest -r buildoptions.DiskMonTest.test_stoptask_behavior
$ yocto-check-layer-wrapper ../meta-mingw
$ bitbake core-image-sato:do_testsdkext core-image-minimal:do_testsdkext
core-image-sato:do_testsdkext (only for qemux86-64, one of the machines
that was failing)
which are the tests that failed with previous patches and now all tests
passed.
Please check the next patch:
https://lists.openembedded.org/g/bitbake-devel/message/12020
Best regards,
Tomasz Dziendzielski
[-- Attachment #2: Type: text/html, Size: 1245 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-02-10 11:13 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-07 11:46 [PATCH v4] event: Prevent bitbake from executing event handler for wrong multiconfig target Tomasz Dziendzielski
2021-02-09 15:50 ` [bitbake-devel] " Richard Purdie
2021-02-09 15:56 ` Tomasz Dziendzielski
[not found] ` <16621F570CFF73B9.16853@lists.openembedded.org>
2021-02-10 11:15 ` Tomasz Dziendzielski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.