All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target
@ 2021-02-04 22:14 Tomasz Dziendzielski
  2021-02-05 12:17 ` [bitbake-devel] " Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-04 22:14 UTC (permalink / raw)
  To: bitbake-devel; +Cc: Tomasz Dziendzielski

When multiconfig is used bitbake might try to run events that don't
exist for specific mc target. In cooker.py we pass
`self.databuilder.mcdata[mc]` data that contains names of events'
handlers per mc target, but fire_class_handlers uses global _handlers
variable that is created during parsing of all the targets.

This leads to a problem where bitbake runs event handler that don't
exist for a target or even overrides them - if multiple targets use
event handler with the same name but different code then only one
version will be executed for all targets.

See [YOCTO #13071] for detailed bug information.

I added a mc target name as a prefix to event handler name so there
won't be two different handlers with the same name.

Signed-off-by: Tomasz Dziendzielski <tomasz.dziendzielski@gmail.com>
---
 lib/bb/cookerdata.py | 3 ++-
 lib/bb/event.py      | 5 +++++
 lib/bb/parse/ast.py  | 7 ++++++-
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index c39b5681..108346ce 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -429,7 +429,8 @@ class CookerDataBuilder(object):
                 parselog.critical("Undefined event handler function '%s'" % var)
                 raise bb.BBHandledException()
             handlerln = int(data.getVarFlag(var, "lineno", False))
-            bb.event.register(var, data.getVar(var, False),  (data.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
+            nvar = mc + var
+            bb.event.register(nvar, data.getVar(var, False),  (data.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
 
         data.setVar('BBINCLUDED',bb.parse.get_file_depends(data))
 
diff --git a/lib/bb/event.py b/lib/bb/event.py
index 694b4705..d0ddac23 100644
--- a/lib/bb/event.py
+++ b/lib/bb/event.py
@@ -118,6 +118,11 @@ def fire_class_handlers(event, d):
             if _eventfilter:
                 if not _eventfilter(name, handler, event, d):
                     continue
+
+            mc_prefix = d.getVar('BB_CURRENT_MC') or ""
+            if mc_prefix and not name[len(mc_prefix):] in (d.getVar("__BBHANDLERS") or []):
+                continue
+
             execute_handler(name, handler, event, d)
 
 ui_queue = []
diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index 0714296a..202cbffb 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -335,7 +335,12 @@ def finalize(fn, d, variant = None):
             if not handlerfn:
                 bb.fatal("Undefined event handler function '%s'" % var)
             handlerln = int(d.getVarFlag(var, "lineno", False))
-            bb.event.register(var, d.getVar(var, False), (d.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
+
+            nvar = var
+            if d.getVar('BB_CURRENT_MC', False):
+                nvar = d.getVar('BB_CURRENT_MC', False) + var
+
+            bb.event.register(nvar, d.getVar(var, False), (d.getVarFlag(var, "eventmask") or "").split(), handlerfn, handlerln)
 
         bb.event.fire(bb.event.RecipePreFinalise(fn), d)
 
-- 
2.30.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target
  2021-02-04 22:14 [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target Tomasz Dziendzielski
@ 2021-02-05 12:17 ` Richard Purdie
  2021-02-05 14:26   ` Tomasz Dziendzielski
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2021-02-05 12:17 UTC (permalink / raw)
  To: Tomasz Dziendzielski, bitbake-devel

On Thu, 2021-02-04 at 23:14 +0100, Tomasz Dziendzielski wrote:
> When multiconfig is used bitbake might try to run events that don't
> exist for specific mc target. In cooker.py we pass
> `self.databuilder.mcdata[mc]` data that contains names of events'
> handlers per mc target, but fire_class_handlers uses global _handlers
> variable that is created during parsing of all the targets.
> 
> This leads to a problem where bitbake runs event handler that don't
> exist for a target or even overrides them - if multiple targets use
> event handler with the same name but different code then only one
> version will be executed for all targets.
> 
> See [YOCTO #13071] for detailed bug information.
> 
> I added a mc target name as a prefix to event handler name so there
> won't be two different handlers with the same name.

This looks like it solves a few issues, thanks!

I'll include it in the next test runs and see where we go from there
but it looks right.

Cheers,

Richard



^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target
  2021-02-05 12:17 ` [bitbake-devel] " Richard Purdie
@ 2021-02-05 14:26   ` Tomasz Dziendzielski
  2021-02-05 14:42     ` Richard Purdie
  0 siblings, 1 reply; 5+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-05 14:26 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

I can already see the bitbake test failed, after adding "if d" it passed.
Sorry that I forgot to run tests before submitting a patch. Another thing,
I ran locally oe-selftest -r multiconfig and build fails with "When
reparsing %s, the basehash value changed from %s to %s. The metadata is not
deterministic and this needs to be fixed". During the build it appeared
only if I made changes to the code during the build. Here it's most
probably caused because of events' handlers renaming.
Do you think there is some way to handle this issue or I should just add:
         ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') ==
'1')
+        if d.getVar("__BBMULTICONFIG", False):
+            ignore_mismatch = True
to bitbake/lib/bb/siggen.py?

Best regards,
Tomasz Dziendzielski

[-- Attachment #2: Type: text/html, Size: 1026 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target
  2021-02-05 14:26   ` Tomasz Dziendzielski
@ 2021-02-05 14:42     ` Richard Purdie
  2021-02-05 15:13       ` Tomasz Dziendzielski
  0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2021-02-05 14:42 UTC (permalink / raw)
  To: Tomasz Dziendzielski; +Cc: bitbake-devel

On Fri, 2021-02-05 at 15:26 +0100, Tomasz Dziendzielski wrote:
> I can already see the bitbake test failed, after adding "if d" it
> passed. Sorry that I forgot to run tests before submitting a patch.
> Another thing, I ran locally oe-selftest -r multiconfig and build
> fails with "When reparsing %s, the basehash value changed from %s to
> %s. The metadata is not deterministic and this needs to be fixed".
> During the build it appeared only if I made changes to the code
> during the build. Here it's most probably caused because of events'
> handlers renaming.

That is going to be a problem, we need to make sure the hashes are
deterministic.

> Do you think there is some way to handle this issue or I should just
> add:
>          ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or
> '') == '1')
> +        if d.getVar("__BBMULTICONFIG", False):
> +            ignore_mismatch = True
> to bitbake/lib/bb/siggen.py?

No, that is definitely going to just mask the problem.

These tests are designed to detect problems which will cause us "pain"
later. The hashes are designed to be stable to whether something is
built as a multiconfig or not shouldn't change its hashes. The tests
are warning us that is happening so we need to fix that.

Cheers,

Richard


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [bitbake-devel] [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target
  2021-02-05 14:42     ` Richard Purdie
@ 2021-02-05 15:13       ` Tomasz Dziendzielski
  0 siblings, 0 replies; 5+ messages in thread
From: Tomasz Dziendzielski @ 2021-02-05 15:13 UTC (permalink / raw)
  To: Richard Purdie; +Cc: bitbake-devel

[-- Attachment #1: Type: text/plain, Size: 2512 bytes --]

>No, that is definitely going to just mask the problem.
>
>These tests are designed to detect problems which will cause us "pain"
>later. The hashes are designed to be stable to whether something is
>built as a multiconfig or not shouldn't change its hashes. The tests
>are warning us that this is happening so we need to fix that.

Thanks, I think I already found the solution. I need to add the mc prefix
before it is written to __BBHANDLERS variable and now I don't see any
metadata hashes changing on test. I will submit a patch after oe-selftest
for multiconfig finishes with success.

@@ -259,6 +259,9 @@ class BBHandlerNode(AstNode):
     def eval(self, data):
         bbhands = data.getVar('__BBHANDLERS', False) or []
         for h in self.hs:
+            if data.getVar('BB_CURRENT_MC', False):
+                h = data.getVar('BB_CURRENT_MC', False) + h
+             bbhands.append(h)
             data.setVarFlag(h, "handler", 1)
         data.setVar('__BBHANDLERS', bbhands)

Best regards,
Tomasz Dziendzielski

pt., 5 lut 2021 o 15:42 Richard Purdie <richard.purdie@linuxfoundation.org>
napisał(a):

> On Fri, 2021-02-05 at 15:26 +0100, Tomasz Dziendzielski wrote:
> > I can already see the bitbake test failed, after adding "if d" it
> > passed. Sorry that I forgot to run tests before submitting a patch.
> > Another thing, I ran locally oe-selftest -r multiconfig and build
> > fails with "When reparsing %s, the basehash value changed from %s to
> > %s. The metadata is not deterministic and this needs to be fixed".
> > During the build it appeared only if I made changes to the code
> > during the build. Here it's most probably caused because of events'
> > handlers renaming.
>
> That is going to be a problem, we need to make sure the hashes are
> deterministic.
>
> > Do you think there is some way to handle this issue or I should just
> > add:
> >          ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or
> > '') == '1')
> > +        if d.getVar("__BBMULTICONFIG", False):
> > +            ignore_mismatch = True
> > to bitbake/lib/bb/siggen.py?
>
> No, that is definitely going to just mask the problem.
>
> These tests are designed to detect problems which will cause us "pain"
> later. The hashes are designed to be stable to whether something is
> built as a multiconfig or not shouldn't change its hashes. The tests
> are warning us that is happening so we need to fix that.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3301 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-02-05 15:12 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-02-04 22:14 [PATCH] event: Prevent bitbake from executing event handler for wrong multiconfig target Tomasz Dziendzielski
2021-02-05 12:17 ` [bitbake-devel] " Richard Purdie
2021-02-05 14:26   ` Tomasz Dziendzielski
2021-02-05 14:42     ` Richard Purdie
2021-02-05 15:13       ` Tomasz Dziendzielski

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.