* [PATCH 1/3] build: Catch and error upon circular task references
@ 2021-09-10 14:07 Richard Purdie
2021-09-10 14:07 ` [PATCH 2/3] data_smart: Improve error display for handled exceptions Richard Purdie
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Richard Purdie @ 2021-09-10 14:07 UTC (permalink / raw)
To: bitbake-devel
If there are circular task references, error on them rather than show
a recursion error. A simple reproducer is:
"""
do_packageswu () {
:
}
addtask do_packageswu after do_image_complete before do_image_qa
"""
into image_types.bbclass. There is code in runqueue to detect these but
we never get that far with the current codebase.
[YOCTO #13140]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/build.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/bb/build.py b/lib/bb/build.py
index 1e062adb51..4249c0be2b 100644
--- a/lib/bb/build.py
+++ b/lib/bb/build.py
@@ -1034,6 +1034,8 @@ def tasksbetween(task_start, task_end, d):
def follow_chain(task, endtask, chain=None):
if not chain:
chain = []
+ if task in chain:
+ bb.fatal("Circular task dependencies as %s depends itself via the chain %s" % (task, " -> ".join(chain)))
chain.append(task)
for othertask in tasks:
if othertask == task:
--
2.32.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/3] data_smart: Improve error display for handled exceptions
2021-09-10 14:07 [PATCH 1/3] build: Catch and error upon circular task references Richard Purdie
@ 2021-09-10 14:07 ` Richard Purdie
2021-09-10 14:07 ` [PATCH 3/3] fetch2: Add recursion guard Richard Purdie
2021-09-10 14:42 ` [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references Quentin Schulz
2 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2021-09-10 14:07 UTC (permalink / raw)
To: bitbake-devel
We don't need tracebacks for BBHandledException. Reduces confusing output like:
ERROR: /meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio
ERROR: ExpansionError during parsing /meta/recipes-core/images/core-image-tiny-initramfs.bb
Traceback (most recent call last):
File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_qa', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
if task in deps:
> follow_chain(othertask, endtask, chain)
chain.pop()
File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
if task in deps:
> follow_chain(othertask, endtask, chain)
chain.pop()
File "/bitbake/lib/bb/build.py", line 1050, in follow_chain(task='do_image_cpio', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
if task in deps:
> follow_chain(othertask, endtask, chain)
chain.pop()
File "/bitbake/lib/bb/build.py", line 1038, in follow_chain(task='do_image_complete', endtask='do_build', chain=['do_image_complete', 'do_packageswu', 'do_image_qa', 'do_image', 'do_image_cpio']):
if task in chain:
> bb.fatal("Circular task dependencies as %s depends itself via the chain %s?!" % (task, " -> ".join(chain)))
chain.append(task)
File "/bitbake/lib/bb/__init__.py", line 165, in fatal:
mainlogger.critical(''.join(args), extra=kwargs)
> raise BBHandledException()
to the real error:
ERROR: /media/build1/poky/meta/recipes-core/images/core-image-tiny-initramfs.bb: Circular task dependencies as do_image_complete depends itself via the chain do_image_complete -> do_packageswu -> do_image_qa -> do_image -> do_image_cpio
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/data_smart.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 515d195fe8..e4afac64b2 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -418,6 +418,8 @@ class DataSmart(MutableMapping):
raise
except bb.parse.SkipRecipe:
raise
+ except bb.BBHandledException:
+ raise
except Exception as exc:
tb = sys.exc_info()[2]
raise ExpansionError(varname, s, exc).with_traceback(tb) from exc
--
2.32.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/3] fetch2: Add recursion guard
2021-09-10 14:07 [PATCH 1/3] build: Catch and error upon circular task references Richard Purdie
2021-09-10 14:07 ` [PATCH 2/3] data_smart: Improve error display for handled exceptions Richard Purdie
@ 2021-09-10 14:07 ` Richard Purdie
2021-09-10 14:48 ` [bitbake-devel] " Quentin Schulz
2021-09-10 14:42 ` [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references Quentin Schulz
2 siblings, 1 reply; 7+ messages in thread
From: Richard Purdie @ 2021-09-10 14:07 UTC (permalink / raw)
To: bitbake-devel
Users sometimes put ${S} references in ${SRC_URI} without realising this can be
problematic. Improve the error messages if they accidentally do.
[YOCTO #11593]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
lib/bb/fetch2/__init__.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
index 47a4943369..d9e1599a05 100644
--- a/lib/bb/fetch2/__init__.py
+++ b/lib/bb/fetch2/__init__.py
@@ -754,6 +754,11 @@ def get_srcrev(d, method_name='sortable_revision'):
that fetcher provides a method with the given name and the same signature as sortable_revision.
"""
+ recursion = d.getVar("__BBINSRCREV")
+ if recursion:
+ raise FetchError("There are recursive references in fetcher variables, likely through SRC_URI")
+ d.setVar("__BBINSRCREV", True)
+
scms = []
fetcher = Fetch(d.getVar('SRC_URI').split(), d)
urldata = fetcher.ud
@@ -768,6 +773,7 @@ def get_srcrev(d, method_name='sortable_revision'):
autoinc, rev = getattr(urldata[scms[0]].method, method_name)(urldata[scms[0]], d, urldata[scms[0]].names[0])
if len(rev) > 10:
rev = rev[:10]
+ d.delVar("__BBINSRCREV")
if autoinc:
return "AUTOINC+" + rev
return rev
@@ -802,6 +808,7 @@ def get_srcrev(d, method_name='sortable_revision'):
if seenautoinc:
format = "AUTOINC+" + format
+ d.delVar("__BBINSRCREV")
return format
def localpath(url, d):
--
2.32.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references
2021-09-10 14:07 [PATCH 1/3] build: Catch and error upon circular task references Richard Purdie
2021-09-10 14:07 ` [PATCH 2/3] data_smart: Improve error display for handled exceptions Richard Purdie
2021-09-10 14:07 ` [PATCH 3/3] fetch2: Add recursion guard Richard Purdie
@ 2021-09-10 14:42 ` Quentin Schulz
2021-09-10 14:54 ` Richard Purdie
2 siblings, 1 reply; 7+ messages in thread
From: Quentin Schulz @ 2021-09-10 14:42 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
Hi Richard,
On Fri, Sep 10, 2021 at 03:07:38PM +0100, Richard Purdie wrote:
> If there are circular task references, error on them rather than show
> a recursion error. A simple reproducer is:
>
> """
> do_packageswu () {
> :
> }
>
> addtask do_packageswu after do_image_complete before do_image_qa
> """
>
> into image_types.bbclass. There is code in runqueue to detect these but
> we never get that far with the current codebase.
>
> [YOCTO #13140]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> lib/bb/build.py | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/lib/bb/build.py b/lib/bb/build.py
> index 1e062adb51..4249c0be2b 100644
> --- a/lib/bb/build.py
> +++ b/lib/bb/build.py
> @@ -1034,6 +1034,8 @@ def tasksbetween(task_start, task_end, d):
> def follow_chain(task, endtask, chain=None):
> if not chain:
> chain = []
> + if task in chain:
> + bb.fatal("Circular task dependencies as %s depends itself via the chain %s" % (task, " -> ".join(chain)))
depends +on itself?
Cheers,
Quentin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake-devel] [PATCH 3/3] fetch2: Add recursion guard
2021-09-10 14:07 ` [PATCH 3/3] fetch2: Add recursion guard Richard Purdie
@ 2021-09-10 14:48 ` Quentin Schulz
2021-09-10 14:49 ` Richard Purdie
0 siblings, 1 reply; 7+ messages in thread
From: Quentin Schulz @ 2021-09-10 14:48 UTC (permalink / raw)
To: Richard Purdie; +Cc: bitbake-devel
Hi Richard,
On Fri, Sep 10, 2021 at 03:07:40PM +0100, Richard Purdie wrote:
> Users sometimes put ${S} references in ${SRC_URI} without realising this can be
> problematic. Improve the error messages if they accidentally do.
>
I think we could also add this check to
insane.bbclass:package_qa_check_src_uri ?
Cheers,
Quentin
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake-devel] [PATCH 3/3] fetch2: Add recursion guard
2021-09-10 14:48 ` [bitbake-devel] " Quentin Schulz
@ 2021-09-10 14:49 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2021-09-10 14:49 UTC (permalink / raw)
To: Quentin Schulz; +Cc: bitbake-devel
On Fri, 2021-09-10 at 16:48 +0200, Quentin Schulz wrote:
> Hi Richard,
>
> On Fri, Sep 10, 2021 at 03:07:40PM +0100, Richard Purdie wrote:
> > Users sometimes put ${S} references in ${SRC_URI} without realising this can be
> > problematic. Improve the error messages if they accidentally do.
> >
>
> I think we could also add this check to
> insane.bbclass:package_qa_check_src_uri ?
It is a nice idea but would be problematic as it causes a parsing failure
unfortunately.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references
2021-09-10 14:42 ` [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references Quentin Schulz
@ 2021-09-10 14:54 ` Richard Purdie
0 siblings, 0 replies; 7+ messages in thread
From: Richard Purdie @ 2021-09-10 14:54 UTC (permalink / raw)
To: Quentin Schulz; +Cc: bitbake-devel
On Fri, 2021-09-10 at 16:42 +0200, Quentin Schulz wrote:
> Hi Richard,
>
> On Fri, Sep 10, 2021 at 03:07:38PM +0100, Richard Purdie wrote:
> > If there are circular task references, error on them rather than show
> > a recursion error. A simple reproducer is:
> >
> > """
> > do_packageswu () {
> > :
> > }
> >
> > addtask do_packageswu after do_image_complete before do_image_qa
> > """
> >
> > into image_types.bbclass. There is code in runqueue to detect these but
> > we never get that far with the current codebase.
> >
> > [YOCTO #13140]
> >
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > lib/bb/build.py | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/lib/bb/build.py b/lib/bb/build.py
> > index 1e062adb51..4249c0be2b 100644
> > --- a/lib/bb/build.py
> > +++ b/lib/bb/build.py
> > @@ -1034,6 +1034,8 @@ def tasksbetween(task_start, task_end, d):
> > def follow_chain(task, endtask, chain=None):
> > if not chain:
> > chain = []
> > + if task in chain:
> > + bb.fatal("Circular task dependencies as %s depends itself via the chain %s" % (task, " -> ".join(chain)))
>
> depends +on itself?
Well spotted, tweaked in master-next, thanks.
Cheers,
Richard
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2021-09-10 14:54 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-10 14:07 [PATCH 1/3] build: Catch and error upon circular task references Richard Purdie
2021-09-10 14:07 ` [PATCH 2/3] data_smart: Improve error display for handled exceptions Richard Purdie
2021-09-10 14:07 ` [PATCH 3/3] fetch2: Add recursion guard Richard Purdie
2021-09-10 14:48 ` [bitbake-devel] " Quentin Schulz
2021-09-10 14:49 ` Richard Purdie
2021-09-10 14:42 ` [bitbake-devel] [PATCH 1/3] build: Catch and error upon circular task references Quentin Schulz
2021-09-10 14:54 ` Richard Purdie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).