* [PATCH 0/2] Fixes for addtask and deltask @ 2019-04-25 10:00 Robert Yang 2019-04-25 10:00 ` [PATCH 1/2] bitbake: BBHandler: Fix " Robert Yang 2019-04-25 10:01 ` [PATCH 2/2] bitbake: build.py: check dependendent task for addtask Robert Yang 0 siblings, 2 replies; 14+ messages in thread From: Robert Yang @ 2019-04-25 10:00 UTC (permalink / raw) To: bitbake-devel The following changes since commit 7f7a7cba4dcd99d7b36a0d747bfd1bba3f9c2288: bitbake: bb: siggen: Print more info when basehash are mis-matched (2019-04-23 23:30:45 +0100) are available in the git repository at: git://git.pokylinux.org/poky-contrib rbt/addtask http://git.pokylinux.org/cgit.cgi//log/?h=rbt/addtask Robert Yang (2): bitbake: BBHandler: Fix addtask and deltask bitbake: build.py: check dependendent task for addtask bitbake/lib/bb/build.py | 3 +++ bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 ++++++++++++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) -- 2.7.4 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask 2019-04-25 10:00 [PATCH 0/2] Fixes for addtask and deltask Robert Yang @ 2019-04-25 10:00 ` Robert Yang 2019-04-25 10:48 ` Richard Purdie 2019-04-25 10:01 ` [PATCH 2/2] bitbake: build.py: check dependendent task for addtask Robert Yang 1 sibling, 1 reply; 14+ messages in thread From: Robert Yang @ 2019-04-25 10:00 UTC (permalink / raw) To: bitbake-devel The following commands are not supported, but they were ignored silently, that may suprise users: * addtask task1 task2 task2 is ignored * addtask task1 before task2 before task3 Should be: addtask task1 before task2 task3 * addtask task1 after task2 after task3 Should be: addtask task1 after task2 task3 * deltask task1 task2 task2 is ignore This patch can check and warn for them. [YOCTO #13282] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py index 9dba5f2..4d90ff0 100644 --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py @@ -42,7 +42,7 @@ __func_start_regexp__ = re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*( __inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) __export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) __addtask_regexp__ = re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*") -__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)") +__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)") __addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) __def_regexp__ = re.compile(r"def\s+(\w+).*:" ) __python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) @@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements, eof=False): m = __addtask_regexp__.match(s) if m: + if len(m.group().split()) == 2: + # Check and warn for "addtask task1 task2" + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", s) + if m2 and m2.group('ignores'): + logger.warning('addtask ignored: "%s"' % m2.group('ignores')) + + # Check and warn for "addtask task1 before task2 before task3", the + # similar to "after" + dup_after = 0 + dup_before = 0 + for tmp in s.split(): + if 'before' == tmp: + dup_before += 1 + if 'after' == tmp: + dup_after += 1 + if dup_after > 1: + logger.warning('addtask found more than 1 keyword "after": "%s"' % s) + if dup_before > 1: + logger.warning('addtask found more than 1 keyword "before": "%s"' % s) + ast.handleAddTask(statements, fn, lineno, m) return m = __deltask_regexp__.match(s) if m: + # Check and warn "for deltask task1 task2" + if m.group('ignores'): + logger.warning('deltask ignored: "%s"' % m.group('ignores')) ast.handleDelTask(statements, fn, lineno, m) return -- 2.7.4 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask 2019-04-25 10:00 ` [PATCH 1/2] bitbake: BBHandler: Fix " Robert Yang @ 2019-04-25 10:48 ` Richard Purdie 2019-04-26 3:17 ` Robert Yang 0 siblings, 1 reply; 14+ messages in thread From: Richard Purdie @ 2019-04-25 10:48 UTC (permalink / raw) To: Robert Yang, bitbake-devel On Thu, 2019-04-25 at 18:00 +0800, Robert Yang wrote: > The following commands are not supported, but they were ignored > silently, that > may suprise users: > > * addtask task1 task2 > task2 is ignored > > * addtask task1 before task2 before task3 > Should be: addtask task1 before task2 task3 > > * addtask task1 after task2 after task3 > Should be: addtask task1 after task2 task3 > > * deltask task1 task2 > task2 is ignore > > This patch can check and warn for them. > > [YOCTO #13282] > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 > ++++++++++++++++++++++++- > 1 file changed, 24 insertions(+), 1 deletion(-) > > diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py > b/bitbake/lib/bb/parse/parse_py/BBHandler.py > index 9dba5f2..4d90ff0 100644 > --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py > +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py > @@ -42,7 +42,7 @@ __func_start_regexp__ = > re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*( > __inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) > __export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) > __addtask_regexp__ = > re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=a > fter))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*") > -__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)") > +__deltask_regexp__ = > re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)") > __addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) > __def_regexp__ = re.compile(r"def\s+(\w+).*:" ) > __python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) > @@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements, > eof=False): > > m = __addtask_regexp__.match(s) > if m: > + if len(m.group().split()) == 2: > + # Check and warn for "addtask task1 task2" > + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", > s) > + if m2 and m2.group('ignores'): > + logger.warning('addtask ignored: "%s"' % > m2.group('ignores')) > + > + # Check and warn for "addtask task1 before task2 before > task3", the > + # similar to "after" > + dup_after = 0 > + dup_before = 0 > + for tmp in s.split(): > + if 'before' == tmp: > + dup_before += 1 > + if 'after' == tmp: > + dup_after += 1 > + if dup_after > 1: taskexpression = s.split() if taskexpression.count("after") > 1: logger.warning("X") if taskexpression.count("before") > 1: logger.warning("Y") > + logger.warning('addtask found more than 1 keyword > "after": "%s"' % s) How about: addtask contained multiple 'after' keywords, only one is supported > + if dup_before > 1: > + logger.warning('addtask found more than 1 keyword > "before": "%s"' % s) and: addtask contained multiple 'before' keywords, only one is supported > ast.handleAddTask(statements, fn, lineno, m) > return > > m = __deltask_regexp__.match(s) > if m: > + # Check and warn "for deltask task1 task2" > + if m.group('ignores'): > + logger.warning('deltask ignored: "%s"' % > m.group('ignores')) deltask listing multiple tasks is not supported I did wonder if we should support multiple tasks for deltask? Also, we might want to consider adding something to lib/bb/tests/ for this? Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask 2019-04-25 10:48 ` Richard Purdie @ 2019-04-26 3:17 ` Robert Yang 0 siblings, 0 replies; 14+ messages in thread From: Robert Yang @ 2019-04-26 3:17 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 4/25/19 6:48 PM, Richard Purdie wrote: > On Thu, 2019-04-25 at 18:00 +0800, Robert Yang wrote: >> The following commands are not supported, but they were ignored >> silently, that >> may suprise users: >> >> * addtask task1 task2 >> task2 is ignored >> >> * addtask task1 before task2 before task3 >> Should be: addtask task1 before task2 task3 >> >> * addtask task1 after task2 after task3 >> Should be: addtask task1 after task2 task3 >> >> * deltask task1 task2 >> task2 is ignore >> >> This patch can check and warn for them. >> >> [YOCTO #13282] >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 >> ++++++++++++++++++++++++- >> 1 file changed, 24 insertions(+), 1 deletion(-) >> >> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py >> b/bitbake/lib/bb/parse/parse_py/BBHandler.py >> index 9dba5f2..4d90ff0 100644 >> --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py >> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py >> @@ -42,7 +42,7 @@ __func_start_regexp__ = >> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*( >> __inherit_regexp__ = re.compile(r"inherit\s+(.+)" ) >> __export_func_regexp__ = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" ) >> __addtask_regexp__ = >> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=a >> fter))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*") >> -__deltask_regexp__ = re.compile(r"deltask\s+(?P<func>\w+)") >> +__deltask_regexp__ = >> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)") >> __addhandler_regexp__ = re.compile(r"addhandler\s+(.+)" ) >> __def_regexp__ = re.compile(r"def\s+(\w+).*:" ) >> __python_func_regexp__ = re.compile(r"(\s+.*)|(^$)|(^#)" ) >> @@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements, >> eof=False): >> >> m = __addtask_regexp__.match(s) >> if m: >> + if len(m.group().split()) == 2: >> + # Check and warn for "addtask task1 task2" >> + m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", >> s) >> + if m2 and m2.group('ignores'): >> + logger.warning('addtask ignored: "%s"' % >> m2.group('ignores')) >> + >> + # Check and warn for "addtask task1 before task2 before >> task3", the >> + # similar to "after" >> + dup_after = 0 >> + dup_before = 0 >> + for tmp in s.split(): >> + if 'before' == tmp: >> + dup_before += 1 >> + if 'after' == tmp: >> + dup_after += 1 >> + if dup_after > 1: > > taskexpression = s.split() > if taskexpression.count("after") > 1: > > logger.warning("X") > if taskexpression.count("before") > 1: > > logger.warning("Y") > >> + logger.warning('addtask found more than 1 keyword >> "after": "%s"' % s) > > How about: > > addtask contained multiple 'after' keywords, only one is supported > >> + if dup_before > 1: >> + logger.warning('addtask found more than 1 keyword >> "before": "%s"' % s) > > and: > > addtask contained multiple 'before' keywords, only one is supported Thanks for all the replies above, I will fix them. > >> ast.handleAddTask(statements, fn, lineno, m) >> return >> >> m = __deltask_regexp__.match(s) >> if m: >> + # Check and warn "for deltask task1 task2" >> + if m.group('ignores'): >> + logger.warning('deltask ignored: "%s"' % >> m.group('ignores')) > > deltask listing multiple tasks is not supported > > I did wonder if we should support multiple tasks for deltask? Yes, that's a good idea, but if we support "deltask multiple tasks", I think that we should also support "addtask multiple tasks" since they are a pair, but I have other concerns on addtask, for example, the following code is hard to read and maintain: __addtask_regexp__ = re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*") I think that we need break it into small pieces. so I gave it up with these concerns which looks risky. > > Also, we might want to consider adding something to lib/bb/tests/ for > this? Thanks, I will add it. // Robert > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-04-25 10:00 [PATCH 0/2] Fixes for addtask and deltask Robert Yang 2019-04-25 10:00 ` [PATCH 1/2] bitbake: BBHandler: Fix " Robert Yang @ 2019-04-25 10:01 ` Robert Yang 2019-04-25 10:50 ` Richard Purdie 1 sibling, 1 reply; 14+ messages in thread From: Robert Yang @ 2019-04-25 10:01 UTC (permalink / raw) To: bitbake-devel The following command is incorrect, but was ignored silently, that may suprise users: addtask task after task_not_existed This patch can check and warn for it. It would be better to also check "before" tasks, but there is no easier way to do it. [YOCTO #13282] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- bitbake/lib/bb/build.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py index 7571421..861e9a9 100644 --- a/bitbake/lib/bb/build.py +++ b/bitbake/lib/bb/build.py @@ -815,6 +815,9 @@ def add_tasks(tasklist, d): task_deps['parents'][task] = [] if 'deps' in flags: for dep in flags['deps']: + # Check and warn for "addtask task after foo" while foo does not exist + if not dep in tasklist: + bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep)) dep = d.expand(dep) task_deps['parents'][task].append(dep) -- 2.7.4 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-04-25 10:01 ` [PATCH 2/2] bitbake: build.py: check dependendent task for addtask Robert Yang @ 2019-04-25 10:50 ` Richard Purdie 2019-04-26 2:49 ` Robert Yang 0 siblings, 1 reply; 14+ messages in thread From: Richard Purdie @ 2019-04-25 10:50 UTC (permalink / raw) To: Robert Yang, bitbake-devel On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote: > The following command is incorrect, but was ignored silently, that may suprise > users: > > addtask task after task_not_existed > > This patch can check and warn for it. It would be better to also check "before" > tasks, but there is no easier way to do it. > > [YOCTO #13282] > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > bitbake/lib/bb/build.py | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py > index 7571421..861e9a9 100644 > --- a/bitbake/lib/bb/build.py > +++ b/bitbake/lib/bb/build.py > @@ -815,6 +815,9 @@ def add_tasks(tasklist, d): > task_deps['parents'][task] = [] > if 'deps' in flags: > for dep in flags['deps']: > + # Check and warn for "addtask task after foo" while foo does not exist > + if not dep in tasklist: > + bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep)) > dep = d.expand(dep) > task_deps['parents'][task].append(dep) I can't help wonder if this change has races, depending on the order of the addtask and deltask expressions. Do most layers parse cleanly with this? At one point bitbake did support "floating" invalid tasks, I can't remember if we changed that or not though. Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-04-25 10:50 ` Richard Purdie @ 2019-04-26 2:49 ` Robert Yang 2019-04-30 12:13 ` Martin Jansa 2019-05-01 10:50 ` Jacob Kroon 0 siblings, 2 replies; 14+ messages in thread From: Robert Yang @ 2019-04-26 2:49 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 4/25/19 6:50 PM, Richard Purdie wrote: > On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote: >> The following command is incorrect, but was ignored silently, that may suprise >> users: >> >> addtask task after task_not_existed >> >> This patch can check and warn for it. It would be better to also check "before" >> tasks, but there is no easier way to do it. >> >> [YOCTO #13282] >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> bitbake/lib/bb/build.py | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py >> index 7571421..861e9a9 100644 >> --- a/bitbake/lib/bb/build.py >> +++ b/bitbake/lib/bb/build.py >> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d): >> task_deps['parents'][task] = [] >> if 'deps' in flags: >> for dep in flags['deps']: >> + # Check and warn for "addtask task after foo" while foo does not exist >> + if not dep in tasklist: >> + bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep)) >> dep = d.expand(dep) >> task_deps['parents'][task].append(dep) > > I can't help wonder if this change has races, depending on the order of > the addtask and deltask expressions. Do most layers parse cleanly with > this? The bb.build.add_tasks() is only called by ast.py's finalize(), I think that everything should be ready in finalize stage, and there should be no races. I tested this with meta-oe, python perl, selinux and others, didn't see any problems (unless I depended on an invalid task). > > At one point bitbake did support "floating" invalid tasks, I can't > remember if we changed that or not though. This warning is good for checking typos during development, I think that user can call "bb.build.addtask()" dynamically in a function to fix the warning for floating tasks? // Robert > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-04-26 2:49 ` Robert Yang @ 2019-04-30 12:13 ` Martin Jansa 2019-05-01 10:50 ` Jacob Kroon 1 sibling, 0 replies; 14+ messages in thread From: Martin Jansa @ 2019-04-30 12:13 UTC (permalink / raw) To: Robert Yang; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 5722 bytes --] I just got few thousands of warnings thanks to this, they might be legitimate as write_bom_data and write_abi_xml_data are our own tasks added in https://github.com/webosose/meta-webosose/blob/master/meta-webos/classes/webos_base.bbclass very long time ago (based on fetchall task which was added in 2006 :)) - there might be better way to add them now, I'll check how to do it better or else I might need to disable this warning locally. Based on "utility-tasks: Drop fetchall and checkuriall tasks" I guess the right way to do the same now will be to convert our scripts to call it with --runall. It's pity that the warning is shown repeatedly. grep -c WARNING:.*dependent.task BUILD/log/cooker/m16p3/console-latest.log 30409 and that was only very small build not even finished yet: Currently 7 running tasks (1142 of 2514) .. WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task write_bom_data does not exist WARNING: cross-localedef-native: dependent task write_abi_xml_data does not exist WARNING: autoconf-archive-native: dependent task do_deploy does not exist WARNING: autoconf-archive-native: dependent task do_package_write_ipk does not exist WARNING: autoconf-archive-native: dependent task do_deploy does not exist WARNING: autoconf-archive-native: dependent task do_package_write_ipk does not exist WARNING: autoconf-archive-native: dependent task write_bom_data does not exist WARNING: autoconf-archive-native: dependent task write_abi_xml_data does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task write_bom_data does not exist WARNING: cross-localedef-native: dependent task write_abi_xml_data does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task write_bom_data does not exist WARNING: cross-localedef-native: dependent task write_abi_xml_data does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task do_deploy does not exist WARNING: cross-localedef-native: dependent task do_package_write_ipk does not exist WARNING: cross-localedef-native: dependent task write_bom_data does not exist WARNING: cross-localedef-native: dependent task write_abi_xml_data does not exist .. On Fri, Apr 26, 2019 at 4:49 AM Robert Yang <liezhi.yang@windriver.com> wrote: > > > On 4/25/19 6:50 PM, Richard Purdie wrote: > > On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote: > >> The following command is incorrect, but was ignored silently, that may > suprise > >> users: > >> > >> addtask task after task_not_existed > >> > >> This patch can check and warn for it. It would be better to also check > "before" > >> tasks, but there is no easier way to do it. > >> > >> [YOCTO #13282] > >> > >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > >> --- > >> bitbake/lib/bb/build.py | 3 +++ > >> 1 file changed, 3 insertions(+) > >> > >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py > >> index 7571421..861e9a9 100644 > >> --- a/bitbake/lib/bb/build.py > >> +++ b/bitbake/lib/bb/build.py > >> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d): > >> task_deps['parents'][task] = [] > >> if 'deps' in flags: > >> for dep in flags['deps']: > >> + # Check and warn for "addtask task after foo" while > foo does not exist > >> + if not dep in tasklist: > >> + bb.warn('%s: dependent task %s does not exist!' % > (d.getVar('PN'), dep)) > >> dep = d.expand(dep) > >> task_deps['parents'][task].append(dep) > > > > I can't help wonder if this change has races, depending on the order of > > the addtask and deltask expressions. Do most layers parse cleanly with > > this? > > The bb.build.add_tasks() is only called by ast.py's finalize(), I think > that > everything should be ready in finalize stage, and there should be no races. > I tested this with meta-oe, python perl, selinux and others, didn't see > any problems (unless I depended on an invalid task). > > > > > At one point bitbake did support "floating" invalid tasks, I can't > > remember if we changed that or not though. > > This warning is good for checking typos during development, I think that > user can call "bb.build.addtask()" dynamically in a function to fix the > warning > for floating tasks? > > // Robert > > > > > Cheers, > > > > Richard > > > > > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel > [-- Attachment #2: Type: text/html, Size: 7461 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-04-26 2:49 ` Robert Yang 2019-04-30 12:13 ` Martin Jansa @ 2019-05-01 10:50 ` Jacob Kroon 2019-05-01 15:30 ` Burton, Ross 1 sibling, 1 reply; 14+ messages in thread From: Jacob Kroon @ 2019-05-01 10:50 UTC (permalink / raw) To: Robert Yang; +Cc: bitbake-devel Hi, On Fri, Apr 26, 2019 at 4:49 AM Robert Yang <liezhi.yang@windriver.com> wrote: > > > > On 4/25/19 6:50 PM, Richard Purdie wrote: > > On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote: > >> The following command is incorrect, but was ignored silently, that may suprise > >> users: > >> > >> addtask task after task_not_existed > >> > >> This patch can check and warn for it. It would be better to also check "before" > >> tasks, but there is no easier way to do it. > >> > >> [YOCTO #13282] > >> > >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > >> --- > >> bitbake/lib/bb/build.py | 3 +++ > >> 1 file changed, 3 insertions(+) > >> > >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py > >> index 7571421..861e9a9 100644 > >> --- a/bitbake/lib/bb/build.py > >> +++ b/bitbake/lib/bb/build.py > >> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d): > >> task_deps['parents'][task] = [] > >> if 'deps' in flags: > >> for dep in flags['deps']: > >> + # Check and warn for "addtask task after foo" while foo does not exist > >> + if not dep in tasklist: > >> + bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep)) > >> dep = d.expand(dep) > >> task_deps['parents'][task].append(dep) > > > > I can't help wonder if this change has races, depending on the order of > > the addtask and deltask expressions. Do most layers parse cleanly with > > this? > > The bb.build.add_tasks() is only called by ast.py's finalize(), I think that > everything should be ready in finalize stage, and there should be no races. > I tested this with meta-oe, python perl, selinux and others, didn't see > any problems (unless I depended on an invalid task). > > > > > At one point bitbake did support "floating" invalid tasks, I can't > > remember if we changed that or not though. > > This warning is good for checking typos during development, I think that > user can call "bb.build.addtask()" dynamically in a function to fix the warning > for floating tasks? > I'm getting a lot of these warnings after removing tmp/ and rebuilding my image from sstate cache. FWIW I use rm_work.bbclass, and regularly use the sstate-cache-management,sh script to prune old cache. /Jacob ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-05-01 10:50 ` Jacob Kroon @ 2019-05-01 15:30 ` Burton, Ross 2019-05-01 16:48 ` Nicolas Dechesne 0 siblings, 1 reply; 14+ messages in thread From: Burton, Ross @ 2019-05-01 15:30 UTC (permalink / raw) To: Jacob Kroon; +Cc: bitbake-devel On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> wrote: > I'm getting a lot of these warnings after removing tmp/ and rebuilding > my image from sstate cache. > FWIW I use rm_work.bbclass, and regularly use the > sstate-cache-management,sh script to prune old cache. There's a patch I hacked up in master-next that shows where these warnings are coming from. Yes, this patch produces two warnings per recipe with rm_work enabled. Ross ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-05-01 15:30 ` Burton, Ross @ 2019-05-01 16:48 ` Nicolas Dechesne 2019-05-01 19:42 ` Richard Purdie 0 siblings, 1 reply; 14+ messages in thread From: Nicolas Dechesne @ 2019-05-01 16:48 UTC (permalink / raw) To: Burton, Ross; +Cc: bitbake-devel On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com> wrote: > > On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> wrote: > > I'm getting a lot of these warnings after removing tmp/ and rebuilding > > my image from sstate cache. > > FWIW I use rm_work.bbclass, and regularly use the > > sstate-cache-management,sh script to prune old cache. > > There's a patch I hacked up in master-next that shows where these > warnings are coming from. Yes, this patch produces two warnings per > recipe with rm_work enabled. argh... i hit the same issue and came up with the exact same conclusion.. i should have checked the mailing list earlier today ;) I am getting thousands of warnings like that with rm_work. I had modified bitbake so that it shows the 'task' name like Ross, in (). WARNING: zip-native: dependent task do_deploy (do_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work) does not exist WARNING: zip-native: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_build_without_rm_work) does not exist WARNING: zip: dependent task do_deploy (do_rm_work) does not exist WARNING: zip: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: zip-native: dependent task do_deploy (do_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work) does not exist WARNING: zip-native: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_build_without_rm_work) does not exist WARNING: zip: dependent task do_deploy (do_rm_work) does not exist WARNING: zip: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: zip-native: dependent task do_deploy (do_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work) does not exist WARNING: zip-native: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: zip-native: dependent task do_package_write_ipk (do_build_without_rm_work) does not exist WARNING: python-cython: dependent task do_deploy (do_rm_work) does not exist WARNING: python-cython: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: python-cython-native: dependent task do_deploy (do_rm_work) does not exist WARNING: python-cython-native: dependent task do_package_write_ipk (do_rm_work) does not exist WARNING: python-cython-native: dependent task do_deploy (do_build_without_rm_work) does not exist WARNING: python-cython-native: dependent task do_package_write_ipk (do_build_without_rm_work) does not exist > > Ross > -- > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-05-01 16:48 ` Nicolas Dechesne @ 2019-05-01 19:42 ` Richard Purdie 2019-05-05 5:56 ` Robert Yang 0 siblings, 1 reply; 14+ messages in thread From: Richard Purdie @ 2019-05-01 19:42 UTC (permalink / raw) To: Nicolas Dechesne, Burton, Ross; +Cc: bitbake-devel On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote: > On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com> > wrote: > > On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> > > wrote: > > > I'm getting a lot of these warnings after removing tmp/ and > > > rebuilding > > > my image from sstate cache. > > > FWIW I use rm_work.bbclass, and regularly use the > > > sstate-cache-management,sh script to prune old cache. > > > > There's a patch I hacked up in master-next that shows where these > > warnings are coming from. Yes, this patch produces two warnings > > per > > recipe with rm_work enabled. > > argh... i hit the same issue and came up with the exact same > conclusion.. i should have checked the mailing list earlier today ;) > > I am getting thousands of warnings like that with rm_work. I had > modified bitbake so that it shows the 'task' name like Ross, in (). I suspect we're going to need to rethink that patch, and/or perhaps rm_work... I did have concerns about it but was assured it was widely tested. We also probably need an rm_work test case on the autobuilder (need a bug for that so we don't forget?). Cheers, Richard ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-05-01 19:42 ` Richard Purdie @ 2019-05-05 5:56 ` Robert Yang 2019-05-05 10:30 ` Robert Yang 0 siblings, 1 reply; 14+ messages in thread From: Robert Yang @ 2019-05-05 5:56 UTC (permalink / raw) To: Richard Purdie, Nicolas Dechesne, Burton, Ross; +Cc: bitbake-devel On 5/2/19 3:42 AM, Richard Purdie wrote: > On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote: >> On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com> >> wrote: >>> On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> >>> wrote: >>>> I'm getting a lot of these warnings after removing tmp/ and >>>> rebuilding >>>> my image from sstate cache. >>>> FWIW I use rm_work.bbclass, and regularly use the >>>> sstate-cache-management,sh script to prune old cache. >>> >>> There's a patch I hacked up in master-next that shows where these >>> warnings are coming from. Yes, this patch produces two warnings >>> per >>> recipe with rm_work enabled. >> >> argh... i hit the same issue and came up with the exact same >> conclusion.. i should have checked the mailing list earlier today ;) >> >> I am getting thousands of warnings like that with rm_work. I had >> modified bitbake so that it shows the 'task' name like Ross, in (). > > I suspect we're going to need to rethink that patch, and/or perhaps > rm_work... Sorry for the warnings, I just came back from the International Labour Day Holiday. Something is wrong with rm_work, I will send a patch to fix it. > > I did have concerns about it but was assured it was widely tested. We > also probably need an rm_work test case on the autobuilder (need a bug > for that so we don't forget?). Rather than add a testcae on autobuilder, how about add a testcase to oe-selftest, please ? // Robert > > Cheers, > > Richard > > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask 2019-05-05 5:56 ` Robert Yang @ 2019-05-05 10:30 ` Robert Yang 0 siblings, 0 replies; 14+ messages in thread From: Robert Yang @ 2019-05-05 10:30 UTC (permalink / raw) To: Richard Purdie, Nicolas Dechesne, Burton, Ross; +Cc: bitbake-devel Hi RP, After more investigations, these warnings are caused by recrdeptask, the rm_work calls: addtask after 'deps + recrdeptask' before do_build Then we will get the warnings, and we can divide these warnings into 3 kinds: 1) deltask only removes task from 'deps', but doesn't remove it from 'recrdeptask', so there are warnings for native recipes like dependent task do_package_write_rpm, although native.bbclass has called "deltask do_package_write_rpm', such a recrdeptask doesn't make any sense for native recipes. But we can't make deltask remove recrdeptask since we may need keep recrdeptask sometimes, for example, the image recipes don't need do_package_write_rpm, but their do_rootfs tasks require all the dependent rpms are done. I think that we need a command such as delrecrdeptask to fix it, but I'm not sure whether it is worth or not. 2) base.bbclass has set 'do_build[recrdeptask] += "do_deploy"', but do_deploy doesn't exist for most of the recipes, so we get the warnings of not do_deploy task, I don't know why do_deploy should be a recrdeptask, and maybe we need set it in rootfs_xxx.bbclass or image.bbclass, or set it conditionally in a func. 3) gcc-source doesn't have do_populate_sysroot or do_populate_lic, but do_build_without_rm_work always depends on them: bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d) so we get warnings that gcc-source has no dependent task do_populate_sysroot or do_populate_lic. I think that we can make do_build_without_rm_work depend on them conditionally to fix the problem. Any comments are appreciated. // Robert On 5/5/19 1:56 PM, Robert Yang wrote: > > > On 5/2/19 3:42 AM, Richard Purdie wrote: >> On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote: >>> On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com> >>> wrote: >>>> On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> >>>> wrote: >>>>> I'm getting a lot of these warnings after removing tmp/ and >>>>> rebuilding >>>>> my image from sstate cache. >>>>> FWIW I use rm_work.bbclass, and regularly use the >>>>> sstate-cache-management,sh script to prune old cache. >>>> >>>> There's a patch I hacked up in master-next that shows where these >>>> warnings are coming from. Yes, this patch produces two warnings >>>> per >>>> recipe with rm_work enabled. >>> >>> argh... i hit the same issue and came up with the exact same >>> conclusion.. i should have checked the mailing list earlier today ;) >>> >>> I am getting thousands of warnings like that with rm_work. I had >>> modified bitbake so that it shows the 'task' name like Ross, in (). >> >> I suspect we're going to need to rethink that patch, and/or perhaps >> rm_work... > > > Sorry for the warnings, I just came back from the International Labour Day > Holiday. Something is wrong with rm_work, I will send a patch to fix it. > >> >> I did have concerns about it but was assured it was widely tested. We >> also probably need an rm_work test case on the autobuilder (need a bug >> for that so we don't forget?). > > Rather than add a testcae on autobuilder, how about add a testcase to > oe-selftest, please ? > > // Robert > >> >> Cheers, >> >> Richard >> >> >> ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2019-05-05 10:31 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-25 10:00 [PATCH 0/2] Fixes for addtask and deltask Robert Yang 2019-04-25 10:00 ` [PATCH 1/2] bitbake: BBHandler: Fix " Robert Yang 2019-04-25 10:48 ` Richard Purdie 2019-04-26 3:17 ` Robert Yang 2019-04-25 10:01 ` [PATCH 2/2] bitbake: build.py: check dependendent task for addtask Robert Yang 2019-04-25 10:50 ` Richard Purdie 2019-04-26 2:49 ` Robert Yang 2019-04-30 12:13 ` Martin Jansa 2019-05-01 10:50 ` Jacob Kroon 2019-05-01 15:30 ` Burton, Ross 2019-05-01 16:48 ` Nicolas Dechesne 2019-05-01 19:42 ` Richard Purdie 2019-05-05 5:56 ` Robert Yang 2019-05-05 10:30 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.