[PATCH 0/2] Fixes for addtask and deltask

[PATCH 0/2] Fixes for addtask and deltask

[Robert Yang]

The following changes since commit 7f7a7cba4dcd99d7b36a0d747bfd1bba3f9c2288:

  bitbake: bb: siggen: Print more info when basehash are mis-matched (2019-04-23 23:30:45 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/addtask
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/addtask

Robert Yang (2):
  bitbake: BBHandler: Fix addtask and deltask
  bitbake: build.py: check dependendent task for addtask

 bitbake/lib/bb/build.py                    |  3 +++
 bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 ++++++++++++++++++++++++-
 2 files changed, 27 insertions(+), 1 deletion(-)

-- 
2.7.4

[PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask

[Robert Yang]

The following commands are not supported, but they were ignored silently, that
may suprise users:

* addtask task1 task2
  task2 is ignored

* addtask task1 before task2 before task3
  Should be: addtask task1 before task2 task3

* addtask task1 after task2 after task3
  Should be: addtask task1 after task2 task3

* deltask task1 task2
  task2 is ignore

This patch can check and warn for them.

[YOCTO #13282]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/parse/parse_py/BBHandler.py | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index 9dba5f2..4d90ff0 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -42,7 +42,7 @@ __func_start_regexp__    = re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(
 __inherit_regexp__       = re.compile(r"inherit\s+(.+)" )
 __export_func_regexp__   = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
 __addtask_regexp__       = re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
-__deltask_regexp__       = re.compile(r"deltask\s+(?P<func>\w+)")
+__deltask_regexp__       = re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
 __addhandler_regexp__    = re.compile(r"addhandler\s+(.+)" )
 __def_regexp__           = re.compile(r"def\s+(\w+).*:" )
 __python_func_regexp__   = re.compile(r"(\s+.*)|(^$)|(^#)" )
@@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements, eof=False):
 
     m = __addtask_regexp__.match(s)
     if m:
+        if len(m.group().split()) == 2:
+            # Check and warn for "addtask task1 task2"
+            m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)", s)
+            if m2 and m2.group('ignores'):
+                logger.warning('addtask ignored: "%s"' % m2.group('ignores'))
+
+        # Check and warn for "addtask task1 before task2 before task3", the
+        # similar to "after"
+        dup_after = 0
+        dup_before = 0
+        for tmp in s.split():
+            if 'before' == tmp:
+                dup_before += 1
+            if 'after' == tmp:
+                dup_after += 1
+        if dup_after > 1:
+            logger.warning('addtask found more than 1 keyword "after": "%s"' % s)
+        if dup_before > 1:
+            logger.warning('addtask found more than 1 keyword "before": "%s"' % s)
+
         ast.handleAddTask(statements, fn, lineno, m)
         return
 
     m = __deltask_regexp__.match(s)
     if m:
+        # Check and warn "for deltask task1 task2"
+        if m.group('ignores'):
+            logger.warning('deltask ignored: "%s"' % m.group('ignores'))
         ast.handleDelTask(statements, fn, lineno, m)
         return
 
-- 
2.7.4

[PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Robert Yang]

The following command is incorrect, but was ignored silently, that may suprise
users:

addtask task after task_not_existed

This patch can check and warn for it. It would be better to also check "before"
tasks, but there is no easier way to do it.

[YOCTO #13282]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/build.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 7571421..861e9a9 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -815,6 +815,9 @@ def add_tasks(tasklist, d):
         task_deps['parents'][task] = []
         if 'deps' in flags:
             for dep in flags['deps']:
+                # Check and warn for "addtask task after foo" while foo does not exist
+                if not dep in tasklist:
+                    bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep))
                 dep = d.expand(dep)
                 task_deps['parents'][task].append(dep)
 
-- 
2.7.4

Re: [PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask

[Richard Purdie]

On Thu, 2019-04-25 at 18:00 +0800, Robert Yang wrote:
> The following commands are not supported, but they were ignored
> silently, that
> may suprise users:
> 
> * addtask task1 task2
>   task2 is ignored
> 
> * addtask task1 before task2 before task3
>   Should be: addtask task1 before task2 task3
> 
> * addtask task1 after task2 after task3
>   Should be: addtask task1 after task2 task3
> 
> * deltask task1 task2
>   task2 is ignore
> 
> This patch can check and warn for them.
> 
> [YOCTO #13282]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/parse/parse_py/BBHandler.py | 25
> ++++++++++++++++++++++++-
>  1 file changed, 24 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> b/bitbake/lib/bb/parse/parse_py/BBHandler.py
> index 9dba5f2..4d90ff0 100644
> --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
> @@ -42,7 +42,7 @@ __func_start_regexp__    =
> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(
>  __inherit_regexp__       = re.compile(r"inherit\s+(.+)" )
>  __export_func_regexp__   = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
>  __addtask_regexp__       =
> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=a
> fter))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
> -__deltask_regexp__       = re.compile(r"deltask\s+(?P<func>\w+)")
> +__deltask_regexp__       =
> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
>  __addhandler_regexp__    = re.compile(r"addhandler\s+(.+)" )
>  __def_regexp__           = re.compile(r"def\s+(\w+).*:" )
>  __python_func_regexp__   = re.compile(r"(\s+.*)|(^$)|(^#)" )
> @@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements,
> eof=False):
>  
>      m = __addtask_regexp__.match(s)
>      if m:
> +        if len(m.group().split()) == 2:
> +            # Check and warn for "addtask task1 task2"
> +            m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
> s)
> +            if m2 and m2.group('ignores'):
> +                logger.warning('addtask ignored: "%s"' %
> m2.group('ignores'))
> +
> +        # Check and warn for "addtask task1 before task2 before
> task3", the
> +        # similar to "after"
> +        dup_after = 0
> +        dup_before = 0
> +        for tmp in s.split():
> +            if 'before' == tmp:
> +                dup_before += 1
> +            if 'after' == tmp:
> +                dup_after += 1
> +        if dup_after > 1:

taskexpression = s.split()
if taskexpression.count("after") > 1:
   
logger.warning("X")
if taskexpression.count("before") > 1:
   
logger.warning("Y")

> +            logger.warning('addtask found more than 1 keyword
> "after": "%s"' % s)

How about:

addtask contained multiple 'after' keywords, only one is supported

> +        if dup_before > 1:
> +            logger.warning('addtask found more than 1 keyword
> "before": "%s"' % s)

and:

addtask contained multiple 'before' keywords, only one is supported

>          ast.handleAddTask(statements, fn, lineno, m)
>          return
>  
>      m = __deltask_regexp__.match(s)
>      if m:
> +        # Check and warn "for deltask task1 task2"
> +        if m.group('ignores'):
> +            logger.warning('deltask ignored: "%s"' %
> m.group('ignores'))

deltask listing multiple tasks is not supported

I did wonder if we should support multiple tasks for deltask?

Also, we might want to consider adding something to lib/bb/tests/ for
this?

Cheers,

Richard

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Richard Purdie]

On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote:
> The following command is incorrect, but was ignored silently, that may suprise
> users:
> 
> addtask task after task_not_existed
> 
> This patch can check and warn for it. It would be better to also check "before"
> tasks, but there is no easier way to do it.
> 
> [YOCTO #13282]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/build.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> index 7571421..861e9a9 100644
> --- a/bitbake/lib/bb/build.py
> +++ b/bitbake/lib/bb/build.py
> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d):
>          task_deps['parents'][task] = []
>          if 'deps' in flags:
>              for dep in flags['deps']:
> +                # Check and warn for "addtask task after foo" while foo does not exist
> +                if not dep in tasklist:
> +                    bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep))
>                  dep = d.expand(dep)
>                  task_deps['parents'][task].append(dep)

I can't help wonder if this change has races, depending on the order of
the addtask and deltask expressions. Do most layers parse cleanly with
this?

At one point bitbake did support "floating" invalid tasks, I can't
remember if we changed that or not though.

Cheers,

Richard

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Robert Yang]

On 4/25/19 6:50 PM, Richard Purdie wrote:
> On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote:
>> The following command is incorrect, but was ignored silently, that may suprise
>> users:
>>
>> addtask task after task_not_existed
>>
>> This patch can check and warn for it. It would be better to also check "before"
>> tasks, but there is no easier way to do it.
>>
>> [YOCTO #13282]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/build.py | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
>> index 7571421..861e9a9 100644
>> --- a/bitbake/lib/bb/build.py
>> +++ b/bitbake/lib/bb/build.py
>> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d):
>>           task_deps['parents'][task] = []
>>           if 'deps' in flags:
>>               for dep in flags['deps']:
>> +                # Check and warn for "addtask task after foo" while foo does not exist
>> +                if not dep in tasklist:
>> +                    bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep))
>>                   dep = d.expand(dep)
>>                   task_deps['parents'][task].append(dep)
> 
> I can't help wonder if this change has races, depending on the order of
> the addtask and deltask expressions. Do most layers parse cleanly with
> this?

The bb.build.add_tasks() is only called by ast.py's finalize(), I think that
everything should be ready in finalize stage, and there should be no races.
I tested this with meta-oe, python perl, selinux and others, didn't see
any problems (unless I depended on an invalid task).

> 
> At one point bitbake did support "floating" invalid tasks, I can't
> remember if we changed that or not though.

This warning is good for checking typos during development, I think that
user can call "bb.build.addtask()" dynamically in a function to fix the warning
for floating tasks?

// Robert

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 1/2] bitbake: BBHandler: Fix addtask and deltask

[Robert Yang]

On 4/25/19 6:48 PM, Richard Purdie wrote:
> On Thu, 2019-04-25 at 18:00 +0800, Robert Yang wrote:
>> The following commands are not supported, but they were ignored
>> silently, that
>> may suprise users:
>>
>> * addtask task1 task2
>>    task2 is ignored
>>
>> * addtask task1 before task2 before task3
>>    Should be: addtask task1 before task2 task3
>>
>> * addtask task1 after task2 after task3
>>    Should be: addtask task1 after task2 task3
>>
>> * deltask task1 task2
>>    task2 is ignore
>>
>> This patch can check and warn for them.
>>
>> [YOCTO #13282]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/parse/parse_py/BBHandler.py | 25
>> ++++++++++++++++++++++++-
>>   1 file changed, 24 insertions(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> b/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> index 9dba5f2..4d90ff0 100644
>> --- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> +++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
>> @@ -42,7 +42,7 @@ __func_start_regexp__    =
>> re.compile(r"(((?P<py>python)|(?P<fr>fakeroot))\s*)*(
>>   __inherit_regexp__       = re.compile(r"inherit\s+(.+)" )
>>   __export_func_regexp__   = re.compile(r"EXPORT_FUNCTIONS\s+(.+)" )
>>   __addtask_regexp__       =
>> re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=a
>> fter))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")
>> -__deltask_regexp__       = re.compile(r"deltask\s+(?P<func>\w+)")
>> +__deltask_regexp__       =
>> re.compile(r"deltask\s+(?P<func>\w+)(?P<ignores>.*)")
>>   __addhandler_regexp__    = re.compile(r"addhandler\s+(.+)" )
>>   __def_regexp__           = re.compile(r"def\s+(\w+).*:" )
>>   __python_func_regexp__   = re.compile(r"(\s+.*)|(^$)|(^#)" )
>> @@ -236,11 +236,34 @@ def feeder(lineno, s, fn, root, statements,
>> eof=False):
>>   
>>       m = __addtask_regexp__.match(s)
>>       if m:
>> +        if len(m.group().split()) == 2:
>> +            # Check and warn for "addtask task1 task2"
>> +            m2 = re.match(r"addtask\s+(?P<func>\w+)(?P<ignores>.*)",
>> s)
>> +            if m2 and m2.group('ignores'):
>> +                logger.warning('addtask ignored: "%s"' %
>> m2.group('ignores'))
>> +
>> +        # Check and warn for "addtask task1 before task2 before
>> task3", the
>> +        # similar to "after"
>> +        dup_after = 0
>> +        dup_before = 0
>> +        for tmp in s.split():
>> +            if 'before' == tmp:
>> +                dup_before += 1
>> +            if 'after' == tmp:
>> +                dup_after += 1
>> +        if dup_after > 1:
> 
> taskexpression = s.split()
> if taskexpression.count("after") > 1:
>     
> logger.warning("X")
> if taskexpression.count("before") > 1:
>     
> logger.warning("Y")
> 
>> +            logger.warning('addtask found more than 1 keyword
>> "after": "%s"' % s)
> 
> How about:
> 
> addtask contained multiple 'after' keywords, only one is supported
> 
>> +        if dup_before > 1:
>> +            logger.warning('addtask found more than 1 keyword
>> "before": "%s"' % s)
> 
> and:
> 
> addtask contained multiple 'before' keywords, only one is supported

Thanks for all the replies above, I will fix them.

> 
>>           ast.handleAddTask(statements, fn, lineno, m)
>>           return
>>   
>>       m = __deltask_regexp__.match(s)
>>       if m:
>> +        # Check and warn "for deltask task1 task2"
>> +        if m.group('ignores'):
>> +            logger.warning('deltask ignored: "%s"' %
>> m.group('ignores'))
> 
> deltask listing multiple tasks is not supported
> 
> I did wonder if we should support multiple tasks for deltask?

Yes, that's a good idea, but if we support "deltask multiple tasks",
I think that we should also support "addtask multiple tasks" since
they are a pair, but I have other concerns on addtask, for example,
the following code is hard to read and maintain:

__addtask_regexp__       = 
re.compile(r"addtask\s+(?P<func>\w+)\s*((before\s*(?P<before>((.*(?=after))|(.*))))|(after\s*(?P<after>((.*(?=before))|(.*)))))*")

I think that we need break it into small pieces. so I gave it up with these
concerns which looks risky.

> 
> Also, we might want to consider adding something to lib/bb/tests/ for
> this?

Thanks, I will add it.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 5722 bytes --]

I just got few thousands of warnings thanks to this, they might be
legitimate as write_bom_data and write_abi_xml_data are our own tasks added
in
https://github.com/webosose/meta-webosose/blob/master/meta-webos/classes/webos_base.bbclass
very
long time ago (based on fetchall task which was added in 2006 :)) - there
might be better way to add them now, I'll check how to do it better or else
I might need to disable this warning locally.

Based on "utility-tasks: Drop fetchall and checkuriall tasks" I guess the
right way to do the same now will be to convert our scripts to call it
with --runall.

It's pity that the warning is shown repeatedly.

grep -c WARNING:.*dependent.task BUILD/log/cooker/m16p3/console-latest.log
30409

and that was only very small build not even finished yet:
Currently  7 running tasks (1142 of 2514)

..
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task write_bom_data does not
exist
WARNING: cross-localedef-native: dependent task write_abi_xml_data does not
exist
WARNING: autoconf-archive-native: dependent task do_deploy does not exist
WARNING: autoconf-archive-native: dependent task do_package_write_ipk does
not exist
WARNING: autoconf-archive-native: dependent task do_deploy does not exist
WARNING: autoconf-archive-native: dependent task do_package_write_ipk does
not exist
WARNING: autoconf-archive-native: dependent task write_bom_data does not
exist
WARNING: autoconf-archive-native: dependent task write_abi_xml_data does
not exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task write_bom_data does not
exist
WARNING: cross-localedef-native: dependent task write_abi_xml_data does not
exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task write_bom_data does not
exist
WARNING: cross-localedef-native: dependent task write_abi_xml_data does not
exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task do_deploy does not exist
WARNING: cross-localedef-native: dependent task do_package_write_ipk does
not exist
WARNING: cross-localedef-native: dependent task write_bom_data does not
exist
WARNING: cross-localedef-native: dependent task write_abi_xml_data does not
exist
..

On Fri, Apr 26, 2019 at 4:49 AM Robert Yang <liezhi.yang@windriver.com>
wrote:

>
>
> On 4/25/19 6:50 PM, Richard Purdie wrote:
> > On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote:
> >> The following command is incorrect, but was ignored silently, that may
> suprise
> >> users:
> >>
> >> addtask task after task_not_existed
> >>
> >> This patch can check and warn for it. It would be better to also check
> "before"
> >> tasks, but there is no easier way to do it.
> >>
> >> [YOCTO #13282]
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> ---
> >>   bitbake/lib/bb/build.py | 3 +++
> >>   1 file changed, 3 insertions(+)
> >>
> >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> >> index 7571421..861e9a9 100644
> >> --- a/bitbake/lib/bb/build.py
> >> +++ b/bitbake/lib/bb/build.py
> >> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d):
> >>           task_deps['parents'][task] = []
> >>           if 'deps' in flags:
> >>               for dep in flags['deps']:
> >> +                # Check and warn for "addtask task after foo" while
> foo does not exist
> >> +                if not dep in tasklist:
> >> +                    bb.warn('%s: dependent task %s does not exist!' %
> (d.getVar('PN'), dep))
> >>                   dep = d.expand(dep)
> >>                   task_deps['parents'][task].append(dep)
> >
> > I can't help wonder if this change has races, depending on the order of
> > the addtask and deltask expressions. Do most layers parse cleanly with
> > this?
>
> The bb.build.add_tasks() is only called by ast.py's finalize(), I think
> that
> everything should be ready in finalize stage, and there should be no races.
> I tested this with meta-oe, python perl, selinux and others, didn't see
> any problems (unless I depended on an invalid task).
>
> >
> > At one point bitbake did support "floating" invalid tasks, I can't
> > remember if we changed that or not though.
>
> This warning is good for checking typos during development, I think that
> user can call "bb.build.addtask()" dynamically in a function to fix the
> warning
> for floating tasks?
>
> // Robert
>
> >
> > Cheers,
> >
> > Richard
> >
> >
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 7461 bytes --]

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Jacob Kroon]

Hi,

On Fri, Apr 26, 2019 at 4:49 AM Robert Yang <liezhi.yang@windriver.com> wrote:
>
>
>
> On 4/25/19 6:50 PM, Richard Purdie wrote:
> > On Thu, 2019-04-25 at 18:01 +0800, Robert Yang wrote:
> >> The following command is incorrect, but was ignored silently, that may suprise
> >> users:
> >>
> >> addtask task after task_not_existed
> >>
> >> This patch can check and warn for it. It would be better to also check "before"
> >> tasks, but there is no easier way to do it.
> >>
> >> [YOCTO #13282]
> >>
> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> >> ---
> >>   bitbake/lib/bb/build.py | 3 +++
> >>   1 file changed, 3 insertions(+)
> >>
> >> diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
> >> index 7571421..861e9a9 100644
> >> --- a/bitbake/lib/bb/build.py
> >> +++ b/bitbake/lib/bb/build.py
> >> @@ -815,6 +815,9 @@ def add_tasks(tasklist, d):
> >>           task_deps['parents'][task] = []
> >>           if 'deps' in flags:
> >>               for dep in flags['deps']:
> >> +                # Check and warn for "addtask task after foo" while foo does not exist
> >> +                if not dep in tasklist:
> >> +                    bb.warn('%s: dependent task %s does not exist!' % (d.getVar('PN'), dep))
> >>                   dep = d.expand(dep)
> >>                   task_deps['parents'][task].append(dep)
> >
> > I can't help wonder if this change has races, depending on the order of
> > the addtask and deltask expressions. Do most layers parse cleanly with
> > this?
>
> The bb.build.add_tasks() is only called by ast.py's finalize(), I think that
> everything should be ready in finalize stage, and there should be no races.
> I tested this with meta-oe, python perl, selinux and others, didn't see
> any problems (unless I depended on an invalid task).
>
> >
> > At one point bitbake did support "floating" invalid tasks, I can't
> > remember if we changed that or not though.
>
> This warning is good for checking typos during development, I think that
> user can call "bb.build.addtask()" dynamically in a function to fix the warning
> for floating tasks?
>

I'm getting a lot of these warnings after removing tmp/ and rebuilding
my image from sstate cache.
FWIW I use rm_work.bbclass, and regularly use the
sstate-cache-management,sh script to prune old cache.

/Jacob

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Burton, Ross]

On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> wrote:
> I'm getting a lot of these warnings after removing tmp/ and rebuilding
> my image from sstate cache.
> FWIW I use rm_work.bbclass, and regularly use the
> sstate-cache-management,sh script to prune old cache.

There's a patch I hacked up in master-next that shows where these
warnings are coming from.  Yes, this patch produces two warnings per
recipe with rm_work enabled.

Ross

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Nicolas Dechesne]

On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com> wrote:
>
> On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com> wrote:
> > I'm getting a lot of these warnings after removing tmp/ and rebuilding
> > my image from sstate cache.
> > FWIW I use rm_work.bbclass, and regularly use the
> > sstate-cache-management,sh script to prune old cache.
>
> There's a patch I hacked up in master-next that shows where these
> warnings are coming from.  Yes, this patch produces two warnings per
> recipe with rm_work enabled.

argh... i hit the same issue and came up with the exact same
conclusion.. i should have checked the mailing list earlier today ;)

I am getting thousands of warnings like that with rm_work. I had
modified bitbake so that it shows the 'task' name like Ross, in ().

WARNING: zip-native: dependent task do_deploy (do_rm_work) does not exist
WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work)
does not exist
WARNING: zip-native: dependent task do_deploy
(do_build_without_rm_work) does not exist
WARNING: zip-native: dependent task do_package_write_ipk
(do_build_without_rm_work) does not exist
WARNING: zip: dependent task do_deploy (do_rm_work) does not exist
WARNING: zip: dependent task do_deploy (do_build_without_rm_work) does
not exist
WARNING: zip-native: dependent task do_deploy (do_rm_work) does not
exist
WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work)
does not exist
WARNING: zip-native: dependent task do_deploy
(do_build_without_rm_work) does not exist
WARNING: zip-native: dependent task do_package_write_ipk
(do_build_without_rm_work) does not exist
WARNING: zip: dependent task do_deploy (do_rm_work) does not exist
WARNING: zip: dependent task do_deploy (do_build_without_rm_work) does
not exist
WARNING: zip-native: dependent task do_deploy (do_rm_work) does not exist
WARNING: zip-native: dependent task do_package_write_ipk (do_rm_work)
does not exist
WARNING: zip-native: dependent task do_deploy
(do_build_without_rm_work) does not exist
WARNING: zip-native: dependent task do_package_write_ipk
(do_build_without_rm_work) does not exist
WARNING: python-cython: dependent task do_deploy (do_rm_work) does not exist
WARNING: python-cython: dependent task do_deploy
(do_build_without_rm_work) does not exist
WARNING: python-cython-native: dependent task do_deploy (do_rm_work)
does not exist
WARNING: python-cython-native: dependent task do_package_write_ipk
(do_rm_work) does not exist
WARNING: python-cython-native: dependent task do_deploy
(do_build_without_rm_work) does not exist
WARNING: python-cython-native: dependent task do_package_write_ipk
(do_build_without_rm_work) does not exist

>
> Ross
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Richard Purdie]

On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote:
> On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com>
> wrote:
> > On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com>
> > wrote:
> > > I'm getting a lot of these warnings after removing tmp/ and
> > > rebuilding
> > > my image from sstate cache.
> > > FWIW I use rm_work.bbclass, and regularly use the
> > > sstate-cache-management,sh script to prune old cache.
> > 
> > There's a patch I hacked up in master-next that shows where these
> > warnings are coming from.  Yes, this patch produces two warnings
> > per
> > recipe with rm_work enabled.
> 
> argh... i hit the same issue and came up with the exact same
> conclusion.. i should have checked the mailing list earlier today ;)
> 
> I am getting thousands of warnings like that with rm_work. I had
> modified bitbake so that it shows the 'task' name like Ross, in ().

I suspect we're going to need to rethink that patch, and/or perhaps
rm_work...

I did have concerns about it but was assured it was widely tested. We
also probably need an rm_work test case on the autobuilder (need a bug
for that so we don't forget?).

Cheers,

Richard

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Robert Yang]

On 5/2/19 3:42 AM, Richard Purdie wrote:
> On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote:
>> On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com>
>> wrote:
>>> On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com>
>>> wrote:
>>>> I'm getting a lot of these warnings after removing tmp/ and
>>>> rebuilding
>>>> my image from sstate cache.
>>>> FWIW I use rm_work.bbclass, and regularly use the
>>>> sstate-cache-management,sh script to prune old cache.
>>>
>>> There's a patch I hacked up in master-next that shows where these
>>> warnings are coming from.  Yes, this patch produces two warnings
>>> per
>>> recipe with rm_work enabled.
>>
>> argh... i hit the same issue and came up with the exact same
>> conclusion.. i should have checked the mailing list earlier today ;)
>>
>> I am getting thousands of warnings like that with rm_work. I had
>> modified bitbake so that it shows the 'task' name like Ross, in ().
> 
> I suspect we're going to need to rethink that patch, and/or perhaps
> rm_work...


Sorry for the warnings, I just came back from the International Labour Day
Holiday. Something is wrong with rm_work, I will send a patch to fix it.

> 
> I did have concerns about it but was assured it was widely tested. We
> also probably need an rm_work test case on the autobuilder (need a bug
> for that so we don't forget?).

Rather than add a testcae on autobuilder, how about add a testcase to
oe-selftest, please ?

// Robert

> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [PATCH 2/2] bitbake: build.py: check dependendent task for addtask

[Robert Yang]

Hi RP,

After more investigations, these warnings are caused by recrdeptask,
the rm_work calls:

addtask after 'deps + recrdeptask' before do_build

Then we will get the warnings, and we can divide these warnings into 3 kinds:

1) deltask only removes task from 'deps', but doesn't remove it from
    'recrdeptask', so there are warnings for native recipes like dependent
    task do_package_write_rpm, although native.bbclass has called
    "deltask do_package_write_rpm', such a recrdeptask doesn't make any sense
    for native recipes. But we can't make deltask remove recrdeptask since we may
    need keep recrdeptask sometimes, for example, the image recipes don't need
    do_package_write_rpm, but their do_rootfs tasks require all the dependent
    rpms are done. I think that we need a command such as delrecrdeptask to fix
    it, but I'm not sure whether it is worth or not.

2) base.bbclass has set 'do_build[recrdeptask] += "do_deploy"', but do_deploy
    doesn't exist for most of the recipes, so we get the warnings of not
    do_deploy task, I don't know why do_deploy should be a recrdeptask, and
    maybe we need set it in rootfs_xxx.bbclass or image.bbclass, or set it
    conditionally in a func.

3) gcc-source doesn't have do_populate_sysroot or do_populate_lic, but
    do_build_without_rm_work always depends on them:

    bb.build.addtask('do_build_without_rm_work', '', ' '.join(deps), d)

    so we get warnings that gcc-source has no dependent task do_populate_sysroot
    or do_populate_lic. I think that we can make do_build_without_rm_work
    depend on them conditionally to fix the problem.

Any comments are appreciated.

// Robert


On 5/5/19 1:56 PM, Robert Yang wrote:
> 
> 
> On 5/2/19 3:42 AM, Richard Purdie wrote:
>> On Wed, 2019-05-01 at 18:48 +0200, Nicolas Dechesne wrote:
>>> On Wed, May 1, 2019 at 5:30 PM Burton, Ross <ross.burton@intel.com>
>>> wrote:
>>>> On Wed, 1 May 2019 at 11:51, Jacob Kroon <jacob.kroon@gmail.com>
>>>> wrote:
>>>>> I'm getting a lot of these warnings after removing tmp/ and
>>>>> rebuilding
>>>>> my image from sstate cache.
>>>>> FWIW I use rm_work.bbclass, and regularly use the
>>>>> sstate-cache-management,sh script to prune old cache.
>>>>
>>>> There's a patch I hacked up in master-next that shows where these
>>>> warnings are coming from.  Yes, this patch produces two warnings
>>>> per
>>>> recipe with rm_work enabled.
>>>
>>> argh... i hit the same issue and came up with the exact same
>>> conclusion.. i should have checked the mailing list earlier today ;)
>>>
>>> I am getting thousands of warnings like that with rm_work. I had
>>> modified bitbake so that it shows the 'task' name like Ross, in ().
>>
>> I suspect we're going to need to rethink that patch, and/or perhaps
>> rm_work...
> 
> 
> Sorry for the warnings, I just came back from the International Labour Day
> Holiday. Something is wrong with rm_work, I will send a patch to fix it.
> 
>>
>> I did have concerns about it but was assured it was widely tested. We
>> also probably need an rm_work test case on the autobuilder (need a bug
>> for that so we don't forget?).
> 
> Rather than add a testcae on autobuilder, how about add a testcase to
> oe-selftest, please ?
> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>
>>
>>