[PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

The following changes since commit c0dc72bad92e2e4dc0aedb48b969709f821baf73:

  coreutils: Fix patch upstream status field (2019-05-08 23:36:21 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/python
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/python

Robert Yang (1):
  bitbake: cookerdata: Avoid double exceptions for bb.fatal()

 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

-- 
2.7.4

[PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

The bb.fatal() raises BBHandledException() which causes double exceptions,
e.g.:

Add 'HOSTTOOLS += "hello"' to conf/local.conf:
$ bitbake -p
[snip]
During handling of the above exception, another exception occurred:
[snip]
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  hello

Use "raise" rather than "raise bb.BBHandledException" to fix the double
exceptions.

[YOCTO #13267]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cookerdata.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index f8ae410..585edc5 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -301,7 +301,9 @@ class CookerDataBuilder(object):
             if multiconfig:
                 bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data)
 
-        except (SyntaxError, bb.BBHandledException):
+        except bb.BBHandledException:
+            raise
+        except SyntaxError:
             raise bb.BBHandledException
         except bb.data_smart.ExpansionError as e:
             logger.error(str(e))
-- 
2.7.4

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Richard Purdie]

On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
> The bb.fatal() raises BBHandledException() which causes double exceptions,
> e.g.:
> 
> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
> $ bitbake -p
> [snip]
> During handling of the above exception, another exception occurred:
> [snip]
> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>   hello
> 
> Use "raise" rather than "raise bb.BBHandledException" to fix the double
> exceptions.
> 
> [YOCTO #13267]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/cookerdata.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cookerdata.py
> b/bitbake/lib/bb/cookerdata.py
> index f8ae410..585edc5 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>              if multiconfig:
>                  bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
> ), self.data)
>  
> -        except (SyntaxError, bb.BBHandledException):
> +        except bb.BBHandledException:
> +            raise
> +        except SyntaxError:
>              raise bb.BBHandledException
>          except bb.data_smart.ExpansionError as e:
>              logger.error(str(e))

Hi Robert,

This doesn't sound right, where is this exception being printed a
second time? The point of "BBHandledException" is to say "don't print
any further traces for this exception". If this fixes the bug, it means
something somewhere is printing a trace for a second time when it
should pass through BBHandledException?

Cheers,

Richard

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 5/12/19 4:28 PM, Richard Purdie wrote:
> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>> e.g.:
>>
>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>> $ bitbake -p
>> [snip]
>> During handling of the above exception, another exception occurred:
>> [snip]
>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>>    hello
>>
>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>> exceptions.
>>
>> [YOCTO #13267]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py
>> index f8ae410..585edc5 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>               if multiconfig:
>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>> ), self.data)
>>   
>> -        except (SyntaxError, bb.BBHandledException):
>> +        except bb.BBHandledException:
>> +            raise
>> +        except SyntaxError:
>>               raise bb.BBHandledException
>>           except bb.data_smart.ExpansionError as e:
>>               logger.error(str(e))
> 
> Hi Robert,
> 
> This doesn't sound right, where is this exception being printed a
> second time? The point of "BBHandledException" is to say "don't print
> any further traces for this exception". If this fixes the bug, it means
> something somewhere is printing a trace for a second time when it
> should pass through BBHandledException?

Hi RP,

I found another serious problem when tried to raise BBHandledException. There
is a deadlock when a recipe is failed to parse, e.g.:

$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ bitbake -p
meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
[hangs]

Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
are still existed, and we need kill them manually, otherwise we can't start
bitbake again.

I think that there are two problems:
1) The cooker.py:parse_generator() raises an exception which makes
    one of result_queue or parser_quit have a deadlock, then
    causes the shutdown():process.join() hanged.

2) The sub processes, parser_quit and result_queue are running dependently,
    we can't make sure that cooker.py:Parser():realrun() can get data from
    parser_quit, so there is a race issue, the Parser's subprocess maybe already
    exited before parser_quit sends data to it.

I think that we can move part of parse_next()'s code into parse_generator()
to fix the deadlock problem, and we may need change the working way of
parser_quit. But I'm not sure, I need your suggestions before work on it.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 5/14/19 7:02 PM, Robert Yang wrote:
> 
> 
> On 5/12/19 4:28 PM, Richard Purdie wrote:
>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>>> e.g.:
>>>
>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>> $ bitbake -p
>>> [snip]
>>> During handling of the above exception, another exception occurred:
>>> [snip]
>>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be 
>>> unavailable in PATH, please install them in order to proceed:
>>>    hello
>>>
>>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>>> exceptions.
>>>
>>> [YOCTO #13267]
>>>
>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>> ---
>>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>> b/bitbake/lib/bb/cookerdata.py
>>> index f8ae410..585edc5 100644
>>> --- a/bitbake/lib/bb/cookerdata.py
>>> +++ b/bitbake/lib/bb/cookerdata.py
>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>               if multiconfig:
>>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>> ), self.data)
>>> -        except (SyntaxError, bb.BBHandledException):
>>> +        except bb.BBHandledException:
>>> +            raise
>>> +        except SyntaxError:
>>>               raise bb.BBHandledException
>>>           except bb.data_smart.ExpansionError as e:
>>>               logger.error(str(e))
>>
>> Hi Robert,
>>
>> This doesn't sound right, where is this exception being printed a
>> second time? The point of "BBHandledException" is to say "don't print
>> any further traces for this exception". If this fixes the bug, it means
>> something somewhere is printing a trace for a second time when it
>> should pass through BBHandledException?
> 
> Hi RP,
> 
> I found another serious problem when tried to raise BBHandledException. There
> is a deadlock when a recipe is failed to parse, e.g.:
> 
> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
> $ bitbake -p
> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
> [hangs]
> 
> Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
> are still existed, and we need kill them manually, otherwise we can't start
> bitbake again.
> 
> I think that there are two problems:
> 1) The cooker.py:parse_generator() raises an exception which makes
>     one of result_queue or parser_quit have a deadlock, then
>     causes the shutdown():process.join() hanged.
> 
> 2) The sub processes, parser_quit and result_queue are running dependently,
>     we can't make sure that cooker.py:Parser():realrun() can get data from
>     parser_quit, so there is a race issue, the Parser's subprocess maybe already
>     exited before parser_quit sends data to it.
> 
> I think that we can move part of parse_next()'s code into parse_generator()
> to fix the deadlock problem, and we may need change the working way of
> parser_quit. But I'm not sure, I need your suggestions before work on it.

Hi RP,

Do you have any suggestions, please?

// Robert

> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>
>>

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

Hi RP,

On 5/12/19 4:28 PM, Richard Purdie wrote:
> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>> e.g.:
>>
>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>> $ bitbake -p
>> [snip]
>> During handling of the above exception, another exception occurred:
>> [snip]
>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
>>    hello
>>
>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>> exceptions.
>>
>> [YOCTO #13267]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cookerdata.py
>> b/bitbake/lib/bb/cookerdata.py
>> index f8ae410..585edc5 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>               if multiconfig:
>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>> ), self.data)
>>   
>> -        except (SyntaxError, bb.BBHandledException):
>> +        except bb.BBHandledException:
>> +            raise
>> +        except SyntaxError:
>>               raise bb.BBHandledException
>>           except bb.data_smart.ExpansionError as e:
>>               logger.error(str(e))
> 
> Hi Robert,
> 
> This doesn't sound right, where is this exception being printed a
> second time? The point of "BBHandledException" is to say "don't print
> any further traces for this exception". If this fixes the bug, it means
> something somewhere is printing a trace for a second time when it
> should pass through BBHandledException?

I think that I've found the root cause, it is because no code handles
BBHandledException during server is starting, so we will *always* get
(double) exceptions when BBHandledException comes during server is starting,
such as call bb.fatal() during parsing configurations files. I will
send a patch to fix it.

// Robert

> 
> Cheers,
> 
> Richard
> 
> 

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 5/14/19 7:02 PM, Robert Yang wrote:
> 
> 
> On 5/12/19 4:28 PM, Richard Purdie wrote:
>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>> The bb.fatal() raises BBHandledException() which causes double exceptions,
>>> e.g.:
>>>
>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>> $ bitbake -p
>>> [snip]
>>> During handling of the above exception, another exception occurred:
>>> [snip]
>>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be 
>>> unavailable in PATH, please install them in order to proceed:
>>>    hello
>>>
>>> Use "raise" rather than "raise bb.BBHandledException" to fix the double
>>> exceptions.
>>>
>>> [YOCTO #13267]
>>>
>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>> ---
>>>   bitbake/lib/bb/cookerdata.py | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>> b/bitbake/lib/bb/cookerdata.py
>>> index f8ae410..585edc5 100644
>>> --- a/bitbake/lib/bb/cookerdata.py
>>> +++ b/bitbake/lib/bb/cookerdata.py
>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>               if multiconfig:
>>>                   bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>> ), self.data)
>>> -        except (SyntaxError, bb.BBHandledException):
>>> +        except bb.BBHandledException:
>>> +            raise
>>> +        except SyntaxError:
>>>               raise bb.BBHandledException
>>>           except bb.data_smart.ExpansionError as e:
>>>               logger.error(str(e))
>>
>> Hi Robert,
>>
>> This doesn't sound right, where is this exception being printed a
>> second time? The point of "BBHandledException" is to say "don't print
>> any further traces for this exception". If this fixes the bug, it means
>> something somewhere is printing a trace for a second time when it
>> should pass through BBHandledException?
> 
> Hi RP,
> 
> I found another serious problem when tried to raise BBHandledException. There
> is a deadlock when a recipe is failed to parse, e.g.:
> 
> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
> $ bitbake -p
> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld'
> [hangs]
> 
> Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes
> are still existed, and we need kill them manually, otherwise we can't start
> bitbake again.

BTW, things becomes much better after the following two patches are merged:
bitbake: knotty: Fix for the Second Keyboard Interrupt
bitbake: cooker: Cleanup the queue before call process.join()

Now we hardly can reproduce the problem:
echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

It's not easy to hang any more, but still hangs sometimes, I tried to debug it,
but didn't find the root cause, the ui/knotty.py can't get event from server,
and goes into a dead loop.

             event = eventHandler.waitEvent(0)
             if event is None:
                 if main.shutdown > 1:
                     break
                 termfilter.updateFooter()
                 event = eventHandler.waitEvent(0.25)
                 if event is None:
                     continue

The main.shutdown is always 0 when it hangs.

// Robert

> 
> I think that there are two problems:
> 1) The cooker.py:parse_generator() raises an exception which makes
>     one of result_queue or parser_quit have a deadlock, then
>     causes the shutdown():process.join() hanged.
> 
> 2) The sub processes, parser_quit and result_queue are running dependently,
>     we can't make sure that cooker.py:Parser():realrun() can get data from
>     parser_quit, so there is a race issue, the Parser's subprocess maybe already
>     exited before parser_quit sends data to it.
> 
> I think that we can move part of parse_next()'s code into parse_generator()
> to fix the deadlock problem, and we may need change the working way of
> parser_quit. But I'm not sure, I need your suggestions before work on it.
> 
> // Robert
> 
>>
>> Cheers,
>>
>> Richard
>>
>>

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[richard.purdie]

On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
> 
> On 5/14/19 7:02 PM, Robert Yang wrote:
> > 
> > On 5/12/19 4:28 PM, Richard Purdie wrote:
> > > On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
> > > > The bb.fatal() raises BBHandledException() which causes double
> > > > exceptions,
> > > > e.g.:
> > > > 
> > > > Add 'HOSTTOOLS += "hello"' to conf/local.conf:
> > > > $ bitbake -p
> > > > [snip]
> > > > During handling of the above exception, another exception
> > > > occurred:
> > > > [snip]
> > > > ERROR: The following required tools (as specified by HOSTTOOLS)
> > > > appear to be 
> > > > unavailable in PATH, please install them in order to proceed:
> > > >    hello
> > > > 
> > > > Use "raise" rather than "raise bb.BBHandledException" to fix
> > > > the double
> > > > exceptions.
> > > > 
> > > > [YOCTO #13267]
> > > > 
> > > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> > > > ---
> > > >   bitbake/lib/bb/cookerdata.py | 4 +++-
> > > >   1 file changed, 3 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/bitbake/lib/bb/cookerdata.py
> > > > b/bitbake/lib/bb/cookerdata.py
> > > > index f8ae410..585edc5 100644
> > > > --- a/bitbake/lib/bb/cookerdata.py
> > > > +++ b/bitbake/lib/bb/cookerdata.py
> > > > @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
> > > >               if multiconfig:
> > > >                  
> > > > bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
> > > > ), self.data)
> > > > -        except (SyntaxError, bb.BBHandledException):
> > > > +        except bb.BBHandledException:
> > > > +            raise
> > > > +        except SyntaxError:
> > > >               raise bb.BBHandledException
> > > >           except bb.data_smart.ExpansionError as e:
> > > >               logger.error(str(e))
> > > 
> > > Hi Robert,
> > > 
> > > This doesn't sound right, where is this exception being printed a
> > > second time? The point of "BBHandledException" is to say "don't
> > > print
> > > any further traces for this exception". If this fixes the bug, it
> > > means
> > > something somewhere is printing a trace for a second time when it
> > > should pass through BBHandledException?
> > 
> > Hi RP,
> > 
> > I found another serious problem when tried to raise
> > BBHandledException. There
> > is a deadlock when a recipe is failed to parse, e.g.:
> > 
> > $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
> > $ bitbake -p
> > meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
> > 'helloworld'
> > [hangs]
> > 
> > Then bitbake hangs, we can use Ctrl-C to break it, but the sub
> > processes
> > are still existed, and we need kill them manually, otherwise we
> > can't start
> > bitbake again.
> 
> BTW, things becomes much better after the following two patches are
> merged:
> bitbake: knotty: Fix for the Second Keyboard Interrupt
> bitbake: cooker: Cleanup the queue before call process.join()
> 
> Now we hardly can reproduce the problem:
> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
> done
> 
> It's not easy to hang any more, but still hangs sometimes, I tried to
> debug it,
> but didn't find the root cause, the ui/knotty.py can't get event from
> server,
> and goes into a dead loop.
> 
>              event = eventHandler.waitEvent(0)
>              if event is None:
>                  if main.shutdown > 1:
>                      break
>                  termfilter.updateFooter()
>                  event = eventHandler.waitEvent(0.25)
>                  if event is None:
>                      continue
> 
> The main.shutdown is always 0 when it hangs.

In theory there are timeouts there so it should never hang waiting for
an event. Is it looping and not getting an event? or is the other end
disconnected?

I guess the question is what we can do to detect a dead connection, or
if the server is still alive, why the server is hanging and not sending
any events?

Thanks for the patches, those were tricky issues to track down and
solve!

Cheers,

Richard

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
>>
>> On 5/14/19 7:02 PM, Robert Yang wrote:
>>>
>>> On 5/12/19 4:28 PM, Richard Purdie wrote:
>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>>>> The bb.fatal() raises BBHandledException() which causes double
>>>>> exceptions,
>>>>> e.g.:
>>>>>
>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>>>> $ bitbake -p
>>>>> [snip]
>>>>> During handling of the above exception, another exception
>>>>> occurred:
>>>>> [snip]
>>>>> ERROR: The following required tools (as specified by HOSTTOOLS)
>>>>> appear to be
>>>>> unavailable in PATH, please install them in order to proceed:
>>>>>     hello
>>>>>
>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix
>>>>> the double
>>>>> exceptions.
>>>>>
>>>>> [YOCTO #13267]
>>>>>
>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>>> ---
>>>>>    bitbake/lib/bb/cookerdata.py | 4 +++-
>>>>>    1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>>>> b/bitbake/lib/bb/cookerdata.py
>>>>> index f8ae410..585edc5 100644
>>>>> --- a/bitbake/lib/bb/cookerdata.py
>>>>> +++ b/bitbake/lib/bb/cookerdata.py
>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>>>                if multiconfig:
>>>>>                   
>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>>>> ), self.data)
>>>>> -        except (SyntaxError, bb.BBHandledException):
>>>>> +        except bb.BBHandledException:
>>>>> +            raise
>>>>> +        except SyntaxError:
>>>>>                raise bb.BBHandledException
>>>>>            except bb.data_smart.ExpansionError as e:
>>>>>                logger.error(str(e))
>>>>
>>>> Hi Robert,
>>>>
>>>> This doesn't sound right, where is this exception being printed a
>>>> second time? The point of "BBHandledException" is to say "don't
>>>> print
>>>> any further traces for this exception". If this fixes the bug, it
>>>> means
>>>> something somewhere is printing a trace for a second time when it
>>>> should pass through BBHandledException?
>>>
>>> Hi RP,
>>>
>>> I found another serious problem when tried to raise
>>> BBHandledException. There
>>> is a deadlock when a recipe is failed to parse, e.g.:
>>>
>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>> $ bitbake -p
>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
>>> 'helloworld'
>>> [hangs]
>>>
>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub
>>> processes
>>> are still existed, and we need kill them manually, otherwise we
>>> can't start
>>> bitbake again.
>>
>> BTW, things becomes much better after the following two patches are
>> merged:
>> bitbake: knotty: Fix for the Second Keyboard Interrupt
>> bitbake: cooker: Cleanup the queue before call process.join()
>>
>> Now we hardly can reproduce the problem:
>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
>> done
>>
>> It's not easy to hang any more, but still hangs sometimes, I tried to
>> debug it,
>> but didn't find the root cause, the ui/knotty.py can't get event from
>> server,
>> and goes into a dead loop.
>>
>>               event = eventHandler.waitEvent(0)
>>               if event is None:
>>                   if main.shutdown > 1:
>>                       break
>>                   termfilter.updateFooter()
>>                   event = eventHandler.waitEvent(0.25)
>>                   if event is None:
>>                       continue
>>
>> The main.shutdown is always 0 when it hangs.
> 
> In theory there are timeouts there so it should never hang waiting for
> an event. Is it looping and not getting an event? or is the other end
> disconnected?
> 
> I guess the question is what we can do to detect a dead connection, or
> if the server is still alive, why the server is hanging and not sending
> any events?

After more investigations, it may hang at two places, they are very rarely to
happen, but does happen, I can use the following command to reproduce it in
10 minutes:

$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

* Hangs #1 in cooker.py:

2065         # Cleanup the queue before call process.join(), otherwise there 
might be
2066         # deadlocks.
2067         while True:
2068             try:
2069                self.result_queue.get(timeout=0.25)
2070             except queue.Empty:
2071                 break

It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
but still have the problem, I think that it's a bug of python3's 
multiprocessing, and we can call self.result_queue.cancel_join_thread()
to fix the problem.


* Hangs #2 in cooker.py:
2073         for process in self.processes:
2074             if force:
2075                 process.join(.1)
2076                 process.terminate()
2077             else:
2078                 process.join()


It hangs at process.join(), I added debug code there, it is because the process
is alive when join() it, I think that we can use a while loop to check whether
the process is alive or not before join(), and force join() after many tries.

I will do more testing before send the patches, make sure it won't hang in hours.

// Robert

> 
> Thanks for the patches, those were tricky issues to track down and
> solve!
> 
> Cheers,
> 
> Richard
> 
> 
> 

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 8/19/19 4:34 PM, Robert Yang wrote:
> 
> On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote:
>> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
>>>
>>> On 5/14/19 7:02 PM, Robert Yang wrote:
>>>>
>>>> On 5/12/19 4:28 PM, Richard Purdie wrote:
>>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>>>>> The bb.fatal() raises BBHandledException() which causes double
>>>>>> exceptions,
>>>>>> e.g.:
>>>>>>
>>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>>>>> $ bitbake -p
>>>>>> [snip]
>>>>>> During handling of the above exception, another exception
>>>>>> occurred:
>>>>>> [snip]
>>>>>> ERROR: The following required tools (as specified by HOSTTOOLS)
>>>>>> appear to be
>>>>>> unavailable in PATH, please install them in order to proceed:
>>>>>>     hello
>>>>>>
>>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix
>>>>>> the double
>>>>>> exceptions.
>>>>>>
>>>>>> [YOCTO #13267]
>>>>>>
>>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>>>> ---
>>>>>>    bitbake/lib/bb/cookerdata.py | 4 +++-
>>>>>>    1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>>>>> b/bitbake/lib/bb/cookerdata.py
>>>>>> index f8ae410..585edc5 100644
>>>>>> --- a/bitbake/lib/bb/cookerdata.py
>>>>>> +++ b/bitbake/lib/bb/cookerdata.py
>>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>>>>                if multiconfig:
>>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>>>>> ), self.data)
>>>>>> -        except (SyntaxError, bb.BBHandledException):
>>>>>> +        except bb.BBHandledException:
>>>>>> +            raise
>>>>>> +        except SyntaxError:
>>>>>>                raise bb.BBHandledException
>>>>>>            except bb.data_smart.ExpansionError as e:
>>>>>>                logger.error(str(e))
>>>>>
>>>>> Hi Robert,
>>>>>
>>>>> This doesn't sound right, where is this exception being printed a
>>>>> second time? The point of "BBHandledException" is to say "don't
>>>>> print
>>>>> any further traces for this exception". If this fixes the bug, it
>>>>> means
>>>>> something somewhere is printing a trace for a second time when it
>>>>> should pass through BBHandledException?
>>>>
>>>> Hi RP,
>>>>
>>>> I found another serious problem when tried to raise
>>>> BBHandledException. There
>>>> is a deadlock when a recipe is failed to parse, e.g.:
>>>>
>>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>>> $ bitbake -p
>>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
>>>> 'helloworld'
>>>> [hangs]
>>>>
>>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub
>>>> processes
>>>> are still existed, and we need kill them manually, otherwise we
>>>> can't start
>>>> bitbake again.
>>>
>>> BTW, things becomes much better after the following two patches are
>>> merged:
>>> bitbake: knotty: Fix for the Second Keyboard Interrupt
>>> bitbake: cooker: Cleanup the queue before call process.join()
>>>
>>> Now we hardly can reproduce the problem:
>>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
>>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
>>> done

Sorry, the full commands should be:


$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done

// Rboert

>>>
>>> It's not easy to hang any more, but still hangs sometimes, I tried to
>>> debug it,
>>> but didn't find the root cause, the ui/knotty.py can't get event from
>>> server,
>>> and goes into a dead loop.
>>>
>>>               event = eventHandler.waitEvent(0)
>>>               if event is None:
>>>                   if main.shutdown > 1:
>>>                       break
>>>                   termfilter.updateFooter()
>>>                   event = eventHandler.waitEvent(0.25)
>>>                   if event is None:
>>>                       continue
>>>
>>> The main.shutdown is always 0 when it hangs.
>>
>> In theory there are timeouts there so it should never hang waiting for
>> an event. Is it looping and not getting an event? or is the other end
>> disconnected?
>>
>> I guess the question is what we can do to detect a dead connection, or
>> if the server is still alive, why the server is hanging and not sending
>> any events?
> 
> After more investigations, it may hang at two places, they are very rarely to
> happen, but does happen, I can use the following command to reproduce it in
> 10 minutes:
> 
> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
> 
> * Hangs #1 in cooker.py:
> 
> 2065         # Cleanup the queue before call process.join(), otherwise there 
> might be
> 2066         # deadlocks.
> 2067         while True:
> 2068             try:
> 2069                self.result_queue.get(timeout=0.25)
> 2070             except queue.Empty:
> 2071                 break
> 
> It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
> here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
> but still have the problem, I think that it's a bug of python3's 
> multiprocessing, and we can call self.result_queue.cancel_join_thread()
> to fix the problem.
> 
> 
> * Hangs #2 in cooker.py:
> 2073         for process in self.processes:
> 2074             if force:
> 2075                 process.join(.1)
> 2076                 process.terminate()
> 2077             else:
> 2078                 process.join()
> 
> 
> It hangs at process.join(), I added debug code there, it is because the process
> is alive when join() it, I think that we can use a while loop to check whether
> the process is alive or not before join(), and force join() after many tries.
> 
> I will do more testing before send the patches, make sure it won't hang in hours.
> 
> // Robert
> 
>>
>> Thanks for the patches, those were tricky issues to track down and
>> solve!
>>
>> Cheers,
>>
>> Richard
>>
>>
>>

Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()

[Robert Yang]

On 8/19/19 4:34 PM, Robert Yang wrote:
> 
> On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote:
>> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
>>>
>>> On 5/14/19 7:02 PM, Robert Yang wrote:
>>>>
>>>> On 5/12/19 4:28 PM, Richard Purdie wrote:
>>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>>>>> The bb.fatal() raises BBHandledException() which causes double
>>>>>> exceptions,
>>>>>> e.g.:
>>>>>>
>>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>>>>> $ bitbake -p
>>>>>> [snip]
>>>>>> During handling of the above exception, another exception
>>>>>> occurred:
>>>>>> [snip]
>>>>>> ERROR: The following required tools (as specified by HOSTTOOLS)
>>>>>> appear to be
>>>>>> unavailable in PATH, please install them in order to proceed:
>>>>>>     hello
>>>>>>
>>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix
>>>>>> the double
>>>>>> exceptions.
>>>>>>
>>>>>> [YOCTO #13267]
>>>>>>
>>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>>>> ---
>>>>>>    bitbake/lib/bb/cookerdata.py | 4 +++-
>>>>>>    1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>>>>> b/bitbake/lib/bb/cookerdata.py
>>>>>> index f8ae410..585edc5 100644
>>>>>> --- a/bitbake/lib/bb/cookerdata.py
>>>>>> +++ b/bitbake/lib/bb/cookerdata.py
>>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>>>>                if multiconfig:
>>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>>>>> ), self.data)
>>>>>> -        except (SyntaxError, bb.BBHandledException):
>>>>>> +        except bb.BBHandledException:
>>>>>> +            raise
>>>>>> +        except SyntaxError:
>>>>>>                raise bb.BBHandledException
>>>>>>            except bb.data_smart.ExpansionError as e:
>>>>>>                logger.error(str(e))
>>>>>
>>>>> Hi Robert,
>>>>>
>>>>> This doesn't sound right, where is this exception being printed a
>>>>> second time? The point of "BBHandledException" is to say "don't
>>>>> print
>>>>> any further traces for this exception". If this fixes the bug, it
>>>>> means
>>>>> something somewhere is printing a trace for a second time when it
>>>>> should pass through BBHandledException?
>>>>
>>>> Hi RP,
>>>>
>>>> I found another serious problem when tried to raise
>>>> BBHandledException. There
>>>> is a deadlock when a recipe is failed to parse, e.g.:
>>>>
>>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>>> $ bitbake -p
>>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
>>>> 'helloworld'
>>>> [hangs]
>>>>
>>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub
>>>> processes
>>>> are still existed, and we need kill them manually, otherwise we
>>>> can't start
>>>> bitbake again.
>>>
>>> BTW, things becomes much better after the following two patches are
>>> merged:
>>> bitbake: knotty: Fix for the Second Keyboard Interrupt
>>> bitbake: cooker: Cleanup the queue before call process.join()
>>>
>>> Now we hardly can reproduce the problem:
>>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
>>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
>>> done
>>>
>>> It's not easy to hang any more, but still hangs sometimes, I tried to
>>> debug it,
>>> but didn't find the root cause, the ui/knotty.py can't get event from
>>> server,
>>> and goes into a dead loop.
>>>
>>>               event = eventHandler.waitEvent(0)
>>>               if event is None:
>>>                   if main.shutdown > 1:
>>>                       break
>>>                   termfilter.updateFooter()
>>>                   event = eventHandler.waitEvent(0.25)
>>>                   if event is None:
>>>                       continue
>>>
>>> The main.shutdown is always 0 when it hangs.
>>
>> In theory there are timeouts there so it should never hang waiting for
>> an event. Is it looping and not getting an event? or is the other end
>> disconnected?
>>
>> I guess the question is what we can do to detect a dead connection, or
>> if the server is still alive, why the server is hanging and not sending
>> any events?
> 
> After more investigations, it may hang at two places, they are very rarely to
> happen, but does happen, I can use the following command to reproduce it in
> 10 minutes:
> 
> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log 
> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
> 
> * Hangs #1 in cooker.py:
> 
> 2065         # Cleanup the queue before call process.join(), otherwise there 
> might be
> 2066         # deadlocks.
> 2067         while True:
> 2068             try:
> 2069                self.result_queue.get(timeout=0.25)
> 2070             except queue.Empty:
> 2071                 break
> 
> It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
> here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
> but still have the problem, I think that it's a bug of python3's 
> multiprocessing, and we can call self.result_queue.cancel_join_thread()
> to fix the problem.
> 
> 
> * Hangs #2 in cooker.py:
> 2073         for process in self.processes:
> 2074             if force:
> 2075                 process.join(.1)
> 2076                 process.terminate()
> 2077             else:
> 2078                 process.join()
> 
> 
> It hangs at process.join(), I added debug code there, it is because the process
> is alive when join() it, I think that we can use a while loop to check whether
> the process is alive or not before join(), and force join() after many tries.
> 
> I will do more testing before send the patches, make sure it won't hang in hours.


After a lot tries, the only reliable way to avoid hanging is os.kill(), I've
sent the patch for it, and added reason in the commit message.

// Robert

> 
> // Robert
> 
>>
>> Thanks for the patches, those were tricky issues to track down and
>> solve!
>>
>> Cheers,
>>
>> Richard
>>
>>
>>