From: Robert Yang <liezhi.yang@windriver.com>
To: <richard.purdie@linuxfoundation.org>,
<bitbake-devel@lists.openembedded.org>
Subject: Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal()
Date: Mon, 19 Aug 2019 17:04:25 +0800 [thread overview]
Message-ID: <57480b17-ce13-1781-9f0a-af9745c6777f@windriver.com> (raw)
In-Reply-To: <e4e11458-d818-db1c-7001-112d303a9edc@windriver.com>
On 8/19/19 4:34 PM, Robert Yang wrote:
>
> On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote:
>> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote:
>>>
>>> On 5/14/19 7:02 PM, Robert Yang wrote:
>>>>
>>>> On 5/12/19 4:28 PM, Richard Purdie wrote:
>>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote:
>>>>>> The bb.fatal() raises BBHandledException() which causes double
>>>>>> exceptions,
>>>>>> e.g.:
>>>>>>
>>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf:
>>>>>> $ bitbake -p
>>>>>> [snip]
>>>>>> During handling of the above exception, another exception
>>>>>> occurred:
>>>>>> [snip]
>>>>>> ERROR: The following required tools (as specified by HOSTTOOLS)
>>>>>> appear to be
>>>>>> unavailable in PATH, please install them in order to proceed:
>>>>>> hello
>>>>>>
>>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix
>>>>>> the double
>>>>>> exceptions.
>>>>>>
>>>>>> [YOCTO #13267]
>>>>>>
>>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>>>> ---
>>>>>> bitbake/lib/bb/cookerdata.py | 4 +++-
>>>>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/bitbake/lib/bb/cookerdata.py
>>>>>> b/bitbake/lib/bb/cookerdata.py
>>>>>> index f8ae410..585edc5 100644
>>>>>> --- a/bitbake/lib/bb/cookerdata.py
>>>>>> +++ b/bitbake/lib/bb/cookerdata.py
>>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object):
>>>>>> if multiconfig:
>>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata
>>>>>> ), self.data)
>>>>>> - except (SyntaxError, bb.BBHandledException):
>>>>>> + except bb.BBHandledException:
>>>>>> + raise
>>>>>> + except SyntaxError:
>>>>>> raise bb.BBHandledException
>>>>>> except bb.data_smart.ExpansionError as e:
>>>>>> logger.error(str(e))
>>>>>
>>>>> Hi Robert,
>>>>>
>>>>> This doesn't sound right, where is this exception being printed a
>>>>> second time? The point of "BBHandledException" is to say "don't
>>>>> print
>>>>> any further traces for this exception". If this fixes the bug, it
>>>>> means
>>>>> something somewhere is printing a trace for a second time when it
>>>>> should pass through BBHandledException?
>>>>
>>>> Hi RP,
>>>>
>>>> I found another serious problem when tried to raise
>>>> BBHandledException. There
>>>> is a deadlock when a recipe is failed to parse, e.g.:
>>>>
>>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>>> $ bitbake -p
>>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line:
>>>> 'helloworld'
>>>> [hangs]
>>>>
>>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub
>>>> processes
>>>> are still existed, and we need kill them manually, otherwise we
>>>> can't start
>>>> bitbake again.
>>>
>>> BTW, things becomes much better after the following two patches are
>>> merged:
>>> bitbake: knotty: Fix for the Second Keyboard Interrupt
>>> bitbake: cooker: Cleanup the queue before call process.join()
>>>
>>> Now we hardly can reproduce the problem:
>>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
>>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
>>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p;
>>> done
Sorry, the full commands should be:
$ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb
$ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
// Rboert
>>>
>>> It's not easy to hang any more, but still hangs sometimes, I tried to
>>> debug it,
>>> but didn't find the root cause, the ui/knotty.py can't get event from
>>> server,
>>> and goes into a dead loop.
>>>
>>> event = eventHandler.waitEvent(0)
>>> if event is None:
>>> if main.shutdown > 1:
>>> break
>>> termfilter.updateFooter()
>>> event = eventHandler.waitEvent(0.25)
>>> if event is None:
>>> continue
>>>
>>> The main.shutdown is always 0 when it hangs.
>>
>> In theory there are timeouts there so it should never hang waiting for
>> an event. Is it looping and not getting an event? or is the other end
>> disconnected?
>>
>> I guess the question is what we can do to detect a dead connection, or
>> if the server is still alive, why the server is hanging and not sending
>> any events?
>
> After more investigations, it may hang at two places, they are very rarely to
> happen, but does happen, I can use the following command to reproduce it in
> 10 minutes:
>
> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log
> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done
>
> * Hangs #1 in cooker.py:
>
> 2065 # Cleanup the queue before call process.join(), otherwise there
> might be
> 2066 # deadlocks.
> 2067 while True:
> 2068 try:
> 2069 self.result_queue.get(timeout=0.25)
> 2070 except queue.Empty:
> 2071 break
>
> It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work
> here, I tried python 3.5.2 and 3.6.7, the later one is a little better,
> but still have the problem, I think that it's a bug of python3's
> multiprocessing, and we can call self.result_queue.cancel_join_thread()
> to fix the problem.
>
>
> * Hangs #2 in cooker.py:
> 2073 for process in self.processes:
> 2074 if force:
> 2075 process.join(.1)
> 2076 process.terminate()
> 2077 else:
> 2078 process.join()
>
>
> It hangs at process.join(), I added debug code there, it is because the process
> is alive when join() it, I think that we can use a while loop to check whether
> the process is alive or not before join(), and force join() after many tries.
>
> I will do more testing before send the patches, make sure it won't hang in hours.
>
> // Robert
>
>>
>> Thanks for the patches, those were tricky issues to track down and
>> solve!
>>
>> Cheers,
>>
>> Richard
>>
>>
>>
next prev parent reply other threads:[~2019-08-19 9:03 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-09 8:03 [PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() Robert Yang
2019-05-09 8:03 ` [PATCH 1/1] " Robert Yang
2019-05-12 8:28 ` Richard Purdie
2019-05-14 11:02 ` Robert Yang
2019-05-20 8:55 ` Robert Yang
2019-08-15 11:29 ` Robert Yang
2019-08-15 23:03 ` richard.purdie
2019-08-19 8:34 ` Robert Yang
2019-08-19 9:04 ` Robert Yang [this message]
2019-08-22 9:06 ` Robert Yang
2019-08-15 10:55 ` Robert Yang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=57480b17-ce13-1781-9f0a-af9745c6777f@windriver.com \
--to=liezhi.yang@windriver.com \
--cc=bitbake-devel@lists.openembedded.org \
--cc=richard.purdie@linuxfoundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.