* [PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() @ 2019-05-09 8:03 Robert Yang 2019-05-09 8:03 ` [PATCH 1/1] " Robert Yang 0 siblings, 1 reply; 11+ messages in thread From: Robert Yang @ 2019-05-09 8:03 UTC (permalink / raw) To: bitbake-devel The following changes since commit c0dc72bad92e2e4dc0aedb48b969709f821baf73: coreutils: Fix patch upstream status field (2019-05-08 23:36:21 +0100) are available in the git repository at: git://git.pokylinux.org/poky-contrib rbt/python http://git.pokylinux.org/cgit.cgi//log/?h=rbt/python Robert Yang (1): bitbake: cookerdata: Avoid double exceptions for bb.fatal() bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) -- 2.7.4 ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-09 8:03 [PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() Robert Yang @ 2019-05-09 8:03 ` Robert Yang 2019-05-12 8:28 ` Richard Purdie 0 siblings, 1 reply; 11+ messages in thread From: Robert Yang @ 2019-05-09 8:03 UTC (permalink / raw) To: bitbake-devel The bb.fatal() raises BBHandledException() which causes double exceptions, e.g.: Add 'HOSTTOOLS += "hello"' to conf/local.conf: $ bitbake -p [snip] During handling of the above exception, another exception occurred: [snip] ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: hello Use "raise" rather than "raise bb.BBHandledException" to fix the double exceptions. [YOCTO #13267] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- bitbake/lib/bb/cookerdata.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py index f8ae410..585edc5 100644 --- a/bitbake/lib/bb/cookerdata.py +++ b/bitbake/lib/bb/cookerdata.py @@ -301,7 +301,9 @@ class CookerDataBuilder(object): if multiconfig: bb.event.fire(bb.event.MultiConfigParsed(self.mcdata), self.data) - except (SyntaxError, bb.BBHandledException): + except bb.BBHandledException: + raise + except SyntaxError: raise bb.BBHandledException except bb.data_smart.ExpansionError as e: logger.error(str(e)) -- 2.7.4 ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-09 8:03 ` [PATCH 1/1] " Robert Yang @ 2019-05-12 8:28 ` Richard Purdie 2019-05-14 11:02 ` Robert Yang 2019-08-15 10:55 ` Robert Yang 0 siblings, 2 replies; 11+ messages in thread From: Richard Purdie @ 2019-05-12 8:28 UTC (permalink / raw) To: Robert Yang, bitbake-devel On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: > The bb.fatal() raises BBHandledException() which causes double exceptions, > e.g.: > > Add 'HOSTTOOLS += "hello"' to conf/local.conf: > $ bitbake -p > [snip] > During handling of the above exception, another exception occurred: > [snip] > ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: > hello > > Use "raise" rather than "raise bb.BBHandledException" to fix the double > exceptions. > > [YOCTO #13267] > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > bitbake/lib/bb/cookerdata.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/bitbake/lib/bb/cookerdata.py > b/bitbake/lib/bb/cookerdata.py > index f8ae410..585edc5 100644 > --- a/bitbake/lib/bb/cookerdata.py > +++ b/bitbake/lib/bb/cookerdata.py > @@ -301,7 +301,9 @@ class CookerDataBuilder(object): > if multiconfig: > bb.event.fire(bb.event.MultiConfigParsed(self.mcdata > ), self.data) > > - except (SyntaxError, bb.BBHandledException): > + except bb.BBHandledException: > + raise > + except SyntaxError: > raise bb.BBHandledException > except bb.data_smart.ExpansionError as e: > logger.error(str(e)) Hi Robert, This doesn't sound right, where is this exception being printed a second time? The point of "BBHandledException" is to say "don't print any further traces for this exception". If this fixes the bug, it means something somewhere is printing a trace for a second time when it should pass through BBHandledException? Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-12 8:28 ` Richard Purdie @ 2019-05-14 11:02 ` Robert Yang 2019-05-20 8:55 ` Robert Yang 2019-08-15 11:29 ` Robert Yang 2019-08-15 10:55 ` Robert Yang 1 sibling, 2 replies; 11+ messages in thread From: Robert Yang @ 2019-05-14 11:02 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 5/12/19 4:28 PM, Richard Purdie wrote: > On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >> The bb.fatal() raises BBHandledException() which causes double exceptions, >> e.g.: >> >> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >> $ bitbake -p >> [snip] >> During handling of the above exception, another exception occurred: >> [snip] >> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: >> hello >> >> Use "raise" rather than "raise bb.BBHandledException" to fix the double >> exceptions. >> >> [YOCTO #13267] >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> bitbake/lib/bb/cookerdata.py | 4 +++- >> 1 file changed, 3 insertions(+), 1 deletion(-) >> >> diff --git a/bitbake/lib/bb/cookerdata.py >> b/bitbake/lib/bb/cookerdata.py >> index f8ae410..585edc5 100644 >> --- a/bitbake/lib/bb/cookerdata.py >> +++ b/bitbake/lib/bb/cookerdata.py >> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >> if multiconfig: >> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >> ), self.data) >> >> - except (SyntaxError, bb.BBHandledException): >> + except bb.BBHandledException: >> + raise >> + except SyntaxError: >> raise bb.BBHandledException >> except bb.data_smart.ExpansionError as e: >> logger.error(str(e)) > > Hi Robert, > > This doesn't sound right, where is this exception being printed a > second time? The point of "BBHandledException" is to say "don't print > any further traces for this exception". If this fixes the bug, it means > something somewhere is printing a trace for a second time when it > should pass through BBHandledException? Hi RP, I found another serious problem when tried to raise BBHandledException. There is a deadlock when a recipe is failed to parse, e.g.: $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ bitbake -p meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' [hangs] Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes are still existed, and we need kill them manually, otherwise we can't start bitbake again. I think that there are two problems: 1) The cooker.py:parse_generator() raises an exception which makes one of result_queue or parser_quit have a deadlock, then causes the shutdown():process.join() hanged. 2) The sub processes, parser_quit and result_queue are running dependently, we can't make sure that cooker.py:Parser():realrun() can get data from parser_quit, so there is a race issue, the Parser's subprocess maybe already exited before parser_quit sends data to it. I think that we can move part of parse_next()'s code into parse_generator() to fix the deadlock problem, and we may need change the working way of parser_quit. But I'm not sure, I need your suggestions before work on it. // Robert > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-14 11:02 ` Robert Yang @ 2019-05-20 8:55 ` Robert Yang 2019-08-15 11:29 ` Robert Yang 1 sibling, 0 replies; 11+ messages in thread From: Robert Yang @ 2019-05-20 8:55 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 5/14/19 7:02 PM, Robert Yang wrote: > > > On 5/12/19 4:28 PM, Richard Purdie wrote: >> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >>> The bb.fatal() raises BBHandledException() which causes double exceptions, >>> e.g.: >>> >>> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >>> $ bitbake -p >>> [snip] >>> During handling of the above exception, another exception occurred: >>> [snip] >>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be >>> unavailable in PATH, please install them in order to proceed: >>> hello >>> >>> Use "raise" rather than "raise bb.BBHandledException" to fix the double >>> exceptions. >>> >>> [YOCTO #13267] >>> >>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>> --- >>> bitbake/lib/bb/cookerdata.py | 4 +++- >>> 1 file changed, 3 insertions(+), 1 deletion(-) >>> >>> diff --git a/bitbake/lib/bb/cookerdata.py >>> b/bitbake/lib/bb/cookerdata.py >>> index f8ae410..585edc5 100644 >>> --- a/bitbake/lib/bb/cookerdata.py >>> +++ b/bitbake/lib/bb/cookerdata.py >>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >>> if multiconfig: >>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >>> ), self.data) >>> - except (SyntaxError, bb.BBHandledException): >>> + except bb.BBHandledException: >>> + raise >>> + except SyntaxError: >>> raise bb.BBHandledException >>> except bb.data_smart.ExpansionError as e: >>> logger.error(str(e)) >> >> Hi Robert, >> >> This doesn't sound right, where is this exception being printed a >> second time? The point of "BBHandledException" is to say "don't print >> any further traces for this exception". If this fixes the bug, it means >> something somewhere is printing a trace for a second time when it >> should pass through BBHandledException? > > Hi RP, > > I found another serious problem when tried to raise BBHandledException. There > is a deadlock when a recipe is failed to parse, e.g.: > > $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb > $ bitbake -p > meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' > [hangs] > > Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes > are still existed, and we need kill them manually, otherwise we can't start > bitbake again. > > I think that there are two problems: > 1) The cooker.py:parse_generator() raises an exception which makes > one of result_queue or parser_quit have a deadlock, then > causes the shutdown():process.join() hanged. > > 2) The sub processes, parser_quit and result_queue are running dependently, > we can't make sure that cooker.py:Parser():realrun() can get data from > parser_quit, so there is a race issue, the Parser's subprocess maybe already > exited before parser_quit sends data to it. > > I think that we can move part of parse_next()'s code into parse_generator() > to fix the deadlock problem, and we may need change the working way of > parser_quit. But I'm not sure, I need your suggestions before work on it. Hi RP, Do you have any suggestions, please? // Robert > > // Robert > >> >> Cheers, >> >> Richard >> >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-14 11:02 ` Robert Yang 2019-05-20 8:55 ` Robert Yang @ 2019-08-15 11:29 ` Robert Yang 2019-08-15 23:03 ` richard.purdie 1 sibling, 1 reply; 11+ messages in thread From: Robert Yang @ 2019-08-15 11:29 UTC (permalink / raw) To: Richard Purdie, bitbake-devel On 5/14/19 7:02 PM, Robert Yang wrote: > > > On 5/12/19 4:28 PM, Richard Purdie wrote: >> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >>> The bb.fatal() raises BBHandledException() which causes double exceptions, >>> e.g.: >>> >>> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >>> $ bitbake -p >>> [snip] >>> During handling of the above exception, another exception occurred: >>> [snip] >>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be >>> unavailable in PATH, please install them in order to proceed: >>> hello >>> >>> Use "raise" rather than "raise bb.BBHandledException" to fix the double >>> exceptions. >>> >>> [YOCTO #13267] >>> >>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>> --- >>> bitbake/lib/bb/cookerdata.py | 4 +++- >>> 1 file changed, 3 insertions(+), 1 deletion(-) >>> >>> diff --git a/bitbake/lib/bb/cookerdata.py >>> b/bitbake/lib/bb/cookerdata.py >>> index f8ae410..585edc5 100644 >>> --- a/bitbake/lib/bb/cookerdata.py >>> +++ b/bitbake/lib/bb/cookerdata.py >>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >>> if multiconfig: >>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >>> ), self.data) >>> - except (SyntaxError, bb.BBHandledException): >>> + except bb.BBHandledException: >>> + raise >>> + except SyntaxError: >>> raise bb.BBHandledException >>> except bb.data_smart.ExpansionError as e: >>> logger.error(str(e)) >> >> Hi Robert, >> >> This doesn't sound right, where is this exception being printed a >> second time? The point of "BBHandledException" is to say "don't print >> any further traces for this exception". If this fixes the bug, it means >> something somewhere is printing a trace for a second time when it >> should pass through BBHandledException? > > Hi RP, > > I found another serious problem when tried to raise BBHandledException. There > is a deadlock when a recipe is failed to parse, e.g.: > > $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb > $ bitbake -p > meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: 'helloworld' > [hangs] > > Then bitbake hangs, we can use Ctrl-C to break it, but the sub processes > are still existed, and we need kill them manually, otherwise we can't start > bitbake again. BTW, things becomes much better after the following two patches are merged: bitbake: knotty: Fix for the Second Keyboard Interrupt bitbake: cooker: Cleanup the queue before call process.join() Now we hardly can reproduce the problem: echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done It's not easy to hang any more, but still hangs sometimes, I tried to debug it, but didn't find the root cause, the ui/knotty.py can't get event from server, and goes into a dead loop. event = eventHandler.waitEvent(0) if event is None: if main.shutdown > 1: break termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None: continue The main.shutdown is always 0 when it hangs. // Robert > > I think that there are two problems: > 1) The cooker.py:parse_generator() raises an exception which makes > one of result_queue or parser_quit have a deadlock, then > causes the shutdown():process.join() hanged. > > 2) The sub processes, parser_quit and result_queue are running dependently, > we can't make sure that cooker.py:Parser():realrun() can get data from > parser_quit, so there is a race issue, the Parser's subprocess maybe already > exited before parser_quit sends data to it. > > I think that we can move part of parse_next()'s code into parse_generator() > to fix the deadlock problem, and we may need change the working way of > parser_quit. But I'm not sure, I need your suggestions before work on it. > > // Robert > >> >> Cheers, >> >> Richard >> >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-08-15 11:29 ` Robert Yang @ 2019-08-15 23:03 ` richard.purdie 2019-08-19 8:34 ` Robert Yang 0 siblings, 1 reply; 11+ messages in thread From: richard.purdie @ 2019-08-15 23:03 UTC (permalink / raw) To: Robert Yang, bitbake-devel On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote: > > On 5/14/19 7:02 PM, Robert Yang wrote: > > > > On 5/12/19 4:28 PM, Richard Purdie wrote: > > > On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: > > > > The bb.fatal() raises BBHandledException() which causes double > > > > exceptions, > > > > e.g.: > > > > > > > > Add 'HOSTTOOLS += "hello"' to conf/local.conf: > > > > $ bitbake -p > > > > [snip] > > > > During handling of the above exception, another exception > > > > occurred: > > > > [snip] > > > > ERROR: The following required tools (as specified by HOSTTOOLS) > > > > appear to be > > > > unavailable in PATH, please install them in order to proceed: > > > > hello > > > > > > > > Use "raise" rather than "raise bb.BBHandledException" to fix > > > > the double > > > > exceptions. > > > > > > > > [YOCTO #13267] > > > > > > > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > > > > --- > > > > bitbake/lib/bb/cookerdata.py | 4 +++- > > > > 1 file changed, 3 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/bitbake/lib/bb/cookerdata.py > > > > b/bitbake/lib/bb/cookerdata.py > > > > index f8ae410..585edc5 100644 > > > > --- a/bitbake/lib/bb/cookerdata.py > > > > +++ b/bitbake/lib/bb/cookerdata.py > > > > @@ -301,7 +301,9 @@ class CookerDataBuilder(object): > > > > if multiconfig: > > > > > > > > bb.event.fire(bb.event.MultiConfigParsed(self.mcdata > > > > ), self.data) > > > > - except (SyntaxError, bb.BBHandledException): > > > > + except bb.BBHandledException: > > > > + raise > > > > + except SyntaxError: > > > > raise bb.BBHandledException > > > > except bb.data_smart.ExpansionError as e: > > > > logger.error(str(e)) > > > > > > Hi Robert, > > > > > > This doesn't sound right, where is this exception being printed a > > > second time? The point of "BBHandledException" is to say "don't > > > print > > > any further traces for this exception". If this fixes the bug, it > > > means > > > something somewhere is printing a trace for a second time when it > > > should pass through BBHandledException? > > > > Hi RP, > > > > I found another serious problem when tried to raise > > BBHandledException. There > > is a deadlock when a recipe is failed to parse, e.g.: > > > > $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb > > $ bitbake -p > > meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: > > 'helloworld' > > [hangs] > > > > Then bitbake hangs, we can use Ctrl-C to break it, but the sub > > processes > > are still existed, and we need kill them manually, otherwise we > > can't start > > bitbake again. > > BTW, things becomes much better after the following two patches are > merged: > bitbake: knotty: Fix for the Second Keyboard Interrupt > bitbake: cooker: Cleanup the queue before call process.join() > > Now we hardly can reproduce the problem: > echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb > $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log > tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; > done > > It's not easy to hang any more, but still hangs sometimes, I tried to > debug it, > but didn't find the root cause, the ui/knotty.py can't get event from > server, > and goes into a dead loop. > > event = eventHandler.waitEvent(0) > if event is None: > if main.shutdown > 1: > break > termfilter.updateFooter() > event = eventHandler.waitEvent(0.25) > if event is None: > continue > > The main.shutdown is always 0 when it hangs. In theory there are timeouts there so it should never hang waiting for an event. Is it looping and not getting an event? or is the other end disconnected? I guess the question is what we can do to detect a dead connection, or if the server is still alive, why the server is hanging and not sending any events? Thanks for the patches, those were tricky issues to track down and solve! Cheers, Richard ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-08-15 23:03 ` richard.purdie @ 2019-08-19 8:34 ` Robert Yang 2019-08-19 9:04 ` Robert Yang 2019-08-22 9:06 ` Robert Yang 0 siblings, 2 replies; 11+ messages in thread From: Robert Yang @ 2019-08-19 8:34 UTC (permalink / raw) To: richard.purdie, bitbake-devel On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote: > On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote: >> >> On 5/14/19 7:02 PM, Robert Yang wrote: >>> >>> On 5/12/19 4:28 PM, Richard Purdie wrote: >>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >>>>> The bb.fatal() raises BBHandledException() which causes double >>>>> exceptions, >>>>> e.g.: >>>>> >>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >>>>> $ bitbake -p >>>>> [snip] >>>>> During handling of the above exception, another exception >>>>> occurred: >>>>> [snip] >>>>> ERROR: The following required tools (as specified by HOSTTOOLS) >>>>> appear to be >>>>> unavailable in PATH, please install them in order to proceed: >>>>> hello >>>>> >>>>> Use "raise" rather than "raise bb.BBHandledException" to fix >>>>> the double >>>>> exceptions. >>>>> >>>>> [YOCTO #13267] >>>>> >>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>>>> --- >>>>> bitbake/lib/bb/cookerdata.py | 4 +++- >>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>> >>>>> diff --git a/bitbake/lib/bb/cookerdata.py >>>>> b/bitbake/lib/bb/cookerdata.py >>>>> index f8ae410..585edc5 100644 >>>>> --- a/bitbake/lib/bb/cookerdata.py >>>>> +++ b/bitbake/lib/bb/cookerdata.py >>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >>>>> if multiconfig: >>>>> >>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >>>>> ), self.data) >>>>> - except (SyntaxError, bb.BBHandledException): >>>>> + except bb.BBHandledException: >>>>> + raise >>>>> + except SyntaxError: >>>>> raise bb.BBHandledException >>>>> except bb.data_smart.ExpansionError as e: >>>>> logger.error(str(e)) >>>> >>>> Hi Robert, >>>> >>>> This doesn't sound right, where is this exception being printed a >>>> second time? The point of "BBHandledException" is to say "don't >>>> print >>>> any further traces for this exception". If this fixes the bug, it >>>> means >>>> something somewhere is printing a trace for a second time when it >>>> should pass through BBHandledException? >>> >>> Hi RP, >>> >>> I found another serious problem when tried to raise >>> BBHandledException. There >>> is a deadlock when a recipe is failed to parse, e.g.: >>> >>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >>> $ bitbake -p >>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: >>> 'helloworld' >>> [hangs] >>> >>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub >>> processes >>> are still existed, and we need kill them manually, otherwise we >>> can't start >>> bitbake again. >> >> BTW, things becomes much better after the following two patches are >> merged: >> bitbake: knotty: Fix for the Second Keyboard Interrupt >> bitbake: cooker: Cleanup the queue before call process.join() >> >> Now we hardly can reproduce the problem: >> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log >> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; >> done >> >> It's not easy to hang any more, but still hangs sometimes, I tried to >> debug it, >> but didn't find the root cause, the ui/knotty.py can't get event from >> server, >> and goes into a dead loop. >> >> event = eventHandler.waitEvent(0) >> if event is None: >> if main.shutdown > 1: >> break >> termfilter.updateFooter() >> event = eventHandler.waitEvent(0.25) >> if event is None: >> continue >> >> The main.shutdown is always 0 when it hangs. > > In theory there are timeouts there so it should never hang waiting for > an event. Is it looping and not getting an event? or is the other end > disconnected? > > I guess the question is what we can do to detect a dead connection, or > if the server is still alive, why the server is hanging and not sending > any events? After more investigations, it may hang at two places, they are very rarely to happen, but does happen, I can use the following command to reproduce it in 10 minutes: $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done * Hangs #1 in cooker.py: 2065 # Cleanup the queue before call process.join(), otherwise there might be 2066 # deadlocks. 2067 while True: 2068 try: 2069 self.result_queue.get(timeout=0.25) 2070 except queue.Empty: 2071 break It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work here, I tried python 3.5.2 and 3.6.7, the later one is a little better, but still have the problem, I think that it's a bug of python3's multiprocessing, and we can call self.result_queue.cancel_join_thread() to fix the problem. * Hangs #2 in cooker.py: 2073 for process in self.processes: 2074 if force: 2075 process.join(.1) 2076 process.terminate() 2077 else: 2078 process.join() It hangs at process.join(), I added debug code there, it is because the process is alive when join() it, I think that we can use a while loop to check whether the process is alive or not before join(), and force join() after many tries. I will do more testing before send the patches, make sure it won't hang in hours. // Robert > > Thanks for the patches, those were tricky issues to track down and > solve! > > Cheers, > > Richard > > > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-08-19 8:34 ` Robert Yang @ 2019-08-19 9:04 ` Robert Yang 2019-08-22 9:06 ` Robert Yang 1 sibling, 0 replies; 11+ messages in thread From: Robert Yang @ 2019-08-19 9:04 UTC (permalink / raw) To: richard.purdie, bitbake-devel On 8/19/19 4:34 PM, Robert Yang wrote: > > On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote: >> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote: >>> >>> On 5/14/19 7:02 PM, Robert Yang wrote: >>>> >>>> On 5/12/19 4:28 PM, Richard Purdie wrote: >>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >>>>>> The bb.fatal() raises BBHandledException() which causes double >>>>>> exceptions, >>>>>> e.g.: >>>>>> >>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >>>>>> $ bitbake -p >>>>>> [snip] >>>>>> During handling of the above exception, another exception >>>>>> occurred: >>>>>> [snip] >>>>>> ERROR: The following required tools (as specified by HOSTTOOLS) >>>>>> appear to be >>>>>> unavailable in PATH, please install them in order to proceed: >>>>>> hello >>>>>> >>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix >>>>>> the double >>>>>> exceptions. >>>>>> >>>>>> [YOCTO #13267] >>>>>> >>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>>>>> --- >>>>>> bitbake/lib/bb/cookerdata.py | 4 +++- >>>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>>> >>>>>> diff --git a/bitbake/lib/bb/cookerdata.py >>>>>> b/bitbake/lib/bb/cookerdata.py >>>>>> index f8ae410..585edc5 100644 >>>>>> --- a/bitbake/lib/bb/cookerdata.py >>>>>> +++ b/bitbake/lib/bb/cookerdata.py >>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >>>>>> if multiconfig: >>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >>>>>> ), self.data) >>>>>> - except (SyntaxError, bb.BBHandledException): >>>>>> + except bb.BBHandledException: >>>>>> + raise >>>>>> + except SyntaxError: >>>>>> raise bb.BBHandledException >>>>>> except bb.data_smart.ExpansionError as e: >>>>>> logger.error(str(e)) >>>>> >>>>> Hi Robert, >>>>> >>>>> This doesn't sound right, where is this exception being printed a >>>>> second time? The point of "BBHandledException" is to say "don't >>>>> print >>>>> any further traces for this exception". If this fixes the bug, it >>>>> means >>>>> something somewhere is printing a trace for a second time when it >>>>> should pass through BBHandledException? >>>> >>>> Hi RP, >>>> >>>> I found another serious problem when tried to raise >>>> BBHandledException. There >>>> is a deadlock when a recipe is failed to parse, e.g.: >>>> >>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >>>> $ bitbake -p >>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: >>>> 'helloworld' >>>> [hangs] >>>> >>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub >>>> processes >>>> are still existed, and we need kill them manually, otherwise we >>>> can't start >>>> bitbake again. >>> >>> BTW, things becomes much better after the following two patches are >>> merged: >>> bitbake: knotty: Fix for the Second Keyboard Interrupt >>> bitbake: cooker: Cleanup the queue before call process.join() >>> >>> Now we hardly can reproduce the problem: >>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log >>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; >>> done Sorry, the full commands should be: $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done // Rboert >>> >>> It's not easy to hang any more, but still hangs sometimes, I tried to >>> debug it, >>> but didn't find the root cause, the ui/knotty.py can't get event from >>> server, >>> and goes into a dead loop. >>> >>> event = eventHandler.waitEvent(0) >>> if event is None: >>> if main.shutdown > 1: >>> break >>> termfilter.updateFooter() >>> event = eventHandler.waitEvent(0.25) >>> if event is None: >>> continue >>> >>> The main.shutdown is always 0 when it hangs. >> >> In theory there are timeouts there so it should never hang waiting for >> an event. Is it looping and not getting an event? or is the other end >> disconnected? >> >> I guess the question is what we can do to detect a dead connection, or >> if the server is still alive, why the server is hanging and not sending >> any events? > > After more investigations, it may hang at two places, they are very rarely to > happen, but does happen, I can use the following command to reproduce it in > 10 minutes: > > $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log > tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done > > * Hangs #1 in cooker.py: > > 2065 # Cleanup the queue before call process.join(), otherwise there > might be > 2066 # deadlocks. > 2067 while True: > 2068 try: > 2069 self.result_queue.get(timeout=0.25) > 2070 except queue.Empty: > 2071 break > > It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work > here, I tried python 3.5.2 and 3.6.7, the later one is a little better, > but still have the problem, I think that it's a bug of python3's > multiprocessing, and we can call self.result_queue.cancel_join_thread() > to fix the problem. > > > * Hangs #2 in cooker.py: > 2073 for process in self.processes: > 2074 if force: > 2075 process.join(.1) > 2076 process.terminate() > 2077 else: > 2078 process.join() > > > It hangs at process.join(), I added debug code there, it is because the process > is alive when join() it, I think that we can use a while loop to check whether > the process is alive or not before join(), and force join() after many tries. > > I will do more testing before send the patches, make sure it won't hang in hours. > > // Robert > >> >> Thanks for the patches, those were tricky issues to track down and >> solve! >> >> Cheers, >> >> Richard >> >> >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-08-19 8:34 ` Robert Yang 2019-08-19 9:04 ` Robert Yang @ 2019-08-22 9:06 ` Robert Yang 1 sibling, 0 replies; 11+ messages in thread From: Robert Yang @ 2019-08-22 9:06 UTC (permalink / raw) To: richard.purdie, bitbake-devel On 8/19/19 4:34 PM, Robert Yang wrote: > > On 8/16/19 7:03 AM, richard.purdie@linuxfoundation.org wrote: >> On Thu, 2019-08-15 at 19:29 +0800, Robert Yang wrote: >>> >>> On 5/14/19 7:02 PM, Robert Yang wrote: >>>> >>>> On 5/12/19 4:28 PM, Richard Purdie wrote: >>>>> On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >>>>>> The bb.fatal() raises BBHandledException() which causes double >>>>>> exceptions, >>>>>> e.g.: >>>>>> >>>>>> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >>>>>> $ bitbake -p >>>>>> [snip] >>>>>> During handling of the above exception, another exception >>>>>> occurred: >>>>>> [snip] >>>>>> ERROR: The following required tools (as specified by HOSTTOOLS) >>>>>> appear to be >>>>>> unavailable in PATH, please install them in order to proceed: >>>>>> hello >>>>>> >>>>>> Use "raise" rather than "raise bb.BBHandledException" to fix >>>>>> the double >>>>>> exceptions. >>>>>> >>>>>> [YOCTO #13267] >>>>>> >>>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >>>>>> --- >>>>>> bitbake/lib/bb/cookerdata.py | 4 +++- >>>>>> 1 file changed, 3 insertions(+), 1 deletion(-) >>>>>> >>>>>> diff --git a/bitbake/lib/bb/cookerdata.py >>>>>> b/bitbake/lib/bb/cookerdata.py >>>>>> index f8ae410..585edc5 100644 >>>>>> --- a/bitbake/lib/bb/cookerdata.py >>>>>> +++ b/bitbake/lib/bb/cookerdata.py >>>>>> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >>>>>> if multiconfig: >>>>>> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >>>>>> ), self.data) >>>>>> - except (SyntaxError, bb.BBHandledException): >>>>>> + except bb.BBHandledException: >>>>>> + raise >>>>>> + except SyntaxError: >>>>>> raise bb.BBHandledException >>>>>> except bb.data_smart.ExpansionError as e: >>>>>> logger.error(str(e)) >>>>> >>>>> Hi Robert, >>>>> >>>>> This doesn't sound right, where is this exception being printed a >>>>> second time? The point of "BBHandledException" is to say "don't >>>>> print >>>>> any further traces for this exception". If this fixes the bug, it >>>>> means >>>>> something somewhere is printing a trace for a second time when it >>>>> should pass through BBHandledException? >>>> >>>> Hi RP, >>>> >>>> I found another serious problem when tried to raise >>>> BBHandledException. There >>>> is a deadlock when a recipe is failed to parse, e.g.: >>>> >>>> $ echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >>>> $ bitbake -p >>>> meta/recipes-extended/bash/bash_4.4.18.bb:42: unparsed line: >>>> 'helloworld' >>>> [hangs] >>>> >>>> Then bitbake hangs, we can use Ctrl-C to break it, but the sub >>>> processes >>>> are still existed, and we need kill them manually, otherwise we >>>> can't start >>>> bitbake again. >>> >>> BTW, things becomes much better after the following two patches are >>> merged: >>> bitbake: knotty: Fix for the Second Keyboard Interrupt >>> bitbake: cooker: Cleanup the queue before call process.join() >>> >>> Now we hardly can reproduce the problem: >>> echo helloworld >> meta/recipes-extended/bash/bash_4.4.18.bb >>> $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log >>> tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; >>> done >>> >>> It's not easy to hang any more, but still hangs sometimes, I tried to >>> debug it, >>> but didn't find the root cause, the ui/knotty.py can't get event from >>> server, >>> and goes into a dead loop. >>> >>> event = eventHandler.waitEvent(0) >>> if event is None: >>> if main.shutdown > 1: >>> break >>> termfilter.updateFooter() >>> event = eventHandler.waitEvent(0.25) >>> if event is None: >>> continue >>> >>> The main.shutdown is always 0 when it hangs. >> >> In theory there are timeouts there so it should never hang waiting for >> an event. Is it looping and not getting an event? or is the other end >> disconnected? >> >> I guess the question is what we can do to detect a dead connection, or >> if the server is still alive, why the server is hanging and not sending >> any events? > > After more investigations, it may hang at two places, they are very rarely to > happen, but does happen, I can use the following command to reproduce it in > 10 minutes: > > $ while true; do kill-bb; rm -fr bitbake-cookerdaemon.log > tmp/cache/default-glibc/qemux86-64/x86_64/bb_cache.dat* ; bitbake -p; done > > * Hangs #1 in cooker.py: > > 2065 # Cleanup the queue before call process.join(), otherwise there > might be > 2066 # deadlocks. > 2067 while True: > 2068 try: > 2069 self.result_queue.get(timeout=0.25) > 2070 except queue.Empty: > 2071 break > > It hangs at self.result_queue.get(timeout=0.25), the timeout doesn't work > here, I tried python 3.5.2 and 3.6.7, the later one is a little better, > but still have the problem, I think that it's a bug of python3's > multiprocessing, and we can call self.result_queue.cancel_join_thread() > to fix the problem. > > > * Hangs #2 in cooker.py: > 2073 for process in self.processes: > 2074 if force: > 2075 process.join(.1) > 2076 process.terminate() > 2077 else: > 2078 process.join() > > > It hangs at process.join(), I added debug code there, it is because the process > is alive when join() it, I think that we can use a while loop to check whether > the process is alive or not before join(), and force join() after many tries. > > I will do more testing before send the patches, make sure it won't hang in hours. After a lot tries, the only reliable way to avoid hanging is os.kill(), I've sent the patch for it, and added reason in the commit message. // Robert > > // Robert > >> >> Thanks for the patches, those were tricky issues to track down and >> solve! >> >> Cheers, >> >> Richard >> >> >> ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() 2019-05-12 8:28 ` Richard Purdie 2019-05-14 11:02 ` Robert Yang @ 2019-08-15 10:55 ` Robert Yang 1 sibling, 0 replies; 11+ messages in thread From: Robert Yang @ 2019-08-15 10:55 UTC (permalink / raw) To: Richard Purdie, bitbake-devel Hi RP, On 5/12/19 4:28 PM, Richard Purdie wrote: > On Thu, 2019-05-09 at 16:03 +0800, Robert Yang wrote: >> The bb.fatal() raises BBHandledException() which causes double exceptions, >> e.g.: >> >> Add 'HOSTTOOLS += "hello"' to conf/local.conf: >> $ bitbake -p >> [snip] >> During handling of the above exception, another exception occurred: >> [snip] >> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed: >> hello >> >> Use "raise" rather than "raise bb.BBHandledException" to fix the double >> exceptions. >> >> [YOCTO #13267] >> >> Signed-off-by: Robert Yang <liezhi.yang@windriver.com> >> --- >> bitbake/lib/bb/cookerdata.py | 4 +++- >> 1 file changed, 3 insertions(+), 1 deletion(-) >> >> diff --git a/bitbake/lib/bb/cookerdata.py >> b/bitbake/lib/bb/cookerdata.py >> index f8ae410..585edc5 100644 >> --- a/bitbake/lib/bb/cookerdata.py >> +++ b/bitbake/lib/bb/cookerdata.py >> @@ -301,7 +301,9 @@ class CookerDataBuilder(object): >> if multiconfig: >> bb.event.fire(bb.event.MultiConfigParsed(self.mcdata >> ), self.data) >> >> - except (SyntaxError, bb.BBHandledException): >> + except bb.BBHandledException: >> + raise >> + except SyntaxError: >> raise bb.BBHandledException >> except bb.data_smart.ExpansionError as e: >> logger.error(str(e)) > > Hi Robert, > > This doesn't sound right, where is this exception being printed a > second time? The point of "BBHandledException" is to say "don't print > any further traces for this exception". If this fixes the bug, it means > something somewhere is printing a trace for a second time when it > should pass through BBHandledException? I think that I've found the root cause, it is because no code handles BBHandledException during server is starting, so we will *always* get (double) exceptions when BBHandledException comes during server is starting, such as call bb.fatal() during parsing configurations files. I will send a patch to fix it. // Robert > > Cheers, > > Richard > > ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2019-08-22 9:05 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-05-09 8:03 [PATCH 0/1] bitbake: cookerdata: Avoid double exceptions for bb.fatal() Robert Yang 2019-05-09 8:03 ` [PATCH 1/1] " Robert Yang 2019-05-12 8:28 ` Richard Purdie 2019-05-14 11:02 ` Robert Yang 2019-05-20 8:55 ` Robert Yang 2019-08-15 11:29 ` Robert Yang 2019-08-15 23:03 ` richard.purdie 2019-08-19 8:34 ` Robert Yang 2019-08-19 9:04 ` Robert Yang 2019-08-22 9:06 ` Robert Yang 2019-08-15 10:55 ` Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.