[bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

[bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

[Nikolay Papenkov]

From: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>

bitbake process has a deadlock in cooker.py, function parse_generator.
The loop expects that all threads will increment self.parsed variable until it
reaches self.toparse. When a random parse thread(s) are killed, the procedure
is not finished and as a result self.parsed never reaches self.toparse.

The fix breaks the loop after 25 seconds of no updates in self.parsed
number.

Signed-off-by: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
---
 bitbake/lib/bb/cooker.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af794b4c42..0c07a073fa 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -2208,10 +2208,19 @@ class CookerParser(object):
             yield not cached, mc, infos
 
     def parse_generator(self):
+        current_parsed = -1
+        no_updates_iter = 0
         while True:
+            if current_parsed == self.parsed:
+                no_updates_iter += 1
+                # Give up after 2 minutes
+                if no_updates_iter > 480:
+                    raise bb.parse.ParseError("No updates of parsed items for 2 minutes. Probably parsing thread was killed. Exiting.", None)
+            else:
+                no_updates_iter = 0
+                current_parsed = self.parsed
             if self.parsed >= self.toparse:
                 break
-
             try:
                 result = self.result_queue.get(timeout=0.25)
             except queue.Empty:
-- 
2.17.1

Re: [bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

[Nikolay Papenkov]

Hello list,

Any comments on the patch? The solution is rough and straightforward but what approach to the problem can be accepted?

Regards,
Nikolay

----- Original Message -----
From: "Nikolay Papenkov" <n.papenkov@inango-systems.com>
To: bitbake-devel@lists.openembedded.org
Cc: "Nikolay Papenkov" <n.papenkov@inango-systems.com>, "Dmytro Slovinskyi" <d.slovinskyi@inango-systems.com>
Sent: Tuesday, October 26, 2021 10:42:17 AM
Subject: [bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

From: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>

bitbake process has a deadlock in cooker.py, function parse_generator.
The loop expects that all threads will increment self.parsed variable until it
reaches self.toparse. When a random parse thread(s) are killed, the procedure
is not finished and as a result self.parsed never reaches self.toparse.

The fix breaks the loop after 25 seconds of no updates in self.parsed
number.

Signed-off-by: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
---
 bitbake/lib/bb/cooker.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af794b4c42..0c07a073fa 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -2208,10 +2208,19 @@ class CookerParser(object):
             yield not cached, mc, infos
 
     def parse_generator(self):
+        current_parsed = -1
+        no_updates_iter = 0
         while True:
+            if current_parsed == self.parsed:
+                no_updates_iter += 1
+                # Give up after 2 minutes
+                if no_updates_iter > 480:
+                    raise bb.parse.ParseError("No updates of parsed items for 2 minutes. Probably parsing thread was killed. Exiting.", None)
+            else:
+                no_updates_iter = 0
+                current_parsed = self.parsed
             if self.parsed >= self.toparse:
                 break
-
             try:
                 result = self.result_queue.get(timeout=0.25)
             except queue.Empty:
-- 
2.17.1
-- 
Nikolay Papenkov 
Architect, Arch group 


Inango. The Service Orchestrator. 
web. [ https://mail.inango.com/inango.com | inango.com ] 
mail. [ mailto:n.papenkov@inango-systems.com | n.papenkov@inango-systems.com ] 
tel. +357 99 615176 

The information contained in this message is proprietary to Inango Systems, protected from disclosure, and may be privileged. The information is intended to be conveyed only to the designated recipient(s) of the message. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.

Re: [bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

[Richard Purdie]

On Tue, 2021-10-26 at 07:42 +0000, Nikolay Papenkov via lists.openembedded.org
wrote:
> From: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
> 
> bitbake process has a deadlock in cooker.py, function parse_generator.
> The loop expects that all threads will increment self.parsed variable until it
> reaches self.toparse. When a random parse thread(s) are killed, the procedure
> is not finished and as a result self.parsed never reaches self.toparse.
> 
> The fix breaks the loop after 25 seconds of no updates in self.parsed
> number.
> 
> Signed-off-by: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
> ---
>  bitbake/lib/bb/cooker.py | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index af794b4c42..0c07a073fa 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -2208,10 +2208,19 @@ class CookerParser(object):
>              yield not cached, mc, infos
>  
>      def parse_generator(self):
> +        current_parsed = -1
> +        no_updates_iter = 0
>          while True:
> +            if current_parsed == self.parsed:
> +                no_updates_iter += 1
> +                # Give up after 2 minutes
> +                if no_updates_iter > 480:
> +                    raise bb.parse.ParseError("No updates of parsed items for 2 minutes. Probably parsing thread was killed. Exiting.", None)
> +            else:
> +                no_updates_iter = 0
> +                current_parsed = self.parsed
>              if self.parsed >= self.toparse:
>                  break
> -
>              try:
>                  result = self.result_queue.get(timeout=0.25)
>              except queue.Empty:

Thanks for reporting this, it does explain a few strange situations we've seen
bitbake hung in and gives us something to fix.

I wasn't sure the timeout was the best way to handle this so I had a further
look at the code and I think we can use the is_alive() test to see when all
parse threads have exited. I've sent out a patch, see what you think and if that
resolves the issue you were seeing?

Cheers,

Richard

Re: [bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

[Nikolay Papenkov]

Hello Richard,

Thanks for proposing a better solution. With this fix we don't observe cooker hang anymore and as soon as the last not-killed thread finishes parsing we see error message:
"ERROR: ParseError in None: Not all recipes parsed, parser thread killed/died? Exiting"
as expected.

Regards,
Nikolay

----- Original Message -----
From: "richard purdie" <richard.purdie@linuxfoundation.org>
To: "Nikolay Papenkov" <n.papenkov@inango-systems.com>, "bitbake-devel" <bitbake-devel@lists.openembedded.org>
Cc: "Dmytro Slovinskyi" <d.slovinskyi@inango-systems.com>
Sent: Monday, November 8, 2021 3:32:14 PM
Subject: Re: [bitbake-devel][PATCH] bitbake: a cooker is stuck for a long time

On Tue, 2021-10-26 at 07:42 +0000, Nikolay Papenkov via lists.openembedded.org
wrote:
> From: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
> 
> bitbake process has a deadlock in cooker.py, function parse_generator.
> The loop expects that all threads will increment self.parsed variable until it
> reaches self.toparse. When a random parse thread(s) are killed, the procedure
> is not finished and as a result self.parsed never reaches self.toparse.
> 
> The fix breaks the loop after 25 seconds of no updates in self.parsed
> number.
> 
> Signed-off-by: Dmytro Slovinskyi <d.slovinskyi@inango-systems.com>
> ---
>  bitbake/lib/bb/cooker.py | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index af794b4c42..0c07a073fa 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -2208,10 +2208,19 @@ class CookerParser(object):
>              yield not cached, mc, infos
>  
>      def parse_generator(self):
> +        current_parsed = -1
> +        no_updates_iter = 0
>          while True:
> +            if current_parsed == self.parsed:
> +                no_updates_iter += 1
> +                # Give up after 2 minutes
> +                if no_updates_iter > 480:
> +                    raise bb.parse.ParseError("No updates of parsed items for 2 minutes. Probably parsing thread was killed. Exiting.", None)
> +            else:
> +                no_updates_iter = 0
> +                current_parsed = self.parsed
>              if self.parsed >= self.toparse:
>                  break
> -
>              try:
>                  result = self.result_queue.get(timeout=0.25)
>              except queue.Empty:

Thanks for reporting this, it does explain a few strange situations we've seen
bitbake hung in and gives us something to fix.

I wasn't sure the timeout was the best way to handle this so I had a further
look at the code and I think we can use the is_alive() test to see when all
parse threads have exited. I've sent out a patch, see what you think and if that
resolves the issue you were seeing?

Cheers,

Richard




-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#12942): https://lists.openembedded.org/g/bitbake-devel/message/12942
Mute This Topic: https://lists.openembedded.org/mt/86597730/6123239
Group Owner: bitbake-devel+owner@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [n.papenkov@inango-systems.com]
-=-=-=-=-=-=-=-=-=-=-=-
-- 
Nikolay Papenkov 
Architect, Arch group 


Inango. The Service Orchestrator. 
web. [ https://mail.inango.com/inango.com | inango.com ] 
mail. [ mailto:n.papenkov@inango-systems.com | n.papenkov@inango-systems.com ] 
tel. +357 99 615176 

The information contained in this message is proprietary to Inango Systems, protected from disclosure, and may be privileged. The information is intended to be conveyed only to the designated recipient(s) of the message. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, use, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer.