Re: [bitbake-devel] [PATCH 4/6] server/process: Avoid hanging if a parser process is terminated

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Sat, 2022-03-26 at 20:34 +0000, Richard Purdie via lists.openembedded.org
wrote:
> If a parser process is terminated while holding a write lock, then it
> will lead to a deadlock (see
> https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Process.terminate).
> 
> With SIGTERM, we don't want to terminate holding the lock. We also don't
> want a SIGINT to cause a partial write to the event stream.
> 
> Use signal masks to avoid this.
> 
> Some ideas from Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/server/process.py | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
> index efc3f04b4c..dc331b3957 100644
> --- a/lib/bb/server/process.py
> +++ b/lib/bb/server/process.py
> @@ -20,6 +20,7 @@ import os
>  import sys
>  import time
>  import select
> +import signal
>  import socket
>  import subprocess
>  import errno
> @@ -739,8 +740,12 @@ class ConnectionWriter(object):
>  
>      def send(self, obj):
>          obj = multiprocessing.reduction.ForkingPickler.dumps(obj)
> +        # We must not terminate holding this lock. For SIGTERM, raising afterwards avoids this.
> +        # For SIGINT, we don't want to have written partial data to the pipe.
> +        signal.pthread_sigmask(signal.SIG_BLOCK, (signal.SIGINT, signal.SIGTERM))
>          with self.wlock:
>              self.writer.send_bytes(obj)
> +        signal.pthread_sigmask(signal.SIG_UNBLOCK, (signal.SIGINT, signal.SIGTERM))
>  
>      def fileno(self):
>          return self.writer.fileno()

This doesn't work as you'd expect, see https://bugs.python.org/issue47139 where
I mentioned the issue upstream.

Cheers,

Richard