[PATCH][master|dunfell|zeus] knotty: ignore pipe errors

[PATCH][master|dunfell|zeus] knotty: ignore pipe errors

[Mikko Rapeli]

Piping bitbake output to another process like less currently
results in an ugly error if the other end of pipe exits
before bitbake has written all output to it.

For example with "bitbake -e linux-yocto | less" and
pressing q to quit less results in:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/mcfrisk/src/yocto/poky/bitbake/lib/bb/event.py", line 133, in print_ui_queue
    sys.stdout.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Fix is to use default signal handler for pipe signal. It's
perfectly normal that on Unix systems the other end of pipe
gets killed before all data is written.

In some cases there seems to also be a loop which results in
a large number of "[Errno 32] Broken pipe" messages printed to
console until knotty process is killed. I think this also
fixes those cases.

This patch applies to master, dunfell and zeus branches.

Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
---
 bitbake/lib/bb/ui/knotty.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/bitbake/lib/bb/ui/knotty.py b/bitbake/lib/bb/ui/knotty.py
index 87e873d644..56ac41d0f9 100644
--- a/bitbake/lib/bb/ui/knotty.py
+++ b/bitbake/lib/bb/ui/knotty.py
@@ -29,6 +29,9 @@ featureSet = [bb.cooker.CookerFeatures.SEND_SANITYEVENTS]
 logger = logging.getLogger("BitBake")
 interactive = sys.stdout.isatty()
 
+# ignore pipe errors since they are normal
+signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+
 class BBProgress(progressbar.ProgressBar):
     def __init__(self, msg, maxval, widgets=None, extrapos=-1, resize_handler=None):
         self.msg = msg
-- 
2.20.1

Re: [bitbake-devel] [PATCH][master|dunfell|zeus] knotty: ignore pipe errors

[Richard Purdie]

On Thu, 2020-07-23 at 17:58 +0300, Mikko Rapeli wrote:
> Piping bitbake output to another process like less currently
> results in an ugly error if the other end of pipe exits
> before bitbake has written all output to it.
> 
> For example with "bitbake -e linux-yocto | less" and
> pressing q to quit less results in:
> 
> Error in atexit._run_exitfuncs:
> Traceback (most recent call last):
>   File "/home/mcfrisk/src/yocto/poky/bitbake/lib/bb/event.py", line
> 133, in print_ui_queue
>     sys.stdout.flush()
> BrokenPipeError: [Errno 32] Broken pipe
> Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w'
> encoding='UTF-8'>
> BrokenPipeError: [Errno 32] Broken pipe
> 
> Fix is to use default signal handler for pipe signal. It's
> perfectly normal that on Unix systems the other end of pipe
> gets killed before all data is written.
> 
> In some cases there seems to also be a loop which results in
> a large number of "[Errno 32] Broken pipe" messages printed to
> console until knotty process is killed. I think this also
> fixes those cases.
> 
> This patch applies to master, dunfell and zeus branches.
> 
> Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> ---
>  bitbake/lib/bb/ui/knotty.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/bitbake/lib/bb/ui/knotty.py
> b/bitbake/lib/bb/ui/knotty.py
> index 87e873d644..56ac41d0f9 100644
> --- a/bitbake/lib/bb/ui/knotty.py
> +++ b/bitbake/lib/bb/ui/knotty.py
> @@ -29,6 +29,9 @@ featureSet =
> [bb.cooker.CookerFeatures.SEND_SANITYEVENTS]
>  logger = logging.getLogger("BitBake")
>  interactive = sys.stdout.isatty()
>  
> +# ignore pipe errors since they are normal
> +signal.signal(signal.SIGPIPE, signal.SIG_DFL)
> +
>  class BBProgress(progressbar.ProgressBar):
>      def __init__(self, msg, maxval, widgets=None, extrapos=-1,
> resize_handler=None):
>          self.msg = msg

I think but have not 100% confirmed this caused the failures in:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1200

(new build running to check)

Cheers,

Richard

Re: [bitbake-devel] [PATCH][master|dunfell|zeus] knotty: ignore pipe errors

[Mikko Rapeli]

On Fri, Jul 24, 2020 at 07:01:46AM +0100, Richard Purdie wrote:
> On Thu, 2020-07-23 at 17:58 +0300, Mikko Rapeli wrote:
> > Piping bitbake output to another process like less currently
> > results in an ugly error if the other end of pipe exits
> > before bitbake has written all output to it.
> > 
> > For example with "bitbake -e linux-yocto | less" and
> > pressing q to quit less results in:
> > 
> > Error in atexit._run_exitfuncs:
> > Traceback (most recent call last):
> >   File "/home/mcfrisk/src/yocto/poky/bitbake/lib/bb/event.py", line
> > 133, in print_ui_queue
> >     sys.stdout.flush()
> > BrokenPipeError: [Errno 32] Broken pipe
> > Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w'
> > encoding='UTF-8'>
> > BrokenPipeError: [Errno 32] Broken pipe
> > 
> > Fix is to use default signal handler for pipe signal. It's
> > perfectly normal that on Unix systems the other end of pipe
> > gets killed before all data is written.
> > 
> > In some cases there seems to also be a loop which results in
> > a large number of "[Errno 32] Broken pipe" messages printed to
> > console until knotty process is killed. I think this also
> > fixes those cases.
> > 
> > This patch applies to master, dunfell and zeus branches.
> > 
> > Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> > ---
> >  bitbake/lib/bb/ui/knotty.py | 3 +++
> >  1 file changed, 3 insertions(+)
> > 
> > diff --git a/bitbake/lib/bb/ui/knotty.py
> > b/bitbake/lib/bb/ui/knotty.py
> > index 87e873d644..56ac41d0f9 100644
> > --- a/bitbake/lib/bb/ui/knotty.py
> > +++ b/bitbake/lib/bb/ui/knotty.py
> > @@ -29,6 +29,9 @@ featureSet =
> > [bb.cooker.CookerFeatures.SEND_SANITYEVENTS]
> >  logger = logging.getLogger("BitBake")
> >  interactive = sys.stdout.isatty()
> >  
> > +# ignore pipe errors since they are normal
> > +signal.signal(signal.SIGPIPE, signal.SIG_DFL)
> > +
> >  class BBProgress(progressbar.ProgressBar):
> >      def __init__(self, msg, maxval, widgets=None, extrapos=-1,
> > resize_handler=None):
> >          self.msg = msg
> 
> I think but have not 100% confirmed this caused the failures in:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1200

Hmm, I can't see the connection based on logs though I don't know the test
setups at all.

> (new build running to check)

Looking forward to these results...

-Mikko

> Cheers,
> 
> Richard

Re: [bitbake-devel] [PATCH][master|dunfell|zeus] knotty: ignore pipe errors

[Richard Purdie]

On Fri, 2020-07-24 at 07:58 +0000, Mikko.Rapeli@bmw.de wrote:
> On Fri, Jul 24, 2020 at 07:01:46AM +0100, Richard Purdie wrote:
> > On Thu, 2020-07-23 at 17:58 +0300, Mikko Rapeli wrote:
> > > Piping bitbake output to another process like less currently
> > > results in an ugly error if the other end of pipe exits
> > > before bitbake has written all output to it.
> > > 
> > > For example with "bitbake -e linux-yocto | less" and
> > > pressing q to quit less results in:
> > > 
> > > Error in atexit._run_exitfuncs:
> > > Traceback (most recent call last):
> > >   File "/home/mcfrisk/src/yocto/poky/bitbake/lib/bb/event.py",
> > > line
> > > 133, in print_ui_queue
> > >     sys.stdout.flush()
> > > BrokenPipeError: [Errno 32] Broken pipe
> > > Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w'
> > > encoding='UTF-8'>
> > > BrokenPipeError: [Errno 32] Broken pipe
> > > 
> > > Fix is to use default signal handler for pipe signal. It's
> > > perfectly normal that on Unix systems the other end of pipe
> > > gets killed before all data is written.
> > > 
> > > In some cases there seems to also be a loop which results in
> > > a large number of "[Errno 32] Broken pipe" messages printed to
> > > console until knotty process is killed. I think this also
> > > fixes those cases.
> > > 
> > > This patch applies to master, dunfell and zeus branches.
> > > 
> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@bmw.de>
> > > ---
> > >  bitbake/lib/bb/ui/knotty.py | 3 +++
> > >  1 file changed, 3 insertions(+)
> > > 
> > > diff --git a/bitbake/lib/bb/ui/knotty.py
> > > b/bitbake/lib/bb/ui/knotty.py
> > > index 87e873d644..56ac41d0f9 100644
> > > --- a/bitbake/lib/bb/ui/knotty.py
> > > +++ b/bitbake/lib/bb/ui/knotty.py
> > > @@ -29,6 +29,9 @@ featureSet =
> > > [bb.cooker.CookerFeatures.SEND_SANITYEVENTS]
> > >  logger = logging.getLogger("BitBake")
> > >  interactive = sys.stdout.isatty()
> > >  
> > > +# ignore pipe errors since they are normal
> > > +signal.signal(signal.SIGPIPE, signal.SIG_DFL)
> > > +
> > >  class BBProgress(progressbar.ProgressBar):
> > >      def __init__(self, msg, maxval, widgets=None, extrapos=-1,
> > > resize_handler=None):
> > >          self.msg = msg
> > 
> > I think but have not 100% confirmed this caused the failures in:
> > 
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1200
> 
> Hmm, I can't see the connection based on logs though I don't know the
> test
> setups at all.
> 
> > (new build running to check)
> 
> Looking forward to these results...

It confirmed this patch is at fault. Looks like devtool/eSDK/tinfoil
are the problem areas at a glance at the failures.

Cheers,

Richard