Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

[Anton Mikanovich]

08.12.2021 19:57, Anton Mikanovich wrote:
> We should not redirect stderr to stdout if we need to get separated
> stdout and stderr contents from Popen.communicate() later.
>
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>   lib/bb/process.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/bb/process.py b/lib/bb/process.py
> index d5a1775f..af5d804a 100644
> --- a/lib/bb/process.py
> +++ b/lib/bb/process.py
> @@ -60,7 +60,7 @@ class Popen(subprocess.Popen):
>           "close_fds": True,
>           "preexec_fn": subprocess_setup,
>           "stdout": subprocess.PIPE,
> -        "stderr": subprocess.STDOUT,
> +        "stderr": subprocess.PIPE,
>           "stdin": subprocess.PIPE,
>           "shell": False,
>       }

Can anyone review this change to proof the fix is correct?

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

It helps if you describe the problem that occurs without the patch and how
it can be observed.

Ale

On Fri 17. Dec 2021 at 19.43, Anton Mikanovich <amikan@ilbers.de> wrote:

> 08.12.2021 19:57, Anton Mikanovich wrote:
> > We should not redirect stderr to stdout if we need to get separated
> > stdout and stderr contents from Popen.communicate() later.
> >
> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> > ---
> >   lib/bb/process.py | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/lib/bb/process.py b/lib/bb/process.py
> > index d5a1775f..af5d804a 100644
> > --- a/lib/bb/process.py
> > +++ b/lib/bb/process.py
> > @@ -60,7 +60,7 @@ class Popen(subprocess.Popen):
> >           "close_fds": True,
> >           "preexec_fn": subprocess_setup,
> >           "stdout": subprocess.PIPE,
> > -        "stderr": subprocess.STDOUT,
> > +        "stderr": subprocess.PIPE,
> >           "stdin": subprocess.PIPE,
> >           "shell": False,
> >       }
>
> Can anyone review this change to proof the fix is correct?
>
> --
> Anton Mikanovich
> Promwad Ltd.
> External service provider of ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn, Germany
> +49 (89) 122 67 24-0
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13190):
> https://lists.openembedded.org/g/bitbake-devel/message/13190
> Mute This Topic: https://lists.openembedded.org/mt/87592650/1686489
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 2917 bytes --]

Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

[Anton Mikanovich]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

17.12.2021 19:54, Alexander Kanavin wrote:
> It helps if you describe the problem that occurs without the patch and 
> how it can be observed.
>
> Ale
>
> On Fri 17. Dec 2021 at 19.43, Anton Mikanovich <amikan@ilbers.de 
> <mailto:amikan@ilbers.de>> wrote:
>
>     08.12.2021 19:57, Anton Mikanovich wrote:
>     > We should not redirect stderr to stdout if we need to get separated
>     > stdout and stderr contents from Popen.communicate() later.
>     >
>     > Signed-off-by: Anton Mikanovich <amikan@ilbers.de
>     <mailto:amikan@ilbers.de>>
>     > ---
>     >   lib/bb/process.py | 2 +-
>     >   1 file changed, 1 insertion(+), 1 deletion(-)
>     >
>     > diff --git a/lib/bb/process.py b/lib/bb/process.py
>     > index d5a1775f..af5d804a 100644
>     > --- a/lib/bb/process.py
>     > +++ b/lib/bb/process.py
>     > @@ -60,7 +60,7 @@ class Popen(subprocess.Popen):
>     >           "close_fds": True,
>     >           "preexec_fn": subprocess_setup,
>     >           "stdout": subprocess.PIPE,
>     > -        "stderr": subprocess.STDOUT,
>     > +        "stderr": subprocess.PIPE,
>     >           "stdin": subprocess.PIPE,
>     >           "shell": False,
>     >       }
>
>     Can anyone review this change to proof the fix is correct?
>
>     -- 
>     Anton Mikanovich
>     Promwad Ltd.
>     External service provider of ilbers GmbH
>     Maria-Merian-Str. 8
>     85521 Ottobrunn, Germany
>     +49 (89) 122 67 24-0
>     Commercial register Munich, HRB 214197
>     General Manager: Baurzhan Ismagulov
>
>
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>     View/Reply Online (#13190):
>     https://lists.openembedded.org/g/bitbake-devel/message/13190
>     <https://lists.openembedded.org/g/bitbake-devel/message/13190>
>     Mute This Topic:
>     https://lists.openembedded.org/mt/87592650/1686489
>     <https://lists.openembedded.org/mt/87592650/1686489>
>     Group Owner: bitbake-devel+owner@lists.openembedded.org
>     <mailto:bitbake-devel%2Bowner@lists.openembedded.org>
>     Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub
>     <https://lists.openembedded.org/g/bitbake-devel/unsub>
>     [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>]
>     -=-=-=-=-=-=-=-=-=-=-=-
>
Hello, Alexander

As I've already mentioned in description the problem is mixing both 
stderr and stdout of the command to stdout output variable when using 
bb.process.run()
Here are minimal steps to reproduce:

$ cat /usr/bin/test.sh
#!/bin/sh
echo "This is stderr" >&2
echo "This is stdout"

Task:

python do_test() {
     out,err = bb.process.run('test.sh')
     bb.warn('STDOUT:', out or 'EMPTY')
     bb.warn('STDERR:', err or 'EMPTY')
}
addtask do_test

Result without the patch:

WARNING: do_test: STDOUT:This is stderr
This is stdout
WARNING: do_test: STDERR:EMPTY

So both stdout and stderr of test.sh were filled to out variable.

Result after the patch:

WARNING: do_test: STDOUT:This is stdout
WARNING: do_test: STDERR:This is stderr

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov


[-- Attachment #2: Type: text/html, Size: 5687 bytes --]

Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3040 bytes --]

Thanks, looks like the issue is real, so the patch should go in unless
there are other concerns.

Alex

On Fri 17. Dec 2021 at 20.49, Anton Mikanovich <amikan@ilbers.de> wrote:

> 17.12.2021 19:54, Alexander Kanavin wrote:
>
> It helps if you describe the problem that occurs without the patch and how
> it can be observed.
>
> Ale
>
> On Fri 17. Dec 2021 at 19.43, Anton Mikanovich <amikan@ilbers.de> wrote:
>
>> 08.12.2021 19:57, Anton Mikanovich wrote:
>> > We should not redirect stderr to stdout if we need to get separated
>> > stdout and stderr contents from Popen.communicate() later.
>> >
>> > Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
>> > ---
>> >   lib/bb/process.py | 2 +-
>> >   1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/lib/bb/process.py b/lib/bb/process.py
>> > index d5a1775f..af5d804a 100644
>> > --- a/lib/bb/process.py
>> > +++ b/lib/bb/process.py
>> > @@ -60,7 +60,7 @@ class Popen(subprocess.Popen):
>> >           "close_fds": True,
>> >           "preexec_fn": subprocess_setup,
>> >           "stdout": subprocess.PIPE,
>> > -        "stderr": subprocess.STDOUT,
>> > +        "stderr": subprocess.PIPE,
>> >           "stdin": subprocess.PIPE,
>> >           "shell": False,
>> >       }
>>
>> Can anyone review this change to proof the fix is correct?
>>
>> --
>> Anton Mikanovich
>> Promwad Ltd.
>> External service provider of ilbers GmbH
>> Maria-Merian-Str. 8
>> 85521 Ottobrunn, Germany
>> +49 (89) 122 67 24-0
>> Commercial register Munich, HRB 214197
>> General Manager: Baurzhan Ismagulov
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#13190):
>> https://lists.openembedded.org/g/bitbake-devel/message/13190
>> Mute This Topic: https://lists.openembedded.org/mt/87592650/1686489
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
>> alex.kanavin@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>> Hello, Alexander
>
> As I've already mentioned in description the problem is mixing both stderr
> and stdout of the command to stdout output variable when using
> bb.process.run()
> Here are minimal steps to reproduce:
>
> $ cat /usr/bin/test.sh
> #!/bin/sh
> echo "This is stderr" >&2
> echo "This is stdout"
>
> Task:
>
> python do_test() {
>     out,err = bb.process.run('test.sh')
>     bb.warn('STDOUT:', out or 'EMPTY')
>     bb.warn('STDERR:', err or 'EMPTY')
> }
> addtask do_test
>
> Result without the patch:
>
> WARNING: do_test: STDOUT:This is stderr
> This is stdout
> WARNING: do_test: STDERR:EMPTY
>
> So both stdout and stderr of test.sh were filled to out variable.
>
> Result after the patch:
>
> WARNING: do_test: STDOUT:This is stdout
> WARNING: do_test: STDERR:This is stderr
>
> --
> Anton Mikanovich
> Promwad Ltd.
> External service provider of ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn, Germany
> +49 (89) 122 67 24-0
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov
>
>

[-- Attachment #2: Type: text/html, Size: 5849 bytes --]

Re: [bitbake-devel] [PATCH] process: Do not mix stderr with stdout

[Richard Purdie]

On Fri, 2021-12-17 at 21:41 +0300, Alexander Kanavin wrote:
> Thanks, looks like the issue is real, so the patch should go in unless there are
> other concerns.
> 
> > 

It already merged:

https://git.yoctoproject.org/poky/commit/bitbake?id=f8474be059bb02dcb46a425c0a1842ff2d4d2c8d

Cheers,

Richard