[oe][Patch] package.bbclass: fix host contamination warnings for source files

[oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939

Source files deployed with the *-dbg packages are owned by the user
running bitbake leading to warnings as the one below.

WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
...

The files are copied as part of the do_package task.
The patch chowns all file in packages/usr/src after cpio copied them into the
package directory.

Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
---


 meta/classes/package.bbclass | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index bdbe96d..d9ef62c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
     # and copied to the destination here.
 
     import stat
+    import subprocess
 
     sourcefile = d.expand("${WORKDIR}/debugsources.list")
     if debugsrcdir and os.path.isfile(sourcefile):
@@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
         if retval:
             bb.fatal("debugsrc symlink fixup failed with exit code %s (cmd was %s)" % (retval, cmd))
 
+        # cpio --no-preserve-owner does not create the destination files with
+        # owner root even when run under pseudo, chown them explicitely.
+        fakerootcmd = d.getVar('FAKEROOTCMD', True)
+        if not os.path.exists(fakerootcmd):
+            logger.error('pseudo executable %s could not be found - have you run a build yet? pseudo-native should install this and if you have run any build then that should have been built')
+            return 2
+        # Set up the appropriate environment
+        newenv = dict(os.environ)
+        fakerootenv = d.getVar('FAKEROOTENV', True)
+        for varvalue in fakerootenv.split():
+            if '=' in varvalue:
+                splitval = varvalue.split('=', 1)
+                newenv[splitval[0]] = splitval[1]
+        kwargs = dict(env = newenv, shell = True)
+
+        cmd = "find '%s%s' -not -uid 0 -exec chown 0:0 {} \;" % (dvar, debugsrcdir)
+        if fakerootcmd is not None:
+            cmd = "%s %s" % (fakerootcmd, cmd)
+        retval = subprocess.call(cmd, **kwargs)
+        if retval:
+            bb.fatal("debugsrc chown failed with code %s (cmd was %s)" % (retval, cmd))
+
         # The copy by cpio may have resulted in some empty directories!  Remove these
         cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
         (retval, output) = oe.utils.getstatusoutput(cmd)
-- 
2.6.2

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Andreas Müller]

On Sun, Apr 3, 2016 at 10:57 PM, Max Krummenacher <max.oss.09@gmail.com> wrote:
> Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
>
> Source files deployed with the *-dbg packages are owned by the user
> running bitbake leading to warnings as the one below.
>
> WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is owned by uid 1000, which is the same as the user running bitbake. This may be due to host contamination
> ...
>
> The files are copied as part of the do_package task.
> The patch chowns all file in packages/usr/src after cpio copied them into the
> package directory.
>
> Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
> ---
>
>
>  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index bdbe96d..d9ef62c 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
>      # and copied to the destination here.
>
>      import stat
> +    import subprocess
>
>      sourcefile = d.expand("${WORKDIR}/debugsources.list")
>      if debugsrcdir and os.path.isfile(sourcefile):
> @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
>          if retval:
>              bb.fatal("debugsrc symlink fixup failed with exit code %s (cmd was %s)" % (retval, cmd))
>
> +        # cpio --no-preserve-owner does not create the destination files with
> +        # owner root even when run under pseudo, chown them explicitely.
> +        fakerootcmd = d.getVar('FAKEROOTCMD', True)
> +        if not os.path.exists(fakerootcmd):
> +            logger.error('pseudo executable %s could not be found - have you run a build yet? pseudo-native should install this and if you have run any build then that should have been built')
> +            return 2
> +        # Set up the appropriate environment
> +        newenv = dict(os.environ)
> +        fakerootenv = d.getVar('FAKEROOTENV', True)
> +        for varvalue in fakerootenv.split():
> +            if '=' in varvalue:
> +                splitval = varvalue.split('=', 1)
> +                newenv[splitval[0]] = splitval[1]
> +        kwargs = dict(env = newenv, shell = True)
> +
> +        cmd = "find '%s%s' -not -uid 0 -exec chown 0:0 {} \;" % (dvar, debugsrcdir)
> +        if fakerootcmd is not None:
> +            cmd = "%s %s" % (fakerootcmd, cmd)
> +        retval = subprocess.call(cmd, **kwargs)
> +        if retval:
> +            bb.fatal("debugsrc chown failed with code %s (cmd was %s)" % (retval, cmd))
> +
>          # The copy by cpio may have resulted in some empty directories!  Remove these
>          cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
>          (retval, output) = oe.utils.getstatusoutput(cmd)
> --
> 2.6.2
>
Thanks a lot - I've applied and started a test build.

Andreas

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Richard Purdie]

On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
> Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
> 
> Source files deployed with the *-dbg packages are owned by the user
> running bitbake leading to warnings as the one below.
> 
> WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc
> -dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid
> 1000, which is the same as the user running bitbake. This may be due
> to host contamination
> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h
> is owned by uid 1000, which is the same as the user running bitbake.
> This may be due to host contamination
> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is
> owned by uid 1000, which is the same as the user running bitbake.
> This may be due to host contamination
> ...
> 
> The files are copied as part of the do_package task.
> The patch chowns all file in packages/usr/src after cpio copied them
> into the
> package directory.
> 
> Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
> ---
> 
> 
>  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/meta/classes/package.bbclass
> b/meta/classes/package.bbclass
> index bdbe96d..d9ef62c 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
>      # and copied to the destination here.
>  
>      import stat
> +    import subprocess
>  
>      sourcefile = d.expand("${WORKDIR}/debugsources.list")
>      if debugsrcdir and os.path.isfile(sourcefile):
> @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
>          if retval:
>              bb.fatal("debugsrc symlink fixup failed with exit code
> %s (cmd was %s)" % (retval, cmd))
>  
> +        # cpio --no-preserve-owner does not create the destination
> files with
> +        # owner root even when run under pseudo, chown them
> explicitely.

How about passing --owner=0:0 to cpio? 

I'm a little worried about why I don't see this failure on my own local
builds.

We have a few cases where things sometimes seem to work out and
sometimes don't and I'd love to get to the bottom of how to reproduce
it and to understand why its different for different people.

Cheers,

Richard

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2889 bytes --]

On Apr 3, 2016 2:45 PM, "Richard Purdie" <richard.purdie@linuxfoundation.org>
wrote:
>
> On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
> > Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
> >
> > Source files deployed with the *-dbg packages are owned by the user
> > running bitbake leading to warnings as the one below.
> >
> > WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc
> > -dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid
> > 1000, which is the same as the user running bitbake. This may be due
> > to host contamination
> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h
> > is owned by uid 1000, which is the same as the user running bitbake.
> > This may be due to host contamination
> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is
> > owned by uid 1000, which is the same as the user running bitbake.
> > This may be due to host contamination
> > ...
> >
> > The files are copied as part of the do_package task.
> > The patch chowns all file in packages/usr/src after cpio copied them
> > into the
> > package directory.
> >
> > Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
> > ---
> >
> >
> >  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> >
> > diff --git a/meta/classes/package.bbclass
> > b/meta/classes/package.bbclass
> > index bdbe96d..d9ef62c 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
> >      # and copied to the destination here.
> >
> >      import stat
> > +    import subprocess
> >
> >      sourcefile = d.expand("${WORKDIR}/debugsources.list")
> >      if debugsrcdir and os.path.isfile(sourcefile):
> > @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
> >          if retval:
> >              bb.fatal("debugsrc symlink fixup failed with exit code
> > %s (cmd was %s)" % (retval, cmd))
> >
> > +        # cpio --no-preserve-owner does not create the destination
> > files with
> > +        # owner root even when run under pseudo, chown them
> > explicitely.
>
> How about passing --owner=0:0 to cpio?
>
> I'm a little worried about why I don't see this failure on my own local
> builds.

Can you dump the dbg rpm from poky and see if the flagged file is there ?
>
> We have a few cases where things sometimes seem to work out and
> sometimes don't and I'd love to get to the bottom of how to reproduce
> it and to understand why its different for different people.
>
> Cheers,
>
> Richard
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 3992 bytes --]

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Hi Khem

2016-04-03 23:53 GMT+02:00 Khem Raj <raj.khem@gmail.com>:
>
> On Apr 3, 2016 2:45 PM, "Richard Purdie"
> <richard.purdie@linuxfoundation.org> wrote:
>>
>> On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
>> > Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
>> >
>> > Source files deployed with the *-dbg packages are owned by the user
>> > running bitbake leading to warnings as the one below.
>> >
>> > WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc
>> > -dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid
>> > 1000, which is the same as the user running bitbake. This may be due
>> > to host contamination
>> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h
>> > is owned by uid 1000, which is the same as the user running bitbake.
>> > This may be due to host contamination
>> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is
>> > owned by uid 1000, which is the same as the user running bitbake.
>> > This may be due to host contamination
>> > ...
>> >
>> > The files are copied as part of the do_package task.
>> > The patch chowns all file in packages/usr/src after cpio copied them
>> > into the
>> > package directory.
>> >
>> > Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
>> > ---
>> >
>> >
>> >  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
>> >  1 file changed, 23 insertions(+)
>> >
>> > diff --git a/meta/classes/package.bbclass
>> > b/meta/classes/package.bbclass
>> > index bdbe96d..d9ef62c 100644
>> > --- a/meta/classes/package.bbclass
>> > +++ b/meta/classes/package.bbclass
>> > @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
>> >      # and copied to the destination here.
>> >
>> >      import stat
>> > +    import subprocess
>> >
>> >      sourcefile = d.expand("${WORKDIR}/debugsources.list")
>> >      if debugsrcdir and os.path.isfile(sourcefile):
>> > @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
>> >          if retval:
>> >              bb.fatal("debugsrc symlink fixup failed with exit code
>> > %s (cmd was %s)" % (retval, cmd))
>> >
>> > +        # cpio --no-preserve-owner does not create the destination
>> > files with
>> > +        # owner root even when run under pseudo, chown them
>> > explicitely.
>>
>> How about passing --owner=0:0 to cpio?
>>
>> I'm a little worried about why I don't see this failure on my own local
>> builds.
>
> Can you dump the dbg rpm from poky and see if the flagged file is there ?
I checked this with -c package followed by -c devshell.
In poky they source files are missing, in angstrom they are in
${WORKDIR}/package/usr/src
but with the 'wrong' owner.

Regards
Max

>>
>> We have a few cases where things sometimes seem to work out and
>> sometimes don't and I'd love to get to the bottom of how to reproduce
>> it and to understand why its different for different people.
>>
>> Cheers,
>>
>> Richard
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

2016-04-03 23:45 GMT+02:00 Richard Purdie <richard.purdie@linuxfoundation.org>:
> On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
>> Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
>>
>> Source files deployed with the *-dbg packages are owned by the user
>> running bitbake leading to warnings as the one below.
>>
>> WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc
>> -dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by uid
>> 1000, which is the same as the user running bitbake. This may be due
>> to host contamination
>> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/monetary.h
>> is owned by uid 1000, which is the same as the user running bitbake.
>> This may be due to host contamination
>> glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h is
>> owned by uid 1000, which is the same as the user running bitbake.
>> This may be due to host contamination
>> ...
>>
>> The files are copied as part of the do_package task.
>> The patch chowns all file in packages/usr/src after cpio copied them
>> into the
>> package directory.
>>
>> Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
>> ---
>>
>>
>>  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
>>  1 file changed, 23 insertions(+)
>>
>> diff --git a/meta/classes/package.bbclass
>> b/meta/classes/package.bbclass
>> index bdbe96d..d9ef62c 100644
>> --- a/meta/classes/package.bbclass
>> +++ b/meta/classes/package.bbclass
>> @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
>>      # and copied to the destination here.
>>
>>      import stat
>> +    import subprocess
>>
>>      sourcefile = d.expand("${WORKDIR}/debugsources.list")
>>      if debugsrcdir and os.path.isfile(sourcefile):
>> @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
>>          if retval:
>>              bb.fatal("debugsrc symlink fixup failed with exit code
>> %s (cmd was %s)" % (retval, cmd))
>>
>> +        # cpio --no-preserve-owner does not create the destination
>> files with
>> +        # owner root even when run under pseudo, chown them
>> explicitely.
>
> How about passing --owner=0:0 to cpio?

That was my first try, it didn't help.
I'm also not sure if do_package() is executed under pseudo by default.
Also since the command is a pipeline, I don't know if pseudo is
propagated from one command to the next.
Actually I feel a bit as a sorcerer apprentice when it comes to
python/pseudo/packaging.

>
> I'm a little worried about why I don't see this failure on my own local
> builds.
I added some text to bugzilla. The gist is: In poky the sources are
not deployed.
No source files, no wrong ownership.

What I did is doing a repo setup of angstrom master.

Removed the following from bblayers.conf because of parsing errors:
  ${TOPDIR}/sources/meta-uav
  ${TOPDIR}/sources/meta-kde4
  ${TOPDIR}/sources/meta-edison
  ${TOPDIR}/sources/meta-atmel
  ${TOPDIR}/sources/meta-minnow
  ${TOPDIR}/sources/meta-altera

And then:
. setup-environment qemux86
bitbake core-image-minimal

Regards
Max

>
> We have a few cases where things sometimes seem to work out and
> sometimes don't and I'd love to get to the bottom of how to reproduce
> it and to understand why its different for different people.
>
> Cheers,
>
> Richard

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Richard Purdie]

On Mon, 2016-04-04 at 00:51 +0200, Max Krummenacher wrote:
> 2016-04-03 23:45 GMT+02:00 Richard Purdie <
> richard.purdie@linuxfoundation.org>:
> > On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
> > How about passing --owner=0:0 to cpio?
> 
> That was my first try, it didn't help.
> I'm also not sure if do_package() is executed under pseudo by
> default.

do_package is definitely run under pseudo.

> Also since the command is a pipeline, I don't know if pseudo is
> propagated from one command to the next.

It almost certainly does stay around. Its one of the reasons I found
the fakeroot changes in your patch odd.

> Actually I feel a bit as a sorcerer apprentice when it comes to
> python/pseudo/packaging.
> 
> > 
> > I'm a little worried about why I don't see this failure on my own
> > local
> > builds.
> I added some text to bugzilla. The gist is: In poky the sources are
> not deployed.

I don't understand that. If you mean playing sources in the -dbg
packages, poky does that too. Is there some specific configuration
Angstrom makes which triggers this?

There was a bug in master recently which broke source file deployment
into -dbg packages but that was a bug everywhere which was then fixed
when we realised.

> No source files, no wrong ownership.
> 
> What I did is doing a repo setup of angstrom master.
> 
> Removed the following from bblayers.conf because of parsing errors:
>   ${TOPDIR}/sources/meta-uav
>   ${TOPDIR}/sources/meta-kde4
>   ${TOPDIR}/sources/meta-edison
>   ${TOPDIR}/sources/meta-atmel
>   ${TOPDIR}/sources/meta-minnow
>   ${TOPDIR}/sources/meta-altera
> 
> And then:
> . setup-environment qemux86
> bitbake core-image-minimal
> 
> Regards
> Max

Cheers,

Richard

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Khem Raj]

On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>> I added some text to bugzilla. The gist is: In poky the sources are
>> not deployed.
>
> I don't understand that. If you mean playing sources in the -dbg
> packages, poky does that too. Is there some specific configuration
> Angstrom makes which triggers this?
>
> There was a bug in master recently which broke source file deployment
> into -dbg packages but that was a bug everywhere which was then fixed
> when we realised.

Max, somewhere in thread you indicated angstrom 2015.12 release which
would be actually using jethro
and if it has been recently fixed in master than it would explain why
it worked on poky/master and not on angstrom
may be this fix should be cherry-picked to jethro.

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Hi Khem

I'm seeing the issue in Angstrom in
 fido, jethro and master, basically since the QA test for host user
contamination has been introduced.

But I only now tried to debug it, meaning I did this on Angstrom
master and Poky master with HEAD of oe-core being no older than 1
week.

2016-04-04 4:36 GMT+02:00 Khem Raj <raj.khem@gmail.com>:
> On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>>> I added some text to bugzilla. The gist is: In poky the sources are
>>> not deployed.
>>
>> I don't understand that. If you mean playing sources in the -dbg
>> packages, poky does that too. Is there some specific configuration
>> Angstrom makes which triggers this?
>>
>> There was a bug in master recently which broke source file deployment
>> into -dbg packages but that was a bug everywhere which was then fixed
>> when we realised.
>
> Max, somewhere in thread you indicated angstrom 2015.12 release which
> would be actually using jethro
> and if it has been recently fixed in master than it would explain why
> it worked on poky/master and not on angstrom
> may be this fix should be cherry-picked to jethro.

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Andreas Müller]

On Mon, Apr 4, 2016 at 4:36 AM, Khem Raj <raj.khem@gmail.com> wrote:
> On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>>> I added some text to bugzilla. The gist is: In poky the sources are
>>> not deployed.
>>
>> I don't understand that. If you mean playing sources in the -dbg
>> packages, poky does that too. Is there some specific configuration
>> Angstrom makes which triggers this?
>>
>> There was a bug in master recently which broke source file deployment
>> into -dbg packages but that was a bug everywhere which was then fixed
>> when we realised.
>
> Max, somewhere in thread you indicated angstrom 2015.12 release which
> would be actually using jethro
> and if it has been recently fixed in master than it would explain why
> it worked on poky/master and not on angstrom
> may be this fix should be cherry-picked to jethro.
I checked Angstrom master but did not find matching commit. Which
commit are you referring to?

Andreas

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Hi Andreas

I guess Richard refers to this one:
http://cgit.openembedded.org/openembedded-core/commit/meta/classes/package.bbclass?h=master-next&id=ecb56a6ae0c870af680da03db9d39703b525fc98
Max

2016-04-04 9:56 GMT+02:00 Andreas Müller <schnitzeltony@googlemail.com>:
> On Mon, Apr 4, 2016 at 4:36 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>>> I added some text to bugzilla. The gist is: In poky the sources are
>>>> not deployed.
>>>
>>> I don't understand that. If you mean playing sources in the -dbg
>>> packages, poky does that too. Is there some specific configuration
>>> Angstrom makes which triggers this?
>>>
>>> There was a bug in master recently which broke source file deployment
>>> into -dbg packages but that was a bug everywhere which was then fixed
>>> when we realised.
>>
>> Max, somewhere in thread you indicated angstrom 2015.12 release which
>> would be actually using jethro
>> and if it has been recently fixed in master than it would explain why
>> it worked on poky/master and not on angstrom
>> may be this fix should be cherry-picked to jethro.
> I checked Angstrom master but did not find matching commit. Which
> commit are you referring to?
>
> Andreas
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Andreas Müller]

On Mon, Apr 4, 2016 at 10:00 AM, Max Krummenacher <max.oss.09@gmail.com> wrote:
> Hi Andreas
>
> I guess Richard refers to this one:
> http://cgit.openembedded.org/openembedded-core/commit/meta/classes/package.bbclass?h=master-next&id=ecb56a6ae0c870af680da03db9d39703b525fc98
> Max
>
Hi Max,

I was addressing Khem as I understood his comment in a way that there
is a related fix in angstrom master (which I cannot find).

Andreas

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Andreas Müller]

On Mon, Apr 4, 2016 at 10:46 AM, Andreas Müller
<schnitzeltony@googlemail.com> wrote:
> On Mon, Apr 4, 2016 at 10:00 AM, Max Krummenacher <max.oss.09@gmail.com> wrote:
>> Hi Andreas
>>
>> I guess Richard refers to this one:
>> http://cgit.openembedded.org/openembedded-core/commit/meta/classes/package.bbclass?h=master-next&id=ecb56a6ae0c870af680da03db9d39703b525fc98
>> Max
>>
Hi Max,

* your patch fixes my warnings
* with Richard's comments I reduced your patch to

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index bdbe96d..64bbeb4 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -410,6 +410,13 @@ def copydebugsources(debugsrcdir, d):
         if retval:
             bb.fatal("debugsrc symlink fixup failed with exit code %s
(cmd was %s)" % (retval, cmd))

+        # cpio --no-preserve-owner does not create the destination files with
+        # owner root even when run under pseudo, chown them explicitely.
+        cmd = "find '%s%s' -not -uid 0 -exec chown 0:0 {} \;" %
(dvar, debugsrcdir)
+        (retval, output) = oe.utils.getstatusoutput(cmd)
+        if retval:
+            bb.fatal("cpio --no-preserve-owner fixup failed with exit
code %s (cmd was %s)" % (retval, cmd))
+
         # The copy by cpio may have resulted in some empty
directories!  Remove these
         cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
         (retval, output) = oe.utils.getstatusoutput(cmd)

and incremental build logs still look quiet (need a scratch build to
be sure). Could you try if that one works for you?

Somehow cpio turns suspicious: my fedora 23 ships 2.11 which is 6
years old. Have checked cpio's (git) logs but did not find something
(there was a matching bugfix 2.10 -> 2.11 - ....)

So how about a survey:

* Which version of cpio does your build host distribution ship
* do you see floods of host contamination warnings for shipped source files?

Andreas

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Richard Purdie]

On Mon, 2016-04-04 at 23:12 +0200, Andreas Müller wrote:
> On Mon, Apr 4, 2016 at 10:46 AM, Andreas Müller
> <schnitzeltony@googlemail.com> wrote:
> > On Mon, Apr 4, 2016 at 10:00 AM, Max Krummenacher <
> > max.oss.09@gmail.com> wrote:
> > > Hi Andreas
> > > 
> > > I guess Richard refers to this one:
> > > http://cgit.openembedded.org/openembedded-core/commit/meta/classe
> > > s/package.bbclass?h=master
> > > -next&id=ecb56a6ae0c870af680da03db9d39703b525fc98
> > > Max
> > > 
> Hi Max,
> 
> * your patch fixes my warnings
> * with Richard's comments I reduced your patch to
> 
> diff --git a/meta/classes/package.bbclass
> b/meta/classes/package.bbclass
> index bdbe96d..64bbeb4 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -410,6 +410,13 @@ def copydebugsources(debugsrcdir, d):
>          if retval:
>              bb.fatal("debugsrc symlink fixup failed with exit code
> %s
> (cmd was %s)" % (retval, cmd))
> 
> +        # cpio --no-preserve-owner does not create the destination
> files with
> +        # owner root even when run under pseudo, chown them
> explicitely.
> +        cmd = "find '%s%s' -not -uid 0 -exec chown 0:0 {} \;" %
> (dvar, debugsrcdir)
> +        (retval, output) = oe.utils.getstatusoutput(cmd)
> +        if retval:
> +            bb.fatal("cpio --no-preserve-owner fixup failed with
> exit
> code %s (cmd was %s)" % (retval, cmd))
> +
>          # The copy by cpio may have resulted in some empty
> directories!  Remove these
>          cmd = "find %s%s -empty -type d -delete" % (dvar,
> debugsrcdir)
>          (retval, output) = oe.utils.getstatusoutput(cmd)
> 
> and incremental build logs still look quiet (need a scratch build to
> be sure). Could you try if that one works for you?
> 
> Somehow cpio turns suspicious: my fedora 23 ships 2.11 which is 6
> years old. Have checked cpio's (git) logs but did not find something
> (there was a matching bugfix 2.10 -> 2.11 - ....)
> 
> So how about a survey:
> 
> * Which version of cpio does your build host distribution ship
> * do you see floods of host contamination warnings for shipped source
> files?

You could install a copy of our buildtools tarball, see if the problem
goes away with that installed?

Cheers,

Richard

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Khem Raj]

On Mon, Apr 4, 2016 at 12:56 AM, Andreas Müller
<schnitzeltony@googlemail.com> wrote:
> On Mon, Apr 4, 2016 at 4:36 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>>> I added some text to bugzilla. The gist is: In poky the sources are
>>>> not deployed.
>>>
>>> I don't understand that. If you mean playing sources in the -dbg
>>> packages, poky does that too. Is there some specific configuration
>>> Angstrom makes which triggers this?
>>>
>>> There was a bug in master recently which broke source file deployment
>>> into -dbg packages but that was a bug everywhere which was then fixed
>>> when we realised.
>>
>> Max, somewhere in thread you indicated angstrom 2015.12 release which
>> would be actually using jethro
>> and if it has been recently fixed in master than it would explain why
>> it worked on poky/master and not on angstrom
>> may be this fix should be cherry-picked to jethro.
> I checked Angstrom master but did not find matching commit. Which
> commit are you referring to?

Just ensure that angstrom master is using master of OE-Core ( latest )
and if problem still is there then second thing would be the toolchain it uses
might be coming from meta-linaro, now see if there are any gcc patches
that are in OE-Core but not in meta-linaro which could cause this kind
of stuff

>
> Andreas

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Hi

2016-04-05 2:44 GMT+02:00 Khem Raj <raj.khem@gmail.com>:
> On Mon, Apr 4, 2016 at 12:56 AM, Andreas Müller
> <schnitzeltony@googlemail.com> wrote:
>> On Mon, Apr 4, 2016 at 4:36 AM, Khem Raj <raj.khem@gmail.com> wrote:
>>> On Sun, Apr 3, 2016 at 4:00 PM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>>>> I added some text to bugzilla. The gist is: In poky the sources are
>>>>> not deployed.
>>>>
>>>> I don't understand that. If you mean playing sources in the -dbg
>>>> packages, poky does that too. Is there some specific configuration
>>>> Angstrom makes which triggers this?
>>>>
>>>> There was a bug in master recently which broke source file deployment
>>>> into -dbg packages but that was a bug everywhere which was then fixed
>>>> when we realised.

I gave some misleading information. I checked out poky a few days
before angstrom and thus poky did not include the commit which fixed
the deployment of the debug sources while angstrom did.
http://cgit.openembedded.org/openembedded-core/commit/meta/classes/package.bbclass?h=master-next&id=ecb56a6ae0c870af680da03db9d39703b525fc98

So with that patch applied my poky build does deploy the sources but
does not show the
host contamination warnings.
Sorry for the red heering.


@ Andreas
* Which version of cpio does your build host distribution ship
openSUSE Leap 42.1: cpio (GNU cpio) 2.11
Fedora 20: cpio (GNU cpio) 2.11

* do you see floods of host contamination warnings for shipped source files?
Yes, on both systems.

@ Richerd
> You could install a copy of our buildtools tarball, see if the problem
> goes away with that installed?

I will try that out. However, as I did setup a poky and an angstrom
build on the same machine likely the host build tools will not be the
culprit.

>>>
>>> Max, somewhere in thread you indicated angstrom 2015.12 release which
>>> would be actually using jethro
>>> and if it has been recently fixed in master than it would explain why
>>> it worked on poky/master and not on angstrom
>>> may be this fix should be cherry-picked to jethro.
>> I checked Angstrom master but did not find matching commit. Which
>> commit are you referring to?
>
> Just ensure that angstrom master is using master of OE-Core ( latest )
> and if problem still is there then second thing would be the toolchain it uses
> might be coming from meta-linaro, now see if there are any gcc patches
> that are in OE-Core but not in meta-linaro which could cause this kind
> of stuff

Note that I tested this with qemux86. For x86 angstrom is not using
the linaro toolchain.
(And I usually build for ARM where I also see the issue)

>>
>> Andreas
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [oe][Patch] package.bbclass: fix host contamination warnings for source files

[Max Krummenacher]

Hi

Am Sonntag, den 03.04.2016, 22:45 +0100 schrieb Richard Purdie:
> On Sun, 2016-04-03 at 22:57 +0200, Max Krummenacher wrote:
> > Addresses https://bugzilla.yoctoproject.org/show_bug.cgi?id=8939
> > 
> > Source files deployed with the *-dbg packages are owned by the user
> > running bitbake leading to warnings as the one below.
> > 
> > WARNING: glibc-2.23-r0 do_package_qa: QA Issue: glibc: /glibc
> > -dbg/usr/src/debug/glibc/2.23-r0/git/include/resolv.h is owned by
> > uid
> > 1000, which is the same as the user running bitbake. This may be
> > due
> > to host contamination
> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23
> > -r0/git/include/monetary.h
> > is owned by uid 1000, which is the same as the user running
> > bitbake.
> > This may be due to host contamination
> > glibc: /glibc-dbg/usr/src/debug/glibc/2.23-r0/git/include/locale.h
> > is
> > owned by uid 1000, which is the same as the user running bitbake.
> > This may be due to host contamination
> > ...
> > 
> > The files are copied as part of the do_package task.
> > The patch chowns all file in packages/usr/src after cpio copied
> > them
> > into the
> > package directory.
> > 
> > Signed-off-by: Max Krummenacher <max.krummenacher@toradex.com>
> > ---
> > 
> > 
> >  meta/classes/package.bbclass | 23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> > 
> > diff --git a/meta/classes/package.bbclass
> > b/meta/classes/package.bbclass
> > index bdbe96d..d9ef62c 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -362,6 +362,7 @@ def copydebugsources(debugsrcdir, d):
> >      # and copied to the destination here.
> >  
> >      import stat
> > +    import subprocess
> >  
> >      sourcefile = d.expand("${WORKDIR}/debugsources.list")
> >      if debugsrcdir and os.path.isfile(sourcefile):
> > @@ -410,6 +411,28 @@ def copydebugsources(debugsrcdir, d):
> >          if retval:
> >              bb.fatal("debugsrc symlink fixup failed with exit code
> > %s (cmd was %s)" % (retval, cmd))
> >  
> > +        # cpio --no-preserve-owner does not create the destination
> > files with
> > +        # owner root even when run under pseudo, chown them
> > explicitely.
> 
> How about passing --owner=0:0 to cpio? 
> 
> I'm a little worried about why I don't see this failure on my own
> local
> builds.
> 
> We have a few cases where things sometimes seem to work out and
> sometimes don't and I'd love to get to the bottom of how to reproduce
> it and to understand why its different for different people.

I finally got enough time to investigate further.

I found cpio -l (i.e. createing hardlinks) under pseudo does not set
the owner to root, neither width --no-preserve-owner nor width -
-owner=0:0.

The file ownership in yocto is corrected later with fs-perms.txt.
Angstrom does provide its own fs perms configuration which disables the
oe-core fs-perms.txt. But due to a bug the angstrom file is not active.
Patch sent to the angstrom ML.
http://article.gmane.org/gmane.linux.distributions.angstrom.devel/7856

Max

> Cheers,
> 
> Richard