[PATCH 0/2] Preserve extended attributes in sstate objects

[PATCH 0/2] Preserve extended attributes in sstate objects

[Joshua Lock]

This small series is part of a larger, on-going, effort to better support  extended attributes (xattr).

The goal of sending these two patches before the whole is complete is to improve support for distros with features that rely on xattr that wish to use meta-swupd.

meta-swupd creates sstate objects of update artefacts so that an OS update delta can be generated against a previous OS release without having to have the full build history in TMPDIR — this is especially useful for CI workflows.
Without these changes sstate objects don't preserve xattr and thus swupd updates artefacts are incorrect/incomplete.

Regards,

Joshua

The following changes since commit 00027aee12f4bbc9a4ba607c91fcc1e0e8257fa2:

  core-image-kernel-dev.bb: Standardize use of _append and leading space. (2016-08-25 23:03:26 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib joshuagl/xattr
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/xattr

Joshua Lock (2):
  buildtools-tarball: add tar
  oe.path: preserve xattr in copytree() and copyhardlinktree()

 meta/lib/oe/path.py                          | 11 ++++++++---
 meta/recipes-core/meta/buildtools-tarball.bb |  1 +
 2 files changed, 9 insertions(+), 3 deletions(-)

-- 
2.7.4

[PATCH 1/2] buildtools-tarball: add tar

[Joshua Lock]

We'd like to make use of features which aren't available in tar
versions prior to 1.27, therefore add tar to the buildtools-tarball
so that older distributions can still perform builds.

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/recipes-core/meta/buildtools-tarball.bb | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/meta/buildtools-tarball.bb b/meta/recipes-core/meta/buildtools-tarball.bb
index 34df531..f28a8c3 100644
--- a/meta/recipes-core/meta/buildtools-tarball.bb
+++ b/meta/recipes-core/meta/buildtools-tarball.bb
@@ -24,6 +24,7 @@ TOOLCHAIN_HOST_TASK ?= "\
     nativesdk-ca-certificates \
     nativesdk-texinfo \
     nativesdk-locale-base-en-us \
+    nativesdk-tar \
     "
 
 SDK_PACKAGE_ARCHS += "buildtools-dummy-${SDKPKGSUFFIX}"
-- 
2.7.4

[PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Joshua Lock]

Pass appropriate options to tar invocations in copytree() and
copyhardlinktree() to ensure that any extended attributes on the files
are preserved during the copy.

We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree()
because cpio doesn't support extended attributes on files. Instead we
revert back to using cp with different patterns depending on whether
or not the directory contains dot files.

[YOCTO #9857]

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
---
 meta/lib/oe/path.py | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 3c07df3..631c3b4 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -65,7 +65,7 @@ def copytree(src, dst):
     # This way we also preserve hardlinks between files in the tree.
 
     bb.utils.mkdirhier(dst)
-    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
+    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
     subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
 
 def copyhardlinktree(src, dst):
@@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
     if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
         # Need to copy directories only with tar first since cp will error if two 
         # writers try and create a directory at the same time
-        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
+        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
-        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)
+        if os.path.isdir(src):
+            import glob
+            if len(glob.glob('%s/.??*' % src)) > 0:
+                src = src + '/.??* '
+            src = src + '/*'
+        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
         subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
     else:
         copytree(src, dst)
-- 
2.7.4

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Mark Hatle]

On 8/26/16 10:43 AM, Joshua Lock wrote:
> Pass appropriate options to tar invocations in copytree() and
> copyhardlinktree() to ensure that any extended attributes on the files
> are preserved during the copy.
> 
> We have to drop the use cpio in "Copy-pass" mode in copyhardlinktree()
> because cpio doesn't support extended attributes on files. Instead we
> revert back to using cp with different patterns depending on whether
> or not the directory contains dot files.
> 
> [YOCTO #9857]
> 
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
>  meta/lib/oe/path.py | 11 ++++++++---
>  1 file changed, 8 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 3c07df3..631c3b4 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -65,7 +65,7 @@ def copytree(src, dst):
>      # This way we also preserve hardlinks between files in the tree.
>  
>      bb.utils.mkdirhier(dst)
> -    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
> +    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
>      subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>  
>  def copyhardlinktree(src, dst):
> @@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
>      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
>          # Need to copy directories only with tar first since cp will error if two 
>          # writers try and create a directory at the same time
> -        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
> +        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
>          subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> -        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)

The reason why cpio was originally used had to do with filenames that had spaces
or other special characters in them.  (this is why the -print0 was used)

I don't object to the change, but we need to make sure that that will still work
properly.

(Also I thought CPIO -could- do xattrs in the current versions...)

--Mark

> +        if os.path.isdir(src):
> +            import glob
> +            if len(glob.glob('%s/.??*' % src)) > 0:
> +                src = src + '/.??* '
> +            src = src + '/*'
> +        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
>          subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>      else:
>          copytree(src, dst)
> 

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Joshua Lock]

On Fri, 2016-08-26 at 10:48 -0500, Mark Hatle wrote:
> On 8/26/16 10:43 AM, Joshua Lock wrote:
> > 
> > Pass appropriate options to tar invocations in copytree() and
> > copyhardlinktree() to ensure that any extended attributes on the
> > files
> > are preserved during the copy.
> > 
> > We have to drop the use cpio in "Copy-pass" mode in
> > copyhardlinktree()
> > because cpio doesn't support extended attributes on files. Instead
> > we
> > revert back to using cp with different patterns depending on
> > whether
> > or not the directory contains dot files.
> > 
> > [YOCTO #9857]
> > 
> > Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> > ---
> >  meta/lib/oe/path.py | 11 ++++++++---
> >  1 file changed, 8 insertions(+), 3 deletions(-)
> > 
> > diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> > index 3c07df3..631c3b4 100644
> > --- a/meta/lib/oe/path.py
> > +++ b/meta/lib/oe/path.py
> > @@ -65,7 +65,7 @@ def copytree(src, dst):
> >      # This way we also preserve hardlinks between files in the
> > tree.
> >  
> >      bb.utils.mkdirhier(dst)
> > -    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
> > +    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . |
> > tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
> >      subprocess.check_output(cmd, shell=True,
> > stderr=subprocess.STDOUT)
> >  
> >  def copyhardlinktree(src, dst):
> > @@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
> >      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
> >          # Need to copy directories only with tar first since cp
> > will error if two 
> >          # writers try and create a directory at the same time
> > -        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
> > --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
> > +        cmd = "cd %s; find . -type d -print | tar --xattrs --
> > xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - |
> > tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
> >          subprocess.check_output(cmd, shell=True,
> > stderr=subprocess.STDOUT)
> > -        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' %
> > (src, dst)
> 
> The reason why cpio was originally used had to do with filenames that
> had spaces
> or other special characters in them.  (this is why the -print0 was
> used)
> 
> I don't object to the change, but we need to make sure that that will
> still work
> properly.
> 
> (Also I thought CPIO -could- do xattrs in the current versions...)

It doesn't preserve xattrs the way we currently invoke it and I can't
see anything about xattrs in the man pages on Ubuntu or Fedora (which
both have different man pages for cpio)

I'm going to submit a v2 of this series that:

a) adds a test to ensure files with spaces and other special characters
are correctly copied.
b) adds a test to ensure that xattr on a file are preserved.

Regards,

Joshua

> 
> --Mark
> 
> > 
> > +        if os.path.isdir(src):
> > +            import glob
> > +            if len(glob.glob('%s/.??*' % src)) > 0:
> > +                src = src + '/.??* '
> > +            src = src + '/*'
> > +        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
> >          subprocess.check_output(cmd, shell=True,
> > stderr=subprocess.STDOUT)
> >      else:
> >          copytree(src, dst)
> > 
> 

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Richard Purdie]

On Fri, 2016-08-26 at 10:48 -0500, Mark Hatle wrote:
> The reason why cpio was originally used had to do with filenames that
> had spaces
> or other special characters in them.  (this is why the -print0 was
> used)
> 
> I don't object to the change, but we need to make sure that that will
> still work
> properly.
> 
> (Also I thought CPIO -could- do xattrs in the current versions...)

FWIW cpio itself doesn't appear to support xattrs, nor is it likely to
any time soon. It seems people had to use bsdcpio to get that
support...

Cheers,

Richard

Re: [PATCH 1/2] buildtools-tarball: add tar

[Paul Eggleton]

On Fri, 26 Aug 2016 16:43:05 Joshua Lock wrote:
> We'd like to make use of features which aren't available in tar
> versions prior to 1.27, therefore add tar to the buildtools-tarball
> so that older distributions can still perform builds.
> 
> Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
> ---
>  meta/recipes-core/meta/buildtools-tarball.bb | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/recipes-core/meta/buildtools-tarball.bb
> b/meta/recipes-core/meta/buildtools-tarball.bb index 34df531..f28a8c3
> 100644
> --- a/meta/recipes-core/meta/buildtools-tarball.bb
> +++ b/meta/recipes-core/meta/buildtools-tarball.bb
> @@ -24,6 +24,7 @@ TOOLCHAIN_HOST_TASK ?= "\
>      nativesdk-ca-certificates \
>      nativesdk-texinfo \
>      nativesdk-locale-base-en-us \
> +    nativesdk-tar \
>      "
> 
>  SDK_PACKAGE_ARCHS += "buildtools-dummy-${SDKPKGSUFFIX}"

Er, tar is already in buildtools-tarball... right?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Nicolas Dechesne]

On Fri, Aug 26, 2016 at 5:43 PM, Joshua Lock <joshua.g.lock@intel.com> wrote:
>
> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
> index 3c07df3..631c3b4 100644
> --- a/meta/lib/oe/path.py
> +++ b/meta/lib/oe/path.py
> @@ -65,7 +65,7 @@ def copytree(src, dst):
>      # This way we also preserve hardlinks between files in the tree.
>
>      bb.utils.mkdirhier(dst)
> -    cmd = 'tar -cf - -C %s -p . | tar -xf - -C %s' % (src, dst)
> +    cmd = "tar --xattrs --xattrs-include='*' -cf - -C %s -p . | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, dst)
>      subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>
>  def copyhardlinktree(src, dst):
> @@ -77,9 +77,14 @@ def copyhardlinktree(src, dst):
>      if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
>          # Need to copy directories only with tar first since cp will error if two
>          # writers try and create a directory at the same time
> -        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --no-recursion --files-from - | tar -xf - -C %s' % (src, src, dst)
> +        cmd = "cd %s; find . -type d -print | tar --xattrs --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
>          subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
> -        cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst)
> +        if os.path.isdir(src):
> +            import glob
> +            if len(glob.glob('%s/.??*' % src)) > 0:
> +                src = src + '/.??* '
> +            src = src + '/*'
> +        cmd = 'cp -afl --preserve=xattr %s %s' % (src, dst)
>          subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>      else:
>          copytree(src, dst)


Now that this got merged, I am seeing the following error:

==
Exception: subprocess.CalledProcessError: Command 'cp -afl
--preserve=xattr
/home/nicolas.dechesne/work/oe-rpb-master/build-rpb/tmp-rpb-glibc/work/x86_64-linux/quilt-native/0.64-r0/license-destdir/*
/home/nicolas.dechesne/work/oe-rpb-master/build-rpb/tmp-rpb-glibc/deploy/licenses'
returned non-zero exit status 1

Subprocess output:
cp: cannot preserve extended attributes, cp is built without xattr support
==

What happens is that I am installing coreutils in my images, and i
have a copy of 'cp' in my sysroot *and* I don't have xattrs in
DISTRO_FEATURES. As a consequence the 'cp' command which is invoked
does not have xattrs support, while my host 'cp' command has it...

e.g.

~/work/oe-rpb-master/build-rpb$
tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr
foo bar
tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve
extended attributes, cp is built without xattr support

while this one works fine:

~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar

Should we revert these patches or am I missing anything?

thanks

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Richard Purdie]

On Tue, 2016-09-06 at 17:22 +0200, Nicolas Dechesne wrote:
> What happens is that I am installing coreutils in my images, and i
> have a copy of 'cp' in my sysroot *and* I don't have xattrs in
> DISTRO_FEATURES. As a consequence the 'cp' command which is invoked
> does not have xattrs support, while my host 'cp' command has it...
> 
> e.g.
> 
> ~/work/oe-rpb-master/build-rpb$
> tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr
> foo bar
> tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve
> extended attributes, cp is built without xattr support
> 
> while this one works fine:
> 
> ~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar
> 
> Should we revert these patches or am I missing anything?

Did you have:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=92fc3ef9738165300a4daed4c90526dbbad43149

Cheers,

Richard

Re: [PATCH 2/2] oe.path: preserve xattr in copytree() and copyhardlinktree()

[Nicolas Dechesne]

On Tue, Sep 6, 2016 at 5:26 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-09-06 at 17:22 +0200, Nicolas Dechesne wrote:
>> What happens is that I am installing coreutils in my images, and i
>> have a copy of 'cp' in my sysroot *and* I don't have xattrs in
>> DISTRO_FEATURES. As a consequence the 'cp' command which is invoked
>> does not have xattrs support, while my host 'cp' command has it...
>>
>> e.g.
>>
>> ~/work/oe-rpb-master/build-rpb$
>> tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp -afl --preserve=xattr
>> foo bar
>> tmp-rpb-glibc/sysroots/x86_64-linux/usr/bin/cp: cannot preserve
>> extended attributes, cp is built without xattr support
>>
>> while this one works fine:
>>
>> ~/work/oe-rpb-master/build-rpb$ cp -afl --preserve=xattr foo bar
>>
>> Should we revert these patches or am I missing anything?
>
> Did you have:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=92fc3ef9738165300a4daed4c90526dbbad43149

right. I am seeing it. The ERROR above happens when building
quilt-native , after I updated my layers. So it's probably a race
conditition. we need to rebuild quilt-native, so that we can rebuild
coreutils with the change you mention.. but with the --preserve=xattr
patch we can't rebuild quilt-native ;-)

so, i have removed coreutils from my sysroot and it is working now..
but we clearly have an 'upgrade' issue.. i am not going to be the only
one..