[Errno 18] Invalid cross-device link warning coming from license file copy

[Errno 18] Invalid cross-device link warning coming from license file copy

[matti kaasinen]

[-- Attachment #1: Type: text/plain, Size: 827 bytes --]

Is there any way of getting rid of warnings of following type:

WARNING: Could not copy license file /YOCTO/poky/LICENSE to
/XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE:
[Errno 18] Invalid cross-device link

Yes, sources and results are located on different logical drives, but
should it still be possible to copy files between them. Everything else
works fine in bitbaking.

MAIN-
Build Configuration:
BB_VERSION        = "1.26.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-14.04"
TARGET_SYS        = "arm-poky-linux-gnueabi"
MACHINE           = "am335x-evm"
DISTRO            = "poky"
DISTRO_VERSION    = "1.8"
TUNE_FEATURES     = "arm armv7a vfp thumb neon callconvention-hard cortexa8"
TARGET_FPU        = "vfp-neon"

Thanks,
Matti

[-- Attachment #2: Type: text/html, Size: 1127 bytes --]

Re: [Errno 18] Invalid cross-device link warning coming from license file copy

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

On 19 August 2015 at 13:48, matti kaasinen <matti.kaasinen@gmail.com> wrote:

> WARNING: Could not copy license file /YOCTO/poky/LICENSE to
> /XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE:
> [Errno 18] Invalid cross-device link
>
> Yes, sources and results are located on different logical drives, but
> should it still be possible to copy files between them. Everything else
> works fine in bitbaking.


What release?  I have my build and checkouts on different disks and I don't
get this warning.

Ross

[-- Attachment #2: Type: text/html, Size: 921 bytes --]

Re: [Errno 18] Invalid cross-device link warning coming from license file copy

[matti kaasinen]

[-- Attachment #1: Type: text/plain, Size: 4188 bytes --]

Hi!
It is not that easy for me to get rid of these mount points on same drive
as they are made in /etc/fstab file. If I unmount any mountpoint, it does
not have any directory to show anymore. Therefore I need to make some
workaround. Really, system call stat does not recognize mount points on the
same drive. However, df command does recognize. Therefore I'm considering
workaround for this problem using df. Unfortunately it is required to use
pipe for getting results from df. I need to modify
if os.access(src, os.W_OK) and (os.stat(src).st_dev ==
os.stat(destdir).st_dev):
line in copy_license_files function as below:

if os.access(src, os.W_OK) and (os.popen('df -P ' + src).read().split()[11]
== (os.popen('df -P ' + destdir).read().split()[11] ):

I'm pretty sure it's much more slowly, but it makes the important
difference. I suppose this is so seldom problem that poky is not too eager
in patching
poky/meta/classes/license.bbclass with this line.

Is there any way overloading system methods with own methods like recipes
with bbappend? I have not found any, so most likely I just need to patch my
poky with this line locally that makes updates pretty awkward, don't I

Cheers,
Matti

2015-08-24 15:52 GMT+03:00 matti kaasinen <matti.kaasinen@gmail.com>:

> Hi!
> This warning/error is coming from copy_license_files (listed below)
> function in do_populate task:
> #------------------------
> def copy_license_files(lic_files_paths, destdir):
>     import shutil
>
>     bb.utils.mkdirhier(destdir)
>     for (basename, path) in lic_files_paths:
>         try:
>             src = path
>             dst = os.path.join(destdir, basename)
>             if os.path.exists(dst):
>                 os.remove(dst)
>             if os.access(src, os.W_OK) and (os.stat(src).st_dev ==
> os.stat(destdir).st_dev):
>                 os.link(src, dst)
>             else:
>                 shutil.copyfile(src, dst)
>         except Exception as e:
>             bb.warn("Could not copy license file %s to %s: %s" % (src,
> dst, e))
> #-----------------------
> I suppose the idea of :
>             if os.access(src, os.W_OK) and (os.stat(src).st_dev ==
> os.stat(destdir).st_dev):
>
> -test is to find out if source directory and destination directories were
> on different drives. In my case that seems kind of failing in that task.
> "Kind of" because they are in different logical drives that are mounted on
> the same physical drive.
>
> Therefore, link is tried to make instead of copy. However, this fails as
> those directories are on the different logical drives and "OSError: [Errno
> 18] Invalid cross-device link" is thrown.
> I suppose, there is no other workaround to this than removing mountpoint
> from this build directory and instead using it as normal direcotry.
> Original idea was that it could have been mapped anywhere - for instance to
> nfs drive.
>
> Cheers,
> Matti
>
>
> By manually executing
>
> 2015-08-20 15:31 GMT+03:00 matti kaasinen <matti.kaasinen@gmail.com>:
>
>> Ross,
>> it is still unclear to me what release are you referring to. Could you
>> please pinpoint it to me.
>> Thanks,
>> Matti
>>
>> 2015-08-19 16:19 GMT+03:00 matti kaasinen <matti.kaasinen@gmail.com>:
>>
>>> I listed my main build configuration in my previous mail. What else do
>>> you need. In fact I have not updated poky recently.
>>> -Matti
>>>
>>> 2015-08-19 16:00 GMT+03:00 Burton, Ross <ross.burton@intel.com>:
>>>
>>>>
>>>> On 19 August 2015 at 13:48, matti kaasinen <matti.kaasinen@gmail.com>
>>>> wrote:
>>>>
>>>>> WARNING: Could not copy license file /YOCTO/poky/LICENSE to
>>>>> /XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE:
>>>>> [Errno 18] Invalid cross-device link
>>>>>
>>>>> Yes, sources and results are located on different logical drives, but
>>>>> should it still be possible to copy files between them. Everything else
>>>>> works fine in bitbaking.
>>>>
>>>>
>>>> What release?  I have my build and checkouts on different disks and I
>>>> don't get this warning.
>>>>
>>>> Ross
>>>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6367 bytes --]

Re: [Errno 18] Invalid cross-device link warning coming from license file copy

[Richard Purdie]

On Wed, 2015-08-19 at 15:48 +0300, matti kaasinen wrote:
> Is there any way of getting rid of warnings of following type:
> 
> WARNING: Could not copy license file /YOCTO/poky/LICENSE
> to /XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE: [Errno 18] Invalid cross-device link
> 
> 
> Yes, sources and results are located on different logical drives, but
> should it still be possible to copy files between them. Everything
> else works fine in bitbaking.

Can you be a little more clear about what you did please? I assume you
changed TMPDIR to point to a different disk to the one where the poky
checkout is located?

I'm still a little puzzled about why the existing check doesn't catch
the fact they're different disks...

Cheers,

Richard

Re: [Errno 18] Invalid cross-device link warning coming from license file copy

[matti kaasinen]

[-- Attachment #1: Type: text/plain, Size: 3565 bytes --]

RIchard,


2015-08-25 17:12 GMT+03:00 Richard Purdie <
richard.purdie@linuxfoundation.org>:

> On Wed, 2015-08-19 at 15:48 +0300, matti kaasinen wrote:
> > Is there any way of getting rid of warnings of following type:
> >
> > WARNING: Could not copy license file /YOCTO/poky/LICENSE
> > to
> /XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE:
> [Errno 18] Invalid cross-device link
> >
> >
> > Yes, sources and results are located on different logical drives, but
> > should it still be possible to copy files between them. Everything
> > else works fine in bitbaking.
>
> Can you be a little more clear about what you did please? I assume you
> changed TMPDIR to point to a different disk to the one where the poky
> checkout is located?
>

Yes, you are absolutely right.
I have a pretty small (120G) ssd disk I have installed my Ubuntu machine
(/dev/sdb2). I try not to pollute it with checkouts and in particular
builds. For that purpose I have reserved one partition from pretty moderate
(1TB) hdd. Both checkouts (I have played with some distros) and TMPDIR live
in there, but in different mount points. In fact, I do have also a mount
point called /TMPDIR there.

>
> I'm still a little puzzled about why the existing check doesn't catch
> the fact they're different disks...
>

Let's study how it works, first the mounts.
As I told before, both the TMPDIR and checkouts (/YOCTO) live in /dev/sdb2
that has been mounted in /mnt/sw. Let's see how they look in df and python
os.stat listings.

==== df shell command
## /mnt is in ssd disk
$ LANG=C df -H /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb2       110G   24G   80G  23% /

## sda2 partition of hdd disk mounted to /mnt/sw
$ LANG=C df -H /mnt/sw
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       572G  148G  395G  28% /mnt/sw

## same results by using /mnt/sw path
$ LANG=C df -H /mnt/sw/TMPDIR
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       572G  148G  395G  28% /mnt/sw

## same results again - in fact as path used same mountpoint /mnt/sw
$ LANG=C df -H /mnt/sw/YOCTO
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       572G  148G  395G  28% /mnt/sw

## let's test mountpoints /TMPDIR and /YOCTO - this is what
copy_license_file tests:

$ LANG=C df -H /TMPDIR
Filesystem      Size  Used Avail Use% Mounted on
/mnt/sw/TMPDIR  572G  148G  395G  28% /TMPDIR

$ LANG=C df -H /YOCTO
Filesystem      Size  Used Avail Use% Mounted on
/mnt/sw/YOCTO   572G  148G  395G  28% /YOCTO

## It made difference both in "Filesystem" and "Mounted on" columns
## Let's make the test more close what copy_license_files function tests:

$ LANG=C df -H /YOCTO/poky /TMPDIR/work
Filesystem      Size  Used Avail Use% Mounted on
/mnt/sw/YOCTO   572G  148G  395G  28% /YOCTO
/mnt/sw/TMPDIR  572G  148G  395G  28% /TMPDIR

==== stat test using ipython
In [9]: os.stat('/mnt').st_dev
Out[9]: 2066L

In [10]: os.stat('/mnt/sw').st_dev
Out[10]: 2050L

In [11]: os.stat('/mnt/sw/TMPDIR/work').st_dev
Out[11]: 2050L

In [12]: os.stat('/mnt/sw/YOCTO/poky').st_dev
Out[12]: 2050L

In [13]: os.stat('/YOCTO/poky').st_dev
Out[13]: 2050L

In [14]: os.stat('/TMPDIR/work').st_dev
Out[14]: 2050L

## Above python tests show that all the tests except os.stat('/mnt').st_dev
return /dev/sda2 mount point number (I guess).

I hope you can get more clear picture of my mounts and how different kind
of tests work.
Cheers,
Matti

[-- Attachment #2: Type: text/html, Size: 4684 bytes --]

Re: [Errno 18] Invalid cross-device link warning coming from license file copy

[matti kaasinen]

[-- Attachment #1: Type: text/plain, Size: 4408 bytes --]

So, it seems that I have following options to choose:
1) change TMPDIR value to point to /mnt/sw/TMPDIR (portable, but config
file templates require editing for next installers).
  This would require also bblayers.conf file editing with /mnt/sw
modifications.
2) move checkouts to my ssd: checkouts are not that big - possible.
3) patch copy_license_files function: easy and working, but makes updates
awkward.
4) wait for poky to patch copy_license_files function - won't happen I
suppose?
5) do nothing and live with all those warnings - most likely to happen.
Cheers,
Matti

2015-08-26 11:19 GMT+03:00 matti kaasinen <matti.kaasinen@gmail.com>:

> RIchard,
>
>
> 2015-08-25 17:12 GMT+03:00 Richard Purdie <
> richard.purdie@linuxfoundation.org>:
>
>> On Wed, 2015-08-19 at 15:48 +0300, matti kaasinen wrote:
>> > Is there any way of getting rid of warnings of following type:
>> >
>> > WARNING: Could not copy license file /YOCTO/poky/LICENSE
>> > to
>> /XXX/work/am335x_evm-poky-linux-gnueabi/xxxx-image/1.0-r0/license-destdir/xxx-image/LICENSE:
>> [Errno 18] Invalid cross-device link
>> >
>> >
>> > Yes, sources and results are located on different logical drives, but
>> > should it still be possible to copy files between them. Everything
>> > else works fine in bitbaking.
>>
>> Can you be a little more clear about what you did please? I assume you
>> changed TMPDIR to point to a different disk to the one where the poky
>> checkout is located?
>>
>
> Yes, you are absolutely right.
> I have a pretty small (120G) ssd disk I have installed my Ubuntu machine
> (/dev/sdb2). I try not to pollute it with checkouts and in particular
> builds. For that purpose I have reserved one partition from pretty moderate
> (1TB) hdd. Both checkouts (I have played with some distros) and TMPDIR live
> in there, but in different mount points. In fact, I do have also a mount
> point called /TMPDIR there.
>
>>
>> I'm still a little puzzled about why the existing check doesn't catch
>> the fact they're different disks...
>>
>
> Let's study how it works, first the mounts.
> As I told before, both the TMPDIR and checkouts (/YOCTO) live in /dev/sdb2
> that has been mounted in /mnt/sw. Let's see how they look in df and python
> os.stat listings.
>
> ==== df shell command
> ## /mnt is in ssd disk
> $ LANG=C df -H /mnt
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sdb2       110G   24G   80G  23% /
>
> ## sda2 partition of hdd disk mounted to /mnt/sw
> $ LANG=C df -H /mnt/sw
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda2       572G  148G  395G  28% /mnt/sw
>
> ## same results by using /mnt/sw path
> $ LANG=C df -H /mnt/sw/TMPDIR
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda2       572G  148G  395G  28% /mnt/sw
>
> ## same results again - in fact as path used same mountpoint /mnt/sw
> $ LANG=C df -H /mnt/sw/YOCTO
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda2       572G  148G  395G  28% /mnt/sw
>
> ## let's test mountpoints /TMPDIR and /YOCTO - this is what
> copy_license_file tests:
>
> $ LANG=C df -H /TMPDIR
> Filesystem      Size  Used Avail Use% Mounted on
> /mnt/sw/TMPDIR  572G  148G  395G  28% /TMPDIR
>
> $ LANG=C df -H /YOCTO
> Filesystem      Size  Used Avail Use% Mounted on
> /mnt/sw/YOCTO   572G  148G  395G  28% /YOCTO
>
> ## It made difference both in "Filesystem" and "Mounted on" columns
> ## Let's make the test more close what copy_license_files function tests:
>
> $ LANG=C df -H /YOCTO/poky /TMPDIR/work
> Filesystem      Size  Used Avail Use% Mounted on
> /mnt/sw/YOCTO   572G  148G  395G  28% /YOCTO
> /mnt/sw/TMPDIR  572G  148G  395G  28% /TMPDIR
>
> ==== stat test using ipython
> In [9]: os.stat('/mnt').st_dev
> Out[9]: 2066L
>
> In [10]: os.stat('/mnt/sw').st_dev
> Out[10]: 2050L
>
> In [11]: os.stat('/mnt/sw/TMPDIR/work').st_dev
> Out[11]: 2050L
>
> In [12]: os.stat('/mnt/sw/YOCTO/poky').st_dev
> Out[12]: 2050L
>
> In [13]: os.stat('/YOCTO/poky').st_dev
> Out[13]: 2050L
>
> In [14]: os.stat('/TMPDIR/work').st_dev
> Out[14]: 2050L
>
> ## Above python tests show that all the tests except
> os.stat('/mnt').st_dev return /dev/sda2 mount point number (I guess).
>
> I hope you can get more clear picture of my mounts and how different kind
> of tests work.
> Cheers,
> Matti
>
>

[-- Attachment #2: Type: text/html, Size: 5790 bytes --]