All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Hombourger, Cedric" <Cedric_Hombourger@mentor.com>
To: Martin Jansa <Martin.Jansa@gmail.com>
Cc: bitbake-devel <bitbake-devel@lists.openembedded.org>,
	Chris Larson <Chris_Larson@mentor.com>
Subject: Re: [bitbake-devel] [PATCH] fetch2: use relative symlinks for anything pulled from PREMIRRORS
Date: Fri, 27 Nov 2020 15:04:02 +0100	[thread overview]
Message-ID: <92262695-8578-94cc-939b-fcc49af7b428@mentor.com> (raw)
In-Reply-To: <CA+chaQez30Xyf-e+PDMOa5G-Phb9L8HiEuqhhAHSCJPgudhKdQ@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 11835 bytes --]


On 11/27/20 2:54 PM, Martin Jansa wrote:
> Unfortunately this seems to completely break SSTATE_MIRROR handling 
> file:// at least in some setups.
thank you so much for the feedback. I will check what's wrong there.
>
> All setscene tasks are failing, because there is a symlink in 
> sstate-cache, but when trying to use it, it points to relative path 
> into wrong directory.
>
> e.g.
> DEBUG: Executing python function do_package_qa_setscene
> DEBUG: Searching for 
> fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> in paths:
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache
> DEBUG: Defaulting to 
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> for fb/a4/sstate:python3-ifcfg:cor
> texa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz
> DEBUG: Testing URL 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz
> DEBUG: For url ['file', '', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz', 
> '', '', OrderedDict([('downloadfilename', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz')])] 
> comparing ['file', '', '.*', '', '', OrderedDict()] to ['file', '', 
> '/mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/PATH', 
> '', '', OrderedDict()]
> DEBUG: For url 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> returning 
> file:///mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/fb/a4/sstate%3Apython3-ifcfg%3Acortexa72-oe-linux%3A0.21%3Ar0%3Acortexa72%3A3%3Afba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz
> DEBUG: Trying PREMIRRORS
> DEBUG: For url ['file', '', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz', 
> '', '', OrderedDict([('downloadfilename', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz')])] 
> comparing ['file', '', '.*', '', '', OrderedDict()] to ['file', '', 
> '/mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/PATH', 
> '', '', OrderedDict()]
> DEBUG: For url 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> returning 
> file:///mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/fb/a4/sstate%3Apython3-ifcfg%3Acortexa72-oe-linux%3A0.21%3Ar0%3Acortexa72%3A3%3Afba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz
> DEBUG: Searching for 
> fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo 
> in paths:
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache
> DEBUG: Defaulting to 
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo 
> for 
> fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo
> DEBUG: Testing URL 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo
> DEBUG: For url ['file', '', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo', 
> '', '', OrderedDict([('downloadfilename', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo')])] 
> comparing ['file', '', '.*', '', '', OrderedDict()] to ['file', '', 
> '/mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/PATH', 
> '', '', OrderedDict()]
> DEBUG: For url 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo 
> returning 
> file:///mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/fb/a4/sstate%3Apython3-ifcfg%3Acortexa72-oe-linux%3A0.21%3Ar0%3Acortexa72%3A3%3Afba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo
> DEBUG: Trying PREMIRRORS
> DEBUG: For url ['file', '', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo', 
> '', '', OrderedDict([('downloadfilename', 
> 'fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo')])] 
> comparing ['file', '', '.*', '', '', OrderedDict()] to ['file', '', 
> '/mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/PATH', 
> '', '', OrderedDict()]
> DEBUG: For url 
> file://fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo 
> returning 
> file:///mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/fb/a4/sstate%3Apython3-ifcfg%3Acortexa72-oe-linux%3A0.21%3Ar0%3Acortexa72%3A3%3Afba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo;downloadfilename=fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz.siginfo
> NOTE: Sstate package 
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> does not exist
> ERROR: No suitable staging package found
> DEBUG: Python function do_package_qa_setscene finished
>
> $ ls -lah 
> "/jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz"
> lrwxrwxrwx 1 mjansa mjansa 229 Nov 27 05:41 
> /jenkins/mjansa/build/ros/ros2-rolling-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz 
> -> 
> ../../../../../../../../mnt/mirror-write-ros/jansa/webos/ros-hardknott/sstate-cache/fb/a4/sstate:python3-ifcfg:cortexa72-oe-linux:0.21:r0:cortexa72:3:fba4a512f137dced352b65244a828d6014453f4f3899c45b178a7cbcf8b5d8b2_package_qa.tgz
>
>
>
> On Wed, Nov 18, 2020 at 4:36 PM Hombourger, Cedric 
> <Cedric_Hombourger@mentor.com <mailto:Cedric_Hombourger@mentor.com>> 
> wrote:
>
>     try_mirror_url() creates a symlink named as the original file to make
>     everything look like files specified in SRC_URI were downloaded from
>     their original location. The link is however created as an absolute
>     reference, this makes DL_DIR non-relocatable. This also causes issues
>     with the Isar project since it bind mounts DL_DIR to /downloads to
>     perform some of its build tasks in a chrooted environment (rendering
>     all symbolic links from DL_DIR invalid). Modify ensure_symlink() to
>     take an optional "relative" argument and have that function use
>     os.path.relpath() to produce a relative symlink.
>
>     Signed-off-by: Cedric Hombourger <Cedric_Hombourger@mentor.com
>     <mailto:Cedric_Hombourger@mentor.com>>
>     Cc: Chris Larson <Chris_Larson@mentor.com
>     <mailto:Chris_Larson@mentor.com>>
>     ---
>      lib/bb/fetch2/__init__.py | 7 +++++--
>      1 file changed, 5 insertions(+), 2 deletions(-)
>
>     diff --git a/lib/bb/fetch2/__init__.py b/lib/bb/fetch2/__init__.py
>     index ca81bd52..e090cd86 100644
>     --- a/lib/bb/fetch2/__init__.py
>     +++ b/lib/bb/fetch2/__init__.py
>     @@ -1026,7 +1026,8 @@ def try_mirror_url(fetch, origud, ud, ld,
>     check = False):
>                          origud.method.build_mirror_data(origud, ld)
>                  return origud.localpath
>              # Otherwise the result is a local file:// and we symlink
>     to it
>     -        ensure_symlink(ud.localpath, origud.localpath)
>     +        ensure_symlink(ud.localpath, origud.localpath, relative=True)
>     +
>              update_stamp(origud, ld)
>              return ud.localpath
>
>     @@ -1060,7 +1061,7 @@ def try_mirror_url(fetch, origud, ud, ld,
>     check = False):
>                  bb.utils.unlockfile(lf)
>
>
>     -def ensure_symlink(target, link_name):
>     +def ensure_symlink(target, link_name, relative=False):
>          if not os.path.exists(link_name):
>              if os.path.islink(link_name):
>                  # Broken symbolic link
>     @@ -1071,6 +1072,8 @@ def ensure_symlink(target, link_name):
>              # same time, in which case we do not want the second task to
>              # fail when the link has already been created by the
>     first task.
>              try:
>     +            if relative is True:
>     +                target = os.path.relpath(target,
>     os.path.dirname(link_name))
>                  os.symlink(target, link_name)
>              except FileExistsError:
>                  pass
>     -- 
>     2.25.1
>
>
>
>
>
> 
>

[-- Attachment #2: Type: text/html, Size: 18476 bytes --]

  reply	other threads:[~2020-11-27 14:04 UTC|newest]

Thread overview: 5+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-18 15:36 [PATCH] fetch2: use relative symlinks for anything pulled from PREMIRRORS Hombourger, Cedric
2020-11-27 13:54 ` [bitbake-devel] " Martin Jansa
2020-11-27 14:04   ` Hombourger, Cedric [this message]
2020-12-02 16:12     ` Martin Jansa
2020-12-09 12:59     ` Richard Purdie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=92262695-8578-94cc-939b-fcc49af7b428@mentor.com \
    --to=cedric_hombourger@mentor.com \
    --cc=Chris_Larson@mentor.com \
    --cc=Martin.Jansa@gmail.com \
    --cc=bitbake-devel@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.