* [morty][PATCH] fetch2: Do not fail to create symbolic links if they already exist
@ 2017-04-03 12:59 Peter Kjellerstedt
2017-04-03 14:08 ` akuster808
0 siblings, 1 reply; 2+ messages in thread
From: Peter Kjellerstedt @ 2017-04-03 12:59 UTC (permalink / raw)
To: bitbake-devel
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
When the fetcher retrieves file:// URLs, there is no lock file being
used. This means that in case two separate tasks (typically from two
concurrent invocations of bitbake) want to download the same file://
URL at the same time, there is a very small chance that they also end
up wanting to create a symbolic link to the file at the same time.
This would previously lead to one of the tasks failing as the other
task would have created the link.
Signed-off-by: Peter Kjellerstedt <pkj@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
This is a backport of 58a03531c8183b165bb7dcad86d8559c92bc150d on
master.
bitbake/lib/bb/fetch2/__init__.py | 15 +++++++++++++--
1 file changed, 13 insertions(+), 2 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index cd7362c44a..f36f01a707 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -967,7 +967,14 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
open(ud.donestamp, 'w').close()
dest = os.path.join(dldir, os.path.basename(ud.localpath))
if not os.path.exists(dest):
- os.symlink(ud.localpath, dest)
+ # In case this is executing without any file locks held (as is
+ # the case for file:// URLs), two tasks may end up here at the
+ # same time, in which case we do not want the second task to
+ # fail when the link has already been created by the first task.
+ try:
+ os.symlink(ud.localpath, dest)
+ except FileExistsError:
+ pass
if not verify_donestamp(origud, ld) or origud.method.need_update(origud, ld):
origud.method.download(origud, ld)
if hasattr(origud.method,"build_mirror_data"):
@@ -979,7 +986,11 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
# Broken symbolic link
os.unlink(origud.localpath)
- os.symlink(ud.localpath, origud.localpath)
+ # As per above, in case two tasks end up here simultaneously.
+ try:
+ os.symlink(ud.localpath, origud.localpath)
+ except FileExistsError:
+ pass
update_stamp(origud, ld)
return ud.localpath
--
2.12.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [morty][PATCH] fetch2: Do not fail to create symbolic links if they already exist
2017-04-03 12:59 [morty][PATCH] fetch2: Do not fail to create symbolic links if they already exist Peter Kjellerstedt
@ 2017-04-03 14:08 ` akuster808
0 siblings, 0 replies; 2+ messages in thread
From: akuster808 @ 2017-04-03 14:08 UTC (permalink / raw)
To: Peter Kjellerstedt, bitbake-devel
On 04/03/2017 05:59 AM, Peter Kjellerstedt wrote:
> From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
>
> When the fetcher retrieves file:// URLs, there is no lock file being
> used. This means that in case two separate tasks (typically from two
> concurrent invocations of bitbake) want to download the same file://
> URL at the same time, there is a very small chance that they also end
> up wanting to create a symbolic link to the file at the same time.
> This would previously lead to one of the tasks failing as the other
> task would have created the link.
>
> Signed-off-by: Peter Kjellerstedt <pkj@axis.com>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
thanks, I am the process of stagging changes.
-armin
> ---
>
> This is a backport of 58a03531c8183b165bb7dcad86d8559c92bc150d on
> master.
>
> bitbake/lib/bb/fetch2/__init__.py | 15 +++++++++++++--
> 1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
> index cd7362c44a..f36f01a707 100644
> --- a/bitbake/lib/bb/fetch2/__init__.py
> +++ b/bitbake/lib/bb/fetch2/__init__.py
> @@ -967,7 +967,14 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
> open(ud.donestamp, 'w').close()
> dest = os.path.join(dldir, os.path.basename(ud.localpath))
> if not os.path.exists(dest):
> - os.symlink(ud.localpath, dest)
> + # In case this is executing without any file locks held (as is
> + # the case for file:// URLs), two tasks may end up here at the
> + # same time, in which case we do not want the second task to
> + # fail when the link has already been created by the first task.
> + try:
> + os.symlink(ud.localpath, dest)
> + except FileExistsError:
> + pass
> if not verify_donestamp(origud, ld) or origud.method.need_update(origud, ld):
> origud.method.download(origud, ld)
> if hasattr(origud.method,"build_mirror_data"):
> @@ -979,7 +986,11 @@ def try_mirror_url(fetch, origud, ud, ld, check = False):
> # Broken symbolic link
> os.unlink(origud.localpath)
>
> - os.symlink(ud.localpath, origud.localpath)
> + # As per above, in case two tasks end up here simultaneously.
> + try:
> + os.symlink(ud.localpath, origud.localpath)
> + except FileExistsError:
> + pass
> update_stamp(origud, ld)
> return ud.localpath
>
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2017-04-03 14:08 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-04-03 12:59 [morty][PATCH] fetch2: Do not fail to create symbolic links if they already exist Peter Kjellerstedt
2017-04-03 14:08 ` akuster808
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.