* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2021-10-22 13:59 Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 1/2] fetch2/git: Avoid races over mirror tarball creation Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 2/2] fetch2/git: Use os.rename instead of mv Steve Sakoman
0 siblings, 2 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-10-22 13:59 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell/1.46 and have comments back by
end of day Monday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2787
The following changes since commit 35f01508d687eb44c2dc488cd74a9b35124859d9:
hashserv: let asyncio discover the running loop (2021-10-21 05:57:38 -1000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
fetch2/git: Avoid races over mirror tarball creation
fetch2/git: Use os.rename instead of mv
lib/bb/fetch2/git.py | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 1/2] fetch2/git: Avoid races over mirror tarball creation
2021-10-22 13:59 [bitbake][dunfell][1.46][PATCH 0/2] Patch review Steve Sakoman
@ 2021-10-22 13:59 ` Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 2/2] fetch2/git: Use os.rename instead of mv Steve Sakoman
1 sibling, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-10-22 13:59 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
There is a potential race over the mirror tarballs where a partial git repo
could be extracted causing fetcher failures if the tarball is being rewritten
whilst another build accesses it.
Create the mirror tarball atomically to avoid this.
[YOCTO #14441]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 3250bc950c56bd7dd2114df26e5a8e13b04ceac8)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/git.py | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 112b833f..81335c11 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -67,6 +67,7 @@ import subprocess
import tempfile
import bb
import bb.progress
+from contextlib import contextmanager
from bb.fetch2 import FetchMethod
from bb.fetch2 import runfetchcmd
from bb.fetch2 import logger
@@ -408,6 +409,20 @@ class Git(FetchMethod):
bb.utils.remove(tmpdir, recurse=True)
def build_mirror_data(self, ud, d):
+
+ # Create as a temp file and move atomically into position to avoid races
+ @contextmanager
+ def create_atomic(filename, d):
+ fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename))
+ try:
+ yield tfile
+ umask = os.umask(0o666)
+ os.umask(umask)
+ os.chmod(tfile, (0o666 & ~umask))
+ runfetchcmd("mv %s %s" % (tfile, filename), d)
+ finally:
+ os.close(fd)
+
if ud.shallow and ud.write_shallow_tarballs:
if not os.path.exists(ud.fullshallow):
if os.path.islink(ud.fullshallow):
@@ -418,7 +433,8 @@ class Git(FetchMethod):
self.clone_shallow_local(ud, shallowclone, d)
logger.info("Creating tarball of git repository")
- runfetchcmd("tar -czf %s ." % ud.fullshallow, d, workdir=shallowclone)
+ with create_atomic(ud.fullshallow, d) as tfile:
+ runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone)
runfetchcmd("touch %s.done" % ud.fullshallow, d)
finally:
bb.utils.remove(tempdir, recurse=True)
@@ -427,7 +443,8 @@ class Git(FetchMethod):
os.unlink(ud.fullmirror)
logger.info("Creating tarball of git repository")
- runfetchcmd("tar -czf %s ." % ud.fullmirror, d, workdir=ud.clonedir)
+ with create_atomic(ud.fullmirror, d) as tfile:
+ runfetchcmd("tar -czf %s ." % tfile, d, workdir=ud.clonedir)
runfetchcmd("touch %s.done" % ud.fullmirror, d)
def clone_shallow_local(self, ud, dest, d):
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 2/2] fetch2/git: Use os.rename instead of mv
2021-10-22 13:59 [bitbake][dunfell][1.46][PATCH 0/2] Patch review Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 1/2] fetch2/git: Avoid races over mirror tarball creation Steve Sakoman
@ 2021-10-22 13:59 ` Steve Sakoman
1 sibling, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-10-22 13:59 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
os.rename will overwrite the destination file if present so we can use this
instead of the process call overhead.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit b3cccaa6a896c41d8c9be5eebc327f726542d16b)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/git.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 81335c11..000aee19 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -412,14 +412,14 @@ class Git(FetchMethod):
# Create as a temp file and move atomically into position to avoid races
@contextmanager
- def create_atomic(filename, d):
+ def create_atomic(filename):
fd, tfile = tempfile.mkstemp(dir=os.path.dirname(filename))
try:
yield tfile
umask = os.umask(0o666)
os.umask(umask)
os.chmod(tfile, (0o666 & ~umask))
- runfetchcmd("mv %s %s" % (tfile, filename), d)
+ os.rename(tfile, filename)
finally:
os.close(fd)
@@ -433,7 +433,7 @@ class Git(FetchMethod):
self.clone_shallow_local(ud, shallowclone, d)
logger.info("Creating tarball of git repository")
- with create_atomic(ud.fullshallow, d) as tfile:
+ with create_atomic(ud.fullshallow) as tfile:
runfetchcmd("tar -czf %s ." % tfile, d, workdir=shallowclone)
runfetchcmd("touch %s.done" % ud.fullshallow, d)
finally:
@@ -443,7 +443,7 @@ class Git(FetchMethod):
os.unlink(ud.fullmirror)
logger.info("Creating tarball of git repository")
- with create_atomic(ud.fullmirror, d) as tfile:
+ with create_atomic(ud.fullmirror) as tfile:
runfetchcmd("tar -czf %s ." % tfile, d, workdir=ud.clonedir)
runfetchcmd("touch %s.done" % ud.fullmirror, d)
--
2.25.1
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2022-11-19 17:51 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2022-11-19 17:51 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4512
The following changes since commit 8ea8e443005ad92f4ad264d9abd9e90e33fb5c17:
tests/fetch: Allow handling of a file:// url within a submodule (2022-10-27 16:38:59 +0100)
are available in the Git repository at:
https://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
utils: Handle lockfile filenames that are too long for filesystems
utils: Fix lockfile path length issues
lib/bb/utils.py | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2022-03-27 16:43 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2022-03-27 16:43 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell/1.46 and have comments back by end of
day Tuesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3433
The following changes since commit d22cc1e587c650fd5f90cda32f5720f8a3105aac:
tests/fetch: Handle upstream master -> main branch change (2022-02-23 18:11:51 +0000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
server/process: Note when commands complete in logs
tinfoil: Allow run_command not to wait on events
lib/bb/server/process.py | 1 +
lib/bb/tinfoil.py | 4 ++--
2 files changed, 3 insertions(+), 2 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2022-01-13 14:10 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2022-01-13 14:10 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell/1.46 and have comments back by
end of day Monday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/3112
The following changes since commit c5969eedd035648e3258bade386cc67ce3bb0e03:
cooker/command: Add a dummy event for tinfoil testing (2022-01-07 23:43:41 +0000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
utils: Update to use exec_module() instead of load_module()
tests/fetch: Drop gnu urls from wget connectivity test
lib/bb/tests/fetch.py | 3 ---
lib/bb/utils.py | 7 +++++--
2 files changed, 5 insertions(+), 5 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2021-12-03 18:27 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-12-03 18:27 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell/1.46 and have comments back by
end of day Tuesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2988
with the exception of a known autobuilder intermittent issue (on qemuppc test)
which passed on subsequent retest:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/4402
The following changes since commit 7fdd43c5cbde38daa013076de2fdedcf3c3d3107:
process/knotty: Improve early exception handling (2021-11-22 04:15:15 -1000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
cooker: Ensure reparsing is handled correctly
bblayers/action: When adding layers, catch BBHandledException
lib/bb/cooker.py | 2 ++
lib/bblayers/action.py | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2021-11-04 14:11 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-11-04 14:11 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for 1.46/dunfell and have comments back by
end of day Monday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2876
The following changes since commit c222eddcebe892ae209aea7776cfc1147ac1df6e:
fetch/git: Handle github dropping git:// support (2021-11-03 11:30:57 +0000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
tests/fetch2: Fix quoting warning
tests/fetch: Update github urls
lib/bb/tests/fetch.py | 34 +++++++++++++++++-----------------
1 file changed, 17 insertions(+), 17 deletions(-)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2021-03-13 19:04 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2021-03-13 19:04 UTC (permalink / raw)
To: bitbake-devel
Please review this next set of patches for dunfell/1.46 and have comments back
by end of day Tuesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1962
The following changes since commit e0dd20a006ce6fdc656e6be6ac581e93e2ed4a95:
__init__.py: Fix bitbake debug log handling (2021-03-09 00:03:26 +0000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
runqueue: Fix task execution corruption issue
runqueue: Add setscene task overlap sanity check
lib/bb/runqueue.py | 7 +++++++
1 file changed, 7 insertions(+)
--
2.25.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2020-10-12 15:07 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2020-10-12 15:07 UTC (permalink / raw)
To: bitbake-devel
Please review this next set of patches for 1.46 (dunfell) and have comments
back by end of day Wednesday.
Passed a-full on autobuilder:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/1466
The following changes since commit 18e1957337fd9f06bc673d28dd4f8277321d07bc:
tests/fetch: Move away from problematic freedesktop.org urls (2020-09-17 09:13:32 -1000)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Charlie Davies (2):
bitbake: fetch/git: add support for SRC_URI containing spaces in url
bitbake: tests/fetch: add unit tests for SRC_URI with spaces in url
lib/bb/fetch2/git.py | 12 +++++------
lib/bb/tests/fetch.py | 47 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 53 insertions(+), 6 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2020-09-14 14:19 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2020-09-14 14:19 UTC (permalink / raw)
To: bitbake-devel
Please review this set of patches for dunfell and have comments back
by end of day Wednesday.
Tested on dunfell autobuilder builds over the past few weeks.
The following changes since commit 838a89141fd6a75da9f982ad2b035e823b692d2a:
server/process: Account for xmlrpc connections (2020-07-29 00:21:29 +0100)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Jean-Francois Dagenais (1):
siggen: clean_basepath: remove recipe full path when virtual:xyz
present
Richard Purdie (1):
fetch2: Drop cups.org from wget status checks
lib/bb/siggen.py | 2 +-
lib/bb/tests/fetch.py | 4 +---
2 files changed, 2 insertions(+), 4 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 12+ messages in thread
* [bitbake][dunfell][1.46][PATCH 0/2] Patch review
@ 2020-07-28 22:58 Steve Sakoman
0 siblings, 0 replies; 12+ messages in thread
From: Steve Sakoman @ 2020-07-28 22:58 UTC (permalink / raw)
To: bitbake-devel
These patches cherry-picked from master should fix the toaster issue.
The following changes since commit cc11dfa4eb3616547a8a3909f89da0cc4f35dc57:
cooker: Handle multiconfig name mappings correctly (2020-07-23 16:50:51 +0100)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.46-nut
http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.46-nut
Richard Purdie (2):
server/process: Fix UI first connection tracking
server/process: Account for xmlrpc connections
lib/bb/server/process.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--
2.17.1
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2022-11-19 17:51 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-22 13:59 [bitbake][dunfell][1.46][PATCH 0/2] Patch review Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 1/2] fetch2/git: Avoid races over mirror tarball creation Steve Sakoman
2021-10-22 13:59 ` [bitbake][dunfell][1.46][PATCH 2/2] fetch2/git: Use os.rename instead of mv Steve Sakoman
-- strict thread matches above, loose matches on Subject: below --
2022-11-19 17:51 [bitbake][dunfell][1.46][PATCH 0/2] Patch review Steve Sakoman
2022-03-27 16:43 Steve Sakoman
2022-01-13 14:10 Steve Sakoman
2021-12-03 18:27 Steve Sakoman
2021-11-04 14:11 Steve Sakoman
2021-03-13 19:04 Steve Sakoman
2020-10-12 15:07 Steve Sakoman
2020-09-14 14:19 Steve Sakoman
2020-07-28 22:58 Steve Sakoman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).