* [1.50][PATCH 0/8] Review request
@ 2021-11-22 2:24 Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation Anuj Mittal
` (7 more replies)
0 siblings, 8 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
Please review and merge these changes for 1.50. No issues seen while
testing on autobuilder.
Thanks,
Anuj
The following changes since commit 15dadb69b4c5d29b80770e55d1e9afbe47084aa4:
runqueue: Fix runall option handling (2021-11-16 11:28:01 +0800)
are available in the Git repository at:
git://push.openembedded.org/bitbake-contrib stable/1.50-next
Alexander Kanavin (3):
bitbake: correct the collections vs collections.abc deprecation
bitbake: correct deprecation warning in process.py
bitbake: adjust parser error check for python 3.10 compatibility
Justin Bronder (1):
hashserv: let asyncio discover the running loop
Richard Purdie (4):
cooker: Handle parse threads disappearing to avoid hangs
cooker: Remove debug code, oops :(
cooker: Handle parsing results queue race
cooker: Fix task-depends.dot for multiconfig targets
lib/bb/cache.py | 3 ++-
lib/bb/cooker.py | 18 ++++++++++++++++--
lib/bb/data_smart.py | 4 ++--
lib/bb/persist_data.py | 5 +++--
lib/bb/server/process.py | 2 +-
lib/hashserv/server.py | 4 ++--
6 files changed, 26 insertions(+), 10 deletions(-)
--
2.33.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py Anuj Mittal
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Alexander Kanavin <alex.kanavin@gmail.com>
This becomes a hard error in python 3.10.
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ae219e1f7460077f4492b31ac91cef4cf9b17277)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/cache.py | 3 ++-
lib/bb/data_smart.py | 2 +-
lib/bb/persist_data.py | 5 +++--
3 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 27eb2717..5f9c0a77 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -19,7 +19,8 @@
import os
import logging
import pickle
-from collections import defaultdict, Mapping
+from collections import defaultdict
+from collections.abc import Mapping
import bb.utils
from bb import PrefixLoggerAdapter
import re
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 8291ca65..aa9ac2c8 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -17,7 +17,7 @@ BitBake build tools.
# Based on functions from the base bb module, Copyright 2003 Holger Schurig
import copy, re, sys, traceback
-from collections import MutableMapping
+from collections.abc import MutableMapping
import logging
import hashlib
import bb, bb.codeparser
diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py
index c6a209fb..6f32d81a 100644
--- a/lib/bb/persist_data.py
+++ b/lib/bb/persist_data.py
@@ -12,6 +12,7 @@ currently, providing a key/value store accessed by 'domain'.
#
import collections
+import collections.abc
import contextlib
import functools
import logging
@@ -19,7 +20,7 @@ import os.path
import sqlite3
import sys
import warnings
-from collections import Mapping
+from collections.abc import Mapping
sqlversion = sqlite3.sqlite_version_info
if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
@@ -29,7 +30,7 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
logger = logging.getLogger("BitBake.PersistData")
@functools.total_ordering
-class SQLTable(collections.MutableMapping):
+class SQLTable(collections.abc.MutableMapping):
class _Decorators(object):
@staticmethod
def retry(*, reconnect=True):
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility Anuj Mittal
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Alexander Kanavin <alex.kanavin@gmail.com>
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aff52fe21a0b27f6302555c1e52a864550eb46ce)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/server/process.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 07bb785a..fcdce197 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -659,7 +659,7 @@ class BBUIEventQueue:
self.reader = ConnectionReader(readfd)
self.t = threading.Thread()
- self.t.setDaemon(True)
+ self.t.daemon = True
self.t.run = self.startCallbackHandler
self.t.start()
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 4/8] hashserv: let asyncio discover the running loop Anuj Mittal
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Alexander Kanavin <alex.kanavin@gmail.com>
The change was introduced in
https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75
Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8d3c6cbbe6ee734495713ae3b99c609527842506)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/data_smart.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index aa9ac2c8..65857a9c 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -403,7 +403,7 @@ class DataSmart(MutableMapping):
s = __expand_python_regexp__.sub(varparse.python_sub, s)
except SyntaxError as e:
# Likely unmatched brackets, just don't expand the expression
- if e.msg != "EOL while scanning string literal":
+ if e.msg != "EOL while scanning string literal" and not e.msg.startswith("unterminated string literal"):
raise
if s == olds:
break
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 4/8] hashserv: let asyncio discover the running loop
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
` (2 preceding siblings ...)
2021-11-22 2:24 ` [1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs Anuj Mittal
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Justin Bronder <jsbronder@cold-front.org>
>From 3.10 documentation [1]:
Deprecated since version 3.8, removed in version 3.10: The loop
parameter. This function has been implicitly getting the current
running loop since 3.7
This is fixed in master as a side-effect of
cf9bc0310b0092bf52b61057405aeb51c86ba137 which is more intrusive but
likewise drops the loop parameter.
1. https://docs.python.org/3/library/asyncio-stream.html#asyncio.open_connection
Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 74a1e71b1e677a482fdedc685a71a1798ad63920)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/hashserv/server.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/hashserv/server.py b/lib/hashserv/server.py
index a0dc0c17..df0fa0a0 100644
--- a/lib/hashserv/server.py
+++ b/lib/hashserv/server.py
@@ -521,7 +521,7 @@ class Server(object):
def start_tcp_server(self, host, port):
self.server = self.loop.run_until_complete(
- asyncio.start_server(self.handle_client, host, port, loop=self.loop)
+ asyncio.start_server(self.handle_client, host, port)
)
for s in self.server.sockets:
@@ -546,7 +546,7 @@ class Server(object):
# Work around path length limits in AF_UNIX
os.chdir(os.path.dirname(path))
self.server = self.loop.run_until_complete(
- asyncio.start_unix_server(self.handle_client, os.path.basename(path), loop=self.loop)
+ asyncio.start_unix_server(self.handle_client, os.path.basename(path))
)
finally:
os.chdir(cwd)
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
` (3 preceding siblings ...)
2021-11-22 2:24 ` [1.50][PATCH 4/8] hashserv: let asyncio discover the running loop Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 6/8] cooker: Remove debug code, oops :( Anuj Mittal
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
If one of the parse threads disappears during parsing for some reason, bitbake
currently hangs. Avoid this (and zombie threads hanging around) by joining()
threads which have exited.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dc86a533d951d13643ce446533370da804782afc)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/cooker.py | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index b041d2a0..f12f4caa 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2036,6 +2036,7 @@ class Parser(multiprocessing.Process):
result = pending.pop()
else:
try:
+ time.sleep(0.25)
job = self.jobs.pop()
except IndexError:
self.results.close()
@@ -2214,7 +2215,7 @@ class CookerParser(object):
yield not cached, mc, infos
def parse_generator(self):
- while True:
+ while self.processes:
if self.parsed >= self.toparse:
break
@@ -2228,6 +2229,14 @@ class CookerParser(object):
raise value
else:
yield result
+ for process in self.processes.copy():
+ if not process.is_alive():
+ process.join()
+ self.processes.remove(process)
+
+ if not (self.parsed >= self.toparse):
+ raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None)
+
def parse_next(self):
result = []
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 6/8] cooker: Remove debug code, oops :(
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
` (4 preceding siblings ...)
2021-11-22 2:24 ` [1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 7/8] cooker: Handle parsing results queue race Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets Anuj Mittal
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19291665fa8b6cc331290f2542af3e8e653203f1)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/cooker.py | 1 -
1 file changed, 1 deletion(-)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index f12f4caa..c952c574 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2036,7 +2036,6 @@ class Parser(multiprocessing.Process):
result = pending.pop()
else:
try:
- time.sleep(0.25)
job = self.jobs.pop()
except IndexError:
self.results.close()
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 7/8] cooker: Handle parsing results queue race
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
` (5 preceding siblings ...)
2021-11-22 2:24 ` [1.50][PATCH 6/8] cooker: Remove debug code, oops :( Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets Anuj Mittal
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous fix introduced a race where the queue might not be empty
but all the parser processes have exited. Handle this correctly to avoid
occasional errors.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/cooker.py | 14 +++++++++-----
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index c952c574..e2a5dc43 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2214,24 +2214,28 @@ class CookerParser(object):
yield not cached, mc, infos
def parse_generator(self):
- while self.processes:
+ empty = False
+ while self.processes or not empty:
+ for process in self.processes.copy():
+ if not process.is_alive():
+ process.join()
+ self.processes.remove(process)
+
if self.parsed >= self.toparse:
break
try:
result = self.result_queue.get(timeout=0.25)
except queue.Empty:
+ empty = True
pass
else:
+ empty = False
value = result[1]
if isinstance(value, BaseException):
raise value
else:
yield result
- for process in self.processes.copy():
- if not process.is_alive():
- process.join()
- self.processes.remove(process)
if not (self.parsed >= self.toparse):
raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None)
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
` (6 preceding siblings ...)
2021-11-22 2:24 ` [1.50][PATCH 7/8] cooker: Handle parsing results queue race Anuj Mittal
@ 2021-11-22 2:24 ` Anuj Mittal
7 siblings, 0 replies; 9+ messages in thread
From: Anuj Mittal @ 2021-11-22 2:24 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The right hand side of dependencies in the task dependency file generated
by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix
this.
[YOCTO #14621]
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
lib/bb/cooker.py | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index e2a5dc43..c946800a 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -815,7 +815,9 @@ class BBCooker:
for dep in rq.rqdata.runtaskentries[tid].depends:
(depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep)
deppn = self.recipecaches[depmc].pkg_fn[deptaskfn]
- depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep)))
+ if depmc:
+ depmc = "mc:" + depmc + ":"
+ depend_tree["tdepends"][dotname].append("%s%s.%s" % (depmc, deppn, bb.runqueue.taskname_from_tid(dep)))
if taskfn not in seen_fns:
seen_fns.append(taskfn)
packages = []
--
2.33.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-11-22 2:25 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-22 2:24 [1.50][PATCH 0/8] Review request Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 4/8] hashserv: let asyncio discover the running loop Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 6/8] cooker: Remove debug code, oops :( Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 7/8] cooker: Handle parsing results queue race Anuj Mittal
2021-11-22 2:24 ` [1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets Anuj Mittal
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.