* [PATCH 0/8] cooker: fixes and parsing performance improve
@ 2018-02-01 15:15 Robert Yang
2018-02-01 15:15 ` [PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty Robert Yang
` (7 more replies)
0 siblings, 8 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
Hi RP,
These patches contain fixes for memory resident bitbake, and
parsing/loading improvement, please see the patches for details.
BTW., do we have a plan to switch to memory resident bitbake by default?
I've tested it a lot with these patches, it works well.
// Robert
The following changes since commit 385944254d0ef88bd6450a221a54cbcb40020b42:
tcmode-default.inc: drop preferred version of gzip-native (2018-01-31 17:01:20 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/inotify
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/inotify
Robert Yang (8):
bitbake: cooker: skip when BBFILE_PATTERN is empty
bitbake: cooker: fix for BBFILE_PATTERN matches bbappend
bitbake: cooker: don't stop file notifier when cooker is shutdown
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cache: improve debug message
bitbake: parse: fixes for resolve_file()
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling
bitbake/lib/bb/cache.py | 14 +++--
bitbake/lib/bb/cooker.py | 79 +++++++++++++++-------------
bitbake/lib/bb/parse/__init__.py | 3 +-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 3 --
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 3 --
bitbake/lib/bb/server/process.py | 2 +
6 files changed, 55 insertions(+), 49 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-01 15:15 ` [PATCH 2/8] bitbake: cooker: fix for BBFILE_PATTERN matches bbappend Robert Yang
` (6 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
There is nothing to do when BBFILE_PATTERN is empty.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index cd365f7..f0dab97 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1170,6 +1170,7 @@ class BBCooker:
elif regex == "":
parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c)
errors = False
+ continue
else:
try:
cre = re.compile(regex)
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 2/8] bitbake: cooker: fix for BBFILE_PATTERN matches bbappend
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
2018-02-01 15:15 ` [PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-01 15:15 ` [PATCH 3/8] bitbake: cooker: don't stop file notifier when cooker is shutdown Robert Yang
` (5 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb
Note there are two layers, parent_layer and sub_layer.
And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"
This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb,
so it warns that no files matched sub_layer, this is the expected behavior, but
it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was
incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it
should warn and let the user fix the problem. Check the bbappend in already
"matched set" before return it as matched by "unmatched set" can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f0dab97..f991c8f 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1808,21 +1808,25 @@ class CookerCollectFiles(object):
realfn, cls, mc = bb.cache.virtualfn2realfn(p)
priorities[p] = self.calc_bbfile_priority(realfn, matched)
- # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
unmatched = set()
for _, _, regex, pri in self.bbfile_config_priorities:
if not regex in matched:
unmatched.add(regex)
- def findmatch(regex):
+ # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
+ def find_bbappend_match(regex):
for b in self.bbappends:
(bbfile, append) = b
if regex.match(append):
+ # If the bbappend is matched by already "matched set", return False
+ for matched_regex in matched:
+ if matched_regex.match(append):
+ return False
return True
return False
for unmatch in unmatched.copy():
- if findmatch(unmatch):
+ if find_bbappend_match(unmatch):
unmatched.remove(unmatch)
for collection, pattern, regex, _ in self.bbfile_config_priorities:
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 3/8] bitbake: cooker: don't stop file notifier when cooker is shutdown
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
2018-02-01 15:15 ` [PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty Robert Yang
2018-02-01 15:15 ` [PATCH 2/8] bitbake: cooker: fix for BBFILE_PATTERN matches bbappend Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-01 15:15 ` [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
` (4 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]
This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 2 --
bitbake/lib/bb/server/process.py | 2 ++
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f991c8f..af482f9 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1604,8 +1604,6 @@ class BBCooker:
if self.parser:
self.parser.shutdown(clean=not force, force=force)
- self.notifier.stop()
- self.confignotifier.stop()
def finishcommand(self):
self.state = state.initial
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 3d31355..828159e 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -223,6 +223,8 @@ class ProcessServer(multiprocessing.Process):
try:
self.cooker.shutdown(True)
+ self.cooker.notifier.stop()
+ self.cooker.confignotifier.stop()
except:
pass
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
` (2 preceding siblings ...)
2018-02-01 15:15 ` [PATCH 3/8] bitbake: cooker: don't stop file notifier when cooker is shutdown Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-05 9:39 ` Richard Purdie
2018-02-01 15:15 ` [PATCH 5/8] bitbake: cache: improve debug message Robert Yang
` (3 subsequent siblings)
7 siblings, 1 reply; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
Fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to-date
one rather than old one.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 8 --------
1 file changed, 8 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af482f9..3f113ae 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -326,11 +326,6 @@ class BBCooker:
self.state = state.initial
self.caches_array = []
- # Need to preserve BB_CONSOLELOG over resets
- consolelog = None
- if hasattr(self, "data"):
- consolelog = self.data.getVar("BB_CONSOLELOG")
-
if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
self.enableDataTracking()
@@ -358,9 +353,6 @@ class BBCooker:
self.data_hash = self.databuilder.data_hash
self.extraconfigdata = {}
- if consolelog:
- self.data.setVar("BB_CONSOLELOG", consolelog)
-
self.data.setVar('BB_CMDLINE', self.ui_cmdline)
#
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 5/8] bitbake: cache: improve debug message
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
` (3 preceding siblings ...)
2018-02-01 15:15 ` [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-01 15:15 ` [PATCH 6/8] bitbake: parse: fixes for resolve_file() Robert Yang
` (2 subsequent siblings)
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
message of "codeparser & file checksum caches", which caused confusion. And
whether the cache file will be used or not is still unknown at that time, so
just print the cache dir.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cache.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 86ce0e7..168a77a 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -395,7 +395,7 @@ class Cache(NoCache):
self.has_cache = True
self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
- logger.debug(1, "Using cache in '%s'", self.cachedir)
+ logger.debug(1, "Cache dir: %s", self.cachedir)
bb.utils.mkdirhier(self.cachedir)
cache_ok = True
@@ -408,6 +408,8 @@ class Cache(NoCache):
self.load_cachefile()
elif os.path.isfile(self.cachefile):
logger.info("Out of date cache found, rebuilding...")
+ else:
+ logger.debug(1, "Cache file %s not found, building..." % self.cachefile)
def load_cachefile(self):
cachesize = 0
@@ -424,6 +426,7 @@ class Cache(NoCache):
for cache_class in self.caches_array:
cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+ logger.debug(1, 'Loading cache file: %s' % cachefile)
with open(cachefile, "rb") as cachefile:
pickled = pickle.Unpickler(cachefile)
# Check cache version information
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 6/8] bitbake: parse: fixes for resolve_file()
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
` (4 preceding siblings ...)
2018-02-01 15:15 ` [PATCH 5/8] bitbake: cache: improve debug message Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-01 15:15 ` [PATCH 7/8] bitbake: cooker: fix event notifications Robert Yang
2018-02-01 15:15 ` [PATCH 8/8] bitbake: cooker: improve inotify handling Robert Yang
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/parse/__init__.py | 3 ++-
bitbake/lib/bb/parse/parse_py/BBHandler.py | 3 ---
bitbake/lib/bb/parse/parse_py/ConfHandler.py | 3 ---
3 files changed, 2 insertions(+), 7 deletions(-)
diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 2fc4002..5397d57 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -134,8 +134,9 @@ def resolve_file(fn, d):
if not newfn:
raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath))
fn = newfn
+ else:
+ mark_dependency(d, fn)
- mark_dependency(d, fn)
if not os.path.isfile(fn):
raise IOError(errno.ENOENT, "file %s not found" % fn)
diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index f89ad24..e5039e3 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -131,9 +131,6 @@ def handle(fn, d, include):
abs_fn = resolve_file(fn, d)
- if include:
- bb.parse.mark_dependency(d, abs_fn)
-
# actual loading
statements = get_statements(fn, abs_fn, base_name)
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index 97aa130..9d3ebe1 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -134,9 +134,6 @@ def handle(fn, data, include):
abs_fn = resolve_file(fn, data)
f = open(abs_fn, 'r')
- if include:
- bb.parse.mark_dependency(data, abs_fn)
-
statements = ast.StatementGroup()
lineno = 0
while True:
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 7/8] bitbake: cooker: fix event notifications
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
` (5 preceding siblings ...)
2018-02-01 15:15 ` [PATCH 6/8] bitbake: parse: fixes for resolve_file() Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
2018-02-02 11:19 ` Richard Purdie
2018-02-01 15:15 ` [PATCH 8/8] bitbake: cooker: improve inotify handling Robert Yang
7 siblings, 1 reply; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
There are two kinds of events:
* directory
We always need handle the event since we don't know what inside the
directory, for example:
$ bitbake --server-only -T -1
$ bitbake -p
$ mkdir ${TOPDIR}/classes
$ cp /path/to/base.bbclass classes
The "${TOPDIR}/classes" is a new directory, the event only reports that the
directory is created, we don't know there is a base.bbclass unless the parser
runs.
* file:
Use bb.parse.supports() to check whether bitbake supports it or not.
And now the bbwatchedfiles is not needed anymore.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
1 file changed, 18 insertions(+), 16 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3f113ae..ffda225 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -177,14 +177,12 @@ class BBCooker:
self.configwatcher = pyinotify.WatchManager()
self.configwatcher.bbseen = []
- self.configwatcher.bbwatchedfiles = []
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
self.watcher = pyinotify.WatchManager()
self.watcher.bbseen = []
- self.watcher.bbwatchedfiles = []
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
# If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
self.baseconfig_valid = False
bb.parse.clear_cache()
return
- if not event.pathname in self.configwatcher.bbwatchedfiles:
- return
- if not event.pathname in self.inotify_modified_files:
- self.inotify_modified_files.append(event.pathname)
+
+ # Check whether it is a supported type when it is a file
+ if not event.dir:
+ if not bb.parse.supports(event.pathname, data):
+ return
+ if not event.pathname in self.inotify_modified_files:
+ self.inotify_modified_files.append(event.pathname)
+
+ bb.debug(1, "Get config inotify event: %s" % event)
self.baseconfig_valid = False
def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
self.parsecache_valid = False
bb.parse.clear_cache()
return
- if event.pathname.endswith("bitbake-cookerdaemon.log") \
- or event.pathname.endswith("bitbake.lock"):
- return
- if not event.pathname in self.inotify_modified_files:
- self.inotify_modified_files.append(event.pathname)
+
+ # Check whether it is a supported type when it is a file
+ if not event.dir:
+ if not bb.parse.supports(event.pathname, data):
+ return
+ if not event.pathname in self.inotify_modified_files:
+ self.inotify_modified_files.append(event.pathname)
+
+ bb.debug(1, "Get inotify event: %s" % event)
self.parsecache_valid = False
def add_filewatch(self, deps, watcher=None, dirs=False):
if not watcher:
watcher = self.watcher
for i in deps:
- watcher.bbwatchedfiles.append(i[0])
if dirs:
f = i[0]
else:
@@ -278,19 +284,15 @@ class BBCooker:
if f in watcher.bbseen:
continue
watcher.bbseen.append(f)
- watchtarget = None
while True:
# We try and add watches for files that don't exist but if they did, would influence
# the parser. The parent directory of these files may not exist, in which case we need
# to watch any parent that does exist for changes.
try:
watcher.add_watch(f, self.watchmask, quiet=False)
- if watchtarget:
- watcher.bbwatchedfiles.append(watchtarget)
break
except pyinotify.WatchManagerError as e:
if 'ENOENT' in str(e):
- watchtarget = f
f = os.path.dirname(f)
if f in watcher.bbseen:
break
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [PATCH 8/8] bitbake: cooker: improve inotify handling
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
` (6 preceding siblings ...)
2018-02-01 15:15 ` [PATCH 7/8] bitbake: cooker: fix event notifications Robert Yang
@ 2018-02-01 15:15 ` Robert Yang
7 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-01 15:15 UTC (permalink / raw)
To: bitbake-devel
Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every
depend file (not only existed files, but also non-exsited/attempted ones),
which caused the parsing and loading very slow. The watchers are global, so we
can only pass new depend files to add_filewatch() to improve the performance.
Here is the testing of 54 layers before and after the patch.
Parsing Loading
Before 35s 33s
Now 11s 9s
24 seconds are saved for parsing, and 24s are saved for loading.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cache.py | 9 ++++++---
bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
2 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 168a77a..7d65939 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -713,12 +713,15 @@ class Cache(NoCache):
def mtime(cachefile):
return bb.parse.cached_mtime_noerror(cachefile)
- def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+ def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
cacheData.add_from_recipeinfo(filename, info_array)
- if watcher:
- watcher(info_array[0].file_depends)
+ if watcher and watch_manager:
+ deps_set = set()
+ for deps in info_array[0].file_depends:
+ deps_set.add(deps[0])
+ watcher(deps_set - watch_manager.bbseen)
if not self.has_cache:
return
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index ffda225..cd98ba0 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -176,13 +176,13 @@ class BBCooker:
self.configuration = configuration
self.configwatcher = pyinotify.WatchManager()
- self.configwatcher.bbseen = []
+ self.configwatcher.bbseen = set()
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
self.watcher = pyinotify.WatchManager()
- self.watcher.bbseen = []
+ self.watcher.bbseen = set()
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
# If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
watcher = self.watcher
for i in deps:
if dirs:
- f = i[0]
+ f = i
else:
- f = os.path.dirname(i[0])
+ f = os.path.dirname(i)
if f in watcher.bbseen:
continue
- watcher.bbseen.append(f)
+ # Add both files and directories to to bbseen
+ watcher.bbseen.add(f)
+ watcher.bbseen.add(i)
while True:
# We try and add watches for files that don't exist but if they did, would influence
# the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
f = os.path.dirname(f)
if f in watcher.bbseen:
break
- watcher.bbseen.append(f)
+ watcher.bbseen.add(f)
continue
if 'ENOSPC' in str(e):
providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
self.disableDataTracking()
self.data.renameVar("__depends", "__base_depends")
- self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+ deps_set = set()
+ for deps in self.data.getVar("__base_depends", False):
+ deps_set.add(deps[0])
+ self.add_filewatch(deps_set, self.configwatcher)
self.baseconfig_valid = True
self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
# Add inotify watches for directories searched for bb/bbappend files
for dirent in searchdirs:
- self.add_filewatch([[dirent]], dirs=True)
+ self.add_filewatch([dirent], dirs=True)
self.parser = CookerParser(self, filelist, masked)
self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
(fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
- parsed=parsed, watcher = self.cooker.add_filewatch)
+ parsed=parsed, watcher=self.cooker.add_filewatch,
+ watch_manager=self.cooker.watcher)
return True
def reparse(self, filename):
--
2.7.4
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH 7/8] bitbake: cooker: fix event notifications
2018-02-01 15:15 ` [PATCH 7/8] bitbake: cooker: fix event notifications Robert Yang
@ 2018-02-02 11:19 ` Richard Purdie
2018-02-06 8:46 ` Robert Yang
0 siblings, 1 reply; 13+ messages in thread
From: Richard Purdie @ 2018-02-02 11:19 UTC (permalink / raw)
To: Robert Yang, bitbake-devel
On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
> There are two kinds of events:
> * directory
> We always need handle the event since we don't know what inside the
> directory, for example:
> $ bitbake --server-only -T -1
> $ bitbake -p
> $ mkdir ${TOPDIR}/classes
> $ cp /path/to/base.bbclass classes
>
> The "${TOPDIR}/classes" is a new directory, the event only reports
> that the
> directory is created, we don't know there is a base.bbclass unless
> the parser
> runs.
>
> * file:
> Use bb.parse.supports() to check whether bitbake supports it or
> not.
>
> And now the bbwatchedfiles is not needed anymore.
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
> bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
> 1 file changed, 18 insertions(+), 16 deletions(-)
I'm not sure you can do this. For example, bitbake itself doesn't
know/care about *.patch files, however if one is referenced in a
SRC_URI and ends up on the watch list, would this change break a patch
file changing causing task checksums to change?
Its looking like we need better tests for some of this code too :/.
Cheers,
Richard
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG
2018-02-01 15:15 ` [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
@ 2018-02-05 9:39 ` Richard Purdie
2018-02-08 4:18 ` Robert Yang
0 siblings, 1 reply; 13+ messages in thread
From: Richard Purdie @ 2018-02-05 9:39 UTC (permalink / raw)
To: Robert Yang, bitbake-devel
On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
> Fixed:
> $ bitbake --server-only -T -1
> Set MACHINE = "qemux86" in conf/local.conf
> $ bitbake quilt
> Set MACHINE = "qemuppc" in conf/local.conf
> $ bitbake quilt
>
> The log still goes into tmp/log/cooker/qemux86 in the second run,
> this is
> incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the
> ui
> initializes it every time when it starts, so let it use the up-to-
> date
> one rather than old one.
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
> bitbake/lib/bb/cooker.py | 8 --------
> 1 file changed, 8 deletions(-)
This effectively reverts:
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de
Sadly I didn't put a bug number in that commit message but I do
remember there being a good reason to do this.
I think the problem is a "UI" can run multiple commands with reset
events within that command stream and it would only have one console
log. I also believe there may have been issues where toaster would have
been unable to find the logs if they kept moving around.
The console log handling probably needs more thought as there are
clearly multiple issues here. We may want to move it out of a directory
which is machine specific?
Cheers,
Richard
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index af482f9..3f113ae 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -326,11 +326,6 @@ class BBCooker:
> self.state = state.initial
> self.caches_array = []
>
> - # Need to preserve BB_CONSOLELOG over resets
> - consolelog = None
> - if hasattr(self, "data"):
> - consolelog = self.data.getVar("BB_CONSOLELOG")
> -
> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
> self.enableDataTracking()
>
> @@ -358,9 +353,6 @@ class BBCooker:
> self.data_hash = self.databuilder.data_hash
> self.extraconfigdata = {}
>
> - if consolelog:
> - self.data.setVar("BB_CONSOLELOG", consolelog)
> -
> self.data.setVar('BB_CMDLINE', self.ui_cmdline)
>
> #
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 7/8] bitbake: cooker: fix event notifications
2018-02-02 11:19 ` Richard Purdie
@ 2018-02-06 8:46 ` Robert Yang
0 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-06 8:46 UTC (permalink / raw)
To: Richard Purdie, bitbake-devel
Hi RP,
On 02/02/2018 07:19 PM, Richard Purdie wrote:
> On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
>> There are two kinds of events:
>> * directory
>> We always need handle the event since we don't know what inside the
>> directory, for example:
>> $ bitbake --server-only -T -1
>> $ bitbake -p
>> $ mkdir ${TOPDIR}/classes
>> $ cp /path/to/base.bbclass classes
>>
>> The "${TOPDIR}/classes" is a new directory, the event only reports
>> that the
>> directory is created, we don't know there is a base.bbclass unless
>> the parser
>> runs.
>>
>> * file:
>> Use bb.parse.supports() to check whether bitbake supports it or
>> not.
>>
>> And now the bbwatchedfiles is not needed anymore.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>> bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
>> 1 file changed, 18 insertions(+), 16 deletions(-)
>
> I'm not sure you can do this. For example, bitbake itself doesn't
> know/care about *.patch files, however if one is referenced in a
> SRC_URI and ends up on the watch list, would this change break a patch
> file changing causing task checksums to change?
I can confirm the the following 2 things:
1) The watcher only watches __depends (and __base_depends) which only contains
files supported by bb.parse.supports(), so use bb.parse.supports() to check
it is OK (or maybe it is the best way).
2) The *.patch is not part of of __depends, but file_checksums, and they are not
watched by watchers. When a patch is updated, it would compare the saved
mtime and current mtime (in lib/bb/checksum.py), and decide whether need
recompute.
So I think that this patch is fine.
The testcase:
$ bitbake --server-only -T -1
$ bitbake quilt-native
# Update quilt-native's patch to test the rebuild:
$ echo "hello world" >>
/path/to/meta/recipes-devtools/quilt/quilt/0001-tests-Allow-different-output-from-mv.patch
# Build quilt-native again
$ bitbake quilt-native
NOTE: Tasks Summary: Attempted 10 tasks of which 0 didn't need to be rerun and
all succeeded.
We can see that it has been rebuilt.
// Robert
>
> Its looking like we need better tests for some of this code too :/.
>
> Cheers,
>
> Richard
>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG
2018-02-05 9:39 ` Richard Purdie
@ 2018-02-08 4:18 ` Robert Yang
0 siblings, 0 replies; 13+ messages in thread
From: Robert Yang @ 2018-02-08 4:18 UTC (permalink / raw)
To: Richard Purdie, bitbake-devel
Hi RP,
On 02/05/2018 05:39 PM, Richard Purdie wrote:
> On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
>> Fixed:
>> $ bitbake --server-only -T -1
>> Set MACHINE = "qemux86" in conf/local.conf
>> $ bitbake quilt
>> Set MACHINE = "qemuppc" in conf/local.conf
>> $ bitbake quilt
>>
>> The log still goes into tmp/log/cooker/qemux86 in the second run,
>> this is
>> incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the
>> ui
>> initializes it every time when it starts, so let it use the up-to-
>> date
>> one rather than old one.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>> bitbake/lib/bb/cooker.py | 8 --------
>> 1 file changed, 8 deletions(-)
>
> This effectively reverts:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de
>
> Sadly I didn't put a bug number in that commit message but I do
> remember there being a good reason to do this.
>
> I think the problem is a "UI" can run multiple commands with reset
> events within that command stream and it would only have one console
> log. I also believe there may have been issues where toaster would have
> been unable to find the logs if they kept moving around.
I've found the bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8411
I think that it has been fixed by:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8373
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a0791c19db7d5b33ce7a4e3614a574ad4a319a84
I've tried the following commands with my patch:
$ . /workspace1/lyang1/poky/bitbake/bin/toaster start noweb
$ MACHINE=qemuppc bitbake quilt-native
The log is tmp/log/cooker/qemuppc/build_20180208_120722.942.log
$ MACHINE=qemux86 bitbake quilt-native
The log is tmp/log/cooker/qemux86/build_20180208_121605.382.log
And the log contents are correct.
So I think that we don't need preserve BB_CONSOLELOG anymore.
// Robert
>
> The console log handling probably needs more thought as there are
> clearly multiple issues here. We may want to move it out of a directory
> which is machine specific?
>
> Cheers,
>
> Richard
>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index af482f9..3f113ae 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -326,11 +326,6 @@ class BBCooker:
>> self.state = state.initial
>> self.caches_array = []
>>
>> - # Need to preserve BB_CONSOLELOG over resets
>> - consolelog = None
>> - if hasattr(self, "data"):
>> - consolelog = self.data.getVar("BB_CONSOLELOG")
>> -
>> if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>> self.enableDataTracking()
>>
>> @@ -358,9 +353,6 @@ class BBCooker:
>> self.data_hash = self.databuilder.data_hash
>> self.extraconfigdata = {}
>>
>> - if consolelog:
>> - self.data.setVar("BB_CONSOLELOG", consolelog)
>> -
>> self.data.setVar('BB_CMDLINE', self.ui_cmdline)
>>
>> #
>> --
>> 2.7.4
>>
>
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2018-02-08 4:17 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-01 15:15 [PATCH 0/8] cooker: fixes and parsing performance improve Robert Yang
2018-02-01 15:15 ` [PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty Robert Yang
2018-02-01 15:15 ` [PATCH 2/8] bitbake: cooker: fix for BBFILE_PATTERN matches bbappend Robert Yang
2018-02-01 15:15 ` [PATCH 3/8] bitbake: cooker: don't stop file notifier when cooker is shutdown Robert Yang
2018-02-01 15:15 ` [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
2018-02-05 9:39 ` Richard Purdie
2018-02-08 4:18 ` Robert Yang
2018-02-01 15:15 ` [PATCH 5/8] bitbake: cache: improve debug message Robert Yang
2018-02-01 15:15 ` [PATCH 6/8] bitbake: parse: fixes for resolve_file() Robert Yang
2018-02-01 15:15 ` [PATCH 7/8] bitbake: cooker: fix event notifications Robert Yang
2018-02-02 11:19 ` Richard Purdie
2018-02-06 8:46 ` Robert Yang
2018-02-01 15:15 ` [PATCH 8/8] bitbake: cooker: improve inotify handling Robert Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.