* [resend][PATCH 0/3 V2] cooker: fixes event notifications
@ 2018-02-27 7:28 Robert Yang
2018-02-27 7:28 ` [PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: Robert Yang @ 2018-02-27 7:28 UTC (permalink / raw)
To: bitbake-devel
Hi RP,
I think that I've answered your questions' in V1, do you have any comments on
these patches, please ?
// Robert
The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:
buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib rbt/cooker
http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker
Robert Yang (3):
bitbake: cooker: don't preserve BB_CONSOLELOG
bitbake: cooker: fix event notifications
bitbake: cooker: improve inotify handling
bitbake/lib/bb/cache.py | 9 ++++---
bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
2 files changed, 39 insertions(+), 36 deletions(-)
--
2.7.4
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG
2018-02-27 7:28 [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
@ 2018-02-27 7:28 ` Robert Yang
2018-02-27 7:28 ` [PATCH 2/3] bitbake: cooker: fix event notifications Robert Yang
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Robert Yang @ 2018-02-27 7:28 UTC (permalink / raw)
To: bitbake-devel
Fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to-date
one rather than old one.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 8 --------
1 file changed, 8 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af482f9..3f113ae 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -326,11 +326,6 @@ class BBCooker:
self.state = state.initial
self.caches_array = []
- # Need to preserve BB_CONSOLELOG over resets
- consolelog = None
- if hasattr(self, "data"):
- consolelog = self.data.getVar("BB_CONSOLELOG")
-
if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
self.enableDataTracking()
@@ -358,9 +353,6 @@ class BBCooker:
self.data_hash = self.databuilder.data_hash
self.extraconfigdata = {}
- if consolelog:
- self.data.setVar("BB_CONSOLELOG", consolelog)
-
self.data.setVar('BB_CMDLINE', self.ui_cmdline)
#
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/3] bitbake: cooker: fix event notifications
2018-02-27 7:28 [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
2018-02-27 7:28 ` [PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
@ 2018-02-27 7:28 ` Robert Yang
2018-02-27 7:28 ` [PATCH 3/3] bitbake: cooker: improve inotify handling Robert Yang
2018-03-15 2:00 ` [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
3 siblings, 0 replies; 5+ messages in thread
From: Robert Yang @ 2018-02-27 7:28 UTC (permalink / raw)
To: bitbake-devel
There are two kinds of events:
* directory
We always need handle the event since we don't know what inside the
directory, for example:
$ bitbake --server-only -T -1
$ bitbake -p
$ mkdir ${TOPDIR}/classes
$ cp /path/to/base.bbclass classes
The "${TOPDIR}/classes" is a new directory, the event only reports that the
directory is created, we don't know there is a base.bbclass unless the parser
runs.
* file:
Use bb.parse.supports() to check whether bitbake supports it or not.
And now the bbwatchedfiles is not needed anymore.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
1 file changed, 18 insertions(+), 16 deletions(-)
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3f113ae..ffda225 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -177,14 +177,12 @@ class BBCooker:
self.configwatcher = pyinotify.WatchManager()
self.configwatcher.bbseen = []
- self.configwatcher.bbwatchedfiles = []
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
self.watcher = pyinotify.WatchManager()
self.watcher.bbseen = []
- self.watcher.bbwatchedfiles = []
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
# If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
self.baseconfig_valid = False
bb.parse.clear_cache()
return
- if not event.pathname in self.configwatcher.bbwatchedfiles:
- return
- if not event.pathname in self.inotify_modified_files:
- self.inotify_modified_files.append(event.pathname)
+
+ # Check whether it is a supported type when it is a file
+ if not event.dir:
+ if not bb.parse.supports(event.pathname, data):
+ return
+ if not event.pathname in self.inotify_modified_files:
+ self.inotify_modified_files.append(event.pathname)
+
+ bb.debug(1, "Get config inotify event: %s" % event)
self.baseconfig_valid = False
def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
self.parsecache_valid = False
bb.parse.clear_cache()
return
- if event.pathname.endswith("bitbake-cookerdaemon.log") \
- or event.pathname.endswith("bitbake.lock"):
- return
- if not event.pathname in self.inotify_modified_files:
- self.inotify_modified_files.append(event.pathname)
+
+ # Check whether it is a supported type when it is a file
+ if not event.dir:
+ if not bb.parse.supports(event.pathname, data):
+ return
+ if not event.pathname in self.inotify_modified_files:
+ self.inotify_modified_files.append(event.pathname)
+
+ bb.debug(1, "Get inotify event: %s" % event)
self.parsecache_valid = False
def add_filewatch(self, deps, watcher=None, dirs=False):
if not watcher:
watcher = self.watcher
for i in deps:
- watcher.bbwatchedfiles.append(i[0])
if dirs:
f = i[0]
else:
@@ -278,19 +284,15 @@ class BBCooker:
if f in watcher.bbseen:
continue
watcher.bbseen.append(f)
- watchtarget = None
while True:
# We try and add watches for files that don't exist but if they did, would influence
# the parser. The parent directory of these files may not exist, in which case we need
# to watch any parent that does exist for changes.
try:
watcher.add_watch(f, self.watchmask, quiet=False)
- if watchtarget:
- watcher.bbwatchedfiles.append(watchtarget)
break
except pyinotify.WatchManagerError as e:
if 'ENOENT' in str(e):
- watchtarget = f
f = os.path.dirname(f)
if f in watcher.bbseen:
break
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 3/3] bitbake: cooker: improve inotify handling
2018-02-27 7:28 [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
2018-02-27 7:28 ` [PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
2018-02-27 7:28 ` [PATCH 2/3] bitbake: cooker: fix event notifications Robert Yang
@ 2018-02-27 7:28 ` Robert Yang
2018-03-15 2:00 ` [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
3 siblings, 0 replies; 5+ messages in thread
From: Robert Yang @ 2018-02-27 7:28 UTC (permalink / raw)
To: bitbake-devel
Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every
depend file (not only existed files, but also non-exsited/attempted ones),
which caused the parsing and loading very slow. The watchers are global, so we
can only pass new depend files to add_filewatch() to improve the performance.
Here is the testing of 54 layers before and after the patch.
Parsing Loading
Before 35s 33s
Now 11s 9s
24 seconds are saved for parsing, and 24s are saved for loading.
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
bitbake/lib/bb/cache.py | 9 ++++++---
bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
2 files changed, 21 insertions(+), 12 deletions(-)
diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 168a77a..7d65939 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -713,12 +713,15 @@ class Cache(NoCache):
def mtime(cachefile):
return bb.parse.cached_mtime_noerror(cachefile)
- def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+ def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
cacheData.add_from_recipeinfo(filename, info_array)
- if watcher:
- watcher(info_array[0].file_depends)
+ if watcher and watch_manager:
+ deps_set = set()
+ for deps in info_array[0].file_depends:
+ deps_set.add(deps[0])
+ watcher(deps_set - watch_manager.bbseen)
if not self.has_cache:
return
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index ffda225..cd98ba0 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -176,13 +176,13 @@ class BBCooker:
self.configuration = configuration
self.configwatcher = pyinotify.WatchManager()
- self.configwatcher.bbseen = []
+ self.configwatcher.bbseen = set()
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
self.watcher = pyinotify.WatchManager()
- self.watcher.bbseen = []
+ self.watcher.bbseen = set()
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
# If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
watcher = self.watcher
for i in deps:
if dirs:
- f = i[0]
+ f = i
else:
- f = os.path.dirname(i[0])
+ f = os.path.dirname(i)
if f in watcher.bbseen:
continue
- watcher.bbseen.append(f)
+ # Add both files and directories to to bbseen
+ watcher.bbseen.add(f)
+ watcher.bbseen.add(i)
while True:
# We try and add watches for files that don't exist but if they did, would influence
# the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
f = os.path.dirname(f)
if f in watcher.bbseen:
break
- watcher.bbseen.append(f)
+ watcher.bbseen.add(f)
continue
if 'ENOSPC' in str(e):
providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
self.disableDataTracking()
self.data.renameVar("__depends", "__base_depends")
- self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+ deps_set = set()
+ for deps in self.data.getVar("__base_depends", False):
+ deps_set.add(deps[0])
+ self.add_filewatch(deps_set, self.configwatcher)
self.baseconfig_valid = True
self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
# Add inotify watches for directories searched for bb/bbappend files
for dirent in searchdirs:
- self.add_filewatch([[dirent]], dirs=True)
+ self.add_filewatch([dirent], dirs=True)
self.parser = CookerParser(self, filelist, masked)
self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
(fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
- parsed=parsed, watcher = self.cooker.add_filewatch)
+ parsed=parsed, watcher=self.cooker.add_filewatch,
+ watch_manager=self.cooker.watcher)
return True
def reparse(self, filename):
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [resend][PATCH 0/3 V2] cooker: fixes event notifications
2018-02-27 7:28 [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
` (2 preceding siblings ...)
2018-02-27 7:28 ` [PATCH 3/3] bitbake: cooker: improve inotify handling Robert Yang
@ 2018-03-15 2:00 ` Robert Yang
3 siblings, 0 replies; 5+ messages in thread
From: Robert Yang @ 2018-03-15 2:00 UTC (permalink / raw)
To: bitbake-devel
Hi RP,
Do you have any comments on this, please ?
// Robert
On 02/27/2018 03:28 PM, Robert Yang wrote:
> Hi RP,
>
> I think that I've answered your questions' in V1, do you have any comments on
> these patches, please ?
>
> // Robert
>
> The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:
>
> buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)
>
> are available in the git repository at:
>
> git://git.pokylinux.org/poky-contrib rbt/cooker
> http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker
>
> Robert Yang (3):
> bitbake: cooker: don't preserve BB_CONSOLELOG
> bitbake: cooker: fix event notifications
> bitbake: cooker: improve inotify handling
>
> bitbake/lib/bb/cache.py | 9 ++++---
> bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
> 2 files changed, 39 insertions(+), 36 deletions(-)
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-03-15 2:00 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-27 7:28 [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
2018-02-27 7:28 ` [PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG Robert Yang
2018-02-27 7:28 ` [PATCH 2/3] bitbake: cooker: fix event notifications Robert Yang
2018-02-27 7:28 ` [PATCH 3/3] bitbake: cooker: improve inotify handling Robert Yang
2018-03-15 2:00 ` [resend][PATCH 0/3 V2] cooker: fixes event notifications Robert Yang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.