[resend][PATCH 0/3 V2] cooker: fixes event notifications

[resend][PATCH 0/3 V2] cooker: fixes event notifications

[Robert Yang]

Hi RP,

I think that I've answered your questions' in V1, do you have any comments on
these patches, please ?

// Robert

The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:

  buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/cooker
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker

Robert Yang (3):
  bitbake: cooker: don't preserve BB_CONSOLELOG
  bitbake: cooker: fix event notifications
  bitbake: cooker: improve inotify handling

 bitbake/lib/bb/cache.py  |  9 ++++---
 bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
 2 files changed, 39 insertions(+), 36 deletions(-)

-- 
2.7.4

[PATCH 1/3] bitbake: cooker: don't preserve BB_CONSOLELOG

[Robert Yang]

Fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to-date
one rather than old one.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af482f9..3f113ae 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -326,11 +326,6 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG")
-
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()
 
@@ -358,9 +353,6 @@ class BBCooker:
         self.data_hash = self.databuilder.data_hash
         self.extraconfigdata = {}
 
-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog)
-
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
         #
-- 
2.7.4

[PATCH 2/3] bitbake: cooker: fix event notifications

[Robert Yang]

There are two kinds of events:
* directory
  We always need handle the event since we don't know what inside the
  directory, for example:
  $ bitbake --server-only -T -1
  $ bitbake -p
  $ mkdir ${TOPDIR}/classes
  $ cp /path/to/base.bbclass classes

  The "${TOPDIR}/classes" is a new directory, the event only reports that the
  directory is created, we don't know there is a base.bbclass unless the parser
  runs.

* file:
  Use bb.parse.supports() to check whether bitbake supports it or not.

And now the bbwatchedfiles is not needed anymore.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3f113ae..ffda225 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -177,14 +177,12 @@ class BBCooker:
 
         self.configwatcher = pyinotify.WatchManager()
         self.configwatcher.bbseen = []
-        self.configwatcher.bbwatchedfiles = []
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
         self.watcher.bbseen = []
-        self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
             self.baseconfig_valid = False
             bb.parse.clear_cache()
             return
-        if not event.pathname in self.configwatcher.bbwatchedfiles:
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get config inotify event: %s" % event)
         self.baseconfig_valid = False
 
     def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
             self.parsecache_valid = False
             bb.parse.clear_cache()
             return
-        if event.pathname.endswith("bitbake-cookerdaemon.log") \
-                or event.pathname.endswith("bitbake.lock"):
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get inotify event: %s" % event)
         self.parsecache_valid = False
 
     def add_filewatch(self, deps, watcher=None, dirs=False):
         if not watcher:
             watcher = self.watcher
         for i in deps:
-            watcher.bbwatchedfiles.append(i[0])
             if dirs:
                 f = i[0]
             else:
@@ -278,19 +284,15 @@ class BBCooker:
             if f in watcher.bbseen:
                 continue
             watcher.bbseen.append(f)
-            watchtarget = None
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
                 # to watch any parent that does exist for changes.
                 try:
                     watcher.add_watch(f, self.watchmask, quiet=False)
-                    if watchtarget:
-                        watcher.bbwatchedfiles.append(watchtarget)
                     break
                 except pyinotify.WatchManagerError as e:
                     if 'ENOENT' in str(e):
-                        watchtarget = f
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-- 
2.7.4

[PATCH 3/3] bitbake: cooker: improve inotify handling

[Robert Yang]

Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every
depend file (not only existed files, but also non-exsited/attempted ones),
which caused the parsing and loading very slow. The watchers are global, so we
can only pass new depend files to add_filewatch() to improve the performance.

Here is the testing of 54 layers before and after the patch.

              Parsing       Loading
Before        35s           33s
Now           11s           9s

24 seconds are saved for parsing, and 24s are saved for loading.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cache.py  |  9 ++++++---
 bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
 2 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 168a77a..7d65939 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -713,12 +713,15 @@ class Cache(NoCache):
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
         if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
             cacheData.add_from_recipeinfo(filename, info_array)
 
-            if watcher:
-                watcher(info_array[0].file_depends)
+            if watcher and watch_manager:
+                deps_set = set()
+                for deps in info_array[0].file_depends:
+                    deps_set.add(deps[0])
+                watcher(deps_set - watch_manager.bbseen)
 
         if not self.has_cache:
             return
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index ffda225..cd98ba0 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -176,13 +176,13 @@ class BBCooker:
         self.configuration = configuration
 
         self.configwatcher = pyinotify.WatchManager()
-        self.configwatcher.bbseen = []
+        self.configwatcher.bbseen = set()
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
-        self.watcher.bbseen = []
+        self.watcher.bbseen = set()
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
             watcher = self.watcher
         for i in deps:
             if dirs:
-                f = i[0]
+                f = i
             else:
-                f = os.path.dirname(i[0])
+                f = os.path.dirname(i)
             if f in watcher.bbseen:
                 continue
-            watcher.bbseen.append(f)
+            # Add both files and directories to to bbseen
+            watcher.bbseen.add(f)
+            watcher.bbseen.add(i)
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-                        watcher.bbseen.append(f)
+                        watcher.bbseen.add(f)
                         continue
                     if 'ENOSPC' in str(e):
                         providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
             self.disableDataTracking()
 
         self.data.renameVar("__depends", "__base_depends")
-        self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+        deps_set = set()
+        for deps in self.data.getVar("__base_depends", False):
+            deps_set.add(deps[0])
+        self.add_filewatch(deps_set, self.configwatcher)
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
 
             # Add inotify watches for directories searched for bb/bbappend files
             for dirent in searchdirs:
-                self.add_filewatch([[dirent]], dirs=True)
+                self.add_filewatch([dirent], dirs=True)
 
             self.parser = CookerParser(self, filelist, masked)
             self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
                 self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
             (fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
             self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
-                                        parsed=parsed, watcher = self.cooker.add_filewatch)
+                                        parsed=parsed, watcher=self.cooker.add_filewatch,
+                                        watch_manager=self.cooker.watcher)
         return True
 
     def reparse(self, filename):
-- 
2.7.4

Re: [resend][PATCH 0/3 V2] cooker: fixes event notifications

[Robert Yang]

Hi RP,

Do you have any comments on this, please ?

// Robert

On 02/27/2018 03:28 PM, Robert Yang wrote:
> Hi RP,
> 
> I think that I've answered your questions' in V1, do you have any comments on
> these patches, please ?
> 
> // Robert
> 
> The following changes since commit 1b7a9d4f63d07d61d53daac12da275e8ef2feb24:
> 
>    buildhistory-diff: honour report_all flag (2018-02-24 10:35:59 +0000)
> 
> are available in the git repository at:
> 
>    git://git.pokylinux.org/poky-contrib rbt/cooker
>    http://git.pokylinux.org/cgit.cgi//log/?h=rbt/cooker
> 
> Robert Yang (3):
>    bitbake: cooker: don't preserve BB_CONSOLELOG
>    bitbake: cooker: fix event notifications
>    bitbake: cooker: improve inotify handling
> 
>   bitbake/lib/bb/cache.py  |  9 ++++---
>   bitbake/lib/bb/cooker.py | 66 ++++++++++++++++++++++++------------------------
>   2 files changed, 39 insertions(+), 36 deletions(-)
>