[PATCH 0/8] cooker: fixes and parsing performance improve

[PATCH 0/8] cooker: fixes and parsing performance improve

[Robert Yang]

Hi RP,

These patches contain fixes for memory resident bitbake, and
parsing/loading improvement, please see the patches for details.

BTW., do we have a plan to switch to memory resident bitbake by default?
I've tested it a lot with these patches, it works well.

// Robert

The following changes since commit 385944254d0ef88bd6450a221a54cbcb40020b42:

  tcmode-default.inc: drop preferred version of gzip-native (2018-01-31 17:01:20 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib rbt/inotify
  http://git.pokylinux.org/cgit.cgi//log/?h=rbt/inotify

Robert Yang (8):
  bitbake: cooker: skip when BBFILE_PATTERN is empty
  bitbake: cooker: fix for BBFILE_PATTERN matches bbappend
  bitbake: cooker: don't stop file notifier when cooker is shutdown
  bitbake: cooker: don't preserve BB_CONSOLELOG
  bitbake: cache: improve debug message
  bitbake: parse: fixes for resolve_file()
  bitbake: cooker: fix event notifications
  bitbake: cooker: improve inotify handling

 bitbake/lib/bb/cache.py                      | 14 +++--
 bitbake/lib/bb/cooker.py                     | 79 +++++++++++++++-------------
 bitbake/lib/bb/parse/__init__.py             |  3 +-
 bitbake/lib/bb/parse/parse_py/BBHandler.py   |  3 --
 bitbake/lib/bb/parse/parse_py/ConfHandler.py |  3 --
 bitbake/lib/bb/server/process.py             |  2 +
 6 files changed, 55 insertions(+), 49 deletions(-)

-- 
2.7.4

[PATCH 1/8] bitbake: cooker: skip when BBFILE_PATTERN is empty

[Robert Yang]

There is nothing to do when BBFILE_PATTERN is empty.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index cd365f7..f0dab97 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1170,6 +1170,7 @@ class BBCooker:
                 elif regex == "":
                     parselog.debug(1, "BBFILE_PATTERN_%s is empty" % c)
                     errors = False
+                    continue
                 else:
                     try:
                         cre = re.compile(regex)
-- 
2.7.4

[PATCH 2/8] bitbake: cooker: fix for BBFILE_PATTERN matches bbappend

[Robert Yang]

The old code couldn't handle nestled layers correctly, e.g.:
parent_layer/sub_layer/foo.bb

Note there are two layers, parent_layer and sub_layer.
And in parent_layer/conf/layer.conf:
BBFILE_PATTERN_parent_layer = ""^${LAYERDIR}/"

This setting is incorrect since it also matches parent_layer/sub_layer/foo.bb,
so it warns that no files matched sub_layer, this is the expected behavior, but
it doesn't warn when there is a parent_layer/sub_layer/bar.bbappend, this was
incorrect since the bbappend is also matched by BBFILE_PATTERN_parent_layer, it
should warn and let the user fix the problem. Check the bbappend in already
"matched set" before return it as matched by "unmatched set" can fix the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f0dab97..f991c8f 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1808,21 +1808,25 @@ class CookerCollectFiles(object):
             realfn, cls, mc = bb.cache.virtualfn2realfn(p)
             priorities[p] = self.calc_bbfile_priority(realfn, matched)
 
-        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
         unmatched = set()
         for _, _, regex, pri in self.bbfile_config_priorities:
             if not regex in matched:
                 unmatched.add(regex)
 
-        def findmatch(regex):
+        # Don't show the warning if the BBFILE_PATTERN did match .bbappend files
+        def find_bbappend_match(regex):
             for b in self.bbappends:
                 (bbfile, append) = b
                 if regex.match(append):
+                    # If the bbappend is matched by already "matched set", return False
+                    for matched_regex in matched:
+                        if matched_regex.match(append):
+                            return False
                     return True
             return False
 
         for unmatch in unmatched.copy():
-            if findmatch(unmatch):
+            if find_bbappend_match(unmatch):
                 unmatched.remove(unmatch)
 
         for collection, pattern, regex, _ in self.bbfile_config_priorities:
-- 
2.7.4

[PATCH 3/8] bitbake: cooker: don't stop file notifier when cooker is shutdown

[Robert Yang]

It should be live/exited with server rather than cooker, fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt
[snip]
ERROR: When reparsing /workspace1/lyang1/poky/meta/recipes-connectivity/openssl/openssl_1.0.2m.bb.do_package, the basehash value changed from c216f7f4fdd3cf4a0b10b975a636426c to d5a8e9431ab261381752d7a64c7b2fa9. The metadata is not deterministic and this needs to be fixed.
[snip]

This is because the server doesn't know local.conf is changed since the
notifiers are stopped, so it doesn't reparse, and then we would get the errors,
let the notifiers live/exited with server can fix the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py         | 2 --
 bitbake/lib/bb/server/process.py | 2 ++
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f991c8f..af482f9 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1604,8 +1604,6 @@ class BBCooker:
 
         if self.parser:
             self.parser.shutdown(clean=not force, force=force)
-        self.notifier.stop()
-        self.confignotifier.stop()
 
     def finishcommand(self):
         self.state = state.initial
diff --git a/bitbake/lib/bb/server/process.py b/bitbake/lib/bb/server/process.py
index 3d31355..828159e 100644
--- a/bitbake/lib/bb/server/process.py
+++ b/bitbake/lib/bb/server/process.py
@@ -223,6 +223,8 @@ class ProcessServer(multiprocessing.Process):
 
         try: 
             self.cooker.shutdown(True)
+            self.cooker.notifier.stop()
+            self.cooker.confignotifier.stop()
         except:
             pass
 
-- 
2.7.4

[PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG

[Robert Yang]

Fixed:
$ bitbake --server-only -T -1
Set MACHINE = "qemux86" in conf/local.conf
$ bitbake quilt
Set MACHINE = "qemuppc" in conf/local.conf
$ bitbake quilt

The log still goes into tmp/log/cooker/qemux86 in the second run, this is
incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the ui
initializes it every time when it starts, so let it use the up-to-date
one rather than old one.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index af482f9..3f113ae 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -326,11 +326,6 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
-        # Need to preserve BB_CONSOLELOG over resets
-        consolelog = None
-        if hasattr(self, "data"):
-            consolelog = self.data.getVar("BB_CONSOLELOG")
-
         if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
             self.enableDataTracking()
 
@@ -358,9 +353,6 @@ class BBCooker:
         self.data_hash = self.databuilder.data_hash
         self.extraconfigdata = {}
 
-        if consolelog:
-            self.data.setVar("BB_CONSOLELOG", consolelog)
-
         self.data.setVar('BB_CMDLINE', self.ui_cmdline)
 
         #
-- 
2.7.4

Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG

[Richard Purdie]

On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
> Fixed:
> $ bitbake --server-only -T -1
> Set MACHINE = "qemux86" in conf/local.conf
> $ bitbake quilt
> Set MACHINE = "qemuppc" in conf/local.conf
> $ bitbake quilt
> 
> The log still goes into tmp/log/cooker/qemux86 in the second run,
> this is
> incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the
> ui
> initializes it every time when it starts, so let it use the up-to-
> date
> one rather than old one.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/cooker.py | 8 --------
>  1 file changed, 8 deletions(-)

This effectively reverts:

http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de

Sadly I didn't put a bug number in that commit message but I do
remember there being a good reason to do this.

I think the problem is a "UI" can run multiple commands with reset
events within that command stream and it would only have one console
log. I also believe there may have been issues where toaster would have
been unable to find the logs if they kept moving around.

The console log handling probably needs more thought as there are
clearly multiple issues here. We may want to move it out of a directory
which is machine specific?

Cheers,

Richard

> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index af482f9..3f113ae 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -326,11 +326,6 @@ class BBCooker:
>          self.state = state.initial
>          self.caches_array = []
>  
> -        # Need to preserve BB_CONSOLELOG over resets
> -        consolelog = None
> -        if hasattr(self, "data"):
> -            consolelog = self.data.getVar("BB_CONSOLELOG")
> -
>          if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>              self.enableDataTracking()
>  
> @@ -358,9 +353,6 @@ class BBCooker:
>          self.data_hash = self.databuilder.data_hash
>          self.extraconfigdata = {}
>  
> -        if consolelog:
> -            self.data.setVar("BB_CONSOLELOG", consolelog)
> -
>          self.data.setVar('BB_CMDLINE', self.ui_cmdline)
>  
>          #
> -- 
> 2.7.4
> 

Re: [PATCH 4/8] bitbake: cooker: don't preserve BB_CONSOLELOG

[Robert Yang]

Hi RP,

On 02/05/2018 05:39 PM, Richard Purdie wrote:
> On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
>> Fixed:
>> $ bitbake --server-only -T -1
>> Set MACHINE = "qemux86" in conf/local.conf
>> $ bitbake quilt
>> Set MACHINE = "qemuppc" in conf/local.conf
>> $ bitbake quilt
>>
>> The log still goes into tmp/log/cooker/qemux86 in the second run,
>> this is
>> incorrect (should be tmp/log/cooker/qemuppc). I checked the code, the
>> ui
>> initializes it every time when it starts, so let it use the up-to-
>> date
>> one rather than old one.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/cooker.py | 8 --------
>>   1 file changed, 8 deletions(-)
> 
> This effectively reverts:
> 
> http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/cooker.py?id=3ebf7617d6c869f798807792918e1030b3ab66de
> 
> Sadly I didn't put a bug number in that commit message but I do
> remember there being a good reason to do this.
> 
> I think the problem is a "UI" can run multiple commands with reset
> events within that command stream and it would only have one console
> log. I also believe there may have been issues where toaster would have
> been unable to find the logs if they kept moving around.

I've found the bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8411

I think that it has been fixed by:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=8373

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=a0791c19db7d5b33ce7a4e3614a574ad4a319a84

I've tried the following commands with my patch:
$ . /workspace1/lyang1/poky/bitbake/bin/toaster start noweb
$ MACHINE=qemuppc bitbake quilt-native

The log is tmp/log/cooker/qemuppc/build_20180208_120722.942.log

$ MACHINE=qemux86 bitbake quilt-native

The log is tmp/log/cooker/qemux86/build_20180208_121605.382.log

And the log contents are correct.

So I think that we don't need preserve BB_CONSOLELOG anymore.

// Robert

> 
> The console log handling probably needs more thought as there are
> clearly multiple issues here. We may want to move it out of a directory
> which is machine specific?
> 
> Cheers,
> 
> Richard
> 
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index af482f9..3f113ae 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -326,11 +326,6 @@ class BBCooker:
>>           self.state = state.initial
>>           self.caches_array = []
>>   
>> -        # Need to preserve BB_CONSOLELOG over resets
>> -        consolelog = None
>> -        if hasattr(self, "data"):
>> -            consolelog = self.data.getVar("BB_CONSOLELOG")
>> -
>>           if CookerFeatures.BASEDATASTORE_TRACKING in self.featureset:
>>               self.enableDataTracking()
>>   
>> @@ -358,9 +353,6 @@ class BBCooker:
>>           self.data_hash = self.databuilder.data_hash
>>           self.extraconfigdata = {}
>>   
>> -        if consolelog:
>> -            self.data.setVar("BB_CONSOLELOG", consolelog)
>> -
>>           self.data.setVar('BB_CMDLINE', self.ui_cmdline)
>>   
>>           #
>> -- 
>> 2.7.4
>>
> 

[PATCH 5/8] bitbake: cache: improve debug message

[Robert Yang]

* Print message when cachefile is found/not can help debug.
* Update "Using cache in" to "Cache dir:" since it was the same as the debug
  message of "codeparser & file checksum caches", which caused confusion. And
  whether the cache file will be used or not is still unknown at that time, so
  just print the cache dir.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cache.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 86ce0e7..168a77a 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -395,7 +395,7 @@ class Cache(NoCache):
         self.has_cache = True
         self.cachefile = getCacheFile(self.cachedir, "bb_cache.dat", self.data_hash)
 
-        logger.debug(1, "Using cache in '%s'", self.cachedir)
+        logger.debug(1, "Cache dir: %s", self.cachedir)
         bb.utils.mkdirhier(self.cachedir)
 
         cache_ok = True
@@ -408,6 +408,8 @@ class Cache(NoCache):
             self.load_cachefile()
         elif os.path.isfile(self.cachefile):
             logger.info("Out of date cache found, rebuilding...")
+        else:
+            logger.debug(1, "Cache file %s not found, building..." % self.cachefile)
 
     def load_cachefile(self):
         cachesize = 0
@@ -424,6 +426,7 @@ class Cache(NoCache):
 
         for cache_class in self.caches_array:
             cachefile = getCacheFile(self.cachedir, cache_class.cachefile, self.data_hash)
+            logger.debug(1, 'Loading cache file: %s' % cachefile)
             with open(cachefile, "rb") as cachefile:
                 pickled = pickle.Unpickler(cachefile)
                 # Check cache version information
-- 
2.7.4

[PATCH 6/8] bitbake: parse: fixes for resolve_file()

[Robert Yang]

The resolve_file() calls mark_dependency(), so the one which calls
resolve_file() doesn't need call mark_dependency() again.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/parse/__init__.py             | 3 ++-
 bitbake/lib/bb/parse/parse_py/BBHandler.py   | 3 ---
 bitbake/lib/bb/parse/parse_py/ConfHandler.py | 3 ---
 3 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/bitbake/lib/bb/parse/__init__.py b/bitbake/lib/bb/parse/__init__.py
index 2fc4002..5397d57 100644
--- a/bitbake/lib/bb/parse/__init__.py
+++ b/bitbake/lib/bb/parse/__init__.py
@@ -134,8 +134,9 @@ def resolve_file(fn, d):
         if not newfn:
             raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath))
         fn = newfn
+    else:
+        mark_dependency(d, fn)
 
-    mark_dependency(d, fn)
     if not os.path.isfile(fn):
         raise IOError(errno.ENOENT, "file %s not found" % fn)
 
diff --git a/bitbake/lib/bb/parse/parse_py/BBHandler.py b/bitbake/lib/bb/parse/parse_py/BBHandler.py
index f89ad24..e5039e3 100644
--- a/bitbake/lib/bb/parse/parse_py/BBHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/BBHandler.py
@@ -131,9 +131,6 @@ def handle(fn, d, include):
 
     abs_fn = resolve_file(fn, d)
 
-    if include:
-        bb.parse.mark_dependency(d, abs_fn)
-
     # actual loading
     statements = get_statements(fn, abs_fn, base_name)
 
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index 97aa130..9d3ebe1 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -134,9 +134,6 @@ def handle(fn, data, include):
     abs_fn = resolve_file(fn, data)
     f = open(abs_fn, 'r')
 
-    if include:
-        bb.parse.mark_dependency(data, abs_fn)
-
     statements = ast.StatementGroup()
     lineno = 0
     while True:
-- 
2.7.4

[PATCH 7/8] bitbake: cooker: fix event notifications

[Robert Yang]

There are two kinds of events:
* directory
  We always need handle the event since we don't know what inside the
  directory, for example:
  $ bitbake --server-only -T -1
  $ bitbake -p
  $ mkdir ${TOPDIR}/classes
  $ cp /path/to/base.bbclass classes

  The "${TOPDIR}/classes" is a new directory, the event only reports that the
  directory is created, we don't know there is a base.bbclass unless the parser
  runs.

* file:
  Use bb.parse.supports() to check whether bitbake supports it or not.

And now the bbwatchedfiles is not needed anymore.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
 1 file changed, 18 insertions(+), 16 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 3f113ae..ffda225 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -177,14 +177,12 @@ class BBCooker:
 
         self.configwatcher = pyinotify.WatchManager()
         self.configwatcher.bbseen = []
-        self.configwatcher.bbwatchedfiles = []
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
         self.watcher.bbseen = []
-        self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -247,10 +245,15 @@ class BBCooker:
             self.baseconfig_valid = False
             bb.parse.clear_cache()
             return
-        if not event.pathname in self.configwatcher.bbwatchedfiles:
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get config inotify event: %s" % event)
         self.baseconfig_valid = False
 
     def notifications(self, event):
@@ -259,18 +262,21 @@ class BBCooker:
             self.parsecache_valid = False
             bb.parse.clear_cache()
             return
-        if event.pathname.endswith("bitbake-cookerdaemon.log") \
-                or event.pathname.endswith("bitbake.lock"):
-            return
-        if not event.pathname in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.pathname)
+
+        # Check whether it is a supported type when it is a file
+        if not event.dir:
+            if not bb.parse.supports(event.pathname, data):
+                return
+            if not event.pathname in self.inotify_modified_files:
+                self.inotify_modified_files.append(event.pathname)
+
+        bb.debug(1, "Get inotify event: %s" % event)
         self.parsecache_valid = False
 
     def add_filewatch(self, deps, watcher=None, dirs=False):
         if not watcher:
             watcher = self.watcher
         for i in deps:
-            watcher.bbwatchedfiles.append(i[0])
             if dirs:
                 f = i[0]
             else:
@@ -278,19 +284,15 @@ class BBCooker:
             if f in watcher.bbseen:
                 continue
             watcher.bbseen.append(f)
-            watchtarget = None
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
                 # to watch any parent that does exist for changes.
                 try:
                     watcher.add_watch(f, self.watchmask, quiet=False)
-                    if watchtarget:
-                        watcher.bbwatchedfiles.append(watchtarget)
                     break
                 except pyinotify.WatchManagerError as e:
                     if 'ENOENT' in str(e):
-                        watchtarget = f
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-- 
2.7.4

Re: [PATCH 7/8] bitbake: cooker: fix event notifications

[Richard Purdie]

On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
> There are two kinds of events:
> * directory
>   We always need handle the event since we don't know what inside the
>   directory, for example:
>   $ bitbake --server-only -T -1
>   $ bitbake -p
>   $ mkdir ${TOPDIR}/classes
>   $ cp /path/to/base.bbclass classes
> 
>   The "${TOPDIR}/classes" is a new directory, the event only reports
> that the
>   directory is created, we don't know there is a base.bbclass unless
> the parser
>   runs.
> 
> * file:
>   Use bb.parse.supports() to check whether bitbake supports it or
> not.
> 
> And now the bbwatchedfiles is not needed anymore.
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
>  1 file changed, 18 insertions(+), 16 deletions(-)

I'm not sure you can do this. For example, bitbake itself doesn't
know/care about *.patch files, however if one is referenced in a
SRC_URI and ends up on the watch list, would this change break a patch
file changing causing task checksums to change?

Its looking like we need better tests for some of this code too :/.

Cheers,

Richard

Re: [PATCH 7/8] bitbake: cooker: fix event notifications

[Robert Yang]

Hi RP,

On 02/02/2018 07:19 PM, Richard Purdie wrote:
> On Thu, 2018-02-01 at 23:15 +0800, Robert Yang wrote:
>> There are two kinds of events:
>> * directory
>>    We always need handle the event since we don't know what inside the
>>    directory, for example:
>>    $ bitbake --server-only -T -1
>>    $ bitbake -p
>>    $ mkdir ${TOPDIR}/classes
>>    $ cp /path/to/base.bbclass classes
>>
>>    The "${TOPDIR}/classes" is a new directory, the event only reports
>> that the
>>    directory is created, we don't know there is a base.bbclass unless
>> the parser
>>    runs.
>>
>> * file:
>>    Use bb.parse.supports() to check whether bitbake supports it or
>> not.
>>
>> And now the bbwatchedfiles is not needed anymore.
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   bitbake/lib/bb/cooker.py | 34 ++++++++++++++++++----------------
>>   1 file changed, 18 insertions(+), 16 deletions(-)
> 
> I'm not sure you can do this. For example, bitbake itself doesn't
> know/care about *.patch files, however if one is referenced in a
> SRC_URI and ends up on the watch list, would this change break a patch
> file changing causing task checksums to change?

I can confirm the the following 2 things:
1) The watcher only watches __depends (and __base_depends) which only contains
    files supported by bb.parse.supports(), so use bb.parse.supports() to check
    it is OK (or maybe it is the best way).

2) The *.patch is not part of of __depends, but file_checksums, and they are not
    watched by watchers. When a patch is updated, it would compare the saved
    mtime and current mtime (in lib/bb/checksum.py), and decide whether need
    recompute.

So I think that this patch is fine.

The testcase:

$ bitbake --server-only -T -1
$ bitbake quilt-native

# Update quilt-native's patch to test the rebuild:
$ echo "hello world" >> 
/path/to/meta/recipes-devtools/quilt/quilt/0001-tests-Allow-different-output-from-mv.patch

# Build quilt-native again
$ bitbake quilt-native

NOTE: Tasks Summary: Attempted 10 tasks of which 0 didn't need to be rerun and 
all succeeded.

We can see that it has been rebuilt.

// Robert

> 
> Its looking like we need better tests for some of this code too :/.
> 
> Cheers,
> 
> Richard
> 
> 

[PATCH 8/8] bitbake: cooker: improve inotify handling

[Robert Yang]

Each recipe calls bb_cache.add_info() and it calls add_filewatch() for every
depend file (not only existed files, but also non-exsited/attempted ones),
which caused the parsing and loading very slow. The watchers are global, so we
can only pass new depend files to add_filewatch() to improve the performance.

Here is the testing of 54 layers before and after the patch.

              Parsing       Loading
Before        35s           33s
Now           11s           9s

24 seconds are saved for parsing, and 24s are saved for loading.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cache.py  |  9 ++++++---
 bitbake/lib/bb/cooker.py | 24 +++++++++++++++---------
 2 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index 168a77a..7d65939 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -713,12 +713,15 @@ class Cache(NoCache):
     def mtime(cachefile):
         return bb.parse.cached_mtime_noerror(cachefile)
 
-    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None):
+    def add_info(self, filename, info_array, cacheData, parsed=None, watcher=None, watch_manager=None):
         if isinstance(info_array[0], CoreRecipeInfo) and (not info_array[0].skipped):
             cacheData.add_from_recipeinfo(filename, info_array)
 
-            if watcher:
-                watcher(info_array[0].file_depends)
+            if watcher and watch_manager:
+                deps_set = set()
+                for deps in info_array[0].file_depends:
+                    deps_set.add(deps[0])
+                watcher(deps_set - watch_manager.bbseen)
 
         if not self.has_cache:
             return
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index ffda225..cd98ba0 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -176,13 +176,13 @@ class BBCooker:
         self.configuration = configuration
 
         self.configwatcher = pyinotify.WatchManager()
-        self.configwatcher.bbseen = []
+        self.configwatcher.bbseen = set()
         self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
         self.watchmask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_CREATE | pyinotify.IN_DELETE | \
                          pyinotify.IN_DELETE_SELF | pyinotify.IN_MODIFY | pyinotify.IN_MOVE_SELF | \
                          pyinotify.IN_MOVED_FROM | pyinotify.IN_MOVED_TO
         self.watcher = pyinotify.WatchManager()
-        self.watcher.bbseen = []
+        self.watcher.bbseen = set()
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
         # If being called by something like tinfoil, we need to clean cached data
@@ -278,12 +278,14 @@ class BBCooker:
             watcher = self.watcher
         for i in deps:
             if dirs:
-                f = i[0]
+                f = i
             else:
-                f = os.path.dirname(i[0])
+                f = os.path.dirname(i)
             if f in watcher.bbseen:
                 continue
-            watcher.bbseen.append(f)
+            # Add both files and directories to to bbseen
+            watcher.bbseen.add(f)
+            watcher.bbseen.add(i)
             while True:
                 # We try and add watches for files that don't exist but if they did, would influence
                 # the parser. The parent directory of these files may not exist, in which case we need
@@ -296,7 +298,7 @@ class BBCooker:
                         f = os.path.dirname(f)
                         if f in watcher.bbseen:
                             break
-                        watcher.bbseen.append(f)
+                        watcher.bbseen.add(f)
                         continue
                     if 'ENOSPC' in str(e):
                         providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
@@ -367,7 +369,10 @@ class BBCooker:
             self.disableDataTracking()
 
         self.data.renameVar("__depends", "__base_depends")
-        self.add_filewatch(self.data.getVar("__base_depends", False), self.configwatcher)
+        deps_set = set()
+        for deps in self.data.getVar("__base_depends", False):
+            deps_set.add(deps[0])
+        self.add_filewatch(deps_set, self.configwatcher)
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
@@ -1510,7 +1515,7 @@ class BBCooker:
 
             # Add inotify watches for directories searched for bb/bbappend files
             for dirent in searchdirs:
-                self.add_filewatch([[dirent]], dirs=True)
+                self.add_filewatch([dirent], dirs=True)
 
             self.parser = CookerParser(self, filelist, masked)
             self.parsecache_valid = True
@@ -2144,7 +2149,8 @@ class CookerParser(object):
                 self.cooker.skiplist[virtualfn] = SkippedPackage(info_array[0])
             (fn, cls, mc) = bb.cache.virtualfn2realfn(virtualfn)
             self.bb_cache.add_info(virtualfn, info_array, self.cooker.recipecaches[mc],
-                                        parsed=parsed, watcher = self.cooker.add_filewatch)
+                                        parsed=parsed, watcher=self.cooker.add_filewatch,
+                                        watch_manager=self.cooker.watcher)
         return True
 
     def reparse(self, filename):
-- 
2.7.4