[PATCH 1/2] cooker: Further fixes to inotify to fix memres bitbake issues

[PATCH 1/2] cooker: Further fixes to inotify to fix memres bitbake issues

[Richard Purdie]

The previous fix for inotify wasn't quite correct as we need to modify
bbseen before calling add_filewatch(). We also need to ensure the parse
mtime cache is cleared when directories are added/removed. There was also
a typo in the original fix and the wrong watcher was being changed. Fix
the various issues which improves memory resident bitbake testing results.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index f435b18c87..1359d33f77 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -254,9 +254,14 @@ class BBCooker:
         if not event.pathname in self.configwatcher.bbwatchedfiles:
             return
         if "IN_ISDIR" in event.maskname:
+            if "IN_CREATE" in event.maskname or "IN_DELETE" in event.maskname:
+                if event.pathname in self.configwatcher.bbseen:
+                    self.configwatcher.bbseen.remove(event.pathname)
+                # Could remove all entries starting with the directory but for now...
+                bb.parse.clear_cache()
             if "IN_CREATE" in event.maskname:
                 self.add_filewatch([[event.pathname]], watcher=self.configwatcher, dirs=True)
-            elif "IN_DELETE" in event.maskname and event.pathname in self.watcher.bbseen:
+            elif "IN_DELETE" in event.maskname and event.pathname in self.configwatcher.bbseen:
                 self.configwatcher.bbseen.remove(event.pathname)
         if not event.pathname in self.inotify_modified_files:
             self.inotify_modified_files.append(event.pathname)
@@ -272,6 +277,11 @@ class BBCooker:
                 or event.pathname.endswith("bitbake.lock"):
             return
         if "IN_ISDIR" in event.maskname:
+            if "IN_CREATE" in event.maskname or "IN_DELETE" in event.maskname:
+                if event.pathname in self.watcher.bbseen:
+                    self.watcher.bbseen.remove(event.pathname)
+                # Could remove all entries starting with the directory but for now...
+                bb.parse.clear_cache()
             if "IN_CREATE" in event.maskname:
                 self.add_filewatch([[event.pathname]], dirs=True)
             elif "IN_DELETE" in event.maskname and event.pathname in self.watcher.bbseen:
-- 
2.32.0

[PATCH 2/2] cooker: Restore sys.path and sys.modules between parses

[Richard Purdie]

When memory resident bitbake is active and we re-parse, the old module
configuration is present which can lead to strange errors. Reset this
when reparsing so the state is consistent. This fixes memory resident
bitbake errors.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 1359d33f77..4753fd16e2 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -159,6 +159,9 @@ class BBCooker:
             for f in featureSet:
                 self.featureset.setFeature(f)
 
+        self.orig_syspath = sys.path.copy()
+        self.orig_sysmodules = sys.modules.copy()
+
         self.configuration = bb.cookerdata.CookerConfiguration()
 
         self.idleCallBackRegister = idleCallBackRegister
@@ -350,6 +353,9 @@ class BBCooker:
         self.state = state.initial
         self.caches_array = []
 
+        sys.path = self.orig_syspath.copy()
+        sys.modules = self.orig_sysmodules.copy()
+
         # Need to preserve BB_CONSOLELOG over resets
         consolelog = None
         if hasattr(self, "data"):
-- 
2.32.0

Re: [bitbake-devel] [PATCH 2/2] cooker: Restore sys.path and sys.modules between parses

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 653 bytes --]

On Thu, Mar 31, 2022 at 4:04 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> When memory resident bitbake is active and we re-parse, the old module
> configuration is present which can lead to strange errors. Reset this
> when reparsing so the state is consistent. This fixes memory resident
> bitbake errors.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>

Looks good!

Acked-by: Christopher Larson <chris_larson@mentor.com>
-- 
Christopher Larson
chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital
Industries Software

[-- Attachment #2: Type: text/html, Size: 1375 bytes --]