[PATCH 1/4] tests.data: Add underscore+numeric in overrides datastore test

[PATCH 1/4] tests.data: Add underscore+numeric in overrides datastore test

[Richard Purdie]

Add a test for x86_64 in overrides where is was being incorrectly handled.
There was a previous fix (3a3be518536acc868c7eeb3c1111ad1b321480b7) but
this ensures we don't regress.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/tests/data.py | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index db3e2010a9..3c511f214b 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -394,6 +394,15 @@ class TestOverrides(unittest.TestCase):
         self.d.setVar("OVERRIDES", "foo:bar:some_val")
         self.assertEqual(self.d.getVar("TEST"), " testvalue5")
 
+    # Test an override with _<numeric> in it based on a real world OE issue
+    def test_underscore_override(self):
+        self.d.setVar("TARGET_ARCH", "x86_64")
+        self.d.setVar("PN", "test-${TARGET_ARCH}")
+        self.d.setVar("VERSION", "1")
+        self.d.setVar("VERSION_pn-test-${TARGET_ARCH}", "2")
+        self.d.setVar("OVERRIDES", "pn-${PN}")
+        bb.data.expandKeys(self.d)
+        self.assertEqual(self.d.getVar("VERSION"), "2")
 
 class TestKeyExpansion(unittest.TestCase):
     def setUp(self):
-- 
2.19.1

[PATCH 2/4] cooker: Remove Feeder() since its no longer needed with moderm multiprocessing

[Richard Purdie]

There used to be many bugs in multiprocessing and we implemented our own
feeder process to avoid them. Now that we have python 3.x, these are fixed
and just using the standard Queue mechanism appears to work fine. We can
therefore drop the unneeded code and simplify.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 42 +++++-------------------------------------
 1 file changed, 5 insertions(+), 37 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index db52964c3a..d1d2868d6f 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1895,35 +1895,6 @@ class ParsingFailure(Exception):
         self.recipe = recipe
         Exception.__init__(self, realexception, recipe)
 
-class Feeder(multiprocessing.Process):
-    def __init__(self, jobs, to_parsers, quit):
-        self.quit = quit
-        self.jobs = jobs
-        self.to_parsers = to_parsers
-        multiprocessing.Process.__init__(self)
-
-    def run(self):
-        while True:
-            try:
-                quit = self.quit.get_nowait()
-            except queue.Empty:
-                pass
-            else:
-                if quit == 'cancel':
-                    self.to_parsers.cancel_join_thread()
-                break
-
-            try:
-                job = self.jobs.pop()
-            except IndexError:
-                break
-
-            try:
-                self.to_parsers.put(job, timeout=0.5)
-            except queue.Full:
-                self.jobs.insert(0, job)
-                continue
-
 class Parser(multiprocessing.Process):
     def __init__(self, jobs, results, quit, init, profile):
         self.jobs = jobs
@@ -2058,12 +2029,13 @@ class CookerParser(object):
                 multiprocessing.util.Finalize(None, bb.codeparser.parser_cache_save, exitpriority=1)
                 multiprocessing.util.Finalize(None, bb.fetch.fetcher_parse_save, exitpriority=1)
 
-            self.feeder_quit = multiprocessing.Queue(maxsize=1)
             self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes)
-            self.jobs = multiprocessing.Queue(maxsize=self.num_processes)
             self.result_queue = multiprocessing.Queue()
-            self.feeder = Feeder(self.willparse, self.jobs, self.feeder_quit)
-            self.feeder.start()
+
+            self.jobs = multiprocessing.Queue()
+            for j in self.willparse:
+                self.jobs.put(j)
+
             for i in range(0, self.num_processes):
                 parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
                 parser.start()
@@ -2086,12 +2058,9 @@ class CookerParser(object):
                                             self.total)
 
             bb.event.fire(event, self.cfgdata)
-            self.feeder_quit.put(None)
             for process in self.processes:
                 self.parser_quit.put(None)
         else:
-            self.feeder_quit.put('cancel')
-
             self.parser_quit.cancel_join_thread()
             for process in self.processes:
                 self.parser_quit.put(None)
@@ -2104,7 +2073,6 @@ class CookerParser(object):
                 process.terminate()
             else:
                 process.join()
-        self.feeder.join()
 
         sync = threading.Thread(target=self.bb_cache.sync)
         sync.start()
-- 
2.19.1

[PATCH 3/4] cooker: Allow faster exitting from parser threads

[Richard Purdie]

We don't push "None" values onto the parser queue so currently idle parsing
threads loop every 0.25s and idle. They may as well exit out of their work
is done.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index d1d2868d6f..9b2a0920f9 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1943,9 +1943,6 @@ class Parser(multiprocessing.Process):
                 try:
                     job = self.jobs.get(timeout=0.25)
                 except queue.Empty:
-                    continue
-
-                if job is None:
                     break
                 result = self.parse(*job)
 
-- 
2.19.1

[PATCH 4/4] cooker: Split recipes to parse amongst threads ahead of time

[Richard Purdie]

We have two choices, split the recipes amongst the parsing threads in
blocks ahead of time, or have a queue which parsers pull from when idle.

The optimum approach depends on how similar the pieces are. For the single
recipe reparse case, there is currently a significant wait for the feeder
thread to start (around 0.25s in a 2s command).

Its possible splitting into blocks in advance may be unluckly for some other
workloads but experimentally it seems to work better overall for me at least.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 9b2a0920f9..e6b8d880ae 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1941,8 +1941,8 @@ class Parser(multiprocessing.Process):
                 result = pending.pop()
             else:
                 try:
-                    job = self.jobs.get(timeout=0.25)
-                except queue.Empty:
+                    job = self.jobs.pop()
+                except IndexError:
                     break
                 result = self.parse(*job)
 
@@ -2029,12 +2029,12 @@ class CookerParser(object):
             self.parser_quit = multiprocessing.Queue(maxsize=self.num_processes)
             self.result_queue = multiprocessing.Queue()
 
-            self.jobs = multiprocessing.Queue()
-            for j in self.willparse:
-                self.jobs.put(j)
+            def chunkify(lst,n):
+                return [lst[i::n] for i in range(n)]
+            self.jobs = chunkify(self.willparse, self.num_processes)
 
             for i in range(0, self.num_processes):
-                parser = Parser(self.jobs, self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
+                parser = Parser(self.jobs[i], self.result_queue, self.parser_quit, init, self.cooker.configuration.profile)
                 parser.start()
                 self.process_names.append(parser.name)
                 self.processes.append(parser)
@@ -2062,8 +2062,6 @@ class CookerParser(object):
             for process in self.processes:
                 self.parser_quit.put(None)
 
-            self.jobs.cancel_join_thread()
-
         for process in self.processes:
             if force:
                 process.join(.1)
-- 
2.19.1

Re: [PATCH 3/4] cooker: Allow faster exitting from parser threads

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

Empty is raised when it times out, not just when it’s actually empty,
though.

On Mon, Jan 7, 2019 at 8:55 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> We don't push "None" values onto the parser queue so currently idle parsing
> threads loop every 0.25s and idle. They may as well exit out of their work
> is done.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/cooker.py | 3 ---
>  1 file changed, 3 deletions(-)
>
> diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
> index d1d2868d6f..9b2a0920f9 100644
> --- a/lib/bb/cooker.py
> +++ b/lib/bb/cooker.py
> @@ -1943,9 +1943,6 @@ class Parser(multiprocessing.Process):
>                  try:
>                      job = self.jobs.get(timeout=0.25)
>                  except queue.Empty:
> -                    continue
> -
> -                if job is None:
>                      break
>                  result = self.parse(*job)
>
> --
> 2.19.1
>
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>


-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2183 bytes --]

Re: [PATCH 3/4] cooker: Allow faster exitting from parser threads

[richard.purdie]

On Mon, 2019-01-07 at 09:02 -0700, Christopher Larson wrote:
> Empty is raised when it times out, not just when it’s actually empty,
> though.

Good point, thanks. I was about to rework it but my next patch removes
it entirely so I guess I can just drop this intermediate one...

Cheers,

Richard