* [1.44 1/4] cooker: Keep track of watched files using a set instead of a list
2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster
@ 2020-01-25 16:01 ` Armin Kuster
2020-01-25 16:01 ` [1.44 2/4] knotty: Hide the footer if a process progress bar is shown Armin Kuster
` (2 subsequent siblings)
3 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw)
To: bitbake-devel
From: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
When there are many watched files, keeping track of them using lists
is suboptimal. Using sets improves the performance considerably.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/cooker.py | 16 ++++++++--------
1 file changed, 8 insertions(+), 8 deletions(-)
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index e6442bff..b74affa7 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -169,8 +169,8 @@ class BBCooker:
bb.debug(1, "BBCooker pyinotify1 %s" % time.time())
sys.stdout.flush()
- self.configwatcher.bbseen = []
- self.configwatcher.bbwatchedfiles = []
+ self.configwatcher.bbseen = set()
+ self.configwatcher.bbwatchedfiles = set()
self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications)
bb.debug(1, "BBCooker pyinotify2 %s" % time.time())
sys.stdout.flush()
@@ -180,8 +180,8 @@ class BBCooker:
self.watcher = pyinotify.WatchManager()
bb.debug(1, "BBCooker pyinotify3 %s" % time.time())
sys.stdout.flush()
- self.watcher.bbseen = []
- self.watcher.bbwatchedfiles = []
+ self.watcher.bbseen = set()
+ self.watcher.bbwatchedfiles = set()
self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
bb.debug(1, "BBCooker pyinotify complete %s" % time.time())
@@ -278,14 +278,14 @@ class BBCooker:
if not watcher:
watcher = self.watcher
for i in deps:
- watcher.bbwatchedfiles.append(i[0])
+ watcher.bbwatchedfiles.add(i[0])
if dirs:
f = i[0]
else:
f = os.path.dirname(i[0])
if f in watcher.bbseen:
continue
- watcher.bbseen.append(f)
+ watcher.bbseen.add(f)
watchtarget = None
while True:
# We try and add watches for files that don't exist but if they did, would influence
@@ -294,7 +294,7 @@ class BBCooker:
try:
watcher.add_watch(f, self.watchmask, quiet=False)
if watchtarget:
- watcher.bbwatchedfiles.append(watchtarget)
+ watcher.bbwatchedfiles.add(watchtarget)
break
except pyinotify.WatchManagerError as e:
if 'ENOENT' in str(e):
@@ -302,7 +302,7 @@ class BBCooker:
f = os.path.dirname(f)
if f in watcher.bbseen:
break
- watcher.bbseen.append(f)
+ watcher.bbseen.add(f)
continue
if 'ENOSPC' in str(e):
providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?")
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 2/4] knotty: Hide the footer if a process progress bar is shown
2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster
2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster
@ 2020-01-25 16:01 ` Armin Kuster
2020-01-25 16:01 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster
2020-01-25 16:01 ` [1.44 4/4] runqueue: Use a set for the setscene tasks list Armin Kuster
3 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw)
To: bitbake-devel
From: Peter Kjellerstedt <pkj@axis.com>
With the introduction of the hash equivalence server, the progress bar
for "Checking sstate mirror object availability" is shown repeatedly
while the tasks are being executed. If the footer is not hidden then,
it will be moved up one line every time, creating a messy interface.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/ui/knotty.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index bd9911cf..ddcb2148 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -477,7 +477,8 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if event is None:
if main.shutdown > 1:
break
- termfilter.updateFooter()
+ if not parseprogress:
+ termfilter.updateFooter()
event = eventHandler.waitEvent(0.25)
if event is None:
continue
@@ -539,6 +540,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
continue
if event.total == 0:
continue
+ termfilter.clearFooter()
parseprogress = new_progress("Parsing recipes", event.total).start()
continue
if isinstance(event, bb.event.ParseProgress):
@@ -638,6 +640,7 @@ def main(server, eventHandler, params, tf = TerminalFilter):
if isinstance(event, bb.event.ProcessStarted):
if params.options.quiet > 1:
continue
+ termfilter.clearFooter()
parseprogress = new_progress(event.processname, event.total)
parseprogress.start(False)
continue
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 3/4] knotty: Be consistent when creating/updating progress bars
2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster
2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster
2020-01-25 16:01 ` [1.44 2/4] knotty: Hide the footer if a process progress bar is shown Armin Kuster
@ 2020-01-25 16:01 ` Armin Kuster
2020-01-25 16:01 ` [1.44 4/4] runqueue: Use a set for the setscene tasks list Armin Kuster
3 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw)
To: bitbake-devel
From: Peter Kjellerstedt <pkj@axis.com>
When creating a new progress bar (using BBProgress), a colon was
appended to the supplied message. However, when updating the message,
no colon was appended.
Change this so that the colon is instead part of the widgets that make
up the progress bar so that it does not matter when and how the
message is updated, it always displays the same.
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/ui/knotty.py | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py
index ddcb2148..c6abb2a1 100644
--- a/lib/bb/ui/knotty.py
+++ b/lib/bb/ui/knotty.py
@@ -35,15 +35,15 @@ class BBProgress(progressbar.ProgressBar):
self.msg = msg
self.extrapos = extrapos
if not widgets:
- widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ',
- progressbar.ETA()]
- self.extrapos = 4
+ widgets = [': ', progressbar.Percentage(), ' ', progressbar.Bar(),
+ ' ', progressbar.ETA()]
+ self.extrapos = 5
if resize_handler:
self._resize_default = resize_handler
else:
self._resize_default = signal.getsignal(signal.SIGWINCH)
- progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout)
+ progressbar.ProgressBar.__init__(self, maxval, [self.msg] + widgets, fd=sys.stdout)
def _handle_resize(self, signum=None, frame=None):
progressbar.ProgressBar._handle_resize(self, signum, frame)
@@ -255,10 +255,10 @@ class TerminalFilter(object):
start_time = activetasks[t].get("starttime", None)
if not pbar or pbar.bouncing != (progress < 0):
if progress < 0:
- pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle)
+ pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.BouncingSlider(), ''], extrapos=3, resize_handler=self.sigwinch_handle)
pbar.bouncing = True
else:
- pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle)
+ pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=5, resize_handler=self.sigwinch_handle)
pbar.bouncing = False
activetasks[t]["progressbar"] = pbar
tasks.append((pbar, progress, rate, start_time))
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 4/4] runqueue: Use a set for the setscene tasks list
2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster
` (2 preceding siblings ...)
2020-01-25 16:01 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster
@ 2020-01-25 16:01 ` Armin Kuster
3 siblings, 0 replies; 6+ messages in thread
From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
This should give performance improvements to functions using this list of
tasks (sets are used for most of the other code for this reason, not sure
why this wasn't a set in the first place).
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f5daef68703481a3c243dfecc7de404e6ebfdbb6)
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
lib/bb/runqueue.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 6e3a91b8..d7186e85 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1124,14 +1124,14 @@ class RunQueueData:
self.init_progress_reporter.next_stage()
# Iterate over the task list looking for tasks with a 'setscene' function
- self.runq_setscene_tids = []
+ self.runq_setscene_tids = set()
if not self.cooker.configuration.nosetscene:
for tid in self.runtaskentries:
(mc, fn, taskname, _) = split_tid_mcfn(tid)
setscenetid = tid + "_setscene"
if setscenetid not in taskData[mc].taskentries:
continue
- self.runq_setscene_tids.append(tid)
+ self.runq_setscene_tids.add(tid)
self.init_progress_reporter.next_stage()
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread