* [1.44 0/4] Pull request
@ 2020-01-25 16:01 Armin Kuster
2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster
` (3 more replies)
0 siblings, 4 replies; 6+ messages in thread
From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw)
To: bitbake-devel
Please concider these changes for 1.44
The following changes since commit 46969195492708a705390896c9dea515f158954c:
cache: Lower debug level for wold build messages (2020-01-02 20:23:09 -0800)
are available in the Git repository at:
git://git.openembedded.org/bitbake-contrib stable/1.44-next
http://cgit.openembedded.org//log/?h=stable/1.44-next
Peter Kjellerstedt (3):
cooker: Keep track of watched files using a set instead of a list
knotty: Hide the footer if a process progress bar is shown
knotty: Be consistent when creating/updating progress bars
Richard Purdie (1):
runqueue: Use a set for the setscene tasks list
lib/bb/cooker.py | 16 ++++++++--------
lib/bb/runqueue.py | 4 ++--
lib/bb/ui/knotty.py | 17 ++++++++++-------
3 files changed, 20 insertions(+), 17 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 6+ messages in thread* [1.44 1/4] cooker: Keep track of watched files using a set instead of a list 2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster @ 2020-01-25 16:01 ` Armin Kuster 2020-01-25 16:01 ` [1.44 2/4] knotty: Hide the footer if a process progress bar is shown Armin Kuster ` (2 subsequent siblings) 3 siblings, 0 replies; 6+ messages in thread From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw) To: bitbake-devel From: Peter Kjellerstedt <peter.kjellerstedt@axis.com> When there are many watched files, keeping track of them using lists is suboptimal. Using sets improves the performance considerably. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 1e96df260e47d160dbd36bfc92c31ef06266f662) Signed-off-by: Armin Kuster <akuster808@gmail.com> --- lib/bb/cooker.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py index e6442bff..b74affa7 100644 --- a/lib/bb/cooker.py +++ b/lib/bb/cooker.py @@ -169,8 +169,8 @@ class BBCooker: bb.debug(1, "BBCooker pyinotify1 %s" % time.time()) sys.stdout.flush() - self.configwatcher.bbseen = [] - self.configwatcher.bbwatchedfiles = [] + self.configwatcher.bbseen = set() + self.configwatcher.bbwatchedfiles = set() self.confignotifier = pyinotify.Notifier(self.configwatcher, self.config_notifications) bb.debug(1, "BBCooker pyinotify2 %s" % time.time()) sys.stdout.flush() @@ -180,8 +180,8 @@ class BBCooker: self.watcher = pyinotify.WatchManager() bb.debug(1, "BBCooker pyinotify3 %s" % time.time()) sys.stdout.flush() - self.watcher.bbseen = [] - self.watcher.bbwatchedfiles = [] + self.watcher.bbseen = set() + self.watcher.bbwatchedfiles = set() self.notifier = pyinotify.Notifier(self.watcher, self.notifications) bb.debug(1, "BBCooker pyinotify complete %s" % time.time()) @@ -278,14 +278,14 @@ class BBCooker: if not watcher: watcher = self.watcher for i in deps: - watcher.bbwatchedfiles.append(i[0]) + watcher.bbwatchedfiles.add(i[0]) if dirs: f = i[0] else: f = os.path.dirname(i[0]) if f in watcher.bbseen: continue - watcher.bbseen.append(f) + watcher.bbseen.add(f) watchtarget = None while True: # We try and add watches for files that don't exist but if they did, would influence @@ -294,7 +294,7 @@ class BBCooker: try: watcher.add_watch(f, self.watchmask, quiet=False) if watchtarget: - watcher.bbwatchedfiles.append(watchtarget) + watcher.bbwatchedfiles.add(watchtarget) break except pyinotify.WatchManagerError as e: if 'ENOENT' in str(e): @@ -302,7 +302,7 @@ class BBCooker: f = os.path.dirname(f) if f in watcher.bbseen: break - watcher.bbseen.append(f) + watcher.bbseen.add(f) continue if 'ENOSPC' in str(e): providerlog.error("No space left on device or exceeds fs.inotify.max_user_watches?") -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 2/4] knotty: Hide the footer if a process progress bar is shown 2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster 2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster @ 2020-01-25 16:01 ` Armin Kuster 2020-01-25 16:01 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster 2020-01-25 16:01 ` [1.44 4/4] runqueue: Use a set for the setscene tasks list Armin Kuster 3 siblings, 0 replies; 6+ messages in thread From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw) To: bitbake-devel From: Peter Kjellerstedt <pkj@axis.com> With the introduction of the hash equivalence server, the progress bar for "Checking sstate mirror object availability" is shown repeatedly while the tasks are being executed. If the footer is not hidden then, it will be moved up one line every time, creating a messy interface. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 56b5ec4c2b3e658e73ca6c3a12feeb96df0977fb) Signed-off-by: Armin Kuster <akuster808@gmail.com> --- lib/bb/ui/knotty.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index bd9911cf..ddcb2148 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -477,7 +477,8 @@ def main(server, eventHandler, params, tf = TerminalFilter): if event is None: if main.shutdown > 1: break - termfilter.updateFooter() + if not parseprogress: + termfilter.updateFooter() event = eventHandler.waitEvent(0.25) if event is None: continue @@ -539,6 +540,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): continue if event.total == 0: continue + termfilter.clearFooter() parseprogress = new_progress("Parsing recipes", event.total).start() continue if isinstance(event, bb.event.ParseProgress): @@ -638,6 +640,7 @@ def main(server, eventHandler, params, tf = TerminalFilter): if isinstance(event, bb.event.ProcessStarted): if params.options.quiet > 1: continue + termfilter.clearFooter() parseprogress = new_progress(event.processname, event.total) parseprogress.start(False) continue -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 3/4] knotty: Be consistent when creating/updating progress bars 2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster 2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster 2020-01-25 16:01 ` [1.44 2/4] knotty: Hide the footer if a process progress bar is shown Armin Kuster @ 2020-01-25 16:01 ` Armin Kuster 2020-01-25 16:01 ` [1.44 4/4] runqueue: Use a set for the setscene tasks list Armin Kuster 3 siblings, 0 replies; 6+ messages in thread From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw) To: bitbake-devel From: Peter Kjellerstedt <pkj@axis.com> When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682) Signed-off-by: Armin Kuster <akuster808@gmail.com> --- lib/bb/ui/knotty.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index ddcb2148..c6abb2a1 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -35,15 +35,15 @@ class BBProgress(progressbar.ProgressBar): self.msg = msg self.extrapos = extrapos if not widgets: - widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ', - progressbar.ETA()] - self.extrapos = 4 + widgets = [': ', progressbar.Percentage(), ' ', progressbar.Bar(), + ' ', progressbar.ETA()] + self.extrapos = 5 if resize_handler: self._resize_default = resize_handler else: self._resize_default = signal.getsignal(signal.SIGWINCH) - progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout) + progressbar.ProgressBar.__init__(self, maxval, [self.msg] + widgets, fd=sys.stdout) def _handle_resize(self, signum=None, frame=None): progressbar.ProgressBar._handle_resize(self, signum, frame) @@ -255,10 +255,10 @@ class TerminalFilter(object): start_time = activetasks[t].get("starttime", None) if not pbar or pbar.bouncing != (progress < 0): if progress < 0: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.BouncingSlider(), ''], extrapos=3, resize_handler=self.sigwinch_handle) pbar.bouncing = True else: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=5, resize_handler=self.sigwinch_handle) pbar.bouncing = False activetasks[t]["progressbar"] = pbar tasks.append((pbar, progress, rate, start_time)) -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 4/4] runqueue: Use a set for the setscene tasks list 2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster ` (2 preceding siblings ...) 2020-01-25 16:01 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster @ 2020-01-25 16:01 ` Armin Kuster 3 siblings, 0 replies; 6+ messages in thread From: Armin Kuster @ 2020-01-25 16:01 UTC (permalink / raw) To: bitbake-devel From: Richard Purdie <richard.purdie@linuxfoundation.org> This should give performance improvements to functions using this list of tasks (sets are used for most of the other code for this reason, not sure why this wasn't a set in the first place). Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit f5daef68703481a3c243dfecc7de404e6ebfdbb6) Signed-off-by: Armin Kuster <akuster808@gmail.com> --- lib/bb/runqueue.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 6e3a91b8..d7186e85 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1124,14 +1124,14 @@ class RunQueueData: self.init_progress_reporter.next_stage() # Iterate over the task list looking for tasks with a 'setscene' function - self.runq_setscene_tids = [] + self.runq_setscene_tids = set() if not self.cooker.configuration.nosetscene: for tid in self.runtaskentries: (mc, fn, taskname, _) = split_tid_mcfn(tid) setscenetid = tid + "_setscene" if setscenetid not in taskData[mc].taskentries: continue - self.runq_setscene_tids.append(tid) + self.runq_setscene_tids.add(tid) self.init_progress_reporter.next_stage() -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [1.44 0/4] Patch review @ 2020-01-22 5:24 Armin Kuster 2020-01-22 5:25 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster 0 siblings, 1 reply; 6+ messages in thread From: Armin Kuster @ 2020-01-22 5:24 UTC (permalink / raw) To: bitbake-devel Most of these have been on the list already. Please havecComments back by Thusday The following changes since commit 46969195492708a705390896c9dea515f158954c: cache: Lower debug level for wold build messages (2020-01-02 20:23:09 -0800) are available in the Git repository at: git://git.openembedded.org/bitbake-contrib stable/1.44-nut http://cgit.openembedded.org/bitbake-contrib/log/?h=stable/1.44-nut Peter Kjellerstedt (3): cooker: Keep track of watched files using a set instead of a list knotty: Hide the footer if a process progress bar is shown knotty: Be consistent when creating/updating progress bars Richard Purdie (1): runqueue: Use a set for the setscene tasks list lib/bb/cooker.py | 16 ++++++++-------- lib/bb/runqueue.py | 4 ++-- lib/bb/ui/knotty.py | 17 ++++++++++------- 3 files changed, 20 insertions(+), 17 deletions(-) -- 2.17.1 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [1.44 3/4] knotty: Be consistent when creating/updating progress bars 2020-01-22 5:24 [1.44 0/4] Patch review Armin Kuster @ 2020-01-22 5:25 ` Armin Kuster 0 siblings, 0 replies; 6+ messages in thread From: Armin Kuster @ 2020-01-22 5:25 UTC (permalink / raw) To: bitbake-devel From: Peter Kjellerstedt <pkj@axis.com> When creating a new progress bar (using BBProgress), a colon was appended to the supplied message. However, when updating the message, no colon was appended. Change this so that the colon is instead part of the widgets that make up the progress bar so that it does not matter when and how the message is updated, it always displays the same. Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> (cherry picked from commit 08f35c04f6e1ce4c4ca5c2bef4cd8a192e12e682) Signed-off-by: Armin Kuster <akuster808@gmail.com> --- lib/bb/ui/knotty.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/bb/ui/knotty.py b/lib/bb/ui/knotty.py index ddcb2148..c6abb2a1 100644 --- a/lib/bb/ui/knotty.py +++ b/lib/bb/ui/knotty.py @@ -35,15 +35,15 @@ class BBProgress(progressbar.ProgressBar): self.msg = msg self.extrapos = extrapos if not widgets: - widgets = [progressbar.Percentage(), ' ', progressbar.Bar(), ' ', - progressbar.ETA()] - self.extrapos = 4 + widgets = [': ', progressbar.Percentage(), ' ', progressbar.Bar(), + ' ', progressbar.ETA()] + self.extrapos = 5 if resize_handler: self._resize_default = resize_handler else: self._resize_default = signal.getsignal(signal.SIGWINCH) - progressbar.ProgressBar.__init__(self, maxval, [self.msg + ": "] + widgets, fd=sys.stdout) + progressbar.ProgressBar.__init__(self, maxval, [self.msg] + widgets, fd=sys.stdout) def _handle_resize(self, signum=None, frame=None): progressbar.ProgressBar._handle_resize(self, signum, frame) @@ -255,10 +255,10 @@ class TerminalFilter(object): start_time = activetasks[t].get("starttime", None) if not pbar or pbar.bouncing != (progress < 0): if progress < 0: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.BouncingSlider(), ''], extrapos=2, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.BouncingSlider(), ''], extrapos=3, resize_handler=self.sigwinch_handle) pbar.bouncing = True else: - pbar = BBProgress("0: %s (pid %s) " % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=4, resize_handler=self.sigwinch_handle) + pbar = BBProgress("0: %s (pid %s)" % (activetasks[t]["title"], activetasks[t]["pid"]), 100, widgets=[' ', progressbar.Percentage(), ' ', progressbar.Bar(), ''], extrapos=5, resize_handler=self.sigwinch_handle) pbar.bouncing = False activetasks[t]["progressbar"] = pbar tasks.append((pbar, progress, rate, start_time)) -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-01-25 16:01 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-01-25 16:01 [1.44 0/4] Pull request Armin Kuster 2020-01-25 16:01 ` [1.44 1/4] cooker: Keep track of watched files using a set instead of a list Armin Kuster 2020-01-25 16:01 ` [1.44 2/4] knotty: Hide the footer if a process progress bar is shown Armin Kuster 2020-01-25 16:01 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster 2020-01-25 16:01 ` [1.44 4/4] runqueue: Use a set for the setscene tasks list Armin Kuster -- strict thread matches above, loose matches on Subject: below -- 2020-01-22 5:24 [1.44 0/4] Patch review Armin Kuster 2020-01-22 5:25 ` [1.44 3/4] knotty: Be consistent when creating/updating progress bars Armin Kuster
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.