* [PATCH 0/3][v4] Fix task buildstats gathering
@ 2016-02-23 12:58 Elliot Smith
2016-02-23 12:58 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith
` (3 more replies)
0 siblings, 4 replies; 12+ messages in thread
From: Elliot Smith @ 2016-02-23 12:58 UTC (permalink / raw)
To: toaster
v4:
I messed up the patch submission, so I'm sending the correct patches this time.
v3:
Only show a single navigation item in the left-hand nav panel for CPU times,
but show both System and User time columns when it is selected.
Add a test for CPU time subpage.
v2:
Store "started" and "ended" times for each task. (This will enable other
improvements later, such as showing when tasks overlap each other.)
Compute elapsed_time from start and end times, rather than in
toaster.bbclass.
Store disk IO in two extra fields, one for read and the other for write.
Compute total disk_io from these two values.
(elapsed_time and disk_io are still stored, as these are used for sorting
and we can't easily sort on a computed field. This can be amended when we
get round to putting the tasks table into ToasterTable; see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.)
Modify the UI to show the CPU times split between system and user.
v1:
Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway.
Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper.
To test:
1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe).
2. Run a build against a local Yocto (as one patch is for toaster.bbclass).
3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842.
Changes since 961893fa05f4d51187f4d5daadacd7927d1a0847 (toaster-next) are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842
Related bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842
NOTE: The first commit in this series is for oe-core.
Elliot Smith (3):
toaster.bbclass: improve how we gather buildstats for Toaster
toaster: rework task buildstats storage and display
toaster-tests: add test for cputime subpage
bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------
.../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++
bitbake/lib/toaster/orm/models.py | 18 +++++-
.../toastergui/templates/basebuildpage.html | 4 +-
.../lib/toaster/toastergui/templates/tasks.html | 20 +++---
bitbake/lib/toaster/toastergui/tests.py | 45 +++++++++++++-
bitbake/lib/toaster/toastergui/urls.py | 2 +-
bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------
meta/classes/toaster.bbclass | 70 +++++++++++++--------
9 files changed, 237 insertions(+), 97 deletions(-)
create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py
--
Elliot Smith
Software Engineer
Intel OTC
---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-02-23 12:58 [PATCH 0/3][v4] Fix task buildstats gathering Elliot Smith @ 2016-02-23 12:58 ` Elliot Smith 2016-02-23 12:58 ` [PATCH 2/3] toaster: rework task buildstats storage and display Elliot Smith ` (2 subsequent siblings) 3 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-23 12:58 UTC (permalink / raw) To: toaster Clean up the code which gathers buildstats for Toaster, and modify the field names so that the correct parts of the buildstats files are used to derive the CPU usage values. Also derive elapsed time for the build here, rather than in Toaster, as we have ready access to the data in the correct format. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass index 51a4c74..b7e8773 100644 --- a/meta/classes/toaster.bbclass +++ b/meta/classes/toaster.bbclass @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { import bb.utils import os + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") + if not e.data.getVar('BUILDSTATS_BASE', True): return # if we don't have buildstats, we cannot collect stats + def stat_to_float(value): + return float(value.strip('% \n\r')) + def _append_read_list(v): lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: + with open(toaster_statlist_file, "a") as fout: taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) bb.utils.unlockfile(lock) def _read_stats(filename): - cpu_usage = 0 - disk_io = 0 - started = '0' - ended = '0' - pn = '' + # seconds + cpu_time_user = 0 + cpu_time_system = 0 + + # bytes + disk_io_read = 0 + disk_io_write = 0 + + started = 0 + ended = 0 + taskname = '' + statinfo = {} with open(filename, 'r') as task_bs: @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { k,v = line.strip().split(": ", 1) statinfo[k] = v - if "CPU usage" in statinfo: - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') - - if "IO write_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) - - if "IO read_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) - if "Started" in statinfo: - started = str(statinfo["Started"]).strip('% \n\r') + started = stat_to_float(statinfo["Started"]) if "Ended" in statinfo: - ended = str(statinfo["Ended"]).strip('% \n\r') + ended = stat_to_float(statinfo["Ended"]) - elapsed_time = float(ended) - float(started) + if "Child rusage ru_utime" in statinfo: + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) - cpu_usage = float(cpu_usage) + if "Child rusage ru_stime" in statinfo: + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} + if "IO write_bytes" in statinfo: + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) + disk_io_write = disk_io_write + write_bytes + if "IO read_bytes" in statinfo: + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) + disk_io_read = disk_io_read + read_bytes + + return { + 'stat_file': filename, + 'cpu_time_user': cpu_time_user, + 'cpu_time_system': cpu_time_system, + 'disk_io_read': disk_io_read, + 'disk_io_write': disk_io_write, + 'started': started, + 'ended': ended + } if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): _append_read_list(e) pass - - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): events = [] - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: + with open(toaster_statlist_file, "r") as fin: for line in fin: (taskfile, taskname, filename, recipename) = line.strip().split("::") - events.append((taskfile, taskname, _read_stats(filename), recipename)) + stats = _read_stats(filename) + events.append((taskfile, taskname, stats, recipename)) bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) + #os.unlink(toaster_statlist_file) } # dump relevant build history data as an event when the build is completed -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/3] toaster: rework task buildstats storage and display 2016-02-23 12:58 [PATCH 0/3][v4] Fix task buildstats gathering Elliot Smith 2016-02-23 12:58 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith @ 2016-02-23 12:58 ` Elliot Smith 2016-02-23 12:58 ` [PATCH 3/3] toaster-tests: add test for cputime subpage Elliot Smith 2016-02-23 14:29 ` [PATCH 0/3][v4] Fix task buildstats gathering Barros Pena, Belen 3 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-23 12:58 UTC (permalink / raw) To: toaster The data available from buildstats is now more fine grained than previously, so take advantage of that to enrich the data we save against tasks: * Store the CPU usage for user and system separately, and display them separately. * Disk IO is now measured in bytes, not ms. Also store the read/write bytes separately. * Store started and ended times, as well as elapsed_time. This will enable future features such as showing which tasks were running at a particular point in the build. There was also a problem with how we were looking up the Task object, which meant that the buildstats were being added to new tasks which weren't correctly associated with the build. Fix how we look up the Task (only looking for tasks which match the build, and the task and recipe names in the build stats data) so the build stats are associated with the correct task. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------ .../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++ bitbake/lib/toaster/orm/models.py | 18 +++++- .../toastergui/templates/basebuildpage.html | 4 +- .../lib/toaster/toastergui/templates/tasks.html | 20 +++--- bitbake/lib/toaster/toastergui/urls.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------ 7 files changed, 148 insertions(+), 71 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py diff --git a/bitbake/lib/bb/ui/buildinfohelper.py b/bitbake/lib/bb/ui/buildinfohelper.py index 81abede..93db067 100644 --- a/bitbake/lib/bb/ui/buildinfohelper.py +++ b/bitbake/lib/bb/ui/buildinfohelper.py @@ -50,6 +50,7 @@ from bb.msg import BBLogFormatter as formatter from django.db import models from pprint import pformat import logging +from datetime import datetime, timedelta from django.db import transaction, connection @@ -120,6 +121,12 @@ class ORMWrapper(object): return vars(self)[dictname][key] + def _timestamp_to_datetime(self, ms): + """ + Convert timestamp in milliseconds ms to Python datetime + """ + return datetime(1970, 1, 1) + timedelta(milliseconds=ms) + # pylint: disable=no-self-use # we disable detection of no self use in functions because the methods actually work on the object # even if they don't touch self anywhere @@ -223,6 +230,28 @@ class ORMWrapper(object): target.license_manifest_path = license_manifest_path target.save() + def update_task_object(self, build, task_name, recipe_name, task_stats): + """ + Find the task for build which matches the recipe and task name + to be stored + """ + task_to_update = Task.objects.get( + build = build, + task_name = task_name, + recipe__name = recipe_name + ) + + task_to_update.started = self._timestamp_to_datetime(task_stats['started']) + task_to_update.ended = self._timestamp_to_datetime(task_stats['ended']) + task_to_update.elapsed_time = (task_stats['ended'] - task_stats['started']) / 1000 + task_to_update.cpu_time_user = task_stats['cpu_time_user'] + task_to_update.cpu_time_system = task_stats['cpu_time_system'] + task_to_update.disk_io_read = task_stats['disk_io_read'] + task_to_update.disk_io_write = task_stats['disk_io_write'] + task_to_update.disk_io = task_stats['disk_io_read'] + task_stats['disk_io_write'] + + task_to_update.save() + def get_update_task_object(self, task_information, must_exist = False): assert 'build' in task_information assert 'recipe' in task_information @@ -259,14 +288,6 @@ class ORMWrapper(object): task_object.sstate_result = Task.SSTATE_FAILED object_changed = True - # mark down duration if we have a start time and a current time - if 'start_time' in task_information.keys() and 'end_time' in task_information.keys(): - duration = task_information['end_time'] - task_information['start_time'] - task_object.elapsed_time = duration - object_changed = True - del task_information['start_time'] - del task_information['end_time'] - if object_changed: task_object.save() return task_object @@ -1091,31 +1112,11 @@ class BuildInfoHelper(object): def store_tasks_stats(self, event): - for (taskfile, taskname, taskstats, recipename) in BuildInfoHelper._get_data_from_event(event): - localfilepath = taskfile.split(":")[-1] - assert localfilepath.startswith("/") + task_data = BuildInfoHelper._get_data_from_event(event) - recipe_information = self._get_recipe_information_from_taskfile(taskfile) - try: - if recipe_information['file_path'].startswith(recipe_information['layer_version'].local_path): - recipe_information['file_path'] = recipe_information['file_path'][len(recipe_information['layer_version'].local_path):].lstrip("/") - - recipe_object = Recipe.objects.get(layer_version = recipe_information['layer_version'], - file_path__endswith = recipe_information['file_path'], - name = recipename) - except Recipe.DoesNotExist: - logger.error("Could not find recipe for recipe_information %s name %s" , pformat(recipe_information), recipename) - raise - - task_information = {} - task_information['build'] = self.internal_state['build'] - task_information['recipe'] = recipe_object - task_information['task_name'] = taskname - task_information['cpu_usage'] = taskstats['cpu_usage'] - task_information['disk_io'] = taskstats['disk_io'] - if 'elapsed_time' in taskstats: - task_information['elapsed_time'] = taskstats['elapsed_time'] - self.orm_wrapper.get_update_task_object(task_information) + for (task_file, task_name, task_stats, recipe_name) in task_data: + build = self.internal_state['build'] + self.orm_wrapper.update_task_object(build, task_name, recipe_name, task_stats) def update_and_store_task(self, event): assert 'taskfile' in vars(event) @@ -1137,13 +1138,6 @@ class BuildInfoHelper(object): recipe = self.orm_wrapper.get_update_recipe_object(recipe_information, True) task_information = self._get_task_information(event,recipe) - if 'time' in vars(event): - if not 'start_time' in self.internal_state['taskdata'][identifier]: - self.internal_state['taskdata'][identifier]['start_time'] = event.time - else: - task_information['end_time'] = event.time - task_information['start_time'] = self.internal_state['taskdata'][identifier]['start_time'] - task_information['outcome'] = self.internal_state['taskdata'][identifier]['outcome'] if 'logfile' in vars(event): diff --git a/bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py b/bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py new file mode 100644 index 0000000..fb1196b --- /dev/null +++ b/bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py @@ -0,0 +1,48 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('orm', '0004_provides'), + ] + + operations = [ + migrations.RemoveField( + model_name='task', + name='cpu_usage', + ), + migrations.AddField( + model_name='task', + name='cpu_time_system', + field=models.DecimalField(null=True, max_digits=8, decimal_places=2), + ), + migrations.AddField( + model_name='task', + name='cpu_time_user', + field=models.DecimalField(null=True, max_digits=8, decimal_places=2), + ), + migrations.AddField( + model_name='task', + name='disk_io_read', + field=models.IntegerField(null=True), + ), + migrations.AddField( + model_name='task', + name='disk_io_write', + field=models.IntegerField(null=True), + ), + migrations.AddField( + model_name='task', + name='ended', + field=models.DateTimeField(null=True), + ), + migrations.AddField( + model_name='task', + name='started', + field=models.DateTimeField(null=True), + ), + ] diff --git a/bitbake/lib/toaster/orm/models.py b/bitbake/lib/toaster/orm/models.py index ab6940f..75218bb 100644 --- a/bitbake/lib/toaster/orm/models.py +++ b/bitbake/lib/toaster/orm/models.py @@ -712,9 +712,23 @@ class Task(models.Model): work_directory = models.FilePathField(max_length=255, blank=True) script_type = models.IntegerField(choices=TASK_CODING, default=CODING_NA) line_number = models.IntegerField(default=0) - disk_io = models.IntegerField(null=True) - cpu_usage = models.DecimalField(max_digits=8, decimal_places=2, null=True) + + # start/end times + started = models.DateTimeField(null=True) + ended = models.DateTimeField(null=True) + + # in seconds; this is stored to enable sorting elapsed_time = models.DecimalField(max_digits=8, decimal_places=2, null=True) + + # in bytes; note that disk_io is stored to enable sorting + disk_io = models.IntegerField(null=True) + disk_io_read = models.IntegerField(null=True) + disk_io_write = models.IntegerField(null=True) + + # in seconds + cpu_time_user = models.DecimalField(max_digits=8, decimal_places=2, null=True) + cpu_time_system = models.DecimalField(max_digits=8, decimal_places=2, null=True) + sstate_result = models.IntegerField(choices=SSTATE_RESULT, default=SSTATE_NA) message = models.CharField(max_length=240) logfile = models.FilePathField(max_length=255, blank=True) diff --git a/bitbake/lib/toaster/toastergui/templates/basebuildpage.html b/bitbake/lib/toaster/toastergui/templates/basebuildpage.html index 3571869..7ebdef9 100644 --- a/bitbake/lib/toaster/toastergui/templates/basebuildpage.html +++ b/bitbake/lib/toaster/toastergui/templates/basebuildpage.html @@ -64,8 +64,8 @@ {% block nav-buildtime %} <li><a href="{% url 'buildtime' build.pk %}">Time</a></li> {% endblock %} - {% block nav-cpuusage %} - <li><a href="{% url 'cpuusage' build.pk %}">CPU usage</a></li> + {% block nav-cputime %} + <li><a href="{% url 'cputime' build.pk %}">CPU time</a></li> {% endblock %} {% block nav-diskio %} <li><a href="{% url 'diskio' build.pk %}">Disk I/O</a></li> diff --git a/bitbake/lib/toaster/toastergui/templates/tasks.html b/bitbake/lib/toaster/toastergui/templates/tasks.html index 353410f..23eb957 100644 --- a/bitbake/lib/toaster/toastergui/templates/tasks.html +++ b/bitbake/lib/toaster/toastergui/templates/tasks.html @@ -1,4 +1,5 @@ {% extends "basebuildpage.html" %} +{% load humanize %} {% load projecttags %} {% block title %} {{mainheading}} - {{build.target_set.all|dictsort:"target"|join:", "}} {{build.machine}} - {{build.project.name}} - Toaster{% endblock %} @@ -20,13 +21,15 @@ <li><a href="{% url 'buildtime' build.pk %}">Time</a></li> {% endif %} {% endblock %} -{% block nav-cpuusage %} - {% if 'CPU usage' == mainheading %} - <li class="active"><a href="{% url 'cpuusage' build.pk %}">CPU usage</a></li> + +{% block nav-cputime %} + {% if 'CPU time' == mainheading %} + <li class="active"><a href="{% url 'cputime' build.pk %}">CPU time</a></li> {% else %} - <li><a href="{% url 'cpuusage' build.pk %}">CPU usage</a></li> + <li><a href="{% url 'cputime' build.pk %}">CPU time</a></li> {% endif %} {% endblock %} + {% block nav-diskio %} {% if 'Disk I/O' == mainheading %} <li class="active"><a href="{% url 'diskio' build.pk %}">Disk I/O</a></li> @@ -107,11 +110,14 @@ <td class="time_taken"> {{task.elapsed_time|format_none_and_zero|floatformat:2}} </td> - <td class="cpu_used"> - {{task.cpu_usage|format_none_and_zero|floatformat:2}}{% if task.cpu_usage %}%{% endif %} + <td class="cpu_time_system"> + {{task.cpu_time_system|format_none_and_zero|floatformat:2}} + </td> + <td class="cpu_time_user"> + {{task.cpu_time_user|format_none_and_zero|floatformat:2}} </td> <td class="disk_io"> - {{task.disk_io|format_none_and_zero}} + {{task.disk_io|format_none_and_zero|intcomma}} </td> </tr> diff --git a/bitbake/lib/toaster/toastergui/urls.py b/bitbake/lib/toaster/toastergui/urls.py index 4aa6488..400580a 100644 --- a/bitbake/lib/toaster/toastergui/urls.py +++ b/bitbake/lib/toaster/toastergui/urls.py @@ -64,7 +64,7 @@ urlpatterns = patterns('toastergui.views', url(r'^build/(?P<build_id>\d+)/configuration$', 'configuration', name='configuration'), url(r'^build/(?P<build_id>\d+)/configvars$', 'configvars', name='configvars'), url(r'^build/(?P<build_id>\d+)/buildtime$', 'buildtime', name='buildtime'), - url(r'^build/(?P<build_id>\d+)/cpuusage$', 'cpuusage', name='cpuusage'), + url(r'^build/(?P<build_id>\d+)/cputime$', 'cputime', name='cputime'), url(r'^build/(?P<build_id>\d+)/diskio$', 'diskio', name='diskio'), # image information dir diff --git a/bitbake/lib/toaster/toastergui/views.py b/bitbake/lib/toaster/toastergui/views.py index 13489af..0392ac4 100755 --- a/bitbake/lib/toaster/toastergui/views.py +++ b/bitbake/lib/toaster/toastergui/views.py @@ -1000,11 +1000,11 @@ def tasks_common(request, build_id, variant, task_anchor): object_search_display="disk I/O data" filter_search_display="tasks" (pagesize, orderby) = _get_parameters_values(request, 25, 'disk_io:-') - elif 'cpuusage' == variant: - title_variant='CPU usage' - object_search_display="CPU usage data" + elif 'cputime' == variant: + title_variant='CPU time' + object_search_display="CPU time data" filter_search_display="tasks" - (pagesize, orderby) = _get_parameters_values(request, 25, 'cpu_usage:-') + (pagesize, orderby) = _get_parameters_values(request, 25, 'cpu_time_system:-') else : title_variant='Tasks' object_search_display="tasks" @@ -1156,23 +1156,38 @@ def tasks_common(request, build_id, variant, task_anchor): del tc_time['clclass'] tc_cache['hidden']='1' - tc_cpu={ - 'name':'CPU usage', - 'qhelp':'The percentage of task CPU utilization', - 'orderfield': _get_toggle_order(request, "cpu_usage", True), - 'ordericon':_get_toggle_order_icon(request, "cpu_usage"), - 'orderkey' : 'cpu_usage', - 'clclass': 'cpu_used', 'hidden' : 1, + tc_cpu_time_system={ + 'name':'System CPU time', + 'qhelp':'Total amount of time spent executing in kernel mode, in \ + seconds. Note that this time can be higher than the task \ + time due to parallel execution.', + 'orderfield': _get_toggle_order(request, "cpu_time_system", True), + 'ordericon':_get_toggle_order_icon(request, "cpu_time_system"), + 'orderkey' : 'cpu_time_system', + 'clclass': 'cpu_time_system', 'hidden' : 1, } - if 'cpuusage' == variant: - tc_cpu['hidden']='0' - del tc_cpu['clclass'] + tc_cpu_time_user={ + 'name':'User CPU time', + 'qhelp':'Total amount of time spent executing in user mode, in seconds.\ + Note that this time can be higher than the task time due to \ + parallel execution.', + 'orderfield': _get_toggle_order(request, "cpu_time_user", True), + 'ordericon':_get_toggle_order_icon(request, "cpu_time_user"), + 'orderkey' : 'cpu_time_user', + 'clclass': 'cpu_time_user', 'hidden' : 1, + } + + if 'cputime' == variant: + tc_cpu_time_system['hidden']='0' + tc_cpu_time_user['hidden']='0' + del tc_cpu_time_system['clclass'] + del tc_cpu_time_user['clclass'] tc_cache['hidden']='1' tc_diskio={ - 'name':'Disk I/O (ms)', - 'qhelp':'Number of miliseconds the task spent doing disk input and output', + 'name':'Disk I/O (bytes)', + 'qhelp':'Number of bytes written to and read from the disk during the task', 'orderfield': _get_toggle_order(request, "disk_io", True), 'ordericon':_get_toggle_order_icon(request, "disk_io"), 'orderkey' : 'disk_io', @@ -1203,7 +1218,8 @@ def tasks_common(request, build_id, variant, task_anchor): tc_outcome, tc_cache, tc_time, - tc_cpu, + tc_cpu_time_system, + tc_cpu_time_user, tc_diskio, ]} @@ -1224,9 +1240,8 @@ def buildtime(request, build_id): def diskio(request, build_id): return tasks_common(request, build_id, 'diskio', '') -def cpuusage(request, build_id): - return tasks_common(request, build_id, 'cpuusage', '') - +def cputime(request, build_id): + return tasks_common(request, build_id, 'cputime', '') def recipes(request, build_id): template = 'recipes.html' -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/3] toaster-tests: add test for cputime subpage 2016-02-23 12:58 [PATCH 0/3][v4] Fix task buildstats gathering Elliot Smith 2016-02-23 12:58 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 2016-02-23 12:58 ` [PATCH 2/3] toaster: rework task buildstats storage and display Elliot Smith @ 2016-02-23 12:58 ` Elliot Smith 2016-02-23 14:29 ` [PATCH 0/3][v4] Fix task buildstats gathering Barros Pena, Belen 3 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-23 12:58 UTC (permalink / raw) To: toaster Add a basic test which checks that the CPU time subpage for the build dashboard shows both the User and System CPU time columns by default. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- bitbake/lib/toaster/toastergui/tests.py | 45 ++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/bitbake/lib/toaster/toastergui/tests.py b/bitbake/lib/toaster/toastergui/tests.py index 4e420ea..978ec44 100644 --- a/bitbake/lib/toaster/toastergui/tests.py +++ b/bitbake/lib/toaster/toastergui/tests.py @@ -29,7 +29,7 @@ from django.utils import timezone from orm.models import Project, Release, BitbakeVersion, Package, LogMessage from orm.models import ReleaseLayerSourcePriority, LayerSource, Layer, Build from orm.models import Layer_Version, Recipe, Machine, ProjectLayer, Target -from orm.models import CustomImageRecipe, ProjectVariable +from orm.models import CustomImageRecipe, ProjectVariable, Task from orm.models import Branch, CustomImagePackage import toastermain @@ -1054,6 +1054,16 @@ class BuildDashboardTests(TestCase): started_on=now, completed_on=now) + # target for build, so breadcrumb can display + Target.objects.create(build=self.build1, target='bash', task='build') + + # layer_version, recipe and task for build, so CPU time data can display + layer = Layer.objects.create() + layer_version = Layer_Version.objects.create(layer=layer) + recipe = Recipe.objects.create(name='zlib', layer_version=layer_version) + Task.objects.create(build=self.build1, recipe=recipe, order=1, + outcome=Task.OUTCOME_SUCCESS) + # exception msg1 = 'an exception was thrown' self.exception_message = LogMessage.objects.create( @@ -1123,3 +1133,36 @@ class BuildDashboardTests(TestCase): section of the page """ self._check_for_log_message(self.critical_message) + + def test_cputime(self): + """ + Check that the system and user CPU time columns are displayed + when the cputime subpage is shown + """ + url = reverse('cputime', args=(self.build1.id,)) + response = self.client.get(url, follow=True) + soup = BeautifulSoup(response.content) + + # check nav item is highlighted + elements = soup.select('#nav li.active') + self.assertEquals(len(elements), 1, + 'should be one active nav element, ' + + 'but found %s' % len(elements)) + link_text = elements[0].find('a').text.strip() + self.assertEquals(link_text, 'CPU time', + 'active nav element should have text "CPU time", ' + + 'but text was %s' % link_text) + + # check page heading + heading = soup.select('.page-header h1')[0] + self.assertEquals(heading.text.strip(), 'CPU time') + + # check CPU column headings are both present + cpu_time_headings = ['System CPU time', 'User CPU time'] + table_heading_links = soup.select('#otable th a') + for link in table_heading_links: + if link.text.strip() in cpu_time_headings: + cpu_time_headings.remove(link.text) + self.assertEquals(len(cpu_time_headings), 0, + 'Both CPU time headings (user, system) ' + + 'must be present; missing %s' % cpu_time_headings) -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 0/3][v4] Fix task buildstats gathering 2016-02-23 12:58 [PATCH 0/3][v4] Fix task buildstats gathering Elliot Smith ` (2 preceding siblings ...) 2016-02-23 12:58 ` [PATCH 3/3] toaster-tests: add test for cputime subpage Elliot Smith @ 2016-02-23 14:29 ` Barros Pena, Belen 2016-02-23 17:55 ` Smith, Elliot 3 siblings, 1 reply; 12+ messages in thread From: Barros Pena, Belen @ 2016-02-23 14:29 UTC (permalink / raw) To: Smith, Elliot, toaster On 23/02/2016 13:58, "toaster-bounces@yoctoproject.org on behalf of Elliot Smith" <toaster-bounces@yoctoproject.org on behalf of elliot.smith@intel.com> wrote: >Changes since 961893fa05f4d51187f4d5daadacd7927d1a0847 (toaster-next) are >in >git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 >http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/bu >ild_data-8842 Yes, this is good. Thank you! Belén ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/3][v4] Fix task buildstats gathering 2016-02-23 14:29 ` [PATCH 0/3][v4] Fix task buildstats gathering Barros Pena, Belen @ 2016-02-23 17:55 ` Smith, Elliot 0 siblings, 0 replies; 12+ messages in thread From: Smith, Elliot @ 2016-02-23 17:55 UTC (permalink / raw) To: Barros Pena, Belen; +Cc: toaster [-- Attachment #1: Type: text/plain, Size: 737 bytes --] I need to do a new version of this; please ignore for now. Elliot On 23 February 2016 at 14:29, Barros Pena, Belen < belen.barros.pena@intel.com> wrote: > > > On 23/02/2016 13:58, "toaster-bounces@yoctoproject.org on behalf of Elliot > Smith" <toaster-bounces@yoctoproject.org on behalf of > elliot.smith@intel.com> wrote: > > >Changes since 961893fa05f4d51187f4d5daadacd7927d1a0847 (toaster-next) are > >in > >git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 > > > http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/bu > >ild_data-8842 > > Yes, this is good. Thank you! > > Belén > > -- Elliot Smith Software Engineer Intel Open Source Technology Centre [-- Attachment #2: Type: text/html, Size: 1579 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 0/3][v7] Fix task buildstats gathering @ 2016-03-07 12:21 Elliot Smith 2016-03-07 12:21 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 0 siblings, 1 reply; 12+ messages in thread From: Elliot Smith @ 2016-03-07 12:21 UTC (permalink / raw) To: toaster v7: Changes by others necessitated a rebase. v6: Fix comments on function which converts started and ended time to a datetime (values are seconds, not milliseconds). Modify the task page: - Show the split CPU times (User and System). - Format Disk IO as done on the tasks page (with commas). - Disk IO value is now in bytes, so modify heading. v5: Fixed an error where the started and ended times for a task were treated as being in milliseconds, when they are in seconds. v4: I messed up the patch submission, so I'm sending the correct patches this time. v3: Only show a single navigation item in the left-hand nav panel for CPU times, but show both System and User time columns when it is selected. Add a test for CPU time subpage. v2: Store "started" and "ended" times for each task. (This will enable other improvements later, such as showing when tasks overlap each other.) Compute elapsed_time from start and end times, rather than in toaster.bbclass. Store disk IO in two extra fields, one for read and the other for write. Compute total disk_io from these two values. (elapsed_time and disk_io are still stored, as these are used for sorting and we can't easily sort on a computed field. This can be amended when we get round to putting the tasks table into ToasterTable; see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.) Modify the UI to show the CPU times split between system and user. v1: Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway. Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper. To test: 1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe). 2. Run a build against a local Yocto (as one patch is for toaster.bbclass). 3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842. Changes since c5194851573c91c0aab8fe4c549086037daf2370 (toaster-next) are in git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842 Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842 Elliot Smith (3): toaster.bbclass: improve how we gather buildstats for Toaster toaster: rework task buildstats storage and display toaster-tests: add test for cputime subpage bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------ .../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++ bitbake/lib/toaster/orm/models.py | 18 +++++- .../toastergui/templates/basebuildpage.html | 4 +- bitbake/lib/toaster/toastergui/templates/task.html | 23 ++++--- .../lib/toaster/toastergui/templates/tasks.html | 20 +++--- bitbake/lib/toaster/toastergui/tests.py | 45 +++++++++++++- bitbake/lib/toaster/toastergui/urls.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------ meta/classes/toaster.bbclass | 70 +++++++++++++-------- 10 files changed, 252 insertions(+), 105 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-03-07 12:21 [PATCH 0/3][v7] " Elliot Smith @ 2016-03-07 12:21 ` Elliot Smith 2016-03-07 18:39 ` Michael Wood 0 siblings, 1 reply; 12+ messages in thread From: Elliot Smith @ 2016-03-07 12:21 UTC (permalink / raw) To: toaster Clean up the code which gathers buildstats for Toaster, and modify the field names so that the correct parts of the buildstats files are used to derive the CPU usage values. Also derive elapsed time for the build here, rather than in Toaster, as we have ready access to the data in the correct format. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass index 51a4c74..b7e8773 100644 --- a/meta/classes/toaster.bbclass +++ b/meta/classes/toaster.bbclass @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { import bb.utils import os + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") + if not e.data.getVar('BUILDSTATS_BASE', True): return # if we don't have buildstats, we cannot collect stats + def stat_to_float(value): + return float(value.strip('% \n\r')) + def _append_read_list(v): lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: + with open(toaster_statlist_file, "a") as fout: taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) bb.utils.unlockfile(lock) def _read_stats(filename): - cpu_usage = 0 - disk_io = 0 - started = '0' - ended = '0' - pn = '' + # seconds + cpu_time_user = 0 + cpu_time_system = 0 + + # bytes + disk_io_read = 0 + disk_io_write = 0 + + started = 0 + ended = 0 + taskname = '' + statinfo = {} with open(filename, 'r') as task_bs: @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { k,v = line.strip().split(": ", 1) statinfo[k] = v - if "CPU usage" in statinfo: - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') - - if "IO write_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) - - if "IO read_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) - if "Started" in statinfo: - started = str(statinfo["Started"]).strip('% \n\r') + started = stat_to_float(statinfo["Started"]) if "Ended" in statinfo: - ended = str(statinfo["Ended"]).strip('% \n\r') + ended = stat_to_float(statinfo["Ended"]) - elapsed_time = float(ended) - float(started) + if "Child rusage ru_utime" in statinfo: + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) - cpu_usage = float(cpu_usage) + if "Child rusage ru_stime" in statinfo: + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} + if "IO write_bytes" in statinfo: + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) + disk_io_write = disk_io_write + write_bytes + if "IO read_bytes" in statinfo: + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) + disk_io_read = disk_io_read + read_bytes + + return { + 'stat_file': filename, + 'cpu_time_user': cpu_time_user, + 'cpu_time_system': cpu_time_system, + 'disk_io_read': disk_io_read, + 'disk_io_write': disk_io_write, + 'started': started, + 'ended': ended + } if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): _append_read_list(e) pass - - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): events = [] - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: + with open(toaster_statlist_file, "r") as fin: for line in fin: (taskfile, taskname, filename, recipename) = line.strip().split("::") - events.append((taskfile, taskname, _read_stats(filename), recipename)) + stats = _read_stats(filename) + events.append((taskfile, taskname, stats, recipename)) bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) + #os.unlink(toaster_statlist_file) } # dump relevant build history data as an event when the build is completed -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-03-07 12:21 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith @ 2016-03-07 18:39 ` Michael Wood 2016-03-07 19:26 ` Smith, Elliot 0 siblings, 1 reply; 12+ messages in thread From: Michael Wood @ 2016-03-07 18:39 UTC (permalink / raw) To: toaster On 07/03/16 12:21, Elliot Smith wrote: > Clean up the code which gathers buildstats for Toaster, > and modify the field names so that the correct parts of the > buildstats files are used to derive the CPU usage values. > > Also derive elapsed time for the build here, rather than in > Toaster, as we have ready access to the data in the correct > format. > > [YOCTO #8842] > > Signed-off-by: Elliot Smith <elliot.smith@intel.com> > --- > meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- > 1 file changed, 45 insertions(+), 25 deletions(-) > > diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass > index 51a4c74..b7e8773 100644 > --- a/meta/classes/toaster.bbclass > +++ b/meta/classes/toaster.bbclass > @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { > import bb.utils > import os > > + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") > + > if not e.data.getVar('BUILDSTATS_BASE', True): > return # if we don't have buildstats, we cannot collect stats > > + def stat_to_float(value): > + return float(value.strip('% \n\r')) > + > def _append_read_list(v): > lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) > > - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: > + with open(toaster_statlist_file, "a") as fout: > taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") > fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) > > bb.utils.unlockfile(lock) > > def _read_stats(filename): > - cpu_usage = 0 > - disk_io = 0 > - started = '0' > - ended = '0' > - pn = '' > + # seconds > + cpu_time_user = 0 > + cpu_time_system = 0 > + > + # bytes > + disk_io_read = 0 > + disk_io_write = 0 > + > + started = 0 > + ended = 0 > + > taskname = '' > + > statinfo = {} > > with open(filename, 'r') as task_bs: > @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { > k,v = line.strip().split(": ", 1) > statinfo[k] = v > > - if "CPU usage" in statinfo: > - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') > - > - if "IO write_bytes" in statinfo: > - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) > - > - if "IO read_bytes" in statinfo: > - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) > - > if "Started" in statinfo: > - started = str(statinfo["Started"]).strip('% \n\r') > + started = stat_to_float(statinfo["Started"]) > > if "Ended" in statinfo: > - ended = str(statinfo["Ended"]).strip('% \n\r') > + ended = stat_to_float(statinfo["Ended"]) > > - elapsed_time = float(ended) - float(started) > + if "Child rusage ru_utime" in statinfo: > + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) > > - cpu_usage = float(cpu_usage) > + if "Child rusage ru_stime" in statinfo: > + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) > > - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} > + if "IO write_bytes" in statinfo: > + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) > + disk_io_write = disk_io_write + write_bytes > > + if "IO read_bytes" in statinfo: > + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) > + disk_io_read = disk_io_read + read_bytes > + > + return { > + 'stat_file': filename, > + 'cpu_time_user': cpu_time_user, > + 'cpu_time_system': cpu_time_system, > + 'disk_io_read': disk_io_read, > + 'disk_io_write': disk_io_write, > + 'started': started, > + 'ended': ended > + } > > if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): > _append_read_list(e) > pass > > - > - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): > + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): > events = [] > - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: > + with open(toaster_statlist_file, "r") as fin: > for line in fin: > (taskfile, taskname, filename, recipename) = line.strip().split("::") > - events.append((taskfile, taskname, _read_stats(filename), recipename)) > + stats = _read_stats(filename) > + events.append((taskfile, taskname, stats, recipename)) > bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) > - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) > + #os.unlink(toaster_statlist_file) Did you mean to leave that 'unlink' commented out? > } > > # dump relevant build history data as an event when the build is completed --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-03-07 18:39 ` Michael Wood @ 2016-03-07 19:26 ` Smith, Elliot 0 siblings, 0 replies; 12+ messages in thread From: Smith, Elliot @ 2016-03-07 19:26 UTC (permalink / raw) To: Michael Wood; +Cc: toaster [-- Attachment #1: Type: text/plain, Size: 6672 bytes --] No, I didn't, I'll fix it. Elliot On 7 March 2016 at 18:39, Michael Wood <michael.g.wood@intel.com> wrote: > On 07/03/16 12:21, Elliot Smith wrote: > >> Clean up the code which gathers buildstats for Toaster, >> and modify the field names so that the correct parts of the >> buildstats files are used to derive the CPU usage values. >> >> Also derive elapsed time for the build here, rather than in >> Toaster, as we have ready access to the data in the correct >> format. >> >> [YOCTO #8842] >> >> Signed-off-by: Elliot Smith <elliot.smith@intel.com> >> --- >> meta/classes/toaster.bbclass | 70 >> ++++++++++++++++++++++++++++---------------- >> 1 file changed, 45 insertions(+), 25 deletions(-) >> >> diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass >> index 51a4c74..b7e8773 100644 >> --- a/meta/classes/toaster.bbclass >> +++ b/meta/classes/toaster.bbclass >> @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { >> import bb.utils >> import os >> + toaster_statlist_file = >> os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") >> + >> if not e.data.getVar('BUILDSTATS_BASE', True): >> return # if we don't have buildstats, we cannot collect stats >> + def stat_to_float(value): >> + return float(value.strip('% \n\r')) >> + >> def _append_read_list(v): >> lock = >> bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) >> - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), >> "toasterstatlist"), "a") as fout: >> + with open(toaster_statlist_file, "a") as fout: >> taskdir = >> e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") >> fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, >> os.path.join(taskdir, e.task), e.data.expand("${PN}"))) >> bb.utils.unlockfile(lock) >> def _read_stats(filename): >> - cpu_usage = 0 >> - disk_io = 0 >> - started = '0' >> - ended = '0' >> - pn = '' >> + # seconds >> + cpu_time_user = 0 >> + cpu_time_system = 0 >> + >> + # bytes >> + disk_io_read = 0 >> + disk_io_write = 0 >> + >> + started = 0 >> + ended = 0 >> + >> taskname = '' >> + >> statinfo = {} >> with open(filename, 'r') as task_bs: >> @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { >> k,v = line.strip().split(": ", 1) >> statinfo[k] = v >> - if "CPU usage" in statinfo: >> - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') >> - >> - if "IO write_bytes" in statinfo: >> - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% >> \n\r')) >> - >> - if "IO read_bytes" in statinfo: >> - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% >> \n\r')) >> - >> if "Started" in statinfo: >> - started = str(statinfo["Started"]).strip('% \n\r') >> + started = stat_to_float(statinfo["Started"]) >> if "Ended" in statinfo: >> - ended = str(statinfo["Ended"]).strip('% \n\r') >> + ended = stat_to_float(statinfo["Ended"]) >> - elapsed_time = float(ended) - float(started) >> + if "Child rusage ru_utime" in statinfo: >> + cpu_time_user = cpu_time_user + >> stat_to_float(statinfo["Child rusage ru_utime"]) >> - cpu_usage = float(cpu_usage) >> + if "Child rusage ru_stime" in statinfo: >> + cpu_time_system = cpu_time_system + >> stat_to_float(statinfo["Child rusage ru_stime"]) >> - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, >> 'elapsed_time': elapsed_time} >> + if "IO write_bytes" in statinfo: >> + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) >> + disk_io_write = disk_io_write + write_bytes >> + if "IO read_bytes" in statinfo: >> + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) >> + disk_io_read = disk_io_read + read_bytes >> + >> + return { >> + 'stat_file': filename, >> + 'cpu_time_user': cpu_time_user, >> + 'cpu_time_system': cpu_time_system, >> + 'disk_io_read': disk_io_read, >> + 'disk_io_write': disk_io_write, >> + 'started': started, >> + 'ended': ended >> + } >> if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): >> _append_read_list(e) >> pass >> - >> - if isinstance(e, bb.event.BuildCompleted) and >> os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), >> "toasterstatlist")): >> + if isinstance(e, bb.event.BuildCompleted) and >> os.path.exists(toaster_statlist_file): >> events = [] >> - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), >> "toasterstatlist"), "r") as fin: >> + with open(toaster_statlist_file, "r") as fin: >> for line in fin: >> (taskfile, taskname, filename, recipename) = >> line.strip().split("::") >> - events.append((taskfile, taskname, >> _read_stats(filename), recipename)) >> + stats = _read_stats(filename) >> + events.append((taskfile, taskname, stats, recipename)) >> bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), >> e.data) >> - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), >> "toasterstatlist")) >> + #os.unlink(toaster_statlist_file) >> > > Did you mean to leave that 'unlink' commented out? > >> } >> # dump relevant build history data as an event when the build is >> completed >> > > --------------------------------------------------------------------- > Intel Corporation (UK) Limited > Registered No. 1134945 (England) > Registered Office: Pipers Way, Swindon SN3 1RJ > VAT No: 860 2173 47 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > -- > _______________________________________________ > toaster mailing list > toaster@yoctoproject.org > https://lists.yoctoproject.org/listinfo/toaster > -- Elliot Smith Software Engineer Intel Open Source Technology Centre [-- Attachment #2: Type: text/html, Size: 8839 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 0/3][v6] Fix task buildstats gathering @ 2016-02-24 11:15 Elliot Smith 2016-02-24 11:15 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 0 siblings, 1 reply; 12+ messages in thread From: Elliot Smith @ 2016-02-24 11:15 UTC (permalink / raw) To: toaster v6: Fix comments on function which converts started and ended time to a datetime (values are seconds, not milliseconds). Modify the task page: - Show the split CPU times (User and System). - Format Disk IO as done on the tasks page (with commas). - Disk IO value is now in bytes, so modify heading. v5: Fixed an error where the started and ended times for a task were treated as being in milliseconds, when they are in seconds. v4: I messed up the patch submission, so I'm sending the correct patches this time. v3: Only show a single navigation item in the left-hand nav panel for CPU times, but show both System and User time columns when it is selected. Add a test for CPU time subpage. v2: Store "started" and "ended" times for each task. (This will enable other improvements later, such as showing when tasks overlap each other.) Compute elapsed_time from start and end times, rather than in toaster.bbclass. Store disk IO in two extra fields, one for read and the other for write. Compute total disk_io from these two values. (elapsed_time and disk_io are still stored, as these are used for sorting and we can't easily sort on a computed field. This can be amended when we get round to putting the tasks table into ToasterTable; see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.) Modify the UI to show the CPU times split between system and user. v1: Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway. Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper. To test: 1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe). 2. Run a build against a local Yocto (as one patch is for toaster.bbclass). 3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842. Changes since a6d307c(toaster-next) are in git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842 Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842 Elliot Smith (3): toaster.bbclass: improve how we gather buildstats for Toaster toaster: rework task buildstats storage and display toaster-tests: add test for cputime subpage bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------ .../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++ bitbake/lib/toaster/orm/models.py | 18 +++++- .../toastergui/templates/basebuildpage.html | 4 +- bitbake/lib/toaster/toastergui/templates/task.html | 23 ++++--- .../lib/toaster/toastergui/templates/tasks.html | 20 +++--- bitbake/lib/toaster/toastergui/tests.py | 45 +++++++++++++- bitbake/lib/toaster/toastergui/urls.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------ meta/classes/toaster.bbclass | 70 +++++++++++++-------- 10 files changed, 252 insertions(+), 105 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-02-24 11:15 [PATCH 0/3][v6] Fix task buildstats gathering Elliot Smith @ 2016-02-24 11:15 ` Elliot Smith 0 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-24 11:15 UTC (permalink / raw) To: toaster Clean up the code which gathers buildstats for Toaster, and modify the field names so that the correct parts of the buildstats files are used to derive the CPU usage values. Also derive elapsed time for the build here, rather than in Toaster, as we have ready access to the data in the correct format. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass index 51a4c74..b7e8773 100644 --- a/meta/classes/toaster.bbclass +++ b/meta/classes/toaster.bbclass @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { import bb.utils import os + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") + if not e.data.getVar('BUILDSTATS_BASE', True): return # if we don't have buildstats, we cannot collect stats + def stat_to_float(value): + return float(value.strip('% \n\r')) + def _append_read_list(v): lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: + with open(toaster_statlist_file, "a") as fout: taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) bb.utils.unlockfile(lock) def _read_stats(filename): - cpu_usage = 0 - disk_io = 0 - started = '0' - ended = '0' - pn = '' + # seconds + cpu_time_user = 0 + cpu_time_system = 0 + + # bytes + disk_io_read = 0 + disk_io_write = 0 + + started = 0 + ended = 0 + taskname = '' + statinfo = {} with open(filename, 'r') as task_bs: @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { k,v = line.strip().split(": ", 1) statinfo[k] = v - if "CPU usage" in statinfo: - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') - - if "IO write_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) - - if "IO read_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) - if "Started" in statinfo: - started = str(statinfo["Started"]).strip('% \n\r') + started = stat_to_float(statinfo["Started"]) if "Ended" in statinfo: - ended = str(statinfo["Ended"]).strip('% \n\r') + ended = stat_to_float(statinfo["Ended"]) - elapsed_time = float(ended) - float(started) + if "Child rusage ru_utime" in statinfo: + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) - cpu_usage = float(cpu_usage) + if "Child rusage ru_stime" in statinfo: + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} + if "IO write_bytes" in statinfo: + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) + disk_io_write = disk_io_write + write_bytes + if "IO read_bytes" in statinfo: + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) + disk_io_read = disk_io_read + read_bytes + + return { + 'stat_file': filename, + 'cpu_time_user': cpu_time_user, + 'cpu_time_system': cpu_time_system, + 'disk_io_read': disk_io_read, + 'disk_io_write': disk_io_write, + 'started': started, + 'ended': ended + } if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): _append_read_list(e) pass - - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): events = [] - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: + with open(toaster_statlist_file, "r") as fin: for line in fin: (taskfile, taskname, filename, recipename) = line.strip().split("::") - events.append((taskfile, taskname, _read_stats(filename), recipename)) + stats = _read_stats(filename) + events.append((taskfile, taskname, stats, recipename)) bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) + #os.unlink(toaster_statlist_file) } # dump relevant build history data as an event when the build is completed -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 0/3][v5] Fix task buildstats gathering @ 2016-02-23 18:17 Elliot Smith 2016-02-23 18:17 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 0 siblings, 1 reply; 12+ messages in thread From: Elliot Smith @ 2016-02-23 18:17 UTC (permalink / raw) To: toaster v5: Fixed an error where the started and ended times for a task were treated as being in milliseconds, when they are in seconds. v4: I messed up the patch submission, so I'm sending the correct patches this time. v3: Only show a single navigation item in the left-hand nav panel for CPU times, but show both System and User time columns when it is selected. Add a test for CPU time subpage. v2: Store "started" and "ended" times for each task. (This will enable other improvements later, such as showing when tasks overlap each other.) Compute elapsed_time from start and end times, rather than in toaster.bbclass. Store disk IO in two extra fields, one for read and the other for write. Compute total disk_io from these two values. (elapsed_time and disk_io are still stored, as these are used for sorting and we can't easily sort on a computed field. This can be amended when we get round to putting the tasks table into ToasterTable; see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.) Modify the UI to show the CPU times split between system and user. v1: Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway. Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper. To test: 1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe). 2. Run a build against a local Yocto (as one patch is for toaster.bbclass). 3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842. Changes since a6d307c (toaster-next) are in git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842 Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842 Elliot Smith (3): toaster.bbclass: improve how we gather buildstats for Toaster toaster: rework task buildstats storage and display toaster-tests: add test for cputime subpage bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------ .../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++ bitbake/lib/toaster/orm/models.py | 18 +++++- .../toastergui/templates/basebuildpage.html | 4 +- .../lib/toaster/toastergui/templates/tasks.html | 20 +++--- bitbake/lib/toaster/toastergui/tests.py | 45 +++++++++++++- bitbake/lib/toaster/toastergui/urls.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------ meta/classes/toaster.bbclass | 70 +++++++++++++-------- 9 files changed, 237 insertions(+), 97 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-02-23 18:17 [PATCH 0/3][v5] Fix task buildstats gathering Elliot Smith @ 2016-02-23 18:17 ` Elliot Smith 0 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-23 18:17 UTC (permalink / raw) To: toaster Clean up the code which gathers buildstats for Toaster, and modify the field names so that the correct parts of the buildstats files are used to derive the CPU usage values. Also derive elapsed time for the build here, rather than in Toaster, as we have ready access to the data in the correct format. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass index 51a4c74..b7e8773 100644 --- a/meta/classes/toaster.bbclass +++ b/meta/classes/toaster.bbclass @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { import bb.utils import os + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") + if not e.data.getVar('BUILDSTATS_BASE', True): return # if we don't have buildstats, we cannot collect stats + def stat_to_float(value): + return float(value.strip('% \n\r')) + def _append_read_list(v): lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: + with open(toaster_statlist_file, "a") as fout: taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) bb.utils.unlockfile(lock) def _read_stats(filename): - cpu_usage = 0 - disk_io = 0 - started = '0' - ended = '0' - pn = '' + # seconds + cpu_time_user = 0 + cpu_time_system = 0 + + # bytes + disk_io_read = 0 + disk_io_write = 0 + + started = 0 + ended = 0 + taskname = '' + statinfo = {} with open(filename, 'r') as task_bs: @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { k,v = line.strip().split(": ", 1) statinfo[k] = v - if "CPU usage" in statinfo: - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') - - if "IO write_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) - - if "IO read_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) - if "Started" in statinfo: - started = str(statinfo["Started"]).strip('% \n\r') + started = stat_to_float(statinfo["Started"]) if "Ended" in statinfo: - ended = str(statinfo["Ended"]).strip('% \n\r') + ended = stat_to_float(statinfo["Ended"]) - elapsed_time = float(ended) - float(started) + if "Child rusage ru_utime" in statinfo: + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) - cpu_usage = float(cpu_usage) + if "Child rusage ru_stime" in statinfo: + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} + if "IO write_bytes" in statinfo: + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) + disk_io_write = disk_io_write + write_bytes + if "IO read_bytes" in statinfo: + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) + disk_io_read = disk_io_read + read_bytes + + return { + 'stat_file': filename, + 'cpu_time_user': cpu_time_user, + 'cpu_time_system': cpu_time_system, + 'disk_io_read': disk_io_read, + 'disk_io_write': disk_io_write, + 'started': started, + 'ended': ended + } if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): _append_read_list(e) pass - - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): events = [] - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: + with open(toaster_statlist_file, "r") as fin: for line in fin: (taskfile, taskname, filename, recipename) = line.strip().split("::") - events.append((taskfile, taskname, _read_stats(filename), recipename)) + stats = _read_stats(filename) + events.append((taskfile, taskname, stats, recipename)) bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) + #os.unlink(toaster_statlist_file) } # dump relevant build history data as an event when the build is completed -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 0/3][v3] Fix task buildstats gathering @ 2016-02-23 12:56 Elliot Smith 2016-02-23 12:56 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 0 siblings, 1 reply; 12+ messages in thread From: Elliot Smith @ 2016-02-23 12:56 UTC (permalink / raw) To: toaster v3: Only show a single navigation item in the left-hand nav panel for CPU times, but show both System and User time columns when it is selected. Add a test for CPU time subpage. v2: Store "started" and "ended" times for each task. (This will enable other improvements later, such as showing when tasks overlap each other.) Compute elapsed_time from start and end times, rather than in toaster.bbclass. Store disk IO in two extra fields, one for read and the other for write. Compute total disk_io from these two values. (elapsed_time and disk_io are still stored, as these are used for sorting and we can't easily sort on a computed field. This can be amended when we get round to putting the tasks table into ToasterTable; see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.) Modify the UI to show the CPU times split between system and user. v1: Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway. Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper. To test: 1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe). 2. Run a build against a local Yocto (as one patch is for toaster.bbclass). 3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842. Changes since 961893fa05f4d51187f4d5daadacd7927d1a0847 (toaster-next) are in git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842 http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842 Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842 NOTE: The first commit in this series is for oe-core. Elliot Smith (3): toaster.bbclass: improve how we gather buildstats for Toaster toaster: rework task buildstats storage and display toaster-tests: add test for cputime subpage bitbake/lib/bb/ui/buildinfohelper.py | 72 ++++++++++------------ .../orm/migrations/0005_task_field_separation.py | 48 +++++++++++++++ bitbake/lib/toaster/orm/models.py | 18 +++++- .../toastergui/templates/basebuildpage.html | 4 +- .../lib/toaster/toastergui/templates/tasks.html | 20 +++--- bitbake/lib/toaster/toastergui/tests.py | 45 +++++++++++++- bitbake/lib/toaster/toastergui/urls.py | 2 +- bitbake/lib/toaster/toastergui/views.py | 55 +++++++++++------ meta/classes/toaster.bbclass | 70 +++++++++++++-------- 9 files changed, 237 insertions(+), 97 deletions(-) create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster 2016-02-23 12:56 [PATCH 0/3][v3] Fix task buildstats gathering Elliot Smith @ 2016-02-23 12:56 ` Elliot Smith 0 siblings, 0 replies; 12+ messages in thread From: Elliot Smith @ 2016-02-23 12:56 UTC (permalink / raw) To: toaster Clean up the code which gathers buildstats for Toaster, and modify the field names so that the correct parts of the buildstats files are used to derive the CPU usage values. Also derive elapsed time for the build here, rather than in Toaster, as we have ready access to the data in the correct format. [YOCTO #8842] Signed-off-by: Elliot Smith <elliot.smith@intel.com> --- meta/classes/toaster.bbclass | 70 ++++++++++++++++++++++++++++---------------- 1 file changed, 45 insertions(+), 25 deletions(-) diff --git a/meta/classes/toaster.bbclass b/meta/classes/toaster.bbclass index 51a4c74..b7e8773 100644 --- a/meta/classes/toaster.bbclass +++ b/meta/classes/toaster.bbclass @@ -202,25 +202,37 @@ python toaster_collect_task_stats() { import bb.utils import os + toaster_statlist_file = os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist") + if not e.data.getVar('BUILDSTATS_BASE', True): return # if we don't have buildstats, we cannot collect stats + def stat_to_float(value): + return float(value.strip('% \n\r')) + def _append_read_list(v): lock = bb.utils.lockfile(e.data.expand("${TOPDIR}/toaster.lock"), False, True) - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "a") as fout: + with open(toaster_statlist_file, "a") as fout: taskdir = e.data.expand("${BUILDSTATS_BASE}/${BUILDNAME}/${PF}") fout.write("%s::%s::%s::%s\n" % (e.taskfile, e.taskname, os.path.join(taskdir, e.task), e.data.expand("${PN}"))) bb.utils.unlockfile(lock) def _read_stats(filename): - cpu_usage = 0 - disk_io = 0 - started = '0' - ended = '0' - pn = '' + # seconds + cpu_time_user = 0 + cpu_time_system = 0 + + # bytes + disk_io_read = 0 + disk_io_write = 0 + + started = 0 + ended = 0 + taskname = '' + statinfo = {} with open(filename, 'r') as task_bs: @@ -228,41 +240,49 @@ python toaster_collect_task_stats() { k,v = line.strip().split(": ", 1) statinfo[k] = v - if "CPU usage" in statinfo: - cpu_usage = str(statinfo["CPU usage"]).strip('% \n\r') - - if "IO write_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO write_bytes"].strip('% \n\r')) - - if "IO read_bytes" in statinfo: - disk_io = disk_io + int(statinfo["IO read_bytes"].strip('% \n\r')) - if "Started" in statinfo: - started = str(statinfo["Started"]).strip('% \n\r') + started = stat_to_float(statinfo["Started"]) if "Ended" in statinfo: - ended = str(statinfo["Ended"]).strip('% \n\r') + ended = stat_to_float(statinfo["Ended"]) - elapsed_time = float(ended) - float(started) + if "Child rusage ru_utime" in statinfo: + cpu_time_user = cpu_time_user + stat_to_float(statinfo["Child rusage ru_utime"]) - cpu_usage = float(cpu_usage) + if "Child rusage ru_stime" in statinfo: + cpu_time_system = cpu_time_system + stat_to_float(statinfo["Child rusage ru_stime"]) - return {'cpu_usage': cpu_usage, 'disk_io': disk_io, 'elapsed_time': elapsed_time} + if "IO write_bytes" in statinfo: + write_bytes = int(statinfo["IO write_bytes"].strip('% \n\r')) + disk_io_write = disk_io_write + write_bytes + if "IO read_bytes" in statinfo: + read_bytes = int(statinfo["IO read_bytes"].strip('% \n\r')) + disk_io_read = disk_io_read + read_bytes + + return { + 'stat_file': filename, + 'cpu_time_user': cpu_time_user, + 'cpu_time_system': cpu_time_system, + 'disk_io_read': disk_io_read, + 'disk_io_write': disk_io_write, + 'started': started, + 'ended': ended + } if isinstance(e, (bb.build.TaskSucceeded, bb.build.TaskFailed)): _append_read_list(e) pass - - if isinstance(e, bb.event.BuildCompleted) and os.path.exists(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")): + if isinstance(e, bb.event.BuildCompleted) and os.path.exists(toaster_statlist_file): events = [] - with open(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist"), "r") as fin: + with open(toaster_statlist_file, "r") as fin: for line in fin: (taskfile, taskname, filename, recipename) = line.strip().split("::") - events.append((taskfile, taskname, _read_stats(filename), recipename)) + stats = _read_stats(filename) + events.append((taskfile, taskname, stats, recipename)) bb.event.fire(bb.event.MetadataEvent("BuildStatsList", events), e.data) - os.unlink(os.path.join(e.data.getVar('BUILDSTATS_BASE', True), "toasterstatlist")) + #os.unlink(toaster_statlist_file) } # dump relevant build history data as an event when the build is completed -- Elliot Smith Software Engineer Intel OTC --------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-03-07 19:26 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-23 12:58 [PATCH 0/3][v4] Fix task buildstats gathering Elliot Smith 2016-02-23 12:58 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 2016-02-23 12:58 ` [PATCH 2/3] toaster: rework task buildstats storage and display Elliot Smith 2016-02-23 12:58 ` [PATCH 3/3] toaster-tests: add test for cputime subpage Elliot Smith 2016-02-23 14:29 ` [PATCH 0/3][v4] Fix task buildstats gathering Barros Pena, Belen 2016-02-23 17:55 ` Smith, Elliot -- strict thread matches above, loose matches on Subject: below -- 2016-03-07 12:21 [PATCH 0/3][v7] " Elliot Smith 2016-03-07 12:21 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 2016-03-07 18:39 ` Michael Wood 2016-03-07 19:26 ` Smith, Elliot 2016-02-24 11:15 [PATCH 0/3][v6] Fix task buildstats gathering Elliot Smith 2016-02-24 11:15 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 2016-02-23 18:17 [PATCH 0/3][v5] Fix task buildstats gathering Elliot Smith 2016-02-23 18:17 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith 2016-02-23 12:56 [PATCH 0/3][v3] Fix task buildstats gathering Elliot Smith 2016-02-23 12:56 ` [PATCH 1/3] toaster.bbclass: improve how we gather buildstats for Toaster Elliot Smith
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.