[PATCH 0/3][v4] Fix task buildstats gathering

[Elliot Smith <elliot.smith@intel.com>]

v4:

I messed up the patch submission, so I'm sending the correct patches this time.

v3:

Only show a single navigation item in the left-hand nav panel for CPU times,
but show both System and User time columns when it is selected.

Add a test for CPU time subpage.

v2:

Store "started" and "ended" times for each task. (This will enable other
improvements later, such as showing when tasks overlap each other.)
Compute elapsed_time from start and end times, rather than in
toaster.bbclass.

Store disk IO in two extra fields, one for read and the other for write.
Compute total disk_io from these two values.

(elapsed_time and disk_io are still stored, as these are used for sorting
and we can't easily sort on a computed field. This can be amended when we
get round to putting the tasks table into ToasterTable; see
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8363.)

Modify the UI to show the CPU times split between system and user.

v1:

Our buildstats gathering for tasks (CPU usage, Disk IO etc.) is broken, partly because of changes to how the data is organised in the buildstats files, and partly because tasks aren't being updated correctly anyway.

Clean up the code which collects the buildstats (it had lots of duplication), collect the stats from the new stat file format (some line leaders had changed), and correctly identify the Task objects to update in BuildInfoHelper.

To test:

1. Remove the content of the directories which cache tasks to see the best effects (I usually delete buildstats, tmp, sstate-cache, and cache, just to be safe).

2. Run a build against a local Yocto (as one patch is for toaster.bbclass).

3. When the build finishes, view the "Tasks" page for the build and enable the "Disk IO" and "CPU usage" columns. Note that some CPU usage percentages are > 100, as discussed in bug 8842.

Changes since 961893fa05f4d51187f4d5daadacd7927d1a0847 (toaster-next) are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/build_data-8842
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/build_data-8842

Related bug:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8842

NOTE: The first commit in this series is for oe-core.

Elliot Smith (3):
  toaster.bbclass: improve how we gather buildstats for Toaster
  toaster: rework task buildstats storage and display
  toaster-tests: add test for cputime subpage

 bitbake/lib/bb/ui/buildinfohelper.py               | 72 ++++++++++------------
 .../orm/migrations/0005_task_field_separation.py   | 48 +++++++++++++++
 bitbake/lib/toaster/orm/models.py                  | 18 +++++-
 .../toastergui/templates/basebuildpage.html        |  4 +-
 .../lib/toaster/toastergui/templates/tasks.html    | 20 +++---
 bitbake/lib/toaster/toastergui/tests.py            | 45 +++++++++++++-
 bitbake/lib/toaster/toastergui/urls.py             |  2 +-
 bitbake/lib/toaster/toastergui/views.py            | 55 +++++++++++------
 meta/classes/toaster.bbclass                       | 70 +++++++++++++--------
 9 files changed, 237 insertions(+), 97 deletions(-)
 create mode 100644 bitbake/lib/toaster/orm/migrations/0005_task_field_separation.py

--
Elliot Smith
Software Engineer
Intel OTC

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.