[PATCH 0/3] Improvements for performance test report view

[PATCH 0/3] Improvements for performance test report view

[Ninette Adhikari]

This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
The current report can be accessed here: 
Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. 
The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.

The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:

- Add Apache echart (https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
- Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the line charts.
- Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
- Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.

Updated report screenshots: 
https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
https://github.com/neighbourhoodie/poky/assets/13760198/1ed43876-73a9-487e-aed3-ca0edf97514c

For local setup, you can do the following:

1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)

2. In the poky repository run the following to build the report HTML:
```bash
./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
```
Note:
- Add your local path to the yocto-buildstats repo
- The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
- This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.


Ninette Adhikari (3):
  oe-build-perf-report: Add apache echarts to make report interactive
  oe-build-perf-report: Display more than 300 commits and date instead
    of commit number
  oe-build-perf-report: Improve report styling and add descriptions

 .../build_perf/html/measurement_chart.html    | 116 +++++++++++-------
 scripts/lib/build_perf/html/report.html       |  96 ++++++++++-----
 scripts/lib/build_perf/report.py              |   4 +-
 scripts/oe-build-perf-report                  |   6 +-
 4 files changed, 143 insertions(+), 79 deletions(-)

-- 
2.44.0

[PATCH 1/3] oe-build-perf-report: Add apache echarts to make report interactive

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

- Add Apache echarts (https://echarts.apache.org/en/index.html) library to create build performance charts.
- Restructure data to time and value array format so that it can be used by echarts.
- This commit also converts test duration to minutes to map against the values axis.
- Zoom is added to the line charts.

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 .../build_perf/html/measurement_chart.html    | 116 +++++++++++-------
 scripts/lib/build_perf/html/report.html       |   6 +-
 2 files changed, 72 insertions(+), 50 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 65f1a227ad..ffec3d09db 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -1,50 +1,76 @@
-<script type="text/javascript">
-  chartsDrawing += 1;
-  google.charts.setOnLoadCallback(drawChart_{{ chart_elem_id }});
-  function drawChart_{{ chart_elem_id }}() {
-    var data = new google.visualization.DataTable();
+<script type="module">
+  // Get raw data
+  const rawData = [
+    {% for sample in measurement.samples %}
+      [{{ sample.commit_num }}, {{ sample.mean.gv_value() }}, {{ sample.start_time }}],
+    {% endfor %}
+  ];
 
-    // Chart options
-    var options = {
-      theme : 'material',
-      legend: 'none',
-      hAxis: { format: '', title: 'Commit number',
-               minValue: {{ chart_opts.haxis.min }},
-               maxValue: {{ chart_opts.haxis.max }} },
-      {% if measurement.type == 'time' %}
-      vAxis: { format: 'h:mm:ss' },
-      {% else %}
-      vAxis: { format: '' },
-      {% endif %}
-      pointSize: 5,
-      chartArea: { left: 80, right: 15 },
-    };
+  const convertToMinute = (time) => {
+    return time[0]*60 + time[1] + time[2]/60 + time[3]/3600;
+  }
 
-    // Define data columns
-    data.addColumn('number', 'Commit');
-    data.addColumn('{{ measurement.value_type.gv_data_type }}',
-                   '{{ measurement.value_type.quantity }}');
-    // Add data rows
-    data.addRows([
-      {% for sample in measurement.samples %}
-        [{{ sample.commit_num }}, {{ sample.mean.gv_value() }}],
-      {% endfor %}
-    ]);
+  // Convert raw data to the format: [time, value]
+  const data = rawData.map(([commit, value, time]) => {
+    return [
+      new Date(time * 1000).getTime(), // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
+      Array.isArray(value) ? convertToMinute(value) : value // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+    ]
+  });
 
-    // Finally, draw the chart
-    chart_div = document.getElementById('{{ chart_elem_id }}');
-    var chart = new google.visualization.LineChart(chart_div);
-    google.visualization.events.addListener(chart, 'ready', function () {
-      //chart_div = document.getElementById('{{ chart_elem_id }}');
-      //chart_div.innerHTML = '<img src="' + chart.getImageURI() + '">';
-      png_div = document.getElementById('{{ chart_elem_id }}_png');
-      png_div.outerHTML = '<a id="{{ chart_elem_id }}_png" href="' + chart.getImageURI() + '">PNG</a>';
-      console.log("CHART READY: {{ chart_elem_id }}");
-      chartsDrawing -= 1;
-      if (chartsDrawing == 0)
-        console.log("ALL CHARTS READY");
-    });
-    chart.draw(data, options);
-}
+  // Set chart options
+  const option = {
+    tooltip: {
+      trigger: 'axis',
+      position: function (pt) {
+        return [pt[0], '10%'];
+      },
+      valueFormatter: (value) => value.toFixed(2)
+    },
+    xAxis: {
+      type: 'time',
+    },
+    yAxis: {
+      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration (minutes)' : 'Disk size (MB)',
+      type: 'value',
+      min: function(value) {
+        return Math.round(value.min - 0.5);
+      },
+      max: function(value) {
+        return Math.round(value.max + 0.5);
+      }
+    },
+    dataZoom: [
+      {
+        type: 'inside',
+        start: 0,
+        end: 100
+      },
+      {
+        start: 0,
+        end: 100
+      }
+    ],
+    series: [
+      {
+        name: '{{ measurement.value_type.quantity }}',
+        type: 'line',
+        smooth: true,
+        symbol: 'none',
+        data: data
+      }
+    ]
+  };
+
+  // Draw chart
+  const chart_div = document.getElementById('{{ chart_elem_id }}');
+  const measurement_chart= echarts.init(chart_div, null, {
+    height: 320
+  });
+  // Change chart size with browser resize
+  window.addEventListener('resize', function() {
+    measurement_chart.resize();
+  });
+  measurement_chart.setOption(option);
 </script>
 
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index d1ba6f2578..653fd985bc 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -3,11 +3,7 @@
 <head>
 {# Scripts, for visualization#}
 <!--START-OF-SCRIPTS-->
-<script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
-<script type="text/javascript">
-google.charts.load('current', {'packages':['corechart']});
-var chartsDrawing = 0;
-</script>
+<script src=" https://cdn.jsdelivr.net/npm/echarts@5.5.0/dist/echarts.min.js "></script>
 
 {# Render measurement result charts #}
 {% for test in test_data %}
-- 
2.44.0

[PATCH 2/3] oe-build-perf-report: Display more than 300 commits and date instead of commit number

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

- This commit updates measurement statistics data to include start_time so that time can be displayed instead of commit numbers on the chart.
- It also updates default commit history length to 300.

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 scripts/lib/build_perf/report.py | 4 +++-
 scripts/oe-build-perf-report     | 6 ++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/build_perf/report.py b/scripts/lib/build_perf/report.py
index ab77424cc7..82c56830d7 100644
--- a/scripts/lib/build_perf/report.py
+++ b/scripts/lib/build_perf/report.py
@@ -294,7 +294,7 @@ class SizeVal(MeasurementVal):
             return "null"
         return self / 1024
 
-def measurement_stats(meas, prefix=''):
+def measurement_stats(meas, prefix='', time=0):
     """Get statistics of a measurement"""
     if not meas:
         return {prefix + 'sample_cnt': 0,
@@ -319,6 +319,7 @@ def measurement_stats(meas, prefix=''):
     stats['quantity'] = val_cls.quantity
     stats[prefix + 'sample_cnt'] = len(values)
 
+    start_time = time # Add start time for both type sysres and disk usage
     mean_val = val_cls(mean(values))
     min_val = val_cls(min(values))
     max_val = val_cls(max(values))
@@ -334,6 +335,7 @@ def measurement_stats(meas, prefix=''):
     stats[prefix + 'max'] = max_val
     stats[prefix + 'minus'] = val_cls(mean_val - min_val)
     stats[prefix + 'plus'] = val_cls(max_val - mean_val)
+    stats[prefix + 'start_time'] = start_time
 
     return stats
 
diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report
index 7812ea4540..266700d294 100755
--- a/scripts/oe-build-perf-report
+++ b/scripts/oe-build-perf-report
@@ -336,7 +336,9 @@ def print_html_report(data, id_comp, buildstats):
                 test_i = test_data['tests'][test]
                 meas_i = test_i['measurements'][meas]
                 commit_num = get_data_item(meta, 'layers.meta.commit_count')
-                samples.append(measurement_stats(meas_i))
+                # Add start_time for both test measurement types of sysres and disk usage
+                start_time = test_i['start_time'][0]
+                samples.append(measurement_stats(meas_i, '', start_time))
                 samples[-1]['commit_num'] = commit_num
 
             absdiff = samples[-1]['val_cls'](samples[-1]['mean'] - samples[id_comp]['mean'])
@@ -473,7 +475,7 @@ Examine build performance test results from a Git repository"""
     group.add_argument('--branch', '-B', default='master', help="Branch to find commit in")
     group.add_argument('--branch2', help="Branch to find comparision revisions in")
     group.add_argument('--machine', default='qemux86')
-    group.add_argument('--history-length', default=25, type=int,
+    group.add_argument('--history-length', default=300, type=int,
                        help="Number of tested revisions to plot in html report")
     group.add_argument('--commit',
                        help="Revision to search for")
-- 
2.44.0

[PATCH 3/3] oe-build-perf-report: Improve report styling and add descriptions

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

Styling updates are added including page margin, labels for x and y axis, tooltip, and section descriptions.

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 .../build_perf/html/measurement_chart.html    | 28 +++---
 scripts/lib/build_perf/html/report.html       | 90 +++++++++++++------
 2 files changed, 78 insertions(+), 40 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index ffec3d09db..9acb3785e2 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -13,8 +13,10 @@
   // Convert raw data to the format: [time, value]
   const data = rawData.map(([commit, value, time]) => {
     return [
-      new Date(time * 1000).getTime(), // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
-      Array.isArray(value) ? convertToMinute(value) : value // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+      // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
+      new Date(time * 1000).getTime(),
+      // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+      Array.isArray(value) ? convertToMinute(value) : value
     ]
   });
 
@@ -22,16 +24,18 @@
   const option = {
     tooltip: {
       trigger: 'axis',
-      position: function (pt) {
-        return [pt[0], '10%'];
-      },
-      valueFormatter: (value) => value.toFixed(2)
+      valueFormatter: (value) => {
+        const hours = Math.floor(value/60)
+        const minutes = Math.floor(value % 60)
+        const seconds = Math.floor((value * 60) % 60)
+        return hours + ':' + minutes + ':' + seconds
+      }
     },
     xAxis: {
       type: 'time',
     },
     yAxis: {
-      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration (minutes)' : 'Disk size (MB)',
+      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration in minutes' : 'Disk size in MB',
       type: 'value',
       min: function(value) {
         return Math.round(value.min - 0.5);
@@ -42,14 +46,10 @@
     },
     dataZoom: [
       {
-        type: 'inside',
-        start: 0,
-        end: 100
+        type: 'slider',
+        xAxisIndex: 0,
+        filterMode: 'none'
       },
-      {
-        start: 0,
-        end: 100
-      }
     ],
     series: [
       {
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index 653fd985bc..4cd240760a 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -24,23 +24,15 @@
   text-align: left;
   border-collapse: collapse;
 }
-.meta-table tr:nth-child(even){background-color: #f2f2f2}
-meta-table th, .meta-table td {
-  padding: 4px;
-}
 .summary {
-  margin: 0;
   font-size: 14px;
   text-align: left;
   border-collapse: collapse;
 }
-summary th, .meta-table td {
-  padding: 4px;
-}
 .measurement {
   padding: 8px 0px 8px 8px;
   border: 2px solid #f0f0f0;
-  margin-bottom: 10px;
+  margin: 1.5rem 0;
 }
 .details {
   margin: 0;
@@ -60,18 +52,58 @@ summary th, .meta-table td {
   background-color: #f0f0f0;
   margin-left: 10px;
 }
-hr {
-  color: #f0f0f0;
+.card-container {
+  border-bottom-width: 1px;
+  padding: 1.25rem 3rem;
+  box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+  border-radius: 0.25rem;
+}
+body {
+  font-family: 'Helvetica', sans-serif;
+  margin: 3rem 8rem;
+}
+h1 {
+  text-align: center;
 }
 h2 {
-  font-size: 20px;
+  font-size: 1.5rem;
   margin-bottom: 0px;
   color: #707070;
+  padding-top: 1.5rem;
 }
 h3 {
-  font-size: 16px;
+  font-size: 1.3rem;
   margin: 0px;
   color: #707070;
+  padding: 1.5rem 0;
+}
+h4 {
+  font-size: 14px;
+  font-weight: lighter;
+  line-height: 1.2rem;
+  margin: auto;
+  padding-top: 1rem;
+}
+table {
+  margin-top: 1.5rem;
+  line-height: 2rem;
+}
+tr {
+  border-bottom: 1px solid #e5e7eb;
+}
+tr:first-child {
+  border-bottom: 1px solid #9ca3af;
+}
+tr:last-child {
+  border-bottom: none;
+}
+a {
+  text-decoration: none;
+  font-weight: bold;
+  color: #0000EE;
+}
+a:hover {
+  color: #8080ff;
 }
 </style>
 
@@ -79,13 +111,14 @@ h3 {
 </head>
 
 {% macro poky_link(commit) -%}
-    <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?id={{ commit }}">{{ commit[0:11] }}</a>
+  <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?id={{ commit }}">{{ commit[0:11] }}</a>
 {%- endmacro %}
 
-<body><div style="width: 700px">
+<body><div>
+  <h1 style="text-align: center;">Performance Test Report</h1>
   {# Test metadata #}
   <h2>General</h2>
-  <hr>
+  <h4>The table provides an overview of the comparison between two selected commits from the same branch.</h4>
   <table class="meta-table" style="width: 100%">
     <tr>
       <th></th>
@@ -108,19 +141,21 @@ h3 {
 
   {# Test result summary #}
   <h2>Test result summary</h2>
-  <hr>
+  <h4>The test summary presents a thorough breakdown of each test conducted on the branch, including details such as build time and disk space consumption. Additionally, it gives insights into the average time taken for test execution, along with absolute and relative values for a better understanding.</h4>
   <table class="summary" style="width: 100%">
+    <tr>
+      <th>Test name</th>
+      <th>Measurement description</th>
+      <th>Mean value</th>
+      <th>Absolute difference</th>
+      <th>Relative difference</th>
+    </tr>
     {% for test in test_data %}
-      {% if loop.index is even %}
-        {% set row_style = 'style="background-color: #f2f2f2"' %}
-      {% else %}
-        {% set row_style = 'style="background-color: #ffffff"' %}
-      {% endif %}
       {% if test.status == 'SUCCESS' %}
         {% for measurement in test.measurements %}
           <tr {{ row_style }}>
             {% if loop.index == 1 %}
-              <td>{{ test.name }}: {{ test.description }}</td>
+              <td><a href=#{{test.name}}>{{ test.name }}: {{ test.description }}</a></td>
             {% else %}
               {# add empty cell in place of the test name#}
               <td></td>
@@ -149,10 +184,12 @@ h3 {
   </table>
 
   {# Detailed test results #}
+  <h2>Test details</h2>
+  <h4>The following section provides details of each test, accompanied by charts representing build time and disk usage over time or by commit number.</h4>
   {% for test in test_data %}
-  <h2>{{ test.name }}: {{ test.description }}</h2>
-  <hr>
+  <h3 style="color: #000;" id={{test.name}}>{{ test.name }}: {{ test.description }}</h3>
     {% if test.status == 'SUCCESS' %}
+    <div class="card-container">
       {% for measurement in test.measurements %}
         <div class="measurement">
           <h3>{{ measurement.description }}</h3>
@@ -271,7 +308,8 @@ h3 {
             {% endif %}
           {% endif %}
         </div>
-      {% endfor %}
+        {% endfor %}
+      </div>
     {# Unsuccessful test #}
     {% else %}
       <span style="font-size: 150%; font-weight: bold; color: red;">{{ test.status }}
-- 
2.44.0

Patchtest results for [PATCH 3/3] oe-build-perf-report: Improve report styling and add descriptions

[patchtest]

[-- Attachment #1: Type: text/plain, Size: 3072 bytes --]

Thank you for your submission. Patchtest identified one
or more issues with the patch. Please see the log below for
more information:

---
Testing patch /home/patchtest/share/mboxes/3-3-oe-build-perf-report-Improve-report-styling-and-add-descriptions.patch

FAIL: test max line length: Patch line too long (current length 308, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)

PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)

SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)

---

Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this
email with an explanation of why the patch should be accepted. If you
believe these results are due to an error in patchtest, please submit a
bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category
under 'Yocto Project Subprojects'). For more information on specific
failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank
you!

Re: [OE-core] [PATCH 0/3] Improvements for performance test report view

[Richard Purdie]

Hi Ninette,

Firstly, thanks for these patches, it is great to be able to try and
improve these and make the data more usable and interactive. 

To help others visualise the new output and test the patches I've made
a test run on the autobuilder and the results are:

http://autobuilder.yocto.io/pub/non-release/20240413-21/testresults/buildperf-alma8/perf-alma8_master-next_20240413135138_904bb385c3.html

and 

http://autobuilder.yocto.io/pub/non-release/20240413-19/testresults/buildperf-debian11/perf-debian11_master-next_20240413064005_7116cd908d.html

In general I love the wider commit range of the new chart and that you
can interact with them, this is great.

The is one big problem which is the change to "start_time" which is
problematic. We need the commit numbers on the graph to be able to
decode which change a given point in time represents. Is there a way we
could show both the commit number and date of the build together?

I think if we could fix that issue these would be good to merge from my
perspective. I have shared links to the examples above in our weekly
status report for others to review as well though.

Cheers,

Richard



On Mon, 2024-04-15 at 16:41 +0200, Ninette Adhikari via
lists.openembedded.org wrote:
> This work is done according to "Milestone 9: Build performance test
> report view" as stated in the Scope of Work with Sovereign Tech Fund
> (STF) (https://www.sovereigntechfund.de/).
> The current report can be accessed here: 
> Performance test report HTML
> (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults
> /buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
> The report is created using the `oe-build-perf-report` script in the
> poky repository. This script generates a performance test report in
> HTML format using the data from the yocto-buildstats
> (https://git.yoctoproject.org/yocto-buildstats/) repository.
> The report displays the performance test results in line chart
> format. The chart x-axis represents the commit numbers, and the y-
> axis represents the test duration in minutes. 
> The report also includes a table that displays the measurement
> statistics data for each test. The report is interactive and allows
> users to zoom in on specific sections of the line chart.
> 
> The current report format required some updates to make it more
> interactive and user-friendly. And this patch addresses such
> improvements:
> 
> - Add Apache echart (https://echarts.apache.org/en/index.html)
> library to create oe build performance report charts and make them
> interactive.
> - Restructure data to time and value array format to be used by
> echarts. It also converts test duration to minutes and adds zoom to
> the line charts.
> - Update measurement statistics data to include `start_time` so that
> time can be displayed instead of commit numbers on the chart. It also
> updates default commit history length to 300.
> - Add styling updates including page margin, labels for x and y axis,
> tooltip, and section descriptions.
> 
> Updated report screenshots: 
> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
> https://github.com/neighbourhoodie/poky/assets/13760198/1ed43876-73a9-487e-aed3-ca0edf97514c
> 
> For local setup, you can do the following:
> 
> 1. Clone the yocto-buildstats
> (https://git.yoctoproject.org/yocto-buildstats/) and the poky
> repository (https://git.yoctoproject.org/poky/)
> 
> 2. In the poky repository run the following to build the report HTML:
> ```bash
> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --
> branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" -
> -branch2 "master" --html > test.html
> ```
> Note:
> - Add your local path to the yocto-buildstats repo
> - The above command builds the report in a file called `test.html`.
> You can access it in the root directory in poky.
> - This exmaple report uses the commit
> `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.

Re: [OE-core] [PATCH 1/3] oe-build-perf-report: Add apache echarts to make report interactive

[Ross Burton]

Hi Ninette,

First, this is awesome, thanks very much!

One thing I noticed is that the ‘rootfs size’ and ’tmpdir size’ tooltips are trying to format the data as a timestamp instead of a size.

Cheers,
Ross

Re: [OE-core] [PATCH 0/3] Improvements for performance test report view

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 6002 bytes --]

On 2024-04-15 10:41 a.m., Ninette Adhikari via lists.openembedded.org wrote:
> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
> The current report can be accessed here:
> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes.
> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.
>
> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:
>
> - Add Apache echart (https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the line charts.
> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
>
> Updated report screenshots:
> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
> https://github.com/neighbourhoodie/poky/assets/13760198/1ed43876-73a9-487e-aed3-ca0edf97514c

Thanks for the nice, dynamic UI web graphs!
Overall, I find them useful and well-designed.


Would it be possible to present the data in a more discrete manner since
the interpolated view presented now, while nice to look at, does not always
reflect the underlying data trend?

For example, if I have this data plotted as 1) stair-steps or 2) line 
segments (I didn't take the time to do a spline fit),

I find that the line style for 1) is usually less misleading about what 
we really know about the data.


1) stair-step



2) straight lines:



and I'd like to see this data:


plotted in the stair-step style.

From:

http://autobuilder.yocto.io/pub/non-release/20240413-19/testresults/buildperf-debian11/perf-debian11_master-next_20240413064005_7116cd908d.html

> For local setup, you can do the following:
>
> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)
>
> 2. In the poky repository run the following to build the report HTML:
> ```bash
> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
> ```
> Note:
> - Add your local path to the yocto-buildstats repo
> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.

I tried that out:

❯ cd .../distro/yocto/yocto-bs-html

❯ ../poky.git/scripts/oe-build-perf-report -r 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-bs-html/../yocto-buildstats.git" 
--branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" 
--branch2 "master" --html > test.html
ERROR: No buildstats found, please try running 'git fetch origin 
refs/notes/buildstats/perf-alma8/master/qemux86:refs/notes/buildstats/perf-alma8/master/qemux86' 
to fetch them from the remote

so I did that.  It was > 3 GB of data...  !!

Receiving objects:  89% (10568/11747), 2.03 GiB | 3.28 MiB/s6 MiB/s

❯ du -sh .
3.2G    .

Also, one shouldn't have to call bash just to run your script since you 
shell
does that already if you have the right #!/path/to/foo and you have made 
the script executable:

❯ file ../poky.git/scripts/oe-build-perf-report
../poky.git/scripts/oe-build-perf-report: Python script, ASCII text 
executable

❯ head -1 ../poky.git/scripts/oe-build-perf-report
#!/usr/bin/env python3


Anyway, that did generate a test.html file in ~ 10 seconds for me on my 
Pop!OS (Ubuntu-based) distro
and it rendered nicely in Firefox 124.0.1 (64-bit) so that's nice to see.


Thanks again,

../Randy



>
>
> Ninette Adhikari (3):
>    oe-build-perf-report: Add apache echarts to make report interactive
>    oe-build-perf-report: Display more than 300 commits and date instead
>      of commit number
>    oe-build-perf-report: Improve report styling and add descriptions
>
>   .../build_perf/html/measurement_chart.html    | 116 +++++++++++-------
>   scripts/lib/build_perf/html/report.html       |  96 ++++++++++-----
>   scripts/lib/build_perf/report.py              |   4 +-
>   scripts/oe-build-perf-report                  |   6 +-
>   4 files changed, 143 insertions(+), 79 deletions(-)
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#198239):https://lists.openembedded.org/g/openembedded-core/message/198239
> Mute This Topic:https://lists.openembedded.org/mt/105537213/3616765
> Group Owner:openembedded-core+owner@lists.openembedded.org
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [randy.macleod@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2.1: Type: text/html, Size: 9421 bytes --]

[-- Attachment #2.2: C0NlrGL0XCzOLWe5.png --]
[-- Type: image/png, Size: 11077 bytes --]

[-- Attachment #2.3: PK30ZAoDLv3EqTSr.png --]
[-- Type: image/png, Size: 12782 bytes --]

[-- Attachment #2.4: YdeAn9rTv39nGigR.png --]
[-- Type: image/png, Size: 54532 bytes --]

[PATCH v2] oe-build-perf-report: Update chart tooltip format

[Ninette Adhikari]

Update chart tooltip format to show value as size in MB for 'rootfs size'
and timestamp for 'tmpdir size'

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 scripts/lib/build_perf/html/measurement_chart.html | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 9acb3785e2..07e92dfe78 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -25,10 +25,13 @@
     tooltip: {
       trigger: 'axis',
       valueFormatter: (value) => {
-        const hours = Math.floor(value/60)
-        const minutes = Math.floor(value % 60)
-        const seconds = Math.floor((value * 60) % 60)
-        return hours + ':' + minutes + ':' + seconds
+        if ('{{ measurement.value_type.quantity }}' == 'time') {
+          const hours = Math.floor(value/60)
+          const minutes = Math.floor(value % 60)
+          const seconds = Math.floor((value * 60) % 60)
+          return hours + ':' + minutes + ':' + seconds
+        }
+        return value.toFixed(2) + ' MB'
       }
     },
     xAxis: {
-- 
2.44.0

[PATCH v2 0/5] Improvements for performance test report view

[Ninette Adhikari]

This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
The current report can be accessed here: 
Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes. 
The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.

The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:

- Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
- Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts.
- Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
- Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
- The charts are created as step charts instead of plain line charts.
- Add start time, size, and commit number to tooltip.
- Add dark mode view

Updated report screenshots: 
https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003
https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a
https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208

For local setup, you can do the following:

1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)

2. In the poky repository run the following to build the report HTML:
```bash
./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
```
Note:
- Add your local path to the yocto-buildstats repo
- The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
- This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.


Ninette Adhikari (5):
  oe-build-perf-report: Add apache echarts to make report interactive
  oe-build-perf-report: Display more than 300 commits and date instead
    of commit number
  oe-build-perf-report: Improve report styling and add descriptions
  oe-build-perf-report: Update chart tooltip and chart type
  oe-build-perf-report: Add dark mode

 .../build_perf/html/measurement_chart.html    | 140 ++++++++++++------
 scripts/lib/build_perf/html/report.html       | 124 +++++++++++-----
 scripts/lib/build_perf/report.py              |   5 +-
 scripts/oe-build-perf-report                  |   6 +-
 4 files changed, 193 insertions(+), 82 deletions(-)

-- 
2.44.0

[PATCH v2 1/5] oe-build-perf-report: Add apache echarts to make report interactive

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

- Add Apache echarts (https://echarts.apache.org/en/index.html) library to create build performance charts.
- Restructure data to time and value array format so that it can be used by echarts.
- This commit also converts test duration to minutes to map against the values axis.
- Zoom is added to the line charts.
---
 .../build_perf/html/measurement_chart.html    | 116 +++++++++++-------
 scripts/lib/build_perf/html/report.html       |   6 +-
 2 files changed, 72 insertions(+), 50 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 65f1a227ad..ffec3d09db 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -1,50 +1,76 @@
-<script type="text/javascript">
-  chartsDrawing += 1;
-  google.charts.setOnLoadCallback(drawChart_{{ chart_elem_id }});
-  function drawChart_{{ chart_elem_id }}() {
-    var data = new google.visualization.DataTable();
+<script type="module">
+  // Get raw data
+  const rawData = [
+    {% for sample in measurement.samples %}
+      [{{ sample.commit_num }}, {{ sample.mean.gv_value() }}, {{ sample.start_time }}],
+    {% endfor %}
+  ];
 
-    // Chart options
-    var options = {
-      theme : 'material',
-      legend: 'none',
-      hAxis: { format: '', title: 'Commit number',
-               minValue: {{ chart_opts.haxis.min }},
-               maxValue: {{ chart_opts.haxis.max }} },
-      {% if measurement.type == 'time' %}
-      vAxis: { format: 'h:mm:ss' },
-      {% else %}
-      vAxis: { format: '' },
-      {% endif %}
-      pointSize: 5,
-      chartArea: { left: 80, right: 15 },
-    };
+  const convertToMinute = (time) => {
+    return time[0]*60 + time[1] + time[2]/60 + time[3]/3600;
+  }
 
-    // Define data columns
-    data.addColumn('number', 'Commit');
-    data.addColumn('{{ measurement.value_type.gv_data_type }}',
-                   '{{ measurement.value_type.quantity }}');
-    // Add data rows
-    data.addRows([
-      {% for sample in measurement.samples %}
-        [{{ sample.commit_num }}, {{ sample.mean.gv_value() }}],
-      {% endfor %}
-    ]);
+  // Convert raw data to the format: [time, value]
+  const data = rawData.map(([commit, value, time]) => {
+    return [
+      new Date(time * 1000).getTime(), // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
+      Array.isArray(value) ? convertToMinute(value) : value // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+    ]
+  });
 
-    // Finally, draw the chart
-    chart_div = document.getElementById('{{ chart_elem_id }}');
-    var chart = new google.visualization.LineChart(chart_div);
-    google.visualization.events.addListener(chart, 'ready', function () {
-      //chart_div = document.getElementById('{{ chart_elem_id }}');
-      //chart_div.innerHTML = '<img src="' + chart.getImageURI() + '">';
-      png_div = document.getElementById('{{ chart_elem_id }}_png');
-      png_div.outerHTML = '<a id="{{ chart_elem_id }}_png" href="' + chart.getImageURI() + '">PNG</a>';
-      console.log("CHART READY: {{ chart_elem_id }}");
-      chartsDrawing -= 1;
-      if (chartsDrawing == 0)
-        console.log("ALL CHARTS READY");
-    });
-    chart.draw(data, options);
-}
+  // Set chart options
+  const option = {
+    tooltip: {
+      trigger: 'axis',
+      position: function (pt) {
+        return [pt[0], '10%'];
+      },
+      valueFormatter: (value) => value.toFixed(2)
+    },
+    xAxis: {
+      type: 'time',
+    },
+    yAxis: {
+      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration (minutes)' : 'Disk size (MB)',
+      type: 'value',
+      min: function(value) {
+        return Math.round(value.min - 0.5);
+      },
+      max: function(value) {
+        return Math.round(value.max + 0.5);
+      }
+    },
+    dataZoom: [
+      {
+        type: 'inside',
+        start: 0,
+        end: 100
+      },
+      {
+        start: 0,
+        end: 100
+      }
+    ],
+    series: [
+      {
+        name: '{{ measurement.value_type.quantity }}',
+        type: 'line',
+        smooth: true,
+        symbol: 'none',
+        data: data
+      }
+    ]
+  };
+
+  // Draw chart
+  const chart_div = document.getElementById('{{ chart_elem_id }}');
+  const measurement_chart= echarts.init(chart_div, null, {
+    height: 320
+  });
+  // Change chart size with browser resize
+  window.addEventListener('resize', function() {
+    measurement_chart.resize();
+  });
+  measurement_chart.setOption(option);
 </script>
 
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index d1ba6f2578..653fd985bc 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -3,11 +3,7 @@
 <head>
 {# Scripts, for visualization#}
 <!--START-OF-SCRIPTS-->
-<script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
-<script type="text/javascript">
-google.charts.load('current', {'packages':['corechart']});
-var chartsDrawing = 0;
-</script>
+<script src=" https://cdn.jsdelivr.net/npm/echarts@5.5.0/dist/echarts.min.js "></script>
 
 {# Render measurement result charts #}
 {% for test in test_data %}
-- 
2.44.0

[PATCH v2 2/5] oe-build-perf-report: Display more than 300 commits and date instead of commit number

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

- This commit updates measurement statistics data to include start_time so that time can be displayed instead of commit numbers on the chart.
- It also updates default commit history length to 300.
---
 scripts/lib/build_perf/report.py | 4 +++-
 scripts/oe-build-perf-report     | 6 ++++--
 2 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/build_perf/report.py b/scripts/lib/build_perf/report.py
index ab77424cc7..82c56830d7 100644
--- a/scripts/lib/build_perf/report.py
+++ b/scripts/lib/build_perf/report.py
@@ -294,7 +294,7 @@ class SizeVal(MeasurementVal):
             return "null"
         return self / 1024
 
-def measurement_stats(meas, prefix=''):
+def measurement_stats(meas, prefix='', time=0):
     """Get statistics of a measurement"""
     if not meas:
         return {prefix + 'sample_cnt': 0,
@@ -319,6 +319,7 @@ def measurement_stats(meas, prefix=''):
     stats['quantity'] = val_cls.quantity
     stats[prefix + 'sample_cnt'] = len(values)
 
+    start_time = time # Add start time for both type sysres and disk usage
     mean_val = val_cls(mean(values))
     min_val = val_cls(min(values))
     max_val = val_cls(max(values))
@@ -334,6 +335,7 @@ def measurement_stats(meas, prefix=''):
     stats[prefix + 'max'] = max_val
     stats[prefix + 'minus'] = val_cls(mean_val - min_val)
     stats[prefix + 'plus'] = val_cls(max_val - mean_val)
+    stats[prefix + 'start_time'] = start_time
 
     return stats
 
diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report
index 7812ea4540..266700d294 100755
--- a/scripts/oe-build-perf-report
+++ b/scripts/oe-build-perf-report
@@ -336,7 +336,9 @@ def print_html_report(data, id_comp, buildstats):
                 test_i = test_data['tests'][test]
                 meas_i = test_i['measurements'][meas]
                 commit_num = get_data_item(meta, 'layers.meta.commit_count')
-                samples.append(measurement_stats(meas_i))
+                # Add start_time for both test measurement types of sysres and disk usage
+                start_time = test_i['start_time'][0]
+                samples.append(measurement_stats(meas_i, '', start_time))
                 samples[-1]['commit_num'] = commit_num
 
             absdiff = samples[-1]['val_cls'](samples[-1]['mean'] - samples[id_comp]['mean'])
@@ -473,7 +475,7 @@ Examine build performance test results from a Git repository"""
     group.add_argument('--branch', '-B', default='master', help="Branch to find commit in")
     group.add_argument('--branch2', help="Branch to find comparision revisions in")
     group.add_argument('--machine', default='qemux86')
-    group.add_argument('--history-length', default=25, type=int,
+    group.add_argument('--history-length', default=300, type=int,
                        help="Number of tested revisions to plot in html report")
     group.add_argument('--commit',
                        help="Revision to search for")
-- 
2.44.0

[PATCH v2 3/5] oe-build-perf-report: Improve report styling and add descriptions

[Ninette Adhikari]

From: Ninette Adhikari <13760198+ninetteadhikari@users.noreply.github.com>

Styling updates are added including page margin, labels for x and y axis, tooltip, and section descriptions.
---
 .../build_perf/html/measurement_chart.html    | 28 +++---
 scripts/lib/build_perf/html/report.html       | 90 +++++++++++++------
 scripts/lib/build_perf/report.py              |  3 +-
 3 files changed, 80 insertions(+), 41 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index ffec3d09db..9acb3785e2 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -13,8 +13,10 @@
   // Convert raw data to the format: [time, value]
   const data = rawData.map(([commit, value, time]) => {
     return [
-      new Date(time * 1000).getTime(), // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
-      Array.isArray(value) ? convertToMinute(value) : value // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+      // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
+      new Date(time * 1000).getTime(),
+      // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+      Array.isArray(value) ? convertToMinute(value) : value
     ]
   });
 
@@ -22,16 +24,18 @@
   const option = {
     tooltip: {
       trigger: 'axis',
-      position: function (pt) {
-        return [pt[0], '10%'];
-      },
-      valueFormatter: (value) => value.toFixed(2)
+      valueFormatter: (value) => {
+        const hours = Math.floor(value/60)
+        const minutes = Math.floor(value % 60)
+        const seconds = Math.floor((value * 60) % 60)
+        return hours + ':' + minutes + ':' + seconds
+      }
     },
     xAxis: {
       type: 'time',
     },
     yAxis: {
-      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration (minutes)' : 'Disk size (MB)',
+      name: '{{ measurement.value_type.quantity }}' == 'time' ? 'Duration in minutes' : 'Disk size in MB',
       type: 'value',
       min: function(value) {
         return Math.round(value.min - 0.5);
@@ -42,14 +46,10 @@
     },
     dataZoom: [
       {
-        type: 'inside',
-        start: 0,
-        end: 100
+        type: 'slider',
+        xAxisIndex: 0,
+        filterMode: 'none'
       },
-      {
-        start: 0,
-        end: 100
-      }
     ],
     series: [
       {
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index 653fd985bc..4cd240760a 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -24,23 +24,15 @@
   text-align: left;
   border-collapse: collapse;
 }
-.meta-table tr:nth-child(even){background-color: #f2f2f2}
-meta-table th, .meta-table td {
-  padding: 4px;
-}
 .summary {
-  margin: 0;
   font-size: 14px;
   text-align: left;
   border-collapse: collapse;
 }
-summary th, .meta-table td {
-  padding: 4px;
-}
 .measurement {
   padding: 8px 0px 8px 8px;
   border: 2px solid #f0f0f0;
-  margin-bottom: 10px;
+  margin: 1.5rem 0;
 }
 .details {
   margin: 0;
@@ -60,18 +52,58 @@ summary th, .meta-table td {
   background-color: #f0f0f0;
   margin-left: 10px;
 }
-hr {
-  color: #f0f0f0;
+.card-container {
+  border-bottom-width: 1px;
+  padding: 1.25rem 3rem;
+  box-shadow: 0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1);
+  border-radius: 0.25rem;
+}
+body {
+  font-family: 'Helvetica', sans-serif;
+  margin: 3rem 8rem;
+}
+h1 {
+  text-align: center;
 }
 h2 {
-  font-size: 20px;
+  font-size: 1.5rem;
   margin-bottom: 0px;
   color: #707070;
+  padding-top: 1.5rem;
 }
 h3 {
-  font-size: 16px;
+  font-size: 1.3rem;
   margin: 0px;
   color: #707070;
+  padding: 1.5rem 0;
+}
+h4 {
+  font-size: 14px;
+  font-weight: lighter;
+  line-height: 1.2rem;
+  margin: auto;
+  padding-top: 1rem;
+}
+table {
+  margin-top: 1.5rem;
+  line-height: 2rem;
+}
+tr {
+  border-bottom: 1px solid #e5e7eb;
+}
+tr:first-child {
+  border-bottom: 1px solid #9ca3af;
+}
+tr:last-child {
+  border-bottom: none;
+}
+a {
+  text-decoration: none;
+  font-weight: bold;
+  color: #0000EE;
+}
+a:hover {
+  color: #8080ff;
 }
 </style>
 
@@ -79,13 +111,14 @@ h3 {
 </head>
 
 {% macro poky_link(commit) -%}
-    <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?id={{ commit }}">{{ commit[0:11] }}</a>
+  <a href="http://git.yoctoproject.org/cgit/cgit.cgi/poky/log/?id={{ commit }}">{{ commit[0:11] }}</a>
 {%- endmacro %}
 
-<body><div style="width: 700px">
+<body><div>
+  <h1 style="text-align: center;">Performance Test Report</h1>
   {# Test metadata #}
   <h2>General</h2>
-  <hr>
+  <h4>The table provides an overview of the comparison between two selected commits from the same branch.</h4>
   <table class="meta-table" style="width: 100%">
     <tr>
       <th></th>
@@ -108,19 +141,21 @@ h3 {
 
   {# Test result summary #}
   <h2>Test result summary</h2>
-  <hr>
+  <h4>The test summary presents a thorough breakdown of each test conducted on the branch, including details such as build time and disk space consumption. Additionally, it gives insights into the average time taken for test execution, along with absolute and relative values for a better understanding.</h4>
   <table class="summary" style="width: 100%">
+    <tr>
+      <th>Test name</th>
+      <th>Measurement description</th>
+      <th>Mean value</th>
+      <th>Absolute difference</th>
+      <th>Relative difference</th>
+    </tr>
     {% for test in test_data %}
-      {% if loop.index is even %}
-        {% set row_style = 'style="background-color: #f2f2f2"' %}
-      {% else %}
-        {% set row_style = 'style="background-color: #ffffff"' %}
-      {% endif %}
       {% if test.status == 'SUCCESS' %}
         {% for measurement in test.measurements %}
           <tr {{ row_style }}>
             {% if loop.index == 1 %}
-              <td>{{ test.name }}: {{ test.description }}</td>
+              <td><a href=#{{test.name}}>{{ test.name }}: {{ test.description }}</a></td>
             {% else %}
               {# add empty cell in place of the test name#}
               <td></td>
@@ -149,10 +184,12 @@ h3 {
   </table>
 
   {# Detailed test results #}
+  <h2>Test details</h2>
+  <h4>The following section provides details of each test, accompanied by charts representing build time and disk usage over time or by commit number.</h4>
   {% for test in test_data %}
-  <h2>{{ test.name }}: {{ test.description }}</h2>
-  <hr>
+  <h3 style="color: #000;" id={{test.name}}>{{ test.name }}: {{ test.description }}</h3>
     {% if test.status == 'SUCCESS' %}
+    <div class="card-container">
       {% for measurement in test.measurements %}
         <div class="measurement">
           <h3>{{ measurement.description }}</h3>
@@ -271,7 +308,8 @@ h3 {
             {% endif %}
           {% endif %}
         </div>
-      {% endfor %}
+        {% endfor %}
+      </div>
     {# Unsuccessful test #}
     {% else %}
       <span style="font-size: 150%; font-weight: bold; color: red;">{{ test.status }}
diff --git a/scripts/lib/build_perf/report.py b/scripts/lib/build_perf/report.py
index 82c56830d7..f4e6a92e09 100644
--- a/scripts/lib/build_perf/report.py
+++ b/scripts/lib/build_perf/report.py
@@ -319,7 +319,8 @@ def measurement_stats(meas, prefix='', time=0):
     stats['quantity'] = val_cls.quantity
     stats[prefix + 'sample_cnt'] = len(values)
 
-    start_time = time # Add start time for both type sysres and disk usage
+    # Add start time for both type sysres and disk usage
+    start_time = time 
     mean_val = val_cls(mean(values))
     min_val = val_cls(min(values))
     max_val = val_cls(max(values))
-- 
2.44.0

[PATCH v2 4/5] oe-build-perf-report: Update chart tooltip and chart type

[Ninette Adhikari]

- Update chart tooltip format to show value as size in MB for 'rootfs size'
and timestamp for 'tmpdir size'
- Add commit number to tooltip
- Update chart type to 'step chart' instead of 'line chart'

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 .../build_perf/html/measurement_chart.html    | 31 ++++++++++++++-----
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 9acb3785e2..7982ec39c2 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -10,13 +10,19 @@
     return time[0]*60 + time[1] + time[2]/60 + time[3]/3600;
   }
 
+  // Update value format to either minutes or leave as size value
+  const updateValue = (value) => {
+    // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
+    return Array.isArray(value) ? convertToMinute(value) : value
+  }
+
   // Convert raw data to the format: [time, value]
   const data = rawData.map(([commit, value, time]) => {
     return [
       // The Date object takes values in milliseconds rather than seconds. So to use a Unix timestamp we have to multiply it by 1000.
       new Date(time * 1000).getTime(),
       // Assuming the array values are duration in the format [hours, minutes, seconds, milliseconds]
-      Array.isArray(value) ? convertToMinute(value) : value
+      updateValue(value)
     ]
   });
 
@@ -25,11 +31,22 @@
     tooltip: {
       trigger: 'axis',
       valueFormatter: (value) => {
-        const hours = Math.floor(value/60)
-        const minutes = Math.floor(value % 60)
-        const seconds = Math.floor((value * 60) % 60)
-        return hours + ':' + minutes + ':' + seconds
-      }
+        const commitNumber  = rawData.filter(([commit, dataValue, time]) => updateValue(dataValue) === value)
+        if ('{{ measurement.value_type.quantity }}' == 'time') {
+          const hours = Math.floor(value/60)
+          const minutes = Math.floor(value % 60)
+          const seconds = Math.floor((value * 60) % 60)
+          return [
+                hours + ':' + minutes + ':' + seconds + ', ' +
+                'commit number: ' + commitNumber[0][0]
+              ]
+        }
+        return [
+          value.toFixed(2) + ' MB' + ', ' +
+          'commit number: ' + commitNumber[0][0]
+        ]
+      },
+
     },
     xAxis: {
       type: 'time',
@@ -55,7 +72,7 @@
       {
         name: '{{ measurement.value_type.quantity }}',
         type: 'line',
-        smooth: true,
+        step: 'start',
         symbol: 'none',
         data: data
       }
-- 
2.44.0

[PATCH v2 5/5] oe-build-perf-report: Add dark mode

[Ninette Adhikari]

Update css to add dark mode when window prefers-color-scheme is dark.

Signed-off-by: Ninette Adhikari <ninette@thehoodiefirm.com>
---
 .../build_perf/html/measurement_chart.html    | 15 +++++---
 scripts/lib/build_perf/html/report.html       | 34 +++++++++++++++----
 2 files changed, 39 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 7982ec39c2..ad4a93ed02 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -81,13 +81,20 @@
 
   // Draw chart
   const chart_div = document.getElementById('{{ chart_elem_id }}');
-  const measurement_chart= echarts.init(chart_div, null, {
-    height: 320
-  });
+  // Set dark mode
+  let measurement_chart
+  if (window.matchMedia('(prefers-color-scheme: dark)').matches) {
+      measurement_chart= echarts.init(chart_div, 'dark', {
+      height: 320
+    });
+  } else {
+      measurement_chart= echarts.init(chart_div, null, {
+      height: 320
+    });
+  }
   // Change chart size with browser resize
   window.addEventListener('resize', function() {
     measurement_chart.resize();
   });
   measurement_chart.setOption(option);
 </script>
-
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index 4cd240760a..537ed3ee52 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -19,6 +19,15 @@
 
 {# Styles #}
 <style>
+:root {
+  --text: #000;
+  --bg: #fff;
+  --h2heading: #707070;
+  --link: #0000EE;
+  --trtopborder: #9ca3af;
+  --trborder: #e5e7eb;
+  --chartborder: #f0f0f0;
+  }
 .meta-table {
   font-size: 14px;
   text-align: left;
@@ -31,7 +40,7 @@
 }
 .measurement {
   padding: 8px 0px 8px 8px;
-  border: 2px solid #f0f0f0;
+  border: 2px solid var(--chartborder);
   margin: 1.5rem 0;
 }
 .details {
@@ -61,6 +70,8 @@
 body {
   font-family: 'Helvetica', sans-serif;
   margin: 3rem 8rem;
+  background-color: var(--bg);
+  color: var(--text);
 }
 h1 {
   text-align: center;
@@ -68,13 +79,13 @@ h1 {
 h2 {
   font-size: 1.5rem;
   margin-bottom: 0px;
-  color: #707070;
+  color: var(--h2heading);
   padding-top: 1.5rem;
 }
 h3 {
   font-size: 1.3rem;
   margin: 0px;
-  color: #707070;
+  color: var(--h2heading);
   padding: 1.5rem 0;
 }
 h4 {
@@ -89,10 +100,10 @@ table {
   line-height: 2rem;
 }
 tr {
-  border-bottom: 1px solid #e5e7eb;
+  border-bottom: 1px solid var(--trborder);
 }
 tr:first-child {
-  border-bottom: 1px solid #9ca3af;
+  border-bottom: 1px solid var(--trtopborder);
 }
 tr:last-child {
   border-bottom: none;
@@ -100,11 +111,22 @@ tr:last-child {
 a {
   text-decoration: none;
   font-weight: bold;
-  color: #0000EE;
+  color: var(--link);
 }
 a:hover {
   color: #8080ff;
 }
+@media (prefers-color-scheme: dark) {
+  :root {
+    --text: #e9e8fa;
+    --bg: #0F0C28;
+    --h2heading: #B8B7CB;
+    --link: #87cefa;
+    --trtopborder: #394150;
+    --trborder: #212936;
+    --chartborder: #b1b0bf;
+  }
+}
 </style>
 
 <title>{{ title }}</title>
-- 
2.44.0

Re: [PATCH v2 0/5] Improvements for performance test report view

[Randy MacLeod]

[-- Attachment #1: Type: text/plain, Size: 8150 bytes --]

On 2024-05-03 10:43 a.m., Ninette Adhikari wrote:
> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
> The current report can be accessed here:
> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes.
> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.
>
> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:
>
> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts.
> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
The tooltips are very nice and useful so
> - The charts are created as step charts instead of plain line charts.
> - Add start time, size, and commit number to tooltip.
> - Add dark mode view
>
> Updated report screenshots:
> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
> https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003
> https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a
> https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208


Very nice! Thanks for the work and the v2 improvements.


I applied the patches by saving them from email and there was a minor 
whitespace warning which
is really nothing to worry about but if you want to avoid such noise, 
you could send yourself
patches by email and then apply them in a different branch if you want 
to check for such problems.


❯ git am /tmp/ninette-v2/*
Applying: oe-build-perf-report: Add apache echarts to make report 
interactive
Applying: oe-build-perf-report: Display more than 300 commits and date 
instead of commit number
Applying: oe-build-perf-report: Improve report styling and add descriptions
.git/rebase-apply/patch:240: trailing whitespace.
     start_time = time
warning: 1 line adds whitespace errors.
Applying: oe-build-perf-report: Update chart tooltip and chart type
Applying: oe-build-perf-report: Add dark mode

poky.git on ninette-v2 [$?]
❯ ls /tmp/ninette-v2/*
'/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add 
apache echarts to make report interactive - "Ninette Adhikari via 
lists.openembedded.org" 
<ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml'
'/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more than 
300 commits and date instead of commit number - Ninette Adhikari 
<ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
'/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report 
styling and add descriptions - Ninette Adhikari 
<ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
'/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart 
tooltip and chart type - Ninette Adhikari <ninette@thehoodiefirm.com> - 
2024-05-03 1043.eml'
'/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode - 
Ninette Adhikari <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'

>
> For local setup, you can do the following:
>
> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)
>
> 2. In the poky repository run the following to build the report HTML:
> ```bash
> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html

These are the same setup step as last time I think and again, they 
aren't quite right but
the script tells you do do:

❯ git fetch origin 
refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86


and that works.


> ```
> Note:
> - Add your local path to the yocto-buildstats repo
> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.

Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a 
commit ID in the yocto-buildstats repo.


One thing that you can think about fixing or just ignore is that the 
--commit arg needs to be the
full commit hash not the shortened version. i.e. this doesn't work:

❯ ./scripts/oe-build-perf-report -r 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" 
--branch "master" --commit "663f180574" --branch2 "master" --html > 
/tmp/ninette-v2-test-663f180574.html
Traceback (most recent call last):
   File 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", 
line 617, in <module>
     sys.exit(main())
   File 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report", 
line 532, in main
     index1 = gitarchive.rev_find(revs, 'commit', args.commit)
   File 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py", 
line 282, in rev_find
     raise ValueError("Unable to find '{}' value '{}'".format(attr, val))
ValueError: Unable to find 'commit' value '663f180574'

Since the script is going to be used in other scripts run on the 
autobuilder, this is a low priority enhancement.


I was happy to see that a newer poky commit, worked fine:

❯ ./scripts/oe-build-perf-report -r 
"/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git" 
--branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787" 
--branch2 "master" --html > 
/tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html
INFO: Parsing buildstats from 
'refs/notes/buildstats/perf-debian11/master/qemux86'


but I did notice:  Stdev: nan


whereas in your linked charts, and in my first run with commit: 
663f1805742ff6fb6955719d0ab7846a425debcf

the correct numerical standard deviation is displayed.

I can debug this is you can't reproduce it yourself.


Again. these reports and beautiful interactive charts are great to see.

Thanks!

../Randy


>
>
> Ninette Adhikari (5):
>    oe-build-perf-report: Add apache echarts to make report interactive
>    oe-build-perf-report: Display more than 300 commits and date instead
>      of commit number
>    oe-build-perf-report: Improve report styling and add descriptions
>    oe-build-perf-report: Update chart tooltip and chart type
>    oe-build-perf-report: Add dark mode
>
>   .../build_perf/html/measurement_chart.html    | 140 ++++++++++++------
>   scripts/lib/build_perf/html/report.html       | 124 +++++++++++-----
>   scripts/lib/build_perf/report.py              |   5 +-
>   scripts/oe-build-perf-report                  |   6 +-
>   4 files changed, 193 insertions(+), 82 deletions(-)
>

-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2.1: Type: text/html, Size: 12105 bytes --]

[-- Attachment #2.2: ipZPodcPGxdg4E4H.png --]
[-- Type: image/png, Size: 65076 bytes --]