[PATCH 0/7] Build perf test enhancements

[PATCH 0/7] Build perf test enhancements

[Markus Lehtonen]

This patchset contains some fixes and new features to build performance tests.

The first three patches fix some minor problems in the oe-build-perf-report
script. The last four patches implement new features (plus fix one bug) to the
build perf test wrapper script - most notably adding support for pushing data
to remote git and sending test reports by email.


The following changes since commit 3b7111b30dbd9a4cdd141b594164da18c15ae970:

  automake: Adjust shebang lines to remove interpreter path hardcode (2017-04-01 08:22:04 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib marquiz/buildperf/scripts
  http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/buildperf/scripts


Markus Lehtonen (7):
  scripts/oe-build-perf-report: better guessing of args
  scripts/oe-build-perf-report: fix 'charts ready' console message
  scripts/oe-build-perf-report: fix typo
  build-perf-test-wrapper.sh: make it possible to specify Git branch
    name
  build-perf-test-wrapper.sh: correctly check test script exit status
  build-perf-test-wrapper.sh: support pushing to remote Git
  build-perf-test-wrapper.sh: support sending email reports

 scripts/contrib/build-perf-test-wrapper.sh         | 70 +++++++++++++++++-----
 scripts/lib/build_perf/html/measurement_chart.html |  6 +-
 scripts/lib/build_perf/html/report.html            |  5 +-
 scripts/oe-build-perf-report                       |  4 +-
 4 files changed, 62 insertions(+), 23 deletions(-)

-- 
2.10.2

[PATCH 1/7] scripts/oe-build-perf-report: better guessing of args

[Markus Lehtonen]

When getting info from the latest commit, don't search all refs but only
branches. We don't get correct data from refs/tags/* or refs/notest/*,
for example.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-report | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report
index 3976613..ca9cf1d 100755
--- a/scripts/oe-build-perf-report
+++ b/scripts/oe-build-perf-report
@@ -380,7 +380,7 @@ def auto_args(repo, args):
     """Guess arguments, if not defined by the user"""
     # Get the latest commit in the repo
     log.debug("Guessing arguments from the latest commit")
-    msg = repo.run_cmd(['log', '-1', '--all', '--format=%b'])
+    msg = repo.run_cmd(['log', '-1', '--branches', '--format=%b'])
     for line in msg.splitlines():
         split = line.split(':', 1)
         if len(split) != 2:
-- 
2.10.2

[PATCH 2/7] scripts/oe-build-perf-report: fix 'charts ready' console message

[Markus Lehtonen]

The javascript console log messages are used in scraping, when
converting an html test report to html email. Before this patch a
console message indicating that all charts have been drawn was not
correctly sent if the last test failed (or didn't have chart data for
some other reason) which, in turn, caused oe-build-perf-report-email.py
script to fail with a timeout.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/lib/build_perf/html/measurement_chart.html | 6 +++---
 scripts/lib/build_perf/html/report.html            | 5 +----
 2 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/build_perf/html/measurement_chart.html b/scripts/lib/build_perf/html/measurement_chart.html
index 26fe145..65f1a22 100644
--- a/scripts/lib/build_perf/html/measurement_chart.html
+++ b/scripts/lib/build_perf/html/measurement_chart.html
@@ -1,4 +1,5 @@
 <script type="text/javascript">
+  chartsDrawing += 1;
   google.charts.setOnLoadCallback(drawChart_{{ chart_elem_id }});
   function drawChart_{{ chart_elem_id }}() {
     var data = new google.visualization.DataTable();
@@ -39,10 +40,9 @@
       png_div = document.getElementById('{{ chart_elem_id }}_png');
       png_div.outerHTML = '<a id="{{ chart_elem_id }}_png" href="' + chart.getImageURI() + '">PNG</a>';
       console.log("CHART READY: {{ chart_elem_id }}");
-      {% if last_chart == true %}
+      chartsDrawing -= 1;
+      if (chartsDrawing == 0)
         console.log("ALL CHARTS READY");
-      {% endif %}
-      //console.log(chart_div.innerHTML);
     });
     chart.draw(data, options);
 }
diff --git a/scripts/lib/build_perf/html/report.html b/scripts/lib/build_perf/html/report.html
index e428711..165cbb8 100644
--- a/scripts/lib/build_perf/html/report.html
+++ b/scripts/lib/build_perf/html/report.html
@@ -6,17 +6,14 @@
 <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
 <script type="text/javascript">
 google.charts.load('current', {'packages':['corechart']});
+var chartsDrawing = 0;
 </script>
 
 {# Render measurement result charts #}
 {% for test in test_data %}
-  {% set test_loop = loop %}
   {% if test.status == 'SUCCESS' %}
     {% for measurement in test.measurements %}
       {% set chart_elem_id = test.name + '_' + measurement.name + '_chart' %}
-      {% if test_loop.last and loop.last %}
-        {% set last_chart = true %}
-      {% endif %}
       {% include 'measurement_chart.html' %}
     {% endfor %}
   {% endif %}
-- 
2.10.2

[PATCH 3/7] scripts/oe-build-perf-report: fix typo

[Markus Lehtonen]

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/oe-build-perf-report | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/oe-build-perf-report b/scripts/oe-build-perf-report
index ca9cf1d..6f2db49 100755
--- a/scripts/oe-build-perf-report
+++ b/scripts/oe-build-perf-report
@@ -234,7 +234,7 @@ def print_diff_report(metadata_l, data_l, metadata_r, data_r):
     meta_diff = metadata_diff(metadata_l, metadata_r)
     rows = []
     row_fmt = ['{:{wid}} ', '{:<{wid}}   ', '{:<{wid}}']
-    rows = [['', 'CURRENT COMMIT', 'OOMPARING WITH']]
+    rows = [['', 'CURRENT COMMIT', 'COMPARING WITH']]
     for key, val in meta_diff.items():
         # Shorten commit hashes
         if key == 'commit':
-- 
2.10.2

[PATCH 4/7] build-perf-test-wrapper.sh: make it possible to specify Git branch name

[Markus Lehtonen]

Support <branch>:<commit> format for the -c argument. This makes it
possible to test older commits of a certain branch (not just the tip of
it) so that the branch name will still be correctly recorded in the test
report data.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 31 ++++++++++++++++++++++++------
 1 file changed, 25 insertions(+), 6 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index 234652d..7b7e4dd 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -29,7 +29,8 @@ Optional arguments:
   -h                show this help and exit.
   -a ARCHIVE_DIR    archive results tarball here, give an empty string to
                     disable tarball archiving (default: $archive_dir)
-  -c COMMITISH      test (checkout) this commit
+  -c COMMITISH      test (checkout) this commit, <branch>:<commit> can be
+                    specified to test specific commit of certain branch
   -C GIT_REPO       commit results into Git
   -w WORK_DIR       work dir for this script
                     (default: GIT_TOP_DIR/build-perf-test)
@@ -90,15 +91,33 @@ fi
 cd "$git_topdir"
 
 if [ -n "$commitish" ]; then
-    # Checkout correct revision
-    echo "Checking out $commitish"
+    echo "Running git fetch"
     git fetch &> /dev/null
     git checkout HEAD^0 &> /dev/null
-    git branch -D $commitish &> /dev/null
-    if ! git checkout -f $commitish &> /dev/null; then
-        echo "Git checkout failed"
+
+    # Handle <branch>:<commit> format
+    if echo "$commitish" | grep -q ":"; then
+        commit=`echo "$commitish" | cut -d":" -f2`
+        branch=`echo "$commitish" | cut -d":" -f1`
+    else
+        commit="$commitish"
+        branch="$commitish"
+    fi
+
+    echo "Checking out $commitish"
+    git branch -D $branch &> /dev/null
+    if ! git checkout -f $branch &> /dev/null; then
+        echo "ERROR: Git checkout failed"
+        exit 1
+    fi
+
+    # Check that the specified branch really contains the commit
+    commit_hash=`git rev-parse --revs-only $commit --`
+    if [ -z "$commit_hash" -o "`git merge-base $branch $commit`" != "$commit_hash" ]; then
+        echo "ERROR: branch $branch does not contain commit $commit"
         exit 1
     fi
+    git reset --hard $commit > /dev/null
 fi
 
 # Setup build environment
-- 
2.10.2

[PATCH 5/7] build-perf-test-wrapper.sh: correctly check test script exit status

[Markus Lehtonen]

Test of the exit code was accidentally moved to wrong place when
oe-git-archive was taken into use.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index 7b7e4dd..e804ac2 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -155,6 +155,14 @@ oe-build-perf-test --out-dir "$results_dir" \
                    "${oe_build_perf_test_extra_opts[@]}" \
                    --lock-file "$base_dir/oe-build-perf.lock"
 
+case $? in
+    1)  echo "ERROR: oe-build-perf-test script failed!"
+        exit 1
+        ;;
+    2)  echo "NOTE: some tests failed!"
+        ;;
+esac
+
 # Commit results to git
 if [ -n "$results_repo" ]; then
     echo -e "\nArchiving results in $results_repo"
@@ -167,14 +175,6 @@ if [ -n "$results_repo" ]; then
         "$results_dir"
 fi
 
-case $? in
-    1)  echo "ERROR: oe-build-perf-test script failed!"
-        exit 1
-        ;;
-    2)  echo "NOTE: some tests failed!"
-        ;;
-esac
-
 echo -ne "\n\n-----------------\n"
 echo "Global results file:"
 echo -ne "\n"
-- 
2.10.2

[PATCH 6/7] build-perf-test-wrapper.sh: support pushing to remote Git

[Markus Lehtonen]

Implement new '-P' option for spefifying a Git remote where to push
results after committing to a local Git repository.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index e804ac2..c847197 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -32,6 +32,7 @@ Optional arguments:
   -c COMMITISH      test (checkout) this commit, <branch>:<commit> can be
                     specified to test specific commit of certain branch
   -C GIT_REPO       commit results into Git
+  -P GIT_REMOTE     push results to a remote Git repository
   -w WORK_DIR       work dir for this script
                     (default: GIT_TOP_DIR/build-perf-test)
   -x                create xml report (instead of json)
@@ -42,7 +43,8 @@ EOF
 # Parse command line arguments
 commitish=""
 oe_build_perf_test_extra_opts=()
-while getopts "ha:c:C:w:x" opt; do
+oe_git_archive_extra_opts=()
+while getopts "ha:c:C:P:w:x" opt; do
     case $opt in
         h)  usage
             exit 0
@@ -53,6 +55,8 @@ while getopts "ha:c:C:w:x" opt; do
             ;;
         C)  results_repo=`realpath -s "$OPTARG"`
             ;;
+        P)  oe_git_archive_extra_opts+=("--push" "$OPTARG")
+            ;;
         w)  base_dir=`realpath -s "$OPTARG"`
             ;;
         x)  oe_build_perf_test_extra_opts+=("--xml")
@@ -172,6 +176,7 @@ if [ -n "$results_repo" ]; then
         --tag-name "{hostname}/{branch}/{machine}/{commit_count}-g{commit}/{tag_number}" \
         --exclude "buildstats.json" \
         --notes "buildstats/{branch_name}" "$results_dir/buildstats.json" \
+        "${oe_git_archive_extra_opts[@]}" \
         "$results_dir"
 fi
 
-- 
2.10.2

[PATCH 7/7] build-perf-test-wrapper.sh: support sending email reports

[Markus Lehtonen]

Add new '-E' command line option for sending an email report to
specified recipient.

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
---
 scripts/contrib/build-perf-test-wrapper.sh | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/scripts/contrib/build-perf-test-wrapper.sh b/scripts/contrib/build-perf-test-wrapper.sh
index c847197..49bc298 100755
--- a/scripts/contrib/build-perf-test-wrapper.sh
+++ b/scripts/contrib/build-perf-test-wrapper.sh
@@ -19,6 +19,7 @@
 # oe-build-perf-test and archives the results.
 
 script=`basename $0`
+script_dir=$(realpath $(dirname $0))
 archive_dir=~/perf-results/archives
 
 usage () {
@@ -32,6 +33,7 @@ Optional arguments:
   -c COMMITISH      test (checkout) this commit, <branch>:<commit> can be
                     specified to test specific commit of certain branch
   -C GIT_REPO       commit results into Git
+  -E EMAIL_ADDR     send email report
   -P GIT_REMOTE     push results to a remote Git repository
   -w WORK_DIR       work dir for this script
                     (default: GIT_TOP_DIR/build-perf-test)
@@ -39,12 +41,16 @@ Optional arguments:
 EOF
 }
 
+get_os_release_var () {
+    ( source /etc/os-release; eval echo '$'$1 )
+}
+
 
 # Parse command line arguments
 commitish=""
 oe_build_perf_test_extra_opts=()
 oe_git_archive_extra_opts=()
-while getopts "ha:c:C:P:w:x" opt; do
+while getopts "ha:c:C:E:P:w:x" opt; do
     case $opt in
         h)  usage
             exit 0
@@ -55,6 +61,8 @@ while getopts "ha:c:C:P:w:x" opt; do
             ;;
         C)  results_repo=`realpath -s "$OPTARG"`
             ;;
+        E)  email_to="$OPTARG"
+            ;;
         P)  oe_git_archive_extra_opts+=("--push" "$OPTARG")
             ;;
         w)  base_dir=`realpath -s "$OPTARG"`
@@ -178,8 +186,18 @@ if [ -n "$results_repo" ]; then
         --notes "buildstats/{branch_name}" "$results_dir/buildstats.json" \
         "${oe_git_archive_extra_opts[@]}" \
         "$results_dir"
+
+    # Send email report
+    if [ -n "$email_to" ]; then
+        echo -e "\nEmailing test report"
+        os_name=`get_os_release_var PRETTY_NAME`
+        oe-build-perf-report -r "$results_repo" > report.txt
+        oe-build-perf-report -r "$results_repo" --html > report.html
+        "$script_dir"/oe-build-perf-report-email.py --to "$email_to" --subject "Build Perf Test Report for $os_name" --text report.txt --html report.html
+    fi
 fi
 
+
 echo -ne "\n\n-----------------\n"
 echo "Global results file:"
 echo -ne "\n"
-- 
2.10.2