* [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools
@ 2020-08-28 10:40 Ahmed Karaman
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
` (8 more replies)
0 siblings, 9 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Greetings,
This series includes all of the scripts, tools, and benchmarks
developed in the TCG Continuous Benchmarking GSoC project for 2020.
The series includes one patch for updating the MAINTAINERS file and
eight patches each with a separate script or tool.
All scripts and tools were thoroughly introduced, explained, and
utilized in the project weekly reports which were posted here on the
list each Monday at 12:30PM for the last three months.
Reports:
Report 1 - Measuring Basic Performance Metrics of QEMU:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg714853.html
Introduced tools:
topN_perf.py and topN_callgrind.py
Report 2 - Dissecting QEMU Into Three Main Parts:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg717592.html
Introduced tools:
dissect.py
Report 3 - QEMU 5.0 and 5.1-pre-soft-freeze Dissect Comparison:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg720321.html
Report 4 - Listing QEMU Helpers and Function Callees:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg722559.html
Introduced tools:
list_fn_callees.py and list_helpers.py
Report 5 - Finding Commits Affecting QEMU Performance:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg724080.html
Introduced tools:
bisect.py
Report 6 - Performance Comparison of Two QEMU Builds:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg725700.html
Report 7 - Measuring QEMU Emulation Efficiency:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg727147.html
Report 8 - QEMU Nightly Performance Tests:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg730444.html
Introduced tools:
nightly-tests/
Report 9 - Measuring QEMU Performance in System Mode:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg732729.html
Introduced tools:
topN_system.py
Report 10 - Measuring QEMU Performance in System Mode - Part Two:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg734485.html
Best regards,
Ahmed Karaman
Ahmed Karaman (9):
scripts/performance: Refactor topN_perf.py
scripts/performance: Refactor topN_callgrind.py
scripts/performance: Refactor dissect.py
scripts/performance: Add list_fn_callees.py script
scripts/performance: Add list_helpers.py script
scripts/performance: Add bisect.py script
tests/performance: Add nightly tests
MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests'
subsection
scripts/performance: Add topN_system.py script
MAINTAINERS | 32 +-
scripts/performance/bisect.py | 425 ++++++++
scripts/performance/dissect.py | 123 +--
scripts/performance/list_fn_callees.py | 245 +++++
scripts/performance/list_helpers.py | 221 +++++
scripts/performance/topN_callgrind.py | 169 ++--
scripts/performance/topN_perf.py | 174 ++--
scripts/performance/topN_system.py | 158 +++
tests/performance/nightly-tests/README.md | 243 +++++
.../source/dijkstra_double/dijkstra_double.c | 194 ++++
.../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
.../source/matmult_double/matmult_double.c | 123 +++
.../source/matmult_int32/matmult_int32.c | 121 +++
.../source/qsort_double/qsort_double.c | 104 ++
.../source/qsort_int32/qsort_int32.c | 103 ++
.../source/qsort_string/qsort_string.c | 122 +++
.../source/search_string/search_string.c | 110 +++
.../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
.../scripts/run_nightly_tests.py | 135 +++
.../nightly-tests/scripts/send_email.py | 56 ++
20 files changed, 3744 insertions(+), 226 deletions(-)
create mode 100755 scripts/performance/bisect.py
create mode 100755 scripts/performance/list_fn_callees.py
create mode 100755 scripts/performance/list_helpers.py
create mode 100755 scripts/performance/topN_system.py
create mode 100644 tests/performance/nightly-tests/README.md
create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py
create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py
create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
--
2.17.1
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 1/9] scripts/performance: Refactor topN_perf.py
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-09-07 20:52 ` Aleksandar Markovic
` (2 more replies)
2020-08-28 10:40 ` [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Ahmed Karaman
` (7 subsequent siblings)
8 siblings, 3 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/topN_perf.py | 174 +++++++++++++++----------------
1 file changed, 87 insertions(+), 87 deletions(-)
diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_perf.py
index 07be195fc8..56b100da87 100755
--- a/scripts/performance/topN_perf.py
+++ b/scripts/performance/topN_perf.py
@@ -1,72 +1,77 @@
#!/usr/bin/env python3
-# Print the top N most executed functions in QEMU using perf.
-# Syntax:
-# topN_perf.py [-h] [-n] <number of displayed top functions> -- \
-# <qemu executable> [<qemu executable options>] \
-# <target executable> [<target execurable options>]
-#
-# [-h] - Print the script arguments help message.
-# [-n] - Specify the number of top functions to print.
-# - If this flag is not specified, the tool defaults to 25.
-#
-# Example of usage:
-# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
-#
-# This file is a part of the project "TCG Continuous Benchmarking".
-#
-# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the top N most executed functions in QEMU using perf.
+
+Syntax:
+topN_perf.py [-h] [-n <number of displayed top functions>] -- \
+ <qemu executable> [<qemu executable options>] \
+ <target executable> [<target execurable options>]
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+ - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
import argparse
import os
import subprocess
import sys
+import tempfile
# Parse the command line arguments
-parser = argparse.ArgumentParser(
- usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- '
+PARSER = argparse.ArgumentParser(
+ usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- '
'<qemu executable> [<qemu executable options>] '
'<target executable> [<target executable options>]')
-parser.add_argument('-n', dest='top', type=int, default=25,
+PARSER.add_argument('-n', dest='top', type=int, default=25,
help='Specify the number of top functions to print.')
-parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
+PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
-args = parser.parse_args()
+ARGS = PARSER.parse_args()
# Extract the needed variables from the args
-command = args.command
-top = args.top
+COMMAND = ARGS.command
+TOP = ARGS.top
# Insure that perf is installed
-check_perf_presence = subprocess.run(["which", "perf"],
- stdout=subprocess.DEVNULL)
-if check_perf_presence.returncode:
+CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+if CHECK_PERF_PRESENCE.returncode:
sys.exit("Please install perf before running the script!")
# Insure user has previllage to run perf
-check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
+CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
stdout=subprocess.DEVNULL,
- stderr=subprocess.DEVNULL)
-if check_perf_executability.returncode:
- sys.exit(
-"""
+ stderr=subprocess.DEVNULL,
+ check=False)
+if CHECK_PERF_EXECUTABILITY.returncode:
+ sys.exit("""
Error:
You may not have permission to collect stats.
@@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
kernel.perf_event_paranoid = -1
* Alternatively, you can run this script under sudo privileges.
-"""
-)
-
-# Run perf record
-perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] +
- command),
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE)
-if perf_record.returncode:
- os.unlink('/tmp/perf.data')
- sys.exit(perf_record.stderr.decode("utf-8"))
-
-# Save perf report output to /tmp/perf_report.out
-with open("/tmp/perf_report.out", "w") as output:
- perf_report = subprocess.run(
- ["perf", "report", "--input=/tmp/perf.data", "--stdio"],
- stdout=output,
- stderr=subprocess.PIPE)
- if perf_report.returncode:
- os.unlink('/tmp/perf.data')
- output.close()
- os.unlink('/tmp/perf_report.out')
- sys.exit(perf_report.stderr.decode("utf-8"))
-
-# Read the reported data to functions[]
-functions = []
-with open("/tmp/perf_report.out", "r") as data:
- # Only read lines that are not comments (comments start with #)
- # Only read lines that are not empty
- functions = [line for line in data.readlines() if line and line[0]
- != '#' and line[0] != "\n"]
-
-# Limit the number of top functions to "top"
-number_of_top_functions = top if len(functions) > top else len(functions)
-
-# Store the data of the top functions in top_functions[]
-top_functions = functions[:number_of_top_functions]
+""")
+
+# Run perf and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+ RECORD_PATH = os.path.join(tmpdir, "record.data")
+ REPORT_PATH = os.path.join(tmpdir, "report.txt")
+
+ PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] +
+ COMMAND),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if PERF_RECORD.returncode:
+ sys.exit(PERF_RECORD.stderr.decode("utf-8"))
+
+ with open(REPORT_PATH, "w") as output:
+ PERF_REPORT = subprocess.run(
+ ["perf", "report", "--input="+RECORD_PATH, "--stdio"],
+ stdout=output,
+ stderr=subprocess.PIPE,
+ check=False)
+ if PERF_REPORT.returncode:
+ sys.exit(PERF_REPORT.stderr.decode("utf-8"))
+
+ # Save the reported data to FUNCTIONS[]
+ with open(REPORT_PATH, "r") as data:
+ # Only read lines that are not comments (comments start with #)
+ # Only read lines that are not empty
+ FUNCTIONS = [line for line in data.readlines() if line and
+ line[0] != '#' and line[0] != "\n"]
+
+# Limit the number of top functions to "TOP"
+NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
+
+# Store the data of the top functions in TOP_FUNCTIONS[]
+TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
# Print table header
print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
@@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
'-' * 25))
# Print top N functions
-for (index, function) in enumerate(top_functions, start=1):
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
function_data = function.split()
function_percentage = function_data[0]
function_name = function_data[-1]
@@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1):
function_percentage,
function_name,
function_invoker))
-
-# Remove intermediate files
-os.unlink('/tmp/perf.data')
-os.unlink('/tmp/perf_report.out')
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-09-07 20:53 ` Aleksandar Markovic
2020-08-28 10:40 ` [PATCH 3/9] scripts/performance: Refactor dissect.py Ahmed Karaman
` (6 subsequent siblings)
8 siblings, 1 reply; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Use 'tempfile' instead of '/tmp' for creating temporary files.
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/topN_callgrind.py | 169 +++++++++++++-------------
1 file changed, 87 insertions(+), 82 deletions(-)
diff --git a/scripts/performance/topN_callgrind.py b/scripts/performance/topN_callgrind.py
index 67c59197af..f8a554f393 100755
--- a/scripts/performance/topN_callgrind.py
+++ b/scripts/performance/topN_callgrind.py
@@ -1,113 +1,122 @@
#!/usr/bin/env python3
-# Print the top N most executed functions in QEMU using callgrind.
-# Syntax:
-# topN_callgrind.py [-h] [-n] <number of displayed top functions> -- \
-# <qemu executable> [<qemu executable options>] \
-# <target executable> [<target execurable options>]
-#
-# [-h] - Print the script arguments help message.
-# [-n] - Specify the number of top functions to print.
-# - If this flag is not specified, the tool defaults to 25.
-#
-# Example of usage:
-# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
-#
-# This file is a part of the project "TCG Continuous Benchmarking".
-#
-# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the top N most executed functions in QEMU using callgrind.
+
+Syntax:
+topN_callgrind.py [-h] [-n <number of displayed top functions>] -- \
+ <qemu executable> [<qemu executable options>] \
+ <target executable> [<target execurable options>]
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+ - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
import argparse
import os
import subprocess
import sys
+import tempfile
# Parse the command line arguments
-parser = argparse.ArgumentParser(
- usage='topN_callgrind.py [-h] [-n] <number of displayed top functions> -- '
+PARSER = argparse.ArgumentParser(
+ usage='topN_callgrind.py [-h] [-n] <number of displayed top functions> -- '
'<qemu executable> [<qemu executable options>] '
'<target executable> [<target executable options>]')
-parser.add_argument('-n', dest='top', type=int, default=25,
+PARSER.add_argument('-n', dest='top', type=int, default=25,
help='Specify the number of top functions to print.')
-parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
+PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
-args = parser.parse_args()
+ARGS = PARSER.parse_args()
# Extract the needed variables from the args
-command = args.command
-top = args.top
+COMMAND = ARGS.command
+TOP = ARGS.top
# Insure that valgrind is installed
-check_valgrind_presence = subprocess.run(["which", "valgrind"],
- stdout=subprocess.DEVNULL)
-if check_valgrind_presence.returncode:
+CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+if CHECK_VALGRIND_PRESENCE.returncode:
sys.exit("Please install valgrind before running the script!")
-# Run callgrind
-callgrind = subprocess.run((
- ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/callgrind.data"]
- + command),
- stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE)
-if callgrind.returncode:
- sys.exit(callgrind.stderr.decode("utf-8"))
-
-# Save callgrind_annotate output to /tmp/callgrind_annotate.out
-with open("/tmp/callgrind_annotate.out", "w") as output:
- callgrind_annotate = subprocess.run(["callgrind_annotate",
- "/tmp/callgrind.data"],
- stdout=output,
- stderr=subprocess.PIPE)
- if callgrind_annotate.returncode:
- os.unlink('/tmp/callgrind.data')
- output.close()
- os.unlink('/tmp/callgrind_annotate.out')
- sys.exit(callgrind_annotate.stderr.decode("utf-8"))
-
-# Read the callgrind_annotate output to callgrind_data[]
-callgrind_data = []
-with open('/tmp/callgrind_annotate.out', 'r') as data:
- callgrind_data = data.readlines()
+# Run callgrind and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+ CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data")
+ ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out")
+
+ # Run callgrind
+ CALLGRIND = subprocess.run((["valgrind",
+ "--tool=callgrind",
+ "--callgrind-out-file="+CALLGRIND_DATA_PATH]
+ + COMMAND),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if CALLGRIND.returncode:
+ sys.exit(CALLGRIND.stderr.decode("utf-8"))
+
+ with open(ANNOTATE_OUT_PATH, "w") as output:
+ CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate",
+ CALLGRIND_DATA_PATH],
+ stdout=output,
+ stderr=subprocess.PIPE,
+ check=False)
+ if CALLGRIND_ANNOTATE.returncode:
+ sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8"))
+
+ # Read the callgrind_annotate output to CALLGRIND_DATA[]
+ CALLGRIND_DATA = []
+ with open(ANNOTATE_OUT_PATH, 'r') as data:
+ CALLGRIND_DATA = data.readlines()
# Line number with the total number of instructions
-total_instructions_line_number = 20
+TOTAL_INSTRUCTIONS_LINE_NO = 20
# Get the total number of instructions
-total_instructions_line_data = callgrind_data[total_instructions_line_number]
-total_number_of_instructions = total_instructions_line_data.split(' ')[0]
-total_number_of_instructions = int(
- total_number_of_instructions.replace(',', ''))
+TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO]
+TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0]
+TOTAL_NUMBER_OF_INSTRUCTIONS = int(
+ TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', ''))
# Line number with the top function
-first_func_line = 25
+FIRST_FUNC_LINE = 25
# Number of functions recorded by callgrind, last two lines are always empty
-number_of_functions = len(callgrind_data) - first_func_line - 2
+NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2
# Limit the number of top functions to "top"
-number_of_top_functions = (top if number_of_functions >
- top else number_of_functions)
+NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS >
+ TOP else NUMBER_OF_FUNCTIONS)
# Store the data of the top functions in top_functions[]
-top_functions = callgrind_data[first_func_line:
- first_func_line + number_of_top_functions]
+TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE:
+ FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS]
# Print table header
print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
@@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
))
# Print top N functions
-for (index, function) in enumerate(top_functions, start=1):
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
function_data = function.split()
# Calculate function percentage
function_instructions = float(function_data[0].replace(',', ''))
function_percentage = (function_instructions /
- total_number_of_instructions)*100
+ TOTAL_NUMBER_OF_INSTRUCTIONS)*100
# Get function name and source files path
function_source_file, function_name = function_data[1].split(':')
# Print extracted data
@@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions, start=1):
round(function_percentage, 3),
function_name,
function_source_file))
-
-# Remove intermediate files
-os.unlink('/tmp/callgrind.data')
-os.unlink('/tmp/callgrind_annotate.out')
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 3/9] scripts/performance: Refactor dissect.py
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
2020-08-28 10:40 ` [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-09-02 8:48 ` Aleksandar Markovic
2020-08-28 10:40 ` [PATCH 4/9] scripts/performance: Add list_fn_callees.py script Ahmed Karaman
` (5 subsequent siblings)
8 siblings, 1 reply; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
- Apply pylint and flake8 formatting rules to the script.
- Move syntax and usage exmaple to main() docstring.
- Update get_jit_line() to only detect the main jit call.
- Use mypy for specifying parameters and return types in functions.
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/dissect.py | 123 ++++++++++++++++++---------------
1 file changed, 68 insertions(+), 55 deletions(-)
diff --git a/scripts/performance/dissect.py b/scripts/performance/dissect.py
index bf24f50922..d4df884b75 100755
--- a/scripts/performance/dissect.py
+++ b/scripts/performance/dissect.py
@@ -1,34 +1,27 @@
#!/usr/bin/env python3
-# Print the percentage of instructions spent in each phase of QEMU
-# execution.
-#
-# Syntax:
-# dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
-# <target executable> [<target executable options>]
-#
-# [-h] - Print the script arguments help message.
-#
-# Example of usage:
-# dissect.py -- qemu-arm coulomb_double-arm
-#
-# This file is a part of the project "TCG Continuous Benchmarking".
-#
-# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
-# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+Print the percentage of instructions spent in each phase of QEMU
+execution.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
import argparse
import os
@@ -36,23 +29,26 @@ import subprocess
import sys
import tempfile
+from typing import List
+
-def get_JIT_line(callgrind_data):
+def get_jit_line(callgrind_data: List[str]) -> int:
"""
Search for the first instance of the JIT call in
the callgrind_annotate output when ran using --tree=caller
This is equivalent to the self number of instructions of JIT.
Parameters:
- callgrind_data (list): callgrind_annotate output
+ callgrind_data (List[str]): callgrind_annotate output
Returns:
(int): Line number
"""
line = -1
- for i in range(len(callgrind_data)):
- if callgrind_data[i].strip('\n') and \
- callgrind_data[i].split()[-1] == "[???]":
+ for (i, callgrind_datum) in enumerate(callgrind_data):
+ if callgrind_datum.strip('\n') and \
+ callgrind_datum.split()[-1] == "[???]" and \
+ callgrind_datum.split()[1] == "*":
line = i
break
if line == -1:
@@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data):
def main():
+ """
+ Parse the command line arguments then start the execution.
+ Syntax:
+ dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
+ <target executable> [<target executable options>]
+
+ [-h] - Print the script arguments help message.
+
+ Example of usage:
+ dissect.py -- qemu-arm coulomb_double-arm
+ """
+
# Parse the command line arguments
parser = argparse.ArgumentParser(
usage='dissect.py [-h] -- '
@@ -76,7 +84,7 @@ def main():
# Insure that valgrind is installed
check_valgrind = subprocess.run(
- ["which", "valgrind"], stdout=subprocess.DEVNULL)
+ ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
if check_valgrind.returncode:
sys.exit("Please install valgrind before running the script.")
@@ -93,7 +101,8 @@ def main():
"--callgrind-out-file=" + data_path]
+ command),
stdout=subprocess.DEVNULL,
- stderr=subprocess.PIPE)
+ stderr=subprocess.PIPE,
+ check=False)
if callgrind.returncode:
sys.exit(callgrind.stderr.decode("utf-8"))
@@ -102,7 +111,8 @@ def main():
callgrind_annotate = subprocess.run(
["callgrind_annotate", data_path, "--tree=caller"],
stdout=output,
- stderr=subprocess.PIPE)
+ stderr=subprocess.PIPE,
+ check=False)
if callgrind_annotate.returncode:
sys.exit(callgrind_annotate.stderr.decode("utf-8"))
@@ -120,25 +130,28 @@ def main():
total_instructions = int(total_instructions.replace(',', ''))
# Line number with the JIT self number of instructions
- JIT_self_instructions_line_number = get_JIT_line(callgrind_data)
+ jit_self_instructions_line_number = get_jit_line(callgrind_data)
# Get the JIT self number of instructions
- JIT_self_instructions_line_data = \
- callgrind_data[JIT_self_instructions_line_number]
- JIT_self_instructions = JIT_self_instructions_line_data.split()[0]
- JIT_self_instructions = int(JIT_self_instructions.replace(',', ''))
+ jit_self_instructions_line_data = \
+ callgrind_data[jit_self_instructions_line_number]
+ jit_self_instructions = jit_self_instructions_line_data.split()[0]
+ jit_self_instructions = int(jit_self_instructions.replace(',', ''))
# Line number with the JIT self + inclusive number of instructions
- # It's the line above the first JIT call when running with --tree=caller
- JIT_total_instructions_line_number = JIT_self_instructions_line_number-1
+ # It's the line above the first JIT call when running with
+ # --tree=caller
+ jit_total_instructions_line_number = \
+ jit_self_instructions_line_number-1
# Get the JIT self + inclusive number of instructions
- JIT_total_instructions_line_data = \
- callgrind_data[JIT_total_instructions_line_number]
- JIT_total_instructions = JIT_total_instructions_line_data.split()[0]
- JIT_total_instructions = int(JIT_total_instructions.replace(',', ''))
+ jit_total_instructions_line_data = \
+ callgrind_data[jit_total_instructions_line_number]
+ jit_total_instructions = jit_total_instructions_line_data.split()[0]
+ jit_total_instructions = int(jit_total_instructions.replace(',', ''))
# Calculate number of instructions in helpers and code generation
- helpers_instructions = JIT_total_instructions-JIT_self_instructions
- code_generation_instructions = total_instructions-JIT_total_instructions
+ helpers_instructions = jit_total_instructions-jit_self_instructions
+ code_generation_instructions = \
+ total_instructions-jit_total_instructions
# Print results (Insert commas in large numbers)
# Print total number of instructions
@@ -149,12 +162,12 @@ def main():
print('{:<20}{:>20}\t{:>6.3f}%'.
format("Code Generation:",
format(code_generation_instructions, ","),
- (code_generation_instructions / total_instructions) * 100))
- # Print JIT instructions and percentage
+ (code_generation_instructions/total_instructions)*100))
+ # Print jit instructions and percentage
print('{:<20}{:>20}\t{:>6.3f}%'.
- format("JIT Execution:",
- format(JIT_self_instructions, ","),
- (JIT_self_instructions / total_instructions) * 100))
+ format("jit Execution:",
+ format(jit_self_instructions, ","),
+ (jit_self_instructions / total_instructions) * 100))
# Print helpers instructions and percentage
print('{:<20}{:>20}\t{:>6.3f}%'.
format("Helpers:",
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 4/9] scripts/performance: Add list_fn_callees.py script
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (2 preceding siblings ...)
2020-08-28 10:40 ` [PATCH 3/9] scripts/performance: Refactor dissect.py Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-08-28 10:40 ` [PATCH 5/9] scripts/performance: Add list_helpers.py script Ahmed Karaman
` (4 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Python script that prints the callees of a given list of QEMU
functions.
Syntax:
list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \
<qemu executable> [<qemu executable options>] \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message.
-f FUNCTION [FUNCTION ...] - List of function names
Example of usage:
list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \
qemu-mips coulomb_double-mips -n10
Example output:
Total number of instructions: 108,952,851
Callees of helper_float_sub_d:
No. Instructions Percentage Calls Ins/Call Function Name Source File
--- ------------ ---------- ------ -------- ------------- ---------------
1 153,160 0.141% 1,305 117 float64_sub <qemu>/fpu/softfloat.c
Callees of helper_float_mul_d:
No. Instructions Percentage Calls Ins/Call Function Name Source File
--- ------------ ---------- ------ -------- ------------- ---------------
1 131,137 0.120% 1,014 129 float64_mul <qemu>/fpu/softfloat.c
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/list_fn_callees.py | 245 +++++++++++++++++++++++++
1 file changed, 245 insertions(+)
create mode 100755 scripts/performance/list_fn_callees.py
diff --git a/scripts/performance/list_fn_callees.py b/scripts/performance/list_fn_callees.py
new file mode 100755
index 0000000000..6aa8f6b6ca
--- /dev/null
+++ b/scripts/performance/list_fn_callees.py
@@ -0,0 +1,245 @@
+#!/usr/bin/env python3
+
+"""
+Print the callees of a given list of QEMU functions.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+from typing import List, Union
+
+
+def find_function_lines(function_name: str,
+ callgrind_data: List[str]) -> List[int]:
+ """
+ Search for the line with the function name in the
+ callgrind_annotate output when ran using --tre=calling.
+ All the function callees should be listed after that line.
+
+ Parameters:
+ function_name (string): The desired function name to print its callees
+ callgrind_data (List[str]): callgrind_annotate output
+
+ Returns:
+ (List[int]): List of function line numbers
+ """
+ lines = []
+ for (i, callgrind_datum) in enumerate(callgrind_data):
+ split_line = callgrind_datum.split()
+ if len(split_line) > 2 and \
+ split_line[1] == "*" and \
+ split_line[2].split(":")[-1] == function_name:
+ # Function might be in the callgrind_annotate output more than
+ # once, so don't break after finding an instance
+ if callgrind_data[i + 1] != "\n":
+ # Only append the line number if the found instance has
+ # callees
+ lines.append(i)
+ return lines
+
+
+def get_function_calles(
+ function_lines: List[int],
+ callgrind_data: List[str]) -> List[List[Union[str, int]]]:
+ """
+ Get all callees data for a function given its list of line numbers in
+ callgrind_annotate output.
+
+ Parameters:
+ function_lines (List[int]): Line numbers of the function to get its callees
+ callgrind_data (List[str]): callgrind_annotate output
+
+ Returns:
+ (List[List[Union[str, int]]]):[[number_of_instructions(int),
+ callee_name(str),
+ number_of_calls(int),
+ source_file(str)],
+ ...]
+ """
+ callees: List[List[Union[str, int]]] = []
+ for function_line in function_lines:
+ next_callee = function_line + 1
+ while callgrind_data[next_callee] != "\n":
+ split_line = callgrind_data[next_callee].split()
+ number_of_instructions = int(split_line[0].replace(",", ""))
+ source_file = split_line[2].split(":")[0]
+ callee_name = split_line[2].split(":")[1]
+ number_of_calls = int(split_line[3][1:-2])
+ callees.append([number_of_instructions, callee_name,
+ number_of_calls, source_file])
+ next_callee += 1
+ return sorted(callees, reverse=True)
+
+
+def main():
+ """
+ Parse the command line arguments then start execution.
+
+ Syntax:
+ list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- \
+ <qemu executable> [<qemu executable options>] \
+ <target executable> [<target executable options>]
+
+ [-h] - Print the script arguments help message.
+ -f FUNCTION [FUNCTION ...] - List of function names
+
+ Example of usage:
+ list_fn_callees.py -f helper_float_sub_d helper_float_mul_d -- \
+ qemu-mips coulomb_double-mips
+ """
+
+ # Parse the command line arguments
+ parser = argparse.ArgumentParser(
+ usage="list_fn_callees.py [-h] -f FUNCTION [FUNCTION ...] -- "
+ "<qemu executable> [<qemu executable options>] "
+ "<target executable> [<target executable options>]")
+
+ parser.add_argument("-f", dest="function", type=str,
+ nargs="+", required=True,
+ help="list of function names to print their callees")
+
+ parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+ args = parser.parse_args()
+
+ # Extract the needed variables from the args
+ command = args.command
+ function_names = args.function
+
+ # Insure that valgrind is installed
+ check_valgrind = subprocess.run(
+ ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
+ if check_valgrind.returncode:
+ sys.exit("Please install valgrind before running the script.")
+
+ # Save all intermediate files in a temporary directory
+ with tempfile.TemporaryDirectory() as tmpdirname:
+ # callgrind output file path
+ data_path = os.path.join(tmpdirname, "callgrind.data")
+ # callgrind_annotate output file path
+ annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out")
+
+ # Run callgrind
+ callgrind = subprocess.run((["valgrind",
+ "--tool=callgrind",
+ "--callgrind-out-file=" + data_path]
+ + command),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if callgrind.returncode:
+ sys.exit(callgrind.stderr.decode("utf-8"))
+
+ # Save callgrind_annotate output
+ with open(annotate_out_path, "w") as output:
+ callgrind_annotate = subprocess.run(["callgrind_annotate",
+ data_path,
+ "--threshold=100",
+ "--tree=calling"],
+ stdout=output,
+ stderr=subprocess.PIPE,
+ check=False)
+ if callgrind_annotate.returncode:
+ sys.exit(callgrind_annotate.stderr.decode("utf-8"))
+
+ # Read the callgrind_annotate output to callgrind_data[]
+ callgrind_data = []
+ with open(annotate_out_path, "r") as data:
+ callgrind_data = data.readlines()
+
+ # Line number with the total number of instructions
+ total_instructions_line_number = 20
+ # Get the total number of instructions
+ total_instructions_line_data = \
+ callgrind_data[total_instructions_line_number]
+ total_instructions = total_instructions_line_data.split()[0]
+
+ print("Total number of instructions: {}\n".format(total_instructions))
+
+ # Remove commas and convert to int
+ total_instructions = int(total_instructions.replace(",", ""))
+
+ for function_name in function_names:
+ # Line numbers with the desired function
+ function_lines = find_function_lines(function_name, callgrind_data)
+
+ if len(function_lines) == 0:
+ print("Couldn't locate function: {}.\n".format(
+ function_name))
+ continue
+
+ # Get function callees
+ function_callees = get_function_calles(
+ function_lines, callgrind_data)
+
+ print("Callees of {}:\n".format(function_name))
+
+ # Print table header
+ print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}".
+ format(
+ "No.",
+ "Instructions",
+ "Percentage",
+ "Calls",
+ "Ins/Call",
+ "Function Name",
+ "Source File")
+ )
+
+ print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}".
+ format(
+ "-" * 4,
+ "-" * 15,
+ "-" * 10,
+ "-" * 15,
+ "-" * 10,
+ "-" * 25,
+ "-" * 30)
+ )
+
+ for (index, callee) in enumerate(function_callees, start=1):
+ instructions = callee[0]
+ percentage = (callee[0] / total_instructions) * 100
+ calls = callee[2]
+ instruction_per_call = int(callee[0] / callee[2])
+ function_name = callee[1]
+ source_file = callee[3]
+ # Print extracted data
+ print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}".
+ format(
+ index,
+ format(instructions, ","),
+ round(percentage, 3),
+ format(calls, ","),
+ format(instruction_per_call, ","),
+ function_name,
+ source_file)
+ )
+
+ print("\n")
+
+
+if __name__ == "__main__":
+ main()
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 5/9] scripts/performance: Add list_helpers.py script
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (3 preceding siblings ...)
2020-08-28 10:40 ` [PATCH 4/9] scripts/performance: Add list_fn_callees.py script Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-08-28 10:40 ` [PATCH 6/9] scripts/performance: Add bisect.py script Ahmed Karaman
` (3 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Python script that prints executed helpers of a QEMU invocation.
Syntax:
list_helpers.py [-h] -- \
<qemu executable> [<qemu executable options>] \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message.
Example of usage:
list_helpers.py -- qemu-mips coulomb_double-mips -n10
Example output:
Total number of instructions: 108,933,695
Executed QEMU Helpers:
No. Ins Percent Calls Ins/Call Helper Name Source File
--- ------- ------- ------ -------- -------------------- ---------------
1 183,021 0.168% 1,305 140 helper_float_sub_d <qemu>/target/mips/fpu_helper.c
2 177,111 0.163% 770 230 helper_float_madd_d <qemu>/target/mips/fpu_helper.c
3 171,537 0.157% 1,014 169 helper_float_mul_d <qemu>/target/mips/fpu_helper.c
4 157,298 0.144% 2,443 64 helper_lookup_tb_ptr <qemu>/accel/tcg/tcg-runtime.c
5 138,123 0.127% 897 153 helper_float_add_d <qemu>/target/mips/fpu_helper.c
6 47,083 0.043% 207 227 helper_float_msub_d <qemu>/target/mips/fpu_helper.c
7 24,062 0.022% 487 49 helper_cmp_d_lt <qemu>/target/mips/fpu_helper.c
8 22,910 0.021% 150 152 helper_float_div_d <qemu>/target/mips/fpu_helper.c
9 15,497 0.014% 321 48 helper_cmp_d_eq <qemu>/target/mips/fpu_helper.c
10 9,100 0.008% 52 175 helper_float_trunc_w_d <qemu>/target/mips/fpu_helper.c
11 7,059 0.006% 10 705 helper_float_sqrt_d <qemu>/target/mips/fpu_helper.c
12 3,000 0.003% 40 75 helper_cmp_d_ule <qemu>/target/mips/fpu_helper.c
13 2,720 0.002% 20 136 helper_float_cvtd_w <qemu>/target/mips/fpu_helper.c
14 2,477 0.002% 27 91 helper_swl <qemu>/target/mips/op_helper.c
15 2,000 0.002% 40 50 helper_cmp_d_le <qemu>/target/mips/fpu_helper.c
16 1,800 0.002% 40 45 helper_cmp_d_un <qemu>/target/mips/fpu_helper.c
17 1,164 0.001% 12 97 helper_raise_exception_ <qemu>/target/mips/op_helper.c
18 720 0.001% 10 72 helper_cmp_d_ult <qemu>/target/mips/fpu_helper.c
19 560 0.001% 140 4 helper_cfc1 <qemu>/target/mips/fpu_helper.c
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/list_helpers.py | 221 ++++++++++++++++++++++++++++
1 file changed, 221 insertions(+)
create mode 100755 scripts/performance/list_helpers.py
diff --git a/scripts/performance/list_helpers.py b/scripts/performance/list_helpers.py
new file mode 100755
index 0000000000..823b1cef66
--- /dev/null
+++ b/scripts/performance/list_helpers.py
@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+
+"""
+Print the executed helpers of a QEMU invocation.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+from typing import List, Union
+
+
+def find_jit_line(callgrind_data: List[str]) -> int:
+ """
+ Search for the line with the JIT call in the callgrind_annotate
+ output when ran using --tre=calling.
+ All the helpers should be listed after that line.
+
+ Parameters:
+ callgrind_data (List[str]): callgrind_annotate output
+
+ Returns:
+ (int): Line number of JIT call
+ """
+ line = -1
+ for (i, callgrind_datum) in enumerate(callgrind_data):
+ split_line = callgrind_datum.split()
+ if len(split_line) > 2 and \
+ split_line[1] == "*" and \
+ split_line[-1] == "[???]":
+ line = i
+ break
+ return line
+
+
+def get_helpers(jit_line: int,
+ callgrind_data: List[str]) -> List[List[Union[str, int]]]:
+ """
+ Get all helpers data given the line number of the JIT call.
+
+ Parameters:
+ jit_line (int): Line number of the JIT call
+ callgrind_data (List[str]): callgrind_annotate output
+
+ Returns:
+ (List[List[Union[str, int]]]):[[number_of_instructions(int),
+ helper_name(str),
+ number_of_calls(int),
+ source_file(str)],
+ ...]
+ """
+ helpers: List[List[Union[str, int]]] = []
+ next_helper = jit_line + 1
+ while callgrind_data[next_helper] != "\n":
+ split_line = callgrind_data[next_helper].split()
+ number_of_instructions = int(split_line[0].replace(",", ""))
+ source_file = split_line[2].split(":")[0]
+ callee_name = split_line[2].split(":")[1]
+ number_of_calls = int(split_line[3][1:-2])
+ helpers.append([number_of_instructions, callee_name,
+ number_of_calls, source_file])
+ next_helper += 1
+ return sorted(helpers, reverse=True)
+
+
+def main():
+ """
+ Parse the command line arguments then start execution
+
+ Syntax:
+ list_helpers.py [-h] -- \
+ <qemu executable> [<qemu executable options>] \
+ <target executable> [<target executable options>]
+
+ [-h] - Print the script arguments help message.
+
+ Example of usage:
+ list_helpers.py -- qemu-mips coulomb_double-mips
+ """
+ # Parse the command line arguments
+ parser = argparse.ArgumentParser(
+ usage="list_helpers.py [-h] -- "
+ "<qemu executable> [<qemu executable options>] "
+ "<target executable> [<target executable options>]")
+
+ parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+ args = parser.parse_args()
+
+ # Extract the needed variables from the args
+ command = args.command
+
+ # Insure that valgrind is installed
+ check_valgrind = subprocess.run(
+ ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
+ if check_valgrind.returncode:
+ sys.exit("Please install valgrind before running the script.")
+
+ # Save all intermediate files in a temporary directory
+ with tempfile.TemporaryDirectory() as tmpdirname:
+ # callgrind output file path
+ data_path = os.path.join(tmpdirname, "callgrind.data")
+ # callgrind_annotate output file path
+ annotate_out_path = os.path.join(tmpdirname, "callgrind_annotate.out")
+
+ # Run callgrind
+ callgrind = subprocess.run((["valgrind",
+ "--tool=callgrind",
+ "--callgrind-out-file=" + data_path]
+ + command),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if callgrind.returncode:
+ sys.exit(callgrind.stderr.decode("utf-8"))
+
+ # Save callgrind_annotate output
+ with open(annotate_out_path, "w") as output:
+ callgrind_annotate = subprocess.run(["callgrind_annotate",
+ data_path,
+ "--threshold=100",
+ "--tree=calling"],
+ stdout=output,
+ stderr=subprocess.PIPE,
+ check=False)
+ if callgrind_annotate.returncode:
+ sys.exit(callgrind_annotate.stderr.decode("utf-8"))
+
+ # Read the callgrind_annotate output to callgrind_data[]
+ callgrind_data = []
+ with open(annotate_out_path, "r") as data:
+ callgrind_data = data.readlines()
+
+ # Line number with the total number of instructions
+ total_instructions_line_number = 20
+ # Get the total number of instructions
+ total_instructions_line_data = \
+ callgrind_data[total_instructions_line_number]
+ total_instructions = total_instructions_line_data.split()[0]
+
+ print("Total number of instructions: {}\n".format(total_instructions))
+
+ # Remove commas and convert to int
+ total_instructions = int(total_instructions.replace(",", ""))
+
+ # Line number with the JIT call
+ jit_line = find_jit_line(callgrind_data)
+ if jit_line == -1:
+ sys.exit("Couldn't locate the JIT call ... Exiting")
+
+ # Get helpers
+ helpers = get_helpers(jit_line, callgrind_data)
+
+ print("Executed QEMU Helpers:\n")
+
+ # Print table header
+ print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}".
+ format(
+ "No.",
+ "Instructions",
+ "Percentage",
+ "Calls",
+ "Ins/Call",
+ "Helper Name",
+ "Source File")
+ )
+
+ print("{:>4} {:>15} {:>10} {:>15} {:>10} {:<25} {}".
+ format(
+ "-" * 4,
+ "-" * 15,
+ "-" * 10,
+ "-" * 15,
+ "-" * 10,
+ "-" * 25,
+ "-" * 30)
+ )
+
+ for (index, callee) in enumerate(helpers, start=1):
+ instructions = callee[0]
+ percentage = (callee[0] / total_instructions) * 100
+ calls = callee[2]
+ instruction_per_call = int(callee[0] / callee[2])
+ helper_name = callee[1]
+ source_file = callee[3]
+ # Print extracted data
+ print("{:>4} {:>15} {:>9.3f}% {:>15} {:>10} {:<25} {}".
+ format(
+ index,
+ format(instructions, ","),
+ round(percentage, 3),
+ format(calls, ","),
+ format(instruction_per_call, ","),
+ helper_name,
+ source_file)
+ )
+
+
+if __name__ == "__main__":
+ main()
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 6/9] scripts/performance: Add bisect.py script
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (4 preceding siblings ...)
2020-08-28 10:40 ` [PATCH 5/9] scripts/performance: Add list_helpers.py script Ahmed Karaman
@ 2020-08-28 10:40 ` Ahmed Karaman
2020-08-28 10:41 ` [PATCH 7/9] tests/performance: Add nightly tests Ahmed Karaman
` (2 subsequent siblings)
8 siblings, 0 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:40 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Python script that locates the commit that caused a performance
degradation or improvement in QEMU using the git bisect command
(binary search).
Syntax:
bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \
--target TARGET --tool {perf,callgrind} -- \
<target executable> [<target executable options>]
[-h] - Print the script arguments help message
-s,--start START - First commit hash in the search range
[-e,--end END] - Last commit hash in the search range
(default: Latest commit)
[-q,--qemu QEMU] - QEMU path.
(default: Path to a GitHub QEMU clone)
--target TARGET - QEMU target name
--tool {perf,callgrind} - Underlying tool used for measurements
Example of usage:
bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \
--tool=perf -- coulomb_double-ppc -n 1000
Example output:
Start Commit Instructions: 12,710,790,060
End Commit Instructions: 13,031,083,512
Performance Change: -2.458%
Estimated Number of Steps: 10
*****************BISECT STEP 1*****************
Instructions: 13,031,097,790
Status: slow commit
*****************BISECT STEP 2*****************
Instructions: 12,710,805,265
Status: fast commit
*****************BISECT STEP 3*****************
Instructions: 13,031,028,053
Status: slow commit
*****************BISECT STEP 4*****************
Instructions: 12,711,763,211
Status: fast commit
*****************BISECT STEP 5*****************
Instructions: 13,031,027,292
Status: slow commit
*****************BISECT STEP 6*****************
Instructions: 12,711,748,738
Status: fast commit
*****************BISECT STEP 7*****************
Instructions: 12,711,748,788
Status: fast commit
*****************BISECT STEP 8*****************
Instructions: 13,031,100,493
Status: slow commit
*****************BISECT STEP 9*****************
Instructions: 12,714,472,954
Status: fast commit
****************BISECT STEP 10*****************
Instructions: 12,715,409,153
Status: fast commit
****************BISECT STEP 11*****************
Instructions: 12,715,394,739
Status: fast commit
*****************BISECT RESULT*****************
commit 0673ecdf6cb2b1445a85283db8cbacb251c46516
Author: Richard Henderson <richard.henderson@linaro.org>
Date: Tue May 5 10:40:23 2020 -0700
softfloat: Inline float64 compare specializations
Replace the float64 compare specializations with inline functions
that call the standard float64_compare{,_quiet} functions.
Use bool as the return type.
***********************************************
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/bisect.py | 425 ++++++++++++++++++++++++++++++++++
1 file changed, 425 insertions(+)
create mode 100755 scripts/performance/bisect.py
diff --git a/scripts/performance/bisect.py b/scripts/performance/bisect.py
new file mode 100755
index 0000000000..0c60be22ce
--- /dev/null
+++ b/scripts/performance/bisect.py
@@ -0,0 +1,425 @@
+#!/usr/bin/env python3
+
+"""
+Locate the commit that caused a performance degradation or improvement in
+QEMU using the git bisect command (binary search).
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import multiprocessing
+import tempfile
+import os
+import shutil
+import subprocess
+import sys
+
+from typing import List
+
+
+# --------------------------- GIT WRAPPERS --------------------------
+def git_bisect(qemu_path: str, qemu_build_path: str, command: str,
+ args: List[str] = None) -> str:
+ """
+ Wrapper function for running git bisect.
+
+ Parameters:
+ qemu_path (str): QEMU path
+ qemu_build_path (str): Path to the build directory with configuration files
+ command (str): bisect command (start|fast|slow|reset)
+ args (list): Optional arguments
+
+ Returns:
+ (str): git bisect stdout.
+ """
+ process = ["git", "bisect", command]
+ if args:
+ process += args
+ bisect = subprocess.run(process,
+ cwd=qemu_path,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=False)
+ if bisect.returncode:
+ clean_exit(qemu_build_path, bisect.stderr.decode("utf-8"))
+
+ return bisect.stdout.decode("utf-8")
+
+
+def git_checkout(commit: str, qemu_path: str, qemu_build_path: str) -> None:
+ """
+ Wrapper function for checking out a given git commit.
+
+ Parameters:
+ commit (str): Commit hash of a git commit
+ qemu_path (str): QEMU path
+ qemu_build_path (str): Path to the build directory with configuration files
+ """
+ checkout_commit = subprocess.run(["git",
+ "checkout",
+ commit],
+ cwd=qemu_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if checkout_commit.returncode:
+ clean_exit(qemu_build_path, checkout_commit.stderr.decode("utf-8"))
+
+
+def git_clone(qemu_path: str) -> None:
+ """
+ Wrapper function for cloning QEMU git repo from GitHub.
+
+ Parameters:
+ qemu_path (str): Path to clone the QEMU repo to
+ """
+ clone_qemu = subprocess.run(["git",
+ "clone",
+ "https://github.com/qemu/qemu.git",
+ qemu_path],
+ stderr=subprocess.STDOUT,
+ check=False)
+ if clone_qemu.returncode:
+ sys.exit("Failed to clone QEMU!")
+# -------------------------------------------------------------------
+
+
+def check_requirements(tool: str) -> None:
+ """
+ Verify that all script requirements are installed (perf|callgrind & git).
+
+ Parameters:
+ tool (str): Tool used for the measurement (perf or callgrind)
+ """
+ if tool == "perf":
+ check_perf_installation = subprocess.run(["which", "perf"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if check_perf_installation.returncode:
+ sys.exit("Please install perf before running the script.")
+
+ # Insure user has previllage to run perf
+ check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ check=False)
+ if check_perf_executability.returncode:
+ sys.exit("""
+ Error:
+ You may not have permission to collect stats.
+ Consider tweaking /proc/sys/kernel/perf_event_paranoid,
+ which controls use of the performance events system by
+ unprivileged users (without CAP_SYS_ADMIN).
+ -1: Allow use of (almost) all events by all users
+ Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
+ 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
+ Disallow raw tracepoint access by users without CAP_SYS_ADMIN
+ 1: Disallow CPU event access by users without CAP_SYS_ADMIN
+ 2: Disallow kernel profiling by users without CAP_SYS_ADMIN
+ To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
+ kernel.perf_event_paranoid = -1
+
+ *Alternatively, you can run this script under sudo privileges.
+ """)
+ elif tool == "callgrind":
+ check_valgrind_installation = subprocess.run(["which", "valgrind"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if check_valgrind_installation.returncode:
+ sys.exit("Please install valgrind before running the script.")
+
+ # Insure that git is installed
+ check_git_installation = subprocess.run(["which", "git"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if check_git_installation.returncode:
+ sys.exit("Please install git before running the script.")
+
+
+def clean_exit(qemu_build_path: str, error_message: str) -> None:
+ """
+ Clean up intermediate files and exit.
+
+ Parameters:
+ qemu_build_path (str): Path to the build directory with configuration files
+ error_message (str): Error message to display after exiting
+ """
+ shutil.rmtree(qemu_build_path)
+ sys.exit(error_message)
+
+
+def make(qemu_build_path: str) -> None:
+ """
+ Build QEMU by running the Makefile.
+
+ Parameters:
+ qemu_build_path (str): Path to the build directory with configuration files
+ """
+ run_make = subprocess.run(["make",
+ "-j",
+ str(multiprocessing.cpu_count())],
+ cwd=qemu_build_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if run_make.returncode:
+ clean_exit(qemu_build_path, run_make.stderr.decode("utf-8"))
+
+
+def measure_instructions(tool: str, qemu_build_path: str, target: str,
+ command: List[str]) -> int:
+ """
+ Measure the number of instructions when running an program with QEMU.
+
+ Parameters:
+ tool (str): Tool used for the measurement (perf|callgrind)
+ qemu_build_path (str): Path to the build directory with configuration files
+ target (str): QEMU target
+ command (list): Program path and arguments
+
+ Returns:
+ (int): Number of instructions.
+ """
+ qemu_exe_path = os.path.join(qemu_build_path,
+ "{}-linux-user".format(target),
+ "qemu-{}".format(target))
+ instructions = 0
+ if tool == "perf":
+ run_perf = subprocess.run((["perf",
+ "stat",
+ "-x",
+ " ",
+ "-e",
+ "instructions",
+ qemu_exe_path]
+ + command),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if run_perf.returncode:
+ clean_exit(qemu_build_path, run_perf.stderr.decode("utf-8"))
+
+ else:
+ perf_output = run_perf.stderr.decode("utf-8").split(" ")
+ instructions = int(perf_output[0])
+
+ elif tool == "callgrind":
+ with tempfile.NamedTemporaryFile() as tmpfile:
+ run_callgrind = subprocess.run((["valgrind",
+ "--tool=callgrind",
+ "--callgrind-out-file={}".format(
+ tmpfile.name),
+ qemu_exe_path]
+ + command),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if run_callgrind.returncode:
+ clean_exit(qemu_build_path, run_callgrind.stderr.decode("utf-8"))
+
+ else:
+ callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n")
+ instructions = int(callgrind_output[8].split(" ")[-1])
+
+ return instructions
+
+
+def main():
+ """
+ Parse the command line arguments then start the execution.
+
+ Syntax:
+ bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU] \
+ --target TARGET --tool {perf,callgrind} -- \
+ <target executable> [<target executable options>]
+
+ Arguments:
+ [-h] - Print the script arguments help message
+ -s,--start START - First commit hash in the search range
+ [-e,--end END] - Last commit hash in the search range
+ (default: Latest commit)
+ [-q,--qemu QEMU] - QEMU path.
+ (default: Path to a GitHub QEMU clone)
+ --target TARGET - QEMU target name
+ --tool {perf,callgrind} - Underlying tool used for measurements
+
+ Example of usage:
+ bisect.py --start=fdd76fecdd --qemu=/path/to/qemu --target=ppc \
+ --tool=perf coulomb_double-ppc -n 1000
+ """
+
+ # Parse the command line arguments
+ parser = argparse.ArgumentParser(
+ usage="bisect.py [-h] -s,--start START [-e,--end END] [-q,--qemu QEMU]"
+ " --target TARGET --tool {perf,callgrind} -- "
+ "<target executable> [<target executable options>]")
+
+ parser.add_argument("-s", "--start", dest="start", type=str, required=True,
+ help="First commit hash in the search range")
+ parser.add_argument("-e", "--end", dest="end", type=str, default="master",
+ help="Last commit hash in the search range")
+ parser.add_argument("-q", "--qemu", dest="qemu", type=str, default="",
+ help="QEMU path")
+ parser.add_argument("--target", dest="target", type=str, required=True,
+ help="QEMU target")
+ parser.add_argument("--tool", dest="tool", choices=["perf", "callgrind"],
+ required=True, help="Tool used for measurements")
+
+ parser.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+ args = parser.parse_args()
+
+ # Extract the needed variables from the args
+ start_commit = args.start
+ end_commit = args.end
+ qemu = args.qemu
+ target = args.target
+ tool = args.tool
+ command = args.command
+
+ # Set QEMU path
+ if qemu == "":
+ # Create a temp directory for cloning QEMU
+ tmpdir = tempfile.TemporaryDirectory()
+ qemu_path = os.path.join(tmpdir.name, "qemu")
+
+ # Clone QEMU into the temporary directory
+ print("Fetching QEMU: ", end="", flush=True)
+ git_clone(qemu_path)
+ print()
+ else:
+ qemu_path = qemu
+
+ # Create the build directory
+ qemu_build_path = os.path.join(qemu_path, "tmp-build-gcc")
+
+ if not os.path.exists(qemu_build_path):
+ os.mkdir(qemu_build_path)
+ else:
+ sys.exit("A build directory with the same name (tmp-build-gcc) used in"
+ " the script is already in the provided QEMU path.")
+
+ # Configure QEMU
+ configure = subprocess.run(["../configure",
+ "--target-list={}-linux-user".format(target)],
+ cwd=qemu_build_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if configure.returncode:
+ clean_exit(qemu_build_path, configure.stderr.decode("utf-8"))
+
+ # Do performance measurements for the start commit
+ git_checkout(start_commit, qemu_path, qemu_build_path)
+ make(qemu_build_path)
+ start_commit_instructions = measure_instructions(tool,
+ qemu_build_path,
+ target,
+ command)
+ print("{:<30} {}".format("Start Commit Instructions:",
+ format(start_commit_instructions, ",")))
+
+ # Do performance measurements for the end commit
+ git_checkout(end_commit, qemu_path, qemu_build_path)
+ make(qemu_build_path)
+ end_commit_instructions = measure_instructions(tool,
+ qemu_build_path,
+ target,
+ command)
+ print("{:<30} {}".format("End Commit Instructions:",
+ format(end_commit_instructions, ",")))
+
+ # Calculate performance difference between start and end commits
+ performance_difference = \
+ (start_commit_instructions - end_commit_instructions) / \
+ max(end_commit_instructions, start_commit_instructions) * 100
+ performance_change = "+" if performance_difference > 0 else "-"
+ print("{:<30} {}".format("Performance Change:",
+ performance_change +
+ str(round(abs(performance_difference), 3))+"%"))
+
+ # Set the custom terms used for progressing in "git bisect"
+ term_old = "fast" if performance_difference < 0 else "slow"
+ term_new = "slow" if term_old == "fast" else "fast"
+
+ # Start git bisect
+ git_bisect(qemu_path, qemu_build_path, "start",
+ ["--term-old", term_old, "--term-new", term_new])
+ # Set start commit state
+ git_bisect(qemu_path, qemu_build_path, term_old, [start_commit])
+ # Set end commit state
+ bisect_output = git_bisect(
+ qemu_path, qemu_build_path, term_new, [end_commit])
+ # Print estimated bisect steps
+ print("\n{:<30} {}\n".format(
+ "Estimated Number of Steps:", bisect_output.split()[9]))
+
+ # Initialize bisect_count to track the number of performed
+ bisect_count = 1
+
+ while True:
+ print("**************BISECT STEP {}**************".
+ format(bisect_count))
+
+ make(qemu_build_path)
+
+ instructions = measure_instructions(tool,
+ qemu_build_path,
+ target,
+ command)
+ # Find the difference between the current instructions and start/end
+ # instructions.
+ diff_end = abs(instructions - end_commit_instructions)
+ diff_start = abs(instructions - start_commit_instructions)
+
+ # If current number of insructions is closer to that of start,
+ # set current commit as term_old.
+ # Else, set current commit as term_new.
+ if diff_end > diff_start:
+ bisect_command = term_old
+ else:
+ bisect_command = term_new
+
+ print("{:<20} {}".format("Instructions:", format(instructions, ",")))
+ print("{:<20} {}".format("Status:", "{} commit".
+ format(bisect_command)))
+
+ bisect_output = git_bisect(qemu_path, qemu_build_path, bisect_command)
+
+ # Continue if still bisecting,
+ # else, print result and break.
+ if not bisect_output.split(" ")[0] == "Bisecting:":
+ print("\n*****************BISECT RESULT*****************")
+ commit_message_start = bisect_output.find("commit\n") + 7
+ commit_message_end = bisect_output.find(":040000") - 1
+ print(bisect_output[commit_message_start:commit_message_end])
+ break
+
+ bisect_count += 1
+
+ # Reset git bisect
+ git_bisect(qemu_path, qemu_build_path, "reset")
+
+ # Delete temp build directory
+ shutil.rmtree(qemu_build_path)
+
+
+if __name__ == "__main__":
+ main()
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 7/9] tests/performance: Add nightly tests
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (5 preceding siblings ...)
2020-08-28 10:40 ` [PATCH 6/9] scripts/performance: Add bisect.py script Ahmed Karaman
@ 2020-08-28 10:41 ` Ahmed Karaman
2020-09-02 8:36 ` Aleksandar Markovic
2020-09-02 13:26 ` Alex Bennée
2020-08-28 10:41 ` [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection Ahmed Karaman
2020-08-28 10:41 ` [PATCH 9/9] scripts/performance: Add topN_system.py script Ahmed Karaman
8 siblings, 2 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:41 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
A nightly performance testing system to monitor any change in QEMU
performance across seventeen different targets.
The system includes eight different benchmarks to provide a variety
of testing workloads.
dijkstra_double:
Find the shortest path between the source node and all other nodes
using Dijkstra’s algorithm. The graph contains n nodes where all nxn
distances are double values. The value of n can be specified using
the -n flag. The default value is 2000.
dijkstra_int32:
Find the shortest path between the source node and all other nodes
using Dijkstra’s algorithm. The graph contains n nodes where all nxn
distances are int32 values. The value of n can be specified using
the -n flag. The default value is 2000.
matmult_double:
Standard matrix multiplication of an n*n matrix of randomly generated
double numbers from 0 to 100. The value of n is passed as an argument
with the -n flag. The default value is 200.
matmult_int32:
Standard matrix multiplication of an n*n matrix of randomly generated
integer numbers from 0 to 100. The value of n is passed as an
argument with the -n flag. The default value is 200.
qsort_double:
Quick sort of an array of n randomly generated double numbers from 0
to 1000. The value of n is passed as an argument with the -n flag.
The default value is 300000.
qsort_int32:
Quick sort of an array of n randomly generated integer numbers from 0
to 50000000. The value of n is passed as an argument with the -n
flag.The default value is 300000.
qsort_string:
Quick sort of an array of 10000 randomly generated strings of size 8
(including null terminating character). The sort process is repeated
n number of times. The value of n is passed as an argument with the
-n flag. The default value is 20.
search_string:
Search for the occurrence of a small string in a much larger random
string (“needle in a hay”). The search process is repeated n number
of times and each time, a different large random string (“hay”) is
generated. The value of n can be specified using the -n flag. The
default value is 20.
Syntax:
nightly_tests_core.py [-h] [-r REF]
Optional arguments:
-h, --help Show this help message and exit
-r REF, --reference REF
Reference QEMU version - Default is v5.1.0
Example of usage:
nightly_tests_core.py -r v5.1.0 2>log.txt
The following report includes detailed setup and execution details
of the system:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
tests/performance/nightly-tests/README.md | 243 +++++
.../source/dijkstra_double/dijkstra_double.c | 194 ++++
.../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
.../source/matmult_double/matmult_double.c | 123 +++
.../source/matmult_int32/matmult_int32.c | 121 +++
.../source/qsort_double/qsort_double.c | 104 ++
.../source/qsort_int32/qsort_int32.c | 103 ++
.../source/qsort_string/qsort_string.c | 122 +++
.../source/search_string/search_string.c | 110 +++
.../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
.../scripts/run_nightly_tests.py | 135 +++
.../nightly-tests/scripts/send_email.py | 56 ++
12 files changed, 2423 insertions(+)
create mode 100644 tests/performance/nightly-tests/README.md
create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
create mode 100644 tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py
create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py
create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/nightly-tests/README.md
new file mode 100644
index 0000000000..6db3b351b3
--- /dev/null
+++ b/tests/performance/nightly-tests/README.md
@@ -0,0 +1,243 @@
+### QEMU Nightly Tests
+
+**Required settings:**
+
+Update the `GMAIL_USER` object in `send_email.py` with your credentials.
+
+For more details on how the system works, please check the [eighth report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos Benchmarking" series.
+
+**Running the System:**
+
+The default reference version is v5.1.0. To specify a custom version, please use the `-r, --reference` flag.
+
+```bash
+./run_nightly_tests.py
+```
+
+**Output:**
+
+```
+Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
+Host Memory : 15.49 GB
+
+Start Time (UTC) : 2020-08-25 21:30:01
+End Time (UTC) : 2020-08-25 22:02:37
+Execution Time : 0:32:35.896990
+
+Status : SUCCESS
+
+Note:
+Changes denoted by '-----' are less than 0.01%.
+
+--------------------------------------------------------
+ SUMMARY REPORT - COMMIT d1a2b51f
+--------------------------------------------------------
+ AVERAGE RESULTS
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 158 355 274 ----- +1.693%
+alpha 1 914 967 171 ----- +3.524%
+arm 8 076 402 940 ----- +2.304%
+hppa 4 261 685 987 -0.182% +3.164%
+m68k 2 690 273 044 ----- +7.131%
+mips 1 862 033 667 ----- +2.494%
+mipsel 2 008 211 069 ----- +2.674%
+mips64 1 918 635 565 ----- +2.818%
+mips64el 2 051 565 677 ----- +3.026%
+ppc 2 480 141 217 ----- +3.107%
+ppc64 2 576 713 959 ----- +3.143%
+ppc64le 2 558 853 539 ----- +3.173%
+riscv64 1 406 704 050 ----- +2.65%
+s390x 3 158 140 046 ----- +3.118%
+sh4 2 364 449 748 ----- +3.33%
+sparc64 3 318 544 783 ----- +3.851%
+x86_64 1 775 844 158 ----- +2.156%
+--------------------------------------------------------
+
+ DETAILED RESULTS
+--------------------------------------------------------
+Test Program: dijkstra_double
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 3 062 583 464 ----- +1.424%
+alpha 3 191 864 698 ----- +3.696%
+arm 16 357 157 526 ----- +2.347%
+hppa 7 228 376 315 -0.139% +3.086%
+m68k 4 294 016 587 ----- +9.692%
+mips 3 051 419 166 ----- +2.427%
+mipsel 3 231 509 618 ----- +2.869%
+mips64 3 245 837 754 ----- +2.596%
+mips64el 3 414 195 796 ----- +3.021%
+ppc 4 914 520 972 -0.041% +4.74%
+ppc64 5 098 154 311 ----- +4.565%
+ppc64le 5 082 419 054 ----- +4.58%
+riscv64 2 192 294 915 ----- +1.955%
+s390x 4 584 503 977 ----- +2.896%
+sh4 3 949 036 447 ----- +3.464%
+sparc64 4 586 203 546 ----- +4.237%
+x86_64 2 484 092 105 ----- +1.75%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: dijkstra_int32
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 210 194 577 ----- +1.493%
+alpha 1 494 133 274 ----- +2.15%
+arm 8 262 935 967 ----- +2.665%
+hppa 5 207 318 306 ----- +3.047%
+m68k 1 725 856 962 ----- +2.527%
+mips 1 495 227 032 ----- +1.492%
+mipsel 1 497 147 869 ----- +1.479%
+mips64 1 715 388 570 ----- +1.892%
+mips64el 1 695 276 864 ----- +1.913%
+ppc 2 014 557 389 ----- +1.819%
+ppc64 2 206 267 901 ----- +2.139%
+ppc64le 2 197 998 781 ----- +2.146%
+riscv64 1 354 912 745 ----- +2.396%
+s390x 2 916 247 062 ----- +1.241%
+sh4 1 990 532 533 ----- +2.669%
+sparc64 2 872 231 051 ----- +3.758%
+x86_64 1 553 981 241 ----- +2.12%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: matmult_double
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 1 412 273 223 ----- +0.302%
+alpha 3 233 991 649 ----- +7.473%
+arm 8 545 173 979 ----- +1.088%
+hppa 3 483 597 802 -1.267% +4.468%
+m68k 3 919 065 529 ----- +18.431%
+mips 2 344 774 894 ----- +4.091%
+mipsel 3 329 886 464 ----- +5.177%
+mips64 2 359 046 988 ----- +4.076%
+mips64el 3 343 664 785 ----- +5.167%
+ppc 3 209 457 051 ----- +3.246%
+ppc64 3 287 503 981 ----- +3.173%
+ppc64le 3 287 189 065 ----- +3.173%
+riscv64 1 221 603 682 ----- +0.277%
+s390x 2 874 199 923 ----- +5.827%
+sh4 3 543 943 634 ----- +6.416%
+sparc64 3 426 150 004 ----- +7.139%
+x86_64 1 248 917 276 ----- +0.322%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: matmult_int32
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 598 681 621 ----- +0.585%
+alpha 372 437 418 ----- +0.677%
+arm 746 583 193 ----- +1.462%
+hppa 674 278 359 ----- +1.183%
+m68k 410 495 553 ----- +0.9%
+mips 499 698 837 ----- +0.531%
+mipsel 499 500 429 ----- +0.497%
+mips64 481 554 664 ----- +0.599%
+mips64el 465 057 054 ----- +0.619%
+ppc 341 334 603 ----- +0.944%
+ppc64 393 796 203 ----- +0.966%
+ppc64le 393 977 298 ----- +0.965%
+riscv64 351 709 769 ----- +0.785%
+s390x 494 427 384 ----- +0.599%
+sh4 402 668 444 ----- +0.899%
+sparc64 495 952 959 ----- +1.192%
+x86_64 402 928 461 ----- +0.833%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_double
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 709 683 624 ----- +2.417%
+alpha 1 969 460 172 ----- +3.68%
+arm 8 322 998 390 ----- +2.587%
+hppa 3 188 301 995 -0.047% +2.9%
+m68k 4 953 930 065 ----- +15.153%
+mips 2 123 919 587 ----- +3.055%
+mipsel 2 124 212 187 ----- +3.048%
+mips64 1 999 047 826 ----- +3.405%
+mips64el 1 996 426 772 ----- +3.409%
+ppc 2 819 267 902 -0.021% +5.435%
+ppc64 2 768 186 548 ----- +5.513%
+ppc64le 2 724 803 772 -0.011% +5.603%
+riscv64 1 638 328 937 ----- +4.021%
+s390x 2 519 081 708 ----- +3.362%
+sh4 2 595 545 154 ----- +2.994%
+sparc64 3 988 986 587 ----- +2.747%
+x86_64 2 033 468 588 ----- +3.234%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_int32
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 193 392 565 ----- +2.916%
+alpha 1 521 291 933 ----- +4.193%
+arm 3 465 445 043 ----- +2.756%
+hppa 2 280 034 340 ----- +3.821%
+m68k 1 843 189 041 ----- +3.583%
+mips 1 558 024 873 ----- +3.863%
+mipsel 1 560 583 980 ----- +3.846%
+mips64 1 563 415 749 ----- +4.412%
+mips64el 1 542 677 320 ----- +4.474%
+ppc 1 728 698 880 ----- +3.665%
+ppc64 1 842 444 545 ----- +3.555%
+ppc64le 1 791 822 067 ----- +3.661%
+riscv64 1 348 866 430 ----- +4.654%
+s390x 2 184 073 151 ----- +3.319%
+sh4 1 946 492 539 ----- +3.624%
+sparc64 3 452 215 585 ----- +2.937%
+x86_64 1 813 544 414 ----- +3.537%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: qsort_string
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 592 218 418 ----- +2.468%
+alpha 1 855 834 626 ----- +3.487%
+arm 7 347 721 165 ----- +2.682%
+hppa 4 758 753 926 ----- +3.543%
+m68k 2 376 811 462 ----- +3.567%
+mips 2 166 608 045 ----- +2.532%
+mipsel 2 163 392 541 ----- +2.528%
+mips64 2 029 251 969 ----- +3.117%
+mips64el 2 011 628 621 ----- +3.145%
+ppc 2 492 942 463 ----- +2.673%
+ppc64 2 464 702 554 ----- +2.488%
+ppc64le 2 445 253 307 ----- +2.505%
+riscv64 1 625 053 328 ----- +3.953%
+s390x 4 194 608 798 ----- +6.623%
+sh4 2 164 142 539 ----- +3.166%
+sparc64 4 299 516 539 ----- +4.065%
+x86_64 2 940 456 780 ----- +2.649%
+--------------------------------------------------------
+--------------------------------------------------------
+Test Program: search_string
+--------------------------------------------------------
+Target Instructions Latest v5.1.0
+---------- -------------------- ---------- ----------
+aarch64 2 487 814 704 ----- +1.94%
+alpha 1 680 723 605 ----- +2.835%
+arm 11 563 208 260 ----- +2.848%
+hppa 7 272 826 858 ----- +3.263%
+m68k 1 998 819 159 ----- +3.198%
+mips 1 656 596 909 ----- +1.959%
+mipsel 1 659 455 464 ----- +1.947%
+mips64 1 955 541 001 ----- +2.447%
+mips64el 1 943 598 207 ----- +2.462%
+ppc 2 320 350 477 ----- +2.332%
+ppc64 2 552 655 634 ----- +2.742%
+ppc64le 2 547 364 971 ----- +2.748%
+riscv64 1 520 862 601 ----- +3.159%
+s390x 5 497 978 370 ----- +1.078%
+sh4 2 323 236 696 ----- +3.41%
+sparc64 3 427 101 999 ----- +4.73%
+x86_64 1 729 364 402 ----- +2.806%
+--------------------------------------------------------
+```
diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
new file mode 100644
index 0000000000..9c0bb804ac
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
@@ -0,0 +1,194 @@
+/*
+ * Source file of a benchmark program involving calculations of the
+ * shortest distances between a source node and all other nodes in a
+ * graph of n nodes in which all nxn distances are defined as "double".
+ * The number n can be given via command line, and the default is 2000.
+ * The algorithm used is Dijsktra's.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <float.h>
+#include <limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_NODE_COUNT 2000
+#define MIN_NODE_COUNT 3
+#define MAX_NODE_COUNT 10000
+
+
+int32_t closest_index(int32_t count, double *distances, bool *flags)
+{
+ int32_t closest;
+ double minimum = DBL_MAX;
+
+ for (size_t i = 0; i < count; i++) {
+ if (flags[i] == false && distances[i] <= minimum) {
+ closest = i;
+ minimum = distances[i];
+ }
+ }
+
+ return closest;
+}
+
+/**
+ * Calculate the shortest distances from the source node using Dijkstra method.
+ * @param (out) distances An array of shortest distances from the source node.
+ * @param (out) via An array of nodes needed to be taken as the the last
+ * before destination, for each destination.
+ * @param (out) eccent Eccentricity of the source node.
+ * @param (in) count The number of nodes.
+ * @param (in) source Source node.
+ * @param (in) matrix Distance matrix.
+ */
+void find_shortest_distances(double *distances, int32_t *via, double *eccent,
+ int32_t count, int32_t source, double **matrix)
+{
+ bool *flags;
+
+ flags = (bool *)malloc(count * sizeof(bool));
+
+ for (size_t i = 0; i < count; i++) {
+ distances[i] = DBL_MAX;
+ flags[i] = false;
+ }
+
+ distances[source] = 0.0;
+ via[source] = source;
+
+ for (size_t i = 0; i < count - 1; i++) {
+ int32_t closest = closest_index(count, distances, flags);
+ flags[closest] = true;
+ for (size_t j = 0; j < count; j++) {
+ if ((!flags[j]) &&
+ (matrix[closest][j]) &&
+ (distances[closest] != DBL_MAX) &&
+ (distances[j] > distances[closest] + matrix[closest][j])) {
+ distances[j] = distances[closest] + matrix[closest][j];
+ via[j] = closest;
+ }
+ }
+ }
+
+ *eccent = 0;
+ for (size_t i = 0; i < count; i++) {
+ if (*eccent < distances[i]) {
+ *eccent = distances[i];
+ }
+ }
+
+ free(flags);
+}
+
+
+void main(int argc, char *argv[])
+{
+ double **distance_matrix;
+ double *shortest_distances;
+ int32_t *via_node;
+ int32_t node_count = DEFAULT_NODE_COUNT;
+ int32_t source_node = 0;
+ double node_eccentricity = 0.0;
+ double range_factor = 999.0 / (double)(RAND_MAX);
+ int32_t option;
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_node_count = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_node_count == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_node_count < MIN_NODE_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_NODE_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_node_count > MAX_NODE_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_NODE_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ node_count = user_node_count;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for all matrixes */
+ distance_matrix = (double **)malloc(node_count * sizeof(double *));
+ for (size_t i = 0; i < node_count; i++) {
+ distance_matrix[i] = (double *)malloc(node_count * sizeof(double));
+ }
+ shortest_distances = (double *)malloc(node_count * sizeof(double));
+ via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
+
+ /* Initialize helper arrays and populate distance_matrix */
+ srand(1);
+ for (size_t i = 0; i < node_count; i++) {
+ shortest_distances[i] = 0.0;
+ via_node[i] = -1;
+ distance_matrix[i][i] = 0.0;
+ }
+ for (size_t i = 0; i < node_count; i++) {
+ for (size_t j = i + 1; j < node_count; j++) {
+ distance_matrix[i][j] = 1.0 + range_factor * (double)rand();
+ distance_matrix[j][i] = distance_matrix[i][j];
+ }
+ }
+
+ find_shortest_distances(shortest_distances, via_node, &node_eccentricity,
+ node_count, source_node, distance_matrix);
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf(" Distance matrix (top left part):\n");
+ for (size_t i = 0; i < 3; i++) {
+ for (size_t j = 0; j < 3; j++) {
+ printf(" %7.2f", distance_matrix[i][j]);
+ }
+ printf("\n");
+ }
+ printf(" Source: %d (eccentricity: %f)\n",
+ source_node, node_eccentricity);
+ printf(" Destination Distance Via Node\n");
+ for (size_t i = 0; i < 3; i++) {
+ printf(" %5d %7.2f %4d\n",
+ i, shortest_distances[i], via_node[i]);
+ }
+
+ /* Free all previously allocated space */
+ for (size_t i = 0; i < node_count; i++) {
+ free(distance_matrix[i]);
+ }
+ free(distance_matrix);
+ free(shortest_distances);
+ free(via_node);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
new file mode 100644
index 0000000000..2663cde943
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
@@ -0,0 +1,192 @@
+/*
+ * Source file of a benchmark program involving calculations of the
+ * shortest distances between a source node and all other nodes in a
+ * graph of n nodes in which all nxn distances are defined as "int32".
+ * The number n can be given via command line, and the default is 2000.
+ * The algorithm used is Dijsktra's.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <limits.h>
+#include <stdbool.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_NODE_COUNT 2000
+#define MIN_NODE_COUNT 3
+#define MAX_NODE_COUNT 10000
+
+
+int32_t closest_index(int32_t count, int32_t *distances, bool *flags)
+{
+ int32_t closest;
+ int32_t minimum = INT_MAX;
+
+ for (size_t i = 0; i < count; i++) {
+ if (flags[i] == false && distances[i] <= minimum) {
+ closest = i;
+ minimum = distances[i];
+ }
+ }
+
+ return closest;
+}
+
+/**
+ * Calculate the shortest distances from the source node using Dijkstra method.
+ * @param (out) distances An array of shortest distances from the source node.
+ * @param (out) via An array of nodes needed to be taken as the the last
+ * before destination, for each destination.
+ * @param (out) eccent Eccentricity of the source node.
+ * @param (in) count The number of nodes.
+ * @param (in) source Source node.
+ * @param (in) matrix Distance matrix.
+ */
+void find_shortest_distances(int32_t *distances, int32_t *via, int32_t *eccent,
+ int32_t count, int32_t source, int32_t **matrix)
+{
+ bool *flags;
+
+ flags = (bool *)malloc(count * sizeof(bool));
+
+ for (size_t i = 0; i < count; i++) {
+ distances[i] = INT_MAX;
+ flags[i] = false;
+ }
+
+ distances[source] = 0;
+ via[source] = source;
+
+ for (size_t i = 0; i < count - 1; i++) {
+ int32_t closest = closest_index(count, distances, flags);
+ flags[closest] = true;
+ for (size_t j = 0; j < count; j++) {
+ if ((!flags[j]) &&
+ (matrix[closest][j]) &&
+ (distances[closest] != INT_MAX) &&
+ (distances[j] > distances[closest] + matrix[closest][j])) {
+ distances[j] = distances[closest] + matrix[closest][j];
+ via[j] = closest;
+ }
+ }
+ }
+
+ *eccent = 0;
+ for (size_t i = 0; i < count; i++) {
+ if (*eccent < distances[i]) {
+ *eccent = distances[i];
+ }
+ }
+
+ free(flags);
+}
+
+
+void main(int argc, char *argv[])
+{
+ int32_t **distance_matrix;
+ int32_t *shortest_distances;
+ int32_t *via_node;
+ int32_t node_count = DEFAULT_NODE_COUNT;
+ int32_t source_node = 0;
+ int32_t node_eccentricity = 0;
+ int32_t option;
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_node_count = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_node_count == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_node_count < MIN_NODE_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_NODE_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_node_count > MAX_NODE_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_NODE_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ node_count = user_node_count;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for all matrixes */
+ distance_matrix = (int32_t **)malloc(node_count * sizeof(int32_t *));
+ for (size_t i = 0; i < node_count; i++) {
+ distance_matrix[i] = (int32_t *)malloc(node_count * sizeof(int32_t));
+ }
+ shortest_distances = (int32_t *)malloc(node_count * sizeof(int32_t));
+ via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
+
+ /* Initialize helper arrays and populate distance_matrix */
+ srand(1);
+ for (size_t i = 0; i < node_count; i++) {
+ shortest_distances[i] = 0;
+ via_node[i] = -1;
+ distance_matrix[i][i] = 0;
+ }
+ for (size_t i = 0; i < node_count; i++) {
+ for (size_t j = i + 1; j < node_count; j++) {
+ distance_matrix[i][j] = 1 + (rand()) / (RAND_MAX / 999);
+ distance_matrix[j][i] = distance_matrix[i][j];
+ }
+ }
+
+ find_shortest_distances(shortest_distances, via_node, &node_eccentricity,
+ node_count, source_node, distance_matrix);
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf(" Distance matrix (top left part):\n");
+ for (size_t i = 0; i < 3; i++) {
+ for (size_t j = 0; j < 3; j++) {
+ printf(" %6d", distance_matrix[i][j]);
+ }
+ printf("\n");
+ }
+ printf(" Source: %d (eccentricity: %d)\n",
+ source_node, node_eccentricity);
+ printf(" Destination Distance Via Node\n");
+ for (size_t i = 0; i < 3; i++) {
+ printf(" %5d %3d %4d\n",
+ i, shortest_distances[i], via_node[i]);
+ }
+
+ /* Free all previously allocated space */
+ for (size_t i = 0; i < node_count; i++) {
+ free(distance_matrix[i]);
+ }
+ free(distance_matrix);
+ free(shortest_distances);
+ free(via_node);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
new file mode 100644
index 0000000000..42bbb4717a
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
@@ -0,0 +1,123 @@
+/*
+ * Source file of a benchmark program involving calculations of
+ * a product of two matrixes nxn whose elements are "double". The
+ * number n can be given via command line, and the default is 200.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_MATRIX_SIZE 200
+#define MIN_MATRIX_SIZE 2
+#define MAX_MATRIX_SIZE 200000
+
+void main(int argc, char *argv[])
+{
+ double **matrix_a;
+ double **matrix_b;
+ double **matrix_res;
+ size_t i;
+ size_t j;
+ size_t k;
+ int32_t matrix_size = DEFAULT_MATRIX_SIZE;
+ int32_t option;
+ double range_factor = 100.0 / (double)(RAND_MAX);
+
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_matrix_size = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_matrix_size == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_matrix_size < MIN_MATRIX_SIZE) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_MATRIX_SIZE);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_matrix_size > MAX_MATRIX_SIZE) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_MATRIX_SIZE);
+ exit(EXIT_FAILURE);
+ }
+ matrix_size = user_matrix_size;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for all matrixes */
+ matrix_a = (double **)malloc(matrix_size * sizeof(double *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_a[i] = (double *)malloc(matrix_size * sizeof(double));
+ }
+ matrix_b = (double **)malloc(matrix_size * sizeof(double *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_b[i] = (double *)malloc(matrix_size * sizeof(double));
+ }
+ matrix_res = (double **)malloc(matrix_size * sizeof(double *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_res[i] = (double *)malloc(matrix_size * sizeof(double));
+ }
+
+ /* Populate matrix_a and matrix_b with random numbers */
+ srand(1);
+ for (i = 0; i < matrix_size; i++) {
+ for (j = 0; j < matrix_size; j++) {
+ matrix_a[i][j] = range_factor * (double)rand();
+ matrix_b[i][j] = range_factor * (double)rand();
+ }
+ }
+
+ /* Calculate the product of two matrixes */
+ for (i = 0; i < matrix_size; i++) {
+ for (j = 0; j < matrix_size; j++) {
+ matrix_res[i][j] = 0.0;
+ for (k = 0; k < matrix_size; k++) {
+ matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
+ }
+ }
+ }
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]);
+ printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]);
+
+ /* Free all previously allocated space */
+ for (i = 0; i < matrix_size; i++) {
+ free(matrix_a[i]);
+ free(matrix_b[i]);
+ free(matrix_res[i]);
+ }
+ free(matrix_a);
+ free(matrix_b);
+ free(matrix_res);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
new file mode 100644
index 0000000000..29a6eb000d
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
@@ -0,0 +1,121 @@
+/*
+ * Source file of a benchmark program involving calculations of
+ * a product of two matrixes nxn whose elements are "int32_t". The
+ * number n can be given via command line, and the default is 200.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of columns and rows in all matrixes*/
+#define DEFAULT_MATRIX_SIZE 200
+#define MIN_MATRIX_SIZE 2
+#define MAX_MATRIX_SIZE 200000
+
+void main(int argc, char *argv[])
+{
+ int32_t **matrix_a;
+ int32_t **matrix_b;
+ int32_t **matrix_res;
+ size_t i;
+ size_t j;
+ size_t k;
+ int32_t matrix_size = DEFAULT_MATRIX_SIZE;
+ int32_t option;
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_matrix_size = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_matrix_size == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_matrix_size < MIN_MATRIX_SIZE) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_MATRIX_SIZE);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_matrix_size > MAX_MATRIX_SIZE) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_MATRIX_SIZE);
+ exit(EXIT_FAILURE);
+ }
+ matrix_size = user_matrix_size;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for all matrixes */
+ matrix_a = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_a[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+ }
+ matrix_b = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_b[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+ }
+ matrix_res = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
+ for (i = 0; i < matrix_size; i++) {
+ matrix_res[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
+ }
+
+ /* Populate matrix_a and matrix_b with random numbers */
+ srand(1);
+ for (i = 0; i < matrix_size; i++) {
+ for (j = 0; j < matrix_size; j++) {
+ matrix_a[i][j] = (rand()) / (RAND_MAX / 100);
+ matrix_b[i][j] = (rand()) / (RAND_MAX / 100);
+ }
+ }
+
+ /* Calculate the product of two matrixes */
+ for (i = 0; i < matrix_size; i++) {
+ for (j = 0; j < matrix_size; j++) {
+ matrix_res[i][j] = 0;
+ for (k = 0; k < matrix_size; k++) {
+ matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
+ }
+ }
+ }
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]);
+ printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]);
+
+ /* Free all previously allocated space */
+ for (i = 0; i < matrix_size; i++) {
+ free(matrix_a[i]);
+ free(matrix_b[i]);
+ free(matrix_res[i]);
+ }
+ free(matrix_a);
+ free(matrix_b);
+ free(matrix_res);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
new file mode 100644
index 0000000000..efc1b2eee1
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
@@ -0,0 +1,104 @@
+/*
+ * Source file of a benchmark program involving sorting of an array
+ * of length n whose elements are "double". The default value for n
+ * is 300000, and it can be set via command line as well.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of elements in the array to be sorted */
+#define DEFAULT_ARRAY_LEN 300000
+#define MIN_ARRAY_LEN 3
+#define MAX_ARRAY_LEN 30000000
+
+/* Upper limit used for generation of random numbers */
+#define UPPER_LIMIT 1000.0
+
+/* Comparison function passed to qsort() */
+static int compare(const void *a, const void *b)
+{
+ if (*(const double *)a > *(const double *)b) {
+ return 1;
+ } else if (*(const double *)a < *(const double *)b) {
+ return -1;
+ }
+ return 0;
+}
+
+void main(int argc, char *argv[])
+{
+ double *array_to_be_sorted;
+ int32_t array_len = DEFAULT_ARRAY_LEN;
+ int32_t option;
+ double range_factor = UPPER_LIMIT / (double)(RAND_MAX);
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_array_len = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_array_len == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_array_len < MIN_ARRAY_LEN) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_ARRAY_LEN);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_array_len > MAX_ARRAY_LEN) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_ARRAY_LEN);
+ exit(EXIT_FAILURE);
+ }
+ array_len = user_array_len;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for the array */
+ array_to_be_sorted = (double *) malloc(array_len * sizeof(double));
+
+ /* Populate the_array with random numbers */
+ srand(1);
+ for (size_t i = 0; i < array_len; i++) {
+ array_to_be_sorted[i] = range_factor * (double)rand();
+ }
+
+ /* Sort the_array using qsort() */
+ qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]),
+ compare);
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf("%14.10f %14.10f %14.10f\n",
+ array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]);
+
+ /* Free all previously allocated space */
+ free(array_to_be_sorted);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
new file mode 100644
index 0000000000..76ca9c3490
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
@@ -0,0 +1,103 @@
+/*
+ * Source file of a benchmark program involving sorting of an array
+ * of length n whose elements are "int32_t". The default value for n
+ * is 300000, and it can be set via command line as well.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+/* Number of elements in the array to be sorted */
+#define DEFAULT_ARRAY_LEN 300000
+#define MIN_ARRAY_LEN 3
+#define MAX_ARRAY_LEN 30000000
+
+/* Upper limit used for generation of random numbers */
+#define UPPER_LIMIT 50000000
+
+/* Comparison function passed to qsort() */
+static int compare(const void *a, const void *b)
+{
+ if (*(const int32_t *)a > *(const int32_t *)b) {
+ return 1;
+ } else if (*(const int32_t *)a < *(const int32_t *)b) {
+ return -1;
+ }
+ return 0;
+}
+
+void main(int argc, char *argv[])
+{
+ int32_t *array_to_be_sorted;
+ int32_t array_len = DEFAULT_ARRAY_LEN;
+ int32_t option;
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_array_len = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_array_len == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_array_len < MIN_ARRAY_LEN) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_ARRAY_LEN);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_array_len > MAX_ARRAY_LEN) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_ARRAY_LEN);
+ exit(EXIT_FAILURE);
+ }
+ array_len = user_array_len;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ /* Allocate the memory space for the array */
+ array_to_be_sorted = (int32_t *) malloc(array_len * sizeof(int32_t));
+
+ /* Populate the_array with random numbers */
+ srand(1);
+ for (size_t i = 0; i < array_len; i++) {
+ array_to_be_sorted[i] = (rand()) / (RAND_MAX / UPPER_LIMIT);
+ }
+
+ /* Sort the_array using qsort() */
+ qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]),
+ compare);
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf("%d %d %d\n",
+ array_to_be_sorted[0], array_to_be_sorted[1], array_to_be_sorted[2]);
+
+ /* Free all previously allocated space */
+ free(array_to_be_sorted);
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
new file mode 100644
index 0000000000..7d582b2dd0
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
@@ -0,0 +1,122 @@
+/*
+ * Source file of a benchmark program involving sorting of an array
+ * of 10000 random strings of length 8 (including terminating zero).
+ * That sorting is repeated a number of times (default is 20 times),
+ * and each time a different array of random strings is generated.
+ * The number of repetitions can be set via command line.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <unistd.h>
+
+/* Length of an individual random string (including terminating zero) */
+#define RANDOM_STRING_LEN 8
+/* Number of elements of the array of random strings */
+#define NUMBER_OF_RANDOM_STRINGS 10000
+
+/* Number of repetitions to be performed each with different input */
+#define DEFAULT_REPETITION_COUNT 20
+#define MIN_REPETITION_COUNT 1
+#define MAX_REPETITION_COUNT 1000
+
+/* Structure that keeps an array of random strings to be sorted */
+struct StringStruct {
+ char chars[RANDOM_STRING_LEN];
+};
+
+/* Comparison function passed to qsort() */
+int compare_strings(const void *element1, const void *element2)
+{
+ int result;
+
+ result = strcmp((*((struct StringStruct *)element1)).chars,
+ (*((struct StringStruct *)element2)).chars);
+
+ return (result < 0) ? -1 : ((result == 0) ? 0 : 1);
+}
+
+/* Generate a random string of given length and containing only small letters */
+static void gen_random_string(char *s, const int len)
+{
+ static const char letters[] = "abcdefghijklmnopqrstuvwxyz";
+
+ for (size_t i = 0; i < (len - 1); i++) {
+ s[i] = letters[rand() % (sizeof(letters) - 1)];
+ }
+
+ s[len - 1] = 0;
+}
+
+void main(int argc, char *argv[])
+{
+ struct StringStruct strings_to_be_sorted[NUMBER_OF_RANDOM_STRINGS];
+ int32_t repetition_count = DEFAULT_REPETITION_COUNT;
+ int32_t option;
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_repetition_count = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_repetition_count == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_repetition_count < MIN_REPETITION_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_REPETITION_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_repetition_count > MAX_REPETITION_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_REPETITION_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ repetition_count = user_repetition_count;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ srand(1);
+
+ for (size_t i = 0; i < repetition_count; ++i) {
+ /* Generate random strings, and, in turn, sort them */
+ for (size_t i = 0; i < NUMBER_OF_RANDOM_STRINGS; ++i) {
+ gen_random_string(strings_to_be_sorted[i].chars, RANDOM_STRING_LEN);
+ }
+ qsort(strings_to_be_sorted, NUMBER_OF_RANDOM_STRINGS,
+ sizeof(struct StringStruct), compare_strings);
+ }
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ for (size_t i = 0; i < 2; ++i) {
+ printf(" %s", strings_to_be_sorted[i].chars);
+ }
+ printf("\n");
+}
diff --git a/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
new file mode 100644
index 0000000000..2827ea032e
--- /dev/null
+++ b/tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
@@ -0,0 +1,110 @@
+/*
+ * Source file of a benchmark program that searches for the occurrence
+ * of a small string in a much larger random string ("needle in a hay").
+ * That searching is repeated a number of times (default is 20 times),
+ * and each time a different large random string ("hay") is generated.
+ * The number of repetitions can be set via command line.
+ *
+ * This file is a part of the project "TCG Continuous Benchmarking".
+ *
+ * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+ * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <https://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <unistd.h>
+
+/* Length of a long string to be searched (including terminating zero) */
+#define HAYSTACK_LEN 30000
+
+/* Number of repetitions to be performed each with different input */
+#define DEFAULT_REPETITION_COUNT 100
+#define MIN_REPETITION_COUNT 1
+#define MAX_REPETITION_COUNT 10000
+
+
+/* Generate a random string of given length and containing only small letters */
+static void gen_random_string(char *s, const int len)
+{
+ static const char letters[] = "abcdefghijklmnopqrstuvwxyz";
+
+ for (size_t i = 0; i < (len - 1); i++) {
+ s[i] = letters[rand() % (sizeof(letters) - 1)];
+ }
+
+ s[len - 1] = 0;
+}
+
+void main(int argc, char *argv[])
+{
+ char haystack[HAYSTACK_LEN];
+ const char needle[] = "aaa ";
+ char *found_needle;
+ int32_t found_cnt = 0;
+ int32_t not_found_cnt = 0;
+ int32_t repetition_count = DEFAULT_REPETITION_COUNT;
+ int32_t option;
+
+ printf("needle is %s, size %d\n", needle, sizeof(needle));
+
+ /* Parse command line options */
+ while ((option = getopt(argc, argv, "n:")) != -1) {
+ if (option == 'n') {
+ int32_t user_repetition_count = atoi(optarg);
+
+ /* Check if the value is a string or zero */
+ if (user_repetition_count == 0) {
+ fprintf(stderr, "Error ... Invalid value for option '-n'.\n");
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is a negative number */
+ if (user_repetition_count < MIN_REPETITION_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be a "
+ "number less than %d.\n", MIN_REPETITION_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ /* Check if the value is too large */
+ if (user_repetition_count > MAX_REPETITION_COUNT) {
+ fprintf(stderr, "Error ... Value for option '-n' cannot be "
+ "more than %d.\n", MAX_REPETITION_COUNT);
+ exit(EXIT_FAILURE);
+ }
+ repetition_count = user_repetition_count;
+ } else {
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ srand(1);
+
+ for (size_t i = 0; i < repetition_count; ++i) {
+ /* Generate random hay, and, in turn, find a needle */
+ gen_random_string(haystack, HAYSTACK_LEN);
+ found_needle = strstr(haystack, needle);
+ if (found_needle != NULL) {
+ found_cnt++;
+ } else {
+ not_found_cnt++;
+ }
+ }
+
+ /* Control printing */
+ printf("CONTROL RESULT:\n");
+ printf(" Found %d times. Not found %d times.\n", found_cnt, not_found_cnt);
+}
diff --git a/tests/performance/nightly-tests/scripts/nightly_tests_core.py b/tests/performance/nightly-tests/scripts/nightly_tests_core.py
new file mode 100755
index 0000000000..da192c704a
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/nightly_tests_core.py
@@ -0,0 +1,920 @@
+#!/usr/bin/env python3
+
+"""
+Core script for performing nightly performance tests on QEMU.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import csv
+import datetime
+import glob
+import multiprocessing
+import os
+import pathlib
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from typing import Dict, List, Optional, Union
+
+
+def get_benchmark_name(benchmark_path: str) -> str:
+ """
+ Return the benchmark name given its path.
+
+ Parameters:
+ benchmarks_path (str): Absolute path to benchmark
+
+ Return:
+ (str): Benchmark name
+ """
+ benchmark_source_file = os.path.split(benchmark_path)[1]
+ return os.path.splitext(benchmark_source_file)[0]
+
+
+def get_benchmark_parent_dir(benchmark_path: str) -> str:
+ """
+ Return the benchmark parent directory name given the benchmark path.
+
+ Parameters:
+ benchmarks_path (str): Absolute path to benchmark
+
+ Return:
+ (str): Benchmark parent directory name
+ """
+ benchmark_parent_dir_path = os.path.split(benchmark_path)[0]
+ benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1]
+
+ return benchmark_parent_dir
+
+
+def get_executable_parent_dir_path(
+ benchmark_path: str, benchmarks_executables_dir_path: str) -> str:
+ """
+ Return the executables parent directory of a benchmark given its path.
+ This is the directory that includes all compiled executables for the
+ benchmark.
+
+ Parameters:
+ benchmarks_path (str): Absolute path to benchmark
+ benchmarks_executables_dir_path (str): Absolute path to the executables
+
+ Return:
+ (str): Executables parent directory path
+ """
+ benchmark_parent_dir_path = os.path.split(benchmark_path)[0]
+ benchmark_parent_dir = os.path.split(benchmark_parent_dir_path)[1]
+ executable_parent_dir_path = os.path.join(benchmarks_executables_dir_path,
+ benchmark_parent_dir)
+
+ return executable_parent_dir_path
+
+
+def get_commit_hash(commit_tag: str, qemu_path: str) -> str:
+ """
+ Find commit hash given the Git commit tag.
+
+ Parameters:
+ commit_tag (str): Commit tag
+ qemu_path (str): Absolute path to QEMU
+
+ Returns:
+ (str): 8 digit commit hash
+ """
+
+ commit_hash = subprocess.run(["git",
+ "rev-parse",
+ commit_tag],
+ cwd=qemu_path,
+ stdout=subprocess.PIPE,
+ check=False)
+ if commit_hash.returncode:
+ clean_exit(qemu_path,
+ "Failed to find the commit hash of {}.".format(commit_tag))
+
+ return commit_hash.stdout.decode("utf-8")[:8]
+
+
+def git_checkout(commit: str, qemu_path: str) -> None:
+ """
+ Checkout a given Git commit.
+ Also pull the latest changes from origin/master if the commit is "master".
+
+ Parameters:
+ commit (str): Commit hash or tag
+ qemu_path (str): Absolute path to QEMU
+ """
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Checking out {}".format(commit), file=sys.stderr, flush=True)
+
+ checkout_commit = subprocess.run(["git",
+ "checkout",
+ commit],
+ cwd=qemu_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if checkout_commit.returncode:
+ clean_exit(qemu_path, checkout_commit.stderr.decode("utf-8"))
+
+ if commit == "master":
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Pulling the latest changes from QEMU master",
+ file=sys.stderr, flush=True)
+ # Try pulling the latest changes.
+ # Limit the number of failed trials to 10.
+ failure_count, failure_limit = 0, 10
+ while True:
+ pull_latest = subprocess.run(["git",
+ "pull",
+ "origin",
+ "master"],
+ cwd=qemu_path,
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if pull_latest.returncode:
+ failure_count += 1
+ if failure_count == failure_limit:
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Trial {}/{}: Failed to pull QEMU".format(
+ failure_count, failure_limit),
+ file=sys.stderr, flush=True)
+ clean_exit(qemu_path, "")
+ else:
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Trial {}/{}: Failed to pull QEMU"
+ " ... retrying again in a minute!".format(
+ failure_count, failure_limit),
+ file=sys.stderr, flush=True)
+ time.sleep(60)
+ else:
+ break
+
+
+def git_clone(qemu_path: str) -> None:
+ """
+ Clone QEMU from Git.
+
+ Parameters:
+ qemu_path (str): Absolute path to clone the QEMU repo to
+ """
+ # Try cloning QEMU.
+ # Limit the number of failed trials to 10.
+ failure_count, failure_limit = 0, 10
+ while True:
+ clone_qemu = subprocess.run(["git",
+ "clone",
+ "https://git.qemu.org/git/qemu.git",
+ qemu_path],
+ check=False)
+ if clone_qemu.returncode:
+ failure_count += 1
+ if failure_count == failure_limit:
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Trial {}/{}: Failed to clone QEMU".format(
+ failure_count, failure_limit),
+ file=sys.stderr, flush=True)
+ clean_exit(qemu_path, "")
+ else:
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Trial {}/{}: Failed to clone QEMU"
+ " ... retrying again in a minute!".format(
+ failure_count, failure_limit),
+ file=sys.stderr, flush=True)
+ time.sleep(60)
+ else:
+ break
+
+
+def build_qemu(qemu_path: str, git_tag: str, targets: List[str]) -> None:
+ """
+ Checkout the Git tag then configure and build QEMU.
+
+ Parameters:
+ qemu_path (str): Absolute path to QEMU
+ git_tag (str): Git tag to checkout before building
+ targets (List[str]): List of targets to configure QEMU for
+ """
+
+ # Clean the QEMU build path
+ qemu_build_path = os.path.join(qemu_path, "build-gcc")
+ if os.path.isdir(qemu_build_path):
+ shutil.rmtree(qemu_build_path)
+ os.mkdir(qemu_build_path)
+
+ git_checkout(git_tag, qemu_path)
+
+ # Specify target list for configuring QEMU
+ target_list = ["{}-linux-user".format(target) for target in targets]
+
+ # Configure QEMU
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Running 'configure' for {}".format(git_tag),
+ file=sys.stderr, flush=True)
+ configure = subprocess.run(["../configure",
+ "--disable-system",
+ "--disable-tools",
+ "--target-list={}".
+ format(",".join(target_list))],
+ cwd=qemu_build_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if configure.returncode:
+ clean_exit(qemu_path, configure.stderr.decode("utf-8"))
+
+ # Run "make -j$(nproc)"
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Running 'make' for {}".format(git_tag), file=sys.stderr,
+ flush=True)
+ make = subprocess.run(["make",
+ "-j",
+ str(multiprocessing.cpu_count())],
+ cwd=qemu_build_path,
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if make.returncode:
+ clean_exit(qemu_path, make.stderr.decode("utf-8"))
+
+
+def compile_target(benchmark_path: str, compiled_benchmark_path: str,
+ target_compiler: str) -> None:
+ """
+ Compile a benchmark using the provided cross compiler.
+
+ Parameters:
+ benchmarks_path (str): Absolute path to benchmark
+ compiled_benchmark_path (str): Path to the output executable
+ target_compiler (str): Cross compiler
+ """
+ compile_benchmark = subprocess.run([target_compiler,
+ "-O2",
+ "-static",
+ "-w",
+ benchmark_path,
+ "-o",
+ compiled_benchmark_path],
+ check=False)
+ if compile_benchmark.returncode:
+ sys.exit("Compilation of {} failed".format(
+ os.path.split(compiled_benchmark_path)[1]))
+
+
+def measure_instructions(
+ benchmark_path: str, benchmarks_executables_dir_path: str,
+ qemu_path: str, targets: List[str]) -> List[List[Union[str, int]]]:
+ """
+ Measure the number of instructions when running an program with QEMU.
+
+ Parameters:
+ benchmarks_path (str): Absolute path to benchmark
+ benchmarks_executables_dir_path (str): Absolute path to the executables
+ qemu_path (str): Absolute path to QEMU
+ targets (List[str]): List of QEMU targets
+
+ Returns:
+ (List[List[Union[str, int]]]): [[target_name, instructions],[...],...]
+ """
+
+ benchmark_name = get_benchmark_name(benchmark_path)
+ executable_parent_dir_path = get_executable_parent_dir_path(
+ benchmark_path, benchmarks_executables_dir_path)
+ qemu_build_path = os.path.join(qemu_path, "build-gcc")
+
+ instructions: List[List[Union[str, int]]] = []
+
+ for target in targets:
+ executable_path = os.path.join(
+ executable_parent_dir_path, "{}-{}".format(benchmark_name, target))
+
+ qemu_exe_path = os.path.join(qemu_build_path,
+ "{}-linux-user".format(target),
+ "qemu-{}".format(target))
+
+ with tempfile.NamedTemporaryFile() as tmpfile:
+ run_callgrind = subprocess.run(["valgrind",
+ "--tool=callgrind",
+ "--callgrind-out-file={}".format(
+ tmpfile.name),
+ qemu_exe_path,
+ executable_path],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if run_callgrind.returncode == 1:
+ clean_exit(qemu_path, run_callgrind.stderr.decode("utf-8"))
+
+ callgrind_output = run_callgrind.stderr.decode("utf-8").split("\n")
+ instructions.append([target, int(callgrind_output[8].split(" ")[-1])])
+
+ return instructions
+
+
+def measure_master_instructions(
+ reference_version_path: str, reference_commit_hash: str,
+ latest_version_path: str, benchmark_path: str,
+ benchmarks_executables_dir_path: str, qemu_path: str,
+ targets: List[str]) -> List[List[Union[str, int]]]:
+ """
+ Measure the latest QEMU "master" instructions and also append the latest
+ instructions and reference version instructions to the instructions list.
+
+ Parameters:
+ reference_version_path (str): Absolute path to reference version results
+ reference_commit_hash (str): Git hash of the reference version
+ latest_version_path (str): Absolute path to the latest version results
+ benchmark_path (str): Absolute path to benchmark
+ benchmarks_executables_dir_path (str):
+ Absolute path to the executables of the benchmark
+ qemu_path (str): Absolute path to QEMU
+ targets (List[str]): List of QEMU targets
+
+
+ Return:
+ (List[List[Union[str, int]]]):
+ [[target_name, instructions, comparison_instructions],[...],...]
+ comparsion_instructions: *[latest, reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ """
+ benchmark_name = get_benchmark_name(benchmark_path)
+
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Measuring instructions for master - {}".format(benchmark_name),
+ file=sys.stderr, flush=True)
+
+ instructions = measure_instructions(
+ benchmark_path, benchmarks_executables_dir_path, qemu_path, targets)
+
+ reference_result = "{}-{}-results.csv".format(
+ reference_commit_hash, benchmark_name)
+ reference_result_path = os.path.join(
+ reference_version_path, reference_result)
+
+ # Find if this benchmark has a record in the latest results
+ latest_result = ""
+ latest_results = os.listdir(latest_version_path)
+ for result in latest_results:
+ if result.split("-")[1] == benchmark_name:
+ latest_result = result
+ break
+
+ # Append instructions from latest version if available
+ if latest_result != "":
+ latest_result_path = os.path.join(latest_version_path, latest_result)
+ with open(latest_result_path, "r") as file:
+ file.readline()
+ for target_instructions in instructions:
+ target_instructions.append(
+ int(file.readline().split('"')[1].replace(",", "")))
+ # Delete the latest results. The directory will contain the new latest
+ # when the new "master" results are stored later.
+ os.unlink(latest_result_path)
+
+ # Append instructions from reference version
+ with open(reference_result_path, "r") as file:
+ file.readline()
+ for target_instructions in instructions:
+ target_instructions.append(
+ int(file.readline().split('"')[1].replace(",", "")))
+
+ return instructions
+
+
+def calculate_percentage(old_instructions: int, new_instructions: int) -> str:
+ """
+ Calculate the change in percentage between two instruction counts
+
+ Parameters:
+ old_instructions (int): Old number
+ new_instructions (int): New number
+
+ Return:
+ (str): [+|-][change][%] or "-----" in case of 0.01% change
+ """
+ percentage = round(((new_instructions - old_instructions) /
+ old_instructions) * 100, 3)
+ return format_percentage(percentage)
+
+
+def format_percentage(percentage: float) -> str:
+ """
+ Format the percentage value to add +|- and %.
+
+ Parameters:
+ percentage (float): Percentage
+
+ Returns:
+ (str): Formatted percentage string
+ """
+ if abs(percentage) <= 0.01:
+ return "-----"
+ return "+" + str(percentage) + "%" if percentage > 0 \
+ else str(percentage) + "%"
+
+
+def calculate_change(instructions: List[List[Union[str, int]]]) -> None:
+ """
+ Calculate the change in the recorded instructions for master compared to
+ latest results and reference version results.
+
+ Parameters:
+ instructions (List[List[Union[str, int]]]):
+ [[target_name, instructions, comparison_instructions],[...],...]
+ comparsion_instructions: *[latest, reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ """
+ for target_instructions in instructions:
+ target_instructions[-1] = calculate_percentage(
+ int(target_instructions[-1]), int(target_instructions[1]))
+ # If latest instructions exists
+ if len(target_instructions) == 4:
+ target_instructions[-2] = calculate_percentage(
+ int(target_instructions[-2]), int(target_instructions[1]))
+
+
+def calculate_average(results: List[List[List[Union[str, int]]]],
+ targets: List[str],
+ num_benchmarks: int) -> List[List[Union[str, int]]]:
+ """
+ Calculate the average results for each target for all benchmarks.
+
+ Parameters:
+ results (List[List[List[Union[str, int]]]]):
+ [[target_name, instructions, comparison_instructions],[...],...]
+ comparsion_instructions: *[latest, reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ targets (List[str]): List of target names
+ num_benchmarks (int): Number of benchmarks
+
+ Return:
+ (List[List[Union[str, int]]]):
+ [[target_name, average_instructions, \
+ comparison_instructions],[...],...]
+ comparsion_instructions: *[average_latest, average_reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ """
+ average_instructions: List[List[Union[str, int]]] = []
+
+ for i, target in enumerate(targets):
+ average_instructions.append([target])
+
+ total_instructions = 0
+ total_latest_percentages: Optional[float] = 0.0
+ total_reference_percentages = 0.0
+
+ for instructions in results:
+ total_instructions += int(instructions[i][1])
+ if instructions[i][3] != "-----":
+ total_reference_percentages += float(
+ str(instructions[i][3])[:-1])
+ if total_latest_percentages is not None:
+ if instructions[i][2] != "N/A":
+ if instructions[i][2] != "-----":
+ total_latest_percentages += float(
+ str(instructions[i][2])[:-1])
+ else:
+ total_latest_percentages = None
+
+ avg_instructions = total_instructions // num_benchmarks
+ avg_reference_percentages = format_percentage(
+ round(total_reference_percentages / num_benchmarks, 3))
+ avg_latest_percentages = format_percentage(
+ round(total_latest_percentages / num_benchmarks, 3)) \
+ if total_latest_percentages is not None else "N/A"
+
+ average_instructions[-1].extend([avg_instructions,
+ avg_latest_percentages,
+ avg_reference_percentages])
+
+ return average_instructions
+
+
+def write_to_csv(instructions: List[List[Union[str, int]]],
+ output_csv_path: str, percentages: bool = False,
+ reference_version: str = "") -> None:
+ """
+ Write the [Target, Instructions] for each target in a CSV file.
+ comparison_instructions are ignored.
+
+ Parameters:
+ instructions (List[List[Union[str, int]]]):
+ [[target_name, instructions, comparison_instructions],[...],...]
+ comparsion_instructions: *[latest, reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ output_csv_path (str): Absolute path to output CSV file
+ percentages (bool): Add percentages to the output CSV file
+ """
+ with open(output_csv_path, "w") as file:
+ writer = csv.writer(file)
+ header = ["Target", "Instructions"]
+ if percentages:
+ header.extend(["Latest", reference_version])
+ writer.writerow(header)
+ for target_instructions in instructions:
+ row = []
+ row.extend([target_instructions[0], format(
+ target_instructions[1], ",")])
+ if percentages:
+ row.extend(target_instructions[2:])
+ writer.writerow(row)
+
+
+def print_table(instructions: str, text: str, reference_version: str) -> None:
+ """
+ Print the results in a tabular form
+
+ Parameters:
+ instructions (List[List[Union[str, int]]]):
+ [[target_name, instructions, comparison_instructions],[...],...]
+ comparsion_instructions: *[latest, reference]
+ If latest is not available,
+ then comparsion_instructions = reference
+ text (str): Text be added to the table header
+ reference_version (str): Reference version used in these results
+ """
+ print("{}\n{}\n{}".
+ format("-" * 56, text, "-" * 56))
+
+ print('{:<10} {:>20} {:>10} {:>10}\n{} {} {} {}'.
+ format('Target',
+ 'Instructions',
+ 'Latest',
+ reference_version,
+ '-' * 10,
+ '-' * 20,
+ '-' * 10,
+ '-' * 10))
+
+ for target_change in instructions:
+ # Replace commas with spaces in instruction count
+ # for easier readability.
+ formatted_instructions = format(
+ target_change[1], ",").replace(",", " ")
+ print('{:<10} {:>20} {:>10} {:>10}'.format(
+ target_change[0], formatted_instructions, *target_change[2:]))
+
+ print("-" * 56)
+
+
+def clean_exit(qemu_path: str, error_message: str) -> None:
+ """
+ Clean up intermediate files and exit.
+
+ Parameters:
+ qemu_path (str): Absolute path to QEMU
+ error_message (str): Error message to display after exiting
+ """
+ # Clean the QEMU build path
+ qemu_build_path = os.path.join(qemu_path, "build-gcc")
+ if os.path.isdir(qemu_build_path):
+ shutil.rmtree(qemu_build_path)
+ sys.exit(error_message)
+
+
+def verify_executables(benchmark_paths: List[str], targets: Dict[str, str],
+ benchmarks: List[Dict[str, str]],
+ benchmarks_executables_dir_path: str) -> None:
+ """
+ Verify that all executables exist for each benchmark.
+
+ Parameters:
+ benchmark_paths (List[str]): List of all paths to benchmarks
+ targets (Dict[str, str]): Dictionary the contains for each target,
+ target_name: target_compiler
+ benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path)
+ benchmarks_executables_dir_path (str): Absolute path to the executables dir
+ """
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Verifying executables of {} benchmarks for {} targets".
+ format(len(benchmark_paths), len(targets)),
+ file=sys.stderr, flush=True)
+
+ for benchmark in benchmarks:
+ executable_parent_dir_path = get_executable_parent_dir_path(
+ benchmark["path"], benchmarks_executables_dir_path)
+
+ # Verify that the exists for this benchmark executables, if not,
+ # create it
+ if not os.path.isdir(executable_parent_dir_path):
+ os.mkdir(executable_parent_dir_path)
+
+ for target_name, target_compiler in targets.items():
+ compiled_benchmark = "{}-{}".format(
+ benchmark["name"], target_name)
+ compiled_benchmark_path = os.path.join(
+ executable_parent_dir_path, compiled_benchmark)
+ # Verify that the the executable for this target is available,
+ # if not, compile it
+ if not os.path.isfile(compiled_benchmark_path):
+ compile_target(benchmark["path"],
+ compiled_benchmark_path,
+ target_compiler)
+
+
+def verify_reference_results(reference_version: str, qemu_path: str,
+ benchmarks: List[Dict[str, str]],
+ reference_version_results_dir_path: str,
+ targets: List[str],
+ benchmarks_executables_dir_path: str) -> None:
+ """
+ Verify that results are available for reference version.
+ If results are missing, build QEMU for the reference version then perform
+ the measurements.
+
+ Paramters:
+ reference_version (str): Reference QEMU version
+ qemu_path (str): Absolute path to QEMU
+ benchmark_paths (List[str]): List of all paths to benchmarks
+ reference_version_results_dir_path (str): Absolute path to the reference
+ version results dir
+ targets (List[str]): Target names
+ benchmarks (List[Dict[str, str]]): Benchmarks data (name, parent_dir, path)
+ benchmarks_executables_dir_path (str): Path to the root executables dir
+ """
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Verifying results of reference version {}".
+ format(reference_version), file=sys.stderr, flush=True)
+
+ # Set flag to know if QEMU was built for reference version before
+ did_build_reference = False
+
+ latest_commit_hash = get_commit_hash(reference_version, qemu_path)
+
+ for benchmark in benchmarks:
+ benchmark_results_dir_path = os.path.join(
+ reference_version_results_dir_path, benchmark["parent_dir"])
+
+ # Verify that the results directory for the benchmark exists, if not,
+ # create it
+ if not os.path.isdir(benchmark_results_dir_path):
+ os.mkdir(benchmark_results_dir_path)
+
+ # Verify that the the results.csv file for the benchmark exits, if not,
+ # create it
+ results_path = os.path.join(benchmark_results_dir_path,
+ "{}-{}-results.csv".
+ format(latest_commit_hash,
+ benchmark["name"]))
+ if not os.path.isfile(results_path):
+ # Only build qemu if reference version wasn't built before
+ if not did_build_reference:
+ build_qemu(qemu_path, reference_version, targets)
+ did_build_reference = True
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Measuring instructions for reference version {} - {}".
+ format(reference_version, benchmark["name"]),
+ file=sys.stderr, flush=True)
+ instructions = measure_instructions(
+ benchmark["path"],
+ benchmarks_executables_dir_path,
+ qemu_path,
+ targets)
+ write_to_csv(instructions, results_path)
+
+
+def verify_requirements() -> None:
+ """
+ Verify that all script requirements are installed (valgrind & git).
+ """
+ # Insure that valgrind is installed
+ check_valgrind_installation = subprocess.run(["which", "valgrind"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if check_valgrind_installation.returncode:
+ sys.exit("Please install valgrind before running the script.")
+
+ # Insure that git is installed
+ check_git_installation = subprocess.run(["which", "git"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+ if check_git_installation.returncode:
+ sys.exit("Please install git before running the script.")
+
+
+def main():
+ """
+ Parse the command line arguments then start the execution.
+ Output on STDOUT represents the nightly test results.
+ Output on STDERR represents the execution log and errors if any.
+ Output on STDERR must be redirected to either /dev/null or to a log file.
+
+ Syntax:
+ nightly_tests_core.py [-h] [-r REF]
+ Optional arguments:
+ -h, --help Show this help message and exit
+ -r REF, --reference REF
+ Reference QEMU version - Default is v5.1.0
+ Example of usage:
+ nightly_tests_core.py -r v5.1.0 2>log.txt
+ """
+ parser = argparse.ArgumentParser()
+ parser.add_argument("-r", "--reference", dest="ref",
+ default="v5.1.0",
+ help="Reference QEMU version - Default is v5.1.0")
+ reference_version = parser.parse_args().ref
+
+ targets = {
+ "aarch64": "aarch64-linux-gnu-gcc",
+ "alpha": "alpha-linux-gnu-gcc",
+ "arm": "arm-linux-gnueabi-gcc",
+ "hppa": "hppa-linux-gnu-gcc",
+ "m68k": "m68k-linux-gnu-gcc",
+ "mips": "mips-linux-gnu-gcc",
+ "mipsel": "mipsel-linux-gnu-gcc",
+ "mips64": "mips64-linux-gnuabi64-gcc",
+ "mips64el": "mips64el-linux-gnuabi64-gcc",
+ "ppc": "powerpc-linux-gnu-gcc",
+ "ppc64": "powerpc64-linux-gnu-gcc",
+ "ppc64le": "powerpc64le-linux-gnu-gcc",
+ "riscv64": "riscv64-linux-gnu-gcc",
+ "s390x": "s390x-linux-gnu-gcc",
+ "sh4": "sh4-linux-gnu-gcc",
+ "sparc64": "sparc64-linux-gnu-gcc",
+ "x86_64": "gcc"
+ }
+
+ # Verify that the script requirements are installed
+ verify_requirements()
+
+ # Get required paths
+ nightly_tests_dir_path = pathlib.Path(__file__).parent.parent.absolute()
+ benchmarks_dir_path = os.path.join(nightly_tests_dir_path, "benchmarks")
+ benchmarks_source_dir_path = os.path.join(benchmarks_dir_path, "source")
+ benchmarks_executables_dir_path = os.path.join(
+ benchmarks_dir_path, "executables")
+
+ # Verify that If the executables directory exists, if not, create it
+ if not os.path.isdir(benchmarks_executables_dir_path):
+ os.mkdir(benchmarks_executables_dir_path)
+
+ # Get absolute path to all available benchmarks
+ benchmark_paths = sorted([y for x in os.walk(benchmarks_source_dir_path)
+ for y in glob.glob(os.path.join(x[0], '*.c'))])
+
+ benchmarks = [{
+ "name": get_benchmark_name(benchmark_path),
+ "parent_dir": get_benchmark_parent_dir(benchmark_path),
+ "path": benchmark_path} for benchmark_path in benchmark_paths]
+
+ # Verify that all executables exist for each benchmark
+ verify_executables(benchmark_paths, targets, benchmarks,
+ benchmarks_executables_dir_path)
+
+ # Set QEMU path and clone from Git if the path doesn't exist
+ qemu_path = os.path.join(nightly_tests_dir_path, "qemu-nightly")
+ if not os.path.isdir(qemu_path):
+ # Clone QEMU into the temporary directory
+ print(datetime.datetime.utcnow().isoformat(),
+ "- Fetching QEMU: ", end="", flush=True, file=sys.stderr)
+ git_clone(qemu_path)
+ print("\n", file=sys.stderr, flush=True)
+
+ # Verify that the results directory exists, if not, create it
+ results_dir_path = os.path.join(nightly_tests_dir_path, "results")
+ if not os.path.isdir(results_dir_path):
+ os.mkdir(results_dir_path)
+
+ # Verify that the reference version results directory exists, if not,
+ # create it
+ reference_version_results_dir_path = os.path.join(
+ results_dir_path, reference_version)
+ if not os.path.isdir(reference_version_results_dir_path):
+ os.mkdir(reference_version_results_dir_path)
+
+ # Verify that previous results are available for reference version
+ verify_reference_results(reference_version, qemu_path, benchmarks,
+ reference_version_results_dir_path,
+ targets.keys(), benchmarks_executables_dir_path)
+
+ # Compare results with the latest QEMU master
+ # -------------------------------------------------------------------------
+ # Verify that the "latest" results directory exists, if not,
+ # create it
+ latest_version_results_dir_path = os.path.join(results_dir_path, "latest")
+ if not os.path.isdir(latest_version_results_dir_path):
+ os.mkdir(latest_version_results_dir_path)
+
+ # Verify that the "history" results directory exists, if not, create it
+ history_results_dir_path = os.path.join(results_dir_path, "history")
+ if not os.path.isdir(history_results_dir_path):
+ os.mkdir(history_results_dir_path)
+
+ # Build QEMU for master
+ build_qemu(qemu_path, "master", targets.keys())
+
+ # Get the commit hash for the top commit at master
+ master_commit_hash = get_commit_hash("master", qemu_path)
+
+ # Print report summary header
+ print("{}\n{}".format("-" * 56, "{} SUMMARY REPORT - COMMIT {}".
+ format(" "*11, master_commit_hash)))
+
+ # For each benchmark, compare the current master results with
+ # latest and reference version
+ results = []
+ for benchmark in benchmarks:
+ reference_version_benchmark_results_dir_path = os.path.join(
+ reference_version_results_dir_path, benchmark["parent_dir"])
+ latest_version_benchmark_results_dir_path = os.path.join(
+ latest_version_results_dir_path, benchmark["parent_dir"])
+ history_benchmark_results_dir_path = os.path.join(
+ history_results_dir_path, benchmark["parent_dir"])
+
+ # Verify that the the benchmark directory exists in the "latest"
+ # directory, if not, create it
+ if not os.path.isdir(latest_version_benchmark_results_dir_path):
+ os.mkdir(latest_version_benchmark_results_dir_path)
+
+ # Verify that the the benchmark directory exists in the "history"
+ # directory, if not, create it
+ if not os.path.isdir(history_benchmark_results_dir_path):
+ os.mkdir(history_benchmark_results_dir_path)
+
+ # Obtain the instructions array which will contain for each target,
+ # the target name, the number of "master" instructions,
+ # "latest" instructions if available,
+ # and "reference version" instructions
+ instructions = measure_master_instructions(
+ reference_version_benchmark_results_dir_path,
+ get_commit_hash(reference_version, qemu_path),
+ latest_version_benchmark_results_dir_path,
+ benchmark["path"],
+ benchmarks_executables_dir_path,
+ qemu_path,
+ targets.keys())
+
+ # Update the "latest" directory with the new results form master
+ updated_latest_version_benchmark_results = os.path.join(
+ latest_version_benchmark_results_dir_path,
+ "{}-{}-results.csv".format(master_commit_hash, benchmark["name"]))
+ write_to_csv(instructions, updated_latest_version_benchmark_results)
+
+ calculate_change(instructions)
+
+ # Insert "N/A" for targets that don't have "latest" instructions
+ for target_instructions in instructions:
+ if len(target_instructions) == 3:
+ target_instructions.insert(2, "N/A")
+
+ history_benchmark_results = os.path.join(
+ history_benchmark_results_dir_path,
+ "{}-{}-results.csv".format(master_commit_hash, benchmark["name"]))
+ write_to_csv(instructions, history_benchmark_results,
+ True, reference_version)
+
+ # Store the results
+ results.append([benchmark["name"], instructions])
+
+ # Calculate the average instructions for each target
+ # Only send the instructions as results without the benchmark names
+ average_instructions = calculate_average(
+ [result[1] for result in results], targets.keys(), len(benchmarks))
+
+ # Save average results to results/history directory
+ average_results = os.path.join(
+ history_results_dir_path,
+ "{}-average-results.csv".format(master_commit_hash))
+ write_to_csv(average_instructions, average_results,
+ True, reference_version)
+
+ # Print results
+ print_table(average_instructions, " "*20 +
+ "AVERAGE RESULTS", reference_version)
+ print("\n", " "*17, "DETAILED RESULTS")
+ for [benchmark_name, instructions] in results:
+ print_table(instructions,
+ "Test Program: " + benchmark_name,
+ reference_version)
+
+ # Cleanup (delete the build directory)
+ shutil.rmtree(os.path.join(qemu_path, "build-gcc"))
+ # -------------------------------------------------------------------------
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/performance/nightly-tests/scripts/run_nightly_tests.py b/tests/performance/nightly-tests/scripts/run_nightly_tests.py
new file mode 100755
index 0000000000..ea65be60dc
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/run_nightly_tests.py
@@ -0,0 +1,135 @@
+#!/usr/bin/env python3
+
+"""
+Entry point script for running nightly performance tests on QEMU.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+import datetime
+import io
+import os
+import subprocess
+import time
+import sys
+from send_email import send_email
+
+
+# Record system hardware information
+with open('/proc/cpuinfo', 'r') as cpuinfo:
+ for line in cpuinfo:
+ if line.startswith('model name'):
+ HOST_CPU = line.rstrip('\n').split(':')[1].strip()
+ break
+with open('/proc/meminfo', 'r') as meminfo:
+ for line in meminfo:
+ if line.startswith('MemTotal'):
+ HOST_MEMORY_KB = int(line.rstrip('\n').split(':')[1].
+ strip().split(' ')[0])
+ HOST_MEMORY = str(round(HOST_MEMORY_KB / (1024 * 1024), 2)) + " GB"
+ break
+
+# Find path for the "nightly_tests_core.py" script
+NIGHTLY_TESTS_CORE_PATH = os.path.join(
+ os.path.dirname(os.path.abspath(__file__)),
+ "nightly_tests_core.py")
+
+NIGHTLY_TESTS_ARGUMENTS = sys.argv[1:]
+
+# Start the nightly test
+START_EPOCH = time.time()
+RUN_NIGHTLY_TESTS = subprocess.run([NIGHTLY_TESTS_CORE_PATH,
+ *NIGHTLY_TESTS_ARGUMENTS],
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ check=False)
+END_EPOCH = time.time()
+
+# Perform time calculations
+EXECUTION_TIME = datetime.timedelta(seconds=END_EPOCH - START_EPOCH)
+TEST_DATE = datetime.datetime.utcfromtimestamp(
+ START_EPOCH).strftime('%A, %B %-d, %Y')
+START_TIME = datetime.datetime.utcfromtimestamp(
+ START_EPOCH).strftime('%Y-%m-%d %H:%M:%S')
+END_TIME = datetime.datetime.utcfromtimestamp(
+ END_EPOCH).strftime('%Y-%m-%d %H:%M:%S')
+
+# Get nightly test status
+if RUN_NIGHTLY_TESTS.returncode:
+ STATUS = "FAILURE"
+else:
+ STATUS = "SUCCESS"
+
+# Initialize a StringIO to print all the output into
+OUTPUT = io.StringIO()
+
+# Print the nightly test statistical information
+print("{:<17}: {}\n{:<17}: {}\n".
+ format("Host CPU",
+ HOST_CPU,
+ "Host Memory",
+ HOST_MEMORY), file=OUTPUT)
+print("{:<17}: {}\n{:<17}: {}\n{:<17}: {}\n".
+ format("Start Time (UTC)",
+ START_TIME,
+ "End Time (UTC)",
+ END_TIME,
+ "Execution Time",
+ EXECUTION_TIME), file=OUTPUT)
+print("{:<17}: {}\n".format("Status", STATUS), file=OUTPUT)
+
+if STATUS == "SUCCESS":
+ print("Note:\nChanges denoted by '-----' are less than 0.01%.\n",
+ file=OUTPUT)
+
+# Print the nightly test stdout (main output)
+print(RUN_NIGHTLY_TESTS.stdout.decode("utf-8"), file=OUTPUT)
+
+# If the nightly test failed, print the stderr (error logs)
+if STATUS == "FAILURE":
+ print("{}\n{}\n{}".format("-" * 56,
+ " " * 18 + "ERROR LOGS",
+ "-" * 56), file=OUTPUT)
+ print(RUN_NIGHTLY_TESTS.stderr.decode("utf-8"), file=OUTPUT)
+
+
+# Temp file to store the output in case sending the email failed
+# with open("temp.txt", "w") as file:
+# file.write(OUTPUT.getvalue())
+
+# Use an HTML message to preserve monospace formatting
+HTML_MESSAGE = """\
+<html><body><pre>
+{body}
+</pre></body></html>
+""".format(body=OUTPUT.getvalue())
+OUTPUT.close()
+
+# Send the nightly test results email to the QEMU mailing list
+while True:
+ try:
+ send_email("[REPORT] Nightly Performance Tests - {}".format(TEST_DATE),
+ ["qemu-devel@nongnu.org"], HTML_MESSAGE)
+ except Exception: # pylint: disable=W0703
+ # Wait for a minute then retry sending
+ time.sleep(60)
+ continue
+ else:
+ break
diff --git a/tests/performance/nightly-tests/scripts/send_email.py b/tests/performance/nightly-tests/scripts/send_email.py
new file mode 100644
index 0000000000..e24e244f51
--- /dev/null
+++ b/tests/performance/nightly-tests/scripts/send_email.py
@@ -0,0 +1,56 @@
+"""
+Helper script for sending emails with the nightly performance test results.
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+
+import smtplib
+from typing import List
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.utils import COMMASPACE, formatdate
+
+
+GMAIL_USER = {"name": "",
+ "email": "",
+ "pass": ""}
+
+
+def send_email(subject: str, send_to: List[str], html: str) -> None:
+ """
+ Send an HTML email.
+
+ Parameters:
+ subject (str): Email subject
+ send_to (List(str)): List of recipients
+ html (str): HTML message
+ """
+ msg = MIMEMultipart('alternative')
+ msg['From'] = "{} <{}>".format(GMAIL_USER["name"], GMAIL_USER["email"])
+ msg['To'] = COMMASPACE.join(send_to)
+ msg['Date'] = formatdate(localtime=True)
+ msg['Subject'] = subject
+
+ msg.attach(MIMEText(html, 'html'))
+
+ server = smtplib.SMTP_SSL('smtp.gmail.com', 465)
+ server.login(GMAIL_USER["email"], GMAIL_USER["pass"])
+ server.sendmail(msg['From'], send_to, msg.as_string())
+ server.quit()
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (6 preceding siblings ...)
2020-08-28 10:41 ` [PATCH 7/9] tests/performance: Add nightly tests Ahmed Karaman
@ 2020-08-28 10:41 ` Ahmed Karaman
2020-09-02 8:37 ` Aleksandar Markovic
2020-08-28 10:41 ` [PATCH 9/9] scripts/performance: Add topN_system.py script Ahmed Karaman
8 siblings, 1 reply; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:41 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Add a new 'tests/performance' directory under the 'Performance Tools
and Tests' subsection.
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
MAINTAINERS | 1 +
1 file changed, 1 insertion(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 5a22c8be42..8923307642 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -3124,3 +3124,4 @@ Performance Tools and Tests
M: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
S: Maintained
F: scripts/performance/
+F: tests/performance/
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* [PATCH 9/9] scripts/performance: Add topN_system.py script
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
` (7 preceding siblings ...)
2020-08-28 10:41 ` [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection Ahmed Karaman
@ 2020-08-28 10:41 ` Ahmed Karaman
8 siblings, 0 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-08-28 10:41 UTC (permalink / raw)
To: qemu-devel, aleksandar.qemu.devel, philmd, alex.bennee, eblake,
ldoktor, jsnow, rth, ehabkost, crosa
Cc: Ahmed Karaman
Python script for listing the topN executed QEMU functions in
system mode.
Syntax:
topN_system.py [-h] [-n <number of displayed top functions>] -- \
<qemu system executable> <qemu system options>
[-h] - Print the script arguments help message.
[-n] - Specify the number of top functions to print.
- If this flag is not specified, the tool defaults to 25.
Example of usage:
topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \
-initrd <initrd>
Example output:
Number of instructions: 150,991,381,071
No. Percentage Name
--- ---------- ------------------------------
1 11.30% helper_lookup_tb_ptr
2 7.01% liveness_pass_1
3 4.48% tcg_gen_code
4 3.41% tcg_optimize
5 1.84% tcg_out_opc.isra.13
6 1.78% helper_pcmpeqb_xmm
7 1.20% object_dynamic_cast_assert
8 1.00% cpu_exec
9 0.99% tcg_temp_new_internal
10 0.88% tb_htable_lookup
11 0.84% object_class_dynamic_cast_assert
12 0.81% init_ts_info
13 0.80% tlb_set_page_with_attrs
14 0.77% victim_tlb_hit
15 0.75% tcg_out_sib_offset
16 0.62% tcg_op_alloc
17 0.61% helper_pmovmskb_xmm
18 0.58% disas_insn.isra.50
19 0.56% helper_pcmpgtb_xmm
20 0.56% address_space_ldq
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
---
scripts/performance/topN_system.py | 158 +++++++++++++++++++++++++++++
1 file changed, 158 insertions(+)
create mode 100755 scripts/performance/topN_system.py
diff --git a/scripts/performance/topN_system.py b/scripts/performance/topN_system.py
new file mode 100755
index 0000000000..9b1f1a66c7
--- /dev/null
+++ b/scripts/performance/topN_system.py
@@ -0,0 +1,158 @@
+#!/usr/bin/env python3
+
+"""
+Print the top N most executed functions in QEMU system mode emulation.
+
+Syntax:
+topN_system.py [-h] [-n <number of displayed top functions>] -- \
+ <qemu system executable> <qemu system options>
+
+[-h] - Print the script arguments help message.
+[-n] - Specify the number of top functions to print.
+ - If this flag is not specified, the tool defaults to 25.
+
+Example of usage:
+topN_system.py -n 20 -- qemu-system-x86_64 -m 1024 -kernel <kernel> \
+ -initrd <initrd>
+
+This file is a part of the project "TCG Continuous Benchmarking".
+
+Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
+Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program. If not, see <https://www.gnu.org/licenses/>.
+"""
+
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+
+
+# Parse the command line arguments
+PARSER = argparse.ArgumentParser(
+ usage="usage: topN_system.py [-h] [-n <number of displayed top functions>]"
+ " -- <qemu system executable> <qemu system options>")
+
+PARSER.add_argument("-n", dest="top", type=int, default=25,
+ help="Specify the number of top functions to print.")
+
+PARSER.add_argument("command", type=str, nargs="+", help=argparse.SUPPRESS)
+
+ARGS = PARSER.parse_args()
+
+# Extract the needed variables from the args
+COMMAND = ARGS.command
+TOP = ARGS.top
+
+# Ensure that perf is installed
+CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
+ stdout=subprocess.DEVNULL,
+ check=False)
+if CHECK_PERF_PRESENCE.returncode:
+ sys.exit("Please install perf before running the script!")
+
+# Ensure user has previllage to run perf
+CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ check=False)
+if CHECK_PERF_EXECUTABILITY.returncode:
+ sys.exit("""
+Error:
+You may not have permission to collect stats.
+Consider tweaking /proc/sys/kernel/perf_event_paranoid,
+which controls use of the performance events system by
+unprivileged users (without CAP_SYS_ADMIN).
+ -1: Allow use of (almost) all events by all users
+ Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK
+ 0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN
+ Disallow raw tracepoint access by users without CAP_SYS_ADMIN
+ 1: Disallow CPU event access by users without CAP_SYS_ADMIN
+ 2: Disallow kernel profiling by users without CAP_SYS_ADMIN
+To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
+ kernel.perf_event_paranoid = -1
+
+* Alternatively, you can run this script under sudo privileges.
+""")
+
+# Run perf and save all intermediate files in a temporary directory
+with tempfile.TemporaryDirectory() as tmpdir:
+ RECORD_PATH = os.path.join(tmpdir, "record.data")
+ REPORT_PATH = os.path.join(tmpdir, "report.txt")
+
+ PERF_RECORD = subprocess.run((["perf",
+ "record",
+ "-e",
+ "instructions",
+ "--output="+RECORD_PATH] +
+ COMMAND),
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.PIPE,
+ check=False)
+ if PERF_RECORD.returncode:
+ sys.exit(PERF_RECORD.stderr.decode("utf-8"))
+
+ with open(REPORT_PATH, "w") as output:
+ PERF_REPORT = subprocess.run(["perf",
+ "report",
+ "--input=" + RECORD_PATH,
+ "--stdio"],
+ stdout=output,
+ stderr=subprocess.PIPE,
+ check=False)
+ if PERF_REPORT.returncode:
+ sys.exit(PERF_REPORT.stderr.decode("utf-8"))
+
+ # Save the reported data to FUNCTIONS[] and INSTRUCTIONS
+ with open(REPORT_PATH, "r") as data:
+ LINES = data.readlines()
+ # Read the number of instructions
+ INSTRUCTIONS = int(LINES[6].split()[-1])
+ # Continue reading:
+ # Only read lines that are not empty
+ # Only read lines that are not comments (comments start with #)
+ # Only read functions executed by qemu-system
+ FUNCTIONS = [line for line in LINES if line
+ and line[0] != "\n"
+ and line[0] != "#"
+ and "qemu-system" in line.split()[2]]
+
+
+# Limit the number of top functions to "TOP"
+NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
+
+# Store the data of the top functions in TOP_FUNCTIONS[]
+TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
+
+# Print total instructions
+print("\nNumber of instructions:", format(INSTRUCTIONS, ","))
+# Print table header
+print("\n{:>4} {:>10} {}\n{} {} {}".format("No.",
+ "Percentage",
+ "Name",
+ "-" * 4,
+ "-" * 10,
+ "-" * 30))
+
+# Print top N functions
+for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
+ function_data = function.split()
+ function_percentage = function_data[0]
+ function_name = function_data[-1]
+ function_invoker = " ".join(function_data[2:-2])
+ print("{:>4} {:>10} {}".format(index,
+ function_percentage,
+ function_name))
--
2.17.1
^ permalink raw reply related [flat|nested] 26+ messages in thread
* Re: [PATCH 7/9] tests/performance: Add nightly tests
2020-08-28 10:41 ` [PATCH 7/9] tests/performance: Add nightly tests Ahmed Karaman
@ 2020-09-02 8:36 ` Aleksandar Markovic
2020-09-02 13:26 ` Alex Bennée
1 sibling, 0 replies; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-02 8:36 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 52022 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> A nightly performance testing system to monitor any change in QEMU
> performance across seventeen different targets.
>
>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> The system includes eight different benchmarks to provide a variety
> of testing workloads.
>
> dijkstra_double:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are double values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> dijkstra_int32:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are int32 values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> matmult_double:
> Standard matrix multiplication of an n*n matrix of randomly generated
> double numbers from 0 to 100. The value of n is passed as an argument
> with the -n flag. The default value is 200.
>
> matmult_int32:
> Standard matrix multiplication of an n*n matrix of randomly generated
> integer numbers from 0 to 100. The value of n is passed as an
> argument with the -n flag. The default value is 200.
>
> qsort_double:
> Quick sort of an array of n randomly generated double numbers from 0
> to 1000. The value of n is passed as an argument with the -n flag.
> The default value is 300000.
>
> qsort_int32:
> Quick sort of an array of n randomly generated integer numbers from 0
> to 50000000. The value of n is passed as an argument with the -n
> flag.The default value is 300000.
>
> qsort_string:
> Quick sort of an array of 10000 randomly generated strings of size 8
> (including null terminating character). The sort process is repeated
> n number of times. The value of n is passed as an argument with the
> -n flag. The default value is 20.
>
> search_string:
> Search for the occurrence of a small string in a much larger random
> string (“needle in a hay”). The search process is repeated n number
> of times and each time, a different large random string (“hay”) is
> generated. The value of n can be specified using the -n flag. The
> default value is 20.
>
> Syntax:
> nightly_tests_core.py [-h] [-r REF]
> Optional arguments:
> -h, --help Show this help message and exit
> -r REF, --reference REF
> Reference QEMU version - Default is v5.1.0
> Example of usage:
> nightly_tests_core.py -r v5.1.0 2>log.txt
>
> The following report includes detailed setup and execution details
> of the system:
> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
> QEMU-Nightly-Performance-Tests/
>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> tests/performance/nightly-tests/README.md | 243 +++++
> .../source/dijkstra_double/dijkstra_double.c | 194 ++++
> .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
> .../source/matmult_double/matmult_double.c | 123 +++
> .../source/matmult_int32/matmult_int32.c | 121 +++
> .../source/qsort_double/qsort_double.c | 104 ++
> .../source/qsort_int32/qsort_int32.c | 103 ++
> .../source/qsort_string/qsort_string.c | 122 +++
> .../source/search_string/search_string.c | 110 +++
> .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
> .../scripts/run_nightly_tests.py | 135 +++
> .../nightly-tests/scripts/send_email.py | 56 ++
> 12 files changed, 2423 insertions(+)
> create mode 100644 tests/performance/nightly-tests/README.md
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_double/dijkstra_double.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_int32/dijkstra_int32.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_double/matmult_double.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_int32/matmult_int32.c
> create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_double/qsort_double.c
> create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_int32/qsort_int32.c
> create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_string/qsort_string.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> search_string/search_string.c
> create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_
> core.py
> create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
> tests.py
> create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
>
> diff --git a/tests/performance/nightly-tests/README.md
> b/tests/performance/nightly-tests/README.md
> new file mode 100644
> index 0000000000..6db3b351b3
> --- /dev/null
> +++ b/tests/performance/nightly-tests/README.md
> @@ -0,0 +1,243 @@
> +### QEMU Nightly Tests
> +
> +**Required settings:**
> +
> +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> +
> +For more details on how the system works, please check the [eighth
> report](https://ahmedkrmn.github.io/TCG-Continuous-
> Benchmarking/QEMU-Nightly-Performance-Tests/) of the "TCG Continuos
> Benchmarking" series.
> +
> +**Running the System:**
> +
> +The default reference version is v5.1.0. To specify a custom version,
> please use the `-r, --reference` flag.
> +
> +```bash
> +./run_nightly_tests.py
> +```
> +
> +**Output:**
> +
> +```
> +Host CPU : Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
> +Host Memory : 15.49 GB
> +
> +Start Time (UTC) : 2020-08-25 21:30:01
> +End Time (UTC) : 2020-08-25 22:02:37
> +Execution Time : 0:32:35.896990
> +
> +Status : SUCCESS
> +
> +Note:
> +Changes denoted by '-----' are less than 0.01%.
> +
> +--------------------------------------------------------
> + SUMMARY REPORT - COMMIT d1a2b51f
> +--------------------------------------------------------
> + AVERAGE RESULTS
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 158 355 274 ----- +1.693%
> +alpha 1 914 967 171 ----- +3.524%
> +arm 8 076 402 940 ----- +2.304%
> +hppa 4 261 685 987 -0.182% +3.164%
> +m68k 2 690 273 044 ----- +7.131%
> +mips 1 862 033 667 ----- +2.494%
> +mipsel 2 008 211 069 ----- +2.674%
> +mips64 1 918 635 565 ----- +2.818%
> +mips64el 2 051 565 677 ----- +3.026%
> +ppc 2 480 141 217 ----- +3.107%
> +ppc64 2 576 713 959 ----- +3.143%
> +ppc64le 2 558 853 539 ----- +3.173%
> +riscv64 1 406 704 050 ----- +2.65%
> +s390x 3 158 140 046 ----- +3.118%
> +sh4 2 364 449 748 ----- +3.33%
> +sparc64 3 318 544 783 ----- +3.851%
> +x86_64 1 775 844 158 ----- +2.156%
> +--------------------------------------------------------
> +
> + DETAILED RESULTS
> +--------------------------------------------------------
> +Test Program: dijkstra_double
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 3 062 583 464 ----- +1.424%
> +alpha 3 191 864 698 ----- +3.696%
> +arm 16 357 157 526 ----- +2.347%
> +hppa 7 228 376 315 -0.139% +3.086%
> +m68k 4 294 016 587 ----- +9.692%
> +mips 3 051 419 166 ----- +2.427%
> +mipsel 3 231 509 618 ----- +2.869%
> +mips64 3 245 837 754 ----- +2.596%
> +mips64el 3 414 195 796 ----- +3.021%
> +ppc 4 914 520 972 -0.041% +4.74%
> +ppc64 5 098 154 311 ----- +4.565%
> +ppc64le 5 082 419 054 ----- +4.58%
> +riscv64 2 192 294 915 ----- +1.955%
> +s390x 4 584 503 977 ----- +2.896%
> +sh4 3 949 036 447 ----- +3.464%
> +sparc64 4 586 203 546 ----- +4.237%
> +x86_64 2 484 092 105 ----- +1.75%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: dijkstra_int32
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 210 194 577 ----- +1.493%
> +alpha 1 494 133 274 ----- +2.15%
> +arm 8 262 935 967 ----- +2.665%
> +hppa 5 207 318 306 ----- +3.047%
> +m68k 1 725 856 962 ----- +2.527%
> +mips 1 495 227 032 ----- +1.492%
> +mipsel 1 497 147 869 ----- +1.479%
> +mips64 1 715 388 570 ----- +1.892%
> +mips64el 1 695 276 864 ----- +1.913%
> +ppc 2 014 557 389 ----- +1.819%
> +ppc64 2 206 267 901 ----- +2.139%
> +ppc64le 2 197 998 781 ----- +2.146%
> +riscv64 1 354 912 745 ----- +2.396%
> +s390x 2 916 247 062 ----- +1.241%
> +sh4 1 990 532 533 ----- +2.669%
> +sparc64 2 872 231 051 ----- +3.758%
> +x86_64 1 553 981 241 ----- +2.12%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: matmult_double
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 1 412 273 223 ----- +0.302%
> +alpha 3 233 991 649 ----- +7.473%
> +arm 8 545 173 979 ----- +1.088%
> +hppa 3 483 597 802 -1.267% +4.468%
> +m68k 3 919 065 529 ----- +18.431%
> +mips 2 344 774 894 ----- +4.091%
> +mipsel 3 329 886 464 ----- +5.177%
> +mips64 2 359 046 988 ----- +4.076%
> +mips64el 3 343 664 785 ----- +5.167%
> +ppc 3 209 457 051 ----- +3.246%
> +ppc64 3 287 503 981 ----- +3.173%
> +ppc64le 3 287 189 065 ----- +3.173%
> +riscv64 1 221 603 682 ----- +0.277%
> +s390x 2 874 199 923 ----- +5.827%
> +sh4 3 543 943 634 ----- +6.416%
> +sparc64 3 426 150 004 ----- +7.139%
> +x86_64 1 248 917 276 ----- +0.322%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: matmult_int32
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 598 681 621 ----- +0.585%
> +alpha 372 437 418 ----- +0.677%
> +arm 746 583 193 ----- +1.462%
> +hppa 674 278 359 ----- +1.183%
> +m68k 410 495 553 ----- +0.9%
> +mips 499 698 837 ----- +0.531%
> +mipsel 499 500 429 ----- +0.497%
> +mips64 481 554 664 ----- +0.599%
> +mips64el 465 057 054 ----- +0.619%
> +ppc 341 334 603 ----- +0.944%
> +ppc64 393 796 203 ----- +0.966%
> +ppc64le 393 977 298 ----- +0.965%
> +riscv64 351 709 769 ----- +0.785%
> +s390x 494 427 384 ----- +0.599%
> +sh4 402 668 444 ----- +0.899%
> +sparc64 495 952 959 ----- +1.192%
> +x86_64 402 928 461 ----- +0.833%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: qsort_double
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 709 683 624 ----- +2.417%
> +alpha 1 969 460 172 ----- +3.68%
> +arm 8 322 998 390 ----- +2.587%
> +hppa 3 188 301 995 -0.047% +2.9%
> +m68k 4 953 930 065 ----- +15.153%
> +mips 2 123 919 587 ----- +3.055%
> +mipsel 2 124 212 187 ----- +3.048%
> +mips64 1 999 047 826 ----- +3.405%
> +mips64el 1 996 426 772 ----- +3.409%
> +ppc 2 819 267 902 -0.021% +5.435%
> +ppc64 2 768 186 548 ----- +5.513%
> +ppc64le 2 724 803 772 -0.011% +5.603%
> +riscv64 1 638 328 937 ----- +4.021%
> +s390x 2 519 081 708 ----- +3.362%
> +sh4 2 595 545 154 ----- +2.994%
> +sparc64 3 988 986 587 ----- +2.747%
> +x86_64 2 033 468 588 ----- +3.234%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: qsort_int32
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 193 392 565 ----- +2.916%
> +alpha 1 521 291 933 ----- +4.193%
> +arm 3 465 445 043 ----- +2.756%
> +hppa 2 280 034 340 ----- +3.821%
> +m68k 1 843 189 041 ----- +3.583%
> +mips 1 558 024 873 ----- +3.863%
> +mipsel 1 560 583 980 ----- +3.846%
> +mips64 1 563 415 749 ----- +4.412%
> +mips64el 1 542 677 320 ----- +4.474%
> +ppc 1 728 698 880 ----- +3.665%
> +ppc64 1 842 444 545 ----- +3.555%
> +ppc64le 1 791 822 067 ----- +3.661%
> +riscv64 1 348 866 430 ----- +4.654%
> +s390x 2 184 073 151 ----- +3.319%
> +sh4 1 946 492 539 ----- +3.624%
> +sparc64 3 452 215 585 ----- +2.937%
> +x86_64 1 813 544 414 ----- +3.537%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: qsort_string
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 592 218 418 ----- +2.468%
> +alpha 1 855 834 626 ----- +3.487%
> +arm 7 347 721 165 ----- +2.682%
> +hppa 4 758 753 926 ----- +3.543%
> +m68k 2 376 811 462 ----- +3.567%
> +mips 2 166 608 045 ----- +2.532%
> +mipsel 2 163 392 541 ----- +2.528%
> +mips64 2 029 251 969 ----- +3.117%
> +mips64el 2 011 628 621 ----- +3.145%
> +ppc 2 492 942 463 ----- +2.673%
> +ppc64 2 464 702 554 ----- +2.488%
> +ppc64le 2 445 253 307 ----- +2.505%
> +riscv64 1 625 053 328 ----- +3.953%
> +s390x 4 194 608 798 ----- +6.623%
> +sh4 2 164 142 539 ----- +3.166%
> +sparc64 4 299 516 539 ----- +4.065%
> +x86_64 2 940 456 780 ----- +2.649%
> +--------------------------------------------------------
> +--------------------------------------------------------
> +Test Program: search_string
> +--------------------------------------------------------
> +Target Instructions Latest v5.1.0
> +---------- -------------------- ---------- ----------
> +aarch64 2 487 814 704 ----- +1.94%
> +alpha 1 680 723 605 ----- +2.835%
> +arm 11 563 208 260 ----- +2.848%
> +hppa 7 272 826 858 ----- +3.263%
> +m68k 1 998 819 159 ----- +3.198%
> +mips 1 656 596 909 ----- +1.959%
> +mipsel 1 659 455 464 ----- +1.947%
> +mips64 1 955 541 001 ----- +2.447%
> +mips64el 1 943 598 207 ----- +2.462%
> +ppc 2 320 350 477 ----- +2.332%
> +ppc64 2 552 655 634 ----- +2.742%
> +ppc64le 2 547 364 971 ----- +2.748%
> +riscv64 1 520 862 601 ----- +3.159%
> +s390x 5 497 978 370 ----- +1.078%
> +sh4 2 323 236 696 ----- +3.41%
> +sparc64 3 427 101 999 ----- +4.73%
> +x86_64 1 729 364 402 ----- +2.806%
> +--------------------------------------------------------
> +```
> diff --git a/tests/performance/nightly-tests/benchmarks/source/
> dijkstra_double/dijkstra_double.c b/tests/performance/nightly-
> tests/benchmarks/source/dijkstra_double/dijkstra_double.c
> new file mode 100644
> index 0000000000..9c0bb804ac
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/
> dijkstra_double/dijkstra_double.c
> @@ -0,0 +1,194 @@
> +/*
> + * Source file of a benchmark program involving calculations of the
> + * shortest distances between a source node and all other nodes in a
> + * graph of n nodes in which all nxn distances are defined as "double".
> + * The number n can be given via command line, and the default is 2000.
> + * The algorithm used is Dijsktra's.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <float.h>
> +#include <limits.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of columns and rows in all matrixes*/
> +#define DEFAULT_NODE_COUNT 2000
> +#define MIN_NODE_COUNT 3
> +#define MAX_NODE_COUNT 10000
> +
> +
> +int32_t closest_index(int32_t count, double *distances, bool *flags)
> +{
> + int32_t closest;
> + double minimum = DBL_MAX;
> +
> + for (size_t i = 0; i < count; i++) {
> + if (flags[i] == false && distances[i] <= minimum) {
> + closest = i;
> + minimum = distances[i];
> + }
> + }
> +
> + return closest;
> +}
> +
> +/**
> + * Calculate the shortest distances from the source node using Dijkstra
> method.
> + * @param (out) distances An array of shortest distances from the source
> node.
> + * @param (out) via An array of nodes needed to be taken as the the last
> + * before destination, for each destination.
> + * @param (out) eccent Eccentricity of the source node.
> + * @param (in) count The number of nodes.
> + * @param (in) source Source node.
> + * @param (in) matrix Distance matrix.
> + */
> +void find_shortest_distances(double *distances, int32_t *via, double
> *eccent,
> + int32_t count, int32_t source, double
> **matrix)
> +{
> + bool *flags;
> +
> + flags = (bool *)malloc(count * sizeof(bool));
> +
> + for (size_t i = 0; i < count; i++) {
> + distances[i] = DBL_MAX;
> + flags[i] = false;
> + }
> +
> + distances[source] = 0.0;
> + via[source] = source;
> +
> + for (size_t i = 0; i < count - 1; i++) {
> + int32_t closest = closest_index(count, distances, flags);
> + flags[closest] = true;
> + for (size_t j = 0; j < count; j++) {
> + if ((!flags[j]) &&
> + (matrix[closest][j]) &&
> + (distances[closest] != DBL_MAX) &&
> + (distances[j] > distances[closest] +
> matrix[closest][j])) {
> + distances[j] = distances[closest] + matrix[closest][j];
> + via[j] = closest;
> + }
> + }
> + }
> +
> + *eccent = 0;
> + for (size_t i = 0; i < count; i++) {
> + if (*eccent < distances[i]) {
> + *eccent = distances[i];
> + }
> + }
> +
> + free(flags);
> +}
> +
> +
> +void main(int argc, char *argv[])
> +{
> + double **distance_matrix;
> + double *shortest_distances;
> + int32_t *via_node;
> + int32_t node_count = DEFAULT_NODE_COUNT;
> + int32_t source_node = 0;
> + double node_eccentricity = 0.0;
> + double range_factor = 999.0 / (double)(RAND_MAX);
> + int32_t option;
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_node_count = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_node_count == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_node_count < MIN_NODE_COUNT) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be a "
> + "number less than %d.\n", MIN_NODE_COUNT);
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is too large */
> + if (user_node_count > MAX_NODE_COUNT) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be "
> + "more than %d.\n", MAX_NODE_COUNT);
> + exit(EXIT_FAILURE);
> + }
> + node_count = user_node_count;
> + } else {
> + exit(EXIT_FAILURE);
> + }
> + }
> +
> + /* Allocate the memory space for all matrixes */
> + distance_matrix = (double **)malloc(node_count * sizeof(double *));
> + for (size_t i = 0; i < node_count; i++) {
> + distance_matrix[i] = (double *)malloc(node_count *
> sizeof(double));
> + }
> + shortest_distances = (double *)malloc(node_count * sizeof(double));
> + via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
> +
> + /* Initialize helper arrays and populate distance_matrix */
> + srand(1);
> + for (size_t i = 0; i < node_count; i++) {
> + shortest_distances[i] = 0.0;
> + via_node[i] = -1;
> + distance_matrix[i][i] = 0.0;
> + }
> + for (size_t i = 0; i < node_count; i++) {
> + for (size_t j = i + 1; j < node_count; j++) {
> + distance_matrix[i][j] = 1.0 + range_factor * (double)rand();
> + distance_matrix[j][i] = distance_matrix[i][j];
> + }
> + }
> +
> + find_shortest_distances(shortest_distances, via_node,
> &node_eccentricity,
> + node_count, source_node, distance_matrix);
> +
> + /* Control printing */
> + printf("CONTROL RESULT:\n");
> + printf(" Distance matrix (top left part):\n");
> + for (size_t i = 0; i < 3; i++) {
> + for (size_t j = 0; j < 3; j++) {
> + printf(" %7.2f", distance_matrix[i][j]);
> + }
> + printf("\n");
> + }
> + printf(" Source: %d (eccentricity: %f)\n",
> + source_node, node_eccentricity);
> + printf(" Destination Distance Via Node\n");
> + for (size_t i = 0; i < 3; i++) {
> + printf(" %5d %7.2f %4d\n",
> + i, shortest_distances[i], via_node[i]);
> + }
> +
> + /* Free all previously allocated space */
> + for (size_t i = 0; i < node_count; i++) {
> + free(distance_matrix[i]);
> + }
> + free(distance_matrix);
> + free(shortest_distances);
> + free(via_node);
> +}
> diff --git a/tests/performance/nightly-tests/benchmarks/source/
> dijkstra_int32/dijkstra_int32.c b/tests/performance/nightly-
> tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
> new file mode 100644
> index 0000000000..2663cde943
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/
> dijkstra_int32/dijkstra_int32.c
> @@ -0,0 +1,192 @@
> +/*
> + * Source file of a benchmark program involving calculations of the
> + * shortest distances between a source node and all other nodes in a
> + * graph of n nodes in which all nxn distances are defined as "int32".
> + * The number n can be given via command line, and the default is 2000.
> + * The algorithm used is Dijsktra's.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <limits.h>
> +#include <stdbool.h>
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of columns and rows in all matrixes*/
> +#define DEFAULT_NODE_COUNT 2000
> +#define MIN_NODE_COUNT 3
> +#define MAX_NODE_COUNT 10000
> +
> +
> +int32_t closest_index(int32_t count, int32_t *distances, bool *flags)
> +{
> + int32_t closest;
> + int32_t minimum = INT_MAX;
> +
> + for (size_t i = 0; i < count; i++) {
> + if (flags[i] == false && distances[i] <= minimum) {
> + closest = i;
> + minimum = distances[i];
> + }
> + }
> +
> + return closest;
> +}
> +
> +/**
> + * Calculate the shortest distances from the source node using Dijkstra
> method.
> + * @param (out) distances An array of shortest distances from the source
> node.
> + * @param (out) via An array of nodes needed to be taken as the the last
> + * before destination, for each destination.
> + * @param (out) eccent Eccentricity of the source node.
> + * @param (in) count The number of nodes.
> + * @param (in) source Source node.
> + * @param (in) matrix Distance matrix.
> + */
> +void find_shortest_distances(int32_t *distances, int32_t *via, int32_t
> *eccent,
> + int32_t count, int32_t source, int32_t
> **matrix)
> +{
> + bool *flags;
> +
> + flags = (bool *)malloc(count * sizeof(bool));
> +
> + for (size_t i = 0; i < count; i++) {
> + distances[i] = INT_MAX;
> + flags[i] = false;
> + }
> +
> + distances[source] = 0;
> + via[source] = source;
> +
> + for (size_t i = 0; i < count - 1; i++) {
> + int32_t closest = closest_index(count, distances, flags);
> + flags[closest] = true;
> + for (size_t j = 0; j < count; j++) {
> + if ((!flags[j]) &&
> + (matrix[closest][j]) &&
> + (distances[closest] != INT_MAX) &&
> + (distances[j] > distances[closest] +
> matrix[closest][j])) {
> + distances[j] = distances[closest] + matrix[closest][j];
> + via[j] = closest;
> + }
> + }
> + }
> +
> + *eccent = 0;
> + for (size_t i = 0; i < count; i++) {
> + if (*eccent < distances[i]) {
> + *eccent = distances[i];
> + }
> + }
> +
> + free(flags);
> +}
> +
> +
> +void main(int argc, char *argv[])
> +{
> + int32_t **distance_matrix;
> + int32_t *shortest_distances;
> + int32_t *via_node;
> + int32_t node_count = DEFAULT_NODE_COUNT;
> + int32_t source_node = 0;
> + int32_t node_eccentricity = 0;
> + int32_t option;
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_node_count = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_node_count == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_node_count < MIN_NODE_COUNT) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be a "
> + "number less than %d.\n", MIN_NODE_COUNT);
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is too large */
> + if (user_node_count > MAX_NODE_COUNT) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be "
> + "more than %d.\n", MAX_NODE_COUNT);
> + exit(EXIT_FAILURE);
> + }
> + node_count = user_node_count;
> + } else {
> + exit(EXIT_FAILURE);
> + }
> + }
> +
> + /* Allocate the memory space for all matrixes */
> + distance_matrix = (int32_t **)malloc(node_count * sizeof(int32_t *));
> + for (size_t i = 0; i < node_count; i++) {
> + distance_matrix[i] = (int32_t *)malloc(node_count *
> sizeof(int32_t));
> + }
> + shortest_distances = (int32_t *)malloc(node_count * sizeof(int32_t));
> + via_node = (int32_t *)malloc(node_count * sizeof(int32_t));
> +
> + /* Initialize helper arrays and populate distance_matrix */
> + srand(1);
> + for (size_t i = 0; i < node_count; i++) {
> + shortest_distances[i] = 0;
> + via_node[i] = -1;
> + distance_matrix[i][i] = 0;
> + }
> + for (size_t i = 0; i < node_count; i++) {
> + for (size_t j = i + 1; j < node_count; j++) {
> + distance_matrix[i][j] = 1 + (rand()) / (RAND_MAX / 999);
> + distance_matrix[j][i] = distance_matrix[i][j];
> + }
> + }
> +
> + find_shortest_distances(shortest_distances, via_node,
> &node_eccentricity,
> + node_count, source_node, distance_matrix);
> +
> + /* Control printing */
> + printf("CONTROL RESULT:\n");
> + printf(" Distance matrix (top left part):\n");
> + for (size_t i = 0; i < 3; i++) {
> + for (size_t j = 0; j < 3; j++) {
> + printf(" %6d", distance_matrix[i][j]);
> + }
> + printf("\n");
> + }
> + printf(" Source: %d (eccentricity: %d)\n",
> + source_node, node_eccentricity);
> + printf(" Destination Distance Via Node\n");
> + for (size_t i = 0; i < 3; i++) {
> + printf(" %5d %3d %4d\n",
> + i, shortest_distances[i], via_node[i]);
> + }
> +
> + /* Free all previously allocated space */
> + for (size_t i = 0; i < node_count; i++) {
> + free(distance_matrix[i]);
> + }
> + free(distance_matrix);
> + free(shortest_distances);
> + free(via_node);
> +}
> diff --git a/tests/performance/nightly-tests/benchmarks/source/
> matmult_double/matmult_double.c b/tests/performance/nightly-
> tests/benchmarks/source/matmult_double/matmult_double.c
> new file mode 100644
> index 0000000000..42bbb4717a
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/
> matmult_double/matmult_double.c
> @@ -0,0 +1,123 @@
> +/*
> + * Source file of a benchmark program involving calculations of
> + * a product of two matrixes nxn whose elements are "double". The
> + * number n can be given via command line, and the default is 200.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of columns and rows in all matrixes*/
> +#define DEFAULT_MATRIX_SIZE 200
> +#define MIN_MATRIX_SIZE 2
> +#define MAX_MATRIX_SIZE 200000
> +
> +void main(int argc, char *argv[])
> +{
> + double **matrix_a;
> + double **matrix_b;
> + double **matrix_res;
> + size_t i;
> + size_t j;
> + size_t k;
> + int32_t matrix_size = DEFAULT_MATRIX_SIZE;
> + int32_t option;
> + double range_factor = 100.0 / (double)(RAND_MAX);
> +
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_matrix_size = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_matrix_size == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_matrix_size < MIN_MATRIX_SIZE) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be a "
> + "number less than %d.\n",
> MIN_MATRIX_SIZE);
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is too large */
> + if (user_matrix_size > MAX_MATRIX_SIZE) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be "
> + "more than %d.\n", MAX_MATRIX_SIZE);
> + exit(EXIT_FAILURE);
> + }
> + matrix_size = user_matrix_size;
> + } else {
> + exit(EXIT_FAILURE);
> + }
> + }
> +
> + /* Allocate the memory space for all matrixes */
> + matrix_a = (double **)malloc(matrix_size * sizeof(double *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_a[i] = (double *)malloc(matrix_size * sizeof(double));
> + }
> + matrix_b = (double **)malloc(matrix_size * sizeof(double *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_b[i] = (double *)malloc(matrix_size * sizeof(double));
> + }
> + matrix_res = (double **)malloc(matrix_size * sizeof(double *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_res[i] = (double *)malloc(matrix_size * sizeof(double));
> + }
> +
> + /* Populate matrix_a and matrix_b with random numbers */
> + srand(1);
> + for (i = 0; i < matrix_size; i++) {
> + for (j = 0; j < matrix_size; j++) {
> + matrix_a[i][j] = range_factor * (double)rand();
> + matrix_b[i][j] = range_factor * (double)rand();
> + }
> + }
> +
> + /* Calculate the product of two matrixes */
> + for (i = 0; i < matrix_size; i++) {
> + for (j = 0; j < matrix_size; j++) {
> + matrix_res[i][j] = 0.0;
> + for (k = 0; k < matrix_size; k++) {
> + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
> + }
> + }
> + }
> +
> + /* Control printing */
> + printf("CONTROL RESULT:\n");
> + printf(" %f %f\n", matrix_res[0][0], matrix_res[0][1]);
> + printf(" %f %f\n", matrix_res[1][0], matrix_res[1][1]);
> +
> + /* Free all previously allocated space */
> + for (i = 0; i < matrix_size; i++) {
> + free(matrix_a[i]);
> + free(matrix_b[i]);
> + free(matrix_res[i]);
> + }
> + free(matrix_a);
> + free(matrix_b);
> + free(matrix_res);
> +}
> diff --git a/tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
> b/tests/performance/nightly-tests/benchmarks/source/
> matmult_int32/matmult_int32.c
> new file mode 100644
> index 0000000000..29a6eb000d
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/
> matmult_int32/matmult_int32.c
> @@ -0,0 +1,121 @@
> +/*
> + * Source file of a benchmark program involving calculations of
> + * a product of two matrixes nxn whose elements are "int32_t". The
> + * number n can be given via command line, and the default is 200.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of columns and rows in all matrixes*/
> +#define DEFAULT_MATRIX_SIZE 200
> +#define MIN_MATRIX_SIZE 2
> +#define MAX_MATRIX_SIZE 200000
> +
> +void main(int argc, char *argv[])
> +{
> + int32_t **matrix_a;
> + int32_t **matrix_b;
> + int32_t **matrix_res;
> + size_t i;
> + size_t j;
> + size_t k;
> + int32_t matrix_size = DEFAULT_MATRIX_SIZE;
> + int32_t option;
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_matrix_size = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_matrix_size == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_matrix_size < MIN_MATRIX_SIZE) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be a "
> + "number less than %d.\n",
> MIN_MATRIX_SIZE);
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is too large */
> + if (user_matrix_size > MAX_MATRIX_SIZE) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be "
> + "more than %d.\n", MAX_MATRIX_SIZE);
> + exit(EXIT_FAILURE);
> + }
> + matrix_size = user_matrix_size;
> + } else {
> + exit(EXIT_FAILURE);
> + }
> + }
> +
> + /* Allocate the memory space for all matrixes */
> + matrix_a = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_a[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
> + }
> + matrix_b = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_b[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
> + }
> + matrix_res = (int32_t **)malloc(matrix_size * sizeof(int32_t *));
> + for (i = 0; i < matrix_size; i++) {
> + matrix_res[i] = (int32_t *)malloc(matrix_size * sizeof(int32_t));
> + }
> +
> + /* Populate matrix_a and matrix_b with random numbers */
> + srand(1);
> + for (i = 0; i < matrix_size; i++) {
> + for (j = 0; j < matrix_size; j++) {
> + matrix_a[i][j] = (rand()) / (RAND_MAX / 100);
> + matrix_b[i][j] = (rand()) / (RAND_MAX / 100);
> + }
> + }
> +
> + /* Calculate the product of two matrixes */
> + for (i = 0; i < matrix_size; i++) {
> + for (j = 0; j < matrix_size; j++) {
> + matrix_res[i][j] = 0;
> + for (k = 0; k < matrix_size; k++) {
> + matrix_res[i][j] += matrix_a[i][k] * matrix_b[k][j];
> + }
> + }
> + }
> +
> + /* Control printing */
> + printf("CONTROL RESULT:\n");
> + printf(" %d %d\n", matrix_res[0][0], matrix_res[0][1]);
> + printf(" %d %d\n", matrix_res[1][0], matrix_res[1][1]);
> +
> + /* Free all previously allocated space */
> + for (i = 0; i < matrix_size; i++) {
> + free(matrix_a[i]);
> + free(matrix_b[i]);
> + free(matrix_res[i]);
> + }
> + free(matrix_a);
> + free(matrix_b);
> + free(matrix_res);
> +}
> diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
> b/tests/performance/nightly-tests/benchmarks/source/qsort_
> double/qsort_double.c
> new file mode 100644
> index 0000000000..efc1b2eee1
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_
> double/qsort_double.c
> @@ -0,0 +1,104 @@
> +/*
> + * Source file of a benchmark program involving sorting of an array
> + * of length n whose elements are "double". The default value for n
> + * is 300000, and it can be set via command line as well.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of elements in the array to be sorted */
> +#define DEFAULT_ARRAY_LEN 300000
> +#define MIN_ARRAY_LEN 3
> +#define MAX_ARRAY_LEN 30000000
> +
> +/* Upper limit used for generation of random numbers */
> +#define UPPER_LIMIT 1000.0
> +
> +/* Comparison function passed to qsort() */
> +static int compare(const void *a, const void *b)
> +{
> + if (*(const double *)a > *(const double *)b) {
> + return 1;
> + } else if (*(const double *)a < *(const double *)b) {
> + return -1;
> + }
> + return 0;
> +}
> +
> +void main(int argc, char *argv[])
> +{
> + double *array_to_be_sorted;
> + int32_t array_len = DEFAULT_ARRAY_LEN;
> + int32_t option;
> + double range_factor = UPPER_LIMIT / (double)(RAND_MAX);
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_array_len = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_array_len == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_array_len < MIN_ARRAY_LEN) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be a "
> + "number less than %d.\n", MIN_ARRAY_LEN);
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is too large */
> + if (user_array_len > MAX_ARRAY_LEN) {
> + fprintf(stderr, "Error ... Value for option '-n' cannot
> be "
> + "more than %d.\n", MAX_ARRAY_LEN);
> + exit(EXIT_FAILURE);
> + }
> + array_len = user_array_len;
> + } else {
> + exit(EXIT_FAILURE);
> + }
> + }
> +
> + /* Allocate the memory space for the array */
> + array_to_be_sorted = (double *) malloc(array_len * sizeof(double));
> +
> + /* Populate the_array with random numbers */
> + srand(1);
> + for (size_t i = 0; i < array_len; i++) {
> + array_to_be_sorted[i] = range_factor * (double)rand();
> + }
> +
> + /* Sort the_array using qsort() */
> + qsort(array_to_be_sorted, array_len, sizeof(array_to_be_sorted[0]),
> + compare);
> +
> + /* Control printing */
> + printf("CONTROL RESULT:\n");
> + printf("%14.10f %14.10f %14.10f\n",
> + array_to_be_sorted[0], array_to_be_sorted[1],
> array_to_be_sorted[2]);
> +
> + /* Free all previously allocated space */
> + free(array_to_be_sorted);
> +}
> diff --git a/tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
> b/tests/performance/nightly-tests/benchmarks/source/qsort_
> int32/qsort_int32.c
> new file mode 100644
> index 0000000000..76ca9c3490
> --- /dev/null
> +++ b/tests/performance/nightly-tests/benchmarks/source/qsort_
> int32/qsort_int32.c
> @@ -0,0 +1,103 @@
> +/*
> + * Source file of a benchmark program involving sorting of an array
> + * of length n whose elements are "int32_t". The default value for n
> + * is 300000, and it can be set via command line as well.
> + *
> + * This file is a part of the project "TCG Continuous Benchmarking".
> + *
> + * Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> + * Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> + *
> + * This program is free software: you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation, either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <https://www.gnu.org/licenses/>.
> + *
> + */
> +
> +#include <stdio.h>
> +#include <stdlib.h>
> +#include <unistd.h>
> +
> +/* Number of elements in the array to be sorted */
> +#define DEFAULT_ARRAY_LEN 300000
> +#define MIN_ARRAY_LEN 3
> +#define MAX_ARRAY_LEN 30000000
> +
> +/* Upper limit used for generation of random numbers */
> +#define UPPER_LIMIT 50000000
> +
> +/* Comparison function passed to qsort() */
> +static int compare(const void *a, const void *b)
> +{
> + if (*(const int32_t *)a > *(const int32_t *)b) {
> + return 1;
> + } else if (*(const int32_t *)a < *(const int32_t *)b) {
> + return -1;
> + }
> + return 0;
> +}
> +
> +void main(int argc, char *argv[])
> +{
> + int32_t *array_to_be_sorted;
> + int32_t array_len = DEFAULT_ARRAY_LEN;
> + int32_t option;
> +
> + /* Parse command line options */
> + while ((option = getopt(argc, argv, "n:")) != -1) {
> + if (option == 'n') {
> + int32_t user_array_len = atoi(optarg);
> +
> + /* Check if the value is a string or zero */
> + if (user_array_len == 0) {
> + fprintf(stderr, "Error ... Invalid value for option
> '-n'.\n");
> + exit(EXIT_FAILURE);
> + }
> + /* Check if the value is a negative number */
> + if (user_array_len < MIN_ARRAY_LEN) {
> +
[-- Attachment #2: Type: text/html, Size: 62990 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection
2020-08-28 10:41 ` [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection Ahmed Karaman
@ 2020-09-02 8:37 ` Aleksandar Markovic
0 siblings, 0 replies; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-02 8:37 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 685 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> Add a new 'tests/performance' directory under the 'Performance Tools
> and Tests' subsection.
>
>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> MAINTAINERS | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5a22c8be42..8923307642 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3124,3 +3124,4 @@ Performance Tools and Tests
> M: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> S: Maintained
> F: scripts/performance/
> +F: tests/performance/
> --
> 2.17.1
>
>
[-- Attachment #2: Type: text/html, Size: 1538 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 3/9] scripts/performance: Refactor dissect.py
2020-08-28 10:40 ` [PATCH 3/9] scripts/performance: Refactor dissect.py Ahmed Karaman
@ 2020-09-02 8:48 ` Aleksandar Markovic
0 siblings, 0 replies; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-02 8:48 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 9658 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> - Apply pylint and flake8 formatting rules to the script.
> - Move syntax and usage exmaple to main() docstring.
> - Update get_jit_line() to only detect the main jit call.
> - Use mypy for specifying parameters and return types in functions.
>
>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> scripts/performance/dissect.py | 123 ++++++++++++++++++---------------
> 1 file changed, 68 insertions(+), 55 deletions(-)
>
> diff --git a/scripts/performance/dissect.py b/scripts/performance/dissect.
> py
> index bf24f50922..d4df884b75 100755
> --- a/scripts/performance/dissect.py
> +++ b/scripts/performance/dissect.py
> @@ -1,34 +1,27 @@
> #!/usr/bin/env python3
>
> -# Print the percentage of instructions spent in each phase of QEMU
> -# execution.
> -#
> -# Syntax:
> -# dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
> -# <target executable> [<target executable options>]
> -#
> -# [-h] - Print the script arguments help message.
> -#
> -# Example of usage:
> -# dissect.py -- qemu-arm coulomb_double-arm
> -#
> -# This file is a part of the project "TCG Continuous Benchmarking".
> -#
> -# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> -#
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 2 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
> +Print the percentage of instructions spent in each phase of QEMU
> +execution.
> +
> +This file is a part of the project "TCG Continuous Benchmarking".
> +
> +Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> +Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> +
> +This program is free software: you can redistribute it and/or modify
> +it under the terms of the GNU General Public License as published by
> +the Free Software Foundation, either version 2 of the License, or
> +(at your option) any later version.
> +
> +This program is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +GNU General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
>
> import argparse
> import os
> @@ -36,23 +29,26 @@ import subprocess
> import sys
> import tempfile
>
> +from typing import List
> +
>
> -def get_JIT_line(callgrind_data):
> +def get_jit_line(callgrind_data: List[str]) -> int:
> """
> Search for the first instance of the JIT call in
> the callgrind_annotate output when ran using --tree=caller
> This is equivalent to the self number of instructions of JIT.
>
> Parameters:
> - callgrind_data (list): callgrind_annotate output
> + callgrind_data (List[str]): callgrind_annotate output
>
> Returns:
> (int): Line number
> """
> line = -1
> - for i in range(len(callgrind_data)):
> - if callgrind_data[i].strip('\n') and \
> - callgrind_data[i].split()[-1] == "[???]":
> + for (i, callgrind_datum) in enumerate(callgrind_data):
> + if callgrind_datum.strip('\n') and \
> + callgrind_datum.split()[-1] == "[???]" and \
> + callgrind_datum.split()[1] == "*":
> line = i
> break
> if line == -1:
> @@ -61,6 +57,18 @@ def get_JIT_line(callgrind_data):
>
>
> def main():
> + """
> + Parse the command line arguments then start the execution.
> + Syntax:
> + dissect.py [-h] -- <qemu executable> [<qemu executable options>] \
> + <target executable> [<target executable options>]
> +
> + [-h] - Print the script arguments help message.
> +
> + Example of usage:
> + dissect.py -- qemu-arm coulomb_double-arm
> + """
> +
> # Parse the command line arguments
> parser = argparse.ArgumentParser(
> usage='dissect.py [-h] -- '
> @@ -76,7 +84,7 @@ def main():
>
> # Insure that valgrind is installed
> check_valgrind = subprocess.run(
> - ["which", "valgrind"], stdout=subprocess.DEVNULL)
> + ["which", "valgrind"], stdout=subprocess.DEVNULL, check=False)
> if check_valgrind.returncode:
> sys.exit("Please install valgrind before running the script.")
>
> @@ -93,7 +101,8 @@ def main():
> "--callgrind-out-file=" + data_path]
> + command),
> stdout=subprocess.DEVNULL,
> - stderr=subprocess.PIPE)
> + stderr=subprocess.PIPE,
> + check=False)
> if callgrind.returncode:
> sys.exit(callgrind.stderr.decode("utf-8"))
>
> @@ -102,7 +111,8 @@ def main():
> callgrind_annotate = subprocess.run(
> ["callgrind_annotate", data_path, "--tree=caller"],
> stdout=output,
> - stderr=subprocess.PIPE)
> + stderr=subprocess.PIPE,
> + check=False)
> if callgrind_annotate.returncode:
> sys.exit(callgrind_annotate.stderr.decode("utf-8"))
>
> @@ -120,25 +130,28 @@ def main():
> total_instructions = int(total_instructions.replace(',', ''))
>
> # Line number with the JIT self number of instructions
> - JIT_self_instructions_line_number = get_JIT_line(callgrind_data)
> + jit_self_instructions_line_number = get_jit_line(callgrind_data)
> # Get the JIT self number of instructions
> - JIT_self_instructions_line_data = \
> - callgrind_data[JIT_self_instructions_line_number]
> - JIT_self_instructions = JIT_self_instructions_line_
> data.split()[0]
> - JIT_self_instructions = int(JIT_self_instructions.replace(',',
> ''))
> + jit_self_instructions_line_data = \
> + callgrind_data[jit_self_instructions_line_number]
> + jit_self_instructions = jit_self_instructions_line_
> data.split()[0]
> + jit_self_instructions = int(jit_self_instructions.replace(',',
> ''))
>
> # Line number with the JIT self + inclusive number of instructions
> - # It's the line above the first JIT call when running with
> --tree=caller
> - JIT_total_instructions_line_number = JIT_self_instructions_line_
> number-1
> + # It's the line above the first JIT call when running with
> + # --tree=caller
> + jit_total_instructions_line_number = \
> + jit_self_instructions_line_number-1
> # Get the JIT self + inclusive number of instructions
> - JIT_total_instructions_line_data = \
> - callgrind_data[JIT_total_instructions_line_number]
> - JIT_total_instructions = JIT_total_instructions_line_
> data.split()[0]
> - JIT_total_instructions = int(JIT_total_instructions.replace(',',
> ''))
> + jit_total_instructions_line_data = \
> + callgrind_data[jit_total_instructions_line_number]
> + jit_total_instructions = jit_total_instructions_line_
> data.split()[0]
> + jit_total_instructions = int(jit_total_instructions.replace(',',
> ''))
>
> # Calculate number of instructions in helpers and code generation
> - helpers_instructions = JIT_total_instructions-JIT_
> self_instructions
> - code_generation_instructions = total_instructions-JIT_total_
> instructions
> + helpers_instructions = jit_total_instructions-jit_
> self_instructions
> + code_generation_instructions = \
> + total_instructions-jit_total_instructions
>
> # Print results (Insert commas in large numbers)
> # Print total number of instructions
> @@ -149,12 +162,12 @@ def main():
> print('{:<20}{:>20}\t{:>6.3f}%'.
> format("Code Generation:",
> format(code_generation_instructions, ","),
> - (code_generation_instructions / total_instructions)
> * 100))
> - # Print JIT instructions and percentage
> + (code_generation_instructions/
> total_instructions)*100))
> + # Print jit instructions and percentage
> print('{:<20}{:>20}\t{:>6.3f}%'.
> - format("JIT Execution:",
> - format(JIT_self_instructions, ","),
> - (JIT_self_instructions / total_instructions) * 100))
> + format("jit Execution:",
> + format(jit_self_instructions, ","),
> + (jit_self_instructions / total_instructions) * 100))
> # Print helpers instructions and percentage
> print('{:<20}{:>20}\t{:>6.3f}%'.
> format("Helpers:",
> --
> 2.17.1
>
>
[-- Attachment #2: Type: text/html, Size: 12762 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 7/9] tests/performance: Add nightly tests
2020-08-28 10:41 ` [PATCH 7/9] tests/performance: Add nightly tests Ahmed Karaman
2020-09-02 8:36 ` Aleksandar Markovic
@ 2020-09-02 13:26 ` Alex Bennée
2020-09-02 17:29 ` Ahmed Karaman
2020-09-15 16:39 ` Aleksandar Markovic
1 sibling, 2 replies; 26+ messages in thread
From: Alex Bennée @ 2020-09-02 13:26 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd,
aleksandar.qemu.devel, crosa, rth
Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
> A nightly performance testing system to monitor any change in QEMU
> performance across seventeen different targets.
>
> The system includes eight different benchmarks to provide a variety
> of testing workloads.
>
> dijkstra_double:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are double values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> dijkstra_int32:
> Find the shortest path between the source node and all other nodes
> using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> distances are int32 values. The value of n can be specified using
> the -n flag. The default value is 2000.
>
> matmult_double:
> Standard matrix multiplication of an n*n matrix of randomly generated
> double numbers from 0 to 100. The value of n is passed as an argument
> with the -n flag. The default value is 200.
>
> matmult_int32:
> Standard matrix multiplication of an n*n matrix of randomly generated
> integer numbers from 0 to 100. The value of n is passed as an
> argument with the -n flag. The default value is 200.
>
> qsort_double:
> Quick sort of an array of n randomly generated double numbers from 0
> to 1000. The value of n is passed as an argument with the -n flag.
> The default value is 300000.
>
> qsort_int32:
> Quick sort of an array of n randomly generated integer numbers from 0
> to 50000000. The value of n is passed as an argument with the -n
> flag.The default value is 300000.
>
> qsort_string:
> Quick sort of an array of 10000 randomly generated strings of size 8
> (including null terminating character). The sort process is repeated
> n number of times. The value of n is passed as an argument with the
> -n flag. The default value is 20.
>
> search_string:
> Search for the occurrence of a small string in a much larger random
> string (“needle in a hay”). The search process is repeated n number
> of times and each time, a different large random string (“hay”) is
> generated. The value of n can be specified using the -n flag. The
> default value is 20.
>
> Syntax:
> nightly_tests_core.py [-h] [-r REF]
> Optional arguments:
> -h, --help Show this help message and exit
> -r REF, --reference REF
> Reference QEMU version - Default is v5.1.0
> Example of usage:
> nightly_tests_core.py -r v5.1.0 2>log.txt
>
> The following report includes detailed setup and execution details
> of the system:
> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> tests/performance/nightly-tests/README.md | 243 +++++
> .../source/dijkstra_double/dijkstra_double.c | 194 ++++
> .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
> .../source/matmult_double/matmult_double.c | 123 +++
> .../source/matmult_int32/matmult_int32.c | 121 +++
> .../source/qsort_double/qsort_double.c | 104 ++
> .../source/qsort_int32/qsort_int32.c | 103 ++
> .../source/qsort_string/qsort_string.c | 122 +++
> .../source/search_string/search_string.c | 110 +++
> .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
> .../scripts/run_nightly_tests.py | 135 +++
> .../nightly-tests/scripts/send_email.py | 56 ++
> 12 files changed, 2423 insertions(+)
> create mode 100644 tests/performance/nightly-tests/README.md
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
> create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
> create mode 100644
> tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
Perhaps we could compress these paths down to:
tests/tcg/benchmarks/foo.c
tests/tcg/benchmarks/bar.c
and then we can also ensure they are built using the existing TCG tests
cross compile framework.
> create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py
> create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py
> create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
>
> diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/nightly-tests/README.md
> new file mode 100644
> index 0000000000..6db3b351b3
> --- /dev/null
> +++ b/tests/performance/nightly-tests/README.md
> @@ -0,0 +1,243 @@
> +### QEMU Nightly Tests
> +
> +**Required settings:**
> +
> +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> +
> +For more details on how the system works, please check the [eighth
> report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/)
> of the "TCG Continuos Benchmarking" series.
As external URLs are potentially unstable I think we want to distil the
details into a rst do in docs/devel/
<snip>
--
Alex Bennée
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 7/9] tests/performance: Add nightly tests
2020-09-02 13:26 ` Alex Bennée
@ 2020-09-02 17:29 ` Ahmed Karaman
2020-09-15 16:39 ` Aleksandar Markovic
1 sibling, 0 replies; 26+ messages in thread
From: Ahmed Karaman @ 2020-09-02 17:29 UTC (permalink / raw)
To: Alex Bennée
Cc: Lukáš Doktor, Eduardo Habkost, John Snow,
QEMU Developers, Philippe Mathieu-Daudé,
Aleksandar Markovic, Cleber Rosa, Richard Henderson
Thanks Mr. Alex,
On Wed, Sep 2, 2020 at 3:26 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>
> > A nightly performance testing system to monitor any change in QEMU
> > performance across seventeen different targets.
> >
> > The system includes eight different benchmarks to provide a variety
> > of testing workloads.
> >
> > dijkstra_double:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are double values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > dijkstra_int32:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are int32 values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > matmult_double:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > double numbers from 0 to 100. The value of n is passed as an argument
> > with the -n flag. The default value is 200.
> >
> > matmult_int32:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > integer numbers from 0 to 100. The value of n is passed as an
> > argument with the -n flag. The default value is 200.
> >
> > qsort_double:
> > Quick sort of an array of n randomly generated double numbers from 0
> > to 1000. The value of n is passed as an argument with the -n flag.
> > The default value is 300000.
> >
> > qsort_int32:
> > Quick sort of an array of n randomly generated integer numbers from 0
> > to 50000000. The value of n is passed as an argument with the -n
> > flag.The default value is 300000.
> >
> > qsort_string:
> > Quick sort of an array of 10000 randomly generated strings of size 8
> > (including null terminating character). The sort process is repeated
> > n number of times. The value of n is passed as an argument with the
> > -n flag. The default value is 20.
> >
> > search_string:
> > Search for the occurrence of a small string in a much larger random
> > string (“needle in a hay”). The search process is repeated n number
> > of times and each time, a different large random string (“hay”) is
> > generated. The value of n can be specified using the -n flag. The
> > default value is 20.
> >
> > Syntax:
> > nightly_tests_core.py [-h] [-r REF]
> > Optional arguments:
> > -h, --help Show this help message and exit
> > -r REF, --reference REF
> > Reference QEMU version - Default is v5.1.0
> > Example of usage:
> > nightly_tests_core.py -r v5.1.0 2>log.txt
> >
> > The following report includes detailed setup and execution details
> > of the system:
> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/
> >
> > Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> > ---
> > tests/performance/nightly-tests/README.md | 243 +++++
> > .../source/dijkstra_double/dijkstra_double.c | 194 ++++
> > .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
> > .../source/matmult_double/matmult_double.c | 123 +++
> > .../source/matmult_int32/matmult_int32.c | 121 +++
> > .../source/qsort_double/qsort_double.c | 104 ++
> > .../source/qsort_int32/qsort_int32.c | 103 ++
> > .../source/qsort_string/qsort_string.c | 122 +++
> > .../source/search_string/search_string.c | 110 +++
> > .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
> > .../scripts/run_nightly_tests.py | 135 +++
> > .../nightly-tests/scripts/send_email.py | 56 ++
> > 12 files changed, 2423 insertions(+)
> > create mode 100644 tests/performance/nightly-tests/README.md
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_double/dijkstra_double.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/dijkstra_int32/dijkstra_int32.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_double/matmult_double.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/matmult_int32/matmult_int32.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_double/qsort_double.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_int32/qsort_int32.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/qsort_string/qsort_string.c
> > create mode 100644
> > tests/performance/nightly-tests/benchmarks/source/search_string/search_string.c
>
> Perhaps we could compress these paths down to:
>
> tests/tcg/benchmarks/foo.c
> tests/tcg/benchmarks/bar.c
>
> and then we can also ensure they are built using the existing TCG tests
> cross compile framework.
>
Can you provide the commands for compiling the benchmarks for the 17 targets?
> > create mode 100755 tests/performance/nightly-tests/scripts/nightly_tests_core.py
> > create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_tests.py
> > create mode 100644 tests/performance/nightly-tests/scripts/send_email.py
> >
> > diff --git a/tests/performance/nightly-tests/README.md b/tests/performance/nightly-tests/README.md
> > new file mode 100644
> > index 0000000000..6db3b351b3
> > --- /dev/null
> > +++ b/tests/performance/nightly-tests/README.md
> > @@ -0,0 +1,243 @@
> > +### QEMU Nightly Tests
> > +
> > +**Required settings:**
> > +
> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> > +
> > +For more details on how the system works, please check the [eighth
> > report](https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/QEMU-Nightly-Performance-Tests/)
> > of the "TCG Continuos Benchmarking" series.
>
> As external URLs are potentially unstable I think we want to distil the
> details into a rst do in docs/devel/
Any advice on converting markdown to rst? Are there any specific rules
that should be followed to write rst documentation for QEMU.
>
> <snip>
>
> --
> Alex Bennée
Best regards,
Ahmed Karaman
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
@ 2020-09-07 20:52 ` Aleksandar Markovic
2020-09-18 20:33 ` Aleksandar Markovic
2020-10-01 20:41 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py John Snow
2 siblings, 0 replies; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-07 20:52 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 10436 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> - Apply pylint and flake8 formatting rules to the script.
> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>
>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
Reply
Reply all
Forward
View Gmail in: *Mobile* | Older version
<https://mail.google.com/mail/mu/mp/966/#> | Desktop
<https://mail.google.com/mail/mu/mp/966/#>
© 2020 Google
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> scripts/performance/topN_perf.py | 174 +++++++++++++++----------------
> 1 file changed, 87 insertions(+), 87 deletions(-)
>
> diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_
> perf.py
> index 07be195fc8..56b100da87 100755
> --- a/scripts/performance/topN_perf.py
> +++ b/scripts/performance/topN_perf.py
> @@ -1,72 +1,77 @@
> #!/usr/bin/env python3
>
> -# Print the top N most executed functions in QEMU using perf.
> -# Syntax:
> -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \
> -# <qemu executable> [<qemu executable options>] \
> -# <target executable> [<target execurable options>]
> -#
> -# [-h] - Print the script arguments help message.
> -# [-n] - Specify the number of top functions to print.
> -# - If this flag is not specified, the tool defaults to 25.
> -#
> -# Example of usage:
> -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> -#
> -# This file is a part of the project "TCG Continuous Benchmarking".
> -#
> -# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> -#
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 2 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
> +Print the top N most executed functions in QEMU using perf.
> +
> +Syntax:
> +topN_perf.py [-h] [-n <number of displayed top functions>] -- \
> + <qemu executable> [<qemu executable options>] \
> + <target executable> [<target execurable options>]
> +
> +[-h] - Print the script arguments help message.
> +[-n] - Specify the number of top functions to print.
> + - If this flag is not specified, the tool defaults to 25.
> +
> +Example of usage:
> +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> +
> +This file is a part of the project "TCG Continuous Benchmarking".
> +
> +Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> +Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> +
> +This program is free software: you can redistribute it and/or modify
> +it under the terms of the GNU General Public License as published by
> +the Free Software Foundation, either version 2 of the License, or
> +(at your option) any later version.
> +
> +This program is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +GNU General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
>
> import argparse
> import os
> import subprocess
> import sys
> +import tempfile
>
>
> # Parse the command line arguments
> -parser = argparse.ArgumentParser(
> - usage='topN_perf.py [-h] [-n] <number of displayed top functions >
> -- '
> +PARSER = argparse.ArgumentParser(
> + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- '
> '<qemu executable> [<qemu executable options>] '
> '<target executable> [<target executable options>]')
>
> -parser.add_argument('-n', dest='top', type=int, default=25,
> +PARSER.add_argument('-n', dest='top', type=int, default=25,
> help='Specify the number of top functions to print.')
>
> -parser.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
> +PARSER.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
>
> -args = parser.parse_args()
> +ARGS = PARSER.parse_args()
>
> # Extract the needed variables from the args
> -command = args.command
> -top = args.top
> +COMMAND = ARGS.command
> +TOP = ARGS.top
>
> # Insure that perf is installed
> -check_perf_presence = subprocess.run(["which", "perf"],
> - stdout=subprocess.DEVNULL)
> -if check_perf_presence.returncode:
> +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
> + stdout=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_PRESENCE.returncode:
> sys.exit("Please install perf before running the script!")
>
> # Insure user has previllage to run perf
> -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
> +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
> stdout=subprocess.DEVNULL,
> - stderr=subprocess.DEVNULL)
> -if check_perf_executability.returncode:
> - sys.exit(
> -"""
> + stderr=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_EXECUTABILITY.returncode:
> + sys.exit("""
> Error:
> You may not have permission to collect stats.
>
> @@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf
> too, e.g.:
> kernel.perf_event_paranoid = -1
>
> * Alternatively, you can run this script under sudo privileges.
> -"""
> -)
> -
> -# Run perf record
> -perf_record = subprocess.run((["perf", "record",
> "--output=/tmp/perf.data"] +
> - command),
> - stdout=subprocess.DEVNULL,
> - stderr=subprocess.PIPE)
> -if perf_record.returncode:
> - os.unlink('/tmp/perf.data')
> - sys.exit(perf_record.stderr.decode("utf-8"))
> -
> -# Save perf report output to /tmp/perf_report.out
> -with open("/tmp/perf_report.out", "w") as output:
> - perf_report = subprocess.run(
> - ["perf", "report", "--input=/tmp/perf.data", "--stdio"],
> - stdout=output,
> - stderr=subprocess.PIPE)
> - if perf_report.returncode:
> - os.unlink('/tmp/perf.data')
> - output.close()
> - os.unlink('/tmp/perf_report.out')
> - sys.exit(perf_report.stderr.decode("utf-8"))
> -
> -# Read the reported data to functions[]
> -functions = []
> -with open("/tmp/perf_report.out", "r") as data:
> - # Only read lines that are not comments (comments start with #)
> - # Only read lines that are not empty
> - functions = [line for line in data.readlines() if line and line[0]
> - != '#' and line[0] != "\n"]
> -
> -# Limit the number of top functions to "top"
> -number_of_top_functions = top if len(functions) > top else len(functions)
> -
> -# Store the data of the top functions in top_functions[]
> -top_functions = functions[:number_of_top_functions]
> +""")
> +
> +# Run perf and save all intermediate files in a temporary directory
> +with tempfile.TemporaryDirectory() as tmpdir:
> + RECORD_PATH = os.path.join(tmpdir, "record.data")
> + REPORT_PATH = os.path.join(tmpdir, "report.txt")
> +
> + PERF_RECORD = subprocess.run((["perf", "record",
> "--output="+RECORD_PATH] +
> + COMMAND),
> + stdout=subprocess.DEVNULL,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_RECORD.returncode:
> + sys.exit(PERF_RECORD.stderr.decode("utf-8"))
> +
> + with open(REPORT_PATH, "w") as output:
> + PERF_REPORT = subprocess.run(
> + ["perf", "report", "--input="+RECORD_PATH, "--stdio"],
> + stdout=output,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_REPORT.returncode:
> + sys.exit(PERF_REPORT.stderr.decode("utf-8"))
> +
> + # Save the reported data to FUNCTIONS[]
> + with open(REPORT_PATH, "r") as data:
> + # Only read lines that are not comments (comments start with #)
> + # Only read lines that are not empty
> + FUNCTIONS = [line for line in data.readlines() if line and
> + line[0] != '#' and line[0] != "\n"]
> +
> +# Limit the number of top functions to "TOP"
> +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
> +
> +# Store the data of the top functions in TOP_FUNCTIONS[]
> +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
>
> # Print table header
> print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
> @@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {}
> {}'.format('No.',
> '-' * 25))
>
> # Print top N functions
> -for (index, function) in enumerate(top_functions, start=1):
> +for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
> function_data = function.split()
> function_percentage = function_data[0]
> function_name = function_data[-1]
> @@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions,
> start=1):
> function_percentage,
> function_name,
> function_invoker))
> -
> -# Remove intermediate files
> -os.unlink('/tmp/perf.data')
> -os.unlink('/tmp/perf_report.out')
> --
> 2.17.1
>
>
[-- Attachment #2: Type: text/html, Size: 14999 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py
2020-08-28 10:40 ` [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Ahmed Karaman
@ 2020-09-07 20:53 ` Aleksandar Markovic
0 siblings, 0 replies; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-07 20:53 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, jsnow, qemu-devel, philmd, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 11158 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> - Apply pylint and flake8 formatting rules to the script.
> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>
>
Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
Reply
Reply all
Forward
View Gmail in: *Mobile* | Older version
<https://mail.google.com/mail/mu/mp/966/#> | Desktop
<https://mail.google.com/mail/mu/mp/966/#>
© 2020 Google
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> scripts/performance/topN_callgrind.py | 169 +++++++++++++-------------
> 1 file changed, 87 insertions(+), 82 deletions(-)
>
> diff --git a/scripts/performance/topN_callgrind.py
> b/scripts/performance/topN_callgrind.py
> index 67c59197af..f8a554f393 100755
> --- a/scripts/performance/topN_callgrind.py
> +++ b/scripts/performance/topN_callgrind.py
> @@ -1,113 +1,122 @@
> #!/usr/bin/env python3
>
> -# Print the top N most executed functions in QEMU using callgrind.
> -# Syntax:
> -# topN_callgrind.py [-h] [-n] <number of displayed top functions> -- \
> -# <qemu executable> [<qemu executable options>] \
> -# <target executable> [<target execurable options>]
> -#
> -# [-h] - Print the script arguments help message.
> -# [-n] - Specify the number of top functions to print.
> -# - If this flag is not specified, the tool defaults to 25.
> -#
> -# Example of usage:
> -# topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
> -#
> -# This file is a part of the project "TCG Continuous Benchmarking".
> -#
> -# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.c
> om>
> -#
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 2 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
> +Print the top N most executed functions in QEMU using callgrind.
> +
> +Syntax:
> +topN_callgrind.py [-h] [-n <number of displayed top functions>] -- \
> + <qemu executable> [<qemu executable options>] \
> + <target executable> [<target execurable options>]
> +
> +[-h] - Print the script arguments help message.
> +[-n] - Specify the number of top functions to print.
> + - If this flag is not specified, the tool defaults to 25.
> +
> +Example of usage:
> +topN_callgrind.py -n 20 -- qemu-arm coulomb_double-arm
> +
> +This file is a part of the project "TCG Continuous Benchmarking".
> +
> +Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> +Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> +
> +This program is free software: you can redistribute it and/or modify
> +it under the terms of the GNU General Public License as published by
> +the Free Software Foundation, either version 2 of the License, or
> +(at your option) any later version.
> +
> +This program is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +GNU General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
>
> import argparse
> import os
> import subprocess
> import sys
> +import tempfile
>
>
> # Parse the command line arguments
> -parser = argparse.ArgumentParser(
> - usage='topN_callgrind.py [-h] [-n] <number of displayed top
> functions> -- '
> +PARSER = argparse.ArgumentParser(
> + usage='topN_callgrind.py [-h] [-n] <number of displayed top
> functions> -- '
> '<qemu executable> [<qemu executable options>] '
> '<target executable> [<target executable options>]')
>
> -parser.add_argument('-n', dest='top', type=int, default=25,
> +PARSER.add_argument('-n', dest='top', type=int, default=25,
> help='Specify the number of top functions to print.')
>
> -parser.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
> +PARSER.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
>
> -args = parser.parse_args()
> +ARGS = PARSER.parse_args()
>
> # Extract the needed variables from the args
> -command = args.command
> -top = args.top
> +COMMAND = ARGS.command
> +TOP = ARGS.top
>
> # Insure that valgrind is installed
> -check_valgrind_presence = subprocess.run(["which", "valgrind"],
> - stdout=subprocess.DEVNULL)
> -if check_valgrind_presence.returncode:
> +CHECK_VALGRIND_PRESENCE = subprocess.run(["which", "valgrind"],
> + stdout=subprocess.DEVNULL,
> + check=False)
> +if CHECK_VALGRIND_PRESENCE.returncode:
> sys.exit("Please install valgrind before running the script!")
>
> -# Run callgrind
> -callgrind = subprocess.run((
> - ["valgrind", "--tool=callgrind", "--callgrind-out-file=/tmp/cal
> lgrind.data"]
> - + command),
> - stdout=subprocess.DEVNULL,
> - stderr=subprocess.PIPE)
> -if callgrind.returncode:
> - sys.exit(callgrind.stderr.decode("utf-8"))
> -
> -# Save callgrind_annotate output to /tmp/callgrind_annotate.out
> -with open("/tmp/callgrind_annotate.out", "w") as output:
> - callgrind_annotate = subprocess.run(["callgrind_annotate",
> - "/tmp/callgrind.data"],
> - stdout=output,
> - stderr=subprocess.PIPE)
> - if callgrind_annotate.returncode:
> - os.unlink('/tmp/callgrind.data')
> - output.close()
> - os.unlink('/tmp/callgrind_annotate.out')
> - sys.exit(callgrind_annotate.stderr.decode("utf-8"))
> -
> -# Read the callgrind_annotate output to callgrind_data[]
> -callgrind_data = []
> -with open('/tmp/callgrind_annotate.out', 'r') as data:
> - callgrind_data = data.readlines()
> +# Run callgrind and save all intermediate files in a temporary directory
> +with tempfile.TemporaryDirectory() as tmpdir:
> + CALLGRIND_DATA_PATH = os.path.join(tmpdir, "callgrind.data")
> + ANNOTATE_OUT_PATH = os.path.join(tmpdir, "callgrind_annotate.out")
> +
> + # Run callgrind
> + CALLGRIND = subprocess.run((["valgrind",
> + "--tool=callgrind",
> + "--callgrind-out-file="+CALLG
> RIND_DATA_PATH]
> + + COMMAND),
> + stdout=subprocess.DEVNULL,
> + stderr=subprocess.PIPE,
> + check=False)
> + if CALLGRIND.returncode:
> + sys.exit(CALLGRIND.stderr.decode("utf-8"))
> +
> + with open(ANNOTATE_OUT_PATH, "w") as output:
> + CALLGRIND_ANNOTATE = subprocess.run(["callgrind_annotate",
> + CALLGRIND_DATA_PATH],
> + stdout=output,
> + stderr=subprocess.PIPE,
> + check=False)
> + if CALLGRIND_ANNOTATE.returncode:
> + sys.exit(CALLGRIND_ANNOTATE.stderr.decode("utf-8"))
> +
> + # Read the callgrind_annotate output to CALLGRIND_DATA[]
> + CALLGRIND_DATA = []
> + with open(ANNOTATE_OUT_PATH, 'r') as data:
> + CALLGRIND_DATA = data.readlines()
>
> # Line number with the total number of instructions
> -total_instructions_line_number = 20
> +TOTAL_INSTRUCTIONS_LINE_NO = 20
>
> # Get the total number of instructions
> -total_instructions_line_data = callgrind_data[total_instructi
> ons_line_number]
> -total_number_of_instructions = total_instructions_line_data.split(' ')[0]
> -total_number_of_instructions = int(
> - total_number_of_instructions.replace(',', ''))
> +TOTAL_INSTRUCTIONS_LINE_DATA = CALLGRIND_DATA[TOTAL_INSTRUCTIONS_LINE_NO]
> +TOTAL_NUMBER_OF_INSTRUCTIONS = TOTAL_INSTRUCTIONS_LINE_DATA.split(' ')[0]
> +TOTAL_NUMBER_OF_INSTRUCTIONS = int(
> + TOTAL_NUMBER_OF_INSTRUCTIONS.replace(',', ''))
>
> # Line number with the top function
> -first_func_line = 25
> +FIRST_FUNC_LINE = 25
>
> # Number of functions recorded by callgrind, last two lines are always
> empty
> -number_of_functions = len(callgrind_data) - first_func_line - 2
> +NUMBER_OF_FUNCTIONS = len(CALLGRIND_DATA) - FIRST_FUNC_LINE - 2
>
> # Limit the number of top functions to "top"
> -number_of_top_functions = (top if number_of_functions >
> - top else number_of_functions)
> +NUMBER_OF_TOP_FUNCTIONS = (TOP if NUMBER_OF_FUNCTIONS >
> + TOP else NUMBER_OF_FUNCTIONS)
>
> # Store the data of the top functions in top_functions[]
> -top_functions = callgrind_data[first_func_line:
> - first_func_line + number_of_top_functions]
> +TOP_FUNCTIONS = CALLGRIND_DATA[FIRST_FUNC_LINE:
> + FIRST_FUNC_LINE + NUMBER_OF_TOP_FUNCTIONS]
>
> # Print table header
> print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
> @@ -121,12 +130,12 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {}
> {}'.format('No.',
> ))
>
> # Print top N functions
> -for (index, function) in enumerate(top_functions, start=1):
> +for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
> function_data = function.split()
> # Calculate function percentage
> function_instructions = float(function_data[0].replace(',', ''))
> function_percentage = (function_instructions /
> - total_number_of_instructions)*100
> + TOTAL_NUMBER_OF_INSTRUCTIONS)*100
> # Get function name and source files path
> function_source_file, function_name = function_data[1].split(':')
> # Print extracted data
> @@ -134,7 +143,3 @@ for (index, function) in enumerate(top_functions,
> start=1):
>
> round(function_percentage, 3),
> function_name,
> function_source_file))
> -
> -# Remove intermediate files
> -os.unlink('/tmp/callgrind.data')
> -os.unlink('/tmp/callgrind_annotate.out')
> --
> 2.17.1
>
>
[-- Attachment #2: Type: text/html, Size: 15908 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 7/9] tests/performance: Add nightly tests
2020-09-02 13:26 ` Alex Bennée
2020-09-02 17:29 ` Ahmed Karaman
@ 2020-09-15 16:39 ` Aleksandar Markovic
2020-09-16 8:31 ` Alex Bennée
1 sibling, 1 reply; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-15 16:39 UTC (permalink / raw)
To: Alex Bennée
Cc: ldoktor, ehabkost, philmd, qemu-devel, Ahmed Karaman, jsnow, crosa, rth
[-- Attachment #1: Type: text/plain, Size: 7279 bytes --]
On Wednesday, September 2, 2020, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>
> > A nightly performance testing system to monitor any change in QEMU
> > performance across seventeen different targets.
> >
> > The system includes eight different benchmarks to provide a variety
> > of testing workloads.
> >
> > dijkstra_double:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are double values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > dijkstra_int32:
> > Find the shortest path between the source node and all other nodes
> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
> > distances are int32 values. The value of n can be specified using
> > the -n flag. The default value is 2000.
> >
> > matmult_double:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > double numbers from 0 to 100. The value of n is passed as an argument
> > with the -n flag. The default value is 200.
> >
> > matmult_int32:
> > Standard matrix multiplication of an n*n matrix of randomly generated
> > integer numbers from 0 to 100. The value of n is passed as an
> > argument with the -n flag. The default value is 200.
> >
> > qsort_double:
> > Quick sort of an array of n randomly generated double numbers from 0
> > to 1000. The value of n is passed as an argument with the -n flag.
> > The default value is 300000.
> >
> > qsort_int32:
> > Quick sort of an array of n randomly generated integer numbers from 0
> > to 50000000. The value of n is passed as an argument with the -n
> > flag.The default value is 300000.
> >
> > qsort_string:
> > Quick sort of an array of 10000 randomly generated strings of size 8
> > (including null terminating character). The sort process is repeated
> > n number of times. The value of n is passed as an argument with the
> > -n flag. The default value is 20.
> >
> > search_string:
> > Search for the occurrence of a small string in a much larger random
> > string (“needle in a hay”). The search process is repeated n number
> > of times and each time, a different large random string (“hay”) is
> > generated. The value of n can be specified using the -n flag. The
> > default value is 20.
> >
> > Syntax:
> > nightly_tests_core.py [-h] [-r REF]
> > Optional arguments:
> > -h, --help Show this help message and exit
> > -r REF, --reference REF
> > Reference QEMU version - Default is v5.1.0
> > Example of usage:
> > nightly_tests_core.py -r v5.1.0 2>log.txt
> >
> > The following report includes detailed setup and execution details
> > of the system:
> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
> QEMU-Nightly-Performance-Tests/
> >
> > Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> > ---
> > tests/performance/nightly-tests/README.md | 243 +++++
> > .../source/dijkstra_double/dijkstra_double.c | 194 ++++
> > .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
> > .../source/matmult_double/matmult_double.c | 123 +++
> > .../source/matmult_int32/matmult_int32.c | 121 +++
> > .../source/qsort_double/qsort_double.c | 104 ++
> > .../source/qsort_int32/qsort_int32.c | 103 ++
> > .../source/qsort_string/qsort_string.c | 122 +++
> > .../source/search_string/search_string.c | 110 +++
> > .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
> > .../scripts/run_nightly_tests.py | 135 +++
> > .../nightly-tests/scripts/send_email.py | 56 ++
> > 12 files changed, 2423 insertions(+)
> > create mode 100644 tests/performance/nightly-tests/README.md
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_double/dijkstra_double.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> dijkstra_int32/dijkstra_int32.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_double/matmult_double.c
> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
> matmult_int32/matmult_int32.c
> > create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_double/qsort_double.c
> > create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_int32/qsort_int32.c
> > create mode 100644 tests/performance/nightly-
> tests/benchmarks/source/qsort_string/qsort_string.c
> > create mode 100644
> > tests/performance/nightly-tests/benchmarks/source/
> search_string/search_string.c
>
> Perhaps we could compress these paths down to:
>
> tests/tcg/benchmarks/foo.c
> tests/tcg/benchmarks/bar.c
>
> and then we can also ensure they are built using the existing TCG tests
> cross compile framework.
>
>
Hi,
May I just bring some alternative views on this topic of
benchmarks/multiple cross-compiling/nightly tests?
Having a known source code of any benchmark (and also license-compatible
with QEMU) is, for sure, a good thing. However, for actual test execution,
source code is not relevant, but the executables built for a diverse set of
targets. Expectation that a test bed should be able to build let's say 17
version for each benchmark is a tall order - and, in fact, unnecessary
self-imposed limitation.
My suggestion is that 17x8 executables needed for nightly tests in its
current form should be prebuilt - and kept permanently in a separate
repository (just not to burden QEMU repository). They are anyway built only
once, there is no need to update them at all.
This will allow nightly test execution on any system that can execute qemu,
without any prerequisite for cross-compiling.
Their source code could and should be integrated into QEMU tree, at the
place of choice, but the test beds should not be subject to any burden of
cross-compiling, simply because there is no need for that.
Yours,
Aleksandar
> > create mode 100755 tests/performance/nightly-
> tests/scripts/nightly_tests_core.py
> > create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
> tests.py
> > create mode 100644 tests/performance/nightly-
> tests/scripts/send_email.py
> >
> > diff --git a/tests/performance/nightly-tests/README.md
> b/tests/performance/nightly-tests/README.md
> > new file mode 100644
> > index 0000000000..6db3b351b3
> > --- /dev/null
> > +++ b/tests/performance/nightly-tests/README.md
> > @@ -0,0 +1,243 @@
> > +### QEMU Nightly Tests
> > +
> > +**Required settings:**
> > +
> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
> > +
> > +For more details on how the system works, please check the [eighth
> > report](https://ahmedkrmn.github.io/TCG-Continuous-
> Benchmarking/QEMU-Nightly-Performance-Tests/)
> > of the "TCG Continuos Benchmarking" series.
>
> As external URLs are potentially unstable I think we want to distil the
> details into a rst do in docs/devel/
>
> <snip>
>
> --
> Alex Bennée
>
[-- Attachment #2: Type: text/html, Size: 9003 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 7/9] tests/performance: Add nightly tests
2020-09-15 16:39 ` Aleksandar Markovic
@ 2020-09-16 8:31 ` Alex Bennée
0 siblings, 0 replies; 26+ messages in thread
From: Alex Bennée @ 2020-09-16 8:31 UTC (permalink / raw)
To: Aleksandar Markovic
Cc: ldoktor, ehabkost, philmd, qemu-devel, Ahmed Karaman, jsnow, crosa, rth
Aleksandar Markovic <aleksandar.qemu.devel@gmail.com> writes:
> On Wednesday, September 2, 2020, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>>
>> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>>
>> > A nightly performance testing system to monitor any change in QEMU
>> > performance across seventeen different targets.
>> >
>> > The system includes eight different benchmarks to provide a variety
>> > of testing workloads.
>> >
>> > dijkstra_double:
>> > Find the shortest path between the source node and all other nodes
>> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
>> > distances are double values. The value of n can be specified using
>> > the -n flag. The default value is 2000.
>> >
>> > dijkstra_int32:
>> > Find the shortest path between the source node and all other nodes
>> > using Dijkstra’s algorithm. The graph contains n nodes where all nxn
>> > distances are int32 values. The value of n can be specified using
>> > the -n flag. The default value is 2000.
>> >
>> > matmult_double:
>> > Standard matrix multiplication of an n*n matrix of randomly generated
>> > double numbers from 0 to 100. The value of n is passed as an argument
>> > with the -n flag. The default value is 200.
>> >
>> > matmult_int32:
>> > Standard matrix multiplication of an n*n matrix of randomly generated
>> > integer numbers from 0 to 100. The value of n is passed as an
>> > argument with the -n flag. The default value is 200.
>> >
>> > qsort_double:
>> > Quick sort of an array of n randomly generated double numbers from 0
>> > to 1000. The value of n is passed as an argument with the -n flag.
>> > The default value is 300000.
>> >
>> > qsort_int32:
>> > Quick sort of an array of n randomly generated integer numbers from 0
>> > to 50000000. The value of n is passed as an argument with the -n
>> > flag.The default value is 300000.
>> >
>> > qsort_string:
>> > Quick sort of an array of 10000 randomly generated strings of size 8
>> > (including null terminating character). The sort process is repeated
>> > n number of times. The value of n is passed as an argument with the
>> > -n flag. The default value is 20.
>> >
>> > search_string:
>> > Search for the occurrence of a small string in a much larger random
>> > string (“needle in a hay”). The search process is repeated n number
>> > of times and each time, a different large random string (“hay”) is
>> > generated. The value of n can be specified using the -n flag. The
>> > default value is 20.
>> >
>> > Syntax:
>> > nightly_tests_core.py [-h] [-r REF]
>> > Optional arguments:
>> > -h, --help Show this help message and exit
>> > -r REF, --reference REF
>> > Reference QEMU version - Default is v5.1.0
>> > Example of usage:
>> > nightly_tests_core.py -r v5.1.0 2>log.txt
>> >
>> > The following report includes detailed setup and execution details
>> > of the system:
>> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/
>> QEMU-Nightly-Performance-Tests/
>> >
>> > Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
>> > ---
>> > tests/performance/nightly-tests/README.md | 243 +++++
>> > .../source/dijkstra_double/dijkstra_double.c | 194 ++++
>> > .../source/dijkstra_int32/dijkstra_int32.c | 192 ++++
>> > .../source/matmult_double/matmult_double.c | 123 +++
>> > .../source/matmult_int32/matmult_int32.c | 121 +++
>> > .../source/qsort_double/qsort_double.c | 104 ++
>> > .../source/qsort_int32/qsort_int32.c | 103 ++
>> > .../source/qsort_string/qsort_string.c | 122 +++
>> > .../source/search_string/search_string.c | 110 +++
>> > .../scripts/nightly_tests_core.py | 920 ++++++++++++++++++
>> > .../scripts/run_nightly_tests.py | 135 +++
>> > .../nightly-tests/scripts/send_email.py | 56 ++
>> > 12 files changed, 2423 insertions(+)
>> > create mode 100644 tests/performance/nightly-tests/README.md
>> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> dijkstra_double/dijkstra_double.c
>> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> dijkstra_int32/dijkstra_int32.c
>> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> matmult_double/matmult_double.c
>> > create mode 100644 tests/performance/nightly-tests/benchmarks/source/
>> matmult_int32/matmult_int32.c
>> > create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_double/qsort_double.c
>> > create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_int32/qsort_int32.c
>> > create mode 100644 tests/performance/nightly-
>> tests/benchmarks/source/qsort_string/qsort_string.c
>> > create mode 100644
>> > tests/performance/nightly-tests/benchmarks/source/
>> search_string/search_string.c
>>
>> Perhaps we could compress these paths down to:
>>
>> tests/tcg/benchmarks/foo.c
>> tests/tcg/benchmarks/bar.c
>>
>> and then we can also ensure they are built using the existing TCG tests
>> cross compile framework.
>>
>>
> Hi,
>
> May I just bring some alternative views on this topic of
> benchmarks/multiple cross-compiling/nightly tests?
>
> Having a known source code of any benchmark (and also license-compatible
> with QEMU) is, for sure, a good thing. However, for actual test execution,
> source code is not relevant, but the executables built for a diverse set of
> targets. Expectation that a test bed should be able to build let's say 17
> version for each benchmark is a tall order - and, in fact, unnecessary
> self-imposed limitation.
>
> My suggestion is that 17x8 executables needed for nightly tests in its
> current form should be prebuilt - and kept permanently in a separate
> repository (just not to burden QEMU repository). They are anyway built only
> once, there is no need to update them at all.
>
> This will allow nightly test execution on any system that can execute qemu,
> without any prerequisite for cross-compiling.
>
> Their source code could and should be integrated into QEMU tree, at the
> place of choice, but the test beds should not be subject to any burden of
> cross-compiling, simply because there is no need for that.
I have no problem with the test binaries being stored somewhere and
accessed when needed - as long as we have a source build-able version in
the tree. It would seem silly not to at least integrate the build for
the 19 different cross compilers we already support for the check-tcg
tests.
I'm not proposing we run them every test either, but at least have the
option to do:
make build-tcg-benchmark
make run-tcg-benchmark
so it's easy for a developer to get to a state when they can tinker and
debug.
>
> Yours,
> Aleksandar
>
>
>
>
>> > create mode 100755 tests/performance/nightly-
>> tests/scripts/nightly_tests_core.py
>> > create mode 100755 tests/performance/nightly-tests/scripts/run_nightly_
>> tests.py
>> > create mode 100644 tests/performance/nightly-
>> tests/scripts/send_email.py
>> >
>> > diff --git a/tests/performance/nightly-tests/README.md
>> b/tests/performance/nightly-tests/README.md
>> > new file mode 100644
>> > index 0000000000..6db3b351b3
>> > --- /dev/null
>> > +++ b/tests/performance/nightly-tests/README.md
>> > @@ -0,0 +1,243 @@
>> > +### QEMU Nightly Tests
>> > +
>> > +**Required settings:**
>> > +
>> > +Update the `GMAIL_USER` object in `send_email.py` with your credentials.
>> > +
>> > +For more details on how the system works, please check the [eighth
>> > report](https://ahmedkrmn.github.io/TCG-Continuous-
>> Benchmarking/QEMU-Nightly-Performance-Tests/)
>> > of the "TCG Continuos Benchmarking" series.
>>
>> As external URLs are potentially unstable I think we want to distil the
>> details into a rst do in docs/devel/
>>
>> <snip>
>>
>> --
>> Alex Bennée
>>
--
Alex Bennée
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
2020-09-07 20:52 ` Aleksandar Markovic
@ 2020-09-18 20:33 ` Aleksandar Markovic
2020-09-19 11:17 ` Bottleneck problem to merge Python patches Philippe Mathieu-Daudé
2020-10-01 20:41 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py John Snow
2 siblings, 1 reply; 26+ messages in thread
From: Aleksandar Markovic @ 2020-09-18 20:33 UTC (permalink / raw)
To: Ahmed Karaman
Cc: ldoktor, ehabkost, philmd, qemu-devel, jsnow,
aleksandar.qemu.devel, crosa, alex.bennee, rth
[-- Attachment #1: Type: text/plain, Size: 10265 bytes --]
On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
wrote:
> - Apply pylint and flake8 formatting rules to the script.
> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
Hello, folks.
This series seems forgotten. Can some of you perhaps take a look, review,
and possibly integrate some patches in a pull request?
Ahmed invested quite of time to improve the functionality and quality of
scripts, and they are truly useful for developers interested in performance
measurement.
Thanks,
Aleksandar
> scripts/performance/topN_perf.py | 174 +++++++++++++++----------------
> 1 file changed, 87 insertions(+), 87 deletions(-)
>
> diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_
> perf.py
> index 07be195fc8..56b100da87 100755
> --- a/scripts/performance/topN_perf.py
> +++ b/scripts/performance/topN_perf.py
> @@ -1,72 +1,77 @@
> #!/usr/bin/env python3
>
> -# Print the top N most executed functions in QEMU using perf.
> -# Syntax:
> -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \
> -# <qemu executable> [<qemu executable options>] \
> -# <target executable> [<target execurable options>]
> -#
> -# [-h] - Print the script arguments help message.
> -# [-n] - Specify the number of top functions to print.
> -# - If this flag is not specified, the tool defaults to 25.
> -#
> -# Example of usage:
> -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> -#
> -# This file is a part of the project "TCG Continuous Benchmarking".
> -#
> -# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.
> com>
> -#
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 2 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
> +Print the top N most executed functions in QEMU using perf.
> +
> +Syntax:
> +topN_perf.py [-h] [-n <number of displayed top functions>] -- \
> + <qemu executable> [<qemu executable options>] \
> + <target executable> [<target execurable options>]
> +
> +[-h] - Print the script arguments help message.
> +[-n] - Specify the number of top functions to print.
> + - If this flag is not specified, the tool defaults to 25.
> +
> +Example of usage:
> +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> +
> +This file is a part of the project "TCG Continuous Benchmarking".
> +
> +Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> +Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> +
> +This program is free software: you can redistribute it and/or modify
> +it under the terms of the GNU General Public License as published by
> +the Free Software Foundation, either version 2 of the License, or
> +(at your option) any later version.
> +
> +This program is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +GNU General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
>
> import argparse
> import os
> import subprocess
> import sys
> +import tempfile
>
>
> # Parse the command line arguments
> -parser = argparse.ArgumentParser(
> - usage='topN_perf.py [-h] [-n] <number of displayed top functions >
> -- '
> +PARSER = argparse.ArgumentParser(
> + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- '
> '<qemu executable> [<qemu executable options>] '
> '<target executable> [<target executable options>]')
>
> -parser.add_argument('-n', dest='top', type=int, default=25,
> +PARSER.add_argument('-n', dest='top', type=int, default=25,
> help='Specify the number of top functions to print.')
>
> -parser.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
> +PARSER.add_argument('command', type=str, nargs='+',
> help=argparse.SUPPRESS)
>
> -args = parser.parse_args()
> +ARGS = PARSER.parse_args()
>
> # Extract the needed variables from the args
> -command = args.command
> -top = args.top
> +COMMAND = ARGS.command
> +TOP = ARGS.top
>
> # Insure that perf is installed
> -check_perf_presence = subprocess.run(["which", "perf"],
> - stdout=subprocess.DEVNULL)
> -if check_perf_presence.returncode:
> +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
> + stdout=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_PRESENCE.returncode:
> sys.exit("Please install perf before running the script!")
>
> # Insure user has previllage to run perf
> -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
> +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
> stdout=subprocess.DEVNULL,
> - stderr=subprocess.DEVNULL)
> -if check_perf_executability.returncode:
> - sys.exit(
> -"""
> + stderr=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_EXECUTABILITY.returncode:
> + sys.exit("""
> Error:
> You may not have permission to collect stats.
>
> @@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf
> too, e.g.:
> kernel.perf_event_paranoid = -1
>
> * Alternatively, you can run this script under sudo privileges.
> -"""
> -)
> -
> -# Run perf record
> -perf_record = subprocess.run((["perf", "record",
> "--output=/tmp/perf.data"] +
> - command),
> - stdout=subprocess.DEVNULL,
> - stderr=subprocess.PIPE)
> -if perf_record.returncode:
> - os.unlink('/tmp/perf.data')
> - sys.exit(perf_record.stderr.decode("utf-8"))
> -
> -# Save perf report output to /tmp/perf_report.out
> -with open("/tmp/perf_report.out", "w") as output:
> - perf_report = subprocess.run(
> - ["perf", "report", "--input=/tmp/perf.data", "--stdio"],
> - stdout=output,
> - stderr=subprocess.PIPE)
> - if perf_report.returncode:
> - os.unlink('/tmp/perf.data')
> - output.close()
> - os.unlink('/tmp/perf_report.out')
> - sys.exit(perf_report.stderr.decode("utf-8"))
> -
> -# Read the reported data to functions[]
> -functions = []
> -with open("/tmp/perf_report.out", "r") as data:
> - # Only read lines that are not comments (comments start with #)
> - # Only read lines that are not empty
> - functions = [line for line in data.readlines() if line and line[0]
> - != '#' and line[0] != "\n"]
> -
> -# Limit the number of top functions to "top"
> -number_of_top_functions = top if len(functions) > top else len(functions)
> -
> -# Store the data of the top functions in top_functions[]
> -top_functions = functions[:number_of_top_functions]
> +""")
> +
> +# Run perf and save all intermediate files in a temporary directory
> +with tempfile.TemporaryDirectory() as tmpdir:
> + RECORD_PATH = os.path.join(tmpdir, "record.data")
> + REPORT_PATH = os.path.join(tmpdir, "report.txt")
> +
> + PERF_RECORD = subprocess.run((["perf", "record",
> "--output="+RECORD_PATH] +
> + COMMAND),
> + stdout=subprocess.DEVNULL,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_RECORD.returncode:
> + sys.exit(PERF_RECORD.stderr.decode("utf-8"))
> +
> + with open(REPORT_PATH, "w") as output:
> + PERF_REPORT = subprocess.run(
> + ["perf", "report", "--input="+RECORD_PATH, "--stdio"],
> + stdout=output,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_REPORT.returncode:
> + sys.exit(PERF_REPORT.stderr.decode("utf-8"))
> +
> + # Save the reported data to FUNCTIONS[]
> + with open(REPORT_PATH, "r") as data:
> + # Only read lines that are not comments (comments start with #)
> + # Only read lines that are not empty
> + FUNCTIONS = [line for line in data.readlines() if line and
> + line[0] != '#' and line[0] != "\n"]
> +
> +# Limit the number of top functions to "TOP"
> +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
> +
> +# Store the data of the top functions in TOP_FUNCTIONS[]
> +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
>
> # Print table header
> print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
> @@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {}
> {}'.format('No.',
> '-' * 25))
>
> # Print top N functions
> -for (index, function) in enumerate(top_functions, start=1):
> +for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
> function_data = function.split()
> function_percentage = function_data[0]
> function_name = function_data[-1]
> @@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions,
> start=1):
> function_percentage,
> function_name,
> function_invoker))
> -
> -# Remove intermediate files
> -os.unlink('/tmp/perf.data')
> -os.unlink('/tmp/perf_report.out')
> --
> 2.17.1
>
>
>
[-- Attachment #2: Type: text/html, Size: 13429 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bottleneck problem to merge Python patches
2020-09-18 20:33 ` Aleksandar Markovic
@ 2020-09-19 11:17 ` Philippe Mathieu-Daudé
2020-09-21 14:49 ` John Snow
` (2 more replies)
0 siblings, 3 replies; 26+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-09-19 11:17 UTC (permalink / raw)
To: Aleksandar Markovic, ehabkost, crosa, Lukas Straub
Cc: ldoktor, Thomas Huth, Daniel P . Berrange, Ben Widawsky,
alex.bennee, qemu-devel, Markus Armbruster, Ahmed Karaman,
aleksandar.qemu.devel, jsnow, rth
Hi Aleksandar,
(extending the Cc list to broader audience)
On 9/18/20 10:33 PM, Aleksandar Markovic wrote:
>
> On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com
> <mailto:ahmedkhaledkaraman@gmail.com>> wrote:
>
> - Apply pylint and flake8 formatting rules to the script.
> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>
> ---
>
> Hello, folks.
>
> This series seems forgotten. Can some of you perhaps take a look,
> review, and possibly integrate some patches in a pull request?
>
> Ahmed invested quite of time to improve the functionality and quality of
> scripts, and they are truly useful for developers interested in
> performance measurement.
The Python patches are indeed being merged slowly, but are not
forgotten :) Eduardo sent just a pull request *yesterday* for
patches he had queued *before* QEMU 5.1, that is more than 1 year
ago!
https://www.mail-archive.com/qemu-devel@nongnu.org/msg742228.html
I hope he will be able to process the other Python patches sent
the last 12 months. He raised the problem few month ago saying he
was overwhelmed and was looking for help from the community.
Cleber helped a pair of times, I helped once, but then nobody
popped up volunteering.
I agree this is a community problem, nobody wants to become the
de-facto Python maintainer. Current maintainers are already busy
solving problem with low-level languages such C.
As a project, QEMU is evolving, using more and more Python, switched
to Meson, we might have Rust code too. Learning that for current
maintainers takes time. I guess we lack new contributors/maintainers
with interest in Python & QEMU.
This is my simple/rough analysis, as John had the same problem
2/3 months ago, his patches were on the list unreviewed for various
weeks. Same problem with Avocado patches, Lukas sent a series a bit
before Ahmed and it is still unreviewed:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg737272.html
Alex Bennée recently said:
"review time is the currency of qemu-devel ;-)"
We might be failing at explaining new contributors the importance
of peer-review, as this helps a lot maintainers. It is described
in the wiki but maybe new contributors don't read it, we don't
remember them to:
https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
and:
https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
My 2 cents...
Regards,
Phil.
>
> Thanks,
> Aleksandar
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bottleneck problem to merge Python patches
2020-09-19 11:17 ` Bottleneck problem to merge Python patches Philippe Mathieu-Daudé
@ 2020-09-21 14:49 ` John Snow
2020-09-21 15:54 ` Eduardo Habkost
2020-09-21 17:57 ` Cleber Rosa
2 siblings, 0 replies; 26+ messages in thread
From: John Snow @ 2020-09-21 14:49 UTC (permalink / raw)
To: Philippe Mathieu-Daudé,
Aleksandar Markovic, ehabkost, crosa, Lukas Straub
Cc: ldoktor, Thomas Huth, Daniel P . Berrange, Ben Widawsky,
alex.bennee, qemu-devel, Markus Armbruster, Ahmed Karaman,
aleksandar.qemu.devel, rth
On 9/19/20 7:17 AM, Philippe Mathieu-Daudé wrote:
> Hi Aleksandar,
>
> (extending the Cc list to broader audience)
>
> On 9/18/20 10:33 PM, Aleksandar Markovic wrote:
>>
>> On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com
>> <mailto:ahmedkhaledkaraman@gmail.com>> wrote:
>>
>> - Apply pylint and flake8 formatting rules to the script.
>> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>>
>> ---
>>
>> Hello, folks.
>>
>> This series seems forgotten. Can some of you perhaps take a look,
>> review, and possibly integrate some patches in a pull request?
>>
>> Ahmed invested quite of time to improve the functionality and quality of
>> scripts, and they are truly useful for developers interested in
>> performance measurement.
>
Hi, I will add it to my queue. I intended to take over Python
maintenance but I have been busy refactoring the QAPI python code and
haven't been reading my mail as regularly.
> The Python patches are indeed being merged slowly, but are not
> forgotten :) Eduardo sent just a pull request *yesterday* for
> patches he had queued *before* QEMU 5.1, that is more than 1 year
> ago!
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg742228.html
>
> I hope he will be able to process the other Python patches sent
> the last 12 months. He raised the problem few month ago saying he
> was overwhelmed and was looking for help from the community.
> Cleber helped a pair of times, I helped once, but then nobody
> popped up volunteering.
>
> I agree this is a community problem, nobody wants to become the
> de-facto Python maintainer. Current maintainers are already busy
> solving problem with low-level languages such C.
> As a project, QEMU is evolving, using more and more Python, switched
> to Meson, we might have Rust code too. Learning that for current
> maintainers takes time. I guess we lack new contributors/maintainers
> with interest in Python & QEMU.
>
I'm volunteering, since I am doing so much work in Python. I could use a
dedicated reviewer to help me, however. I prefer a maintainer policy
where all patches get at least ONE review by someone other than the
primary author.
In the case that I am writing so much Python, I still need a
co-maintainer to help review *my* patches.
> This is my simple/rough analysis, as John had the same problem
> 2/3 months ago, his patches were on the list unreviewed for various
> weeks. Same problem with Avocado patches, Lukas sent a series a bit
> before Ahmed and it is still unreviewed:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg737272.html
>
> Alex Bennée recently said:
>
> "review time is the currency of qemu-devel ;-)"
>
> We might be failing at explaining new contributors the importance
> of peer-review, as this helps a lot maintainers. It is described
> in the wiki but maybe new contributors don't read it, we don't
> remember them to:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
> and:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
>
> My 2 cents...
>
> Regards,
>
> Phil.
>
>>
>> Thanks,
>> Aleksandar
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bottleneck problem to merge Python patches
2020-09-19 11:17 ` Bottleneck problem to merge Python patches Philippe Mathieu-Daudé
2020-09-21 14:49 ` John Snow
@ 2020-09-21 15:54 ` Eduardo Habkost
2020-09-21 17:57 ` Cleber Rosa
2 siblings, 0 replies; 26+ messages in thread
From: Eduardo Habkost @ 2020-09-21 15:54 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: ldoktor, Thomas Huth, Lukas Straub, Daniel P . Berrange,
Ben Widawsky, qemu-devel, Markus Armbruster, Ahmed Karaman,
aleksandar.qemu.devel, jsnow, crosa, alex.bennee,
Aleksandar Markovic, rth
On Sat, Sep 19, 2020 at 01:17:06PM +0200, Philippe Mathieu-Daudé wrote:
> Hi Aleksandar,
>
> (extending the Cc list to broader audience)
>
> On 9/18/20 10:33 PM, Aleksandar Markovic wrote:
> >
> > On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com
> > <mailto:ahmedkhaledkaraman@gmail.com>> wrote:
> >
> > - Apply pylint and flake8 formatting rules to the script.
> > - Use 'tempfile' instead of '/tmp' for creating temporary files.
> >
> > ---
> >
> > Hello, folks.
> >
> > This series seems forgotten. Can some of you perhaps take a look,
> > review, and possibly integrate some patches in a pull request?
> >
> > Ahmed invested quite of time to improve the functionality and quality of
> > scripts, and they are truly useful for developers interested in
> > performance measurement.
>
> The Python patches are indeed being merged slowly, but are not
> forgotten :) Eduardo sent just a pull request *yesterday* for
> patches he had queued *before* QEMU 5.1, that is more than 1 year
> ago!
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg742228.html
>
> I hope he will be able to process the other Python patches sent
> the last 12 months. He raised the problem few month ago saying he
> was overwhelmed and was looking for help from the community.
> Cleber helped a pair of times, I helped once, but then nobody
> popped up volunteering.
The patches were from July this year. But, yeah, I'm not being
able to keep up with the maintainer role for Python code, and
plan to change my M: line to R: in the "Python scripts" section.
>
> I agree this is a community problem, nobody wants to become the
> de-facto Python maintainer. Current maintainers are already busy
> solving problem with low-level languages such C.
> As a project, QEMU is evolving, using more and more Python, switched
> to Meson, we might have Rust code too. Learning that for current
> maintainers takes time. I guess we lack new contributors/maintainers
> with interest in Python & QEMU.
I believe the solution here is to not have a single owner for all
Python code, but owners for specific areas. We already have 36
F: entries in MAINTAINERS for files inside `scripts/`, and each
of those maintainers can send pull requests containing Python
code in the areas they maintain. Including Ahmed, who is already
listed as the maintainer of `scripts/performance/`.
>
> This is my simple/rough analysis, as John had the same problem
> 2/3 months ago, his patches were on the list unreviewed for various
> weeks. Same problem with Avocado patches, Lukas sent a series a bit
> before Ahmed and it is still unreviewed:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg737272.html
>
> Alex Bennée recently said:
>
> "review time is the currency of qemu-devel ;-)"
>
> We might be failing at explaining new contributors the importance
> of peer-review, as this helps a lot maintainers. It is described
> in the wiki but maybe new contributors don't read it, we don't
> remember them to:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
> and:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
>
> My 2 cents...
>
> Regards,
>
> Phil.
>
> >
> > Thanks,
> > Aleksandar
>
--
Eduardo
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: Bottleneck problem to merge Python patches
2020-09-19 11:17 ` Bottleneck problem to merge Python patches Philippe Mathieu-Daudé
2020-09-21 14:49 ` John Snow
2020-09-21 15:54 ` Eduardo Habkost
@ 2020-09-21 17:57 ` Cleber Rosa
2 siblings, 0 replies; 26+ messages in thread
From: Cleber Rosa @ 2020-09-21 17:57 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: ldoktor, Thomas Huth, Lukas Straub, Daniel P . Berrange,
ehabkost, Ben Widawsky, qemu-devel, Markus Armbruster,
Ahmed Karaman, aleksandar.qemu.devel, jsnow, Willian Rampazzo,
alex.bennee, Aleksandar Markovic, rth
[-- Attachment #1: Type: text/plain, Size: 3768 bytes --]
On Sat, Sep 19, 2020 at 01:17:06PM +0200, Philippe Mathieu-Daudé wrote:
> Hi Aleksandar,
>
> (extending the Cc list to broader audience)
>
> On 9/18/20 10:33 PM, Aleksandar Markovic wrote:
> >
> > On Friday, August 28, 2020, Ahmed Karaman <ahmedkhaledkaraman@gmail.com
> > <mailto:ahmedkhaledkaraman@gmail.com>> wrote:
> >
> > - Apply pylint and flake8 formatting rules to the script.
> > - Use 'tempfile' instead of '/tmp' for creating temporary files.
> >
> > ---
> >
> > Hello, folks.
> >
> > This series seems forgotten. Can some of you perhaps take a look,
> > review, and possibly integrate some patches in a pull request?
> >
> > Ahmed invested quite of time to improve the functionality and quality of
> > scripts, and they are truly useful for developers interested in
> > performance measurement.
>
> The Python patches are indeed being merged slowly, but are not
> forgotten :) Eduardo sent just a pull request *yesterday* for
> patches he had queued *before* QEMU 5.1, that is more than 1 year
> ago!
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg742228.html
>
> I hope he will be able to process the other Python patches sent
> the last 12 months. He raised the problem few month ago saying he
> was overwhelmed and was looking for help from the community.
> Cleber helped a pair of times, I helped once, but then nobody
> popped up volunteering.
>
Yes, Python patches are not forgotten... they have been haunting me
every day and night for the last few months. I have a *lot* of blame
to take here, so my sincere apologies.
One word of hope is that my resources (which were exhausted during the
last months towards the Avocado 82.0 LTS release, released last week)
are now freed. And *a lot* of the Avocado features were eyeing QEMU,
so it's just natural that I will (because I want and have to) get
back to being more active and responsive.
> I agree this is a community problem, nobody wants to become the
> de-facto Python maintainer. Current maintainers are already busy
> solving problem with low-level languages such C.
> As a project, QEMU is evolving, using more and more Python, switched
> to Meson, we might have Rust code too. Learning that for current
> maintainers takes time. I guess we lack new contributors/maintainers
> with interest in Python & QEMU.
>
> This is my simple/rough analysis, as John had the same problem
> 2/3 months ago, his patches were on the list unreviewed for various
> weeks. Same problem with Avocado patches, Lukas sent a series a bit
> before Ahmed and it is still unreviewed:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg737272.html
>
ACK. This is a good real example of my previous abstract explanation
above.
So, besides me getting back with more resources to review and maintain
the Python pathes, I have also requested Willian Rampazzo (thanks!) to
keep an eye on those patches and help with reviews.
> Alex Bennée recently said:
>
> "review time is the currency of qemu-devel ;-)"
>
> We might be failing at explaining new contributors the importance
> of peer-review, as this helps a lot maintainers. It is described
> in the wiki but maybe new contributors don't read it, we don't
> remember them to:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Participating_in_Code_Review
> and:
> https://wiki.qemu.org/Contribute/SubmitAPatch#Return_the_favor
>
> My 2 cents...
>
> Regards,
>
> Phil.
>
So while more help is always welcome, I do sincerely hope with myself
and Willian back into the loop, the Python patches won't lag behind
anymore.
And thanks for raising the awareness of the issue Phil!
- Cleber.
> >
> > Thanks,
> > Aleksandar
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
2020-09-07 20:52 ` Aleksandar Markovic
2020-09-18 20:33 ` Aleksandar Markovic
@ 2020-10-01 20:41 ` John Snow
2020-10-01 21:59 ` John Snow
2 siblings, 1 reply; 26+ messages in thread
From: John Snow @ 2020-10-01 20:41 UTC (permalink / raw)
To: Ahmed Karaman, qemu-devel, aleksandar.qemu.devel, philmd,
alex.bennee, eblake, ldoktor, rth, ehabkost, crosa
I realize this review comes well after you are no longer being paid to
work on this, so I am offering my time to help polish your patches if
you would like.
On 8/28/20 6:40 AM, Ahmed Karaman wrote:
> - Apply pylint and flake8 formatting rules to the script.
> - Use 'tempfile' instead of '/tmp' for creating temporary files.
>
I had meant to maybe consider using some helper functions so that you
didn't need to rename the globals, for instance:
> Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> ---
> scripts/performance/topN_perf.py | 174 +++++++++++++++----------------
> 1 file changed, 87 insertions(+), 87 deletions(-)
>
> diff --git a/scripts/performance/topN_perf.py b/scripts/performance/topN_perf.py
> index 07be195fc8..56b100da87 100755
> --- a/scripts/performance/topN_perf.py
> +++ b/scripts/performance/topN_perf.py
> @@ -1,72 +1,77 @@
> #!/usr/bin/env python3
>
> -# Print the top N most executed functions in QEMU using perf.
> -# Syntax:
> -# topN_perf.py [-h] [-n] <number of displayed top functions> -- \
> -# <qemu executable> [<qemu executable options>] \
> -# <target executable> [<target execurable options>]
> -#
> -# [-h] - Print the script arguments help message.
> -# [-n] - Specify the number of top functions to print.
> -# - If this flag is not specified, the tool defaults to 25.
> -#
> -# Example of usage:
> -# topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> -# > -# This file is a part of the project "TCG Continuous Benchmarking".
> -#
> -# Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> -# Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> -#
> -# This program is free software: you can redistribute it and/or modify
> -# it under the terms of the GNU General Public License as published by
> -# the Free Software Foundation, either version 2 of the License, or
> -# (at your option) any later version.
> -#
> -# This program is distributed in the hope that it will be useful,
> -# but WITHOUT ANY WARRANTY; without even the implied warranty of
> -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> -# GNU General Public License for more details.
> -#
> -# You should have received a copy of the GNU General Public License
> -# along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
> +Print the top N most executed functions in QEMU using perf.
> +
> +Syntax:
> +topN_perf.py [-h] [-n <number of displayed top functions>] -- \
> + <qemu executable> [<qemu executable options>] \
> + <target executable> [<target execurable options>]
> +
> +[-h] - Print the script arguments help message.
> +[-n] - Specify the number of top functions to print.
> + - If this flag is not specified, the tool defaults to 25.
> +
> +Example of usage:
> +topN_perf.py -n 20 -- qemu-arm coulomb_double-arm
> +
Based on discussion we've had upstream since you sent this, I think we
will be keeping license and authorship information out of the
docstrings, so this part can stay a comment.
> +This file is a part of the project "TCG Continuous Benchmarking".
> +
> +Copyright (C) 2020 Ahmed Karaman <ahmedkhaledkaraman@gmail.com>
> +Copyright (C) 2020 Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
> +
> +This program is free software: you can redistribute it and/or modify
> +it under the terms of the GNU General Public License as published by
> +the Free Software Foundation, either version 2 of the License, or
> +(at your option) any later version.
> +
> +This program is distributed in the hope that it will be useful,
> +but WITHOUT ANY WARRANTY; without even the implied warranty of
> +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +GNU General Public License for more details.
> +
> +You should have received a copy of the GNU General Public License
> +along with this program. If not, see <https://www.gnu.org/licenses/>.
> +"""
>
> import argparse
> import os
> import subprocess
> import sys
> +import tempfile
>
>
> # Parse the command line arguments
> -parser = argparse.ArgumentParser(
> - usage='topN_perf.py [-h] [-n] <number of displayed top functions > -- '
> +PARSER = argparse.ArgumentParser(
> + usage='topN_perf.py [-h] [-n <number of displayed top functions>] -- '
> '<qemu executable> [<qemu executable options>] '
> '<target executable> [<target executable options>]')
>
This is a little odd; generally we can avoid having such globals by
making a main() function that defines a parser as a local instead.
e.g.,
def main():
parser = ...
parser.add_argument(...)
args = parser.parse_args()
...
return 0
if __name__ == '__main__':
sys.exit(main())
> -parser.add_argument('-n', dest='top', type=int, default=25,
> +PARSER.add_argument('-n', dest='top', type=int, default=25,
> help='Specify the number of top functions to print.')
>
> -parser.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
> +PARSER.add_argument('command', type=str, nargs='+', help=argparse.SUPPRESS)
>
> -args = parser.parse_args()
> +ARGS = PARSER.parse_args()
>
> # Extract the needed variables from the args
> -command = args.command
> -top = args.top
> +COMMAND = ARGS.command
> +TOP = ARGS.top
>
> # Insure that perf is installed
> -check_perf_presence = subprocess.run(["which", "perf"],
> - stdout=subprocess.DEVNULL)
> -if check_perf_presence.returncode:
> +CHECK_PERF_PRESENCE = subprocess.run(["which", "perf"],
> + stdout=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_PRESENCE.returncode:
> sys.exit("Please install perf before running the script!")
>
> # Insure user has previllage to run perf
> -check_perf_executability = subprocess.run(["perf", "stat", "ls", "/"],
> +CHECK_PERF_EXECUTABILITY = subprocess.run(["perf", "stat", "ls", "/"],
> stdout=subprocess.DEVNULL,
> - stderr=subprocess.DEVNULL)
> -if check_perf_executability.returncode:
> - sys.exit(
> -"""
> + stderr=subprocess.DEVNULL,
> + check=False)
> +if CHECK_PERF_EXECUTABILITY.returncode:
> + sys.exit("""
> Error:
> You may not have permission to collect stats.
>
> @@ -85,43 +90,42 @@ To make this setting permanent, edit /etc/sysctl.conf too, e.g.:
> kernel.perf_event_paranoid = -1
>
> * Alternatively, you can run this script under sudo privileges.
> -"""
> -)
> -
> -# Run perf record
> -perf_record = subprocess.run((["perf", "record", "--output=/tmp/perf.data"] +
> - command),
> - stdout=subprocess.DEVNULL,
> - stderr=subprocess.PIPE)
> -if perf_record.returncode:
> - os.unlink('/tmp/perf.data')
> - sys.exit(perf_record.stderr.decode("utf-8"))
> -
> -# Save perf report output to /tmp/perf_report.out
> -with open("/tmp/perf_report.out", "w") as output:
> - perf_report = subprocess.run(
> - ["perf", "report", "--input=/tmp/perf.data", "--stdio"],
> - stdout=output,
> - stderr=subprocess.PIPE)
> - if perf_report.returncode:
> - os.unlink('/tmp/perf.data')
> - output.close()
> - os.unlink('/tmp/perf_report.out')
> - sys.exit(perf_report.stderr.decode("utf-8"))
> -
> -# Read the reported data to functions[]
> -functions = []
> -with open("/tmp/perf_report.out", "r") as data:
> - # Only read lines that are not comments (comments start with #)
> - # Only read lines that are not empty
> - functions = [line for line in data.readlines() if line and line[0]
> - != '#' and line[0] != "\n"]
> -
> -# Limit the number of top functions to "top"
> -number_of_top_functions = top if len(functions) > top else len(functions)
> -
> -# Store the data of the top functions in top_functions[]
> -top_functions = functions[:number_of_top_functions]
> +""")
> +
> +# Run perf and save all intermediate files in a temporary directory
> +with tempfile.TemporaryDirectory() as tmpdir:
> + RECORD_PATH = os.path.join(tmpdir, "record.data")
> + REPORT_PATH = os.path.join(tmpdir, "report.txt")
> +
> + PERF_RECORD = subprocess.run((["perf", "record", "--output="+RECORD_PATH] +
> + COMMAND),
> + stdout=subprocess.DEVNULL,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_RECORD.returncode:
> + sys.exit(PERF_RECORD.stderr.decode("utf-8"))
> +
> + with open(REPORT_PATH, "w") as output:
> + PERF_REPORT = subprocess.run(
> + ["perf", "report", "--input="+RECORD_PATH, "--stdio"],
> + stdout=output,
> + stderr=subprocess.PIPE,
> + check=False)
> + if PERF_REPORT.returncode:
> + sys.exit(PERF_REPORT.stderr.decode("utf-8"))
> +
> + # Save the reported data to FUNCTIONS[]
> + with open(REPORT_PATH, "r") as data:
> + # Only read lines that are not comments (comments start with #)
> + # Only read lines that are not empty
> + FUNCTIONS = [line for line in data.readlines() if line and
> + line[0] != '#' and line[0] != "\n"]
> +
> +# Limit the number of top functions to "TOP"
> +NO_TOP_FUNCTIONS = TOP if len(FUNCTIONS) > TOP else len(FUNCTIONS)
> +
> +# Store the data of the top functions in TOP_FUNCTIONS[]
> +TOP_FUNCTIONS = FUNCTIONS[:NO_TOP_FUNCTIONS]
>
> # Print table header
> print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
> @@ -134,7 +138,7 @@ print('{:>4} {:>10} {:<30} {}\n{} {} {} {}'.format('No.',
> '-' * 25))
>
> # Print top N functions
> -for (index, function) in enumerate(top_functions, start=1):
> +for (index, function) in enumerate(TOP_FUNCTIONS, start=1):
> function_data = function.split()
> function_percentage = function_data[0]
> function_name = function_data[-1]
> @@ -143,7 +147,3 @@ for (index, function) in enumerate(top_functions, start=1):
> function_percentage,
> function_name,
> function_invoker))
> -
> -# Remove intermediate files
> -os.unlink('/tmp/perf.data')
> -os.unlink('/tmp/perf_report.out')
>
^ permalink raw reply [flat|nested] 26+ messages in thread
* Re: [PATCH 1/9] scripts/performance: Refactor topN_perf.py
2020-10-01 20:41 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py John Snow
@ 2020-10-01 21:59 ` John Snow
0 siblings, 0 replies; 26+ messages in thread
From: John Snow @ 2020-10-01 21:59 UTC (permalink / raw)
To: Ahmed Karaman, qemu-devel, aleksandar.qemu.devel, philmd,
alex.bennee, eblake, ldoktor, rth, ehabkost, crosa
On 10/1/20 4:41 PM, John Snow wrote:
> I realize this review comes well after you are no longer being paid to
> work on this, so I am offering my time to help polish your patches if
> you would like.
Actually, I see now that you are adding your name to the MAINTAINERS
file here, so I suspect you probably rather want to be more involved
than not.
I cleaned up patch 1/9 provisionally with my own style preferences, but
it's all just style stuff, and it's mostly things I wouldn't actually
require you to do (...I went way overboard.)
https://gitlab.com/jsnow/qemu/-/commit/c66a4a6ca8ccc3d406b92796935f92057bf1e48d
What I'd recommend for your cleanup is actually *much* simpler;
Use pylint 2.6.0 and flake8 3.8.3:
> pip3 install --user pylint==2.6.0 flake8==3.8.3
flake8's default settings should be pretty good, but pylint has a lot of
warnings you can ignore.
In particular, it's OK to use script-style python (Scripts with a
#!/usr/bin/env python3, and where you do not use python functions to
avoid side-effects that occur on 'import'.) In this case, IGNORE any of
pylint's warnings telling you that you have too many lines, that you
need to UPPERCASE variable names, etc. It just hurts readability here.
So I'd actually ask that you revise these patches to remove all of the
UPPERCASE variable names, and then check your code with these:
flake8 topN_perf.py
pylint --disable=invalid-name topN_perf.py
Use your best judgment -- If something seems like it looks worse, it
probably is. If in doubt, please reach out and ask.
--js
^ permalink raw reply [flat|nested] 26+ messages in thread
end of thread, other threads:[~2020-10-01 22:01 UTC | newest]
Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-08-28 10:40 [PATCH 0/9] GSoC 2020 - TCG Continuous Benchmarking scripts and tools Ahmed Karaman
2020-08-28 10:40 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py Ahmed Karaman
2020-09-07 20:52 ` Aleksandar Markovic
2020-09-18 20:33 ` Aleksandar Markovic
2020-09-19 11:17 ` Bottleneck problem to merge Python patches Philippe Mathieu-Daudé
2020-09-21 14:49 ` John Snow
2020-09-21 15:54 ` Eduardo Habkost
2020-09-21 17:57 ` Cleber Rosa
2020-10-01 20:41 ` [PATCH 1/9] scripts/performance: Refactor topN_perf.py John Snow
2020-10-01 21:59 ` John Snow
2020-08-28 10:40 ` [PATCH 2/9] scripts/performance: Refactor topN_callgrind.py Ahmed Karaman
2020-09-07 20:53 ` Aleksandar Markovic
2020-08-28 10:40 ` [PATCH 3/9] scripts/performance: Refactor dissect.py Ahmed Karaman
2020-09-02 8:48 ` Aleksandar Markovic
2020-08-28 10:40 ` [PATCH 4/9] scripts/performance: Add list_fn_callees.py script Ahmed Karaman
2020-08-28 10:40 ` [PATCH 5/9] scripts/performance: Add list_helpers.py script Ahmed Karaman
2020-08-28 10:40 ` [PATCH 6/9] scripts/performance: Add bisect.py script Ahmed Karaman
2020-08-28 10:41 ` [PATCH 7/9] tests/performance: Add nightly tests Ahmed Karaman
2020-09-02 8:36 ` Aleksandar Markovic
2020-09-02 13:26 ` Alex Bennée
2020-09-02 17:29 ` Ahmed Karaman
2020-09-15 16:39 ` Aleksandar Markovic
2020-09-16 8:31 ` Alex Bennée
2020-08-28 10:41 ` [PATCH 8/9] MAINTAINERS: Add 'tests/performance' to 'Performance Tools and Tests' subsection Ahmed Karaman
2020-09-02 8:37 ` Aleksandar Markovic
2020-08-28 10:41 ` [PATCH 9/9] scripts/performance: Add topN_system.py script Ahmed Karaman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.