From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A43AC433EF for ; Sat, 2 Apr 2022 17:07:39 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 32A93839A1; Sat, 2 Apr 2022 19:06:43 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="L/Ed9UsF"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 46FC683989; Sat, 2 Apr 2022 19:06:40 +0200 (CEST) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 63DEE83989 for ; Sat, 2 Apr 2022 19:06:35 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ej1-x636.google.com with SMTP id p15so11866825ejc.7 for ; Sat, 02 Apr 2022 10:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=via69kdHL1YS9S11WxDiMptMuY4fTzrLr7zLzHEGjMU=; b=L/Ed9UsF0YUHBp2sSULJM6YRtLehVOmcNQG/0ndi/Ql5zyHw33oxdxKfGvVNCWGbUT qtetpIp3ciyHlgPNvzAVttHGVNg5aF3BG3VEZkMPFQC7cFeUS3lPD0uQDJQ9bRSCZn8W aFurvblBF5YZIw6v/dvhVCZeRQzlIqDXF6opBnh0tkjcMif0WsulPLUk/xBmMT1hs1PB kpt8azUyiZSuC46bYfv5CNcfU8VFjfOyJ200UqVluenjOmfum9lwAEZmDxm5vIfW/ftT upS/9P6jdwnyqshXAlAke+jrQr0ybVAIrHO9CzMCeZ0pNxeonCgp68Ddnj84GvMAFwje lveQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=via69kdHL1YS9S11WxDiMptMuY4fTzrLr7zLzHEGjMU=; b=SzhTb2/0jszOathE28TV0c+D1ruq+KC6s+7pnuspE7g7UK1O9kvNkijxeWGVYGh4Pl TkKpg0g0haLcB/aQ9K/fNBeiYZsW2kc3uQogulkk1qR/J91MVBYUofjF4Xoeo9SURQbg yXgj0eC8s+jMstkAQ1lsSuEL5aEEyy1ks2nXhEZKEi2fwFUKHg1weYqipfky6A1b7mNh hQhCbZtqKRsP91O+hBnoH4brPVEkHEmOHrFnpbSGlpVqWAEZXouxPSv/l7zG6dMTMl/I eeiW0+Durx4M30XFX6Hoy14G2RTaapi2N6+khw1zwFZG7rflZADO58cU1qDzZjStfXl/ D12w== X-Gm-Message-State: AOAM532/ROpd1hMyrQfQa6T3cp+4qUa5YbcO5nPRDt/VkoRLO98Q/u8H 0PaqDkS6RqNa7uB7xtlfVs2tLffxJgc= X-Google-Smtp-Source: ABdhPJx2EzT6k4LUXpZaRANyFJ8/lCgC+M8NmB0/AGiKgoua5dZFKbKwzAs29SClzXvEN8QsaljIgQ== X-Received: by 2002:a17:907:daa:b0:6e4:9b0d:3f1 with SMTP id go42-20020a1709070daa00b006e49b0d03f1mr4431933ejc.37.1648919189983; Sat, 02 Apr 2022 10:06:29 -0700 (PDT) Received: from localhost.localdomain ([178.233.178.185]) by smtp.gmail.com with ESMTPSA id qb24-20020a1709077e9800b006e029bd4c24sm2268198ejc.193.2022.04.02.10.06.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Apr 2022 10:06:29 -0700 (PDT) From: Alper Nebi Yasak To: u-boot@lists.denx.de Cc: Walter Lozano , Simon Glass , Alper Nebi Yasak Subject: [PATCH 3/5] patman: test_util: Use unittest text runner to print test results Date: Sat, 2 Apr 2022 20:06:06 +0300 Message-Id: <20220402170609.17790-4-alpernebiyasak@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220402170609.17790-1-alpernebiyasak@gmail.com> References: <20220402170609.17790-1-alpernebiyasak@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean The python tools' test utilities handle printing test results, but the output is quite bare compared to an ordinary unittest run. Delegate printing the results to a unittest text runner, which gives us niceties like clear separation between each test's result and how long it took to run the test suite. Unfortunately it does not print info for skipped tests by default, but this can be handled later by a custom test result subclass. It also does not print the tool name; manually print a heading that includes the toolname so that the outputs of each tool's tests are distinguishable in the CI output. Signed-off-by: Alper Nebi Yasak --- tools/binman/main.py | 8 ++---- tools/buildman/main.py | 8 ++---- tools/dtoc/main.py | 9 +++--- tools/dtoc/test_fdt.py | 8 +++--- tools/patman/main.py | 8 ++---- tools/patman/test_util.py | 58 ++++++++++++++------------------------- 6 files changed, 38 insertions(+), 61 deletions(-) diff --git a/tools/binman/main.py b/tools/binman/main.py index 9392b59adb15..d2eea1c4bb76 100755 --- a/tools/binman/main.py +++ b/tools/binman/main.py @@ -14,7 +14,6 @@ import site import sys import traceback -import unittest # Get the absolute path to this file at run-time our_path = os.path.dirname(os.path.realpath(__file__)) @@ -80,19 +79,18 @@ def RunTests(debug, verbosity, processes, test_preserve_dirs, args, toolpath): from binman import image_test import doctest - result = unittest.TestResult() test_name = args and args[0] or None # Run the entry tests first ,since these need to be the first to import the # 'entry' module. - test_util.run_test_suites( - result, debug, verbosity, test_preserve_dirs, processes, test_name, + result = test_util.run_test_suites( + 'binman', debug, verbosity, test_preserve_dirs, processes, test_name, toolpath, [bintool_test.TestBintool, entry_test.TestEntry, ftest.TestFunctional, fdt_test.TestFdt, elf_test.TestElf, image_test.TestImage, cbfs_util_test.TestCbfs, fip_util_test.TestFip]) - return test_util.report_result('binman', test_name, result) + return (0 if result.wasSuccessful() else 1) def RunTestCoverage(toolpath): """Run the tests and check that we get 100% coverage""" diff --git a/tools/buildman/main.py b/tools/buildman/main.py index 3b6af240802d..67c560c48d37 100755 --- a/tools/buildman/main.py +++ b/tools/buildman/main.py @@ -11,7 +11,6 @@ import os import re import sys -import unittest # Bring in the patman libraries our_path = os.path.dirname(os.path.realpath(__file__)) @@ -34,19 +33,18 @@ def RunTests(skip_net_tests, verboose, args): from buildman import test import doctest - result = unittest.TestResult() test_name = args and args[0] or None if skip_net_tests: test.use_network = False # Run the entry tests first ,since these need to be the first to import the # 'entry' module. - test_util.run_test_suites( - result, False, verboose, False, None, test_name, [], + result = test_util.run_test_suites( + 'buildman', False, verboose, False, None, test_name, [], [test.TestBuild, func_test.TestFunctional, 'buildman.toolchain', 'patman.gitutil']) - return test_util.report_result('buildman', test_name, result) + return (0 if result.wasSuccessful() else 1) options, args = cmdline.ParseArgs() diff --git a/tools/dtoc/main.py b/tools/dtoc/main.py index fac9db9c786f..5508759d4d56 100755 --- a/tools/dtoc/main.py +++ b/tools/dtoc/main.py @@ -24,7 +24,6 @@ from argparse import ArgumentParser import os import sys -import unittest # Bring in the patman libraries our_path = os.path.dirname(os.path.realpath(__file__)) @@ -49,18 +48,18 @@ def run_tests(processes, args): from dtoc import test_src_scan from dtoc import test_dtoc - result = unittest.TestResult() sys.argv = [sys.argv[0]] test_name = args.files and args.files[0] or None test_dtoc.setup() - test_util.run_test_suites( - result, debug=True, verbosity=1, test_preserve_dirs=False, + result = test_util.run_test_suites( + toolname='dtoc', debug=True, verbosity=1, test_preserve_dirs=False, processes=processes, test_name=test_name, toolpath=[], class_and_module_list=[test_dtoc.TestDtoc,test_src_scan.TestSrcScan]) - return test_util.report_result('binman', test_name, result) + return (0 if result.wasSuccessful() else 1) + def RunTestCoverage(): """Run the tests and check that we get 100% coverage""" diff --git a/tools/dtoc/test_fdt.py b/tools/dtoc/test_fdt.py index 3859af8d032b..3baf4437cdd3 100755 --- a/tools/dtoc/test_fdt.py +++ b/tools/dtoc/test_fdt.py @@ -784,13 +784,13 @@ def RunTests(args): Returns: Return code, 0 on success """ - result = unittest.TestResult() test_name = args and args[0] or None - test_util.run_test_suites( - result, False, False, False, None, test_name, None, + result = test_util.run_test_suites( + 'test_fdt', False, False, False, None, test_name, None, [TestFdt, TestNode, TestProp, TestFdtUtil]) - return test_util.report_result('fdt', test_name, result) + return (0 if result.wasSuccessful() else 1) + if __name__ != '__main__': sys.exit(1) diff --git a/tools/patman/main.py b/tools/patman/main.py index 2a2ac4570931..66d4806c8d8f 100755 --- a/tools/patman/main.py +++ b/tools/patman/main.py @@ -12,7 +12,6 @@ import shutil import sys import traceback -import unittest if __name__ == "__main__": # Allow 'from patman import xxx to work' @@ -134,13 +133,12 @@ import doctest from patman import func_test - result = unittest.TestResult() - test_util.run_test_suites( - result, False, False, False, None, None, None, + result = test_util.run_test_suites( + 'patman', False, False, False, None, None, None, [test_checkpatch.TestPatch, func_test.TestFunctional, 'gitutil', 'settings', 'terminal']) - sys.exit(test_util.report_result('patman', args.testname, result)) + sys.exit(0 if result.wasSuccessful() else 1) # Process commits, produce patches files, check them, email them elif args.cmd == 'send': diff --git a/tools/patman/test_util.py b/tools/patman/test_util.py index a4c2a2c3c0b2..ba8f87f75f03 100644 --- a/tools/patman/test_util.py +++ b/tools/patman/test_util.py @@ -102,36 +102,12 @@ def capture_sys_output(): sys.stdout, sys.stderr = old_out, old_err -def report_result(toolname:str, test_name: str, result: unittest.TestResult): - """Report the results from a suite of tests - - Args: - toolname: Name of the tool that ran the tests - test_name: Name of test that was run, or None for all - result: A unittest.TestResult object containing the results - """ - print(result) - for test, err in result.errors: - print(test.id(), err) - for test, err in result.failures: - print(test.id(), err) - if result.skipped: - print('%d %s test%s SKIPPED:' % (len(result.skipped), toolname, - 's' if len(result.skipped) > 1 else '')) - for skip_info in result.skipped: - print('%s: %s' % (skip_info[0], skip_info[1])) - if result.errors or result.failures: - print('%s tests FAILED' % toolname) - return 1 - return 0 - - -def run_test_suites(result, debug, verbosity, test_preserve_dirs, processes, +def run_test_suites(toolname, debug, verbosity, test_preserve_dirs, processes, test_name, toolpath, class_and_module_list): """Run a series of test suites and collect the results Args: - result: A unittest.TestResult object to add the results to + toolname: Name of the tool that ran the tests debug: True to enable debugging, which shows a full stack trace on error verbosity: Verbosity level to use (0-4) test_preserve_dirs: True to preserve the input directory used by tests @@ -145,11 +121,6 @@ def run_test_suites(result, debug, verbosity, test_preserve_dirs, processes, class_and_module_list: List of test classes (type class) and module names (type str) to run """ - for module in class_and_module_list: - if isinstance(module, str) and (not test_name or test_name == module): - suite = doctest.DocTestSuite(module) - suite.run(result) - sys.argv = [sys.argv[0]] if debug: sys.argv.append('-D') @@ -161,6 +132,19 @@ def run_test_suites(result, debug, verbosity, test_preserve_dirs, processes, suite = unittest.TestSuite() loader = unittest.TestLoader() + runner = unittest.TextTestRunner( + stream=sys.stdout, + verbosity=(1 if verbosity is None else verbosity), + ) + + if use_concurrent and processes != 1: + suite = ConcurrentTestSuite(suite, + fork_for_tests(processes or multiprocessing.cpu_count())) + + for module in class_and_module_list: + if isinstance(module, str) and (not test_name or test_name == module): + suite.addTests(doctest.DocTestSuite(module)) + for module in class_and_module_list: if isinstance(module, str): continue @@ -179,9 +163,9 @@ def run_test_suites(result, debug, verbosity, test_preserve_dirs, processes, suite.addTests(loader.loadTestsFromName(test_name, module)) else: suite.addTests(loader.loadTestsFromTestCase(module)) - if use_concurrent and processes != 1: - concurrent_suite = ConcurrentTestSuite(suite, - fork_for_tests(processes or multiprocessing.cpu_count())) - concurrent_suite.run(result) - else: - suite.run(result) + + print(f" Running {toolname} tests ".center(70, "=")) + result = runner.run(suite) + print() + + return result -- 2.35.1