From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EDD47C433F5 for ; Sat, 2 Apr 2022 17:07:23 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 8056883989; Sat, 2 Apr 2022 19:06:41 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Oih1V5EF"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 4172A8396C; Sat, 2 Apr 2022 19:06:37 +0200 (CEST) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id E743E83966 for ; Sat, 2 Apr 2022 19:06:32 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ej1-x633.google.com with SMTP id ot30so1971672ejb.12 for ; Sat, 02 Apr 2022 10:06:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=v1MeL2F2OiN0i/JKegfdkQhpL2m56olAgbbH02d4xqk=; b=Oih1V5EFZvJ4JeKj12T8XVIo1W8UtD7y6XI0yxChM1iDdaBCJm/riReanRrn0TGv5U YvmelMfa6Sbx+Y5T+hUrYdwgJGzVPu4PHI4cxCIVPkplV5twEyczNGFU5iGWpiXQxhL9 SS6UrLb8bnzQwrMzzuSQdlfo7w1UiuNl7HtttHf4ErQD2ZExNNzGik9Fiiuvn/IYW9zl Sq6QDWG2xS492tVS9HW7mZnBuOAosrJAfa8dhPQW4BkxE/ay+J5zSoPjIbGQqXkx4gWz PzCi5wl4Qzk30z1dGvrmxsOnvuD/HlB4/0IsbYGOnrz8McYG1+j//HOcSmf+1id9wfjk v/2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=v1MeL2F2OiN0i/JKegfdkQhpL2m56olAgbbH02d4xqk=; b=CrNMl3HjxT5wCpUwc/qHwUdoInvf7IDl8LcBHcrG6TOgqL1vKb92teC4VThTsiQ4GH +2+MJa4dwTLlfGr8kHBw6JIqtG17bAVLnIbAbg9ujscF+uQlZXY8K9Q8zMlgfRG5UK4T 9RA/nasV2jfIEnuiHArhTugdKNw7QMpNXI6LfFhZ9lP4FC5iVFPtJdQxmqbUpIhiMqVB VX2XQxgk2X5xPl7wH8DLWpEFbjV79Zj6j6cnK94ikxd4bOUySWq0gTx9ayCy318rNvoF fJw6VsKNPgCyeQLO8WFOROTrEL6aNzI5OyJIGn+M2ARHLePiHGnYJKW+9sqKPOT+xVDz nFVg== X-Gm-Message-State: AOAM53141gPoHzNkN2wyvI1f1R+44dQmWSUQgvNOu7vE8yqBvzMqLwbs qGu4VEWdbVGetCYcbqrDuStX0KJo/Gw= X-Google-Smtp-Source: ABdhPJz31dYYJ76zOyKPbn8y+8twZj5H6/dF99CesMbXSqeJlBJXq5xoPi9BcBF17ht7N/yWooqpHw== X-Received: by 2002:a17:907:c16:b0:6da:7d72:1357 with SMTP id ga22-20020a1709070c1600b006da7d721357mr4415443ejc.251.1648919192511; Sat, 02 Apr 2022 10:06:32 -0700 (PDT) Received: from localhost.localdomain ([178.233.178.185]) by smtp.gmail.com with ESMTPSA id qb24-20020a1709077e9800b006e029bd4c24sm2268198ejc.193.2022.04.02.10.06.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Apr 2022 10:06:32 -0700 (PDT) From: Alper Nebi Yasak To: u-boot@lists.denx.de Cc: Walter Lozano , Simon Glass , Alper Nebi Yasak Subject: [PATCH 4/5] patman: test_util: Customize unittest test results for more info Date: Sat, 2 Apr 2022 20:06:07 +0300 Message-Id: <20220402170609.17790-5-alpernebiyasak@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220402170609.17790-1-alpernebiyasak@gmail.com> References: <20220402170609.17790-1-alpernebiyasak@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean By default, unittest test summaries only print extended info about tests that failed or couldn't run due to an error. Use a custom text result class to print info about more cases: skipped tests, expected failures and unexpected successes. Signed-off-by: Alper Nebi Yasak --- This could be squashed into the previous patch, but makes the diff ugly. tools/patman/test_util.py | 46 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/tools/patman/test_util.py b/tools/patman/test_util.py index ba8f87f75f03..130d91409140 100644 --- a/tools/patman/test_util.py +++ b/tools/patman/test_util.py @@ -102,6 +102,51 @@ def capture_sys_output(): sys.stdout, sys.stderr = old_out, old_err +class FullTextTestResult(unittest.TextTestResult): + """A test result class that can print extended text results to a stream + + This is meant to be used by a TestRunner as a result class. Like + TextTestResult, this prints out the names of tests as they are run, + errors as they occur, and a summary of the results at the end of the + test run. Beyond those, this prints information about skipped tests, + expected failures and unexpected successes. + + Args: + stream: A file-like object to write results to + descriptions (bool): True to print descriptions with test names + verbosity (int): Detail of printed output per test as they run + Test stdout and stderr always get printed when buffering + them is disabled by the test runner. In addition to that, + 0: Print nothing + 1: Print a dot per test + 2: Print test names + """ + def __init__(self, stream, descriptions, verbosity): + self.verbosity = verbosity + super().__init__(stream, descriptions, verbosity) + + def printErrors(self): + "Called by TestRunner after test run to summarize the tests" + # The parent class doesn't keep unexpected successes in the same + # format as the rest. Adapt it to what printErrorList expects. + unexpected_successes = [ + (test, 'Test was expected to fail, but succeeded.\n') + for test in self.unexpectedSuccesses + ] + + super().printErrors() # FAIL and ERROR + self.printErrorList('SKIP', self.skipped) + self.printErrorList('XFAIL', self.expectedFailures) + self.printErrorList('XPASS', unexpected_successes) + + def addSkip(self, test, reason): + """Called when a test is skipped.""" + # Add empty line to keep spacing consistent with other results + if not reason.endswith('\n'): + reason += '\n' + super().addSkip(test, reason) + + def run_test_suites(toolname, debug, verbosity, test_preserve_dirs, processes, test_name, toolpath, class_and_module_list): """Run a series of test suites and collect the results @@ -135,6 +180,7 @@ def run_test_suites(toolname, debug, verbosity, test_preserve_dirs, processes, runner = unittest.TextTestRunner( stream=sys.stdout, verbosity=(1 if verbosity is None else verbosity), + resultclass=FullTextTestResult, ) if use_concurrent and processes != 1: -- 2.35.1