From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AF04EC433F5 for ; Sat, 2 Apr 2022 17:06:35 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 2F82C8395A; Sat, 2 Apr 2022 19:06:32 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="mwyowKiI"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 1A9D683970; Sat, 2 Apr 2022 19:06:30 +0200 (CEST) Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 100E283938 for ; Sat, 2 Apr 2022 19:06:23 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=alpernebiyasak@gmail.com Received: by mail-ej1-x629.google.com with SMTP id qh7so1886312ejb.11 for ; Sat, 02 Apr 2022 10:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=x6JJj6md6VKtAFRrtkIPUTet1pg4CepQywlbhHPTXz0=; b=mwyowKiI52dM1ys6rJ13ujYekwfTfBnceYteVphUXZPpL3rX2phNJVVnknWrSsYZWZ OlJdO/+v+zRZsMfRAaeDairfStQtcmUUzn+vG0nlSh1e7spREDYa21ghxi/53fNXsR/7 gMRlo5XmECXMxE8dSzCYX9kxwCKRkAhEf1tJbMHKZV5fLAkVRt96yWEIxcN9jbKn931/ 2KSm5oU0pqwQiXrK9OL6i5Xmnh8ZpYbv1bhhjNQORP/BFNrU6EjA+Yhcv2W7H1pQUgdt +P0xCI8UWSOrE9XgIONan0zhOl/BLOXabexsExQEwAqkhnT2vIdWdRUgGzORATnACIEu 1KdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=x6JJj6md6VKtAFRrtkIPUTet1pg4CepQywlbhHPTXz0=; b=BR5YAASJ0fD2iuOHni9yVMkvVAG7wQdFNqaryuesFKIM3V4o8OcFjiZERXn7zsbQLG iTYz5wOs5NQ9lbJs6BXPKBrmVyrIlCwaZZhShWm5NhLDW+WQLmlai5EOHRyIfozSIFP1 +n+cQLSTX0+HecZiYzzufxaXOwcVfXUVUi94Y7nSBQFwo7jPhxgTYUt5P3/6d/Ll2ujZ 28qdJbAWdTSXHxA/LyQZz+C9C7AeM6K/AY+VC0rue/J1yIvCBlS92hG5Ku2HoPWPGeQs 1DjPmePHpOYQN8jr3lAsmGqYjWCTzLGVodTsNdBzaFFkvv8ObBJf6bAXeBVkKhTgBVHy qOVg== X-Gm-Message-State: AOAM532Dm7R0pCobscWCwt2Fm+LHQIP1GMiBwYthKwWohVuBgN8upw5p 8Uyc+FC+WjnF6Xe6flVY9Zss/EZl2dc= X-Google-Smtp-Source: ABdhPJxuHHdUfRqI44HFuZ0+cUopgcb/cuCWx7rpdzPpjs/e1HbgBbSccOJHFmRuEYjlP1DAy6E54g== X-Received: by 2002:a17:907:d13:b0:6e0:b799:8fcc with SMTP id gn19-20020a1709070d1300b006e0b7998fccmr4542492ejc.11.1648919182461; Sat, 02 Apr 2022 10:06:22 -0700 (PDT) Received: from localhost.localdomain ([178.233.178.185]) by smtp.gmail.com with ESMTPSA id qb24-20020a1709077e9800b006e029bd4c24sm2268198ejc.193.2022.04.02.10.06.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 02 Apr 2022 10:06:21 -0700 (PDT) From: Alper Nebi Yasak To: u-boot@lists.denx.de Cc: Walter Lozano , Simon Glass , Alper Nebi Yasak Subject: [PATCH 0/5] patman: test_util: Prettify test report outputs for python tools Date: Sat, 2 Apr 2022 20:06:03 +0300 Message-Id: <20220402170609.17790-1-alpernebiyasak@gmail.com> X-Mailer: git-send-email 2.35.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.5 at phobos.denx.de X-Virus-Status: Clean These are a few changes to the test utilities shared by binman, patman, buildman and dtoc to make their printed test results cleaner. Simon has sent a patch making test_fdt use these utils [1], this applies on top of that and affects its output as well. [1] dtoc: Update fdt tests to use test_util https://patchwork.ozlabs.org/project/uboot/patch/20220319000150.1722595-1-sjg@chromium.org/ As a comparison, here's how the report currently looks when I add three bad tests to patman (one failing, one error, one skip). Line wrapped with backslashes at the ends for convenience, most things are otherwise jammed on one line: > Printing text block to stderr: > ************** > * test_error * > ************** > Printing text block to stdout: > ++++++++++++++++ > + test_failure + > ++++++++++++++++ > > patman.func_test.TestFunctional.test_error \ > testtools.testresult.real._StringException: \ > RuntimeError: Uncaught error during test > > testtools.testresult.real._StringException: \ > AssertionError: False is not true > > [( description='patman.func_test.TestFunctional.test_failure'>, \ > 'testtools.testresult.real._StringException: \ > AssertionError: False is not true\n\n')] > 1 patman test SKIPPED: > patman.func_test.TestFunctional.test_skipped \ > (subunit.RemotedTestCase): Example skip reason > > patman tests FAILED This quickly gets very bad the more test failures there are. Here's what I get after this series: > ======================== Running patman tests ======================== > ====================================================================== > ERROR: patman.func_test.TestFunctional.test_error (subunit.RemotedTestCase) > patman.func_test.TestFunctional.test_error > ---------------------------------------------------------------------- > testtools.testresult.real._StringException: stderr: {{{ > Printing text block to stderr: > ************** > * test_error * > ************** > }}} > > RuntimeError: Uncaught error during test > > ====================================================================== > FAIL: patman.func_test.TestFunctional.test_failure (subunit.RemotedTestCase) > patman.func_test.TestFunctional.test_failure > ---------------------------------------------------------------------- > testtools.testresult.real._StringException: stdout: {{{ > Printing text block to stdout: > ++++++++++++++++ > + test_failure + > ++++++++++++++++ > }}} > > AssertionError: False is not true > > ====================================================================== > SKIP: patman.func_test.TestFunctional.test_skipped (subunit.RemotedTestCase) > patman.func_test.TestFunctional.test_skipped > ---------------------------------------------------------------------- > Example skip reason > > ---------------------------------------------------------------------- > Ran 48 tests in 4.469s > > FAILED (failures=1, errors=1, skipped=1) There are some weird parts in this as well, like the stdout/stderr being quoted with braces and test descriptions being the same as test names, but they're quirks of the libraries used for concurrency test flows. The report looks slightly better when tests are run non-concurrently. Alper Nebi Yasak (5): patman: test_util: Fix printing results for failed tests patman: test_util: Handle nonexistent tests while loading tests patman: test_util: Use unittest text runner to print test results patman: test_util: Customize unittest test results for more info patman: test_util: Print test stdout/stderr within test summaries tools/binman/main.py | 8 +- tools/buildman/main.py | 8 +- tools/concurrencytest/concurrencytest.py | 83 ++++++++++++- tools/dtoc/main.py | 9 +- tools/dtoc/test_fdt.py | 8 +- tools/patman/main.py | 8 +- tools/patman/test_util.py | 150 +++++++++++++++-------- 7 files changed, 197 insertions(+), 77 deletions(-) -- 2.35.1