From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 769F2C43334 for ; Tue, 28 Jun 2022 13:43:55 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 05BB684516; Tue, 28 Jun 2022 15:39:41 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="i+j8MMHa"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2DB7484511; Tue, 28 Jun 2022 15:39:36 +0200 (CEST) Received: from mail-ua1-x936.google.com (mail-ua1-x936.google.com [IPv6:2607:f8b0:4864:20::936]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 4D882844D4 for ; Tue, 28 Jun 2022 15:38:50 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@google.com Received: by mail-ua1-x936.google.com with SMTP id b7so4560398uap.5 for ; Tue, 28 Jun 2022 06:38:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=mime-version:sender:from:in-reply-to:references:date:message-id :subject:to:cc; bh=tyDq+2xVR+fAbKqV2hQPUPcdMD78A5RqvVXOHjxeWj8=; b=i+j8MMHaDE5sfqJpjiJ10QpZNNvYdGxajiHSFYV0Ud64F9obv01YWBT3rYVZK1cZIT nN9BKthQUwQrAYNTwhWdYO109Qwwf8qPAt/7hpOon2nCGBcEoZQlYc9VJSMtrFNx9L5n 1WMDuAFgEWGSBBmUGKP0buST3L3A34MeVcX6w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:sender:from:in-reply-to:references :date:message-id:subject:to:cc; bh=tyDq+2xVR+fAbKqV2hQPUPcdMD78A5RqvVXOHjxeWj8=; b=DyaqszdYhyccszak3UYKVWiAJtGTuleE+O0AaEN9Lxcw7kCrhR65dA3TLgVmVNm8lC u80hhLc6XqOWS4HK78FDepYbk+RHa+GgitCrKU7y6BKnuDUDNnE6Xc+6l4MnYyuqaI+4 eQGeIl2AZ15y21M9qHt0OtZHOTSauA9D4MniwnKlReGc1Xc5K5zhKTbQkhpYAnwKkhzp VSpouoizb+1KEGUW4Cjx5tc5S/nsHanP1xL5nvYsAsC+NFMWOLOP/aJE3VO94LnCeEMH zqsYiGTUEsucfOoRtCc7mPwjHjEGgl6rTjC2fk9tcBgugOuaX0MMQCFty7QEEGB/Wm+T f/eA== X-Gm-Message-State: AJIora9dyXNmvu5wCC1Vilw0deKRB3/yTfUaYqmz8cAqgbDVOEabJEj2 wiGZsbplVsARMyRBIfGVC70haSuDiVgF0w6Ye3ZGUA== X-Google-Smtp-Source: AGRyM1s+LJH9hiRri0znTREa//5gRDEi+HZJHv/AT8+dCfwQXiMtvuImdj1TsfXk7nfRuYrPPhjkiGuw9VEQvnnTFwc= X-Received: by 2002:a9f:2f07:0:b0:37f:163a:904a with SMTP id x7-20020a9f2f07000000b0037f163a904amr6582125uaj.82.1656423494427; Tue, 28 Jun 2022 06:38:14 -0700 (PDT) Received: from 480794996271 named unknown by gmailapi.google.com with HTTPREST; Tue, 28 Jun 2022 09:38:12 -0400 MIME-Version: 1.0 From: Simon Glass In-Reply-To: References: <20220402170609.17790-1-alpernebiyasak@gmail.com> <20220402170609.17790-6-alpernebiyasak@gmail.com> Date: Tue, 28 Jun 2022 09:38:12 -0400 X-Google-Sender-Auth: mx8NJnReRuXx1oKGAJuciZn8HA0 Message-ID: Subject: Re: [PATCH 5/5] patman: test_util: Print test stdout/stderr within test summaries To: Simon Glass Cc: U-Boot Mailing List , Walter Lozano , Alper Nebi Yasak Content-Type: text/plain; charset="UTF-8" X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean On Sat, 2 Apr 2022 at 11:06, Alper Nebi Yasak wrote: > > While running tests for a python tool, the tests' outputs get printed in > whatever order they happen to run, without any indication as to which > output belongs to which test. Unittest supports capturing these outputs > and printing them as part of the test summaries, but when a failure or > error occurs it switches back to printing as the tests run. Testtools > and subunit tests can do the same as their parts inherit from unittest, > but they don't outright expose this functionality. > > On the unittest side, enable output buffering for the custom test result > class. Try to avoid ugly outputs by not printing stdout/stderr before > the test summary for low verbosity levels and for successful tests. > > On the subunit side, implement a custom TestProtocolClient that enables > the same underlying functionality and injects the captured streams as > additional test details. This causes them to be merged into their test's > error traceback message, which is later rebuilt into an exception and > passed to our unittest report class. > > Signed-off-by: Alper Nebi Yasak > --- > The way I had to do this for concurrencytest feels hacky, so I am > ambivalent towards this patch. I'm actually fine with re-running a > failing binman test alone with --debug -v6 to get better, isolated > output from it. Yes that's what I tend to do, but I think we should try this patch. We can always drop it in a future release if it causes problems. > > tools/concurrencytest/concurrencytest.py | 83 +++++++++++++++++++++++- > tools/patman/test_util.py | 33 +++++++++- > 2 files changed, 112 insertions(+), 4 deletions(-) Reviewed-by: Simon Glass Applied to u-boot-dm, thanks!