From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12CD8C32771 for ; Wed, 28 Sep 2022 10:22:11 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id ACD6384C5B; Wed, 28 Sep 2022 12:21:42 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="jltfHX8Y"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 70E0484B6B; Wed, 28 Sep 2022 12:21:08 +0200 (CEST) Received: from mail-oi1-x22c.google.com (mail-oi1-x22c.google.com [IPv6:2607:f8b0:4864:20::22c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 4BF7484B6B for ; Wed, 28 Sep 2022 12:21:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@google.com Received: by mail-oi1-x22c.google.com with SMTP id n83so14828634oif.11 for ; Wed, 28 Sep 2022 03:21:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date; bh=yyOeu8qBB2wDKratj0SDTkROiZOp126ol26Gha2E25U=; b=jltfHX8YP0DB4z5c22BMu8dV6VjCx+mNyRdllyrpUgGxJ+Du+JzJzGCUAxn+Y/DWRZ cDanhYMpgjE5soyZ1PNO71baAakD7iA+J6j0GR6dpXwsD+uJLPKrwhign+GuT/jNBpuA O3dlnUzb1j0Q4oKZqwLNb0pAGxCg3ghoPMGNE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date; bh=yyOeu8qBB2wDKratj0SDTkROiZOp126ol26Gha2E25U=; b=U6mDwnJ1LNCtHJv3gt7EhqcdMlFfy7o6aVtG9duoBFrW6F/z6e+KbzfzVx7d3Q4Wkh Cw5z+E5oRn/dfA2NYR/c/A6o3fkmT27v6UbEPxYg9BpatHr3qz4YZhXEs+HbKp8LW8rV rSkTnAE/ycUaW6+zcuh4Lv2L7qo3BkwaXyLpHJN3xBMBV6xa6irst3XfVQ5nQlqFrq6C OMGEDf0gGWKM8KNFbCMul1sCsN2xYhah8FR2FdIAcmDwhF7bwTuTzH5gMdfvi+2qNlbV 1mtuWLV2B8V1Pf/K/xZuHZepA7l/Z+plRTS/yr9XnoM18OsfbYZ+69455f4EsAB65brA n//w== X-Gm-Message-State: ACrzQf2yOj3xyknTzdgGtf5D582atzA0TKmRnlJFe9iSIKUNdJpeA7N4 gI3PnjHbWgnxhrnwIB14s19epPeM1BQ3L/151bervg== X-Google-Smtp-Source: AMsMyM5Y6md6fqSXs7sosXTZ/RICTpcoj3JkUUIotXLMZTA2/dMEOnxXQkLfJPpLZousLE9K1JmqEt+CwDeMk7sWrLY= X-Received: by 2002:a05:6808:d53:b0:351:1cab:9c29 with SMTP id w19-20020a0568080d5300b003511cab9c29mr4098495oik.64.1664360461814; Wed, 28 Sep 2022 03:21:01 -0700 (PDT) MIME-Version: 1.0 References: <20220925150248.2524421-1-sjg@chromium.org> <20220925150248.2524421-26-sjg@chromium.org> In-Reply-To: From: Simon Glass Date: Wed, 28 Sep 2022 04:20:42 -0600 Message-ID: Subject: Re: [PATCH 25/45] test: Support tests which can only be run manually To: Heinrich Schuchardt Cc: Tom Rini , Rui Miguel Silva , Stefan Roese , U-Boot Mailing List Content-Type: text/plain; charset="UTF-8" X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean Hi Heinrich, On Mon, 26 Sept 2022 at 00:56, Heinrich Schuchardt wrote: > > On 9/25/22 17:02, Simon Glass wrote: > > At present we normally write tests either in Python or in C. But most > > Python tests end up doing a lot of checks which would be better done in C. > > Checks done in C are orders of magnitude faster and it is possible to get > > full access to U-Boot's internal workings, rather than just relying on > > the command line. > > > > The model is to have a Python test set up some things and then use C code > > (in a unit test) to check that they were done correctly. But we don't want > > those checks to happen as part of normal test running, since each C unit > > tests is dependent on the associate Python tests, so cannot run without > > it. > > > > To acheive this, add a new UT_TESTF_MANUAL flag to use with the C 'check' > > tests, so that they can be skipped by default when the 'ut' command is > > used. Require that tests have a name ending with '_norun', so that pytest > > knows to skip them. > > > > Signed-off-by: Simon Glass > > --- > > > > arch/sandbox/cpu/spl.c | 2 +- > > doc/develop/tests_writing.rst | 22 ++++++++++++++++++++++ > > include/test/test.h | 8 ++++++++ > > include/test/ut.h | 4 +++- > > test/cmd_ut.c | 16 +++++++++++++--- > > test/dm/test-dm.c | 2 +- > > test/py/conftest.py | 8 +++++++- > > test/test-main.c | 27 ++++++++++++++++++++++++++- > > 8 files changed, 81 insertions(+), 8 deletions(-) > > > > diff --git a/arch/sandbox/cpu/spl.c b/arch/sandbox/cpu/spl.c > > index 6d4981152bb..2678370481a 100644 > > --- a/arch/sandbox/cpu/spl.c > > +++ b/arch/sandbox/cpu/spl.c > > @@ -90,7 +90,7 @@ void spl_board_init(void) > > int ret; > > > > ret = ut_run_list("spl", NULL, tests, count, > > - state->select_unittests, 1); > > + state->select_unittests, 1, false); > > /* continue execution into U-Boot */ > > } > > } > > diff --git a/doc/develop/tests_writing.rst b/doc/develop/tests_writing.rst > > index 1ddf7a353a7..d3ffc550300 100644 > > --- a/doc/develop/tests_writing.rst > > +++ b/doc/develop/tests_writing.rst > > @@ -74,6 +74,28 @@ NOT rely on running with sandbox, but instead should function correctly on any > > board supported by U-Boot. > > > > > > +Mixing Python and C > > +------------------- > > + > > +The best of both worlds is sometimes to have a Python test set things up and > > +perform some operations, with a 'checker' C unit test doing the checks > > +afterwards. This can be achieved with these steps: > > + > > +- Add the `UT_TESTF_MANUAL` flag to the checker test so that the `ut` command > > + does not run it by default > > +- Add a `_norun` suffix to the name so that pytest knows to skip it too > > + > > +In your Python test use the `-f` flag to the `ut` command to force the checker > > +test to run it, e.g.:: > > + > > + # Do the Python part > > + host load ... > > + bootm ... > > + > > + # Run the checker to make sure that everything worked > > + ut -f bootstd vbe_test_fixup_norun > > Shouldn't the documentation describes what a test looks like that can > only be run manually? It is just a normal test and I figure there is an example in the code. What sort of info is missing here? Regards, Simon