From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E5AAC19F2B for ; Mon, 1 Aug 2022 13:59:22 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 3D3C68449C; Mon, 1 Aug 2022 15:59:06 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="k6VR2+fa"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id AE9D9844D6; Mon, 1 Aug 2022 15:59:03 +0200 (CEST) Received: from mail-oo1-xc31.google.com (mail-oo1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id C720681115 for ; Mon, 1 Aug 2022 15:58:58 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-oo1-xc31.google.com with SMTP id d12-20020a4aeb8c000000b004214e709b72so2037093ooj.6 for ; Mon, 01 Aug 2022 06:58:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc; bh=tg7Zg1YgiBQpRpfOk/wPUkw25F3FqMXQBwWwrdgYm+Y=; b=k6VR2+fa/zXq2UUK9+t6qxutgbDbHTAtECapIl7HXlsOQ6E7538px3AEeMVTQwthhc wnUm/aiPEQrvHcYgW72w5GTuB3UKZwdJ3DWAg3pq/fpTfA/LxBcqa+PeKwY/BqoVw34a PrF3gUmdUmqbZLV4J1vlmtJ1cADq19nAem8IE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc; bh=tg7Zg1YgiBQpRpfOk/wPUkw25F3FqMXQBwWwrdgYm+Y=; b=5ApQaMwjrZyb/c4Jl9i3M/a0hk4Tn5nf40UD3L8DEY7xjJpIU/JlqH3gntXFs/rSlG xX9//kcxu/+F8LRZoUPeUSDNv0wJH5ZG7vg5cBJ4GkcOFiUdt5MAactfUTmU07omNKjo kJWR9mUCtnofyRYm+DVnKtM0E2aGL15TAHBmDF1cnjclVwNvIjT9aScZqcdPUznG0HVZ HufPWS/rNJ/VESpXqA9A3bmEd668NaaaLQpYaPaYtrf5Zw4RmND1n8zI7dZ/wjoIiFa2 J8O092Urqp7SQmh7uketZxlDJKAFeNLmEtB477s163s14bKj26QgPprDAyD4lYrDkURd QYMA== X-Gm-Message-State: AJIora/vZlea97on7GWrm74KNq2kfHttVldgyA5aoUdfya6nGvU2RvtJ b6Hl3c68BKa6n5VU1N5u++6827eHig3jkQ== X-Google-Smtp-Source: AGRyM1s1hX5G1v7cjNQbWg6LZJXRk/rwbLp/D8Ki/cUkOuce1g2Mb69QqBdRNq/nPVXWcmvAdR0kug== X-Received: by 2002:a4a:8512:0:b0:435:77c5:f0e6 with SMTP id k18-20020a4a8512000000b0043577c5f0e6mr5266866ooh.74.1659362337166; Mon, 01 Aug 2022 06:58:57 -0700 (PDT) Received: from sjg1.lan (c-67-190-102-125.hsd1.co.comcast.net. [67.190.102.125]) by smtp.gmail.com with ESMTPSA id a14-20020a9d3e0e000000b0061cf85b2a0csm2743161otd.5.2022.08.01.06.58.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Aug 2022 06:58:56 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Simon Glass , Andrew Scull , Rui Miguel Silva Subject: [PATCH 2/5] test: Allow running tests multiple times Date: Mon, 1 Aug 2022 07:58:45 -0600 Message-Id: <20220801135848.992449-3-sjg@chromium.org> X-Mailer: git-send-email 2.37.1.455.g008518b4e5-goog In-Reply-To: <20220801135848.992449-1-sjg@chromium.org> References: <20220801135848.992449-1-sjg@chromium.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean Some tests can have race conditions which are hard to detect on a single one. Add a way to run tests more than once, to help with this. Each individual test is run the requested number of times before moving to the next test. If any runs failed, a message is shown. This is most useful when running a single test, since running all tests multiple times can take a while. Signed-off-by: Simon Glass --- arch/sandbox/cpu/spl.c | 2 +- doc/develop/tests_sandbox.rst | 24 ++++++++++++++++++++++++ include/test/test.h | 2 ++ include/test/ut.h | 3 ++- test/cmd_ut.c | 12 ++++++++++-- test/dm/test-dm.c | 13 ++++++++++--- test/test-main.c | 14 +++++++++++--- 7 files changed, 60 insertions(+), 10 deletions(-) diff --git a/arch/sandbox/cpu/spl.c b/arch/sandbox/cpu/spl.c index fe5d44d36ed..1d49a9bd102 100644 --- a/arch/sandbox/cpu/spl.c +++ b/arch/sandbox/cpu/spl.c @@ -89,7 +89,7 @@ void spl_board_init(void) int ret; ret = ut_run_list("spl", NULL, tests, count, - state->select_unittests); + state->select_unittests, 1); /* continue execution into U-Boot */ } } diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 40cf8ecdd7f..8e42a32afb9 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -119,6 +119,30 @@ You can easily use gdb on these tests, without needing --gdbserver:: You can then single-step and look at variables as needed. +Running tests multiple times +---------------------------- + +Some tests can have race conditions which are hard to detect on a single +one. It is possible to run each individual test multiple times, before moving +to the next test, with the '-r' flag. + +This is most useful when running a single test, since running all tests +multiple times can take a while. + +For example:: + + => ut dm -r1000 dm_test_rtc_set_get + ... + Test: dm_test_rtc_set_get: rtc.c (flat tree) + Test: dm_test_rtc_set_get: rtc.c + test/dm/rtc.c:257, dm_test_rtc_reset(): old_base_time == base_time: Expected 0x62e7453c (1659323708), got 0x62e7453d (1659323709) + Test: dm_test_rtc_set_get: rtc.c (flat tree) + Test: dm_test_rtc_set_get: rtc.c + Test: dm_test_rtc_set_get: rtc.c (flat tree) + ... + Test dm_test_rtc_reset failed 3 times + + Running sandbox_spl tests directly ---------------------------------- diff --git a/include/test/test.h b/include/test/test.h index 0104e189f63..66fece48b2a 100644 --- a/include/test/test.h +++ b/include/test/test.h @@ -20,6 +20,7 @@ * @testdev: Test device * @force_fail_alloc: Force all memory allocs to fail * @skip_post_probe: Skip uclass post-probe processing + * @runs_per_test: Number of times to run each test (typically 1) * @expect_str: Temporary string used to hold expected string value * @actual_str: Temporary string used to hold actual string value */ @@ -32,6 +33,7 @@ struct unit_test_state { struct udevice *testdev; int force_fail_alloc; int skip_post_probe; + int runs_per_test; char expect_str[512]; char actual_str[512]; }; diff --git a/include/test/ut.h b/include/test/ut.h index 18740f5807c..f7d1d18f7c1 100644 --- a/include/test/ut.h +++ b/include/test/ut.h @@ -403,9 +403,10 @@ void test_set_state(struct unit_test_state *uts); * @count: Number of tests to run * @select_name: Name of a single test to run (from the list provided). If NULL * then all tests are run + * @runs_per_test: Number of times to run each test (typically 1) * Return: 0 if all tests passed, -1 if any failed */ int ut_run_list(const char *name, const char *prefix, struct unit_test *tests, - int count, const char *select_name); + int count, const char *select_name, int runs_per_test); #endif diff --git a/test/cmd_ut.c b/test/cmd_ut.c index 3789c6b784c..11c219b48ac 100644 --- a/test/cmd_ut.c +++ b/test/cmd_ut.c @@ -18,10 +18,17 @@ int cmd_ut_category(const char *name, const char *prefix, struct unit_test *tests, int n_ents, int argc, char *const argv[]) { + int runs_per_text = 1; int ret; + if (argc > 1 && !strncmp("-r", argv[1], 2)) { + runs_per_text = dectoul(argv[1] + 2, NULL); + argv++; + argc++; + } + ret = ut_run_list(name, prefix, tests, n_ents, - argc > 1 ? argv[1] : NULL); + argc > 1 ? argv[1] : NULL, runs_per_text); return ret ? CMD_RET_FAILURE : 0; } @@ -168,7 +175,8 @@ static char ut_help_text[] = #ifdef CONFIG_CMD_LOADM "ut loadm [test-name]- test of parameters and load memory blob\n" #endif - ; + "All commands accept an optional [-r] flag before [test-name], to\n" + "run each test multiple times ( is in decimal)"; #endif /* CONFIG_SYS_LONGHELP */ U_BOOT_CMD( diff --git a/test/dm/test-dm.c b/test/dm/test-dm.c index f5cda81bbfc..eb3581333b9 100644 --- a/test/dm/test-dm.c +++ b/test/dm/test-dm.c @@ -29,13 +29,14 @@ DECLARE_GLOBAL_DATA_PTR; * "fdt_pre_reloc"), or NULL to run all * Return: 0 if all tests passed, 1 if not */ -static int dm_test_run(const char *test_name) +static int dm_test_run(const char *test_name, int runs_per_text) { struct unit_test *tests = UNIT_TEST_SUITE_START(dm_test); const int n_ents = UNIT_TEST_SUITE_COUNT(dm_test); int ret; - ret = ut_run_list("driver model", "dm_test_", tests, n_ents, test_name); + ret = ut_run_list("driver model", "dm_test_", tests, n_ents, test_name, + runs_per_text); return ret ? CMD_RET_FAILURE : 0; } @@ -43,9 +44,15 @@ static int dm_test_run(const char *test_name) int do_ut_dm(struct cmd_tbl *cmdtp, int flag, int argc, char *const argv[]) { const char *test_name = NULL; + int runs_per_text = 1; + if (argc > 1 && !strncmp("-r", argv[1], 2)) { + runs_per_text = dectoul(argv[1] + 2, NULL); + argv++; + argc++; + } if (argc > 1) test_name = argv[1]; - return dm_test_run(test_name); + return dm_test_run(test_name, runs_per_text); } diff --git a/test/test-main.c b/test/test-main.c index ee38d1faea8..9b19f967499 100644 --- a/test/test-main.c +++ b/test/test-main.c @@ -387,11 +387,17 @@ static int ut_run_tests(struct unit_test_state *uts, const char *prefix, for (test = tests; test < tests + count; test++) { const char *test_name = test->name; - int ret; + int ret, i, old_fail_count; if (!test_matches(prefix, test_name, select_name)) continue; - ret = ut_run_test_live_flat(uts, test, select_name); + old_fail_count = uts->fail_count; + for (i = 0; i < uts->runs_per_test; i++) + ret = ut_run_test_live_flat(uts, test, select_name); + if (uts->fail_count != old_fail_count) { + printf("Test %s failed %d times\n", select_name, + uts->fail_count - old_fail_count); + } found++; if (ret == -EAGAIN) continue; @@ -405,7 +411,8 @@ static int ut_run_tests(struct unit_test_state *uts, const char *prefix, } int ut_run_list(const char *category, const char *prefix, - struct unit_test *tests, int count, const char *select_name) + struct unit_test *tests, int count, const char *select_name, + int runs_per_test) { struct unit_test_state uts = { .fail_count = 0 }; bool has_dm_tests = false; @@ -429,6 +436,7 @@ int ut_run_list(const char *category, const char *prefix, printf("Running %d %s tests\n", count, category); uts.of_root = gd_of_root(); + uts.runs_per_test = runs_per_test; ret = ut_run_tests(&uts, prefix, tests, count, select_name); if (ret == -ENOENT) -- 2.37.1.455.g008518b4e5-goog