From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECA13C433F5 for ; Wed, 29 Sep 2021 19:54:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CD41A61527 for ; Wed, 29 Sep 2021 19:54:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344161AbhI2T41 (ORCPT ); Wed, 29 Sep 2021 15:56:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237351AbhI2T4Z (ORCPT ); Wed, 29 Sep 2021 15:56:25 -0400 Received: from mail-qv1-xf4a.google.com (mail-qv1-xf4a.google.com [IPv6:2607:f8b0:4864:20::f4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56E78C061764 for ; Wed, 29 Sep 2021 12:54:44 -0700 (PDT) Received: by mail-qv1-xf4a.google.com with SMTP id s10-20020a05621406aa00b003821a333809so8748819qvz.21 for ; Wed, 29 Sep 2021 12:54:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=OiLVv+xlAY408jKup4fnR/QKBpKolunAgpCER+09La0=; b=NVz31hGg+OiDBrChin3fOWkXkIWb3ISmobKjwkzs0XPlc28pv+wuq00CWHagraj773 zo3U/yk6XMMeIMJs2xYUJGGHz0uhr59tErdoYHAezMShRzTpFlOOswzlw/e+lJUMeWu9 +rVzvPIZWHnQnFU173saT2ILUVo9wigRlPWMZxm0ydxYTDLpRMKvGNGelx+4/Hj+CNr0 L+qDrKc5z6C8XdHySjQCvR75HnznMfvEmh6urSK7kzY67YeQE1xK81qNvIYHCDvbTnR8 v6Jugm8TQ1RHqxOpl0sGZHnYB7mSP5couet8WA/K3pmqi7UBDJ/iygBfJ42TJ7VZPJuP d+pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=OiLVv+xlAY408jKup4fnR/QKBpKolunAgpCER+09La0=; b=ttj1Au5/6xiNi+ZKa8fo0YSQ4H96Ml4joMpD6qEuAtPvv2Ys0U8didRba0N27zcd/z I86EhmWg/UhFnH/rMC5Bs8FAYJMMWHJiwxcogjj+JEW9EgtKslDM0mT8r4f3B3tRVJtb zr+7Jhu4M582CObroG/FJbtx2LlAa5h0ayektgNz6YCRVV2Pgb5+LCdsaExF6RZR4zd7 cZ5G7nGl2FRiGImTv/K8KPeY5eB+lmjIEMiLKIL+VED7Ik8pdeTjkc0dPHoZdHGdr9j1 X0ihg/QjAnTxDr1NY7DGUr7VGdgycFFt2hPtXMnl9+jzsDtEC7Daapo//9PbpvRlOx+P 3v2g== X-Gm-Message-State: AOAM533jxeHRK3I2QSj6L32oEHC+qoC/KV+J040KCZ6pL58LWQ2/g7mL M1KMv5WZ8Jaf/LUg5ko0o4YAdJ8w+iq0pQ== X-Google-Smtp-Source: ABdhPJyiigPPp4XqOmZYp9GJQy5+YfZfy+Mm5NDfc+aIOt8KOrP4lLxG7SliycGcOcw426+rX1fOQPhhvOIN8w== X-Received: from dlatypov.svl.corp.google.com ([2620:15c:2cd:202:e473:7cd7:9986:85b7]) (user=dlatypov job=sendgmr) by 2002:a05:6214:1046:: with SMTP id l6mr287892qvr.6.1632945283492; Wed, 29 Sep 2021 12:54:43 -0700 (PDT) Date: Wed, 29 Sep 2021 12:54:32 -0700 Message-Id: <20210929195436.1405996-1-dlatypov@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.33.0.685.g46640cef36-goog Subject: [PATCH v3 0/3] kunit: allow running test suites/cases individually From: Daniel Latypov To: brendanhiggins@google.com, davidgow@google.com Cc: linux-kernel@vger.kernel.org, kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org, skhan@linuxfoundation.org, Daniel Latypov Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Allow running each suite or each test case alone per kernel boot. The motivation for this is to debug "test hermeticity" issues. This new --run_isolated flag would be a good first step to try and narrow down root causes. Context: sometimes tests pass/fail depending on what ran before them. Memory corruption errors in particular might only cause noticeable issues later on. But you can also have the opposite, where "fixing" one test causes another to start failing. Usage: $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=suite $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=test $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit --run_isolated=test example The last one would provide output like ======== [PASSED] example ======== [PASSED] example_simple_test ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 0 skipped. Starting KUnit Kernel (2/3)... ============================================================ ======== [SKIPPED] example ======== [SKIPPED] example_skip_test # SKIP this test should be skipped ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 1 skipped. Starting KUnit Kernel (3/3)... ============================================================ ======== [SKIPPED] example ======== [SKIPPED] example_mark_skipped_test # SKIP this test should be skipped ============================================================ Testing complete. 1 tests run. 0 failed. 0 crashed. 1 skipped. See the last patch's description for a bit more detail. Meta: The first patch is from another series with just a reworded commit message, https://lore.kernel.org/linux-kselftest/20210805235145.2528054-2-dlatypov@google.com/ This patch series is based on Shuah's kunit branch: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/?h=kunit Changes: v1 -> v2: rebase onto Shuah's kunit branch, fix missing code in patch 1. v2 -> v3: fix mypy errors, drop test plan from output, fix pre-existing bug where kunit was not actually tracking test execution time (new patch 3). Daniel Latypov (4): kunit: add 'kunit.action' param to allow listing out tests kunit: tool: factor exec + parse steps into a function kunit: tool: actually track how long it took to run tests kunit: tool: support running each suite/test separately lib/kunit/executor.c | 45 ++++++++- tools/testing/kunit/kunit.py | 129 +++++++++++++++++-------- tools/testing/kunit/kunit_tool_test.py | 40 ++++++++ 3 files changed, 169 insertions(+), 45 deletions(-) base-commit: 3b29021ddd10cfb6b2565c623595bd3b02036f33 -- 2.33.0.685.g46640cef36-goog