From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36E80C47254 for ; Tue, 5 May 2020 08:13:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1589120675 for ; Tue, 5 May 2020 08:13:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linaro.org header.i=@linaro.org header.b="Ds/TGo88" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728401AbgEEINQ (ORCPT ); Tue, 5 May 2020 04:13:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726551AbgEEINP (ORCPT ); Tue, 5 May 2020 04:13:15 -0400 Received: from mail-lj1-x241.google.com (mail-lj1-x241.google.com [IPv6:2a00:1450:4864:20::241]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72579C0610D5 for ; Tue, 5 May 2020 01:13:15 -0700 (PDT) Received: by mail-lj1-x241.google.com with SMTP id e25so661754ljg.5 for ; Tue, 05 May 2020 01:13:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cCtbnn/mPSYMTovrACT2kKrxnwGdokswItkXqrNBN20=; b=Ds/TGo88InOgXEcC2Gt5/VgiFUU8yBoN4mf50voWf0ReQi8v5PKgYBXhp/82AjMhqB rTJsuC5+jFz4XMfzl/6020xwtmeCq6QkJs2++a/n+/kMYDSpiRAHYh1yF8UfJO5L7UwR 2DAf67oLsyBJ5mLJtL91w5TAlbn6wdGv9X5gfILWFZD9RbWmIJkxZ8WXX6BiBOpJ4ZZs H0UY8zhoYXsuMzNUNAukprACSGCpQMCw7tvWQYxBu6+4K9+5sgqLkPHgIEuklCZyYvqd iSoCM2UNPS79xcfwFMFQgG3A2znyPS1q8ogklxD926Qj892yKoqT7ioU+9X6DS6s78N4 KFkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cCtbnn/mPSYMTovrACT2kKrxnwGdokswItkXqrNBN20=; b=Ln+jeXjeFOQgL4S5pc3V9KfnKjzE2RpNAZjESodYEVppnr3BKejZjPCz7zTWm7A9pf 8hFYB2F2R7e7/NuXrs3CG0Q8QdFYcndlgFcYu2Defq7nAL+hQ8u3P4uPuiE0UuRVx6R5 N4fYKTGguzZNAL2/KUiOvY47fjb1BHrDEZ6eH12jFbTOSm8Y7e+CP+TBVJ2UJOnK4Ltm dzUz+wlF7qSHMQ/82Sv2saRlVV2sZkBcXXQcA3p9LNQJaXFiueDXIp66KM+e3OYo6NpW XNqEs85vqhmNEdY7HNPJx8bm5DzY8ZtrS/B4NCFQa7KWvuUfIfgxlEiv0hGy/xOwVzJE t+hQ== X-Gm-Message-State: AGi0PubbAgvt6zzEwvNkPWrrpzQYorAp111TMTIzAc5IUFzcIcnT4eUH ZXAYE29Mc4HQSz9db2cmPoiJ0OD+oxVrc3HF/4AOfg== X-Google-Smtp-Source: APiQypJUpeM7dR82Dn4AY7wcHcK7ejTccxr/K+9YNhjOMoBLs1Z8eehQdWNAzTelKtxHX1iwrzM/YGhrn32nrSdwdS8= X-Received: by 2002:a2e:6a08:: with SMTP id f8mr1135471ljc.8.1588666393650; Tue, 05 May 2020 01:13:13 -0700 (PDT) MIME-Version: 1.0 References: <20200501083510.1413-1-anders.roxell@linaro.org> In-Reply-To: From: Anders Roxell Date: Tue, 5 May 2020 10:13:02 +0200 Message-ID: Subject: Re: [PATCH] kunit: Kconfig: enable a KUNIT_RUN_ALL fragment To: David Gow Cc: Brendan Higgins , Greg KH , "Theodore Ts'o" , adilger.kernel@dilger.ca, Marco Elver , John Johansen , James Morris , "Serge E. Hallyn" , Linux Kernel Mailing List , linux-ext4@vger.kernel.org, kasan-dev , "open list:KERNEL SELFTEST FRAMEWORK" , KUnit Development , linux-security-module Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, 2 May 2020 at 04:11, David Gow wrote: > > On Sat, May 2, 2020 at 4:31 AM Brendan Higgins > wrote: > > > > On Fri, May 1, 2020 at 1:35 AM Anders Roxell wrote: > > > > > > Make it easier to enable all KUnit fragments. This is needed for kernel > > > test-systems, so its easy to get all KUnit tests enabled and if new gets > > > added they will be enabled as well. Fragments that has to be builtin > > > will be missed if CONFIG_KUNIT_RUN_ALL is set as a module. > > > > > > Adding 'if !KUNIT_RUN_ALL' so individual test can be turned of if > > > someone wants that even though KUNIT_RUN_ALL is enabled. > > > > I would LOVE IT, if you could make this work! I have been trying to > > figure out the best way to run all KUnit tests for a long time now. > > > > That being said, I am a bit skeptical that this approach will be much > > more successful than just using allyesconfig. Either way, there are > > tests coming down the pipeline that are incompatible with each other > > (the KASAN test and the KCSAN test will be incompatible). Even so, > > tests like the apparmor test require a lot of non-default > > configuration to compile. In the end, I am not sure how many tests we > > will really be able to turn on this way. > > > > Thoughts? > > I think there's still some value in this which the allyesconfig option > doesn't provide. As you point out, it's not possible to have a generic > "run all tests" option due to potential conflicting dependencies, but > this does provide a way to run all tests for things enabled in the > current config. This could be really useful for downstream developers > who want a way of running all tests relevant to their config without > the overhead of running irrelevant tests (e.g., for drivers they don't > build). It will solve that as well as for a tester doesn't have to go through all KUnit tests fragments to turn them on. > Using allyesconfig doesn't make that distinction. We could also create a config fragment file in kernel/configs/kunit.config where we set ------start CONFIG_KUNIT=y CONFIG_KUNIT_RUN_ALL=y CONFIG_SECURITY_APPARMOR=y ------end So, these two can only be enabled if KUNIT=y CONFIG_KUNIT_DRIVER_PE_TEST=y CONFIG_PM_QOS_KUNIT_TEST=y and for this one we have a pre-request of SECURITY_APPARMOR=y CONFIG_SECURITY_APPARMOR_KUNIT_TEST=y Other tests solves the dependencies with 'select' like CONFIG_EXT4_KUNIT_TESTS, that adds this row in fs/ext4/Kconfig, 'select EXT4_FS' But I think we should try to minimize the number of 'select' statements, in order to avoid circular dependencies and unexpected behaviours. Maybe we should add the CONFIG_EXT4_FS=y into the kunit.config file instead ? > > Ultimately, we'll probably still want something which enables a > broader set of tests for upstream development: whether that's based on > this, allyesconfig, or something else entirely remains to be seen, I > think. I suspect we're going to end up with something > subsystem-specific (having a kunitconfig per subsystem, or a testing > line in MAINTAINERS or similar are ideas which have been brought up in > the past). > > This is a great looking tool to have in the toolbox, though. I agree! I'll prepare a patchset with individual patches as was suggested by Marco shortly. Cheers, Anders