From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.2 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8A97CA9EA0 for ; Fri, 18 Oct 2019 15:28:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BAF23222BD for ; Fri, 18 Oct 2019 15:28:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2410507AbfJRP2B (ORCPT ); Fri, 18 Oct 2019 11:28:01 -0400 Received: from outgoing-auth-1.mit.edu ([18.9.28.11]:37286 "EHLO outgoing.mit.edu" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2394233AbfJRP2A (ORCPT ); Fri, 18 Oct 2019 11:28:00 -0400 Received: from callcc.thunk.org (guestnat-104-133-0-98.corp.google.com [104.133.0.98] (may be forged)) (authenticated bits=0) (User authenticated as tytso@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id x9IFRk2g032466 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 18 Oct 2019 11:27:47 -0400 Received: by callcc.thunk.org (Postfix, from userid 15806) id A1830420458; Fri, 18 Oct 2019 11:27:46 -0400 (EDT) Date: Fri, 18 Oct 2019 11:27:46 -0400 From: "Theodore Y. Ts'o" To: Tim.Bird@sony.com Cc: skhan@linuxfoundation.org, brendanhiggins@google.com, yzaikin@google.com, linux-kselftest@vger.kernel.org, linux-ext4@vger.kernel.org, adilger.kernel@dilger.ca, kunit-dev@googlegroups.com Subject: Re: [PATCH linux-kselftest/test v2] ext4: add kunit test for decoding extended timestamps Message-ID: <20191018152746.GF21137@mit.edu> References: <20191011131902.GC16225@mit.edu> <1e6611e6-2fa6-6f7d-bc7f-0bc2243d9342@linuxfoundation.org> <20191017120833.GA25548@mit.edu> <20191017225637.GB6371@mit.edu> <20191018014027.GA21137@mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-ext4-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-ext4@vger.kernel.org On Fri, Oct 18, 2019 at 02:40:50AM +0000, Tim.Bird@sony.com wrote: > We're just talking past each other. My original e-mail was a rebuttal > to your assertion that any test that was data-driven or non-deterministic > was a fuzzer. I still believe that's just not the case. This is independent > of the mechanics or speed of how the data is input. Apologies, I was still focused on the original context of this thread, which was about suggested improvements to Iurii's ext4 kunit test, or perhaps adding new features to Kunit. > I also conceded (multiple times) that externally data-driven > techniques are probably more aptly applied to non-unit tests. I've > heard your pitch about speed, and I'm sympathetic. My point is that > I believe there is a place for data-driven tests. I guess I would put it differently. The key goal is it should be really easy for developers to run, create, and extend tests. Data-driven tests is certainly one technique to make it easier to extend tests, and indeed fs/ext4/inode-test.c is data-driven with the goal to make it easier to add additional tests. Having the data for the test be external is certainly one option, and there will be cases where it will make sense. However, the overhead in creating the parser for the data, and additional complexity required to get the test data to be fed to the test program means that that benefits need to be pretty large in order to balance the additional costs of having an external data file, especially for Kunit. In terms of the abstract question, is there a place for data-driven tests, I'm in complete agreement with you. I've used this many times personally, especially when writing tests which are implemented in terms of shell scripts. Examples of this include e2fsprogs's regression test suite and xfstests. I don't consider that a terribly interesting question though; I view that as on the same order as "is the sky blue?" or "are apple pies yummy?" The more interesting, and more concrete question is whether there is a place for external data-driven tests in Kunit, and there I am *much* more skeptical. Sure, I could imagine adding some infrastructure where user-mode linux could read files from its "host" system. But there are those people who want to run KUnit tests by cross-compiling an ARM kernel and then shipping the resulting kernel to an test board, and then reading the output from the ARM board's serial console. In that case, we can't make the assumption that Kunit tests will be running under user-mode linux, so the complexity of how we get the external data file into the KUnit test just went up by another order of magnitude. It's going to be ***so*** much simpler if the test data is embedded in the kernel built by KUnit. That way, it works both in the ARCH=um case, and in the "build an ARM kernel and push it to a test board" case. Cheers, - Ted