From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kees Cook Subject: Re: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests Date: Mon, 6 May 2019 10:43:55 -0700 Message-ID: References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Return-path: In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> Sender: linux-kernel-owner@vger.kernel.org To: Frank Rowand , Brendan Higgins , Shuah Khan Cc: Greg KH , Kieran Bingham , Luis Chamberlain , Rob Herring , Stephen Boyd , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , "linux-fsdevel@vger.kernel.org" , linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir List-Id: linux-nvdimm@lists.01.org On Sun, May 5, 2019 at 5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_PASS,T_DKIMWL_WL_MED,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 442E9C04AAD for ; Mon, 6 May 2019 17:44:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1ACFC2082F for ; Mon, 6 May 2019 17:44:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="m4Zw2fBY" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726756AbfEFRoJ (ORCPT ); Mon, 6 May 2019 13:44:09 -0400 Received: from mail-vs1-f65.google.com ([209.85.217.65]:32812 "EHLO mail-vs1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726428AbfEFRoI (ORCPT ); Mon, 6 May 2019 13:44:08 -0400 Received: by mail-vs1-f65.google.com with SMTP id z145so8675921vsc.0 for ; Mon, 06 May 2019 10:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OeGgjZBJ9uWD1pdsX/+zOfgOjAibZGBp9WPaU0eToUc=; b=m4Zw2fBY+VMLzhBfTBbz4Dtlq0Cc6l6MqEwoJ7YTjrYPzeO8QeIYW3yIFLyCL117Kt axxocOSPh/a+gWkz5TWdPKqb2BPRt/jtybJOfLKxxemlLxpId3LZKDrEsHYJlkXqRfwz fCTrKPtDSaBjY4auU4otyGauXVMtlpMTLmviNmwp8xF+tgZuQMZPsaKxN5HR6oBL5vv2 Zb040i1FzkyXpGkN4I4IP7WFVE6iEZwE10S2R1vSIzVYWL40r/PqrLJElx8kQUpZMdyS 9343jPbZPHbXIDd2SJlyBmLnUf59HZ8PtlBE/IOAyuF/EUfnmI/1F3+fvCafLr94a9B9 gs7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OeGgjZBJ9uWD1pdsX/+zOfgOjAibZGBp9WPaU0eToUc=; b=BOuW8QUf3wvVOopzxiV8NwY2R4LWntbzc5QXkqz+yecsWm8OtnnaPLOm371sZjk/Hi QC8FK/T5VTUf6qHldz4W26CWkiRtWf506ibRcBYjKftYfcr55hUrMJIpLXI7PMU+o31c 6uY3UB2LbUomfc6pYuGHhRLWEedNQQsF1Xxz5Hu336IhDGAspLT2YnBjQw0YnL6GcdNv McPp//aX67TBIJ/xgEJJl0qlrCm9dvjUYPUGMglh0nOA3ezjVn91vU1ayGLhDghWJ8EG sv0jylSMOAO+EFrH55yjNjeezD0DRkY6u9OPxKDEaQ45wppTkJcs3ejYPngN+BKsA98W zQsA== X-Gm-Message-State: APjAAAW7l8eY71oH3grwkekUroKv0LJB6GsGdzMC0gwpqErW7rcnWDQI N+6GWwfBOjm1Wm+4q8h71x4xh521KWnV7RtgDRbO8A== X-Google-Smtp-Source: APXvYqxj9ho/3KigVFISlPQqfamyJ4vl1xCgP8i9j8q0ibA/ApDLHmmjNXKUf72ouvJJD/44drdXvLamKur++kGc2Rs= X-Received: by 2002:a67:dd95:: with SMTP id i21mr8883021vsk.48.1557164647231; Mon, 06 May 2019 10:44:07 -0700 (PDT) MIME-Version: 1.0 References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> From: Kees Cook Date: Mon, 6 May 2019 10:43:55 -0700 Message-ID: Subject: Re: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests To: Frank Rowand , Brendan Higgins , Shuah Khan Cc: Greg KH , Kieran Bingham , Luis Chamberlain , Rob Herring , Stephen Boyd , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , "linux-fsdevel@vger.kernel.org" , linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Dan Williams , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com, Felix Guo Content-Type: text/plain; charset="UTF-8" Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sun, May 5, 2019 at 5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook From mboxrd@z Thu Jan 1 00:00:00 1970 From: keescook at google.com (Kees Cook) Date: Mon, 6 May 2019 10:43:55 -0700 Subject: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> Message-ID: On Sun, May 5, 2019 at 5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook From mboxrd@z Thu Jan 1 00:00:00 1970 From: keescook@google.com (Kees Cook) Date: Mon, 6 May 2019 10:43:55 -0700 Subject: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> Message-ID: Content-Type: text/plain; charset="UTF-8" Message-ID: <20190506174355.5MayLn720WQXp2eRZu8yuvHLoS_f9naFuDrEAwT47V0@z> On Sun, May 5, 2019@5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-f68.google.com ([209.85.217.68]:45653 "EHLO mail-vs1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726406AbfEFRoI (ORCPT ); Mon, 6 May 2019 13:44:08 -0400 Received: by mail-vs1-f68.google.com with SMTP id o10so8647895vsp.12 for ; Mon, 06 May 2019 10:44:07 -0700 (PDT) MIME-Version: 1.0 References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> From: Kees Cook Date: Mon, 6 May 2019 10:43:55 -0700 Message-ID: Subject: Re: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests Content-Type: text/plain; charset="UTF-8" Sender: linux-kbuild-owner@vger.kernel.org List-ID: To: Frank Rowand , Brendan Higgins , Shuah Khan Cc: Greg KH , Kieran Bingham , Luis Chamberlain , Rob Herring , Stephen Boyd , devicetree , dri-devel , kunit-dev@googlegroups.com, "open list:DOCUMENTATION" , "linux-fsdevel@vger.kernel.org" , linux-kbuild , Linux Kernel Mailing List , "open list:KERNEL SELFTEST FRAMEWORK" , linux-nvdimm , linux-um@lists.infradead.org, Sasha Levin , "Bird, Timothy" , Amir Goldstein , Dan Carpenter , Dan Williams , Daniel Vetter , Jeff Dike , Joel Stanley , Julia Lawall , Kevin Hilman , Knut Omang , Logan Gunthorpe , Michael Ellerman , Petr Mladek , Richard Weinberger , David Rientjes , Steven Rostedt , wfg@linux.intel.com, Felix Guo On Sun, May 5, 2019 at 5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-vs1-xe42.google.com ([2607:f8b0:4864:20::e42]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1hNhet-0005G1-Sh for linux-um@lists.infradead.org; Mon, 06 May 2019 17:44:13 +0000 Received: by mail-vs1-xe42.google.com with SMTP id r62so1867798vsc.9 for ; Mon, 06 May 2019 10:44:08 -0700 (PDT) MIME-Version: 1.0 References: <20190501230126.229218-1-brendanhiggins@google.com> <20190501230126.229218-13-brendanhiggins@google.com> <20190502110220.GD12416@kroah.com> <1a5f3c44-9fa9-d423-66bf-45255a90c468@gmail.com> <052fa196-4ea9-8384-79b7-fe6bacc0ee82@gmail.com> <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> In-Reply-To: <63f63c7c-6185-5e64-b338-6a5e7fb9e27c@gmail.com> From: Kees Cook Date: Mon, 6 May 2019 10:43:55 -0700 Message-ID: Subject: Re: [PATCH v2 12/17] kunit: tool: add Python wrappers for running KUnit tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-um" Errors-To: linux-um-bounces+geert=linux-m68k.org@lists.infradead.org To: Frank Rowand , Brendan Higgins , Shuah Khan Cc: Petr Mladek , "open list:DOCUMENTATION" , Amir Goldstein , dri-devel , Sasha Levin , "open list:KERNEL SELFTEST FRAMEWORK" , Rob Herring , linux-nvdimm , Kevin Hilman , Knut Omang , Kieran Bingham , Felix Guo , wfg@linux.intel.com, Joel Stanley , David Rientjes , Jeff Dike , Dan Carpenter , devicetree , linux-kbuild , "Bird, Timothy , linux-um@lists.infradead.org, Steven Rostedt" , Julia Lawall , Dan Williams , kunit-dev@googlegroups.com, Richard Weinberger , Stephen Boyd , Greg KH , Linux Kernel Mailing List , Luis Chamberlain , Daniel Vetter , Michael Ellerman , "linux-fsdevel@vger.kernel.org" , Logan Gunthorpe On Sun, May 5, 2019 at 5:19 PM Frank Rowand wrote: > You can see the full version 14 document in the submitter's repo: > > $ git clone https://github.com/isaacs/testanything.github.io.git > $ cd testanything.github.io > $ git checkout tap14 > $ ls tap-version-14-specification.md > > My understanding is the the version 14 specification is not trying to > add new features, but instead capture what is already implemented in > the wild. Oh! I didn't know about the work on TAP 14. I'll go read through this. > > ## Here is what I propose for this patchset: > > > > - Print out test number range at the beginning of each test suite. > > - Print out log lines as soon as they happen as diagnostics. > > - Print out the lines that state whether a test passes or fails as a > > ok/not ok line. > > > > This would be technically conforming with TAP13 and is consistent with > > what some kselftests have done. This is what I fixed kselftest to actually do (it wasn't doing correct TAP13), and Shuah is testing the series now: https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=ksft-tap-refactor I'll go read TAP 14 now... -- Kees Cook _______________________________________________ linux-um mailing list linux-um@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-um