From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CC94C433DF for ; Wed, 17 Jun 2020 03:36:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 38F67206F1 for ; Wed, 17 Jun 2020 03:36:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chromium.org header.i=@chromium.org header.b="NcmAqGk+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726568AbgFQDgK (ORCPT ); Tue, 16 Jun 2020 23:36:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726497AbgFQDgJ (ORCPT ); Tue, 16 Jun 2020 23:36:09 -0400 Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C977C06174E for ; Tue, 16 Jun 2020 20:36:09 -0700 (PDT) Received: by mail-pg1-x52b.google.com with SMTP id s135so604295pgs.2 for ; Tue, 16 Jun 2020 20:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=DLJBB3pRoWG5efQHi3veUFRcagcqJ+oWknNXTwAk0o0=; b=NcmAqGk+iRuqRcsBVf4/+Ex0k/oFPhiAYwMxrL0tiAKhUfjO6L9sXlM9BOpAI6QVbu KDVYyPvTr3mHFIUXiuuzh8dTwGEfhsa0pRlk7EZ2vdpybUMt71pw5yS5SLUilMgf9lvA KLYKhZNTSD40IU6tRKvcLbIBTkgWccHUzP0bQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=DLJBB3pRoWG5efQHi3veUFRcagcqJ+oWknNXTwAk0o0=; b=oSVr0KXlortnlaoKZMjUX9mbrXNiIhymICH/evglag6zXk7wGYz/4Cn9wdCpnr73L0 gkgvhfAFUntICi+0Z+93jIavZLQzhnMuSitVeNA4Z3R+Agv2W5nMFH3oEMPIZgzm4RHt FRWI2slPfKALB5yy2/Gj/Mi0YDeggCL1l6fpVVK5cBHwpVgfXz8VgHsNVOpgL65DdBPF 3P4eJY5ApARuklO/gAeDIxZ+V0iOoNni26r4v+zrILO3SXJf+QbW4d/0vlxNvYgc/Bcz Z8GswVl4BVYm10PmN1XKYLrw8NZ8gkydhgHggyZb//Br1do39op0OenWQp+L18HR8nGb E47g== X-Gm-Message-State: AOAM533ezg5UKWp27VbErDiGz0p1Ik06pbBu8fjq70v6Nl1LxUnAM0Cq 08hdmDUmqnurh+vgM731CjoDhw== X-Google-Smtp-Source: ABdhPJzt2IyfSrsCjbnEWsH+80i2Sph0IP1u82KvDONyDI5ns9Z8WN7IKClvSiOHQihmzNQ32Orptg== X-Received: by 2002:a63:778c:: with SMTP id s134mr4407440pgc.273.1592364968589; Tue, 16 Jun 2020 20:36:08 -0700 (PDT) Received: from www.outflux.net (smtp.outflux.net. [198.145.64.163]) by smtp.gmail.com with ESMTPSA id lt14sm3797226pjb.52.2020.06.16.20.36.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Jun 2020 20:36:07 -0700 (PDT) Date: Tue, 16 Jun 2020 20:36:06 -0700 From: Kees Cook To: "Bird, Tim" Cc: Brendan Higgins , "shuah@kernel.org" , "linux-kselftest@vger.kernel.org" , "linux-kernel@vger.kernel.org" , Paolo Bonzini , David Gow Subject: Re: RFC - kernel selftest result documentation (KTAP) Message-ID: <202006162032.9BF6F8F4E@keescook> References: <20200616204817.GA212825@google.com> <202006161703.B2E51605@keescook> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-kselftest-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kselftest@vger.kernel.org On Wed, Jun 17, 2020 at 02:30:45AM +0000, Bird, Tim wrote: > Agreed. You only need machine-parsable data if you expect the CI > system to do something more with the data than just present it. > What that would be, that would be common for all tests (or at least > many test), is unclear. Maybe there are patterns in the diagnostic > data that could lead to higher-level analysis, or even automated > fixes, that don't become apparent if the data is unstructured. But > it's hard to know until you have lots of data. I think just getting > the other things consistent is a good priority right now. Yeah. I think the main place for this is performance analysis, but I think that's a separate system entirely. TAP is really strictly yes/no, where as performance analysis a whole other thing. The only other thing I can think of is some kind of feature analysis, but that would be built out of the standard yes/no output. i.e. if I create a test that checks for specific security mitigation features (*cough*LKDTM*cough*), having a dashboard that shows features down one axis and architectures and/or kernel versions on other axes, then I get a pretty picture. But it's still being built out of the yes/no info. *shrug* I think diagnostic should be expressly non-machine-oriented. -- Kees Cook