From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, USER_AGENT_MUTT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE1C7C10F04 for ; Thu, 14 Feb 2019 21:28:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BB93D2147C for ; Thu, 14 Feb 2019 21:28:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=osandov-com.20150623.gappssmtp.com header.i=@osandov-com.20150623.gappssmtp.com header.b="K4YVpkwd" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2440162AbfBNV2m (ORCPT ); Thu, 14 Feb 2019 16:28:42 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:45500 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2440158AbfBNV2m (ORCPT ); Thu, 14 Feb 2019 16:28:42 -0500 Received: by mail-pl1-f194.google.com with SMTP id r14so3791779pls.12 for ; Thu, 14 Feb 2019 13:28:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osandov-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Rw2+D7+us+wJqdcSNjpNfeFzsqQkBVrdgzooPvFEdxI=; b=K4YVpkwdteoladCHicIieFXiZ7fB+i90feP85dmyIF5dcABOwDVuSTpQ1b9kRGlJzo eaIG0S5wDpd/6wXcwnoF0Th1mcoDt8M4yxjtVNg/sEsPLiSSfQSdAUR6Ep9jtLXZStSz iqapTyleGfw+ouwzYdZnwj12qCCF/0jEZH2RxbP7k7AL+PUR0lQYVRnN9Bb+0jetMOl6 XnH3/nLP13UiLIbByo/IuMKItcFQjLMk+GxNO4XCq2ebdsGtZBY2TVkgs2jyiZEOs31u hIgt4gpjppGYZrQBgz1HLn8sjY1FQ+0UKrqPSoTUlJQ0EZZRwje3yFSkE6o3sAnsNB+Z 6Nsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Rw2+D7+us+wJqdcSNjpNfeFzsqQkBVrdgzooPvFEdxI=; b=WgTu/zREj1nyMboQamVeP5JwimMkbs/D80gtehrNfRbc65zg0yGKJutMTh6O74CKdF ktpdA4DJFmGAuTcii1fHnA7OdMKWoI0I5HbWXePag4VtjRhi5gmPxb+SbSEhzUcdnXn5 ePSb55LPt3L15ikJimA7ngR1VWIjPod4YBsLQLhlFRQGmSq5SOyqnq9d4Ws9NDQh8WYx KuX+vuQTOEE3GrrpcGxHTu27GupGV7oI0ZQM3iywtTipNQpiqBMURjE13XUZMn/tIhZ5 I/ELeRl0pvYsVUPdJ/wnFy7DBq+T7U0qKNA9X9QHAaosHUDTX493BFTgmqQrf33hRuJc /vZQ== X-Gm-Message-State: AHQUAuYdE6nYVM4+uU9TLFLBEPV1DF/gcPNqcbU50PtfUPpiVjnB7Afx LPxWjKltL25ya8OwgGdZo+qwkg== X-Google-Smtp-Source: AHgI3IY6rX7RIIlCeCrNvJREsSaqZdL3ScFlFbtTs2q9LgS0Lj1DfIS44AkoSz5E/2wFsLKWPbKfuQ== X-Received: by 2002:a17:902:583:: with SMTP id f3mr6565146plf.202.1550179720276; Thu, 14 Feb 2019 13:28:40 -0800 (PST) Received: from vader ([2620:10d:c090:200::7:c363]) by smtp.gmail.com with ESMTPSA id c17sm4103576pgg.0.2019.02.14.13.28.39 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 14 Feb 2019 13:28:39 -0800 (PST) Date: Thu, 14 Feb 2019 13:28:38 -0800 From: Omar Sandoval To: Lukas Czerner Cc: "Theodore Y. Ts'o" , lsf-pc@lists.linux-foundation.org, linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: [LSF/MM TOPIC] improving storage testing Message-ID: <20190214212838.GC9819@vader> References: <20190213180754.GX23000@mit.edu> <20190214121040.vckivyo5qcp6iyc4@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20190214121040.vckivyo5qcp6iyc4@localhost.localdomain> User-Agent: Mutt/1.11.3 (2019-02-01) Sender: linux-fsdevel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-fsdevel@vger.kernel.org On Thu, Feb 14, 2019 at 01:10:40PM +0100, Lukas Czerner wrote: > On Wed, Feb 13, 2019 at 01:07:54PM -0500, Theodore Y. Ts'o wrote: > > > > 2) Documenting what are known failures should be for various tests on > > different file systems and kernel versions. I think we all have our > > own way of excluding tests which are known to fail. One extreme case > > is where the test case was added to xfstests (generic/484), but the > > patch to fix it got hung up because it was somewhat controversial, so > > it was failing on all file systems. > > > > Other cases might be when fixing a particular test failure is too > > complex to backport to stable (maybe because it would drag in all > > sorts of other changes in other subsystems), so that test is Just > > Going To Fail for a particular stable kernel series. > > > > It probably doesn't make sense to do this in xfstests, which is why we > > all have our own individual test runners that are layered on top of > > xfstests. But if we want to automate running xfstests for stable > > kernel series, some way of annotating fixes for different kernel > > versions would be useful, perhaps some kind of centralized clearing > > house of this information would be useful. > > I think that the first step can be to require the new test to go in > "after" the respective kernel fix. And related to that, require the test > to include a well-defined tag (preferably both in the test itself and > commit description) saying which commit fixed this particular problem. For blktests, I require that regression tests include what commit they are testing in the test comment. For xfstests, sometimes the test mentions it, sometimes the commit mentions it, but more often you have to search for keywords in kernel commit messages... It'd be great if xfstests also required that the test file mentioned the commit/patch it tests.