From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70AD4C433FE for ; Fri, 18 Mar 2022 00:34:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231205AbiCRAfb (ORCPT ); Thu, 17 Mar 2022 20:35:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229499AbiCRAfa (ORCPT ); Thu, 17 Mar 2022 20:35:30 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0301231939 for ; Thu, 17 Mar 2022 17:34:11 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id m42-20020a05600c3b2a00b00382ab337e14so5898671wms.3 for ; Thu, 17 Mar 2022 17:34:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C0uroXwvLwFDNfnRfL3JRvpsqoCFeg89n7g8kFyVFB4=; b=bAdZ6/TRSHnY1+l03d/HyPkWLAR13MAFl1mymoWJJ6hx5RkPwAPgBvQO7uFsJV5xhM qafg3KiBQCPrqff1C46wY6nggfEteAFHF15RZ8d3p8u+YcyJ2Jan4Be95w7HsQQdHBkH cJ4McWvgNo/ZpKEGJv81/mIWksTO4wofb35IPQHv0sSHcCC48nbo99uiBA68+d3RhCvW 2/MqOFmmJzPSsoYjMja5PHeVPsgCkroBpoZHhLBiXWlnVVtgufseIKnJYAeyICUh8/gB oJ/dZWRIv85waaQTAC01ZEJXV/WbRqBwn11KGnN2z51zBNxBpDkXN/SvVQclIHjNJHOm sbIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C0uroXwvLwFDNfnRfL3JRvpsqoCFeg89n7g8kFyVFB4=; b=uYvqM2732Ufgz0klz7cmCMWsjQQsf1XxEX2WXjYi/aAcH9zfKolm+TdpCNQaE0q2tT 2xG5/ik4Wyw4d1ysbubgfZC1lF39Vi/SDTkiVPg+DG1RhhItoLdIVrPknMf0Y1kaIz/g CZu46S0y65Y+tgiKPRI8qExltWN6xzmz0mZ/ppAOqdrazzslaecL4Cu6VNQh+ICFN9d4 9CWvjpeVuPv8HocL++HbZB47ONUA4tKydfQwgH24+K7YecTGnyeSlH+YB2nCs0tSR08w O/dXTZgMJuS/P0IZiipvKVBvbpmCSlUY0Ijqa6Tq4cErBce6GAsOmn90TM7hlNhEMf5o Ebbw== X-Gm-Message-State: AOAM531086/oTorrLzddG5QuDv2kESKZ95prtl7dJwvXbZAkH5KxSPJy V/7kwRNeuQ78QhZqDs8eDYA7RIu8mzDifw== X-Google-Smtp-Source: ABdhPJwKkG/GbjFLMYIiBw2AEvIb11TKELC7Wfqrc9ytog4ItRkkpycxrhY0jtUULm68kp/amxZLfA== X-Received: by 2002:a7b:cc12:0:b0:37c:1ae:100a with SMTP id f18-20020a7bcc12000000b0037c01ae100amr13789784wmh.54.1647563650130; Thu, 17 Mar 2022 17:34:10 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id l4-20020adff484000000b00203dde8d82bsm5096548wro.88.2022.03.17.17.34.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Mar 2022 17:34:09 -0700 (PDT) From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= To: git@vger.kernel.org Cc: Junio C Hamano , John Cai , Elijah Newren , Derrick Stolee , =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Subject: [PATCH 1/7] test-lib: add a "test_expect_todo", similar to "test_expect_failure" Date: Fri, 18 Mar 2022 01:33:56 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1436.g756b814e59f In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add an alternative to the "test_expect_failure" function. Like "test_expect_failure" it will marks a test as "not ok ... TODO" in the TAP output, and thus document that it's a "TODO" test that fails currently. Unlike "test_expect_failure" it asserts that the tested code in exactly one manner, and not any other. We'll thus stop conflating e.g. segfaults with other sorts of errors, and generally be able to tell if our "expected to fail" tests start failing in new and unexpected ways. In more detail, the these problems of "test_expect_failure" are solved by this new function. * When "test_expect_failure" is used in combination with "test_{must,might}_fail" it will hide segfaults or abort() failures, such as failures due to LSAN. This is because we do do the right thing in those helpers, but they themselves are expected to be used within "test_expect_success" and return 1. The "test_expect_failure" can't differentiate this from a "real" failure. We could change "test_{must,might}_fail" to appropriately carry forward the status code to work around this specific issue, but doing so would be a large semantic change in the current test suite. * More generally, "test_expect_failure" by design doesn't test what does, but just asserts that the test fails in some way. For some tests that truly fail in unpredictable ways this behavior makes sense, but it's almost never what our tests want. We know we e.g. have a revision at HEAD~ instead of HEAD, and would like to know conflate that with a potential segfault or other behavior change. In summary, this new function behaves exactly like "test_expect_success", except that its "success" is then reported as a "not ok .. TODO". Let's convert a few "test_expect_failure" uses to the new "test_expect_todo". For previous discussion on this topic see [1] and [2], in particular the points by Junio that it's desired that the "test_expect_failure" output differentiate the TODO tests from "test_expect_success". 1. https://lore.kernel.org/git/87tuhmk19c.fsf@evledraar.gmail.com/ 2. https://lore.kernel.org/git/xmqqo824e145.fsf@gitster.g/ Signed-off-by: Ævar Arnfjörð Bjarmason --- t/README | 20 +++++++++++++++++++- t/t0000-basic.sh | 11 ++++++++--- t/t1060-object-corruption.sh | 4 ++-- t/t7815-grep-binary.sh | 4 ++-- t/test-lib-functions.sh | 28 ++++++++++++++++++++++++---- t/test-lib.sh | 32 ++++++++++++++++++++++++++++---- 6 files changed, 83 insertions(+), 16 deletions(-) diff --git a/t/README b/t/README index f48e0542cdc..e0aa8ebb0eb 100644 --- a/t/README +++ b/t/README @@ -805,10 +805,28 @@ see test-lib-functions.sh for the full list and their options. test_expect_success PERL,PYTHON 'yo dawg' \ ' test $(perl -E 'print eval "1 +" . qx[python -c "print 2"]') == "4" ' + - test_expect_todo []