git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Jeff King <peff@peff.net>
To: Stefan Beller <sbeller@google.com>
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	git <git@vger.kernel.org>, "Junio C Hamano" <gitster@pobox.com>
Subject: Re: [PATCH 2/4] t5310: test delta reuse with bitmaps
Date: Tue, 4 Sep 2018 16:02:25 -0400	[thread overview]
Message-ID: <20180904200224.GB17481@sigill.intra.peff.net> (raw)
In-Reply-To: <CAGZ79ka8e2-f4fYgy+=HUDdvugefvQ5TnDG0v8YmUn7kGhTdaQ@mail.gmail.com>

On Tue, Sep 04, 2018 at 12:05:58PM -0700, Stefan Beller wrote:

> Yeah, maybe we need to ask for more tests in the 'real' test suite, and not
> just in some special corner (such as t/perf/ or any of the environment
> variable proposals nearby).
> 
> I wonder if we can make use of git.git in the test suite for similar things,
> e.g. after reading the thread about "index corruption with git commit -p" [1],
> I thought that we only have these toy examples in the test suite. Toy examples
> show that the new feature barely works, and doesn't show it is working at scale.

I think the toy examples do both. Often they drill down directly to a
useful but rare corner case that _wouldn't_ be hit during normal
operation. And being toys, they are a lot quicker to set up then trying
to work with a 50MB repository.

Take the "commit -p" one for example. It's not really about the
repository shape but about a particular set of actions. If you don't
test those actions, you won't reproduce the bug.

> > There may be a larger lesson about tracking code coverage, but I don't
> > know that most general code coverage tools would have helped (any
> > overall percentage number would be too large to move). A tool that
> > looked at the diff and said "of the N lines you added/touched, this
> > percent is exercised in the test suite" might have been useful.
> 
> From some offline discussion, maybe we want to adapt a philosophy of
> 
>   Each patch needs to add a test, that fails when the patch
>   is not applied, but succeeds when it is applied. This shows
>   that _some_ code in the patch is exercised at least.
> 
> (and automatically/strongly enforce this going forwards; however
> enforcing such a strict thing is hard, not sure how we'd do it.)

I wouldn't want a hard-and-fast rule like that. If you're fixing a bug,
sure, I think it's good to make sure it's exercised. And if you're
adding a feature, you almost always add some basic tests (and almost
certainly leave some corner without code coverage).

But if you're writing an optimization, there's often no before/after
test.  Presumably it worked before, and hopefully it still works after,
and it just got faster. You're generally relying on existing regression
tests (from when that code was introduced) to save you from bugs. You
might need to _write_ those tests if nobody did before. But it's hard to
know without digging if there are decent tests or not. That's why I
think code coverage of the lines in your diff is probably the best
measure.

-Peff

  parent reply	other threads:[~2018-09-04 20:02 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-08-21 18:41 [PATCH] test-tool.h: include git-compat-util.h Jeff King
2018-08-21 19:03 ` Junio C Hamano
2018-08-21 19:06 ` [PATCH 1/6] t/perf: factor boilerplate out of test_perf Jeff King
2018-08-21 19:06 ` [PATCH 2/6] t/perf: factor out percent calculations Jeff King
2018-08-21 19:06 ` [PATCH 3/6] t/perf: add infrastructure for measuring sizes Jeff King
2018-08-22 13:40   ` Derrick Stolee
2018-08-22 15:31     ` Jeff King
2018-08-21 19:06 ` [PATCH 4/6] t/perf: add perf tests for fetches from a bitmapped server Jeff King
2018-08-21 19:07 ` [PATCH 5/6] pack-bitmap: save "have" bitmap from walk Jeff King
2018-08-21 19:47   ` Derrick Stolee
2018-08-21 19:54     ` Jeff King
2018-08-31 15:23   ` Ævar Arnfjörð Bjarmason
2018-08-31 22:55     ` Jeff King
2018-09-01  7:41       ` [PATCH 0/4] un-breaking pack-objects with bitmaps Jeff King
2018-09-01  7:44         ` [PATCH 1/4] bitmap_has_sha1_in_uninteresting(): drop BUG check Jeff King
2018-09-01  7:48         ` [PATCH 2/4] t5310: test delta reuse with bitmaps Jeff King
2018-09-01  8:03           ` Jeff King
2018-09-01 20:29             ` Ævar Arnfjörð Bjarmason
2018-09-01 22:46               ` Ben Peart
2018-09-02  5:51               ` Jeff King
2018-09-04 19:05             ` Stefan Beller
2018-09-04 19:45               ` Junio C Hamano
2018-09-04 20:02               ` Jeff King [this message]
2018-09-01  7:49         ` [PATCH 3/4] traverse_bitmap_commit_list(): don't free result Jeff King
2018-09-01  7:50         ` [PATCH 4/4] pack-bitmap: drop "loaded" flag Jeff King
2018-09-04 19:30         ` [PATCH 0/4] un-breaking pack-objects with bitmaps Stefan Beller
2018-09-04 20:03           ` Jeff King
2018-09-08  6:43         ` Ævar Arnfjörð Bjarmason
2018-09-10 16:53           ` Junio C Hamano
2018-09-10 18:48             ` Jeff King
2018-09-10 19:23               ` Junio C Hamano
2018-08-21 19:07 ` [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects Jeff King
2018-08-21 19:43   ` Junio C Hamano
2018-08-21 19:50     ` Junio C Hamano
2018-08-21 20:07       ` Jeff King
2018-08-21 20:14         ` Jeff King
2018-08-21 20:52           ` Junio C Hamano
2018-08-21 21:30             ` Jeff King
2018-08-21 20:57         ` Junio C Hamano
2018-08-21 21:32           ` Jeff King
2018-08-23  0:43           ` [PATCH 0/9] trailer-parsing false positives Jeff King
2018-08-23  0:44             ` [PATCH 1/9] trailer: use size_t for string offsets Jeff King
2018-08-23  0:45             ` [PATCH 2/9] trailer: use size_t for iterating trailer list Jeff King
2018-08-23  0:46             ` [PATCH 3/9] trailer: pass process_trailer_opts to trailer_info_get() Jeff King
2018-08-23  0:48             ` [PATCH 4/9] interpret-trailers: tighten check for "---" patch boundary Jeff King
2018-08-23  0:49             ` [PATCH 5/9] interpret-trailers: allow suppressing "---" divider Jeff King
2018-08-23  0:50             ` [PATCH 6/9] pretty, ref-filter: format %(trailers) with no_divider option Jeff King
2018-08-23  0:50             ` [PATCH 7/9] sequencer: ignore "---" divider when parsing trailers Jeff King
2018-08-23  0:50             ` [PATCH 8/9] append_signoff: use size_t for string offsets Jeff King
2018-08-23  0:51             ` [PATCH 9/9] sequencer: handle ignore_footer when parsing trailers Jeff King
2018-08-23 18:30             ` [PATCH 0/9] trailer-parsing false positives Junio C Hamano
2018-08-24  7:26               ` Jeff King
2018-08-21 20:00     ` [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects Jeff King

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180904200224.GB17481@sigill.intra.peff.net \
    --to=peff@peff.net \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=sbeller@google.com \
    --subject='Re: [PATCH 2/4] t5310: test delta reuse with bitmaps' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).