From: Stefan Beller <email@example.com> To: Jeff King <firstname.lastname@example.org>, Johannes Schindelin <Johannes.Schindelin@gmx.de> Cc: "Ævar Arnfjörð Bjarmason" <email@example.com>, git <firstname.lastname@example.org>, "Junio C Hamano" <email@example.com> Subject: Re: [PATCH 2/4] t5310: test delta reuse with bitmaps Date: Tue, 4 Sep 2018 12:05:58 -0700 [thread overview] Message-ID: <CAGZ79ka8e2-f4fYgy+=HUDdvugefvQ5TnDG0v8YmUn7kGhTdaQ@mail.gmail.com> (raw) In-Reply-To: <20180901080316.GA25852@sigill.intra.peff.net> On Sat, Sep 1, 2018 at 1:03 AM Jeff King <firstname.lastname@example.org> wrote: > > On Sat, Sep 01, 2018 at 03:48:13AM -0400, Jeff King wrote: > > > Commit 6a1e32d532 (pack-objects: reuse on-disk deltas for > > thin "have" objects, 2018-08-21) taught pack-objects a new > > optimization trick. Since this wasn't meant to change > > user-visible behavior, but only produce smaller packs more > > quickly, testing focused on t/perf/p5311. > > > > However, since people don't run perf tests very often, we > > should make sure that the feature is exercised in the > > regular test suite. This patch does so. > > This, by the way, is the crux of how such an obvious and severe bug made > it to 'next'. > > The original series was tested quite extensively via t/perf and in > production at GitHub. When I re-rolled v2, the only change was the > addition of the assertion, so I didn't bother re-doing the perf tests, > since they're slow and there wouldn't be a measurable impact. > > I did run the normal test suite (as I'm sure Junio did, too) as a > double-check for correctness, but as we noticed, the code wasn't > actually exercised there. > > Nor had I yet backported the revised series to the version we run at > GitHub, so it hadn't been run there, either. > > And all of that coupled with the fact that it only triggers with > bitmaps, so day-to-day use of the buggy Git (like Junio trying to push > out the result ;) ) wouldn't show it. > > Anyway. Not that exciting, and kind of obviously dumb in retrospect. But > I think it was worth analyzing to see what went wrong. If there's an > immediate lesson, it is probably: add tests even for changes that aren't > really user-visible to make sure the code is exercised. Yeah, maybe we need to ask for more tests in the 'real' test suite, and not just in some special corner (such as t/perf/ or any of the environment variable proposals nearby). I wonder if we can make use of git.git in the test suite for similar things, e.g. after reading the thread about "index corruption with git commit -p" , I thought that we only have these toy examples in the test suite. Toy examples show that the new feature barely works, and doesn't show it is working at scale.  https://email@example.com/ > There may be a larger lesson about tracking code coverage, but I don't > know that most general code coverage tools would have helped (any > overall percentage number would be too large to move). A tool that > looked at the diff and said "of the N lines you added/touched, this > percent is exercised in the test suite" might have been useful. From some offline discussion, maybe we want to adapt a philosophy of Each patch needs to add a test, that fails when the patch is not applied, but succeeds when it is applied. This shows that _some_ code in the patch is exercised at least. (and automatically/strongly enforce this going forwards; however enforcing such a strict thing is hard, not sure how we'd do it.) Stefan
next prev parent reply other threads:[~2018-09-04 19:06 UTC|newest] Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-08-21 18:41 [PATCH] test-tool.h: include git-compat-util.h Jeff King 2018-08-21 19:03 ` Junio C Hamano 2018-08-21 19:06 ` [PATCH 1/6] t/perf: factor boilerplate out of test_perf Jeff King 2018-08-21 19:06 ` [PATCH 2/6] t/perf: factor out percent calculations Jeff King 2018-08-21 19:06 ` [PATCH 3/6] t/perf: add infrastructure for measuring sizes Jeff King 2018-08-22 13:40 ` Derrick Stolee 2018-08-22 15:31 ` Jeff King 2018-08-21 19:06 ` [PATCH 4/6] t/perf: add perf tests for fetches from a bitmapped server Jeff King 2018-08-21 19:07 ` [PATCH 5/6] pack-bitmap: save "have" bitmap from walk Jeff King 2018-08-21 19:47 ` Derrick Stolee 2018-08-21 19:54 ` Jeff King 2018-08-31 15:23 ` Ævar Arnfjörð Bjarmason 2018-08-31 22:55 ` Jeff King 2018-09-01 7:41 ` [PATCH 0/4] un-breaking pack-objects with bitmaps Jeff King 2018-09-01 7:44 ` [PATCH 1/4] bitmap_has_sha1_in_uninteresting(): drop BUG check Jeff King 2018-09-01 7:48 ` [PATCH 2/4] t5310: test delta reuse with bitmaps Jeff King 2018-09-01 8:03 ` Jeff King 2018-09-01 20:29 ` Ævar Arnfjörð Bjarmason 2018-09-01 22:46 ` Ben Peart 2018-09-02 5:51 ` Jeff King 2018-09-04 19:05 ` Stefan Beller [this message] 2018-09-04 19:45 ` Junio C Hamano 2018-09-04 20:02 ` Jeff King 2018-09-01 7:49 ` [PATCH 3/4] traverse_bitmap_commit_list(): don't free result Jeff King 2018-09-01 7:50 ` [PATCH 4/4] pack-bitmap: drop "loaded" flag Jeff King 2018-09-04 19:30 ` [PATCH 0/4] un-breaking pack-objects with bitmaps Stefan Beller 2018-09-04 20:03 ` Jeff King 2018-09-08 6:43 ` Ævar Arnfjörð Bjarmason 2018-09-10 16:53 ` Junio C Hamano 2018-09-10 18:48 ` Jeff King 2018-09-10 19:23 ` Junio C Hamano 2018-08-21 19:07 ` [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects Jeff King 2018-08-21 19:43 ` Junio C Hamano 2018-08-21 19:50 ` Junio C Hamano 2018-08-21 20:07 ` Jeff King 2018-08-21 20:14 ` Jeff King 2018-08-21 20:52 ` Junio C Hamano 2018-08-21 21:30 ` Jeff King 2018-08-21 20:57 ` Junio C Hamano 2018-08-21 21:32 ` Jeff King 2018-08-23 0:43 ` [PATCH 0/9] trailer-parsing false positives Jeff King 2018-08-23 0:44 ` [PATCH 1/9] trailer: use size_t for string offsets Jeff King 2018-08-23 0:45 ` [PATCH 2/9] trailer: use size_t for iterating trailer list Jeff King 2018-08-23 0:46 ` [PATCH 3/9] trailer: pass process_trailer_opts to trailer_info_get() Jeff King 2018-08-23 0:48 ` [PATCH 4/9] interpret-trailers: tighten check for "---" patch boundary Jeff King 2018-08-23 0:49 ` [PATCH 5/9] interpret-trailers: allow suppressing "---" divider Jeff King 2018-08-23 0:50 ` [PATCH 6/9] pretty, ref-filter: format %(trailers) with no_divider option Jeff King 2018-08-23 0:50 ` [PATCH 7/9] sequencer: ignore "---" divider when parsing trailers Jeff King 2018-08-23 0:50 ` [PATCH 8/9] append_signoff: use size_t for string offsets Jeff King 2018-08-23 0:51 ` [PATCH 9/9] sequencer: handle ignore_footer when parsing trailers Jeff King 2018-08-23 18:30 ` [PATCH 0/9] trailer-parsing false positives Junio C Hamano 2018-08-24 7:26 ` Jeff King 2018-08-21 20:00 ` [PATCH 6/6] pack-objects: reuse on-disk deltas for thin "have" objects Jeff King
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to='CAGZ79ka8e2-f4fYgy+=HUDdvugefvQ5TnDG0v8YmUn7kGhTdaQ@mail.gmail.com' \ --firstname.lastname@example.org \ --cc=Johannes.Schindelin@gmx.de \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --firstname.lastname@example.org \ --subject='Re: [PATCH 2/4] t5310: test delta reuse with bitmaps' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).