Re: SHA-256 transition

From: Jeff King <peff@peff.net>
To: "brian m. carlson" <sandals@crustytoothpaste.net>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Stephen Smith" <ischis2@cox.net>, git <git@vger.kernel.org>
Subject: Re: SHA-256 transition
Date: Fri, 1 Jul 2022 14:00:12 -0400	[thread overview]
Message-ID: <Yr82LITalM8JX753@coredump.intra.peff.net> (raw)
In-Reply-To: <YrbNIUnftj+Ooumo@tapette.crustytoothpaste.net>

On Sat, Jun 25, 2022 at 08:53:53AM +0000, brian m. carlson wrote:

> > I'm curious if you have numbers on this. I naively converted linux.git
> > to sha256 by doing "fast-export | fast-import" (the latter in a sha256
> > repo, of course, and then both repacked with "-f --window=250" to get
> > reasonable apples-to-apples packs).
> 
> I did the same thing, except I just did a regular gc and not a custom
> repack, and I created both a SHA-1 and SHA-256 repo from the same
> original.

That _might_ influence your timings a bit, just because the fast-import
packs have lousy deltas. I think my linux.git was something like 6GB
from fast-import, packed down to 1.5GB after "repack -f".

But I'm not sure if it would change the direction of the trend of what
you were measuring, only the magnitude. We'll hash the same bytes in
either case, but in the fast-import pack we'd spend more time on zlib
inflating and less time on delta reconstruction. Which one is more
expensive probably depends on a lot of factors, but it's entirely
possible that running your test after a "repack -f" would actually show
a greater change between the two cases.

> Here are my results:
> 
> [sha256]
> time ~/checkouts/git/git index-pack --verify .git/objects/pack/pack-*.pack
> ~/checkouts/git/git index-pack --verify .git/objects/pack/pack-*.pack  2768.42s user 181.00s system 185% cpu 26:31.70 total
> 
> [sha1dc]
> time ~/checkouts/git/git index-pack --verify .git/objects/pack/pack-*.pack
> ~/checkouts/git/git index-pack --verify .git/objects/pack/pack-*.pack  3041.28s user 184.84s system 199% cpu 26:54.74 total
> 
> Note that in my case, I'm using an accelerated hardware-based SHA-256
> implementation (Nettle, which I will send a patch for soon).  This is a
> brand new ThinkPad X1 Carbon Gen 10 with an i7-1280P (with 20 "cores" of
> different sizes).

OK, that probably explains the difference in results we saw. Thanks for
sharing your numbers.  I think that's pretty "apples to apples" since
we'd hope that sha256 will eventually be accelerated, but sha1dc never
will be.

-Peff