Re: [PATCH 3/3] index-pack: adjust default threading cap

From: Jeff King <peff@peff.net>
To: "brian m. carlson" <sandals@crustytoothpaste.net>
Cc: git@vger.kernel.org
Subject: Re: [PATCH 3/3] index-pack: adjust default threading cap
Date: Mon, 24 Aug 2020 13:37:35 -0400	[thread overview]
Message-ID: <20200824173735.GA673908@coredump.intra.peff.net> (raw)
In-Reply-To: <20200822011607.GX8085@camp.crustytoothpaste.net>

On Sat, Aug 22, 2020 at 01:16:07AM +0000, brian m. carlson wrote:

> > +		if (nr_threads < 4)
> > +			; /* too few cores to consider capping */
> > +		else if (nr_threads < 6)
> > +			nr_threads = 3; /* historic cap */
> > +		else if (nr_threads < 40)
> > +			nr_threads /= 2;
> 
> I was going to ask if we could make the halving conditional on x86_64,
> but it turns out POWER and UltraSPARC also have SMT, so that doesn't
> make sense.  I expect that most users who have more than 6 "cores" are
> going to be on one of those systems or possibly ARM64, and since the
> performance penalty of using half as many cores isn't that significant,
> I'm not sure it's worth worrying about further.  This will be an
> improvement regardless.
> 
> Which is just a long way of saying, this patch seems fine to me.

OK, good. :) I agree there may be room for more improvement on those
systems. But lacking access to any, my goal was to make things better on
systems I _could_ test on, and not make things worse on other systems.
So I'd be very happy if people on other platforms (especially non-intel
ones) wanted to run:

  cd t/perf
  GIT_PERF_EXTRA=1 \
  GIT_PERF_LARGE_REPO=/path/to/clone/of/linux.git \
  ./p5302-pack-index.sh

and report the results.

I do have a slightly-old AMD machine with 4 cores (an A8-7600). Here's
what it says:

  5302.3: index-pack 0 threads                   447.67(436.62+6.57)
  5302.4: index-pack 1 threads                   450.80(441.26+7.20)
  5302.5: index-pack 2 threads                   265.62(459.56+7.30)
  5302.6: index-pack 4 threads                   177.06(477.56+8.22)
  5302.7: index-pack default number of threads   202.60(473.15+7.61)

So it does get better with 4 threads (but we continue to cap it at 3).
I wonder whether we should just do:

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 9721bf1ffe..d7453d0c09 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1809,7 +1809,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 		if (nr_threads < 4)
 			; /* too few cores to consider capping */
 		else if (nr_threads < 6)
-			nr_threads = 3; /* historic cap */
+			nr_threads = nr_threads;
 		else if (nr_threads < 40)
 			nr_threads /= 2;
 		else

That does probably make things slightly worse for a 6-core hyperthreaded
Intel machine. And it doesn't help an actual 8-core AMD machine at all.

-Peff
