Re: Partial-clone cause big performance impact on server

From: Jeff King <peff@peff.net>
To: 程洋 <chengyang@xiaomi.com>
Cc: "git@vger.kernel.org" <git@vger.kernel.org>,
	何浩 <hehao@xiaomi.com>, "Xin7 Ma 马鑫" <maxin7@xiaomi.com>,
	石奉兵 <shifengbing@xiaomi.com>, 凡军辉 <fanjunhui@xiaomi.com>,
	王汉基 <wanghanji@xiaomi.com>
Subject: Re: Partial-clone cause big performance impact on server
Date: Sun, 14 Aug 2022 02:48:01 -0400	[thread overview]
Message-ID: <YviaoXRctE9fg/mB@coredump.intra.peff.net> (raw)
In-Reply-To: <bfa3de4485614badb4a27d8cfba99968@xiaomi.com>

On Thu, Aug 11, 2022 at 08:09:56AM +0000, 程洋 wrote:

>     4. And we then track our server(which is gerrit with jgit). We
>        found the server is couting objects. Then we check those 40k
>        objects, most of them are blobs rather than commit. (which
>        means they're not in bitmap)
>     5. We believe that's the root cause of our problem. Git sends too
>        many "want SHA1" which are not in bitmap, cause the server to
>        count objects  frequently, which then slow down the server.

I'd be surprised if bitmaps make a big difference either way here, since
blobs are very quick in the "counting" phase of pack-objects. They can't
link to anything else, so we should not be opening the object contents
at all! We just need to find them on disk, and then in many cases we can
send them over the wire without even decompressing (the exception is if
they are stored as deltas against an object the client doesn't have).

I didn't generate a test case, but I'm pretty sure that is how git.git's
pack-objects should behave. But you mentioned that the server is jgit;
it's possible that it isn't as optimized in that area.

-Peff