Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases

From: Jeff King <peff@peff.net>
To: Jonathan Tan <jonathantanmy@google.com>
Cc: git@vger.kernel.org
Subject: Re: [PATCH 2/2] index-pack: prefetch missing REF_DELTA bases
Date: Fri, 17 May 2019 00:42:52 -0400	[thread overview]
Message-ID: <20190517044252.GA12107@sigill.intra.peff.net> (raw)
In-Reply-To: <20190517043939.GA12063@sigill.intra.peff.net>

On Fri, May 17, 2019 at 12:39:39AM -0400, Jeff King wrote:

> > I think to trigger it you'd have to manually assemble an evil pack as I
> > described (e.g., using the routines in t/lib-pack.sh). I'm going offline
> > for a bit, but I may have a go at it later tonight or tomorrow.
> 
> OK, doing so wasn't _too_ bad, though I did have to resurrect the
> horrible generator patch from [1]. My results are below. But more
> importantly, there is good news.

Oh, I forgot my [1]. It was this message:

  https://public-inbox.org/git/20130823182409.GA30130@sigill.intra.peff.net/

which explained how I hacked up pack-objects to generate the binary goo
in lib-pack.sh. Here's my forward-ported version of it, in case we need
it again.

I do think it would be useful for debugging and experimenting to have a
mode for pack-objects that takes in a list of instructions (add object
X, make it a delta against object Y) that skips its usual ordering and
delta constraints. This isn't it, but it's enough to hackily get what
you need. ;)

---

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 9f424aabab..87955d9404 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -3234,6 +3234,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	struct argv_array rp = ARGV_ARRAY_INIT;
 	int rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0;
 	int rev_list_index = 0;
+	int magic = 0;
 	struct string_list keep_pack_list = STRING_LIST_INIT_NODUP;
 	struct option pack_objects_options[] = {
 		OPT_SET_INT('q', "quiet", &progress,
@@ -3321,6 +3322,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 			 N_("do not pack objects in promisor packfiles")),
 		OPT_BOOL(0, "delta-islands", &use_delta_islands,
 			 N_("respect islands during delta compression")),
+		OPT_BOOL(0, "magic", &magic, "make deltas"),
 		OPT_END(),
 	};
 
@@ -3337,6 +3339,34 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	argc = parse_options(argc, argv, prefix, pack_objects_options,
 			     pack_usage, 0);
 
+	if (magic) {
+		struct object_id oid;
+		struct delta_index *index;
+		void *src, *trg, *delta;
+		enum object_type src_type, trg_type;
+		unsigned long src_size, trg_size, delta_size, z_delta_size;
+		unsigned char header[10];
+		unsigned long header_len;
+
+		get_oid(argv[0], &oid);
+		trg = read_object_file(&oid, &trg_type, &trg_size);
+
+		get_oid(argv[1], &oid);
+		src = read_object_file(&oid, &src_type, &src_size);
+
+		index = create_delta_index(src, src_size);
+		delta = create_delta(index, trg, trg_size, &delta_size, 8192);
+
+		z_delta_size = do_compress(&delta, delta_size);
+		header_len = encode_in_pack_object_header(header, sizeof(header),
+							  OBJ_REF_DELTA,
+							  delta_size);
+		fwrite(header, 1, header_len, stdout);
+		fwrite(oid.hash, 1, the_hash_algo->rawsz, stdout);
+		fwrite(delta, 1, z_delta_size, stdout);
+		return 0;
+	}
+
 	if (argc) {
 		base_name = argv[0];
 		argc--;
