[PATCH] midx: apply gitconfig to midx repack

* [PATCH] midx: apply gitconfig to midx repack
@ 2020-05-05 13:06 Son Luong Ngoc via GitGitGadget
  2020-05-05 13:50 ` Derrick Stolee
  2020-05-06  9:43 ` [PATCH v2 0/2] " Son Luong Ngoc via GitGitGadget
  0 siblings, 2 replies; 26+ messages in thread
From: Son Luong Ngoc via GitGitGadget @ 2020-05-05 13:06 UTC (permalink / raw)
  To: git; +Cc: Son Luong Ngoc, Son Luong Ngoc

From: Son Luong Ngoc <sluongng@gmail.com>

Multi-Pack-Index repack is an incremental, repack solutions
that allows user to consolidate multiple packfiles in a non-disruptive
way. However the new packfile could be created without some of the
capabilities of a packfile that is created by calling `git repack`.

This is because with `git repack`, there are configuration that would
enable different flags to be passed down to `git pack-objects` plumbing.

In this patch, I applies those flags into `git multi-pack-index repack`
so that it respect the `repack.*` config series.

Note: I left out `repack.packKeptObjects` intentionally as I dont think
its relevant to midx repack use case.

Signed-off-by: Son Luong Ngoc <sluongng@gmail.com>
---
    midx: apply gitconfig to midx repack
    
    Midx repack has largely been used in Microsoft Scalar on the client side
    to optimize the repository multiple packs state. However when I tried to
    apply this onto the server-side, I realized that there are certain
    features that were lacking compare to git repack. Most of these features
    are highly desirable on the server-side to create the most optimized
    pack possible.
    
    One of the example is delta_base_offset, comparing an midx repack
    with/without delta_base_offset, we can observe significant size
    differences.
    
    > du objects/pack/*pack
    14536   objects/pack/pack-08a017b424534c88191addda1aa5dd6f24bf7a29.pack
    9435280 objects/pack/pack-8829c53ad1dca02e7311f8e5b404962ab242e8f1.pack
    
    Latest 2.26.2 (without delta_base_offset)
    > git multi-pack-index write
    > git multi-pack-index repack
    > git multi-pack-index expire
    > du objects/pack/*pack
    9446096 objects/pack/pack-366c75e2c2f987b9836d3bf0bf5e4a54b6975036.pack
    
    With delta_base_offset
    > git version
    git version 2.26.2.672.g232c24e857.dirty
    > git multi-pack-index write
    > git multi-pack-index repack
    > git multi-pack-index expire
    > du objects/pack/*pack
    9152512 objects/pack/pack-3bc8c1ec496ab95d26875f8367ff6807081e9e7d.pack
    
    In this patch, I intentionally leaving out repack.packKeptObjects as I
    don't think its very relevant to midx repack use case:
    
     * One could always exclude biggest packs with --batch-size option
       
       
     * For non-biggest-packs exclusion use case, its rather rare (unless you
       want to have a special pack with only commits and trees being
       excluded from repack to serve partial clone better?)
       
       
    
    Please let me know if anyone think that we should include that option
    for the sake of completions.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-626%2Fsluongng%2Fsluongngoc%2Fmidx-config-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-626/sluongng/sluongngoc/midx-config-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/626

 midx.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/midx.c b/midx.c
index 9a61d3b37d9..88f16594268 100644
--- a/midx.c
+++ b/midx.c
@@ -1361,6 +1361,10 @@ static int fill_included_packs_batch(struct repository *r,
 	return 0;
 }
 
+static int delta_base_offset = 1;
+static int write_bitmaps = -1;
+static int use_delta_islands;
+
 int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, unsigned flags)
 {
 	int result = 0;
@@ -1381,12 +1385,25 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
 	} else if (fill_included_packs_all(m, include_pack))
 		goto cleanup;
 
+  git_config_get_bool("repack.usedeltabaseoffset", &delta_base_offset);
+  git_config_get_bool("repack.writebitmaps", &write_bitmaps);
+  git_config_get_bool("repack.usedeltaislands", &use_delta_islands);
+
 	argv_array_push(&cmd.args, "pack-objects");
 
 	strbuf_addstr(&base_name, object_dir);
 	strbuf_addstr(&base_name, "/pack/pack");
 	argv_array_push(&cmd.args, base_name.buf);
 
+	if (delta_base_offset)
+		argv_array_push(&cmd.args, "--delta-base-offset");
+	if (use_delta_islands)
+		argv_array_push(&cmd.args, "--delta-islands");
+	if (write_bitmaps > 0)
+		argv_array_push(&cmd.args, "--write-bitmap-index");
+	else if (write_bitmaps < 0)
+		argv_array_push(&cmd.args, "--write-bitmap-index-quiet");
+
 	if (flags & MIDX_PROGRESS)
 		argv_array_push(&cmd.args, "--progress");
 	else

base-commit: b34789c0b0d3b137f0bb516b417bd8d75e0cb306
-- 
gitgitgadget

^ permalink raw reply related	[flat|nested] 26+ messages in thread