Re: [PATCH v5 3/6] pack-bitmap-write: learn pack.writeBitmapLookupTable and add tests

[Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>]

On Wed, Aug 10, 2022 at 2:50 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Abhradeep,
> I instrumented this, and saw that the `multi-pack-index` and
> `multi-pack-index*.bitmap` files were unchanged by the `git repack`
> invocation.

Yeah, those two files remain unchanged here.

> Re-generating the MIDX bitmap forcefully after the repack seems to fix
> things over here:
>
> -- snip --
> diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh
> index a95537e759b..564124bda27 100644
> --- a/t/lib-bitmap.sh
> +++ b/t/lib-bitmap.sh
> @@ -438,7 +438,10 @@ midx_bitmap_partial_tests () {
>
>         test_expect_success 'setup partial bitmaps' '
>                 test_commit packed &&
> +ls -l .git/objects/pack/ &&
>                 git repack &&
> +git multi-pack-index write --bitmap &&
> +ls -l .git/objects/pack/ &&
>                 test_commit loose &&
>                 git multi-pack-index write --bitmap 2>err &&
>                 test_path_is_file $midx &&
> -- snap --
>
> This suggests to me that the `multi-pack-index write --bitmap 2>err` call
> in this hunk might reuse a stale MIDX bitmap, and that _that_  might be
> the root cause of this breakage.

Yeah, the `multi-pack-index write --bitmap 2>err` is creating the
problem. More specifically the `multi-pack-index write` part. As you
can see in my previous  comment (if you get the comment), I shared a
screenshot there which pointed out that the multi-pack-index files in
both cases are different. The portion from which it started to differ
belongs to the `RIDX` chunk.

So, I used some debug lines in `midx_pack_order()` function[1] and
found that the objects are sorted differently in those cases (i.e.
passing case and failing case). For passing case, the RIDX chunk
contents are like below -

pack_order = [ 1, 36, 11, 6, 18, 3, 19, 12, 5, 31, 27, 23, 29, 8, 38,
22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,
32, 0, 21, 16, 25, 13, 40, 20,]

And in the failing case, this is -

pack_order = [ 12, 18, 3, 19, 1, 36, 11, 6, 5, 31, 27, 23, 29, 8, 38,
22, 9, 15, 14, 24, 37, 28, 7, 39, 10, 34, 26, 4, 30, 33, 2, 35, 17,
32, 0, 21, 16, 25, 13, 40, 20,]

I went further and realized that this is due to the line[2] -

    if (!e->preferred)
        data[i].pack |= (1U << 31);

I.e. 4- 5 `pack_midx_entry` objects have different `preferred` values
in those cases. For example,
"46193a971f5045cb3ca6022957541f9ccddfbfe78591d8506e2d952f8113059b"
(with pack order 12) is `preferred` in failing case (that's why it is
in the first position) and the same is `not preferred` in the passing
case.

It may be because of reusing a stale midx bitmap (as you said). But I
am not sure. Just to ensure myself, I compared all the other
packfiles, idx files and a pack `.bitmap` file (which you can see
using ls command) of failing and passing cases and found that they are
the same.

Thanks :)

[1] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L861
[2] https://github.com/git/git/blob/c50926e1f48891e2671e1830dbcd2912a4563450/midx.c#L872