[PATCH 0/6] midx: permit changing the preferred pack when reusing the MIDX

[Taylor Blau <me@ttaylorr.com>]

This series resolves a bug that was reported[1] by Johannes, and
investigated by him, Abhradeep, and Stolee in that same sub-thread.

The crux of the issue is that a MIDX bitmap can enter a corrupt state
when changing the preferred pack from its value in an existing MIDX in
certain circumstances as described in the first and final patches.

This series is structured as follows:

  - The first patch of this series adds a test which demonstrates the
    problem. (This is an improvement from the debugging in [1], where we
    only noticed the problem racily in an existing test, and only in
    SHA-256 mode).

  - The next small handful of patches refactor midx.c's
    `get_sorted_entries()` function to make fixing this bug more
    straightforward.

  - The final patch resolves the bug and updates the test to no longer
    expect failure.

A couple of meta-notes:

  - This bug has existed since the introduction of MIDX bitmaps, but
    probably wasn't noticed until now since it is only triggerable when
    the `--stdin-packs` mode *isn't* passed, so it never occurs when
    invoked via `git repack`.

  - We could likely change the behavior of 56d863e979 (midx: expose
    `write_midx_file_only()` publicly, 2021-09-28), which explicitly
    disables reusing the existing MIDX (by avoiding loading it
    altogether) when `--stdin-packs` is given.

    The rationale in the comment added by 56d863e979 is somewhat unclear,
    but I have a vague recollection of running into a bug that was
    squashed by avoiding reusing an existing MIDX to write one with
    bitmaps. This was likely that bug.

Thanks in advance for your review, and thanks to Johannes, Abhradeep,
and Stolee for investigating this bug while I was on vacation.

[1]: https://lore.kernel.org/git/p3r70610-8n52-s8q0-n641-onp4ps01330n@tzk.qr/

Taylor Blau (6):
  t5326: demonstrate potential bitmap corruption
  t/lib-bitmap.sh: avoid silencing stderr
  midx.c: extract `struct midx_fanout`
  midx.c: extract `midx_fanout_add_midx_fanout()`
  midx.c: extract `midx_fanout_add_pack_fanout()`
  midx.c: include preferred pack correctly with existing MIDX

 midx.c                        | 128 ++++++++++++++++++++++------------
 t/lib-bitmap.sh               |   2 +-
 t/t5326-multi-pack-bitmaps.sh |  43 ++++++++++++
 3 files changed, 127 insertions(+), 46 deletions(-)

-- 
2.37.0.1.g1379af2e9d