Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension

From: Philip Oakley <philipoakley@iee.email>
To: Abhradeep Chakraborty <chakrabortyabhradeep79@gmail.com>
Cc: Git <git@vger.kernel.org>,
	Kaartic Sivaraam <kaartic.sivaraam@gmail.com>,
	Taylor Blau <me@ttaylorr.com>,
	Derrick Stolee <derrickstolee@github.com>,
	Junio C Hamano <gitster@pobox.com>
Subject: Re: [PATCH v3 1/6] Documentation/technical: describe bitmap lookup table extension
Date: Sun, 10 Jul 2022 16:01:11 +0100	[thread overview]
Message-ID: <d70a4505-60ef-82c4-5497-499ac788782a@iee.email> (raw)
In-Reply-To: <20220709075310.83848-1-chakrabortyabhradeep79@gmail.com>

On 09/07/2022 08:53, Abhradeep Chakraborty wrote:
> Hello Philip,
>
> Philip Oakley <philipoakley@iee.email> wrote:
>
>> Not sure if this is new in this extension, but should there be a link or
>> two to the basics of XOR compression and some of the bitmap look up
>> techniques?
>>
>> It's not always obvious if these techniques are 'heuristic' and only
>> have partial commit data, or they have all the commits listed, Nor
>> how/why they work. My point is more about giving new readers a hand-up
>> in their understanding, rather than simple implementation details for
>> those who already know what is going on. For example, are there any
>> external articles that you found helpful in getting started that could
>> be referenced somewhere in the docs?
> As this series is only about adding a lookup-table extension (and not
> about bitmap itself), I am not sure whether it's good to include those
> things in this series. 

Thanks for the clarification. I must have slight misread some of the
discussions and falsely thought it was the XOR compression (which is a
technique I wasn't really aware of), that was being provided by the
extension - Where would it be best for me to look up the background to
your "extension" project?

> But I agree with your point that it should be
> able build a logical understanding among the new readers.

*nod*
>
> There are some external articles[1] which talk about bitmap internals.
> But I think it would be better if we can make a new doc file (may be
> `Documentation/technical/reachability-bitmaps.txt` or similar) rather
> than putting those details in the `bitmap-format.txt` 

Thanks for the two links. In general I agree about the format document.

> (As the name 
> suggests, this file should only contain format details of bitmaps).
> That file would provide the answers of "Why bitmaps", "how they are
> stored",  "How they are fetched", "how they work with pack-objects,
> git-fetch, midx etc.", "Detailed explanation of each bitmap extension"
> , and lastly "what are the future works" (if any).

One thing I've realised on reflection is that I'm unclear how the
'reachability bitmaps' and the 'commit-graph file' techniques relate to
each other (and to the ODB DAG), and what features they pick out within
their heuristic, explained at just enough level to allow folks to
appreciate what the options that select them will do for their use case.

>
> What do you think?

I'd be happy to collate contributions, suggestions and thoughts.

Trying to create these good introductory descriptions can be really
difficult, as you can only step into the same river once (the 'reading
for the first time problem' of not being able to un-hear the
explanations of others when reading a 2nd draft...)
>
>> Separately I'm preparing a short series on adding 'reachability bitmap'
>> and 'commit graph' (among other stuff) to the glossary as part of giving
>> folks [0] stepping stones to cross the chasm of understanding
> Great!
>
> Thanks :)
>
> [1] https://github.blog/2015-09-22-counting-objects/, https://github.blog/2021-04-29-scaling-monorepo-maintenance/
Thank you.