Re: [PATCH 00/30] [RFC] extensions.refFormat and packed-refs v2 file format

From: Junio C Hamano <gitster@pobox.com>
To: Derrick Stolee <derrickstolee@github.com>
Cc: Elijah Newren <newren@gmail.com>,
	Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>,
	git@vger.kernel.org, jrnieder@gmail.com
Subject: Re: [PATCH 00/30] [RFC] extensions.refFormat and packed-refs v2 file format
Date: Fri, 18 Nov 2022 15:31:18 -0800	[thread overview]
Message-ID: <xmqqiljbkfg9.fsf@gitster.g> (raw)
In-Reply-To: <0e156172-0670-2832-78cb-c7dfe2599192@github.com> (Derrick Stolee's message of "Sun, 13 Nov 2022 19:07:21 -0500")

Derrick Stolee <derrickstolee@github.com> writes:

> On 11/11/22 6:28 PM, Elijah Newren wrote:
>> On Mon, Nov 7, 2022 at 11:01 AM Derrick Stolee via GitGitGadget
>> <gitgitgadget@gmail.com> wrote:
>>>
>>> Introduction
>>> ============
>>>
>>> I became interested in our packed-ref format based on the asymmetry between
>>> ref updates and ref deletions: if we delete a packed ref, then the
>>> packed-refs file needs to be rewritten. Compared to writing a loose ref,
>>> this is an O(N) cost instead of O(1).
>>>
>>> In this way, I set out with some goals:
>>>
>>>  * (Primary) Make packed ref deletions be nearly as fast as loose ref
>>>    updates.
>> 
>> Performance is always nice.  :-)
>> 
>>>  * (Secondary) Allow using a packed ref format for all refs, dropping loose
>>>    refs and creating a clear way to snapshot all refs at a given point in
>>>    time.
>> 
>> Is this secondary goal the actual goal you have, or just the
>> implementation by which you get the real underlying goal?
>
> To me, the primary goal takes precedence. It turns out that the best
> way to solve for that goal happens to also make it possible to store
> all refs in a packed form, because we can update the packed form
> much faster than our current setup. There are alternatives that I
> considered (and prototyped) that were more specific to the deletions
> case, but they were not actually as fast as the stacked method. Those
> alternatives also would never help reach the secondary goal, but I
> probably would have considered them anyway if they were faster, if
> only for their simplicity.

I have been and am still offline and haven't examined this proposal
in detail, but would it be a better longer-term approach to improve
reftable backend, instead of piling more effort on loose+packed
filesystem based backend?