On Tue, Mar 01, 2022 at 09:06:44AM -0500, Derrick Stolee wrote:
> On 3/1/2022 5:35 AM, Patrick Steinhardt wrote:
> > On Tue, Mar 01, 2022 at 10:46:14AM +0100, Patrick Steinhardt wrote:
> >> On Mon, Feb 28, 2022 at 01:44:01PM -0500, Derrick Stolee wrote:
> >>> On 2/28/2022 11:59 AM, Patrick Steinhardt wrote:
> >>>> On Mon, Feb 28, 2022 at 11:23:38AM -0500, Derrick Stolee wrote:
> >>>>> On 2/28/2022 10:18 AM, Patrick Steinhardt wrote:
> >>>>>> I haven't yet found the time to dig deeper into why this is happening.
> >>>>>> While the repository is publicly accessible at [1], unfortunately the
> >>>>>> bug seems to be triggered by a commit that's only kept alive by an
> >>>>>> internal reference.
> >>>>>>
> >>>>>> Patrick
> >>>>>>
> >>>>>> [1]: https://gitlab.com/gitlab-com/www-gitlab-com.git
> >>>>>
> >>>>> Thanks for including this information. Just to be clear: did you
> >>>>> include patch 4 in your tests, or not? Patch 4 includes a fix
> >>>>> related to overflow values, so it would be helpful to know if you
> >>>>> found a _different_ bug or if it is the same one.
> >>>>>
> >>>>> Thanks,
> >>>>> -Stolee
> >>>>
> >>>> I initially only applied the first three patches, but after having hit
> >>>> the fatal error I also applied the rest of this series to have a look at
> >>>> whether it is indeed fixed already by one of your later patches. The
> >>>> error remains the same though.
> >>>
> >>> Thanks for this extra context. Is this a commit-graph that you wrote
> >>> with the first three patches and then you get an error when reading it?
> >>>
> >>> Do you get the same error when deleting that file and rewriting it with
> >>> all patches included?
> >>>
> >>> Thanks,
> >>> -Stolee
> >>
> >> Yes, I do. I've applied all four patches from v2 on top of 715d08a9e5
> >> (The eighth batch, 2022-02-25) and still get the same results:
> >>
> >>     $ find objects/info/commit-graphs/
> >>     objects/info/commit-graphs/
> >>     objects/info/commit-graphs/graph-607e641165f3e83a82d5b14af4e611bf2a688f35.graph
> >>     objects/info/commit-graphs/commit-graph-chain
> >>     objects/info/commit-graphs/graph-5f357c7573c0075d42d82b28e660bc3eac01bfe8.graph
> >>     objects/info/commit-graphs/graph-e0c12ead1b61c7c30720ae372e8a9f98d95dfb2d.graph
> >>     objects/info/commit-graphs/graph-c96723b133c2d81106a01ecd7a8773bb2ef6c2e1.graph
> >>
> >>      $ git commit-graph verify
> >>     fatal: commit-graph requires overflow generation data but has none
> >>
> >>      $ git commit-graph write
> >>     Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.
> >>     Expanding reachable commits in commit graph: 2197197, done.
> >>     Finding extra edges in commit graph: 100% (2197197/2197197), done.
> >>     fatal: commit-graph requires overflow generation data but has none
> >>
> >>      $ rm -rf objects/info/commit-graphs/
> >>
> >>      $ git commit-graph write
> >>     Finding commits for commit graph among packed objects: 100% (10235119/10235119), done.
> >>     Expanding reachable commits in commit graph: 2197197, done.
> >>     Finding extra edges in commit graph: 100% (2197197/2197197), done.
> >>     fatal: commit-graph requires overflow generation data but has none)
> >>
> >> So even generating them completely anew doesn't seem to generate the
> >> overflow generation data.
> >>
> >> Patrick
> > 
> > I stand corrected. I forgot that the repository at hand was connected to
> > another one via `objects/info/alternates`. If I prune commit-graphs from
> > that alternate, too, then it works alright with your patches.
> 
> OK, thanks. That clarifies the situation.
> 
> I ordered the patches such that the fix in patch 4 could be immediately
> testable, which is not the case without patch 3. However, it does leave
> this temporary state where information can be incorrect if only a subset
> of the series is applied.
> 
> > This makes me wonder how such a bugfix should be handled though. As this
> > series is right now, users will be faced with repository corruption as
> > soon as they upgrade their Git version to one that contains this patch
> > series. This corruption needs manual action: they have to go into the
> > repository, delete the commit-graphs and then optionally create new
> > ones.
> > 
> > This is not a good user experience, and it's worse on the server-side
> > where we now have a timeframe where all commit-graphs are potentially
> > corrupt. This effectively leads to us being unable to serve those repos
> > at all until we have rewritten the commit-graphs because all commands
> > which make use of the commit-graph will now die:
> > 
> >     $ git log
> >     fatal: commit-graph requires overflow generation data but has none
> > 
> > So the question is whether this is a change that needs to be rolled out
> > over multiple releases. First we'd get in the bug fix such that we write
> > correct commit-graphs, and after this fix has been released we can also
> > release the fix that starts to actually parse the generation. This
> > ensures there's a grace period during which we can hopefully correct the
> > data on-disk such that users are not faced with failures.
> 
> You are right that we need to be careful here, but I also think that
> previous versions of Git always wrote the correct data. Here is my
> thought process:
> 
> 1. To get this bug, we need to have parsed the corrected commit date
>    from an existing commit-graph in order to under-count the number
>    of overflow values.
> 
> 2. Before this series, Git versions were not parsing the corrected
>    commit date, so they recompute the corrected commit date every
>    time the commit-graph is written, getting the proper count of
>    overflow values.
> 
> For these reasons, data written by previous versions of Git are
> correct and can be trusted without a staged release.
> 
> Does this make sense? Or, do you experience a different result when
> you build commit-graphs with a released Git version and then when
> writing on top with all patches applied?

Just to verify my understanding: you claim that the bug I was hitting
shouldn't be encountered in the wild when the release , but
only if one were to write a commit-graph with the intermediate stafe
until patch 3/4 of your patch series?

Hum. I have re-verified, and this indeed seems to play out. So I must've
accidentally ran all my testing with the state generated without the
final patch which fixes the corruption. I do see lots of the following
warnings, but overall I can verify and write the commit-graph just fine:

    commit-graph generation for commit c80a42de8803e2d77818d0c82f88e748d7f9425f is 1623362063 < 1623362139

Thanks for your patience, and sorry for the noise :)

> > The better alternative would probably be to just gracefully handle
> > commit-graphs which are corrupted in such a way. Can we maybe just
> > continue to not parse generations in case we find that the commit-graph
> > doesn't have overflow generation data?
> >
> > This is more of a general issue though: commit-graphs are an auxiliary
> > cache that is not required for proper operation at all. If we fail to
> > parse it, then Git shouldn't die but instead fail gracefully just ignore
> > it. Furthermore, if we notice that graphs are corrupt when we try to
> > write new ones, we may just delete the corrupt versions automatically
> > and generate completely new ones.
> 
> You are right that we can be better about failures here and report
> and error instead of a die(). Especially in this case, we could just
> revert to topological levels instead of throwing out the commit-graph
> entirely.
> 
> This seems like something for another series, so we can be sure to
> audit all cases of fatal errors when parsing the commit-graph so we
> catch all of them and do the "best" thing in each case.

I agree.

Patrick