git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Taylor Blau <me@ttaylorr.com>
To: Gregory Szorc <gregory.szorc@gmail.com>
Cc: git@vger.kernel.org, stolee@gmail.com
Subject: Re: Race condition between repack and loose-objects maintenance task
Date: Wed, 29 Jun 2022 13:03:54 -0400	[thread overview]
Message-ID: <YryF+vkosJOXf+mQ@nand.local> (raw)
In-Reply-To: <CAKQoGamCrRMqtzziuzi8mL6E7uA3SC1WXiMGT_4rpbk1jcu_OQ@mail.gmail.com>

Hi Greg,

On Wed, Jun 29, 2022 at 09:55:54AM -0700, Gregory Szorc wrote:
> 1. `git repack -A` creates loose objects
> 2. `git maintenance`'s `loose-objects` task deletes those loose objects
> 3. `git repack -A` fails to find the loose objects it just created and
> aborts with `fatal: unable to add recent objects`

This is a somewhat well-known race that occurs when one Git process
decides unreachable objects are safe to be deleted, but an incoming push
or reference update makes those to-be-deleted objects reachable before
they are actually removed, leaving the repository in a corrupt state.

I'm surprised that the loose-objects maintenance task deletes those
objects, though, since it just runs `prune-packed` or (the equivalent
of) `repack -d`, neither of which will actually delete objects from the
repository.

I see that Stolee is already on the CC list, so perhaps he could chime
in on the above.

In either case, my recommendation would be to keep those unreachable
objects which haven't yet aged out of the repository around for longer,
which will decrease the likelihood of seeing the race. If your
repository has a lot of unreachable objects (such that storing each
one of them individually as loose is impractical), I would recommend
using cruft packs (e.g., by running either `git repack --cruft -d` or
`git gc --cruft`) to collect those unreachable objects together into a
single pack.

See Documentation/technical/cruft-packs.txt for more information about
cruft packs.

Thanks,
Taylor

  reply	other threads:[~2022-06-29 17:04 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-06-29 16:55 Race condition between repack and loose-objects maintenance task Gregory Szorc
2022-06-29 17:03 ` Taylor Blau [this message]
2022-06-29 17:10   ` Taylor Blau
2022-06-29 17:16   ` Gregory Szorc
2022-06-29 17:21     ` Taylor Blau
2022-06-30  0:44       ` Gregory Szorc
2022-06-30  3:19         ` Taylor Blau
2022-06-30  3:23           ` Taylor Blau
2022-06-30 20:12             ` Junio C Hamano
2022-07-05 18:43               ` Gregory Szorc
2022-07-06  8:50                 ` Ævar Arnfjörð Bjarmason
2022-07-20  1:40             ` Gregory Szorc
2022-07-20  9:52               ` Ævar Arnfjörð Bjarmason
2022-07-26 16:22                 ` Gregory Szorc
2022-07-26 18:11                   ` Ævar Arnfjörð Bjarmason
2022-07-20  1:41             ` Gregory Szorc

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YryF+vkosJOXf+mQ@nand.local \
    --to=me@ttaylorr.com \
    --cc=git@vger.kernel.org \
    --cc=gregory.szorc@gmail.com \
    --cc=stolee@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).