From: Jeff King <peff@peff.net>
To: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: Neeraj Singh <neerajsi@microsoft.com>,
Patrick Steinhardt <ps@pks.im>,
git@vger.kernel.org
Subject: Re: [PATCH] refs: sync loose refs to disk before committing them
Date: Fri, 5 Nov 2021 05:34:44 -0400 [thread overview]
Message-ID: <YYT6tDyfBbwot2br@coredump.intra.peff.net> (raw)
In-Reply-To: <nycvar.QRO.7.76.6.2111051010070.56@tvgsbejvaqbjf.bet>
On Fri, Nov 05, 2021 at 10:12:25AM +0100, Johannes Schindelin wrote:
> > So this will definitely hurt at edge / pathological cases.
>
> Ouch.
>
> I wonder whether this could be handled similarly to the
> `core.fsyncObjectFiles=batch` mode that has been proposed in
> https://lore.kernel.org/git/pull.1076.v8.git.git.1633366667.gitgitgadget@gmail.com/
Yeah, that was along the lines I was thinking.
I hadn't really looked at the details of the batch-fsync there. The big
trick seems to be doing a pagecache writeback for each file, and then
stimulating an actual disk write (by fsyncing a tempfile) after the
batch is done.
That would be pretty easy to do for the refs (it's git_fsync() while
closing each file where Patrick is calling fsync(), followed by a single
do_batch_fsync() after everything is closed but before we rename).
> Essentially, we would have to find a better layer to do this, where we
> can synchronize after a potentially quite large number of ref updates has
> happened. That would definitely be a different layer than the file-based
> refs backend, of course, and would probably apply in a different way to
> other refs backends.
We do have the concept of a ref_transaction, so that would be the
natural place for it. Not every caller uses it, though, because it
implies atomicity of the transaction (so some may do a sequence of N
independent transactions, because they don't want failure of one to
impact others). I think that could be changed, if the ref_transaction
learned about non-atomicity, but it may take some surgery.
I expect that reftables would similarly benefit; it is probably much
more efficient to write a table slice with N entries than it is to write
N slices, even before accounting for fsync(). And once doing that, then
the fsync() part becomes trivial.
-Peff
next prev parent reply other threads:[~2021-11-05 9:34 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-04 12:38 [PATCH] refs: sync loose refs to disk before committing them Patrick Steinhardt
2021-11-04 13:14 ` Ævar Arnfjörð Bjarmason
2021-11-04 14:51 ` Patrick Steinhardt
2021-11-04 21:24 ` Junio C Hamano
2021-11-04 22:36 ` Neeraj Singh
2021-11-05 1:40 ` Junio C Hamano
2021-11-05 6:36 ` Jeff King
2021-11-05 8:35 ` Ævar Arnfjörð Bjarmason
2021-11-05 9:04 ` Jeff King
2021-11-05 7:07 ` Jeff King
2021-11-05 7:17 ` Jeff King
2021-11-05 9:12 ` Johannes Schindelin
2021-11-05 9:22 ` Patrick Steinhardt
2021-11-05 9:34 ` Jeff King [this message]
2021-11-09 11:25 ` Patrick Steinhardt
2021-11-10 8:36 ` Jeff King
2021-11-10 9:16 ` Patrick Steinhardt
2021-11-10 11:40 ` [PATCH v2 0/3] " Patrick Steinhardt
2021-11-10 11:40 ` [PATCH v2 1/3] wrapper: handle EINTR in `git_fsync()` Patrick Steinhardt
2021-11-10 14:33 ` Johannes Schindelin
2021-11-10 14:39 ` Ævar Arnfjörð Bjarmason
2021-11-10 11:40 ` [PATCH v2 2/3] wrapper: provide function to sync directories Patrick Steinhardt
2021-11-10 14:40 ` Ævar Arnfjörð Bjarmason
2021-11-10 11:41 ` [PATCH v2 3/3] refs: add configuration to enable flushing of refs Patrick Steinhardt
2021-11-10 14:49 ` Ævar Arnfjörð Bjarmason
2021-11-10 19:15 ` Neeraj Singh
2021-11-10 20:23 ` Ævar Arnfjörð Bjarmason
2021-11-11 0:03 ` Neeraj Singh
2021-11-11 12:14 ` Patrick Steinhardt
2021-11-11 12:06 ` Patrick Steinhardt
2021-11-11 0:18 ` Neeraj Singh
2021-11-10 14:44 ` [PATCH v2 0/3] refs: sync loose refs to disk before committing them Johannes Schindelin
2021-11-10 20:45 ` Jeff King
2021-11-11 11:47 ` Patrick Steinhardt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YYT6tDyfBbwot2br@coredump.intra.peff.net \
--to=peff@peff.net \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
--cc=neerajsi@microsoft.com \
--cc=ps@pks.im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).