All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jeff King <peff@peff.net>
To: Elijah Newren <newren@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>,
	Elijah Newren via GitGitGadget <gitgitgadget@gmail.com>,
	Git Mailing List <git@vger.kernel.org>,
	Jonathan Nieder <jrnieder@gmail.com>,
	Sergey Organov <sorganov@gmail.com>
Subject: Re: [PATCH 5/7] tmp-objdir: new API for creating and removing primary object dirs
Date: Fri, 1 Oct 2021 01:27:44 -0400	[thread overview]
Message-ID: <YVacUCcl6USJSY5T@coredump.intra.peff.net> (raw)
In-Reply-To: <CABPp-BF20vpC4m5V4JtS91fuA+PMiEN78J9OpVEPDpnKsTcP8A@mail.gmail.com>

On Thu, Sep 30, 2021 at 09:26:37PM -0700, Elijah Newren wrote:

> >  - This side-steps all of our usual code for getting object data into
> >    memory. In general, I'd expect this content to not be too enormous,
> >    but it _could_ be if there are many / large blobs in the result. So
> >    we may end up with large maps. Probably not a big deal on modern
> >    64-bit systems. Maybe an issue on 32-bit systems, just because of
> >    virtual address space.
> >
> >    Likewise, we do support systems with NO_MMAP. They'd work here, but
> >    it would probably mean putting all that object data into the heap. I
> >    could live with that, given how rare such systems are these days, and
> >    that it only matters if you're using --remerge-diff with big blobs.
> 
> Um, I'm starting to get uncomfortable with this pretend_object stuff.
> Part of the reason that merge-ort isn't truly "in memory" despite
> attempting to do exactly that, was because for large enough repos with
> enough files modified on both sides, I wasn't comfortable assuming
> that all new files from three-way content merges and all new trees fit
> into memory.  I'm sure we'd be fine with current-day linux kernel
> sized repos.  No big deal.  In fact, most merges probably don't add
> more than a few dozen new files.  But for microsoft-sized repos, and
> with repos tending to grow over time, more so when the tools
> themselves scale nicely (which we've all been working on enabling),
> makes me worry there might be enough new objects within a single merge
> (especially given the recursive inner merges) that we might need to
> worry about this.

I do think we need to consider that the content might be larger than
will comfortably fit in memory. But the point of using mmap is that we
don't have to care. The OS is taking care of it for us (just like it
would in regular object files).

The question is just whether we're comfortable assuming that mmap
exists if you're working on such a large repository. I'd guess that big
repos are pretty painful with out it (and again, I'm not even clear
which systems Git runs on even lack mmap these days). So IMHO this isn't
really a blocker for going in this direction.

> >    The whole pretend-we-have-this-object thing
> >    may want to likewise make sure we don't write out objects that we
> >    already have in the real odb.
> 
> Right, so I'd have to copy the relevant logic from write_object_file()
> -- I think that means instead of write_object_file()'s calls to
> freshen_packed_object() and freshen_loose_object() that I instead call
> find_pack_entry() and make has_loose_object() in object-file.c not be
> static and then call it.  Does that sound right?

Yes. You _could_ call the same freshening functions, but I think since
we know that our objects are throw-away anyway, we do not even need to
perform that freshening operation. I think just has_object_file() would
be sufficient for your needs.

-Peff

  reply	other threads:[~2021-10-01  5:27 UTC|newest]

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-08-31  2:26 [PATCH 0/7] Add a new --remerge-diff capability to show & log Elijah Newren via GitGitGadget
2021-08-31  2:26 ` [PATCH 1/7] merge-ort: mark a few more conflict messages as omittable Elijah Newren via GitGitGadget
2021-08-31 21:06   ` Junio C Hamano
2021-09-01  0:03     ` Elijah Newren
2021-09-01 17:19       ` Junio C Hamano
2021-08-31  2:26 ` [PATCH 2/7] merge-ort: add ability to record conflict messages in a file Elijah Newren via GitGitGadget
2021-09-28 22:29   ` Jeff King
2021-09-29  6:25     ` Elijah Newren
2021-09-29 16:14       ` Junio C Hamano
2021-09-29 16:31         ` Elijah Newren
2021-09-30  7:58       ` Jeff King
2021-09-30  8:09         ` Ævar Arnfjörð Bjarmason
2021-10-01  2:07         ` Elijah Newren
2021-10-01  5:28           ` Jeff King
2021-08-31  2:26 ` [PATCH 3/7] ll-merge: add API for capturing warnings in a strbuf instead of stderr Elijah Newren via GitGitGadget
2021-09-28 22:37   ` Jeff King
2021-09-28 23:49     ` Junio C Hamano
2021-09-29  4:03     ` Elijah Newren
2021-08-31  2:26 ` [PATCH 4/7] merge-ort: capture and print ll-merge warnings in our preferred fashion Elijah Newren via GitGitGadget
2021-09-28 22:39   ` Jeff King
2021-09-30 16:53   ` Ævar Arnfjörð Bjarmason
2021-10-01  1:54     ` Elijah Newren
2021-10-01  7:23       ` Ævar Arnfjörð Bjarmason
2021-08-31  2:26 ` [PATCH 5/7] tmp-objdir: new API for creating and removing primary object dirs Elijah Newren via GitGitGadget
2021-09-28  7:55   ` Ævar Arnfjörð Bjarmason
2021-09-29  4:22     ` Elijah Newren
2021-09-30  7:41       ` Jeff King
2021-09-30 14:17       ` Ævar Arnfjörð Bjarmason
2021-10-01  3:55         ` Elijah Newren
2021-09-28 23:17   ` Jeff King
2021-09-29  4:08     ` Junio C Hamano
2021-09-30  7:33       ` Jeff King
2021-09-30 13:16         ` Ævar Arnfjörð Bjarmason
2021-09-30 21:00           ` Jeff King
2021-10-01  3:11           ` Elijah Newren
2021-10-01  7:30             ` Ævar Arnfjörð Bjarmason
2021-10-01  8:03               ` Elijah Newren
2021-10-01  4:26         ` Elijah Newren
2021-10-01  5:27           ` Jeff King [this message]
2021-10-01  7:43             ` Ævar Arnfjörð Bjarmason
2021-09-29  5:05     ` Elijah Newren
2021-09-30  7:26       ` Jeff King
2021-09-30  7:46         ` Jeff King
2021-09-30 20:06           ` Junio C Hamano
2021-10-01  3:59             ` Elijah Newren
2021-10-01 16:36               ` Junio C Hamano
2021-10-01  2:31           ` Elijah Newren
2021-10-01  5:21             ` Jeff King
2021-09-30 19:25         ` Neeraj Singh
2021-09-30 20:19           ` Junio C Hamano
2021-09-30 20:00         ` Junio C Hamano
2021-10-01  2:23         ` Elijah Newren
2021-08-31  2:26 ` [PATCH 6/7] show, log: provide a --remerge-diff capability Elijah Newren via GitGitGadget
2021-08-31  9:19   ` Sergey Organov
2021-09-28  8:01   ` Ævar Arnfjörð Bjarmason
2021-09-29  4:00     ` Elijah Newren
2021-08-31  2:26 ` [PATCH 7/7] doc/diff-options: explain the new --remerge-diff option Elijah Newren via GitGitGadget
2021-08-31 11:05 ` [PATCH 0/7] Add a new --remerge-diff capability to show & log Bagas Sanjaya
2021-08-31 16:16   ` Elijah Newren
2021-08-31 20:03 ` Junio C Hamano
2021-08-31 20:23   ` Elijah Newren
2021-09-01 21:07 ` Junio C Hamano
2021-09-01 21:42   ` Elijah Newren
2021-09-01 21:55     ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YVacUCcl6USJSY5T@coredump.intra.peff.net \
    --to=peff@peff.net \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=gitster@pobox.com \
    --cc=jrnieder@gmail.com \
    --cc=newren@gmail.com \
    --cc=sorganov@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.