From: Derrick Stolee <stolee@gmail.com>
To: Jeff King <peff@peff.net>, Geert Jansen <gerardu@amazon.com>
Cc: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>,
"git@vger.kernel.org" <git@vger.kernel.org>,
"René Scharfe" <l.s.r@web.de>,
"Takuto Ikuta" <tikuta@chromium.org>
Subject: Re: [PATCH 5/9] handle alternates paths the same as the main object dir
Date: Mon, 12 Nov 2018 10:38:28 -0500 [thread overview]
Message-ID: <56c50f51-5d9c-1166-75b9-418e384fc672@gmail.com> (raw)
In-Reply-To: <20181112144935.GE7400@sigill.intra.peff.net>
On 11/12/2018 9:49 AM, Jeff King wrote:
> When we generate loose file paths for the main object directory, the
> caller provides a buffer to loose_object_path (formerly sha1_file_name).
> The callers generally keep their own static buffer to avoid excessive
> reallocations.
>
> But for alternate directories, each struct carries its own scratch
> buffer. This is needlessly different; let's unify them.
>
> We could go either direction here, but this patch moves the alternates
> struct over to the main directory style (rather than vice-versa).
> Technically the alternates style is more efficient, as it avoids
> rewriting the object directory name on each call. But this is unlikely
> to matter in practice, as we avoid reallocations either way (and nobody
> has ever noticed or complained that the main object directory is copying
> a few extra bytes before making a much more expensive system call).
Hm. I've complained in the past [1] about a simple method like
strbuf_addf() over loose objects, but that was during abbreviation
checks so we were adding the string for every loose object but not
actually reading the objects.
[1]
https://public-inbox.org/git/20171201174956.143245-1-dstolee@microsoft.com/
The other concern I have is for alternates that may have long-ish paths
to their object directories.
So, this is worth keeping an eye on, but is likely to be fine.
> And this has the advantage that the reusable buffers are tied to
> particular calls, which makes the invalidation rules simpler (for
> example, the return value from stat_sha1_file() used to be invalidated
> by basically any other object call, but now it is affected only by other
> calls to stat_sha1_file()).
>
> We do steal the trick from alt_sha1_path() of returning a pointer to the
> filled buffer, which makes a few conversions more convenient.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> object-store.h | 14 +-------------
> object.c | 1 -
> sha1-file.c | 44 ++++++++++++++++----------------------------
> sha1-name.c | 8 ++++++--
> 4 files changed, 23 insertions(+), 44 deletions(-)
>
> diff --git a/object-store.h b/object-store.h
> index fefa17e380..b2fa0d0df0 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -10,10 +10,6 @@
> struct object_directory {
> struct object_directory *next;
>
> - /* see alt_scratch_buf() */
> - struct strbuf scratch;
> - size_t base_len;
> -
> /*
> * Used to store the results of readdir(3) calls when searching
> * for unique abbreviated hashes. This cache is never
> @@ -54,14 +50,6 @@ void add_to_alternates_file(const char *dir);
> */
> void add_to_alternates_memory(const char *dir);
>
> -/*
> - * Returns a scratch strbuf pre-filled with the alternate object directory,
> - * including a trailing slash, which can be used to access paths in the
> - * alternate. Always use this over direct access to alt->scratch, as it
> - * cleans up any previous use of the scratch buffer.
> - */
> -struct strbuf *alt_scratch_buf(struct object_directory *odb);
> -
> struct packed_git {
> struct packed_git *next;
> struct list_head mru;
> @@ -157,7 +145,7 @@ void raw_object_store_clear(struct raw_object_store *o);
> * Put in `buf` the name of the file in the local object database that
> * would be used to store a loose object with the specified sha1.
> */
> -void loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
> +const char *loose_object_path(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
>
> void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
>
> diff --git a/object.c b/object.c
> index 6af8e908bb..dd485ac629 100644
> --- a/object.c
> +++ b/object.c
> @@ -484,7 +484,6 @@ struct raw_object_store *raw_object_store_new(void)
>
> static void free_alt_odb(struct object_directory *odb)
> {
> - strbuf_release(&odb->scratch);
> oid_array_clear(&odb->loose_objects_cache);
> free(odb);
> }
> diff --git a/sha1-file.c b/sha1-file.c
> index 478eac326b..15db6b61a9 100644
> --- a/sha1-file.c
> +++ b/sha1-file.c
> @@ -346,27 +346,20 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
> }
> }
>
> -void loose_object_path(struct repository *r, struct strbuf *buf,
> - const unsigned char *sha1)
> +static const char *odb_loose_path(const char *path, struct strbuf *buf,
> + const unsigned char *sha1)
> {
> strbuf_reset(buf);
> - strbuf_addstr(buf, r->objects->objectdir);
> + strbuf_addstr(buf, path);
> strbuf_addch(buf, '/');
> fill_sha1_path(buf, sha1);
> + return buf->buf;
> }
>
> -struct strbuf *alt_scratch_buf(struct object_directory *odb)
> +const char *loose_object_path(struct repository *r, struct strbuf *buf,
> + const unsigned char *sha1)
> {
> - strbuf_setlen(&odb->scratch, odb->base_len);
> - return &odb->scratch;
> -}
> -
> -static const char *alt_sha1_path(struct object_directory *odb,
> - const unsigned char *sha1)
> -{
> - struct strbuf *buf = alt_scratch_buf(odb);
> - fill_sha1_path(buf, sha1);
> - return buf->buf;
> + return odb_loose_path(r->objects->objectdir, buf, sha1);
> }
>
> /*
> @@ -547,9 +540,6 @@ struct object_directory *alloc_alt_odb(const char *dir)
> struct object_directory *ent;
>
> FLEX_ALLOC_STR(ent, path, dir);
> - strbuf_init(&ent->scratch, 0);
> - strbuf_addf(&ent->scratch, "%s/", dir);
> - ent->base_len = ent->scratch.len;
>
> return ent;
> }
> @@ -745,10 +735,12 @@ static int check_and_freshen_local(const struct object_id *oid, int freshen)
> static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)
> {
> struct object_directory *odb;
> + static struct strbuf path = STRBUF_INIT;
> +
> prepare_alt_odb(the_repository);
> for (odb = the_repository->objects->alt_odb_list; odb; odb = odb->next) {
> - const char *path = alt_sha1_path(odb, oid->hash);
> - if (check_and_freshen_file(path, freshen))
> + odb_loose_path(odb->path, &path, oid->hash);
> + if (check_and_freshen_file(path.buf, freshen))
> return 1;
> }
> return 0;
> @@ -889,7 +881,7 @@ int git_open_cloexec(const char *name, int flags)
> *
> * The "path" out-parameter will give the path of the object we found (if any).
> * Note that it may point to static storage and is only valid until another
> - * call to loose_object_path(), etc.
> + * call to stat_sha1_file().
> */
> static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
> struct stat *st, const char **path)
> @@ -897,16 +889,14 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
> struct object_directory *odb;
> static struct strbuf buf = STRBUF_INIT;
>
> - loose_object_path(r, &buf, sha1);
> - *path = buf.buf;
> -
> + *path = loose_object_path(r, &buf, sha1);
> if (!lstat(*path, st))
> return 0;
>
> prepare_alt_odb(r);
> errno = ENOENT;
> for (odb = r->objects->alt_odb_list; odb; odb = odb->next) {
> - *path = alt_sha1_path(odb, sha1);
> + *path = odb_loose_path(odb->path, &buf, sha1);
> if (!lstat(*path, st))
> return 0;
> }
> @@ -926,9 +916,7 @@ static int open_sha1_file(struct repository *r,
> int most_interesting_errno;
> static struct strbuf buf = STRBUF_INIT;
>
> - loose_object_path(r, &buf, sha1);
> - *path = buf.buf;
> -
> + *path = loose_object_path(r, &buf, sha1);
> fd = git_open(*path);
> if (fd >= 0)
> return fd;
> @@ -936,7 +924,7 @@ static int open_sha1_file(struct repository *r,
>
> prepare_alt_odb(r);
> for (odb = r->objects->alt_odb_list; odb; odb = odb->next) {
> - *path = alt_sha1_path(odb, sha1);
> + *path = odb_loose_path(odb->path, &buf, sha1);
> fd = git_open(*path);
> if (fd >= 0)
> return fd;
> diff --git a/sha1-name.c b/sha1-name.c
> index 2594aa79f8..96a8e71482 100644
> --- a/sha1-name.c
> +++ b/sha1-name.c
> @@ -97,6 +97,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
> int subdir_nr = ds->bin_pfx.hash[0];
> struct object_directory *odb;
> static struct object_directory *fakeent;
> + struct strbuf buf = STRBUF_INIT;
>
> if (!fakeent) {
> /*
> @@ -114,8 +115,9 @@ static void find_short_object_filename(struct disambiguate_state *ds)
> int pos;
>
> if (!odb->loose_objects_subdir_seen[subdir_nr]) {
> - struct strbuf *buf = alt_scratch_buf(odb);
> - for_each_file_in_obj_subdir(subdir_nr, buf,
> + strbuf_reset(&buf);
> + strbuf_addstr(&buf, odb->path);
> + for_each_file_in_obj_subdir(subdir_nr, &buf,
> append_loose_object,
> NULL, NULL,
> &odb->loose_objects_cache);
> @@ -134,6 +136,8 @@ static void find_short_object_filename(struct disambiguate_state *ds)
> pos++;
> }
> }
> +
> + strbuf_release(&buf);
> }
>
> static int match_sha(unsigned len, const unsigned char *a, const unsigned char *b)
next prev parent reply other threads:[~2018-11-12 15:38 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-25 18:38 [RFC PATCH] index-pack: improve performance on NFS Jansen, Geert
2018-10-26 0:21 ` Junio C Hamano
2018-10-26 20:38 ` Ævar Arnfjörð Bjarmason
2018-10-27 7:26 ` Junio C Hamano
2018-10-27 9:33 ` Jeff King
2018-10-27 11:22 ` Ævar Arnfjörð Bjarmason
2018-10-28 22:50 ` [PATCH 0/4] index-pack: optionally turn off SHA-1 collision checking Ævar Arnfjörð Bjarmason
2018-10-30 2:49 ` Geert Jansen
2018-10-30 9:04 ` Junio C Hamano
2018-10-30 18:43 ` [PATCH v2 0/3] index-pack: test updates Ævar Arnfjörð Bjarmason
2018-11-13 20:19 ` [PATCH v3] index-pack: add ability to disable SHA-1 collision check Ævar Arnfjörð Bjarmason
2018-11-14 7:09 ` Junio C Hamano
2018-11-14 12:40 ` Ævar Arnfjörð Bjarmason
2018-10-30 18:43 ` [PATCH v2 1/3] pack-objects test: modernize style Ævar Arnfjörð Bjarmason
2018-10-30 18:43 ` [PATCH v2 2/3] pack-objects tests: don't leave test .git corrupt at end Ævar Arnfjörð Bjarmason
2018-10-30 18:43 ` [PATCH v2 3/3] index-pack tests: don't leave test repo dirty " Ævar Arnfjörð Bjarmason
2018-10-28 22:50 ` [PATCH 1/4] pack-objects test: modernize style Ævar Arnfjörð Bjarmason
2018-10-28 22:50 ` [PATCH 2/4] pack-objects tests: don't leave test .git corrupt at end Ævar Arnfjörð Bjarmason
2018-10-28 22:50 ` [PATCH 3/4] index-pack tests: don't leave test repo dirty " Ævar Arnfjörð Bjarmason
2018-10-28 22:50 ` [PATCH 4/4] index-pack: add ability to disable SHA-1 collision check Ævar Arnfjörð Bjarmason
2018-10-29 15:04 ` [RFC PATCH] index-pack: improve performance on NFS Jeff King
2018-10-29 15:09 ` Jeff King
2018-10-29 19:36 ` Ævar Arnfjörð Bjarmason
2018-10-29 23:27 ` Jeff King
2018-11-07 22:55 ` Geert Jansen
2018-11-08 12:02 ` Jeff King
2018-11-08 20:58 ` Geert Jansen
2018-11-08 21:18 ` Jeff King
2018-11-08 21:55 ` Geert Jansen
2018-11-08 22:20 ` Ævar Arnfjörð Bjarmason
2018-11-09 10:11 ` Ævar Arnfjörð Bjarmason
2018-11-12 14:31 ` Jeff King
2018-11-12 14:46 ` [PATCH 0/9] caching loose objects Jeff King
2018-11-12 14:46 ` [PATCH 1/9] fsck: do not reuse child_process structs Jeff King
2018-11-12 15:26 ` Derrick Stolee
2018-11-12 14:47 ` [PATCH 2/9] submodule--helper: prefer strip_suffix() to ends_with() Jeff King
2018-11-12 18:23 ` Stefan Beller
2018-11-12 14:48 ` [PATCH 3/9] rename "alternate_object_database" to "object_directory" Jeff King
2018-11-12 15:30 ` Derrick Stolee
2018-11-12 15:36 ` Jeff King
2018-11-12 19:41 ` Ramsay Jones
2018-11-12 14:48 ` [PATCH 4/9] sha1_file_name(): overwrite buffer instead of appending Jeff King
2018-11-12 15:32 ` Derrick Stolee
2018-11-12 14:49 ` [PATCH 5/9] handle alternates paths the same as the main object dir Jeff King
2018-11-12 15:38 ` Derrick Stolee [this message]
2018-11-12 15:46 ` Jeff King
2018-11-12 15:50 ` Derrick Stolee
2018-11-12 14:50 ` [PATCH 6/9] sha1-file: use an object_directory for " Jeff King
2018-11-12 15:48 ` Derrick Stolee
2018-11-12 16:09 ` Jeff King
2018-11-12 19:04 ` Stefan Beller
2018-11-22 17:42 ` Jeff King
2018-11-12 18:48 ` Stefan Beller
2018-11-12 14:50 ` [PATCH 7/9] object-store: provide helpers for loose_objects_cache Jeff King
2018-11-12 19:24 ` René Scharfe
2018-11-12 20:16 ` Jeff King
2018-11-12 14:54 ` [PATCH 8/9] sha1-file: use loose object cache for quick existence check Jeff King
2018-11-12 16:00 ` Derrick Stolee
2018-11-12 16:01 ` Ævar Arnfjörð Bjarmason
2018-11-12 16:21 ` Jeff King
2018-11-12 22:18 ` Ævar Arnfjörð Bjarmason
2018-11-12 22:30 ` Ævar Arnfjörð Bjarmason
2018-11-13 10:02 ` Ævar Arnfjörð Bjarmason
2018-11-14 18:21 ` René Scharfe
2018-12-02 10:52 ` René Scharfe
2018-12-03 22:04 ` Jeff King
2018-12-04 21:45 ` René Scharfe
2018-12-05 4:46 ` Jeff King
2018-12-05 6:02 ` René Scharfe
2018-12-05 6:51 ` Jeff King
2018-12-05 8:15 ` Jeff King
2018-12-05 18:41 ` René Scharfe
2018-12-05 20:17 ` Jeff King
2018-11-12 22:44 ` Geert Jansen
2018-11-27 20:48 ` René Scharfe
2018-12-01 19:49 ` Jeff King
2018-11-12 14:55 ` [PATCH 9/9] fetch-pack: drop custom loose object cache Jeff King
2018-11-12 19:25 ` René Scharfe
2018-11-12 19:32 ` Ævar Arnfjörð Bjarmason
2018-11-12 20:07 ` Jeff King
2018-11-12 20:13 ` René Scharfe
2018-11-12 16:02 ` [PATCH 0/9] caching loose objects Derrick Stolee
2018-11-12 19:10 ` Stefan Beller
2018-11-09 13:43 ` [RFC PATCH] index-pack: improve performance on NFS Ævar Arnfjörð Bjarmason
2018-11-09 16:08 ` Duy Nguyen
2018-11-10 14:04 ` Ævar Arnfjörð Bjarmason
2018-11-12 14:34 ` Jeff King
2018-11-12 22:58 ` Geert Jansen
2018-10-27 14:04 ` Duy Nguyen
2018-10-29 15:18 ` Jeff King
2018-10-29 0:48 ` Junio C Hamano
2018-10-29 15:20 ` Jeff King
2018-10-29 18:43 ` Ævar Arnfjörð Bjarmason
2018-10-29 21:34 ` Geert Jansen
2018-10-29 21:50 ` Jeff King
2018-10-29 22:21 ` Geert Jansen
2018-10-29 22:27 ` Jeff King
2018-10-29 22:35 ` Stefan Beller
2018-10-29 23:29 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56c50f51-5d9c-1166-75b9-418e384fc672@gmail.com \
--to=stolee@gmail.com \
--cc=avarab@gmail.com \
--cc=gerardu@amazon.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=tikuta@chromium.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).