[PATCH 00/26] Moving global state into the repository object (part 1)

[PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which 
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (8):
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to map_sha1_file
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (18):
  repository: introduce raw object store field
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c                                    |   2 +-
 builtin/clone.c                                 |   2 +-
 builtin/count-objects.c                         |   6 +-
 builtin/fetch.c                                 |   2 +-
 builtin/fsck.c                                  |  13 ++-
 builtin/gc.c                                    |   4 +-
 builtin/grep.c                                  |   2 +-
 builtin/index-pack.c                            |   1 +
 builtin/merge.c                                 |   2 +-
 builtin/pack-objects.c                          |  21 ++--
 builtin/pack-redundant.c                        |   6 +-
 builtin/receive-pack.c                          |   3 +-
 cache.h                                         |  46 ++------
 contrib/coccinelle/refactoring/packed_git.cocci |   7 ++
 environment.c                                   |   5 +-
 fast-import.c                                   |   6 +-
 http-backend.c                                  |   6 +-
 http-push.c                                     |   1 +
 http-walker.c                                   |   4 +-
 http.c                                          |   6 +-
 mru.h                                           |   1 +
 object-store.h                                  |  75 +++++++++++++
 object.c                                        |  26 +++++
 pack-bitmap.c                                   |   4 +-
 pack-check.c                                    |   1 +
 pack-revindex.c                                 |   1 +
 packfile.c                                      |  64 +++++------
 packfile.h                                      |   2 +-
 path.c                                          |   2 +-
 reachable.c                                     |   1 +
 repository.c                                    |  17 ++-
 repository.h                                    |   7 +-
 server-info.c                                   |   6 +-
 sha1_file.c                                     | 135 +++++++++++++-----------
 sha1_name.c                                     |  11 +-
 streaming.c                                     |   5 +-
 36 files changed, 321 insertions(+), 182 deletions(-)
 create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci
 create mode 100644 object-store.h

-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 01/26] repository: introduce raw object store field

[Stefan Beller]

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 17 +++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 42 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index 63ac38a46f..5d7ffb3a3f 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -242,9 +243,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..cf35760ceb
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 0afdfd19b7..9b5b65e189 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index f66fcb1342..bd2ad578de 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store

[Stefan Beller]

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 cache.h        |  4 ++--
 object-store.h |  7 ++++++-
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 6 files changed, 27 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 04846d46f9..1048255da1 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/cache.h b/cache.h
index d8b975a571..918c2f15b4 100644
--- a/cache.h
+++ b/cache.h
@@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -1591,7 +1591,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
diff --git a/object-store.h b/object-store.h
index cf35760ceb..44b8f84753 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,14 +1,19 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+#include "cache.h"
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 4a5fe7ab18..d61076faaf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "mru.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -880,7 +881,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 3da70ac650..2826d5d6ed 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "mru.h"
@@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -673,12 +671,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.73.ga2c3e9663f.dirty

Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store

[Brandon Williams]

On 02/12, Stefan Beller wrote:
> In a process with multiple repositories open, alternates should be
> associated to a single repository and not shared globally. Move
> alt_odb_list and alt_odb_tail into the_repository and adjust callers
> to reflect this.
> 
> Now that the alternative object data base is per repository, we're
> leaking its memory upon freeing a repository. The next patch plugs
> this hole.
> 
> No functional change intended.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  builtin/fsck.c |  4 +++-
>  cache.h        |  4 ++--
>  object-store.h |  7 ++++++-
>  packfile.c     |  3 ++-
>  sha1_file.c    | 25 ++++++++++++-------------
>  sha1_name.c    |  3 ++-
>  6 files changed, 27 insertions(+), 19 deletions(-)
> 
> diff --git a/builtin/fsck.c b/builtin/fsck.c
> index 04846d46f9..1048255da1 100644
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -1,5 +1,6 @@
>  #include "builtin.h"
>  #include "cache.h"
> +#include "repository.h"
>  #include "config.h"
>  #include "commit.h"
>  #include "tree.h"
> @@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  		fsck_object_dir(get_object_directory());
>  
>  		prepare_alt_odb();
> -		for (alt = alt_odb_list; alt; alt = alt->next)
> +		for (alt = the_repository->objects.alt_odb_list;
> +				alt; alt = alt->next)
>  			fsck_object_dir(alt->path);
>  
>  		if (check_full) {
> diff --git a/cache.h b/cache.h
> index d8b975a571..918c2f15b4 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
>  extern void schedule_dir_for_removal(const char *name, int len);
>  extern void remove_scheduled_dirs(void);
>  
> -extern struct alternate_object_database {
> +struct alternate_object_database {
>  	struct alternate_object_database *next;
>  
>  	/* see alt_scratch_buf() */
> @@ -1591,7 +1591,7 @@ extern struct alternate_object_database {
>  	struct oid_array loose_objects_cache;
>  
>  	char path[FLEX_ARRAY];
> -} *alt_odb_list;
> +};
>  extern void prepare_alt_odb(void);
>  extern char *compute_alternate_path(const char *path, struct strbuf *err);
>  typedef int alt_odb_fn(struct alternate_object_database *, void *);
> diff --git a/object-store.h b/object-store.h
> index cf35760ceb..44b8f84753 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -1,14 +1,19 @@
>  #ifndef OBJECT_STORE_H
>  #define OBJECT_STORE_H
>  
> +#include "cache.h"
> +

Maybe we would want to move the definition of the alternate store to
this header file?  That way we don't need to include cache.h here.

>  struct raw_object_store {
>  	/*
>  	 * Path to the repository's object store.
>  	 * Cannot be NULL after initialization.
>  	 */
>  	char *objectdir;
> +
> +	struct alternate_object_database *alt_odb_list;
> +	struct alternate_object_database **alt_odb_tail;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
>  
>  void raw_object_store_clear(struct raw_object_store *o);
>  
> diff --git a/packfile.c b/packfile.c
> index 4a5fe7ab18..d61076faaf 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "mru.h"
>  #include "pack.h"
> +#include "repository.h"
>  #include "dir.h"
>  #include "mergesort.h"
>  #include "packfile.h"
> @@ -880,7 +881,7 @@ void prepare_packed_git(void)
>  		return;
>  	prepare_packed_git_one(get_object_directory(), 1);
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next)
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
>  		prepare_packed_git_one(alt->path, 0);
>  	rearrange_packed_git();
>  	prepare_packed_git_mru();
> diff --git a/sha1_file.c b/sha1_file.c
> index 3da70ac650..2826d5d6ed 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -22,6 +22,7 @@
>  #include "pack-revindex.h"
>  #include "sha1-lookup.h"
>  #include "bulk-checkin.h"
> +#include "repository.h"
>  #include "streaming.h"
>  #include "dir.h"
>  #include "mru.h"
> @@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
>  	return buf->buf;
>  }
>  
> -struct alternate_object_database *alt_odb_list;
> -static struct alternate_object_database **alt_odb_tail;
> -
>  /*
>   * Return non-zero iff the path is usable as an alternate object database.
>   */
> @@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
>  	 * Prevent the common mistake of listing the same
>  	 * thing twice, or object directory itself.
>  	 */
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		if (!fspathcmp(path->buf, alt->path))
>  			return 0;
>  	}
> @@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
>  	ent = alloc_alt_odb(pathbuf.buf);
>  
>  	/* add the alternate entry */
> -	*alt_odb_tail = ent;
> -	alt_odb_tail = &(ent->next);
> +	*the_repository->objects.alt_odb_tail = ent;
> +	the_repository->objects.alt_odb_tail = &(ent->next);
>  	ent->next = NULL;
>  
>  	/* recursively add alternates */
> @@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)
>  		fprintf_or_die(out, "%s\n", reference);
>  		if (commit_lock_file(&lock))
>  			die_errno("unable to move new alternates file into place");
> -		if (alt_odb_tail)
> +		if (the_repository->objects.alt_odb_tail)
>  			link_alt_odb_entries(reference, '\n', NULL, 0);
>  	}
>  	free(alts);
> @@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
>  	int r = 0;
>  
>  	prepare_alt_odb();
> -	for (ent = alt_odb_list; ent; ent = ent->next) {
> +	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
>  		r = fn(ent, cb);
>  		if (r)
>  			break;
> @@ -673,12 +671,13 @@ void prepare_alt_odb(void)
>  {
>  	const char *alt;
>  
> -	if (alt_odb_tail)
> +	if (the_repository->objects.alt_odb_tail)
>  		return;
>  
>  	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>  
> -	alt_odb_tail = &alt_odb_list;
> +	the_repository->objects.alt_odb_tail =
> +			&the_repository->objects.alt_odb_list;
>  	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
>  
>  	read_info_alternates(get_object_directory(), 0);
> @@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
>  {
>  	struct alternate_object_database *alt;
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		const char *path = alt_sha1_path(alt, sha1);
>  		if (check_and_freshen_file(path, freshen))
>  			return 1;
> @@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
>  
>  	prepare_alt_odb();
>  	errno = ENOENT;
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		*path = alt_sha1_path(alt, sha1);
>  		if (!lstat(*path, st))
>  			return 0;
> @@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
>  	most_interesting_errno = errno;
>  
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		*path = alt_sha1_path(alt, sha1);
>  		fd = git_open(*path);
>  		if (fd >= 0)
> diff --git a/sha1_name.c b/sha1_name.c
> index 611c7d24dd..957ce25680 100644
> --- a/sha1_name.c
> +++ b/sha1_name.c
> @@ -10,6 +10,7 @@
>  #include "dir.h"
>  #include "sha1-array.h"
>  #include "packfile.h"
> +#include "repository.h"
>  
>  static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
>  
> @@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
>  		 */
>  		fakeent = alloc_alt_odb(get_object_directory());
>  	}
> -	fakeent->next = alt_odb_list;
> +	fakeent->next = the_repository->objects.alt_odb_list;
>  
>  	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
>  		int pos;
> -- 
> 2.16.1.73.ga2c3e9663f.dirty
> 

-- 
Brandon Williams

Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store

[Stefan Beller]

On Tue, Feb 13, 2018 at 10:51 AM, Brandon Williams <bmwill@google.com> wrote:

>> +#include "cache.h"
>> +
>
> Maybe we would want to move the definition of the alternate store to
> this header file?  That way we don't need to include cache.h here.
>

Sounds good. I'll take a look at that.

Stefan

[PATCH 03/26] object-store: free alt_odb_list

[Stefan Beller]

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 9b5b65e189..d778f09717 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 04/26] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch
    contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
    This semantic patch is placed in a sub directory of the coccinelle
    contrib dir, as this semantic patch is not expected to be of general
    usefulness; it is only useful during developing this series and
    merging it with other topics in flight. At a later date, just
    delete that semantic patch.

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c                         |  6 ++--
 builtin/fsck.c                                  |  7 +++--
 builtin/gc.c                                    |  4 ++-
 builtin/index-pack.c                            |  1 +
 builtin/pack-objects.c                          | 21 ++++++++-----
 builtin/pack-redundant.c                        |  6 ++--
 builtin/receive-pack.c                          |  1 +
 cache.h                                         | 28 ------------------
 contrib/coccinelle/refactoring/packed_git.cocci |  7 +++++
 fast-import.c                                   |  6 ++--
 http-backend.c                                  |  6 ++--
 http-push.c                                     |  1 +
 http-walker.c                                   |  1 +
 http.c                                          |  1 +
 mru.h                                           |  1 +
 object-store.h                                  | 32 +++++++++++++++++++-
 object.c                                        |  6 ++++
 pack-bitmap.c                                   |  4 ++-
 pack-check.c                                    |  1 +
 pack-revindex.c                                 |  1 +
 packfile.c                                      | 39 +++++++++++++------------
 reachable.c                                     |  1 +
 server-info.c                                   |  6 ++--
 sha1_name.c                                     |  5 ++--
 24 files changed, 120 insertions(+), 72 deletions(-)
 create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 1048255da1..d4d249c2ed 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -707,7 +708,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -715,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 3c5eae0edf..6970f325e5 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 4c51aec81f..1902d4e096 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 6b9cfc289d..2ee8b69238 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1023,7 +1025,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			return want;
 	}
 
-	for (entry = packed_git_mru.head; entry; entry = entry->next) {
+	for (entry = the_repository->objects.packed_git_mru.head; entry;
+	     entry = entry->next) {
 		struct packed_git *p = entry->item;
 		off_t offset;
 
@@ -1041,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				mru_mark(&packed_git_mru, entry);
+				mru_mark(&the_repository->objects.packed_git_mru,
+					 entry);
 			if (want != -1)
 				return want;
 		}
@@ -2648,7 +2652,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2711,7 +2715,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2720,7 +2725,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2756,7 +2761,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3119,7 +3124,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3132,7 +3137,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index 918c2f15b4..459fd01dbb 100644
--- a/cache.h
+++ b/cache.h
@@ -1633,34 +1633,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list, which can
- * be iterated instead of packed_git (and marked via mru_mark).
- */
-extern struct mru packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/contrib/coccinelle/refactoring/packed_git.cocci b/contrib/coccinelle/refactoring/packed_git.cocci
new file mode 100644
index 0000000000..da317a51a9
--- /dev/null
+++ b/contrib/coccinelle/refactoring/packed_git.cocci
@@ -0,0 +1,7 @@
+@@ @@
+- packed_git
++ the_repository->objects.packed_git
+
+@@ @@
+- packed_git_mru
++ the_repository->objects.packed_git_mru
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 14435ab65d..34c5d030b7 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 1ae8363de2..a1c6f2639b 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 5977712712..ab989b88dd 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/mru.h b/mru.h
index 42e4aeaa10..d7de81efb9 100644
--- a/mru.h
+++ b/mru.h
@@ -37,6 +37,7 @@ struct mru_entry {
 struct mru {
 	struct mru_entry *head, *tail;
 };
+#define MRU_INIT { NULL, NULL }
 
 void mru_append(struct mru *mru, void *item);
 void mru_mark(struct mru *mru, struct mru_entry *entry);
diff --git a/object-store.h b/object-store.h
index 44b8f84753..8778ab9685 100644
--- a/object-store.h
+++ b/object-store.h
@@ -2,6 +2,7 @@
 #define OBJECT_STORE_H
 
 #include "cache.h"
+#include "mru.h"
 
 struct raw_object_store {
 	/*
@@ -10,11 +11,40 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct mru packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+struct packed_git {
+	struct packed_git *next;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index d778f09717..81d083bec1 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	mru_clear(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index 1b7ebd8d7e..82c3119652 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index d61076faaf..11b0e6613c 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 
@@ -40,8 +41,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-struct mru packed_git_mru;
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -241,7 +240,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -311,7 +310,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -379,7 +378,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -677,8 +676,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -760,7 +759,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -810,7 +810,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -859,17 +859,18 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	mru_clear(&packed_git_mru);
-	for (p = packed_git; p; p = p->next)
-		mru_append(&packed_git_mru, p);
+	mru_clear(&the_repository->objects.packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		mru_append(&the_repository->objects.packed_git_mru, p);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1003,7 +1004,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1835,12 +1836,12 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct mru_entry *p;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	for (p = packed_git_mru.head; p; p = p->next) {
+	for (p = the_repository->objects.packed_git_mru.head; p; p = p->next) {
 		if (fill_pack_entry(sha1, e, p->item)) {
-			mru_mark(&packed_git_mru, p);
+			mru_mark(&the_repository->objects.packed_git_mru, p);
 			return 1;
 		}
 	}
@@ -1887,7 +1888,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if (open_pack_index(p)) {
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 05/26] object-store: close all packs upon clearing the object store

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index acfe9d3c8c..75e16dfec6 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 284651797e..7ec08d4696 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1200,7 +1200,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7bbcd26faf..70015ccdc7 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1407,7 +1407,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index 81d083bec1..79c2c447bc 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -468,8 +469,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 	o->alt_odb_tail = NULL;
 
 	mru_clear(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index 11b0e6613c..2ce1b2cef7 100644
--- a/packfile.c
+++ b/packfile.c
@@ -306,11 +306,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index 0cdeb54dcd..fa547226b7 100644
--- a/packfile.h
+++ b/packfile.h
@@ -61,7 +61,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 06/26] pack: move prepare_packed_git_run_once to object store

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 8778ab9685..9fd109fa9c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,8 +20,14 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 2ce1b2cef7..c107fce171 100644
--- a/packfile.c
+++ b/packfile.c
@@ -873,12 +873,11 @@ static void prepare_packed_git_mru(void)
 		mru_append(&the_repository->objects.packed_git_mru, p);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -886,13 +885,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 07/26] pack: move approximate object count to object store

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 9fd109fa9c..86f1ec2cd0 100644
--- a/object-store.h
+++ b/object-store.h
@@ -21,13 +21,21 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
 	 */
 	unsigned packed_git_initialized : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index c107fce171..a53706cf16 100644
--- a/packfile.c
+++ b/packfile.c
@@ -793,8 +793,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -804,8 +802,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -815,8 +813,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -890,7 +889,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 08/26] sha1_file: add raw_object_store argument to alt_odb_usable

[Stefan Beller]

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 2826d5d6ed..d25b68a8fb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -350,7 +350,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -366,7 +368,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -418,7 +420,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 09/26] sha1_file: add repository argument to link_alt_odb_entry

[Stefan Beller]

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index d25b68a8fb..7dc5f690e2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -394,8 +394,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -492,7 +493,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 10/26] sha1_file: add repository argument to read_info_alternates

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 7dc5f690e2..4fdfdd945a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -393,7 +393,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -434,7 +436,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -500,7 +502,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -684,7 +687,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 11/26] sha1_file: add repository argument to link_alt_odb_entries

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 4fdfdd945a..b090f403d8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -471,8 +471,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -515,7 +519,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -569,7 +573,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -582,7 +587,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -685,7 +691,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 12/26] sha1_file: add repository argument to prepare_alt_odb

[Stefan Beller]

See previous patch for explanation.

While at it, move the declaration to object-store.h,
where it should be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 cache.h        |  1 -
 object-store.h |  3 +++
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  3 ++-
 6 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index d4d249c2ed..00ec8eecf0 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -695,7 +695,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/cache.h b/cache.h
index 459fd01dbb..70518e24ce 100644
--- a/cache.h
+++ b/cache.h
@@ -1592,7 +1592,6 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 extern int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/object-store.h b/object-store.h
index 86f1ec2cd0..d96a16edd1 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,4 +61,7 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+extern void prepare_alt_odb_the_repository(void);
+
 #endif /* OBJECT_STORE_H */
diff --git a/packfile.c b/packfile.c
index a53706cf16..9c49b34d2b 100644
--- a/packfile.c
+++ b/packfile.c
@@ -879,7 +879,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index b090f403d8..d7f271fe6e 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "mru.h"
@@ -585,7 +586,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -671,7 +672,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -680,7 +681,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -729,7 +730,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -884,7 +885,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -911,7 +912,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..3e490ee8f6 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 13/26] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories

[Stefan Beller]

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h     |  4 ++++
 sha1_file.c | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/cache.h b/cache.h
index 70518e24ce..09d37cb651 100644
--- a/cache.h
+++ b/cache.h
@@ -1590,6 +1590,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
diff --git a/sha1_file.c b/sha1_file.c
index d7f271fe6e..d18ce3aeba 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -394,11 +394,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -424,7 +423,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -432,12 +431,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -472,12 +471,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -491,7 +486,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -500,15 +495,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -520,7 +516,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 21 +++++++++++----------
 2 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/object-store.h b/object-store.h
index d96a16edd1..add1d4e27c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,7 +61,6 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-extern void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index d18ce3aeba..f046d560f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+
+	if (!r->ignore_env) {
+		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+		if (!alt)
+			alt = "";
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	}
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.73.ga2c3e9663f.dirty

Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Duy Nguyen]

On Tue, Feb 13, 2018 at 8:22 AM, Stefan Beller <sbeller@google.com> wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  object-store.h |  3 +--
>  sha1_file.c    | 21 +++++++++++----------
>  2 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/object-store.h b/object-store.h
> index d96a16edd1..add1d4e27c 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -61,7 +61,6 @@ struct packed_git {
>         char pack_name[FLEX_ARRAY]; /* more */
>  };
>
> -#define prepare_alt_odb(r) prepare_alt_odb_##r()
> -extern void prepare_alt_odb_the_repository(void);
> +void prepare_alt_odb(struct repository *r);
>
>  #endif /* OBJECT_STORE_H */
> diff --git a/sha1_file.c b/sha1_file.c
> index d18ce3aeba..f046d560f8 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
>         return r;
>  }
>
> -void prepare_alt_odb_the_repository(void)
> +void prepare_alt_odb(struct repository *r)
>  {
> -       const char *alt;
> -
> -       if (the_repository->objects.alt_odb_tail)
> +       if (r->objects.alt_odb_tail)
>                 return;
>
> -       alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +       r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> +
> +       if (!r->ignore_env) {
> +               const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);

If one day the majority of git moves to use 'struct repository', then
ALTERNATE_DB_ENVIRONMENT is always ignored because ignore_env is
always true. I think if you ignore_env, then you still need to get
this "alt" from  'struct raw_object_store' (or 'struct repository').

Since you get lots of getenv() in repo_setup_env(), I think this
getenv(ALTERNATE_DB_ENVIRONMENT) belongs there too. Then here, if
ignore_env is true, you read r->objects.alt or something instead of
doing getenv().

I really want to kill this getenv() in this code, which is basically
delayed initialization because we haven't done proper init on
the_repo. I realize that it cannot be done earlier, when
prepare_alt_odb() does not even have a  'struct repository *' to work
with. Would it be ok if I contributed a patch on top of your series to
basically do repo_init(&the_repo) for all builtin/external commands
(and fix all the bugs that come with it)? Then we would not need
ignore_env here anymore.

> +               if (!alt)
> +                       alt = "";
>
> -       the_repository->objects.alt_odb_tail =
> -                       &the_repository->objects.alt_odb_list;
> -       link_alt_odb_entries(the_repository, alt,
> -                            PATH_SEP, NULL, 0);
> +               link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> +       }
>
> -       read_info_alternates(the_repository, get_object_directory(), 0);
> +       read_info_alternates(r, r->objects.objectdir, 0);
>  }
>
>  /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
> --
> 2.16.1.73.ga2c3e9663f.dirty
>



-- 
Duy

Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Brandon Williams]

On 02/14, Duy Nguyen wrote:
> On Tue, Feb 13, 2018 at 8:22 AM, Stefan Beller <sbeller@google.com> wrote:
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
> >  object-store.h |  3 +--
> >  sha1_file.c    | 21 +++++++++++----------
> >  2 files changed, 12 insertions(+), 12 deletions(-)
> >
> > diff --git a/object-store.h b/object-store.h
> > index d96a16edd1..add1d4e27c 100644
> > --- a/object-store.h
> > +++ b/object-store.h
> > @@ -61,7 +61,6 @@ struct packed_git {
> >         char pack_name[FLEX_ARRAY]; /* more */
> >  };
> >
> > -#define prepare_alt_odb(r) prepare_alt_odb_##r()
> > -extern void prepare_alt_odb_the_repository(void);
> > +void prepare_alt_odb(struct repository *r);
> >
> >  #endif /* OBJECT_STORE_H */
> > diff --git a/sha1_file.c b/sha1_file.c
> > index d18ce3aeba..f046d560f8 100644
> > --- a/sha1_file.c
> > +++ b/sha1_file.c
> > @@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
> >         return r;
> >  }
> >
> > -void prepare_alt_odb_the_repository(void)
> > +void prepare_alt_odb(struct repository *r)
> >  {
> > -       const char *alt;
> > -
> > -       if (the_repository->objects.alt_odb_tail)
> > +       if (r->objects.alt_odb_tail)
> >                 return;
> >
> > -       alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> > +       r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> > +
> > +       if (!r->ignore_env) {
> > +               const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> 
> If one day the majority of git moves to use 'struct repository', then
> ALTERNATE_DB_ENVIRONMENT is always ignored because ignore_env is
> always true. I think if you ignore_env, then you still need to get
> this "alt" from  'struct raw_object_store' (or 'struct repository').
> 
> Since you get lots of getenv() in repo_setup_env(), I think this
> getenv(ALTERNATE_DB_ENVIRONMENT) belongs there too. Then here, if
> ignore_env is true, you read r->objects.alt or something instead of
> doing getenv().
> 
> I really want to kill this getenv() in this code, which is basically
> delayed initialization because we haven't done proper init on
> the_repo. I realize that it cannot be done earlier, when
> prepare_alt_odb() does not even have a  'struct repository *' to work
> with. Would it be ok if I contributed a patch on top of your series to
> basically do repo_init(&the_repo) for all builtin/external commands
> (and fix all the bugs that come with it)? Then we would not need
> ignore_env here anymore.

At some point yes we would definitely want the setup code to fully
initialize a repository object (in this case the_repository).  And I
would even like to change the function signatures of all the builtin
commands to take a repository object so that they don't implicitly rely
on the_repository at all.

When I introduced struct repository I seem to remember there being a
couple things which were different about setup that made it difficult to
simply call repo_init() on the_repository.  If you can fix whatever
those issues with setup were (I can't remember all of them) then that
would be great :)

> 
> > +               if (!alt)
> > +                       alt = "";
> >
> > -       the_repository->objects.alt_odb_tail =
> > -                       &the_repository->objects.alt_odb_list;
> > -       link_alt_odb_entries(the_repository, alt,
> > -                            PATH_SEP, NULL, 0);
> > +               link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> > +       }
> >
> > -       read_info_alternates(the_repository, get_object_directory(), 0);
> > +       read_info_alternates(r, r->objects.objectdir, 0);
> >  }
> >
> >  /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
> > --
> > 2.16.1.73.ga2c3e9663f.dirty
> >
> 
> 
> 
> -- 
> Duy

-- 
Brandon Williams

[PATCH 0/2] Fix initializing the_hash_algo

[Nguyễn Thái Ngọc Duy]

On Thu, Feb 15, 2018 at 1:08 AM, Brandon Williams <bmwill@google.com> wrote:
> At some point yes we would definitely want the setup code to fully
> initialize a repository object (in this case the_repository).  And I
> would even like to change the function signatures of all the builtin
> commands to take a repository object so that they don't implicitly rely
> on the_repository at all.
>
> When I introduced struct repository I seem to remember there being a
> couple things which were different about setup that made it difficult to
> simply call repo_init() on the_repository.  If you can fix whatever
> those issues with setup were (I can't remember all of them) then that
> would be great :)

I can certainly try! I start to remember all the hairy details in that
setup code.

The first step may be something like this, which identifies all the
"repo init" entry points. This is basically a revert of e26f7f19b6
(repository: pre-initialize hash algo pointer - 2018-01-19) and doing
things the proper way, hopefully.

This is on 'master', independent from Stefan's series. I have another
patch on top of that series to remove the use of ignore_env in
sha1_file.c (and things seem to work). Basically whenever you have to
initialize the hash algorithm, there's a good chance you need to
initialize object store as well. But I'll hold that off until
Stefan's and this one are both merged.

But yeah, it looks like we need some surgery in setup.c if we want
something as pretty as repo_submodule_init() but for the main repo.

Nguyễn Thái Ngọc Duy (2):
  setup.c: initialize the_repository correctly in all cases
  Revert "repository: pre-initialize hash algo pointer"

 builtin/index-pack.c             | 5 +++++
 builtin/init-db.c                | 3 ++-
 cache.h                          | 3 ++-
 common-main.c                    | 4 ++++
 diff-no-index.c                  | 5 +++++
 path.c                           | 2 +-
 repository.c                     | 2 +-
 setup.c                          | 5 ++++-
 t/helper/test-dump-split-index.c | 2 ++
 9 files changed, 26 insertions(+), 5 deletions(-)

-- 
2.16.1.435.g8f24da2e1a

[PATCH 1/2] setup.c: initialize the_repository correctly in all cases

[Nguyễn Thái Ngọc Duy]

There are many ways for any command to access a git repository:

- most of them will try to discover the .git dir via
  setup_git_directory() and friends

- the server side programs already know where the repo is and prepare
  with enter_repo()

- special commands that deal with repo creation (init/clone) use
  init_db() once the new repo is ready for access.

- somebody accesses $GIT_DIR before any of above functions are called
  and accidentally sets up a git repository by set_git_dir() alone

"the_repository" is partially set up via set_git_dir() at some point
in all four cases. The hash algorithm though is configured later after
.git/config is read.

So far proper repo initialization is done only for the first case [1].
The second case is not covered (but that's fine [3]). The third case
was found and worked around in [2]. The fourth case is a buggy one,
which should be fixed already by jk/no-looking-at-dotgit-outside-repo
and never happens again.

This patch makes sure all cases initialize the hash algorithm in
the_repository correctly. Both second and third cases must run
check_repo_format() before "entering" it. Eventually we probably just
rename this function to init_repo() or something.

[1] 78a6766802 (Integrate hash algorithm support with repo setup -
    2017-11-12)

[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
    2018-01-19)

[3] the reason server side is still running ok with no hash algo
    before [2] is because the programs that use enter_repo() do very
    little then spawn a new program (like pack-objects or
    upload-archive) to do the heavy lifting. These programs already
    use setup_git_dir..()

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/init-db.c | 3 ++-
 cache.h           | 3 ++-
 path.c            | 2 +-
 setup.c           | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index c9b7946bad..d119d9906b 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -9,6 +9,7 @@
 #include "builtin.h"
 #include "exec_cmd.h"
 #include "parse-options.h"
+#include "repository.h"
 
 #ifndef DEFAULT_GIT_TEMPLATE_DIR
 #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	 * config file, so this will not fail.  What we are catching
 	 * is an attempt to reinitialize new repository with an old tool.
 	 */
-	check_repository_format();
+	check_repository_format(the_repository);
 
 	reinit = create_default_files(template_dir, original_git_dir);
 
diff --git a/cache.h b/cache.h
index 21fbcc2414..6b97138264 100644
--- a/cache.h
+++ b/cache.h
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;
 extern char *repository_format_partial_clone;
 extern const char *core_partial_clone_filter_default;
 
+struct repository;
 struct repository_format {
 	int version;
 	int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,
  * set_git_dir() before calling this, and use it only for "are we in a valid
  * repo?".
  */
-extern void check_repository_format(void);
+extern void check_repository_format(struct repository *);
 
 #define MTIME_CHANGED	0x0001
 #define CTIME_CHANGED	0x0002
diff --git a/path.c b/path.c
index da8b655730..a544252198 100644
--- a/path.c
+++ b/path.c
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)
 
 	if (is_git_directory(".")) {
 		set_git_dir(".");
-		check_repository_format();
+		check_repository_format(the_repository);
 		return path;
 	}
 
diff --git a/setup.c b/setup.c
index c5d55dcee4..a82103832e 100644
--- a/setup.c
+++ b/setup.c
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)
 	return -(i & 0666);
 }
 
-void check_repository_format(void)
+/* optionally configure "repo" to the correct format */
+void check_repository_format(struct repository *repo)
 {
 	struct repository_format repo_fmt;
 	check_repository_format_gently(get_git_dir(), &repo_fmt, NULL);
 	startup_info->have_repository = 1;
+	if (repo)
+		repo_set_hash_algo(repo, repo_fmt.hash_algo);
 }
 
 /*
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH 1/2] setup.c: initialize the_repository correctly in all cases

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

On Fri, Feb 23, 2018 at 04:56:39PM +0700, Nguyễn Thái Ngọc Duy wrote:
> [3] the reason server side is still running ok with no hash algo
>     before [2] is because the programs that use enter_repo() do very
>     little then spawn a new program (like pack-objects or
>     upload-archive) to do the heavy lifting. These programs already
>     use setup_git_dir..()

You have "..()" here.  Did you want to say "()." instead?
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH 1/2] setup.c: initialize the_repository correctly in all cases

[Duy Nguyen]

On Sat, Feb 24, 2018 at 5:17 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Fri, Feb 23, 2018 at 04:56:39PM +0700, Nguyễn Thái Ngọc Duy wrote:
>> [3] the reason server side is still running ok with no hash algo
>>     before [2] is because the programs that use enter_repo() do very
>>     little then spawn a new program (like pack-objects or
>>     upload-archive) to do the heavy lifting. These programs already
>>     use setup_git_dir..()
>
> You have "..()" here.  Did you want to say "()." instead?

It's because we have two functions, setup_git_directory() and
setup_git_directory_gently() and I tend to just put '*' instead of
listing both. But since it causes confusion, I'm listing both in v2.
-- 
Duy

[PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Nguyễn Thái Ngọc Duy]

This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
problem, git clone not setting up the_hash_algo, has been fixed in the
previous patch.

As a result of the revert, some code paths that use the_hash_algo
without initialization is revealed and fixed here. It's basically
commands that are allowed to run without a repository. The fix here is
not the best. We probably could figure out the hash algorithm from input
somehow.

Since this is a dangerous move and could potentially break stuff after
release (and leads to workaround like the reverted commit), the
workaround technically remains, but is hidden behind a new environment
variable GIT_HASH_FIXUP. This should let the users continue to use git
while we fix the problem.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/index-pack.c             | 5 +++++
 common-main.c                    | 4 ++++
 diff-no-index.c                  | 5 +++++
 repository.c                     | 2 +-
 t/helper/test-dump-split-index.c | 2 ++
 5 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 7e3e1a461c..8ee935504e 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	if (prefix && chdir(prefix))
 		die(_("Cannot come back to cwd"));
 
+	if (!the_hash_algo) {
+		warning(_("Running without a repository, assuming SHA-1 hash"));
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+	}
+
 	for (i = 1; i < argc; i++) {
 		const char *arg = argv[i];
 
diff --git a/common-main.c b/common-main.c
index 6a689007e7..12aec36794 100644
--- a/common-main.c
+++ b/common-main.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "exec_cmd.h"
 #include "attr.h"
+#include "repository.h"
 
 /*
  * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,8 @@ int main(int argc, const char **argv)
 
 	restore_sigpipe_to_default();
 
+	if (getenv("GIT_HASH_FIXUP"))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+
 	return cmd_main(argc, argv);
 }
diff --git a/diff-no-index.c b/diff-no-index.c
index 0ed5f0f496..f038f665bc 100644
--- a/diff-no-index.c
+++ b/diff-no-index.c
@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
 	struct strbuf replacement = STRBUF_INIT;
 	const char *prefix = revs->prefix;
 
+	if (!the_hash_algo) {
+		warning(_("Running without a repository, assuming SHA-1 hash"));
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+	}
+
 	diff_setup(&revs->diffopt);
 	for (i = 1; i < argc - 2; ) {
 		int j;
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..0d715f4fdb 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
 };
 struct repository *the_repository = &the_repo;
 
diff --git a/t/helper/test-dump-split-index.c b/t/helper/test-dump-split-index.c
index e44430b699..b759fe3fc1 100644
--- a/t/helper/test-dump-split-index.c
+++ b/t/helper/test-dump-split-index.c
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)
 	struct split_index *si;
 	int i;
 
+	setup_git_directory();
+
 	do_read_index(&the_index, av[1], 1);
 	printf("own %s\n", sha1_to_hex(the_index.sha1));
 	si = the_index.split_index;
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Stefan Beller]

On Fri, Feb 23, 2018 at 1:56 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
> problem, git clone not setting up the_hash_algo, has been fixed in the
> previous patch.
>
> As a result of the revert, some code paths that use the_hash_algo
> without initialization is revealed and fixed here. It's basically
> commands that are allowed to run without a repository. The fix here is
> not the best. We probably could figure out the hash algorithm from input
> somehow.
>
> Since this is a dangerous move and could potentially break stuff after
> release (and leads to workaround like the reverted commit), the
> workaround technically remains, but is hidden behind a new environment
> variable GIT_HASH_FIXUP. This should let the users continue to use git
> while we fix the problem.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  builtin/index-pack.c             | 5 +++++
>  common-main.c                    | 4 ++++
>  diff-no-index.c                  | 5 +++++
>  repository.c                     | 2 +-
>  t/helper/test-dump-split-index.c | 2 ++
>  5 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 7e3e1a461c..8ee935504e 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
>         if (prefix && chdir(prefix))
>                 die(_("Cannot come back to cwd"));
>
> +       if (!the_hash_algo) {
> +               warning(_("Running without a repository, assuming SHA-1 hash"));
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +       }
> +
>         for (i = 1; i < argc; i++) {
>                 const char *arg = argv[i];
>
> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..12aec36794 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
>
>         restore_sigpipe_to_default();
>
> +       if (getenv("GIT_HASH_FIXUP"))
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +
>         return cmd_main(argc, argv);
>  }
> diff --git a/diff-no-index.c b/diff-no-index.c
> index 0ed5f0f496..f038f665bc 100644
> --- a/diff-no-index.c
> +++ b/diff-no-index.c
> @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
>         struct strbuf replacement = STRBUF_INIT;
>         const char *prefix = revs->prefix;
>
> +       if (!the_hash_algo) {
> +               warning(_("Running without a repository, assuming SHA-1 hash"));
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +       }
> +
>         diff_setup(&revs->diffopt);
>         for (i = 1; i < argc - 2; ) {
>                 int j;
> diff --git a/repository.c b/repository.c
> index 4ffbe9bc94..0d715f4fdb 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -5,7 +5,7 @@
>
>  /* The main repository */
>  static struct repository the_repo = {
> -       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
> +       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0

I am not sure I agree with this defense in depth, because it would add
a lot to maintenance burden.
Specifically this part. The series that I sent out usually clashes
here as this is currently a hot area
of the code touched by many different series in flight.

However this is the long term correct thing to do? We assume no algorithm until
the repository can tell us from its config (or we default to sha1 if there is no
configuration present).

I wonder if there is yet another missing case in the enumeration of
the previous patch:
Some commands are able to operate on GIT_OBJECT_DIR instead
of GIT_DIR (git repack?), which may not even explore the full git directory,
and so doesn't know about the hash value.

In the cover letter you reference my series, but comparing the diffstats
(and looking through the patches), I would only expect this one place
to have merge conflicts, which ought to be easy to resolve.
(In my series I break the initializer into multiple lines to help the
future, too)

After some thought, I like this series.

Thanks,
Stefan

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> I wonder if there is yet another missing case in the enumeration of
> the previous patch:
> Some commands are able to operate on GIT_OBJECT_DIR instead
> of GIT_DIR (git repack?), which may not even explore the full git directory,
> and so doesn't know about the hash value.

... because GIT_DIR/config is not known?  "repack" is not one of
them, though---it needs to at least use refs as the starting point
so a standalone OBJECT_DIR is insufficient.

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Stefan Beller]

On Fri, Feb 23, 2018 at 11:50 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> I wonder if there is yet another missing case in the enumeration of
>> the previous patch:
>> Some commands are able to operate on GIT_OBJECT_DIR instead
>> of GIT_DIR (git repack?), which may not even explore the full git directory,
>> and so doesn't know about the hash value.
>
> ... because GIT_DIR/config is not known?  "repack" is not one of
> them, though---it needs to at least use refs as the starting point
> so a standalone OBJECT_DIR is insufficient.

Yes, I could have worded this as a question:
Is there any command that operates on GIT_OBJECT_DIR
without trying to discover GIT_DIR ?

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Fri, Feb 23, 2018 at 11:50 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Stefan Beller <sbeller@google.com> writes:
>>
>>> I wonder if there is yet another missing case in the enumeration of
>>> the previous patch:
>>> Some commands are able to operate on GIT_OBJECT_DIR instead
>>> of GIT_DIR (git repack?), which may not even explore the full git directory,
>>> and so doesn't know about the hash value.
>>
>> ... because GIT_DIR/config is not known?  "repack" is not one of
>> them, though---it needs to at least use refs as the starting point
>> so a standalone OBJECT_DIR is insufficient.
>
> Yes, I could have worded this as a question:
> Is there any command that operates on GIT_OBJECT_DIR
> without trying to discover GIT_DIR ?

If somebody finds one that would be a good argument not to pursue
the approach.  Lack of response to the question would not amount to
that much---it is possible all people who bothered to think of
overlooked something obvious, though.  "When I asked around nobody
thought of a possibly way for this to cause breakages, so let's
declare it is safe to do so and do it" is not how we want to do
things X-<.

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

On Fri, Feb 23, 2018 at 04:56:40PM +0700, Nguyễn Thái Ngọc Duy wrote:
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 7e3e1a461c..8ee935504e 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
>  	if (prefix && chdir(prefix))
>  		die(_("Cannot come back to cwd"));
>  
> +	if (!the_hash_algo) {
> +		warning(_("Running without a repository, assuming SHA-1 hash"));
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +	}

Is this warning going to be visible to users in the normal course of
operation?  If so, people are probably going to find this bothersome or
alarming.

>  	for (i = 1; i < argc; i++) {
>  		const char *arg = argv[i];
>  
> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..12aec36794 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>  
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
>  
>  	restore_sigpipe_to_default();
>  
> +	if (getenv("GIT_HASH_FIXUP"))
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);

I'm lukewarm on adding this environment variable, but considering our
history here, we had probably better.  We can always remove it after a
few releases.

>  	return cmd_main(argc, argv);
>  }
> diff --git a/diff-no-index.c b/diff-no-index.c
> index 0ed5f0f496..f038f665bc 100644
> --- a/diff-no-index.c
> +++ b/diff-no-index.c
> @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
>  	struct strbuf replacement = STRBUF_INIT;
>  	const char *prefix = revs->prefix;
>  
> +	if (!the_hash_algo) {
> +		warning(_("Running without a repository, assuming SHA-1 hash"));
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +	}

Again, same concern.  I can imagine scripts that will blow up loudly if
git diff --no-index spews things to standard error.

I'm not opposed to making this more visible, but I wonder if maybe it
should only be visible to developers or such.  The only way I can think
of doing is that is with an advice options, but maybe there's a better
way.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Brandon Williams]

On 02/23, brian m. carlson wrote:
> On Fri, Feb 23, 2018 at 04:56:40PM +0700, Nguyễn Thái Ngọc Duy wrote:
> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> > index 7e3e1a461c..8ee935504e 100644
> > --- a/builtin/index-pack.c
> > +++ b/builtin/index-pack.c
> > @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
> >  	if (prefix && chdir(prefix))
> >  		die(_("Cannot come back to cwd"));
> >  
> > +	if (!the_hash_algo) {
> > +		warning(_("Running without a repository, assuming SHA-1 hash"));
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> > +	}
> 
> Is this warning going to be visible to users in the normal course of
> operation?  If so, people are probably going to find this bothersome or
> alarming.
> 
> >  	for (i = 1; i < argc; i++) {
> >  		const char *arg = argv[i];
> >  
> > diff --git a/common-main.c b/common-main.c
> > index 6a689007e7..12aec36794 100644
> > --- a/common-main.c
> > +++ b/common-main.c
> > @@ -1,6 +1,7 @@
> >  #include "cache.h"
> >  #include "exec_cmd.h"
> >  #include "attr.h"
> > +#include "repository.h"
> >  
> >  /*
> >   * Many parts of Git have subprograms communicate via pipe, expect the
> > @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
> >  
> >  	restore_sigpipe_to_default();
> >  
> > +	if (getenv("GIT_HASH_FIXUP"))
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> 
> I'm lukewarm on adding this environment variable, but considering our
> history here, we had probably better.  We can always remove it after a
> few releases.

Yeah I don't know what the best thing to do here would be.  I mean we
could always just init the hash_algo for the_repository here so that we
don't ever have to worry about it, but I don't know if even that is the
right approach.

> 
> >  	return cmd_main(argc, argv);
> >  }
> > diff --git a/diff-no-index.c b/diff-no-index.c
> > index 0ed5f0f496..f038f665bc 100644
> > --- a/diff-no-index.c
> > +++ b/diff-no-index.c
> > @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
> >  	struct strbuf replacement = STRBUF_INIT;
> >  	const char *prefix = revs->prefix;
> >  
> > +	if (!the_hash_algo) {
> > +		warning(_("Running without a repository, assuming SHA-1 hash"));
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> > +	}
> 
> Again, same concern.  I can imagine scripts that will blow up loudly if
> git diff --no-index spews things to standard error.
> 
> I'm not opposed to making this more visible, but I wonder if maybe it
> should only be visible to developers or such.  The only way I can think
> of doing is that is with an advice options, but maybe there's a better
> way.
> -- 
> brian m. carlson / brian with sandals: Houston, Texas, US
> https://www.crustytoothpaste.net/~bmc | My opinion only
> OpenPGP: https://keybase.io/bk2204



-- 
Brandon Williams

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Duy Nguyen]

On Sat, Feb 24, 2018 at 5:29 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>> @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
>>
>>       restore_sigpipe_to_default();
>>
>> +     if (getenv("GIT_HASH_FIXUP"))
>> +             repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
>
> I'm lukewarm on adding this environment variable, but considering our
> history here, we had probably better.  We can always remove it after a
> few releases.

Yes that's the intention. But after writing cover letter for v2 and
sending it out, it looks to me that this thing must stay until all our
code is converted to using the_hash_algo (I don't know if there are
more to convert or it's finished already). So an alternative is we do
the opposite: default to GIT_HASH_SHA1, but when an env variable is
set, reset it back to NULL. This env variable will _always_ be set by
the test suite to help us catch problems.
-- 
Duy

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> Yes that's the intention. But after writing cover letter for v2 and
> sending it out, it looks to me that this thing must stay until all our
> code is converted to using the_hash_algo (I don't know if there are
> more to convert or it's finished already). So an alternative is we do
> the opposite: default to GIT_HASH_SHA1, but when an env variable is
> set, reset it back to NULL. This env variable will _always_ be set by
> the test suite to help us catch problems.

If we were to do the "do not blindly initialize and always
initialize explicitly for debugging" (which I doubt is a good
direction to go in), then what you outlined above certainly is a
less evil way to do so than what the patch under discussion did, I
would think.

Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"

[Duy Nguyen]

On Tue, Feb 27, 2018 at 6:09 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
>
>> Yes that's the intention. But after writing cover letter for v2 and
>> sending it out, it looks to me that this thing must stay until all our
>> code is converted to using the_hash_algo (I don't know if there are
>> more to convert or it's finished already). So an alternative is we do
>> the opposite: default to GIT_HASH_SHA1, but when an env variable is
>> set, reset it back to NULL. This env variable will _always_ be set by
>> the test suite to help us catch problems.
>
> If we were to do the "do not blindly initialize and always
> initialize explicitly for debugging" (which I doubt is a good
> direction to go in), then what you outlined above certainly is a
> less evil way to do so than what the patch under discussion did, I
> would think.

So v3 [1] should be in good condition to go to 'pu', I think?

[1] https://public-inbox.org/git/20180225111840.16421-1-pclouds@gmail.com/
-- 
Duy

Re: [PATCH 0/2] Fix initializing the_hash_algo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Fri, Feb 23, 2018 at 04:56:38PM +0700, Nguyễn Thái Ngọc Duy wrote:
> I can certainly try! I start to remember all the hairy details in that
> setup code.
> 
> The first step may be something like this, which identifies all the
> "repo init" entry points. This is basically a revert of e26f7f19b6
> (repository: pre-initialize hash algo pointer - 2018-01-19) and doing
> things the proper way, hopefully.
> 
> This is on 'master', independent from Stefan's series. I have another
> patch on top of that series to remove the use of ignore_env in
> sha1_file.c (and things seem to work). Basically whenever you have to
> initialize the hash algorithm, there's a good chance you need to
> initialize object store as well. But I'll hold that off until
> Stefan's and this one are both merged.

I definitely think this series is an improvement over my previous patch.

My major concern is alarming users (or breaking scripts) with the
warning message.  I wonder if deferring the use of the warning until we
have multiple hash algorithms might be a better idea.  At that point,
the warning would become something people could act upon.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH v2 0/5] Fix initializing the_hash_algo

[Nguyễn Thái Ngọc Duy]

Brian questioned the unnecessary alarms in v1 when "git diff" or "git
index-pack" run in no-repository mode. I had the same feeling after
sending v1. But instead of suppresing alarms, we could do better.

v2 breaks those "fall back to SHA-1" code into separate patches and
handles it properly (I hope) instead of blind fall back like v1:

- for index-pack, we can determine the needed algorithm from the pack
  file. I'm making an assumption here that pack files with new hash
  algo must step up file format version. But I think it's a reasonable
  assumption.

- for diff --no-index, I still fall back to SHA-1 to generate the
  hashes like before. We could probably introduce a new command line
  option to use a different hash. But that work could be done later
  when an actual new hash has come

Note, I didn't test but this series could potentially break 'pu' (in a
good way). I initially was puzzled why the test suite didn't fail when
the_repository->hash_algo was NULL (i.e. before Brian's fix). Then I
found out that more work to abstract away SHA-1 hashing functions have
been done since then. But because the_hash_algo is pre-initialized
with SHA-1, these special cases did not show up until now. If there
are more the_hash_algo conversion on 'pu', more "no-repo" cases
could be spotted by the test suite.

I think this makes pre-initializing the_hash_algo to NULL a very good
point: during the abstraction work, we at least must identify the use
cases like this (code running without repo). We don't have to fix it
right away, but we can start thinking about how to deal with it. If we
ignore it and switch to a new hash, these code will keep using SHA-1
and cause more headache in future.

Nguyễn Thái Ngọc Duy (5):
  setup.c: initialize the_repository correctly in all cases
  sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
  index-pack: check (and optionally set) hash algo based on input file
  diff.c: initialize hash algo when running in --no-index mode
  Revert "repository: pre-initialize hash algo pointer"

 builtin/index-pack.c             | 26 +++++++++++++++++++++++++-
 builtin/init-db.c                |  3 ++-
 cache.h                          |  3 ++-
 common-main.c                    | 10 ++++++++++
 diff.c                           | 12 ++++++++++++
 path.c                           |  2 +-
 repository.c                     |  2 +-
 setup.c                          |  5 ++++-
 sha1_file.c                      |  2 +-
 t/helper/test-dump-split-index.c |  2 ++
 10 files changed, 60 insertions(+), 7 deletions(-)

-- 
2.16.1.435.g8f24da2e1a

[PATCH v2 1/5] setup.c: initialize the_repository correctly in all cases

[Nguyễn Thái Ngọc Duy]

There are many ways for any command to access a git repository:

- most of them will try to discover the .git dir via
  setup_git_directory() and friends

- the server side programs already know where the repo is and prepare
  with enter_repo()

- special commands that deal with repo creation (init/clone) use
  init_db() once the new repo is ready for access.

- somebody accesses $GIT_DIR before any of above functions are called
  and accidentally sets up a git repository by set_git_dir() alone

"the_repository" is partially set up via set_git_dir() at some point
in all four cases. The hash algorithm though is configured later after
.git/config is read.

So far proper repo initialization is done only for the first case [1].
The second case is not covered (but that's fine [3]). The third case
was found and worked around in [2]. The fourth case is a buggy one,
which should be fixed already by jk/no-looking-at-dotgit-outside-repo
and never happens again.

This patch makes sure all cases initialize the hash algorithm in
the_repository correctly. Both second and third cases must run
check_repo_format() before "entering" it. Eventually we probably just
rename this function to init_repo() or something.

[1] 78a6766802 (Integrate hash algorithm support with repo setup -
    2017-11-12)

[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
    2018-01-19)

[3] the reason server side is still running ok with no hash algo before
    [2] is because the programs that use enter_repo() do very
    little (and unlikely to hash anything) then spawn a new
    program (like pack-objects or upload-archive) to do the heavy
    lifting. These programs already use setup_git_directory() or the
    gently version

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/init-db.c | 3 ++-
 cache.h           | 3 ++-
 path.c            | 2 +-
 setup.c           | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index c9b7946bad..d119d9906b 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -9,6 +9,7 @@
 #include "builtin.h"
 #include "exec_cmd.h"
 #include "parse-options.h"
+#include "repository.h"
 
 #ifndef DEFAULT_GIT_TEMPLATE_DIR
 #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	 * config file, so this will not fail.  What we are catching
 	 * is an attempt to reinitialize new repository with an old tool.
 	 */
-	check_repository_format();
+	check_repository_format(the_repository);
 
 	reinit = create_default_files(template_dir, original_git_dir);
 
diff --git a/cache.h b/cache.h
index 21fbcc2414..6b97138264 100644
--- a/cache.h
+++ b/cache.h
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;
 extern char *repository_format_partial_clone;
 extern const char *core_partial_clone_filter_default;
 
+struct repository;
 struct repository_format {
 	int version;
 	int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,
  * set_git_dir() before calling this, and use it only for "are we in a valid
  * repo?".
  */
-extern void check_repository_format(void);
+extern void check_repository_format(struct repository *);
 
 #define MTIME_CHANGED	0x0001
 #define CTIME_CHANGED	0x0002
diff --git a/path.c b/path.c
index da8b655730..a544252198 100644
--- a/path.c
+++ b/path.c
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)
 
 	if (is_git_directory(".")) {
 		set_git_dir(".");
-		check_repository_format();
+		check_repository_format(the_repository);
 		return path;
 	}
 
diff --git a/setup.c b/setup.c
index c5d55dcee4..a82103832e 100644
--- a/setup.c
+++ b/setup.c
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)
 	return -(i & 0666);
 }
 
-void check_repository_format(void)
+/* optionally configure "repo" to the correct format */
+void check_repository_format(struct repository *repo)
 {
 	struct repository_format repo_fmt;
 	check_repository_format_gently(get_git_dir(), &repo_fmt, NULL);
 	startup_info->have_repository = 1;
+	if (repo)
+		repo_set_hash_algo(repo, repo_fmt.hash_algo);
 }
 
 /*
-- 
2.16.1.435.g8f24da2e1a

[PATCH v2 2/5] sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]

[Nguyễn Thái Ngọc Duy]

This is mostly for displaying the hash algorithm name when we report
errors. Printing "unknown" with '%s' is much better than '(null)' in
glibc printf version (and probably could crash if other implementations
do not check for NULL pointer)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 sha1_file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sha1_file.c b/sha1_file.c
index 826d7a0ae3..4b266b1c68 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -71,7 +71,7 @@ static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)
 
 const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 	{
-		NULL,
+		"unknown",
 		0x00000000,
 		0,
 		0,
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 2/5] sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Sat, Feb 24, 2018 at 10:34:26AM +0700, Nguyễn Thái Ngọc Duy wrote:
> This is mostly for displaying the hash algorithm name when we report
> errors. Printing "unknown" with '%s' is much better than '(null)' in
> glibc printf version (and probably could crash if other implementations
> do not check for NULL pointer)

This is a good change.  Using a NULL pointer with %s on the BSDs
definitely did cause a crash, last I checked.

I looked at the comment I wrote in hash.h, and based on that, it is
indeed very likely that someone would accidentally write code this way.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH v2 3/5] index-pack: check (and optionally set) hash algo based on input file

[Nguyễn Thái Ngọc Duy]

After 454253f059 (builtin/index-pack: improve hash function abstraction
- 2018-02-01), index-pack uses the_hash_algo for hashing. If "git
index-pack" is executed without a repository, we do not know what hash
algorithm to be used and the_hash_algo in theory could be undefined.

Since there should be some information about the hash algorithm in the
input pack file, we can initialize the correct hash algorithm with that
if the_hash_algo is not yet initialized. This assumes that pack files
with new hash algorithm MUST step up pack version.

While at there, make sure the hash algorithm requested by the pack file
and configured by the repository (if we're running with a repo) are
consistent.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/index-pack.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 7e3e1a461c..1144458140 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -326,10 +326,31 @@ static const char *open_pack_file(const char *pack_name)
 		output_fd = -1;
 		nothread_data.pack_fd = input_fd;
 	}
-	the_hash_algo->init_fn(&input_ctx);
 	return pack_name;
 }
 
+static void prepare_hash_algo(uint32_t pack_version)
+{
+	const struct git_hash_algo *pack_algo;
+
+	switch (pack_version) {
+	case 2:
+	case 3:
+		pack_algo = &hash_algos[GIT_HASH_SHA1];
+		break;
+	default:
+		die("BUG: how to determine hash algo for new version?");
+	}
+
+	if (!the_hash_algo)	/* running without repo */
+		the_hash_algo = pack_algo;
+
+	if (the_hash_algo != pack_algo)
+		die(_("incompatible hash algorithm, "
+		      "configured for %s but the pack file needs %s"),
+		    the_hash_algo->name, pack_algo->name);
+}
+
 static void parse_pack_header(void)
 {
 	struct pack_header *hdr = fill(sizeof(struct pack_header));
@@ -341,6 +362,9 @@ static void parse_pack_header(void)
 		die(_("pack version %"PRIu32" unsupported"),
 			ntohl(hdr->hdr_version));
 
+	prepare_hash_algo(ntohl(hdr->hdr_version));
+	the_hash_algo->init_fn(&input_ctx);
+
 	nr_objects = ntohl(hdr->hdr_entries);
 	use(sizeof(struct pack_header));
 }
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 3/5] index-pack: check (and optionally set) hash algo based on input file

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

On Sat, Feb 24, 2018 at 10:34:27AM +0700, Nguyễn Thái Ngọc Duy wrote:
> After 454253f059 (builtin/index-pack: improve hash function abstraction
> - 2018-02-01), index-pack uses the_hash_algo for hashing. If "git
> index-pack" is executed without a repository, we do not know what hash
> algorithm to be used and the_hash_algo in theory could be undefined.
> 
> Since there should be some information about the hash algorithm in the
> input pack file, we can initialize the correct hash algorithm with that
> if the_hash_algo is not yet initialized. This assumes that pack files
> with new hash algorithm MUST step up pack version.
> 
> While at there, make sure the hash algorithm requested by the pack file
> and configured by the repository (if we're running with a repo) are
> consistent.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  builtin/index-pack.c | 26 +++++++++++++++++++++++++-
>  1 file changed, 25 insertions(+), 1 deletion(-)
> 
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 7e3e1a461c..1144458140 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -326,10 +326,31 @@ static const char *open_pack_file(const char *pack_name)
>  		output_fd = -1;
>  		nothread_data.pack_fd = input_fd;
>  	}
> -	the_hash_algo->init_fn(&input_ctx);
>  	return pack_name;
>  }
>  
> +static void prepare_hash_algo(uint32_t pack_version)
> +{
> +	const struct git_hash_algo *pack_algo;
> +
> +	switch (pack_version) {
> +	case 2:
> +	case 3:
> +		pack_algo = &hash_algos[GIT_HASH_SHA1];
> +		break;
> +	default:
> +		die("BUG: how to determine hash algo for new version?");
> +	}
> +
> +	if (!the_hash_algo)	/* running without repo */
> +		the_hash_algo = pack_algo;
> +
> +	if (the_hash_algo != pack_algo)
> +		die(_("incompatible hash algorithm, "
> +		      "configured for %s but the pack file needs %s"),
> +		    the_hash_algo->name, pack_algo->name);

I like this.  It's a nice improvement and it should be easy for us to
pass additional information into the function when our pack format
understands multiple algorithms.

I might have done the comparison using the format_id members instead of
the pointers themselves, but that's more a personal preference than
anything.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode

[Nguyễn Thái Ngọc Duy]

Our "git diff" command supports running as a standalone tool. In this
code path, we try to hash the file content but after
18e2588e11 (sha1_file: switch uses of SHA-1 to the_hash_algo -
2018-02-01), there is a chance that the_hash_algo (required by
index_path) may still be uninitialized if no repository is found.

Executing index_path() when the_hash_algo is NULL (or points to unknown
algo) either crashes or dies. Let's make it a bit safer by explicitly
falling back to SHA-1 (so that the diff output remains the same as
before, compared to the alternative that we simply do not hash).

dòng được

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 diff.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/diff.c b/diff.c
index 21c3838b25..f5755425b6 100644
--- a/diff.c
+++ b/diff.c
@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
 		return;
 	}
 
+	/*
+	 * NEEDSWORK: When running in no-index mode (and no repo is
+	 * found, thus no hash algo conifugred), fall back to SHA-1
+	 * hashing (which is used by diff_fill_oid_info below) to
+	 * avoid regression in diff output.
+	 *
+	 * In future, perhaps we can allow the user to specify their
+	 * hash algorithm from command line in this mode.
+	 */
+	if (o->flags.no_index && !the_hash_algo)
+		the_hash_algo = &hash_algos[GIT_HASH_SHA1];
+
 	diff_fill_oid_info(one);
 	diff_fill_oid_info(two);
 
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode

[Eric Sunshine]

On Fri, Feb 23, 2018 at 10:34 PM, Nguyễn Thái Ngọc Duy
<pclouds@gmail.com> wrote:
> Our "git diff" command supports running as a standalone tool. In this
> code path, we try to hash the file content but after
> 18e2588e11 (sha1_file: switch uses of SHA-1 to the_hash_algo -
> 2018-02-01), there is a chance that the_hash_algo (required by
> index_path) may still be uninitialized if no repository is found.
>
> Executing index_path() when the_hash_algo is NULL (or points to unknown
> algo) either crashes or dies. Let's make it a bit safer by explicitly
> falling back to SHA-1 (so that the diff output remains the same as
> before, compared to the alternative that we simply do not hash).
>
> dòng được

Accidental paste?

> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> diff --git a/diff.c b/diff.c
> @@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
> +       /*
> +        * NEEDSWORK: When running in no-index mode (and no repo is
> +        * found, thus no hash algo conifugred), fall back to SHA-1

s/conifugred/configured/

> +        * hashing (which is used by diff_fill_oid_info below) to
> +        * avoid regression in diff output.
> +        *
> +        * In future, perhaps we can allow the user to specify their
> +        * hash algorithm from command line in this mode.
> +        */
> +       if (o->flags.no_index && !the_hash_algo)
> +               the_hash_algo = &hash_algos[GIT_HASH_SHA1];
> +
>         diff_fill_oid_info(one);
>         diff_fill_oid_info(two);

Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode

[Duy Nguyen]

On Sat, Feb 24, 2018 at 3:15 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Fri, Feb 23, 2018 at 10:34 PM, Nguyễn Thái Ngọc Duy
> <pclouds@gmail.com> wrote:
>> before, compared to the alternative that we simply do not hash).
>>
>> dòng được
>
> Accidental paste?

Oops. I blame Emacs.
-- 
Duy

Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode

[Duy Nguyen]

On Sat, Feb 24, 2018 at 10:34 AM, Nguyễn Thái Ngọc Duy
<pclouds@gmail.com> wrote:
> @@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
>                 return;
>         }
>
> +       /*
> +        * NEEDSWORK: When running in no-index mode (and no repo is
> +        * found, thus no hash algo conifugred), fall back to SHA-1
> +        * hashing (which is used by diff_fill_oid_info below) to
> +        * avoid regression in diff output.
> +        *
> +        * In future, perhaps we can allow the user to specify their
> +        * hash algorithm from command line in this mode.
> +        */
> +       if (o->flags.no_index && !the_hash_algo)
> +               the_hash_algo = &hash_algos[GIT_HASH_SHA1];

Brian, are we supposed to use the_hash_algo this way (i.e. as a
writable var)? Or should I stick to something like

    repo_set_hash_algo(the_repository, GIT_HASH_SHA1);

which allows us to notify other parts inside struct repository about
the hash algorithm change, if we ever need to?

If the_hash_algo is supposed to be read-only, maybe I should convert
that macro to an inline function to prevent people from accidentally
reassigning it?

> +
>         diff_fill_oid_info(one);
>         diff_fill_oid_info(two);
>
> --
> 2.16.1.435.g8f24da2e1a
>
-- 
Duy

Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

On Sat, Feb 24, 2018 at 09:36:03PM +0700, Duy Nguyen wrote:
> On Sat, Feb 24, 2018 at 10:34 AM, Nguyễn Thái Ngọc Duy
> <pclouds@gmail.com> wrote:
> > @@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
> >                 return;
> >         }
> >
> > +       /*
> > +        * NEEDSWORK: When running in no-index mode (and no repo is
> > +        * found, thus no hash algo conifugred), fall back to SHA-1
> > +        * hashing (which is used by diff_fill_oid_info below) to
> > +        * avoid regression in diff output.
> > +        *
> > +        * In future, perhaps we can allow the user to specify their
> > +        * hash algorithm from command line in this mode.
> > +        */
> > +       if (o->flags.no_index && !the_hash_algo)
> > +               the_hash_algo = &hash_algos[GIT_HASH_SHA1];
> 
> Brian, are we supposed to use the_hash_algo this way (i.e. as a
> writable var)? Or should I stick to something like
> 
>     repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> 
> which allows us to notify other parts inside struct repository about
> the hash algorithm change, if we ever need to?

I would definitely recommend using the function.  As you pointed out, it
makes our code future-proof against needing to more work when setting
the value.

> If the_hash_algo is supposed to be read-only, maybe I should convert
> that macro to an inline function to prevent people from accidentally
> reassigning it?

You could if you want to, although I don't really see a need to, since
we can just catch it in review.  If you wanted to, I'd make it an inline
function for performance reasons.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"

[Nguyễn Thái Ngọc Duy]

This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
problem, git clone not setting up the_hash_algo, has been fixed in the
previous patch.

Since this is a dangerous move and could potentially break stuff after
release (and leads to workaround like the reverted commit), the
workaround technically remains, but is hidden behind a new environment
variable GIT_HASH_FIXUP. This should let the users continue to use git
while we fix the problem. This variable can be deleted after one or two
releases.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 common-main.c                    | 10 ++++++++++
 repository.c                     |  2 +-
 t/helper/test-dump-split-index.c |  2 ++
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/common-main.c b/common-main.c
index 6a689007e7..fbfa98c3b8 100644
--- a/common-main.c
+++ b/common-main.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "exec_cmd.h"
 #include "attr.h"
+#include "repository.h"
 
 /*
  * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,14 @@ int main(int argc, const char **argv)
 
 	restore_sigpipe_to_default();
 
+	/*
+	 * Temporary recovery measure while hashing code is being
+	 * refactored. This variable should be gone after everybody
+	 * has used the_hash_algo in one or two releases and nobody
+	 * complains anything.
+	 */
+	if (getenv("GIT_HASH_FIXUP"))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+
 	return cmd_main(argc, argv);
 }
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..0d715f4fdb 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
 };
 struct repository *the_repository = &the_repo;
 
diff --git a/t/helper/test-dump-split-index.c b/t/helper/test-dump-split-index.c
index e44430b699..b759fe3fc1 100644
--- a/t/helper/test-dump-split-index.c
+++ b/t/helper/test-dump-split-index.c
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)
 	struct split_index *si;
 	int i;
 
+	setup_git_directory();
+
 	do_read_index(&the_index, av[1], 1);
 	printf("own %s\n", sha1_to_hex(the_index.sha1));
 	si = the_index.split_index;
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2489 bytes --]

On Sat, Feb 24, 2018 at 10:34:29AM +0700, Nguyễn Thái Ngọc Duy wrote:
> This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
> problem, git clone not setting up the_hash_algo, has been fixed in the
> previous patch.
> 
> Since this is a dangerous move and could potentially break stuff after
> release (and leads to workaround like the reverted commit), the
> workaround technically remains, but is hidden behind a new environment
> variable GIT_HASH_FIXUP. This should let the users continue to use git
> while we fix the problem. This variable can be deleted after one or two
> releases.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  common-main.c                    | 10 ++++++++++
>  repository.c                     |  2 +-
>  t/helper/test-dump-split-index.c |  2 ++
>  3 files changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..fbfa98c3b8 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>  
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -40,5 +41,14 @@ int main(int argc, const char **argv)
>  
>  	restore_sigpipe_to_default();
>  
> +	/*
> +	 * Temporary recovery measure while hashing code is being
> +	 * refactored. This variable should be gone after everybody
> +	 * has used the_hash_algo in one or two releases and nobody
> +	 * complains anything.
> +	 */
> +	if (getenv("GIT_HASH_FIXUP"))
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +
>  	return cmd_main(argc, argv);
>  }
> diff --git a/repository.c b/repository.c
> index 4ffbe9bc94..0d715f4fdb 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -5,7 +5,7 @@
>  
>  /* The main repository */
>  static struct repository the_repo = {
> -	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
> +	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0

I'm wondering, now that you have the name field for the unknown value,
if that might be a better choice here than NULL.  I don't have a strong
preference either way, so whatever you decide here is fine.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"

[Duy Nguyen]

On Sun, Feb 25, 2018 at 5:58 AM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>> diff --git a/repository.c b/repository.c
>> index 4ffbe9bc94..0d715f4fdb 100644
>> --- a/repository.c
>> +++ b/repository.c
>> @@ -5,7 +5,7 @@
>>
>>  /* The main repository */
>>  static struct repository the_repo = {
>> -     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
>> +     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
>
> I'm wondering, now that you have the name field for the unknown value,
> if that might be a better choice here than NULL.  I don't have a strong
> preference either way, so whatever you decide here is fine.

I did try that first, but for the purpose of catching uninitialized
algo use, NULL is better.

When I set unknown hash algo here, I think some test failed
mysteriously because it used rawsz field (which has value zero), it
didn't match some expectation, the code went to the error handling
path, which eventually failed with some error message, but it's not
obvious that the problem was rawsz being zero and back tracking that
took me some time.

With NULL hash_algo, any dereferencing fails immediately with a nice
stack trace. Another reason to push me towards NULL hash algo is, even
if we prefer nice messages over segmentation faults, we can't avoid it
completely anyway (empty_tree and empty_blob are still NULL in unknown
hash algo and will cause segfaults). Might as well make things
consistent and always segfault.
-- 
Duy

Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

On Sun, Feb 25, 2018 at 10:29:29AM +0700, Duy Nguyen wrote:
> When I set unknown hash algo here, I think some test failed
> mysteriously because it used rawsz field (which has value zero), it
> didn't match some expectation, the code went to the error handling
> path, which eventually failed with some error message, but it's not
> obvious that the problem was rawsz being zero and back tracking that
> took me some time.
> 
> With NULL hash_algo, any dereferencing fails immediately with a nice
> stack trace. Another reason to push me towards NULL hash algo is, even
> if we prefer nice messages over segmentation faults, we can't avoid it
> completely anyway (empty_tree and empty_blob are still NULL in unknown
> hash algo and will cause segfaults). Might as well make things
> consistent and always segfault.

That makes sense.  Thanks for the explanation.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

[PATCH v3 0/6] Fix initializing the_hash_algo

[Nguyễn Thái Ngọc Duy]

v3 refines v2 a bit more:

- fix configure typo (and stray words in commit message)
- use repo_set_hash_algo() instead of reassigning the_hash_algo
- compare hash algos by format_id
- catch NULL hash algo, report nicely and suggest GIT_HASH_FIXUP

The last point makes me much happier about keeping this workaround
around until we are confident we can live without it. Interdiff

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 1144458140..2c75f28b41 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -331,21 +331,24 @@ static const char *open_pack_file(const char *pack_name)
 
 static void prepare_hash_algo(uint32_t pack_version)
 {
+	int pack_algo_id;
 	const struct git_hash_algo *pack_algo;
 
 	switch (pack_version) {
 	case 2:
 	case 3:
-		pack_algo = &hash_algos[GIT_HASH_SHA1];
+		pack_algo_id = GIT_HASH_SHA1;
 		break;
 	default:
-		die("BUG: how to determine hash algo for new version?");
+		die("BUG: how to determine hash algo for version %d?",
+		    pack_version);
 	}
 
-	if (!the_hash_algo)	/* running without repo */
-		the_hash_algo = pack_algo;
+	if (!repo_has_valid_hash_algo(the_repository)) /* running without repo */
+		repo_set_hash_algo(the_repository, pack_algo_id);
 
-	if (the_hash_algo != pack_algo)
+	pack_algo = &hash_algos[pack_algo_id];
+	if (the_hash_algo->format_id != pack_algo->format_id)
 		die(_("incompatible hash algorithm, "
 		      "configured for %s but the pack file needs %s"),
 		    the_hash_algo->name, pack_algo->name);
diff --git a/cache.h b/cache.h
index 6b97138264..55b31e9756 100644
--- a/cache.h
+++ b/cache.h
@@ -53,7 +53,7 @@ struct object_id {
 	unsigned char hash[GIT_MAX_RAWSZ];
 };
 
-#define the_hash_algo the_repository->hash_algo
+#define the_hash_algo repo_get_hash_algo(the_repository)
 
 #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)
 #define DTYPE(de)	((de)->d_type)
diff --git a/diff.c b/diff.c
index f5755425b6..5f28d84b87 100644
--- a/diff.c
+++ b/diff.c
@@ -3997,15 +3997,15 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
 
 	/*
 	 * NEEDSWORK: When running in no-index mode (and no repo is
-	 * found, thus no hash algo conifugred), fall back to SHA-1
+	 * found, thus no hash algo configured), fall back to SHA-1
 	 * hashing (which is used by diff_fill_oid_info below) to
 	 * avoid regression in diff output.
 	 *
 	 * In future, perhaps we can allow the user to specify their
 	 * hash algorithm from command line in this mode.
 	 */
-	if (o->flags.no_index && !the_hash_algo)
-		the_hash_algo = &hash_algos[GIT_HASH_SHA1];
+	if (o->flags.no_index && !repo_has_valid_hash_algo(the_repository))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
 
 	diff_fill_oid_info(one);
 	diff_fill_oid_info(two);
diff --git a/repository.h b/repository.h
index 0329e40c7f..40bd49611f 100644
--- a/repository.h
+++ b/repository.h
@@ -107,4 +107,20 @@ extern void repo_clear(struct repository *repo);
  */
 extern int repo_read_index(struct repository *repo);
 
+static inline const struct git_hash_algo *repo_get_hash_algo(
+	const struct repository *repo)
+{
+	if (!repo->hash_algo)
+		die("BUG: hash_algo is not initialized!\n%s",
+		    _("You can work around this by setting environment"
+		      " variable GIT_HASH_FIXUP=1.\n"
+		      "Please report this to git@vger.kernel.org"));
+	return repo->hash_algo;
+}
+
+static inline int repo_has_valid_hash_algo(const struct repository *repo)
+{
+	return repo->hash_algo != NULL;
+}
+
 #endif /* REPOSITORY_H */

Nguyễn Thái Ngọc Duy (6):
  setup.c: initialize the_repository correctly in all cases
  sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
  cache.h: make the_hash_algo read-only
  index-pack: check (and optionally set) hash algo based on input file
  diff.c: initialize hash algo when running in --no-index mode
  Revert "repository: pre-initialize hash algo pointer"

 builtin/index-pack.c             | 29 ++++++++++++++++++++++++++++-
 builtin/init-db.c                |  3 ++-
 cache.h                          |  5 +++--
 common-main.c                    | 10 ++++++++++
 diff.c                           | 12 ++++++++++++
 path.c                           |  2 +-
 repository.c                     |  2 +-
 repository.h                     | 16 ++++++++++++++++
 setup.c                          |  5 ++++-
 sha1_file.c                      |  2 +-
 t/helper/test-dump-split-index.c |  2 ++
 11 files changed, 80 insertions(+), 8 deletions(-)

-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 1/6] setup.c: initialize the_repository correctly in all cases

[Nguyễn Thái Ngọc Duy]

There are many ways for any command to access a git repository:

- most of them will try to discover the .git dir via
  setup_git_directory() and friends

- the server side programs already know where the repo is and prepare
  with enter_repo()

- special commands that deal with repo creation (init/clone) use
  init_db() once the new repo is ready for access.

- somebody accesses $GIT_DIR before any of above functions are called
  and accidentally sets up a git repository by set_git_dir() alone

"the_repository" is partially set up via set_git_dir() at some point
in all four cases. The hash algorithm though is configured later after
.git/config is read.

So far proper repo initialization is done only for the first case [1].
The second case is not covered (but that's fine [3]). The third case
was found and worked around in [2]. The fourth case is a buggy one,
which should be fixed already by jk/no-looking-at-dotgit-outside-repo
and never happens again.

This patch makes sure all cases initialize the hash algorithm in
the_repository correctly. Both second and third cases must run
check_repo_format() before "entering" it. Eventually we probably just
rename this function to init_repo() or something.

[1] 78a6766802 (Integrate hash algorithm support with repo setup -
    2017-11-12)

[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
    2018-01-19)

[3] the reason server side is still running ok with no hash algo before
    [2] is because the programs that use enter_repo() do very
    little (and unlikely to hash anything) then spawn a new
    program (like pack-objects or upload-archive) to do the heavy
    lifting. These programs already use setup_git_directory() or the
    gently version

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/init-db.c | 3 ++-
 cache.h           | 3 ++-
 path.c            | 2 +-
 setup.c           | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index c9b7946bad..d119d9906b 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -9,6 +9,7 @@
 #include "builtin.h"
 #include "exec_cmd.h"
 #include "parse-options.h"
+#include "repository.h"
 
 #ifndef DEFAULT_GIT_TEMPLATE_DIR
 #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	 * config file, so this will not fail.  What we are catching
 	 * is an attempt to reinitialize new repository with an old tool.
 	 */
-	check_repository_format();
+	check_repository_format(the_repository);
 
 	reinit = create_default_files(template_dir, original_git_dir);
 
diff --git a/cache.h b/cache.h
index 21fbcc2414..6b97138264 100644
--- a/cache.h
+++ b/cache.h
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;
 extern char *repository_format_partial_clone;
 extern const char *core_partial_clone_filter_default;
 
+struct repository;
 struct repository_format {
 	int version;
 	int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,
  * set_git_dir() before calling this, and use it only for "are we in a valid
  * repo?".
  */
-extern void check_repository_format(void);
+extern void check_repository_format(struct repository *);
 
 #define MTIME_CHANGED	0x0001
 #define CTIME_CHANGED	0x0002
diff --git a/path.c b/path.c
index da8b655730..a544252198 100644
--- a/path.c
+++ b/path.c
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)
 
 	if (is_git_directory(".")) {
 		set_git_dir(".");
-		check_repository_format();
+		check_repository_format(the_repository);
 		return path;
 	}
 
diff --git a/setup.c b/setup.c
index c5d55dcee4..a82103832e 100644
--- a/setup.c
+++ b/setup.c
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)
 	return -(i & 0666);
 }
 
-void check_repository_format(void)
+/* optionally configure "repo" to the correct format */
+void check_repository_format(struct repository *repo)
 {
 	struct repository_format repo_fmt;
 	check_repository_format_gently(get_git_dir(), &repo_fmt, NULL);
 	startup_info->have_repository = 1;
+	if (repo)
+		repo_set_hash_algo(repo, repo_fmt.hash_algo);
 }
 
 /*
-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 2/6] sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]

[Nguyễn Thái Ngọc Duy]

This is mostly for displaying the hash algorithm name when we report
errors. Printing "unknown" with '%s' is much better than '(null)' in
glibc printf version (and probably could crash if other implementations
do not check for NULL pointer)

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 sha1_file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/sha1_file.c b/sha1_file.c
index 826d7a0ae3..4b266b1c68 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -71,7 +71,7 @@ static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)
 
 const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 	{
-		NULL,
+		"unknown",
 		0x00000000,
 		0,
 		0,
-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 3/6] cache.h: make the_hash_algo read-only

[Nguyễn Thái Ngọc Duy]

By returning an R-value in the_hash_algo we make sure people can't
accidentally change hash algorithm with

    the_hash_algo = &hash_algos[something];

and go with repo_set_hash_algo() instead. Of course they can still do

    the_repository->hash_algo = ...

but that is more obvious and easily caught in review.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h      | 2 +-
 repository.h | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/cache.h b/cache.h
index 6b97138264..55b31e9756 100644
--- a/cache.h
+++ b/cache.h
@@ -53,7 +53,7 @@ struct object_id {
 	unsigned char hash[GIT_MAX_RAWSZ];
 };
 
-#define the_hash_algo the_repository->hash_algo
+#define the_hash_algo repo_get_hash_algo(the_repository)
 
 #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)
 #define DTYPE(de)	((de)->d_type)
diff --git a/repository.h b/repository.h
index 0329e40c7f..5092df3700 100644
--- a/repository.h
+++ b/repository.h
@@ -107,4 +107,10 @@ extern void repo_clear(struct repository *repo);
  */
 extern int repo_read_index(struct repository *repo);
 
+static inline const struct git_hash_algo *repo_get_hash_algo(
+	const struct repository *repo)
+{
+	return repo->hash_algo;
+}
+
 #endif /* REPOSITORY_H */
-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 4/6] index-pack: check (and optionally set) hash algo based on input file

[Nguyễn Thái Ngọc Duy]

After 454253f059 (builtin/index-pack: improve hash function abstraction
- 2018-02-01), index-pack uses the_hash_algo for hashing. If "git
index-pack" is executed without a repository, we do not know what hash
algorithm to be used and the_hash_algo in theory could be undefined.

Since there should be some information about the hash algorithm in the
input pack file, we can initialize the correct hash algorithm with that
if the_hash_algo is not yet initialized. This assumes that pack files
with new hash algorithm MUST step up pack version.

While at there, make sure the hash algorithm requested by the pack file
and configured by the repository (if we're running with a repo) are
consistent.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/index-pack.c | 29 ++++++++++++++++++++++++++++-
 repository.h         |  5 +++++
 2 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 7e3e1a461c..2c75f28b41 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -326,10 +326,34 @@ static const char *open_pack_file(const char *pack_name)
 		output_fd = -1;
 		nothread_data.pack_fd = input_fd;
 	}
-	the_hash_algo->init_fn(&input_ctx);
 	return pack_name;
 }
 
+static void prepare_hash_algo(uint32_t pack_version)
+{
+	int pack_algo_id;
+	const struct git_hash_algo *pack_algo;
+
+	switch (pack_version) {
+	case 2:
+	case 3:
+		pack_algo_id = GIT_HASH_SHA1;
+		break;
+	default:
+		die("BUG: how to determine hash algo for version %d?",
+		    pack_version);
+	}
+
+	if (!repo_has_valid_hash_algo(the_repository)) /* running without repo */
+		repo_set_hash_algo(the_repository, pack_algo_id);
+
+	pack_algo = &hash_algos[pack_algo_id];
+	if (the_hash_algo->format_id != pack_algo->format_id)
+		die(_("incompatible hash algorithm, "
+		      "configured for %s but the pack file needs %s"),
+		    the_hash_algo->name, pack_algo->name);
+}
+
 static void parse_pack_header(void)
 {
 	struct pack_header *hdr = fill(sizeof(struct pack_header));
@@ -341,6 +365,9 @@ static void parse_pack_header(void)
 		die(_("pack version %"PRIu32" unsupported"),
 			ntohl(hdr->hdr_version));
 
+	prepare_hash_algo(ntohl(hdr->hdr_version));
+	the_hash_algo->init_fn(&input_ctx);
+
 	nr_objects = ntohl(hdr->hdr_entries);
 	use(sizeof(struct pack_header));
 }
diff --git a/repository.h b/repository.h
index 5092df3700..02c695bc74 100644
--- a/repository.h
+++ b/repository.h
@@ -113,4 +113,9 @@ static inline const struct git_hash_algo *repo_get_hash_algo(
 	return repo->hash_algo;
 }
 
+static inline int repo_has_valid_hash_algo(const struct repository *repo)
+{
+	return repo->hash_algo != NULL;
+}
+
 #endif /* REPOSITORY_H */
-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 5/6] diff.c: initialize hash algo when running in --no-index mode

[Nguyễn Thái Ngọc Duy]

Our "git diff" command supports running as a standalone tool. In this
code path, we try to hash the file content but after
18e2588e11 (sha1_file: switch uses of SHA-1 to the_hash_algo -
2018-02-01), there is a chance that the_hash_algo (required by
index_path) may still be uninitialized if no repository is found.

Executing index_path() when the_hash_algo is NULL (or points to unknown
algo) either crashes or dies. Let's make it a bit safer by explicitly
falling back to SHA-1 (so that the diff output remains the same as
before, compared to the alternative that we simply do not hash).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 diff.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/diff.c b/diff.c
index 21c3838b25..5f28d84b87 100644
--- a/diff.c
+++ b/diff.c
@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)
 		return;
 	}
 
+	/*
+	 * NEEDSWORK: When running in no-index mode (and no repo is
+	 * found, thus no hash algo configured), fall back to SHA-1
+	 * hashing (which is used by diff_fill_oid_info below) to
+	 * avoid regression in diff output.
+	 *
+	 * In future, perhaps we can allow the user to specify their
+	 * hash algorithm from command line in this mode.
+	 */
+	if (o->flags.no_index && !repo_has_valid_hash_algo(the_repository))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+
 	diff_fill_oid_info(one);
 	diff_fill_oid_info(two);
 
-- 
2.16.1.435.g8f24da2e1a

[PATCH v3 6/6] Revert "repository: pre-initialize hash algo pointer"

[Nguyễn Thái Ngọc Duy]

This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
problem, git clone not setting up the_hash_algo, has been fixed in the
previous patch.

Since this is a dangerous move and could potentially break stuff after
release (and leads to workaround like the reverted commit), the
workaround technically remains, but is hidden behind a new environment
variable GIT_HASH_FIXUP. This should let the users continue to use git
while we fix the problem. This variable can be deleted after one or two
releases.

test-dump-split-index.c needs to call setup_git_directory() after this
to configure hash algorithm before parsing the index file. It's not the
best way (i.e. check index file version) but it's a test tool, this is
good enough.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 common-main.c                    | 10 ++++++++++
 repository.c                     |  2 +-
 repository.h                     |  5 +++++
 t/helper/test-dump-split-index.c |  2 ++
 4 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/common-main.c b/common-main.c
index 6a689007e7..fbfa98c3b8 100644
--- a/common-main.c
+++ b/common-main.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "exec_cmd.h"
 #include "attr.h"
+#include "repository.h"
 
 /*
  * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,14 @@ int main(int argc, const char **argv)
 
 	restore_sigpipe_to_default();
 
+	/*
+	 * Temporary recovery measure while hashing code is being
+	 * refactored. This variable should be gone after everybody
+	 * has used the_hash_algo in one or two releases and nobody
+	 * complains anything.
+	 */
+	if (getenv("GIT_HASH_FIXUP"))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+
 	return cmd_main(argc, argv);
 }
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..0d715f4fdb 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
 };
 struct repository *the_repository = &the_repo;
 
diff --git a/repository.h b/repository.h
index 02c695bc74..40bd49611f 100644
--- a/repository.h
+++ b/repository.h
@@ -110,6 +110,11 @@ extern int repo_read_index(struct repository *repo);
 static inline const struct git_hash_algo *repo_get_hash_algo(
 	const struct repository *repo)
 {
+	if (!repo->hash_algo)
+		die("BUG: hash_algo is not initialized!\n%s",
+		    _("You can work around this by setting environment"
+		      " variable GIT_HASH_FIXUP=1.\n"
+		      "Please report this to git@vger.kernel.org"));
 	return repo->hash_algo;
 }
 
diff --git a/t/helper/test-dump-split-index.c b/t/helper/test-dump-split-index.c
index e44430b699..b759fe3fc1 100644
--- a/t/helper/test-dump-split-index.c
+++ b/t/helper/test-dump-split-index.c
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)
 	struct split_index *si;
 	int i;
 
+	setup_git_directory();
+
 	do_read_index(&the_index, av[1], 1);
 	printf("own %s\n", sha1_to_hex(the_index.sha1));
 	si = the_index.split_index;
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v3 0/6] Fix initializing the_hash_algo

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

On Sun, Feb 25, 2018 at 06:18:34PM +0700, Nguyễn Thái Ngọc Duy wrote:
> v3 refines v2 a bit more:
> 
> - fix configure typo (and stray words in commit message)
> - use repo_set_hash_algo() instead of reassigning the_hash_algo
> - compare hash algos by format_id
> - catch NULL hash algo, report nicely and suggest GIT_HASH_FIXUP
> 
> The last point makes me much happier about keeping this workaround
> around until we are confident we can live without it. Interdiff

This looks sane to me.

Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net>
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

Re: [PATCH v3 0/6] Fix initializing the_hash_algo

[Stefan Beller]

On Sun, Feb 25, 2018 at 12:34 PM, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> On Sun, Feb 25, 2018 at 06:18:34PM +0700, Nguyễn Thái Ngọc Duy wrote:
>> v3 refines v2 a bit more:
>>
>> - fix configure typo (and stray words in commit message)
>> - use repo_set_hash_algo() instead of reassigning the_hash_algo
>> - compare hash algos by format_id
>> - catch NULL hash algo, report nicely and suggest GIT_HASH_FIXUP
>>
>> The last point makes me much happier about keeping this workaround
>> around until we are confident we can live without it. Interdiff
>
> This looks sane to me.
>
> Reviewed-by: brian m. carlson <sandals@crustytoothpaste.net>

I agree with this version as well.

Thanks,
Stefan

[PATCH 0/4] Delete ignore_env member in struct repository

[Nguyễn Thái Ngọc Duy]

It turns out I don't need my other series [1] in order to delete this
field. This series moves getenv() calls from
repo_set_gitdir()/repo_setup_env() and prepare_alt_odb() back in
environment.c where they belong in my opinion.

The repo_set_gitdir() now takes $GIT_DIR and optionally all other
configurable paths. If those paths are NULL, default repo layout will
be used. With getenv() no longer called inside repo_set_gitdir(),
ignore_env has no reason to stay. This is in 1/4.

The getenv() in prepare_alt_odb() is also moved back to
setup_git_env() in 3/4. It demonstrates how we could move other
getenv() back to if we want.

This series is built on top of Stefan's object-store-part1, v4. I
could rebase it on 'master' too, but then Junio may need to resolve
some conflicts.

[1] https://public-inbox.org/git/20180225111840.16421-1-pclouds@gmail.com/

Nguyễn Thái Ngọc Duy (4):
  repository.c: move env-related setup code back to environment.c
  repository.c: delete dead functions
  sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
  repository: delete ignore_env member

 cache.h        |  2 +-
 environment.c  | 13 +++++++--
 object-store.h |  5 +++-
 object.c       |  1 +
 repository.c   | 79 ++++++++++++++++++++++----------------------------
 repository.h   | 21 +++++++-------
 setup.c        |  3 +-
 sha1_file.c    |  6 +---
 8 files changed, 64 insertions(+), 66 deletions(-)

-- 
2.16.1.435.g8f24da2e1a

[PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Nguyễn Thái Ngọc Duy]

It does not make sense that generic repository code contains handling
of environment variables, which are specific for the main repository
only. Refactor repo_set_gitdir() function to take $GIT_DIR and
optionally _all_ other customizable paths. These optional paths can be
NULL and will be calculated according to the default directory layout.

Note that some dead functions are left behind to reduce diff
noise. They will be deleted in the next patch.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h       |  2 +-
 environment.c | 12 +++++++++---
 repository.c  | 48 ++++++++++++++++++++++++++++++++++++++++++------
 repository.h  | 11 ++++++++++-
 setup.c       |  3 +--
 5 files changed, 63 insertions(+), 13 deletions(-)

diff --git a/cache.h b/cache.h
index 5717399183..b164a407eb 100644
--- a/cache.h
+++ b/cache.h
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)
  */
 extern const char * const local_repo_env[];
 
-extern void setup_git_env(void);
+extern void setup_git_env(const char *git_dir);
 
 /*
  * Returns true iff we have a configured git repository (either via
diff --git a/environment.c b/environment.c
index ec10b062e6..2905042b16 100644
--- a/environment.c
+++ b/environment.c
@@ -148,10 +148,17 @@ static char *expand_namespace(const char *raw_namespace)
 	return strbuf_detach(&buf, NULL);
 }
 
-void setup_git_env(void)
+void setup_git_env(const char *git_dir)
 {
 	const char *shallow_file;
 	const char *replace_ref_base;
+	struct set_gitdir_args args = { NULL };
+
+	args.shared_root = getenv(GIT_COMMON_DIR_ENVIRONMENT);
+	args.object_dir = getenv(DB_ENVIRONMENT);
+	args.graft_file = getenv(GRAFT_ENVIRONMENT);
+	args.index_file = getenv(INDEX_ENVIRONMENT);
+	repo_set_gitdir(the_repository, git_dir, &args);
 
 	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
 		check_replace_refs = 0;
@@ -301,8 +308,7 @@ int set_git_dir(const char *path)
 {
 	if (setenv(GIT_DIR_ENVIRONMENT, path, 1))
 		return error("Could not set GIT_DIR to '%s'", path);
-	repo_set_gitdir(the_repository, path);
-	setup_git_env();
+	setup_git_env(path);
 	return 0;
 }
 
diff --git a/repository.c b/repository.c
index a069b1b640..70dc8dc661 100644
--- a/repository.c
+++ b/repository.c
@@ -61,15 +61,50 @@ static void repo_setup_env(struct repository *repo)
 					     "index", !repo->ignore_env);
 }
 
-void repo_set_gitdir(struct repository *repo, const char *path)
+static void expand_base_dir(char **out, const char *in,
+			    const char *base_dir, const char *def_in)
 {
-	const char *gitfile = read_gitfile(path);
-	char *old_gitdir = repo->gitdir;
+	free(*out);
+	if (in)
+		*out = xstrdup(in);
+	else
+		*out = xstrfmt("%s/%s", base_dir, def_in);
+}
+
+static void repo_set_commondir(struct repository *repo,
+			       const char *shared_root)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	free(repo->commondir);
 
-	repo->gitdir = xstrdup(gitfile ? gitfile : path);
-	repo_setup_env(repo);
+	if (shared_root) {
+		repo->different_commondir = 1;
+		repo->commondir = xstrdup(shared_root);
+		return;
+	}
 
+	repo->different_commondir = get_common_dir_noenv(&sb, repo->gitdir);
+	repo->commondir = strbuf_detach(&sb, NULL);
+}
+
+void repo_set_gitdir(struct repository *repo,
+		     const char *root,
+		     const struct set_gitdir_args *o)
+{
+	const char *gitfile = read_gitfile(root);
+	char *old_gitdir = repo->gitdir;
+
+	repo->gitdir = xstrdup(gitfile ? gitfile : root);
 	free(old_gitdir);
+
+	repo_set_commondir(repo, o->shared_root);
+	expand_base_dir(&repo->objects.objectdir, o->object_dir,
+			repo->commondir, "objects");
+	expand_base_dir(&repo->graft_file, o->graft_file,
+			repo->commondir, "info/grafts");
+	expand_base_dir(&repo->index_file, o->index_file,
+			repo->gitdir, "index");
 }
 
 void repo_set_hash_algo(struct repository *repo, int hash_algo)
@@ -87,6 +122,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 	int error = 0;
 	char *abspath = NULL;
 	const char *resolved_gitdir;
+	struct set_gitdir_args args = { NULL };
 
 	abspath = real_pathdup(gitdir, 0);
 	if (!abspath) {
@@ -101,7 +137,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 		goto out;
 	}
 
-	repo_set_gitdir(repo, resolved_gitdir);
+	repo_set_gitdir(repo, resolved_gitdir, &args);
 
 out:
 	free(abspath);
diff --git a/repository.h b/repository.h
index fa73ab8e93..b5b5d138aa 100644
--- a/repository.h
+++ b/repository.h
@@ -89,7 +89,16 @@ struct repository {
 
 extern struct repository *the_repository;
 
-extern void repo_set_gitdir(struct repository *repo, const char *path);
+struct set_gitdir_args {
+	const char *shared_root;
+	const char *object_dir;
+	const char *graft_file;
+	const char *index_file;
+};
+
+extern void repo_set_gitdir(struct repository *repo,
+			    const char *root,
+			    const struct set_gitdir_args *optional);
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
 extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index c5d55dcee4..6fac1bb58a 100644
--- a/setup.c
+++ b/setup.c
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
 			if (!gitdir)
 				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
-			repo_set_gitdir(the_repository, gitdir);
-			setup_git_env();
+			setup_git_env(gitdir);
 		}
 		if (startup_info->have_repository)
 			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Eric Sunshine]

On Mon, Feb 26, 2018 at 5:30 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> It does not make sense that generic repository code contains handling
> of environment variables, which are specific for the main repository
> only. Refactor repo_set_gitdir() function to take $GIT_DIR and
> optionally _all_ other customizable paths. These optional paths can be
> NULL and will be calculated according to the default directory layout.
>
> Note that some dead functions are left behind to reduce diff
> noise. They will be deleted in the next patch.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> diff --git a/environment.c b/environment.c
> @@ -148,10 +148,17 @@ static char *expand_namespace(const char *raw_namespace)
> -void setup_git_env(void)
> +void setup_git_env(const char *git_dir)
>  {
>         const char *shallow_file;
>         const char *replace_ref_base;
> +       struct set_gitdir_args args = { NULL };
> +
> +       args.shared_root = getenv(GIT_COMMON_DIR_ENVIRONMENT);
> +       args.object_dir = getenv(DB_ENVIRONMENT);
> +       args.graft_file = getenv(GRAFT_ENVIRONMENT);
> +       args.index_file = getenv(INDEX_ENVIRONMENT);

According to POSIX[1], the result of getenv() may be invalidated by
another call to getenv() (or setenv() or unsetenv() or putenv()).

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html

> +       repo_set_gitdir(the_repository, git_dir, &args);
> diff --git a/repository.c b/repository.c
> @@ -61,15 +61,50 @@ static void repo_setup_env(struct repository *repo)
> +void repo_set_gitdir(struct repository *repo,
> +                    const char *root,
> +                    const struct set_gitdir_args *o)
> +{
> +       const char *gitfile = read_gitfile(root);
> +       char *old_gitdir = repo->gitdir;
> +
> +       repo->gitdir = xstrdup(gitfile ? gitfile : root);
>         free(old_gitdir);

Can:

    char *old_gitdir = repo->gitdir;
    repo->gitdir = xstrdup(...);
    free(old_gitdir);

be simplified to:

    free(repo->gitdir);
    repo->gitdir = xstrdup(...);

?

> +       repo_set_commondir(repo, o->shared_root);

The repo_set_gitdir() prototype (below) makes it seem as if the last
argument ('o', in this case) is optional, presumably by passing in
NULL, however, this code does not appear to be prepared to deal with
NULL.

> +       expand_base_dir(&repo->objects.objectdir, o->object_dir,
> +                       repo->commondir, "objects");
> +       expand_base_dir(&repo->graft_file, o->graft_file,
> +                       repo->commondir, "info/grafts");
> +       expand_base_dir(&repo->index_file, o->index_file,
> +                       repo->gitdir, "index");
>  }
> diff --git a/repository.h b/repository.h
> @@ -89,7 +89,16 @@ struct repository {
> +struct set_gitdir_args {
> +       [...]
> +};
> +
> +extern void repo_set_gitdir(struct repository *repo,
> +                           const char *root,
> +                           const struct set_gitdir_args *optional);

Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Duy Nguyen]

On Tue, Feb 27, 2018 at 1:57 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:

>> +       repo_set_commondir(repo, o->shared_root);
>
> The repo_set_gitdir() prototype (below) makes it seem as if the last
> argument ('o', in this case) is optional, presumably by passing in
> NULL, however, this code does not appear to be prepared to deal with
> NULL.

I know. I went this this struct because I really hate to write lots of
NULL without names

    repo_set_gitdir(gitdir, NULL, NULL, NULL, ...);

but I could not find a nicer way to make the whole struct optional,
not without repeating more code or using preprocessor. Given that this
function has 3-4 call sites max, I think it's still ok.

>
>> +       expand_base_dir(&repo->objects.objectdir, o->object_dir,
>> +                       repo->commondir, "objects");
>> +       expand_base_dir(&repo->graft_file, o->graft_file,
>> +                       repo->commondir, "info/grafts");
>> +       expand_base_dir(&repo->index_file, o->index_file,
>> +                       repo->gitdir, "index");
>>  }
-- 
Duy

Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Duy Nguyen]

On Tue, Feb 27, 2018 at 1:57 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> Can:
>
>     char *old_gitdir = repo->gitdir;
>     repo->gitdir = xstrdup(...);
>     free(old_gitdir);
>
> be simplified to:
>
>     free(repo->gitdir);
>     repo->gitdir = xstrdup(...);
>
> ?

No because I think "root" (not quoted here) could point to the same
value as repo->gitdir, if you free that first, the second xstrdup()
will duplicate a freed memory. See 1fb2b636c6 (set_git_dir: handle
feeding gitdir to itself - 2017-09-05)

I will add a comment about this since it's not very clear from this code.
-- 
Duy

Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Stefan Beller]

> diff --git a/setup.c b/setup.c
> index c5d55dcee4..6fac1bb58a 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
>                         const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
>                         if (!gitdir)
>                                 gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
> -                       repo_set_gitdir(the_repository, gitdir);
> -                       setup_git_env();
> +                       setup_git_env(gitdir);
>                 }

What makes git_dir special, such that we have to pass it in here?
Could the check for getenv(GIT_DIR_ENVIRONMENT) and fallback to
DEFAULT_... be part of setup_git_env() ?
Oh I guess that cannot be done easily as the submodule code
spcifically doesn't want to discover the git dir itself.


>                 if (startup_info->have_repository)
>                         repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
> --
> 2.16.1.435.g8f24da2e1a
>

Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c

[Duy Nguyen]

On Tue, Feb 27, 2018 at 3:30 AM, Stefan Beller <sbeller@google.com> wrote:
>> diff --git a/setup.c b/setup.c
>> index c5d55dcee4..6fac1bb58a 100644
>> --- a/setup.c
>> +++ b/setup.c
>> @@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
>>                         const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
>>                         if (!gitdir)
>>                                 gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
>> -                       repo_set_gitdir(the_repository, gitdir);
>> -                       setup_git_env();
>> +                       setup_git_env(gitdir);
>>                 }
>
> What makes git_dir special, such that we have to pass it in here?
> Could the check for getenv(GIT_DIR_ENVIRONMENT) and fallback to
> DEFAULT_... be part of setup_git_env() ?
> Oh I guess that cannot be done easily as the submodule code
> spcifically doesn't want to discover the git dir itself.

No, submodule code already does not care about setup_git_env().

Back to your first question, this is related to the "NEEDSWORK"
comment in this code. We _should_ always set_git_dir() before leaving
setup_git_directory_gently() then everybody behaves the same way and
we don't need this special setup_git_env() here at all. BUT we need to
be careful about setting environment variable $GIT_DIR, done inside
set_git_dir(), because sub-processes will inherit it and if we're not
careful, we'll break .git discovery in those. There's another thing I
don't like about this is setup_work_tree() using set_git_dir() the
second time to adjust relative $GIT_DIR and friends when it moves
$CWD. I'll need to fix this, soon.

So in short, it's special because the current situation is messy and
(hopefully) temporary. But it should eventually be gone.
-- 
Duy

[PATCH 2/4] repository.c: delete dead functions

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 45 ---------------------------------------------
 1 file changed, 45 deletions(-)

diff --git a/repository.c b/repository.c
index 70dc8dc661..be86da82cc 100644
--- a/repository.c
+++ b/repository.c
@@ -16,51 +16,6 @@ static struct repository the_repo = {
 };
 struct repository *the_repository = &the_repo;
 
-static char *git_path_from_env(const char *envvar, const char *git_dir,
-			       const char *path, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(envvar);
-		if (value)
-			return xstrdup(value);
-	}
-
-	return xstrfmt("%s/%s", git_dir, path);
-}
-
-static int find_common_dir(struct strbuf *sb, const char *gitdir, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(GIT_COMMON_DIR_ENVIRONMENT);
-		if (value) {
-			strbuf_addstr(sb, value);
-			return 1;
-		}
-	}
-
-	return get_common_dir_noenv(sb, gitdir);
-}
-
-static void repo_setup_env(struct repository *repo)
-{
-	struct strbuf sb = STRBUF_INIT;
-
-	repo->different_commondir = find_common_dir(&sb, repo->gitdir,
-						    !repo->ignore_env);
-	free(repo->commondir);
-	repo->commondir = strbuf_detach(&sb, NULL);
-	raw_object_store_clear(&repo->objects);
-	repo->objects.objectdir =
-		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-				  "objects", !repo->ignore_env);
-	free(repo->graft_file);
-	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
-					     "info/grafts", !repo->ignore_env);
-	free(repo->index_file);
-	repo->index_file = git_path_from_env(INDEX_ENVIRONMENT, repo->gitdir,
-					     "index", !repo->ignore_env);
-}
-
 static void expand_base_dir(char **out, const char *in,
 			    const char *base_dir, const char *def_in)
 {
-- 
2.16.1.435.g8f24da2e1a

[PATCH 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

[Nguyễn Thái Ngọc Duy]

getenv() is supposed to work on the main repository only. This delayed
getenv() code in sha1_file.c makes it more difficult to convert
sha1_file.c to a generic object store that could be used by both
submodule and main repositories.

Move the getenv() back in setup_git_env() where other env vars are
also fetched.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 environment.c  | 1 +
 object-store.h | 5 ++++-
 object.c       | 1 +
 repository.c   | 2 ++
 repository.h   | 1 +
 sha1_file.c    | 6 +-----
 6 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/environment.c b/environment.c
index 2905042b16..95de419de8 100644
--- a/environment.c
+++ b/environment.c
@@ -158,6 +158,7 @@ void setup_git_env(const char *git_dir)
 	args.object_dir = getenv(DB_ENVIRONMENT);
 	args.graft_file = getenv(GRAFT_ENVIRONMENT);
 	args.index_file = getenv(INDEX_ENVIRONMENT);
+	args.alternate_db = getenv(ALTERNATE_DB_ENVIRONMENT);
 	repo_set_gitdir(the_repository, git_dir, &args);
 
 	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
diff --git a/object-store.h b/object-store.h
index afe2f93459..9b1303549b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -87,6 +87,9 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	/* Path to extra alternate object database if not NULL */
+	char *alternate_db;
+
 	struct packed_git *packed_git;
 	/* A most-recently-used ordered version of the packed_git list. */
 	struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index a7c238339b..5317cfc390 100644
--- a/object.c
+++ b/object.c
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	FREE_AND_NULL(o->objectdir);
+	FREE_AND_NULL(o->alternate_db);
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
diff --git a/repository.c b/repository.c
index be86da82cc..19532ec5ff 100644
--- a/repository.c
+++ b/repository.c
@@ -56,6 +56,8 @@ void repo_set_gitdir(struct repository *repo,
 	repo_set_commondir(repo, o->shared_root);
 	expand_base_dir(&repo->objects.objectdir, o->object_dir,
 			repo->commondir, "objects");
+	free(repo->objects.alternate_db);
+	repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);
 	expand_base_dir(&repo->graft_file, o->graft_file,
 			repo->commondir, "info/grafts");
 	expand_base_dir(&repo->index_file, o->index_file,
diff --git a/repository.h b/repository.h
index b5b5d138aa..b1da2a6384 100644
--- a/repository.h
+++ b/repository.h
@@ -94,6 +94,7 @@ struct set_gitdir_args {
 	const char *object_dir;
 	const char *graft_file;
 	const char *index_file;
+	const char *alternate_db;
 };
 
 extern void repo_set_gitdir(struct repository *repo,
diff --git a/sha1_file.c b/sha1_file.c
index dfc8deec38..ad1cd441e6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 
 void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
 	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
-
 	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
-	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(r, r->objects.alternate_db, PATH_SEP, NULL, 0);
 
 	read_info_alternates(r, r->objects.objectdir, 0);
 }
-- 
2.16.1.435.g8f24da2e1a

[PATCH 4/4] repository: delete ignore_env member

[Nguyễn Thái Ngọc Duy]

This variable was added because the repo_set_gitdir() was created to
cover both submodule and main repos, but these two are initialized a
bit differently so ignore_env == 0 means main repo, while ignore_env
!= 0 is submodules.

Since the difference part (env variables) has been moved out of
repo_set_gitdir(), this function works the same way for both repo
types and ignore_env is not needed anymore.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 4 +---
 repository.h | 9 ---------
 2 files changed, 1 insertion(+), 12 deletions(-)

diff --git a/repository.c b/repository.c
index 19532ec5ff..8f6386022f 100644
--- a/repository.c
+++ b/repository.c
@@ -12,7 +12,7 @@ static struct repository the_repo = {
 	NULL, NULL, NULL,
 	&the_index,
 	&hash_algos[GIT_HASH_SHA1],
-	0, 0
+	0
 };
 struct repository *the_repository = &the_repo;
 
@@ -134,8 +134,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
 
-	repo->ignore_env = 1;
-
 	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
 
 	if (repo_init_gitdir(repo, gitdir))
diff --git a/repository.h b/repository.h
index b1da2a6384..07e8971428 100644
--- a/repository.h
+++ b/repository.h
@@ -73,15 +73,6 @@ struct repository {
 	const struct git_hash_algo *hash_algo;
 
 	/* Configurations */
-	/*
-	 * Bit used during initialization to indicate if repository state (like
-	 * the location of the 'objectdir') should be read from the
-	 * environment.  By default this bit will be set at the begining of
-	 * 'repo_init()' so that all repositories will ignore the environment.
-	 * The exception to this is 'the_repository', which doesn't go through
-	 * the normal 'repo_init()' process.
-	 */
-	unsigned ignore_env:1;
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
 	unsigned different_commondir:1;
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH 0/4] Delete ignore_env member in struct repository

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

> This series is built on top of Stefan's object-store-part1, v4. I
> could rebase it on 'master' too, but then Junio may need to resolve
> some conflicts.

As a follow-up fix for combined "move things to the_repo as much as
possible" efforts I think what you did makes most sense.  I would
say that it makes even more sense to make these part of the
object-store series if/when the series needs rerolling--after all
the topic is already a multi-author effort.

I'll have to read them carefully before commenting on the actual
patches ;-)  Thanks.

>
> [1] https://public-inbox.org/git/20180225111840.16421-1-pclouds@gmail.com/
>
> Nguyễn Thái Ngọc Duy (4):
>   repository.c: move env-related setup code back to environment.c
>   repository.c: delete dead functions
>   sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
>   repository: delete ignore_env member
>
>  cache.h        |  2 +-
>  environment.c  | 13 +++++++--
>  object-store.h |  5 +++-
>  object.c       |  1 +
>  repository.c   | 79 ++++++++++++++++++++++----------------------------
>  repository.h   | 21 +++++++-------
>  setup.c        |  3 +-
>  sha1_file.c    |  6 +---
>  8 files changed, 64 insertions(+), 66 deletions(-)

Re: [PATCH 0/4] Delete ignore_env member in struct repository

[Stefan Beller]

On Mon, Feb 26, 2018 at 2:30 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> It turns out I don't need my other series [1] in order to delete this
> field. This series moves getenv() calls from
> repo_set_gitdir()/repo_setup_env() and prepare_alt_odb() back in
> environment.c where they belong in my opinion.
>
> The repo_set_gitdir() now takes $GIT_DIR and optionally all other
> configurable paths. If those paths are NULL, default repo layout will
> be used. With getenv() no longer called inside repo_set_gitdir(),
> ignore_env has no reason to stay. This is in 1/4.
>
> The getenv() in prepare_alt_odb() is also moved back to
> setup_git_env() in 3/4. It demonstrates how we could move other
> getenv() back to if we want.
>
> This series is built on top of Stefan's object-store-part1, v4. I
> could rebase it on 'master' too, but then Junio may need to resolve
> some conflicts.
>

Thanks for working on this,
I found this series a pleasant read, the only issue I saw was Erics upbringing
of multiple getenv calls without strdup()ing the content.

What is the plan from here on?
Should I build further series on top of yours? The next series will
focus on the pack side of things (pack.h, packfile.{c,h})

So maybe we'll have Junio merge down my series (and yours as it
needs one reroll?) and then build on top of that?

Thanks,
Stefan

Re: [PATCH 0/4] Delete ignore_env member in struct repository

[Duy Nguyen]

On Tue, Feb 27, 2018 at 3:46 AM, Stefan Beller <sbeller@google.com> wrote:
> What is the plan from here on?
> Should I build further series on top of yours? The next series will
> focus on the pack side of things (pack.h, packfile.{c,h})
>
> So maybe we'll have Junio merge down my series (and yours as it
> needs one reroll?) and then build on top of that?

I think that's the less painful way for everybody.
-- 
Duy

[PATCH v2 0/4] Delete ignore_env member in struct repository

[Nguyễn Thái Ngọc Duy]

v2 fixes the incorrect use of consecutive getenv() and adds a comment
to clarify the role of old_gitdir

Interdiff:

diff --git a/environment.c b/environment.c
index 95de419de8..47c6e31559 100644
--- a/environment.c
+++ b/environment.c
@@ -14,6 +14,7 @@
 #include "fmt-merge-msg.h"
 #include "commit.h"
 #include "object-store.h"
+#include "argv-array.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace)
 	return strbuf_detach(&buf, NULL);
 }
 
+/* Wrapper of getenv() that returns a strdup value. This value is kept
+ * in argv to be freed later.
+ */
+static const char *getenv_safe(struct argv_array *argv, const char *name)
+{
+	const char *value = getenv(name);
+
+	if (!value)
+		return NULL;
+
+	argv_array_push(argv, value);
+	return argv->argv[argv->argc - 1];
+}
+
 void setup_git_env(const char *git_dir)
 {
 	const char *shallow_file;
 	const char *replace_ref_base;
 	struct set_gitdir_args args = { NULL };
+	struct argv_array to_free = ARGV_ARRAY_INIT;
 
-	args.shared_root = getenv(GIT_COMMON_DIR_ENVIRONMENT);
-	args.object_dir = getenv(DB_ENVIRONMENT);
-	args.graft_file = getenv(GRAFT_ENVIRONMENT);
-	args.index_file = getenv(INDEX_ENVIRONMENT);
-	args.alternate_db = getenv(ALTERNATE_DB_ENVIRONMENT);
+	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
+	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
+	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
+	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
+	args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
 	repo_set_gitdir(the_repository, git_dir, &args);
+	argv_array_clear(&to_free);
 
 	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
 		check_replace_refs = 0;
diff --git a/repository.c b/repository.c
index 8f6386022f..c555dacad2 100644
--- a/repository.c
+++ b/repository.c
@@ -48,6 +48,11 @@ void repo_set_gitdir(struct repository *repo,
 		     const struct set_gitdir_args *o)
 {
 	const char *gitfile = read_gitfile(root);
+	/*
+	 * repo->gitdir is saved because the caller could pass "root"
+	 * that also points to repo->gitdir. We want to keep it alive
+	 * until after xstrdup(root). Then we can free it.
+	 */
 	char *old_gitdir = repo->gitdir;
 
 	repo->gitdir = xstrdup(gitfile ? gitfile : root);

Nguyễn Thái Ngọc Duy (4):
  repository.c: move env-related setup code back to environment.c
  repository.c: delete dead functions
  sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
  repository: delete ignore_env member

 cache.h        |  2 +-
 environment.c  | 30 ++++++++++++++++--
 object-store.h |  5 ++-
 object.c       |  1 +
 repository.c   | 84 ++++++++++++++++++++++++--------------------------
 repository.h   | 21 +++++++------
 setup.c        |  3 +-
 sha1_file.c    |  6 +---
 8 files changed, 86 insertions(+), 66 deletions(-)

-- 
2.16.1.435.g8f24da2e1a

[PATCH v2 1/4] repository.c: move env-related setup code back to environment.c

[Nguyễn Thái Ngọc Duy]

It does not make sense that generic repository code contains handling
of environment variables, which are specific for the main repository
only. Refactor repo_set_gitdir() function to take $GIT_DIR and
optionally _all_ other customizable paths. These optional paths can be
NULL and will be calculated according to the default directory layout.

Note that some dead functions are left behind to reduce diff
noise. They will be deleted in the next patch.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h       |  2 +-
 environment.c | 29 +++++++++++++++++++++++++---
 repository.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++------
 repository.h  | 11 ++++++++++-
 setup.c       |  3 +--
 5 files changed, 85 insertions(+), 13 deletions(-)

diff --git a/cache.h b/cache.h
index 5717399183..b164a407eb 100644
--- a/cache.h
+++ b/cache.h
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)
  */
 extern const char * const local_repo_env[];
 
-extern void setup_git_env(void);
+extern void setup_git_env(const char *git_dir);
 
 /*
  * Returns true iff we have a configured git repository (either via
diff --git a/environment.c b/environment.c
index ec10b062e6..74a2900ddf 100644
--- a/environment.c
+++ b/environment.c
@@ -14,6 +14,7 @@
 #include "fmt-merge-msg.h"
 #include "commit.h"
 #include "object-store.h"
+#include "argv-array.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -148,10 +149,33 @@ static char *expand_namespace(const char *raw_namespace)
 	return strbuf_detach(&buf, NULL);
 }
 
-void setup_git_env(void)
+/* Wrapper of getenv() that returns a strdup value. This value is kept
+ * in argv to be freed later.
+ */
+static const char *getenv_safe(struct argv_array *argv, const char *name)
+{
+	const char *value = getenv(name);
+
+	if (!value)
+		return NULL;
+
+	argv_array_push(argv, value);
+	return argv->argv[argv->argc - 1];
+}
+
+void setup_git_env(const char *git_dir)
 {
 	const char *shallow_file;
 	const char *replace_ref_base;
+	struct set_gitdir_args args = { NULL };
+	struct argv_array to_free = ARGV_ARRAY_INIT;
+
+	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
+	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
+	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
+	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
+	repo_set_gitdir(the_repository, git_dir, &args);
+	argv_array_clear(&to_free);
 
 	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
 		check_replace_refs = 0;
@@ -301,8 +325,7 @@ int set_git_dir(const char *path)
 {
 	if (setenv(GIT_DIR_ENVIRONMENT, path, 1))
 		return error("Could not set GIT_DIR to '%s'", path);
-	repo_set_gitdir(the_repository, path);
-	setup_git_env();
+	setup_git_env(path);
 	return 0;
 }
 
diff --git a/repository.c b/repository.c
index a069b1b640..343efe7282 100644
--- a/repository.c
+++ b/repository.c
@@ -61,15 +61,55 @@ static void repo_setup_env(struct repository *repo)
 					     "index", !repo->ignore_env);
 }
 
-void repo_set_gitdir(struct repository *repo, const char *path)
+static void expand_base_dir(char **out, const char *in,
+			    const char *base_dir, const char *def_in)
 {
-	const char *gitfile = read_gitfile(path);
-	char *old_gitdir = repo->gitdir;
+	free(*out);
+	if (in)
+		*out = xstrdup(in);
+	else
+		*out = xstrfmt("%s/%s", base_dir, def_in);
+}
+
+static void repo_set_commondir(struct repository *repo,
+			       const char *shared_root)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	free(repo->commondir);
 
-	repo->gitdir = xstrdup(gitfile ? gitfile : path);
-	repo_setup_env(repo);
+	if (shared_root) {
+		repo->different_commondir = 1;
+		repo->commondir = xstrdup(shared_root);
+		return;
+	}
 
+	repo->different_commondir = get_common_dir_noenv(&sb, repo->gitdir);
+	repo->commondir = strbuf_detach(&sb, NULL);
+}
+
+void repo_set_gitdir(struct repository *repo,
+		     const char *root,
+		     const struct set_gitdir_args *o)
+{
+	const char *gitfile = read_gitfile(root);
+	/*
+	 * repo->gitdir is saved because the caller could pass "root"
+	 * that also points to repo->gitdir. We want to keep it alive
+	 * until after xstrdup(root). Then we can free it.
+	 */
+	char *old_gitdir = repo->gitdir;
+
+	repo->gitdir = xstrdup(gitfile ? gitfile : root);
 	free(old_gitdir);
+
+	repo_set_commondir(repo, o->shared_root);
+	expand_base_dir(&repo->objects.objectdir, o->object_dir,
+			repo->commondir, "objects");
+	expand_base_dir(&repo->graft_file, o->graft_file,
+			repo->commondir, "info/grafts");
+	expand_base_dir(&repo->index_file, o->index_file,
+			repo->gitdir, "index");
 }
 
 void repo_set_hash_algo(struct repository *repo, int hash_algo)
@@ -87,6 +127,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 	int error = 0;
 	char *abspath = NULL;
 	const char *resolved_gitdir;
+	struct set_gitdir_args args = { NULL };
 
 	abspath = real_pathdup(gitdir, 0);
 	if (!abspath) {
@@ -101,7 +142,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 		goto out;
 	}
 
-	repo_set_gitdir(repo, resolved_gitdir);
+	repo_set_gitdir(repo, resolved_gitdir, &args);
 
 out:
 	free(abspath);
diff --git a/repository.h b/repository.h
index fa73ab8e93..b5b5d138aa 100644
--- a/repository.h
+++ b/repository.h
@@ -89,7 +89,16 @@ struct repository {
 
 extern struct repository *the_repository;
 
-extern void repo_set_gitdir(struct repository *repo, const char *path);
+struct set_gitdir_args {
+	const char *shared_root;
+	const char *object_dir;
+	const char *graft_file;
+	const char *index_file;
+};
+
+extern void repo_set_gitdir(struct repository *repo,
+			    const char *root,
+			    const struct set_gitdir_args *optional);
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
 extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index c5d55dcee4..6fac1bb58a 100644
--- a/setup.c
+++ b/setup.c
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
 			if (!gitdir)
 				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
-			repo_set_gitdir(the_repository, gitdir);
-			setup_git_env();
+			setup_git_env(gitdir);
 		}
 		if (startup_info->have_repository)
 			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 1/4] repository.c: move env-related setup code back to environment.c

[Brandon Williams]

On 02/27, Nguyễn Thái Ngọc Duy wrote:
> It does not make sense that generic repository code contains handling
> of environment variables, which are specific for the main repository
> only. Refactor repo_set_gitdir() function to take $GIT_DIR and
> optionally _all_ other customizable paths. These optional paths can be
> NULL and will be calculated according to the default directory layout.
> 
> Note that some dead functions are left behind to reduce diff
> noise. They will be deleted in the next patch.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  cache.h       |  2 +-
>  environment.c | 29 +++++++++++++++++++++++++---
>  repository.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++------
>  repository.h  | 11 ++++++++++-
>  setup.c       |  3 +--
>  5 files changed, 85 insertions(+), 13 deletions(-)
> 
> diff --git a/cache.h b/cache.h
> index 5717399183..b164a407eb 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)
>   */
>  extern const char * const local_repo_env[];
>  
> -extern void setup_git_env(void);
> +extern void setup_git_env(const char *git_dir);
>  
>  /*
>   * Returns true iff we have a configured git repository (either via
> diff --git a/environment.c b/environment.c
> index ec10b062e6..74a2900ddf 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -14,6 +14,7 @@
>  #include "fmt-merge-msg.h"
>  #include "commit.h"
>  #include "object-store.h"
> +#include "argv-array.h"
>  
>  int trust_executable_bit = 1;
>  int trust_ctime = 1;
> @@ -148,10 +149,33 @@ static char *expand_namespace(const char *raw_namespace)
>  	return strbuf_detach(&buf, NULL);
>  }
>  
> -void setup_git_env(void)
> +/* Wrapper of getenv() that returns a strdup value. This value is kept
> + * in argv to be freed later.
> + */
> +static const char *getenv_safe(struct argv_array *argv, const char *name)
> +{
> +	const char *value = getenv(name);
> +
> +	if (!value)
> +		return NULL;
> +
> +	argv_array_push(argv, value);
> +	return argv->argv[argv->argc - 1];
> +}
> +
> +void setup_git_env(const char *git_dir)
>  {
>  	const char *shallow_file;
>  	const char *replace_ref_base;
> +	struct set_gitdir_args args = { NULL };
> +	struct argv_array to_free = ARGV_ARRAY_INIT;
> +
> +	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
> +	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
> +	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
> +	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
> +	repo_set_gitdir(the_repository, git_dir, &args);
> +	argv_array_clear(&to_free);
>  
>  	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
>  		check_replace_refs = 0;
> @@ -301,8 +325,7 @@ int set_git_dir(const char *path)
>  {
>  	if (setenv(GIT_DIR_ENVIRONMENT, path, 1))
>  		return error("Could not set GIT_DIR to '%s'", path);
> -	repo_set_gitdir(the_repository, path);
> -	setup_git_env();
> +	setup_git_env(path);
>  	return 0;
>  }
>  
> diff --git a/repository.c b/repository.c
> index a069b1b640..343efe7282 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -61,15 +61,55 @@ static void repo_setup_env(struct repository *repo)
>  					     "index", !repo->ignore_env);
>  }
>  
> -void repo_set_gitdir(struct repository *repo, const char *path)
> +static void expand_base_dir(char **out, const char *in,
> +			    const char *base_dir, const char *def_in)
>  {
> -	const char *gitfile = read_gitfile(path);
> -	char *old_gitdir = repo->gitdir;
> +	free(*out);
> +	if (in)
> +		*out = xstrdup(in);
> +	else
> +		*out = xstrfmt("%s/%s", base_dir, def_in);
> +}
> +
> +static void repo_set_commondir(struct repository *repo,
> +			       const char *shared_root)
> +{
> +	struct strbuf sb = STRBUF_INIT;
> +
> +	free(repo->commondir);
>  
> -	repo->gitdir = xstrdup(gitfile ? gitfile : path);
> -	repo_setup_env(repo);
> +	if (shared_root) {
> +		repo->different_commondir = 1;
> +		repo->commondir = xstrdup(shared_root);
> +		return;
> +	}
>  
> +	repo->different_commondir = get_common_dir_noenv(&sb, repo->gitdir);
> +	repo->commondir = strbuf_detach(&sb, NULL);
> +}
> +
> +void repo_set_gitdir(struct repository *repo,
> +		     const char *root,
> +		     const struct set_gitdir_args *o)
> +{
> +	const char *gitfile = read_gitfile(root);
> +	/*
> +	 * repo->gitdir is saved because the caller could pass "root"
> +	 * that also points to repo->gitdir. We want to keep it alive
> +	 * until after xstrdup(root). Then we can free it.
> +	 */
> +	char *old_gitdir = repo->gitdir;
> +
> +	repo->gitdir = xstrdup(gitfile ? gitfile : root);
>  	free(old_gitdir);
> +
> +	repo_set_commondir(repo, o->shared_root);
> +	expand_base_dir(&repo->objects.objectdir, o->object_dir,
> +			repo->commondir, "objects");
> +	expand_base_dir(&repo->graft_file, o->graft_file,
> +			repo->commondir, "info/grafts");
> +	expand_base_dir(&repo->index_file, o->index_file,
> +			repo->gitdir, "index");
>  }
>  
>  void repo_set_hash_algo(struct repository *repo, int hash_algo)
> @@ -87,6 +127,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
>  	int error = 0;
>  	char *abspath = NULL;
>  	const char *resolved_gitdir;
> +	struct set_gitdir_args args = { NULL };
>  
>  	abspath = real_pathdup(gitdir, 0);
>  	if (!abspath) {
> @@ -101,7 +142,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
>  		goto out;
>  	}
>  
> -	repo_set_gitdir(repo, resolved_gitdir);
> +	repo_set_gitdir(repo, resolved_gitdir, &args);
>  
>  out:
>  	free(abspath);
> diff --git a/repository.h b/repository.h
> index fa73ab8e93..b5b5d138aa 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -89,7 +89,16 @@ struct repository {
>  
>  extern struct repository *the_repository;
>  
> -extern void repo_set_gitdir(struct repository *repo, const char *path);
> +struct set_gitdir_args {
> +	const char *shared_root;

Can you add a comment explaining what shared_root is? From reading the
code it seems to be the common gitdir but its not clear from just
reading this.

> +	const char *object_dir;
> +	const char *graft_file;
> +	const char *index_file;
> +};
> +
> +extern void repo_set_gitdir(struct repository *repo,
> +			    const char *root,
> +			    const struct set_gitdir_args *optional);
>  extern void repo_set_worktree(struct repository *repo, const char *path);
>  extern void repo_set_hash_algo(struct repository *repo, int algo);
>  extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
> diff --git a/setup.c b/setup.c
> index c5d55dcee4..6fac1bb58a 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
>  			const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
>  			if (!gitdir)
>  				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
> -			repo_set_gitdir(the_repository, gitdir);
> -			setup_git_env();
> +			setup_git_env(gitdir);
>  		}
>  		if (startup_info->have_repository)
>  			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
> -- 
> 2.16.1.435.g8f24da2e1a
> 

-- 
Brandon Williams

[PATCH v2 2/4] repository.c: delete dead functions

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 45 ---------------------------------------------
 1 file changed, 45 deletions(-)

diff --git a/repository.c b/repository.c
index 343efe7282..7654b8ada9 100644
--- a/repository.c
+++ b/repository.c
@@ -16,51 +16,6 @@ static struct repository the_repo = {
 };
 struct repository *the_repository = &the_repo;
 
-static char *git_path_from_env(const char *envvar, const char *git_dir,
-			       const char *path, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(envvar);
-		if (value)
-			return xstrdup(value);
-	}
-
-	return xstrfmt("%s/%s", git_dir, path);
-}
-
-static int find_common_dir(struct strbuf *sb, const char *gitdir, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(GIT_COMMON_DIR_ENVIRONMENT);
-		if (value) {
-			strbuf_addstr(sb, value);
-			return 1;
-		}
-	}
-
-	return get_common_dir_noenv(sb, gitdir);
-}
-
-static void repo_setup_env(struct repository *repo)
-{
-	struct strbuf sb = STRBUF_INIT;
-
-	repo->different_commondir = find_common_dir(&sb, repo->gitdir,
-						    !repo->ignore_env);
-	free(repo->commondir);
-	repo->commondir = strbuf_detach(&sb, NULL);
-	raw_object_store_clear(&repo->objects);
-	repo->objects.objectdir =
-		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-				  "objects", !repo->ignore_env);
-	free(repo->graft_file);
-	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
-					     "info/grafts", !repo->ignore_env);
-	free(repo->index_file);
-	repo->index_file = git_path_from_env(INDEX_ENVIRONMENT, repo->gitdir,
-					     "index", !repo->ignore_env);
-}
-
 static void expand_base_dir(char **out, const char *in,
 			    const char *base_dir, const char *def_in)
 {
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 2/4] repository.c: delete dead functions

[Brandon Williams]

On 02/27, Nguyễn Thái Ngọc Duy wrote:
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>

These could technically be removed in the previous patch, it may make
understanding what is happening in the previous patch a bit clearer too.

> ---
>  repository.c | 45 ---------------------------------------------
>  1 file changed, 45 deletions(-)
> 
> diff --git a/repository.c b/repository.c
> index 343efe7282..7654b8ada9 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -16,51 +16,6 @@ static struct repository the_repo = {
>  };
>  struct repository *the_repository = &the_repo;
>  
> -static char *git_path_from_env(const char *envvar, const char *git_dir,
> -			       const char *path, int fromenv)
> -{
> -	if (fromenv) {
> -		const char *value = getenv(envvar);
> -		if (value)
> -			return xstrdup(value);
> -	}
> -
> -	return xstrfmt("%s/%s", git_dir, path);
> -}
> -
> -static int find_common_dir(struct strbuf *sb, const char *gitdir, int fromenv)
> -{
> -	if (fromenv) {
> -		const char *value = getenv(GIT_COMMON_DIR_ENVIRONMENT);
> -		if (value) {
> -			strbuf_addstr(sb, value);
> -			return 1;
> -		}
> -	}
> -
> -	return get_common_dir_noenv(sb, gitdir);
> -}
> -
> -static void repo_setup_env(struct repository *repo)
> -{
> -	struct strbuf sb = STRBUF_INIT;
> -
> -	repo->different_commondir = find_common_dir(&sb, repo->gitdir,
> -						    !repo->ignore_env);
> -	free(repo->commondir);
> -	repo->commondir = strbuf_detach(&sb, NULL);
> -	raw_object_store_clear(&repo->objects);
> -	repo->objects.objectdir =
> -		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
> -				  "objects", !repo->ignore_env);
> -	free(repo->graft_file);
> -	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
> -					     "info/grafts", !repo->ignore_env);
> -	free(repo->index_file);
> -	repo->index_file = git_path_from_env(INDEX_ENVIRONMENT, repo->gitdir,
> -					     "index", !repo->ignore_env);
> -}
> -
>  static void expand_base_dir(char **out, const char *in,
>  			    const char *base_dir, const char *def_in)
>  {
> -- 
> 2.16.1.435.g8f24da2e1a
> 

-- 
Brandon Williams

[PATCH v2 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

[Nguyễn Thái Ngọc Duy]

getenv() is supposed to work on the main repository only. This delayed
getenv() code in sha1_file.c makes it more difficult to convert
sha1_file.c to a generic object store that could be used by both
submodule and main repositories.

Move the getenv() back in setup_git_env() where other env vars are
also fetched.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 environment.c  | 1 +
 object-store.h | 5 ++++-
 object.c       | 1 +
 repository.c   | 2 ++
 repository.h   | 1 +
 sha1_file.c    | 6 +-----
 6 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/environment.c b/environment.c
index 74a2900ddf..47c6e31559 100644
--- a/environment.c
+++ b/environment.c
@@ -174,6 +174,7 @@ void setup_git_env(const char *git_dir)
 	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
 	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
 	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
+	args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
 	repo_set_gitdir(the_repository, git_dir, &args);
 	argv_array_clear(&to_free);
 
diff --git a/object-store.h b/object-store.h
index afe2f93459..9b1303549b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -87,6 +87,9 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	/* Path to extra alternate object database if not NULL */
+	char *alternate_db;
+
 	struct packed_git *packed_git;
 	/* A most-recently-used ordered version of the packed_git list. */
 	struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index a7c238339b..5317cfc390 100644
--- a/object.c
+++ b/object.c
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	FREE_AND_NULL(o->objectdir);
+	FREE_AND_NULL(o->alternate_db);
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
diff --git a/repository.c b/repository.c
index 7654b8ada9..e326f0fcbc 100644
--- a/repository.c
+++ b/repository.c
@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,
 	repo_set_commondir(repo, o->shared_root);
 	expand_base_dir(&repo->objects.objectdir, o->object_dir,
 			repo->commondir, "objects");
+	free(repo->objects.alternate_db);
+	repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);
 	expand_base_dir(&repo->graft_file, o->graft_file,
 			repo->commondir, "info/grafts");
 	expand_base_dir(&repo->index_file, o->index_file,
diff --git a/repository.h b/repository.h
index b5b5d138aa..b1da2a6384 100644
--- a/repository.h
+++ b/repository.h
@@ -94,6 +94,7 @@ struct set_gitdir_args {
 	const char *object_dir;
 	const char *graft_file;
 	const char *index_file;
+	const char *alternate_db;
 };
 
 extern void repo_set_gitdir(struct repository *repo,
diff --git a/sha1_file.c b/sha1_file.c
index dfc8deec38..ad1cd441e6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 
 void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
 	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
-
 	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
-	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(r, r->objects.alternate_db, PATH_SEP, NULL, 0);
 
 	read_info_alternates(r, r->objects.objectdir, 0);
 }
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

[Brandon Williams]

On 02/27, Nguyễn Thái Ngọc Duy wrote:
> getenv() is supposed to work on the main repository only. This delayed
> getenv() code in sha1_file.c makes it more difficult to convert
> sha1_file.c to a generic object store that could be used by both
> submodule and main repositories.
> 
> Move the getenv() back in setup_git_env() where other env vars are
> also fetched.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  environment.c  | 1 +
>  object-store.h | 5 ++++-
>  object.c       | 1 +
>  repository.c   | 2 ++
>  repository.h   | 1 +
>  sha1_file.c    | 6 +-----
>  6 files changed, 10 insertions(+), 6 deletions(-)
> 
> diff --git a/environment.c b/environment.c
> index 74a2900ddf..47c6e31559 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -174,6 +174,7 @@ void setup_git_env(const char *git_dir)
>  	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
>  	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
>  	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
> +	args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
>  	repo_set_gitdir(the_repository, git_dir, &args);
>  	argv_array_clear(&to_free);
>  
> diff --git a/object-store.h b/object-store.h
> index afe2f93459..9b1303549b 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -87,6 +87,9 @@ struct raw_object_store {
>  	 */
>  	char *objectdir;
>  
> +	/* Path to extra alternate object database if not NULL */
> +	char *alternate_db;
> +
>  	struct packed_git *packed_git;
>  	/* A most-recently-used ordered version of the packed_git list. */
>  	struct list_head packed_git_mru;
> @@ -109,7 +112,7 @@ struct raw_object_store {
>  	unsigned packed_git_initialized : 1;
>  };
>  
> -#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
> +#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
>  
>  void raw_object_store_clear(struct raw_object_store *o);
>  
> diff --git a/object.c b/object.c
> index a7c238339b..5317cfc390 100644
> --- a/object.c
> +++ b/object.c
> @@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)
>  void raw_object_store_clear(struct raw_object_store *o)
>  {
>  	FREE_AND_NULL(o->objectdir);
> +	FREE_AND_NULL(o->alternate_db);
>  
>  	free_alt_odbs(o);
>  	o->alt_odb_tail = NULL;
> diff --git a/repository.c b/repository.c
> index 7654b8ada9..e326f0fcbc 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,
>  	repo_set_commondir(repo, o->shared_root);
>  	expand_base_dir(&repo->objects.objectdir, o->object_dir,
>  			repo->commondir, "objects");
> +	free(repo->objects.alternate_db);
> +	repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);

Just a note that this only works because the object store is embedded in
the repository struct, it isn't a pointer to an object store.

Patch looks good.

>  	expand_base_dir(&repo->graft_file, o->graft_file,
>  			repo->commondir, "info/grafts");
>  	expand_base_dir(&repo->index_file, o->index_file,
> diff --git a/repository.h b/repository.h
> index b5b5d138aa..b1da2a6384 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -94,6 +94,7 @@ struct set_gitdir_args {
>  	const char *object_dir;
>  	const char *graft_file;
>  	const char *index_file;
> +	const char *alternate_db;
>  };
>  
>  extern void repo_set_gitdir(struct repository *repo,
> diff --git a/sha1_file.c b/sha1_file.c
> index dfc8deec38..ad1cd441e6 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
>  
>  void prepare_alt_odb(struct repository *r)
>  {
> -	const char *alt;
> -
>  	if (r->objects.alt_odb_tail)
>  		return;
>  
> -	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> -
>  	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> -	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> +	link_alt_odb_entries(r, r->objects.alternate_db, PATH_SEP, NULL, 0);
>  
>  	read_info_alternates(r, r->objects.objectdir, 0);
>  }
> -- 
> 2.16.1.435.g8f24da2e1a
> 

-- 
Brandon Williams

Re: [PATCH v2 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

[Duy Nguyen]

On Wed, Feb 28, 2018 at 3:12 AM, Brandon Williams <bmwill@google.com> wrote:
> On 02/27, Nguyễn Thái Ngọc Duy wrote:
>> diff --git a/repository.c b/repository.c
>> index 7654b8ada9..e326f0fcbc 100644
>> --- a/repository.c
>> +++ b/repository.c
>> @@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,
>>       repo_set_commondir(repo, o->shared_root);
>>       expand_base_dir(&repo->objects.objectdir, o->object_dir,
>>                       repo->commondir, "objects");
>> +     free(repo->objects.alternate_db);
>> +     repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);
>
> Just a note that this only works because the object store is embedded in
> the repository struct, it isn't a pointer to an object store.

It is possible to make it work even if object store is not embedded
though. From my point of view, this function is about "give me
$GIT_DIR and optionally the where all other parts are, if you ignore
default repo layout and tear the repository apart". We could
initialize the object store later when it's created and pass the
relevant paths to it. Exactly where it's safe to "create and
initialize object store" is harder to see now because the whole main
repo setup is spread out over many many functions.
-- 
Duy

[PATCH v2 4/4] repository: delete ignore_env member

[Nguyễn Thái Ngọc Duy]

This variable was added because the repo_set_gitdir() was created to
cover both submodule and main repos, but these two are initialized a
bit differently so ignore_env == 0 means main repo, while ignore_env
!= 0 is submodules.

Since the difference part (env variables) has been moved out of
repo_set_gitdir(), this function works the same way for both repo
types and ignore_env is not needed anymore.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 4 +---
 repository.h | 9 ---------
 2 files changed, 1 insertion(+), 12 deletions(-)

diff --git a/repository.c b/repository.c
index e326f0fcbc..c555dacad2 100644
--- a/repository.c
+++ b/repository.c
@@ -12,7 +12,7 @@ static struct repository the_repo = {
 	NULL, NULL, NULL,
 	&the_index,
 	&hash_algos[GIT_HASH_SHA1],
-	0, 0
+	0
 };
 struct repository *the_repository = &the_repo;
 
@@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
 
-	repo->ignore_env = 1;
-
 	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
 
 	if (repo_init_gitdir(repo, gitdir))
diff --git a/repository.h b/repository.h
index b1da2a6384..07e8971428 100644
--- a/repository.h
+++ b/repository.h
@@ -73,15 +73,6 @@ struct repository {
 	const struct git_hash_algo *hash_algo;
 
 	/* Configurations */
-	/*
-	 * Bit used during initialization to indicate if repository state (like
-	 * the location of the 'objectdir') should be read from the
-	 * environment.  By default this bit will be set at the begining of
-	 * 'repo_init()' so that all repositories will ignore the environment.
-	 * The exception to this is 'the_repository', which doesn't go through
-	 * the normal 'repo_init()' process.
-	 */
-	unsigned ignore_env:1;
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
 	unsigned different_commondir:1;
-- 
2.16.1.435.g8f24da2e1a

Re: [PATCH v2 4/4] repository: delete ignore_env member

[Brandon Williams]

On 02/27, Nguyễn Thái Ngọc Duy wrote:
> This variable was added because the repo_set_gitdir() was created to
> cover both submodule and main repos, but these two are initialized a
> bit differently so ignore_env == 0 means main repo, while ignore_env
> != 0 is submodules.
> 
> Since the difference part (env variables) has been moved out of
> repo_set_gitdir(), this function works the same way for both repo
> types and ignore_env is not needed anymore.
> 
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  repository.c | 4 +---
>  repository.h | 9 ---------
>  2 files changed, 1 insertion(+), 12 deletions(-)
> 
> diff --git a/repository.c b/repository.c
> index e326f0fcbc..c555dacad2 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -12,7 +12,7 @@ static struct repository the_repo = {
>  	NULL, NULL, NULL,
>  	&the_index,
>  	&hash_algos[GIT_HASH_SHA1],
> -	0, 0
> +	0
>  };
>  struct repository *the_repository = &the_repo;
>  
> @@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
>  	struct repository_format format;
>  	memset(repo, 0, sizeof(*repo));
>  
> -	repo->ignore_env = 1;
> -
>  	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
>  
>  	if (repo_init_gitdir(repo, gitdir))
> diff --git a/repository.h b/repository.h
> index b1da2a6384..07e8971428 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -73,15 +73,6 @@ struct repository {
>  	const struct git_hash_algo *hash_algo;
>  
>  	/* Configurations */
> -	/*
> -	 * Bit used during initialization to indicate if repository state (like
> -	 * the location of the 'objectdir') should be read from the
> -	 * environment.  By default this bit will be set at the begining of
> -	 * 'repo_init()' so that all repositories will ignore the environment.
> -	 * The exception to this is 'the_repository', which doesn't go through
> -	 * the normal 'repo_init()' process.
> -	 */
> -	unsigned ignore_env:1;

Nice, glad to see this get a bit less complicated.  I'm also very glad I
didn't end up using this variable when doing anything else but
initializing a repo object otherwise you would have had a bit more
difficult time getting rid of it :)

Thanks!

>  
>  	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
>  	unsigned different_commondir:1;
> -- 
> 2.16.1.435.g8f24da2e1a
> 

-- 
Brandon Williams

Re: [PATCH v2 0/4] Delete ignore_env member in struct repository

[Eric Sunshine]

On Tue, Feb 27, 2018 at 4:58 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> v2 fixes the incorrect use of consecutive getenv() and adds a comment
> to clarify the role of old_gitdir
>
> diff --git a/environment.c b/environment.c
> @@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace)
> +/* Wrapper of getenv() that returns a strdup value. This value is kept
> + * in argv to be freed later.
> + */

    /*
     * Comment style.
     */

> +static const char *getenv_safe(struct argv_array *argv, const char *name)
> +{
> +       const char *value = getenv(name);
> +
> +       if (!value)
> +               return NULL;
> +
> +       argv_array_push(argv, value);
> +       return argv->argv[argv->argc - 1];
> +}
> +
>  void setup_git_env(const char *git_dir)
>  {
>         const char *shallow_file;
>         const char *replace_ref_base;
>         struct set_gitdir_args args = { NULL };
> +       struct argv_array to_free = ARGV_ARRAY_INIT;

Cute. Another example[1] showing that renaming argv_array to something
else might be a good idea.

[1]: https://public-inbox.org/git/20180227062128.GG65699@aiede.svl.corp.google.com/

> -       args.shared_root = getenv(GIT_COMMON_DIR_ENVIRONMENT);
> -       args.object_dir = getenv(DB_ENVIRONMENT);
> -       args.graft_file = getenv(GRAFT_ENVIRONMENT);
> -       args.index_file = getenv(INDEX_ENVIRONMENT);
> -       args.alternate_db = getenv(ALTERNATE_DB_ENVIRONMENT);
> +       args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
> +       args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
> +       args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
> +       args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
> +       args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
>         repo_set_gitdir(the_repository, git_dir, &args);
> +       argv_array_clear(&to_free);
> diff --git a/repository.c b/repository.c
> @@ -48,6 +48,11 @@ void repo_set_gitdir(struct repository *repo,
>         const char *gitfile = read_gitfile(root);
> +       /*
> +        * repo->gitdir is saved because the caller could pass "root"
> +        * that also points to repo->gitdir. We want to keep it alive
> +        * until after xstrdup(root). Then we can free it.
> +        */
>         char *old_gitdir = repo->gitdir;

Good. This comment makes the reasoning clear.

>         repo->gitdir = xstrdup(gitfile ? gitfile : root);

Re: [PATCH v2 0/4] Delete ignore_env member in struct repository

[Stefan Beller]

On Tue, Feb 27, 2018 at 1:58 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> v2 fixes the incorrect use of consecutive getenv() and adds a comment
> to clarify the role of old_gitdir
>
> Interdiff:
>
> diff --git a/environment.c b/environment.c
> index 95de419de8..47c6e31559 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -14,6 +14,7 @@
>  #include "fmt-merge-msg.h"
>  #include "commit.h"
>  #include "object-store.h"
> +#include "argv-array.h"
>
>  int trust_executable_bit = 1;
>  int trust_ctime = 1;
> @@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace)
>         return strbuf_detach(&buf, NULL);
>  }
>
> +/* Wrapper of getenv() that returns a strdup value. This value is kept
> + * in argv to be freed later.
> + */

/* comment style, see also Erics reply */

I do not understand why we need to use an argv_array here?
I see that the push calls xstrdup and then puts it into the array.
So to me this looks like a clever way that we can easily free them all
at once using the predefined argv_array_clear() after calling
repo_set_gitdir.

That makes sense, though is confusing at first; I would have expected
a set_gitdir_args_clear() function that just frees all its strings, whether they
are NULL or not.

But given that we are clever with not pushing onto the argv_array in case the
getenv returns NULL, this seems to be less work in the usual case of no
env variables set.

Ok, I think I like it.

> +static const char *getenv_safe(struct argv_array *argv, const char *name)

I would have expected that we already have such a (or similar) helper
in wrapper.c, but it turns out we always take care of getenv manually,
and we do not have a wrapper yet.

So only the comment style remains as a nit.

Thanks,
Stefan

[PATCH v3 0/4] Delete ignore_env member in struct repository

[Nguyễn Thái Ngọc Duy]

v3 fixes comment style. Also since Brandon raised a question about
shared_root, it's obviously not a good name, so I renamed it to
commondir.

I still keep the delete patch 2/4, but I move the repo_setup_env()
deletion back to 1/4 so all env logic is in one patch (the
introduction of new helper functions in 1/4 and deletion in 2/4 are
still diff noise if 2/4 is completely merged back).

Interdiff:

diff --git a/environment.c b/environment.c
index 47c6e31559..b2128c1188 100644
--- a/environment.c
+++ b/environment.c
@@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)
 	return strbuf_detach(&buf, NULL);
 }
 
-/* Wrapper of getenv() that returns a strdup value. This value is kept
+/*
+ * Wrapper of getenv() that returns a strdup value. This value is kept
  * in argv to be freed later.
  */
 static const char *getenv_safe(struct argv_array *argv, const char *name)
@@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)
 	struct set_gitdir_args args = { NULL };
 	struct argv_array to_free = ARGV_ARRAY_INIT;
 
-	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
+	args.commondir = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
 	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
 	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
 	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
diff --git a/repository.c b/repository.c
index c555dacad2..4f44384dde 100644
--- a/repository.c
+++ b/repository.c
@@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in,
 }
 
 static void repo_set_commondir(struct repository *repo,
-			       const char *shared_root)
+			       const char *commondir)
 {
 	struct strbuf sb = STRBUF_INIT;
 
 	free(repo->commondir);
 
-	if (shared_root) {
+	if (commondir) {
 		repo->different_commondir = 1;
-		repo->commondir = xstrdup(shared_root);
+		repo->commondir = xstrdup(commondir);
 		return;
 	}
 
@@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo,
 	repo->gitdir = xstrdup(gitfile ? gitfile : root);
 	free(old_gitdir);
 
-	repo_set_commondir(repo, o->shared_root);
+	repo_set_commondir(repo, o->commondir);
 	expand_base_dir(&repo->objects.objectdir, o->object_dir,
 			repo->commondir, "objects");
 	free(repo->objects.alternate_db);
diff --git a/repository.h b/repository.h
index 07e8971428..e05a77a099 100644
--- a/repository.h
+++ b/repository.h
@@ -81,7 +81,7 @@ struct repository {
 extern struct repository *the_repository;
 
 struct set_gitdir_args {
-	const char *shared_root;
+	const char *commondir;
 	const char *object_dir;
 	const char *graft_file;
 	const char *index_file;

Nguyễn Thái Ngọc Duy (4):
  repository.c: move env-related setup code back to environment.c
  repository.c: delete dead functions
  sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
  repository: delete ignore_env member

 cache.h        |  2 +-
 environment.c  | 31 +++++++++++++++++--
 object-store.h |  5 ++-
 object.c       |  1 +
 repository.c   | 84 ++++++++++++++++++++++++--------------------------
 repository.h   | 21 +++++++------
 setup.c        |  3 +-
 sha1_file.c    |  6 +---
 8 files changed, 87 insertions(+), 66 deletions(-)

-- 
2.16.1.399.g632f88eed1

[PATCH v3 1/4] repository.c: move env-related setup code back to environment.c

[Nguyễn Thái Ngọc Duy]

It does not make sense that generic repository code contains handling
of environment variables, which are specific for the main repository
only. Refactor repo_set_gitdir() function to take $GIT_DIR and
optionally _all_ other customizable paths. These optional paths can be
NULL and will be calculated according to the default directory layout.

Note that some dead functions are left behind to reduce diff
noise. They will be deleted in the next patch.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 cache.h       |  2 +-
 environment.c | 30 +++++++++++++++++++++++---
 repository.c  | 59 ++++++++++++++++++++++++++++++++++-----------------
 repository.h  | 11 +++++++++-
 setup.c       |  3 +--
 5 files changed, 79 insertions(+), 26 deletions(-)

diff --git a/cache.h b/cache.h
index 5717399183..b164a407eb 100644
--- a/cache.h
+++ b/cache.h
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)
  */
 extern const char * const local_repo_env[];
 
-extern void setup_git_env(void);
+extern void setup_git_env(const char *git_dir);
 
 /*
  * Returns true iff we have a configured git repository (either via
diff --git a/environment.c b/environment.c
index ec10b062e6..454e435bed 100644
--- a/environment.c
+++ b/environment.c
@@ -14,6 +14,7 @@
 #include "fmt-merge-msg.h"
 #include "commit.h"
 #include "object-store.h"
+#include "argv-array.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -148,10 +149,34 @@ static char *expand_namespace(const char *raw_namespace)
 	return strbuf_detach(&buf, NULL);
 }
 
-void setup_git_env(void)
+/*
+ * Wrapper of getenv() that returns a strdup value. This value is kept
+ * in argv to be freed later.
+ */
+static const char *getenv_safe(struct argv_array *argv, const char *name)
+{
+	const char *value = getenv(name);
+
+	if (!value)
+		return NULL;
+
+	argv_array_push(argv, value);
+	return argv->argv[argv->argc - 1];
+}
+
+void setup_git_env(const char *git_dir)
 {
 	const char *shallow_file;
 	const char *replace_ref_base;
+	struct set_gitdir_args args = { NULL };
+	struct argv_array to_free = ARGV_ARRAY_INIT;
+
+	args.commondir = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
+	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
+	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
+	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
+	repo_set_gitdir(the_repository, git_dir, &args);
+	argv_array_clear(&to_free);
 
 	if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT))
 		check_replace_refs = 0;
@@ -301,8 +326,7 @@ int set_git_dir(const char *path)
 {
 	if (setenv(GIT_DIR_ENVIRONMENT, path, 1))
 		return error("Could not set GIT_DIR to '%s'", path);
-	repo_set_gitdir(the_repository, path);
-	setup_git_env();
+	setup_git_env(path);
 	return 0;
 }
 
diff --git a/repository.c b/repository.c
index a069b1b640..ae117efbf0 100644
--- a/repository.c
+++ b/repository.c
@@ -41,35 +41,55 @@ static int find_common_dir(struct strbuf *sb, const char *gitdir, int fromenv)
 	return get_common_dir_noenv(sb, gitdir);
 }
 
-static void repo_setup_env(struct repository *repo)
+static void expand_base_dir(char **out, const char *in,
+			    const char *base_dir, const char *def_in)
+{
+	free(*out);
+	if (in)
+		*out = xstrdup(in);
+	else
+		*out = xstrfmt("%s/%s", base_dir, def_in);
+}
+
+static void repo_set_commondir(struct repository *repo,
+			       const char *commondir)
 {
 	struct strbuf sb = STRBUF_INIT;
 
-	repo->different_commondir = find_common_dir(&sb, repo->gitdir,
-						    !repo->ignore_env);
 	free(repo->commondir);
+
+	if (commondir) {
+		repo->different_commondir = 1;
+		repo->commondir = xstrdup(commondir);
+		return;
+	}
+
+	repo->different_commondir = get_common_dir_noenv(&sb, repo->gitdir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	raw_object_store_clear(&repo->objects);
-	repo->objects.objectdir =
-		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-				  "objects", !repo->ignore_env);
-	free(repo->graft_file);
-	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
-					     "info/grafts", !repo->ignore_env);
-	free(repo->index_file);
-	repo->index_file = git_path_from_env(INDEX_ENVIRONMENT, repo->gitdir,
-					     "index", !repo->ignore_env);
 }
 
-void repo_set_gitdir(struct repository *repo, const char *path)
+void repo_set_gitdir(struct repository *repo,
+		     const char *root,
+		     const struct set_gitdir_args *o)
 {
-	const char *gitfile = read_gitfile(path);
+	const char *gitfile = read_gitfile(root);
+	/*
+	 * repo->gitdir is saved because the caller could pass "root"
+	 * that also points to repo->gitdir. We want to keep it alive
+	 * until after xstrdup(root). Then we can free it.
+	 */
 	char *old_gitdir = repo->gitdir;
 
-	repo->gitdir = xstrdup(gitfile ? gitfile : path);
-	repo_setup_env(repo);
-
+	repo->gitdir = xstrdup(gitfile ? gitfile : root);
 	free(old_gitdir);
+
+	repo_set_commondir(repo, o->commondir);
+	expand_base_dir(&repo->objects.objectdir, o->object_dir,
+			repo->commondir, "objects");
+	expand_base_dir(&repo->graft_file, o->graft_file,
+			repo->commondir, "info/grafts");
+	expand_base_dir(&repo->index_file, o->index_file,
+			repo->gitdir, "index");
 }
 
 void repo_set_hash_algo(struct repository *repo, int hash_algo)
@@ -87,6 +107,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 	int error = 0;
 	char *abspath = NULL;
 	const char *resolved_gitdir;
+	struct set_gitdir_args args = { NULL };
 
 	abspath = real_pathdup(gitdir, 0);
 	if (!abspath) {
@@ -101,7 +122,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)
 		goto out;
 	}
 
-	repo_set_gitdir(repo, resolved_gitdir);
+	repo_set_gitdir(repo, resolved_gitdir, &args);
 
 out:
 	free(abspath);
diff --git a/repository.h b/repository.h
index fa73ab8e93..f917baa584 100644
--- a/repository.h
+++ b/repository.h
@@ -89,7 +89,16 @@ struct repository {
 
 extern struct repository *the_repository;
 
-extern void repo_set_gitdir(struct repository *repo, const char *path);
+struct set_gitdir_args {
+	const char *commondir;
+	const char *object_dir;
+	const char *graft_file;
+	const char *index_file;
+};
+
+extern void repo_set_gitdir(struct repository *repo,
+			    const char *root,
+			    const struct set_gitdir_args *optional);
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
 extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index c5d55dcee4..6fac1bb58a 100644
--- a/setup.c
+++ b/setup.c
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			const char *gitdir = getenv(GIT_DIR_ENVIRONMENT);
 			if (!gitdir)
 				gitdir = DEFAULT_GIT_DIR_ENVIRONMENT;
-			repo_set_gitdir(the_repository, gitdir);
-			setup_git_env();
+			setup_git_env(gitdir);
 		}
 		if (startup_info->have_repository)
 			repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
-- 
2.16.1.399.g632f88eed1

[PATCH v3 2/4] repository.c: delete dead functions

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 25 -------------------------
 1 file changed, 25 deletions(-)

diff --git a/repository.c b/repository.c
index ae117efbf0..89e76173a3 100644
--- a/repository.c
+++ b/repository.c
@@ -16,31 +16,6 @@ static struct repository the_repo = {
 };
 struct repository *the_repository = &the_repo;
 
-static char *git_path_from_env(const char *envvar, const char *git_dir,
-			       const char *path, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(envvar);
-		if (value)
-			return xstrdup(value);
-	}
-
-	return xstrfmt("%s/%s", git_dir, path);
-}
-
-static int find_common_dir(struct strbuf *sb, const char *gitdir, int fromenv)
-{
-	if (fromenv) {
-		const char *value = getenv(GIT_COMMON_DIR_ENVIRONMENT);
-		if (value) {
-			strbuf_addstr(sb, value);
-			return 1;
-		}
-	}
-
-	return get_common_dir_noenv(sb, gitdir);
-}
-
 static void expand_base_dir(char **out, const char *in,
 			    const char *base_dir, const char *def_in)
 {
-- 
2.16.1.399.g632f88eed1

[PATCH v3 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()

[Nguyễn Thái Ngọc Duy]

getenv() is supposed to work on the main repository only. This delayed
getenv() code in sha1_file.c makes it more difficult to convert
sha1_file.c to a generic object store that could be used by both
submodule and main repositories.

Move the getenv() back in setup_git_env() where other env vars are
also fetched.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 environment.c  | 1 +
 object-store.h | 5 ++++-
 object.c       | 1 +
 repository.c   | 2 ++
 repository.h   | 1 +
 sha1_file.c    | 6 +-----
 6 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/environment.c b/environment.c
index 454e435bed..b2128c1188 100644
--- a/environment.c
+++ b/environment.c
@@ -175,6 +175,7 @@ void setup_git_env(const char *git_dir)
 	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
 	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
 	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
+	args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT);
 	repo_set_gitdir(the_repository, git_dir, &args);
 	argv_array_clear(&to_free);
 
diff --git a/object-store.h b/object-store.h
index afe2f93459..9b1303549b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -87,6 +87,9 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	/* Path to extra alternate object database if not NULL */
+	char *alternate_db;
+
 	struct packed_git *packed_git;
 	/* A most-recently-used ordered version of the packed_git list. */
 	struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index a7c238339b..5317cfc390 100644
--- a/object.c
+++ b/object.c
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	FREE_AND_NULL(o->objectdir);
+	FREE_AND_NULL(o->alternate_db);
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
diff --git a/repository.c b/repository.c
index 89e76173a3..b5306ddaa2 100644
--- a/repository.c
+++ b/repository.c
@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,
 	repo_set_commondir(repo, o->commondir);
 	expand_base_dir(&repo->objects.objectdir, o->object_dir,
 			repo->commondir, "objects");
+	free(repo->objects.alternate_db);
+	repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);
 	expand_base_dir(&repo->graft_file, o->graft_file,
 			repo->commondir, "info/grafts");
 	expand_base_dir(&repo->index_file, o->index_file,
diff --git a/repository.h b/repository.h
index f917baa584..1b6afd0926 100644
--- a/repository.h
+++ b/repository.h
@@ -94,6 +94,7 @@ struct set_gitdir_args {
 	const char *object_dir;
 	const char *graft_file;
 	const char *index_file;
+	const char *alternate_db;
 };
 
 extern void repo_set_gitdir(struct repository *repo,
diff --git a/sha1_file.c b/sha1_file.c
index dfc8deec38..ad1cd441e6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 
 void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
 	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
-
 	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
-	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(r, r->objects.alternate_db, PATH_SEP, NULL, 0);
 
 	read_info_alternates(r, r->objects.objectdir, 0);
 }
-- 
2.16.1.399.g632f88eed1

[PATCH v3 4/4] repository: delete ignore_env member

[Nguyễn Thái Ngọc Duy]

This variable was added because the repo_set_gitdir() was created to
cover both submodule and main repos, but these two are initialized a
bit differently so ignore_env == 0 means main repo, while ignore_env
!= 0 is submodules.

Since the difference part (env variables) has been moved out of
repo_set_gitdir(), this function works the same way for both repo
types and ignore_env is not needed anymore.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 repository.c | 4 +---
 repository.h | 9 ---------
 2 files changed, 1 insertion(+), 12 deletions(-)

diff --git a/repository.c b/repository.c
index b5306ddaa2..4f44384dde 100644
--- a/repository.c
+++ b/repository.c
@@ -12,7 +12,7 @@ static struct repository the_repo = {
 	NULL, NULL, NULL,
 	&the_index,
 	&hash_algos[GIT_HASH_SHA1],
-	0, 0
+	0
 };
 struct repository *the_repository = &the_repo;
 
@@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
 
-	repo->ignore_env = 1;
-
 	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
 
 	if (repo_init_gitdir(repo, gitdir))
diff --git a/repository.h b/repository.h
index 1b6afd0926..e05a77a099 100644
--- a/repository.h
+++ b/repository.h
@@ -73,15 +73,6 @@ struct repository {
 	const struct git_hash_algo *hash_algo;
 
 	/* Configurations */
-	/*
-	 * Bit used during initialization to indicate if repository state (like
-	 * the location of the 'objectdir') should be read from the
-	 * environment.  By default this bit will be set at the begining of
-	 * 'repo_init()' so that all repositories will ignore the environment.
-	 * The exception to this is 'the_repository', which doesn't go through
-	 * the normal 'repo_init()' process.
-	 */
-	unsigned ignore_env:1;
 
 	/* Indicate if a repository has a different 'commondir' from 'gitdir' */
 	unsigned different_commondir:1;
-- 
2.16.1.399.g632f88eed1

Re: [PATCH v3 0/4] Delete ignore_env member in struct repository

[Brandon Williams]

On 02/28, Nguyễn Thái Ngọc Duy wrote:
> v3 fixes comment style. Also since Brandon raised a question about
> shared_root, it's obviously not a good name, so I renamed it to
> commondir.

Thanks, its a bit clearer and more consistent with the rest of the
terminology we already have :)

> 
> I still keep the delete patch 2/4, but I move the repo_setup_env()
> deletion back to 1/4 so all env logic is in one patch (the
> introduction of new helper functions in 1/4 and deletion in 2/4 are
> still diff noise if 2/4 is completely merged back).
> 
> Interdiff:
> 
> diff --git a/environment.c b/environment.c
> index 47c6e31559..b2128c1188 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)
>  	return strbuf_detach(&buf, NULL);
>  }
>  
> -/* Wrapper of getenv() that returns a strdup value. This value is kept
> +/*
> + * Wrapper of getenv() that returns a strdup value. This value is kept
>   * in argv to be freed later.
>   */
>  static const char *getenv_safe(struct argv_array *argv, const char *name)
> @@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)
>  	struct set_gitdir_args args = { NULL };
>  	struct argv_array to_free = ARGV_ARRAY_INIT;
>  
> -	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
> +	args.commondir = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
>  	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
>  	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
>  	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
> diff --git a/repository.c b/repository.c
> index c555dacad2..4f44384dde 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in,
>  }
>  
>  static void repo_set_commondir(struct repository *repo,
> -			       const char *shared_root)
> +			       const char *commondir)
>  {
>  	struct strbuf sb = STRBUF_INIT;
>  
>  	free(repo->commondir);
>  
> -	if (shared_root) {
> +	if (commondir) {
>  		repo->different_commondir = 1;
> -		repo->commondir = xstrdup(shared_root);
> +		repo->commondir = xstrdup(commondir);
>  		return;
>  	}
>  
> @@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo,
>  	repo->gitdir = xstrdup(gitfile ? gitfile : root);
>  	free(old_gitdir);
>  
> -	repo_set_commondir(repo, o->shared_root);
> +	repo_set_commondir(repo, o->commondir);
>  	expand_base_dir(&repo->objects.objectdir, o->object_dir,
>  			repo->commondir, "objects");
>  	free(repo->objects.alternate_db);
> diff --git a/repository.h b/repository.h
> index 07e8971428..e05a77a099 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -81,7 +81,7 @@ struct repository {
>  extern struct repository *the_repository;
>  
>  struct set_gitdir_args {
> -	const char *shared_root;
> +	const char *commondir;
>  	const char *object_dir;
>  	const char *graft_file;
>  	const char *index_file;
> 
> Nguyễn Thái Ngọc Duy (4):
>   repository.c: move env-related setup code back to environment.c
>   repository.c: delete dead functions
>   sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
>   repository: delete ignore_env member
> 
>  cache.h        |  2 +-
>  environment.c  | 31 +++++++++++++++++--
>  object-store.h |  5 ++-
>  object.c       |  1 +
>  repository.c   | 84 ++++++++++++++++++++++++--------------------------
>  repository.h   | 21 +++++++------
>  setup.c        |  3 +-
>  sha1_file.c    |  6 +---
>  8 files changed, 87 insertions(+), 66 deletions(-)
> 
> -- 
> 2.16.1.399.g632f88eed1
> 

-- 
Brandon Williams

Re: [PATCH v3 0/4] Delete ignore_env member in struct repository

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

> v3 fixes comment style. Also since Brandon raised a question about
> shared_root, it's obviously not a good name, so I renamed it to
> commondir.
>
> I still keep the delete patch 2/4, but I move the repo_setup_env()
> deletion back to 1/4 so all env logic is in one patch (the
> introduction of new helper functions in 1/4 and deletion in 2/4 are
> still diff noise if 2/4 is completely merged back).

Thanks.  After replacing sb/object-store with the last one posted
and re-tweaking resolution of conflicts with ds/commit-graph that is
in flight, I think I can squeeze this topic in.  Among the three, I
find that this kind of clean-up is the most interesting/urgent thing
we need before we can move forward ;-)



>
> Interdiff:
>
> diff --git a/environment.c b/environment.c
> index 47c6e31559..b2128c1188 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)
>  	return strbuf_detach(&buf, NULL);
>  }
>  
> -/* Wrapper of getenv() that returns a strdup value. This value is kept
> +/*
> + * Wrapper of getenv() that returns a strdup value. This value is kept
>   * in argv to be freed later.
>   */
>  static const char *getenv_safe(struct argv_array *argv, const char *name)
> @@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)
>  	struct set_gitdir_args args = { NULL };
>  	struct argv_array to_free = ARGV_ARRAY_INIT;
>  
> -	args.shared_root = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
> +	args.commondir = getenv_safe(&to_free, GIT_COMMON_DIR_ENVIRONMENT);
>  	args.object_dir = getenv_safe(&to_free, DB_ENVIRONMENT);
>  	args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT);
>  	args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT);
> diff --git a/repository.c b/repository.c
> index c555dacad2..4f44384dde 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in,
>  }
>  
>  static void repo_set_commondir(struct repository *repo,
> -			       const char *shared_root)
> +			       const char *commondir)
>  {
>  	struct strbuf sb = STRBUF_INIT;
>  
>  	free(repo->commondir);
>  
> -	if (shared_root) {
> +	if (commondir) {
>  		repo->different_commondir = 1;
> -		repo->commondir = xstrdup(shared_root);
> +		repo->commondir = xstrdup(commondir);
>  		return;
>  	}
>  
> @@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo,
>  	repo->gitdir = xstrdup(gitfile ? gitfile : root);
>  	free(old_gitdir);
>  
> -	repo_set_commondir(repo, o->shared_root);
> +	repo_set_commondir(repo, o->commondir);
>  	expand_base_dir(&repo->objects.objectdir, o->object_dir,
>  			repo->commondir, "objects");
>  	free(repo->objects.alternate_db);
> diff --git a/repository.h b/repository.h
> index 07e8971428..e05a77a099 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -81,7 +81,7 @@ struct repository {
>  extern struct repository *the_repository;
>  
>  struct set_gitdir_args {
> -	const char *shared_root;
> +	const char *commondir;
>  	const char *object_dir;
>  	const char *graft_file;
>  	const char *index_file;
>
> Nguyễn Thái Ngọc Duy (4):
>   repository.c: move env-related setup code back to environment.c
>   repository.c: delete dead functions
>   sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
>   repository: delete ignore_env member
>
>  cache.h        |  2 +-
>  environment.c  | 31 +++++++++++++++++--
>  object-store.h |  5 ++-
>  object.c       |  1 +
>  repository.c   | 84 ++++++++++++++++++++++++--------------------------
>  repository.h   | 21 +++++++------
>  setup.c        |  3 +-
>  sha1_file.c    |  6 +---
>  8 files changed, 87 insertions(+), 66 deletions(-)

[PATCH 15/26] sha1_file: add repository argument to sha1_file_name

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  8 --------
 http-walker.c  |  3 ++-
 http.c         |  5 +++--
 object-store.h |  9 +++++++++
 sha1_file.c    | 11 ++++++-----
 5 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/cache.h b/cache.h
index 09d37cb651..4b01c0de10 100644
--- a/cache.h
+++ b/cache.h
@@ -956,14 +956,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Return the name of the file in the local object database that would
- * be used to store a loose object with the specified sha1.  The
- * return value is a pointer to a statically allocated buffer that is
- * overwritten each time the function is called.
- */
-extern const char *sha1_file_name(const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index a1c6f2639b..96873bdfed 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		ret = error("unable to write sha1 filename %s",
-			    sha1_file_name(req->sha1));
+			    sha1_file_name(the_repository, req->sha1));
 	}
 
 	release_http_object_request(req);
diff --git a/http.c b/http.c
index ab989b88dd..c7d2b74ce2 100644
--- a/http.c
+++ b/http.c
@@ -2181,7 +2181,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	filename = sha1_file_name(sha1);
+	filename = sha1_file_name(the_repository, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename);
 
@@ -2329,7 +2329,8 @@ int finish_http_object_request(struct http_object_request *freq)
 		return -1;
 	}
 	freq->rename =
-		finalize_object_file(freq->tmpfile, sha1_file_name(freq->sha1));
+		finalize_object_file(freq->tmpfile,
+				     sha1_file_name(the_repository, freq->sha1));
 
 	return freq->rename;
 }
diff --git a/object-store.h b/object-store.h
index add1d4e27c..5422e80c08 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,6 +61,15 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
+/*
+ * Return the name of the file in a repository's local object database
+ * that would be used to store a loose object with the specified sha1.
+ * The return value is a pointer to a statically allocated buffer that
+ * is overwritten each time the function is called.
+ */
+#define sha1_file_name(r, s) sha1_file_name_##r(s)
+const char *sha1_file_name_the_repository(const unsigned char *sha1);
+
 void prepare_alt_odb(struct repository *r);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index f046d560f8..65fefdf4ac 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-const char *sha1_file_name(const unsigned char *sha1)
+const char *sha1_file_name_the_repository(const unsigned char *sha1)
 {
 	static struct strbuf buf = STRBUF_INIT;
 
@@ -721,7 +721,8 @@ int check_and_freshen_file(const char *fn, int freshen)
 
 static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 {
-	return check_and_freshen_file(sha1_file_name(sha1), freshen);
+	return check_and_freshen_file(sha1_file_name(the_repository, sha1),
+				      freshen);
 }
 
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
@@ -878,7 +879,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 {
 	struct alternate_object_database *alt;
 
-	*path = sha1_file_name(sha1);
+	*path = sha1_file_name(the_repository, sha1);
 	if (!lstat(*path, st))
 		return 0;
 
@@ -903,7 +904,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	struct alternate_object_database *alt;
 	int most_interesting_errno;
 
-	*path = sha1_file_name(sha1);
+	*path = sha1_file_name(the_repository, sha1);
 	fd = git_open(*path);
 	if (fd >= 0)
 		return fd;
@@ -1568,7 +1569,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	git_SHA_CTX c;
 	unsigned char parano_sha1[20];
 	static struct strbuf tmp_file = STRBUF_INIT;
-	const char *filename = sha1_file_name(sha1);
+	const char *filename = sha1_file_name(the_repository, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename);
 	if (fd < 0) {
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 16/26] sha1_file: add repository argument to stat_sha1_file

[Stefan Beller]

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 65fefdf4ac..1d81bafe56 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 
@@ -1173,7 +1174,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1372,7 +1373,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 17/26] sha1_file: add repository argument to open_sha1_file

[Stefan Beller]

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1d81bafe56..e9e01fb471 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -899,7 +899,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -938,7 +940,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 18/26] sha1_file: add repository argument to map_sha1_file_1

[Stefan Beller]

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index e9e01fb471..f706526bc2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -930,9 +930,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -961,7 +962,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2173,7 +2174,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 19/26] sha1_file: add repository argument to map_sha1_file

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 2 ++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index 4b01c0de10..d0a1236dac 100644
--- a/cache.h
+++ b/cache.h
@@ -1237,7 +1237,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 5422e80c08..0a4561f476 100644
--- a/object-store.h
+++ b/object-store.h
@@ -69,6 +69,8 @@ struct packed_git {
  */
 #define sha1_file_name(r, s) sha1_file_name_##r(s)
 const char *sha1_file_name_the_repository(const unsigned char *sha1);
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
 
 void prepare_alt_odb(struct repository *r);
 
diff --git a/sha1_file.c b/sha1_file.c
index f706526bc2..1ce4058644 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -960,7 +960,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1184,7 +1184,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..22d27df55e 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 20/26] sha1_file: add repository argument to sha1_loose_object_info

[Stefan Beller]

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1ce4058644..d0bc09a089 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1151,9 +1151,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1266,7 +1267,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 	if (!find_pack_entry(real, &e)) {
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 21/26] sha1_file: allow sha1_file_name to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index 0a4561f476..b4756444bb 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,8 +67,7 @@ struct packed_git {
  * The return value is a pointer to a statically allocated buffer that
  * is overwritten each time the function is called.
  */
-#define sha1_file_name(r, s) sha1_file_name_##r(s)
-const char *sha1_file_name_the_repository(const unsigned char *sha1);
+const char *sha1_file_name(struct repository *r, const unsigned char *sha1);
 #define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
 void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
 
diff --git a/sha1_file.c b/sha1_file.c
index d0bc09a089..fddada5756 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,12 +323,12 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-const char *sha1_file_name_the_repository(const unsigned char *sha1)
+const char *sha1_file_name(struct repository *r, const unsigned char *sha1)
 {
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/", get_object_directory());
+	strbuf_addf(&buf, "%s/", r->objects.objectdir);
 
 	fill_sha1_path(&buf, sha1);
 	return buf.buf;
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 22/26] sha1_file: allow stat_sha1_file to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index fddada5756..21ddbff846 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,19 +874,18 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
-static int stat_sha1_file_the_repository(const unsigned char *sha1,
-					 struct stat *st, const char **path)
+static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 
-	*path = sha1_file_name(the_repository, sha1);
+	*path = sha1_file_name(r, sha1);
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(the_repository);
+	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 23/26] sha1_file: allow open_sha1_file to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 21ddbff846..50202f0959 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -898,22 +898,21 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
-static int open_sha1_file_the_repository(const unsigned char *sha1,
-					 const char **path)
+static int open_sha1_file(struct repository *r,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
 	int most_interesting_errno;
 
-	*path = sha1_file_name(the_repository, sha1);
+	*path = sha1_file_name(r, sha1);
 	fd = git_open(*path);
 	if (fd >= 0)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 24/26] sha1_file: allow map_sha1_file_1 to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 50202f0959..86b0ca7089 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -928,10 +928,8 @@ static int open_sha1_file(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
-static void *map_sha1_file_1_the_repository(const char *path,
-					    const unsigned char *sha1,
-					    unsigned long *size)
+static void *map_sha1_file_1(struct repository *r, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -939,7 +937,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(the_repository, sha1, &path);
+		fd = open_sha1_file(r, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 25/26] sha1_file: allow map_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 5 +++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index b4756444bb..f164c4e5c9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -68,8 +68,7 @@ struct packed_git {
  * is overwritten each time the function is called.
  */
 const char *sha1_file_name(struct repository *r, const unsigned char *sha1);
-#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+extern void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
 void prepare_alt_odb(struct repository *r);
 
diff --git a/sha1_file.c b/sha1_file.c
index 86b0ca7089..6237d59a59 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -956,9 +956,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,
 	return map;
 }
 
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct repository *r,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(the_repository, NULL, sha1, size);
+	return map_sha1_file_1(r, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
-- 
2.16.1.73.ga2c3e9663f.dirty

[PATCH 26/26] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6237d59a59..c3f35914ce 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1148,10 +1148,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
-static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
-						 struct object_info *oi,
-						 int flags)
+static int sha1_loose_object_info(struct repository *r,
+				  const unsigned char *sha1,
+				  struct object_info *oi, int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1175,14 +1174,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
+		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(the_repository, sha1, &mapsize);
+	map = map_sha1_file(r, sha1, &mapsize);
 	if (!map)
 		return -1;
 
-- 
2.16.1.73.ga2c3e9663f.dirty

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

On Mon, Feb 12, 2018 at 5:22 PM, Stefan Beller <sbeller@google.com> wrote:

> I developed this series [...] on top of current master.

also available at https://github.com/stefanbeller/git/tree/object-store-part1

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Duy Nguyen]

On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
> This is a real take on the first part of the recent RFC[1].
>
> ...
> 
> Duy suggested that we shall not use the repository blindly, but
> should carefully examine whether to pass on an object store or the
> refstore or such[4], which I agree with if it makes sense. This
> series unfortunately has an issue with that as I would not want to
> pass down the `ignore_env` flag separately from the object store, so
> I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full
> repository.

Second proposal :) How about you store ignore_env in raw_object_store?
This would not be the first time an object has some configuration
passed in at construction time. And it has a "constructor" now,
raw_object_store_init() (I probably should merge _setup in it too)

The core changes look like this. I have a full commit on top of your
series [1] that keeps sha1_file.c functions take 'struct
raw_object_store' instead.

[1] https://github.com/pclouds/git/tree/object-store-part1

-- 8< --
diff --git a/object-store.h b/object-store.h
index f164c4e5c9..a8bf1738f2 100644
--- a/object-store.h
+++ b/object-store.h
@@ -34,9 +34,13 @@ struct raw_object_store {
 	 * packs.
 	 */
 	unsigned packed_git_initialized : 1;
+
+	unsigned ignore_env : 1;
 };
 #define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 }
 
+void raw_object_store_init(struct raw_object_store *ros, struct repository *r);
+void raw_object_store_setup(struct raw_object_store *ros, char *);
 void raw_object_store_clear(struct raw_object_store *o);
 
 struct packed_git {
diff --git a/object.c b/object.c
index 79c2c447bc..a4534bf4c4 100644
--- a/object.c
+++ b/object.c
@@ -461,6 +461,20 @@ static void free_alt_odbs(struct raw_object_store *o)
 	}
 }
 
+void raw_object_store_init(struct raw_object_store *o,
+			   struct repository *r)
+{
+	/* FIXME: memset? */
+	o->ignore_env = r->ignore_env;
+}
+
+void raw_object_store_setup(struct raw_object_store *o,
+			    char *objectdir)
+{
+	free(o->objectdir);
+	o->objectdir = objectdir;
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
diff --git a/repository.c b/repository.c
index bd2ad578de..ac5863d7e1 100644
--- a/repository.c
+++ b/repository.c
@@ -49,9 +49,9 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objects.objectdir);
-	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-					    "objects", !repo->ignore_env);
+	raw_object_store_setup(&repo->objects,
+			       git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+						 "objects", !repo->ignore_env));
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	raw_object_store_init(&repo->objects, repo);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/sha1_file.c b/sha1_file.c
index c3f35914ce..3be58651a1 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -677,14 +677,14 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(struct repository *r)
+void prepare_alt_odb(struct raw_object_store *ros)
 {
 	if (r->objects.alt_odb_tail)
 		return;
 
 	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
 
-	if (!r->ignore_env) {
+	if (!ros->ignore_env) {
 		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 		if (!alt)
 			alt = "";
-- 8< --


--
Duy

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Duy Nguyen]

On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
>> This is a real take on the first part of the recent RFC[1].
>>
>> ...
>>
>> Duy suggested that we shall not use the repository blindly, but
>> should carefully examine whether to pass on an object store or the
>> refstore or such[4], which I agree with if it makes sense. This
>> series unfortunately has an issue with that as I would not want to
>> pass down the `ignore_env` flag separately from the object store, so
>> I made all functions that only take the object store to have the raw
>> object store as the first parameter, and others using the full
>> repository.
>
> Second proposal :) How about you store ignore_env in raw_object_store?
> This would not be the first time an object has some configuration
> passed in at construction time. And it has a "constructor" now,
> raw_object_store_init() (I probably should merge _setup in it too)

A bit more on this configuration parameters. Down the road I think we
need something like this anyway to delete global config vars like
packed_git_window_size, delta_base_cache_limit...  Either all these
end up in raw_object_store, or raw_object_store holds a link to
"struct config_set".

The ignore_env specifically though looks to me like a stop gap
solution until everything goes through repo_init() first. At that
point we don't have to delay getenv() anymore. We can getenv() all at
repo_init() then pass them in raw_object_store and ignore_env should
be gone. So sticking it inside raw_object_store _temporarily_ does not
sound so bad.
-- 
Duy

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Brandon Williams]

On 02/13, Duy Nguyen wrote:
> On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> > On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
> >> This is a real take on the first part of the recent RFC[1].
> >>
> >> ...
> >>
> >> Duy suggested that we shall not use the repository blindly, but
> >> should carefully examine whether to pass on an object store or the
> >> refstore or such[4], which I agree with if it makes sense. This
> >> series unfortunately has an issue with that as I would not want to
> >> pass down the `ignore_env` flag separately from the object store, so
> >> I made all functions that only take the object store to have the raw
> >> object store as the first parameter, and others using the full
> >> repository.
> >
> > Second proposal :) How about you store ignore_env in raw_object_store?
> > This would not be the first time an object has some configuration
> > passed in at construction time. And it has a "constructor" now,
> > raw_object_store_init() (I probably should merge _setup in it too)
> 
> A bit more on this configuration parameters. Down the road I think we
> need something like this anyway to delete global config vars like
> packed_git_window_size, delta_base_cache_limit...  Either all these
> end up in raw_object_store, or raw_object_store holds a link to
> "struct config_set".
> 
> The ignore_env specifically though looks to me like a stop gap
> solution until everything goes through repo_init() first. At that
> point we don't have to delay getenv() anymore. We can getenv() all at
> repo_init() then pass them in raw_object_store and ignore_env should
> be gone. So sticking it inside raw_object_store _temporarily_ does not
> sound so bad.

I like this approach, I mean at the moment we are replicating a single
bit of data but that allows us to be able to limit the scope of where a
repository struct is passed, giving us a better abstraction layer.

> -- 
> Duy

-- 
Brandon Williams

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 13, 2018 at 4:13 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
>>> This is a real take on the first part of the recent RFC[1].
>>>
>>> ...
>>>
>>> Duy suggested that we shall not use the repository blindly, but
>>> should carefully examine whether to pass on an object store or the
>>> refstore or such[4], which I agree with if it makes sense. This
>>> series unfortunately has an issue with that as I would not want to
>>> pass down the `ignore_env` flag separately from the object store, so
>>> I made all functions that only take the object store to have the raw
>>> object store as the first parameter, and others using the full
>>> repository.
>>
>> Second proposal :) How about you store ignore_env in raw_object_store?
>> This would not be the first time an object has some configuration
>> passed in at construction time. And it has a "constructor" now,
>> raw_object_store_init() (I probably should merge _setup in it too)
>
> A bit more on this configuration parameters. Down the road I think we
> need something like this anyway to delete global config vars like
> packed_git_window_size, delta_base_cache_limit...  Either all these
> end up in raw_object_store, or raw_object_store holds a link to
> "struct config_set".

That makes sense long term.

> The ignore_env specifically though looks to me like a stop gap
> solution until everything goes through repo_init() first. At that
> point we don't have to delay getenv() anymore. We can getenv() all at
> repo_init() then pass them in raw_object_store and ignore_env should
> be gone. So sticking it inside raw_object_store _temporarily_ does not
> sound so bad.

Oh, that is an interesting perspective. Here is how I arrived at the opposite
conclusion initially: Searching for 'ignore_env' shows that we care about it
as well for the index and graft paths, which are not the object store, hence
it would be better kept in the repository. (The alternative would be to
duplicate it into the raw object store, but I do not like data duplication)

But maybe it is better to duplicate this one bit instead of passing through
a larger scoped object.

I'll rework the patches.

Thanks!
Stefan

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Oh, that is an interesting perspective. Here is how I arrived at the opposite
> conclusion initially: Searching for 'ignore_env' shows that we care about it
> as well for the index and graft paths, which are not the object store, hence
> it would be better kept in the repository. (The alternative would be to
> duplicate it into the raw object store, but I do not like data duplication)
>
> But maybe it is better to duplicate this one bit instead of passing through
> a larger scoped object.

If a larger scoped repository refers to a smaller scoped
object-store, is there still a need to duplicate that bit, instead
of referring to the bit the smaller scoped one has when asking about
the bit in the larger scoped one?

I am not sure if these "do not look at environment variables" is an
attribute of these entities---it sounds more like an attribute for
each invocation of an operation, i.e. "I want to learn where the
index is but would ignore GIT_INDEX environment for this particular
query." and "What's the value of this ref?  Please honor the
common-dir environment during this query".

So from that point of view, it may not matter where the "bit" lives,
among repository, object-store, or ref-store.

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 13, 2018 at 10:57 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> Oh, that is an interesting perspective. Here is how I arrived at the opposite
>> conclusion initially: Searching for 'ignore_env' shows that we care about it
>> as well for the index and graft paths, which are not the object store, hence
>> it would be better kept in the repository. (The alternative would be to
>> duplicate it into the raw object store, but I do not like data duplication)
>>
>> But maybe it is better to duplicate this one bit instead of passing through
>> a larger scoped object.
>
> If a larger scoped repository refers to a smaller scoped
> object-store, is there still a need to duplicate that bit, instead
> of referring to the bit the smaller scoped one has when asking about
> the bit in the larger scoped one?

No (in theory). But in practice it may be worthwhile:

    "What's the value of this ref?"

    "Oh let me check the ignore_env bit that happens
    to live in the object store first."

would be super confusing to me.

> I am not sure if these "do not look at environment variables" is an
> attribute of these entities---it sounds more like an attribute for
> each invocation of an operation, i.e. "I want to learn where the
> index is but would ignore GIT_INDEX environment for this particular
> query." and "What's the value of this ref?  Please honor the
> common-dir environment during this query".

That sounds like we want to have a configset struct eventually.

For now the ignore_env bit lives in the repository, as that helps
when working with submodules, when reading its comments.
Unfortunately 359efeffc1 (repository: introduce the repository
object, 2017-06-22) did not reason about the existence of the ignore_env
flag in its commit message.

> So from that point of view, it may not matter where the "bit" lives,
> among repository, object-store, or ref-store.

It matters on the scale of confusing the developer?

Stefan

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> For now the ignore_env bit lives in the repository, as that helps
> when working with submodules, when reading its comments.
> Unfortunately 359efeffc1 (repository: introduce the repository
> object, 2017-06-22) did not reason about the existence of the ignore_env
> flag in its commit message.
>
>> So from that point of view, it may not matter where the "bit" lives,
>> among repository, object-store, or ref-store.
>
> It matters on the scale of confusing the developer?

What I meant is that from the point of view, having the bit in the
data (not on the invocation) is confusing no matter which data
structure holds it--they are equally bad.

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 13, 2018 at 11:35 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> For now the ignore_env bit lives in the repository, as that helps
>> when working with submodules, when reading its comments.
>> Unfortunately 359efeffc1 (repository: introduce the repository
>> object, 2017-06-22) did not reason about the existence of the ignore_env
>> flag in its commit message.
>>
>>> So from that point of view, it may not matter where the "bit" lives,
>>> among repository, object-store, or ref-store.
>>
>> It matters on the scale of confusing the developer?
>
> What I meant is that from the point of view, having the bit in the
> data (not on the invocation) is confusing no matter which data
> structure holds it--they are equally bad.

Right.

Which is why I'd strongly consider having it only in the repository
object as that is the largest-scoped thing we'd want. e.g. submodules
should care about environment variables differently:

    GIT_WORK_TREE=~/mysuperproject git checkout \
            --recurse-submodules master

 So with such a command in mind, the environment variable would
only apply to the superproject and the nested submodules should
ignore the env, but compute their paths off the superproject, i.e.
the superproject repository, not its object store?

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> Which is why I'd strongly consider having it only in the repository
> object as that is the largest-scoped thing we'd want. e.g. submodules
> should care about environment variables differently:
>
>     GIT_WORK_TREE=~/mysuperproject git checkout \
>             --recurse-submodules master
>
>  So with such a command in mind, the environment variable would
> only apply to the superproject and the nested submodules should
> ignore the env, but compute their paths off the superproject, i.e.
> the superproject repository, not its object store?

In the longer term endgame state, what is in the repository object
cannot just be a simple "do we honor environment variable" bit
anymore.  

It will be more like "we may (or may not) look at environment when
we create a repository object", i.e. a bit passed to repo_init()
constructor, and from then on, the repository object knows where the
object store and its alternates are, where the top of the working
tree is, where the repository (i.e. the directory that has "refs/"
in it) is, what "worktree" of the repository we are talking about by
itself.  There is no need for a bit "do we or do we not look at
environment?" that needs to be consulted at runtime, which is quite
a round-about thing.

In your example, the repository object that represents the
superproject will pay attention to GIT_WORK_TREE environment when it
is consturcted, and repository objects dynamically constructed while
the command "recurse-submodules" through them will be constructed
with their "where is the top of my working tree?" set appropriately.
They won't be storing "when figuring out where my working tree is,
do not pay attention to GIT_WORK_TREE environment variable" bit.

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> This is a real take on the first part of the recent RFC[1].
>
> Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
> might be a good breaking point for a first part at that RFC at patch 38.
> This series is smaller and contains only 26 patches as the patches in the big
> RFC were slightly out of order.

Thanks.  This looks like a nice reviewable series, so I'm happy to see
it broken out.

[...]
> Comments in the early range of that RFC were on 003 where Junio pointed out
> that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
> in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

Can you say a little more about this?  Was the problem that the
semantic patch wasn't idempotent, that it was too slow to run, or
something else?

If we're including the semantic patch for reference but never running
it, I think I'd prefer it to go in the commit message.  But if it's
useful to run then we should make it idempotent so it can go in
contrib/coccinelle.

[...]
> Duy suggested that we shall not use the repository blindly, but should carefully
> examine whether to pass on an object store or the refstore or such[4], which
> I agree with if it makes sense. This series unfortunately has an issue with that
> as I would not want to pass down the `ignore_env` flag separately from the object
> store, so I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full repository.

I think I want to push back on this a little.

The advantage of a function taking e.g. an object_store as an argument
instead of a repository is that it increases its flexibility, since it
allows callers that do not have access to a repository to call it.  The
disadvantage is also that it increases the flexibility without any
callers benefitting from that:

 1. It ties us to assumptions from today.  If e.g. an object access in
    the future starts relying on some other information from the
    repository (e.g. its config) then we'd have to either add a
    back-pointer from the object store to its repository or add
    additional arguments for that additional data at that point.

    If all callers already have a repository, it is simpler to pass
    that repository as context so that we have the flexibility to make
    more use of it later.

 2. It complicates the caller.  Instead of consistently passing the
    same repository argument as context to functions that access that
    repository, the caller would have to pull out relevant fields like
    the object store from it.

 3. It prevents us from making opportunistic use of other information
    from the repository, such as its name for use in error messages.

In lower-level funcitons that need to be usable by callers without a
repository (e.g. to find packfiles in an alternate) it makes sense to
not pass a repository, but without such a use case in mind I don't
think it needs to be a general goal.

To put it another way, most callers do not *care* whether they are
working with a repository's object store, ref database, or some other
aspect of the repository.  They just know they want to e.g. read an
object from this repository.

It's similar to how FILE * works: some operations rely on the buffer
the FILE * manages and some other operations only rely on the
underlying file descriptor, but using the FILE * consistently provides
a clean abstraction that generally makes life easier.

> Eric Sunshine brought up memory leaks with the RFC, and I would think to
> have plugged all holes.

Yay, thank you!

I'll try to find time to look at the patches in detail soon, but no
promises (i.e. if someone else reviews them first, then even better
;-)).

Sincerely,
Jonathan

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Duy Nguyen]

On Wed, Feb 14, 2018 at 2:26 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Duy suggested that we shall not use the repository blindly, but should carefully
>> examine whether to pass on an object store or the refstore or such[4], which
>> I agree with if it makes sense. This series unfortunately has an issue with that
>> as I would not want to pass down the `ignore_env` flag separately from the object
>> store, so I made all functions that only take the object store to have the raw
>> object store as the first parameter, and others using the full repository.
>
> I think I want to push back on this a little.
>
> The advantage of a function taking e.g. an object_store as an argument
> instead of a repository is that it increases its flexibility, since it
> allows callers that do not have access to a repository to call it.  The
> disadvantage is also that it increases the flexibility without any
> callers benefitting from that:
>
>  1. It ties us to assumptions from today.  If e.g. an object access in
>     the future starts relying on some other information from the
>     repository (e.g. its config) then we'd have to either add a
>     back-pointer from the object store to its repository or add
>     additional arguments for that additional data at that point.
>
>     If all callers already have a repository, it is simpler to pass
>     that repository as context so that we have the flexibility to make
>     more use of it later.

It's essentially putting all global variables in the same place again.
Only this time it's not global namespace, but "struct repository".
It's no worse than the current state though.

>
>  2. It complicates the caller.  Instead of consistently passing the
>     same repository argument as context to functions that access that
>     repository, the caller would have to pull out relevant fields like
>     the object store from it.

Well, I see that as a good point :)

>
>  3. It prevents us from making opportunistic use of other information
>     from the repository, such as its name for use in error messages.

It does not exactly prevent us. It's just more effort to pass this
around. The repository name for example, there's no reason we can't
have object store name (which could be initialized the same as repo
name).

> In lower-level funcitons that need to be usable by callers without a
> repository (e.g. to find packfiles in an alternate) it makes sense to
> not pass a repository, but without such a use case in mind

I do agree with your benefit argument. But I'd like to point out
though that having all input to object store visible from something
like "struct raw_object_store" makes it easier to reason about the
code. I know how object store works, but occasionally I'm still
surprised when it getenv (or read $GIT_DIR/index, but not in object
store code) behind the scene. Imagine how hard it is for newcomers.

I would count that as benefit, even though it's not a use case per se.
Another potential benefit is writing unit tests will be much easier
(you can configure object store through struct repository too, but
setting one piece here, one piece there to control object store
behavior is not a nice experience). It's a nice thing to have, but not
a deciding factor.

> I don't think it needs to be a general goal.

My stand is a bit more aggressive. We should try to achieve better
abstraction if possible. But if it makes Stefan's life hell, it's not
worth doing. Converting to 'struct repository' is already a step
forward. Actually if it discourages him from finishing this work, it's
already not worth doing.
-- 
Duy

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Brandon Williams]

On 02/12, Stefan Beller wrote:
> This is a real take on the first part of the recent RFC[1].
> 
> Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
> might be a good breaking point for a first part at that RFC at patch 38.
> This series is smaller and contains only 26 patches as the patches in the big
> RFC were slightly out of order.
> 
> I developed this series partly by writing patches, but mostly by cherrypicking
> from that RFC on top of current master. I noticed no external conflicts apart
> from one addition to the repositories _INIT macro, which was easy to resolve.
> 
> Comments in the early range of that RFC were on 003 where Junio pointed out
> that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
> in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
> 
> brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
> without any suggestion to include into this series[3].
> 
> Duy suggested that we shall not use the repository blindly, but should carefully
> examine whether to pass on an object store or the refstore or such[4], which 
> I agree with if it makes sense. This series unfortunately has an issue with that
> as I would not want to pass down the `ignore_env` flag separately from the object
> store, so I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full repository.
> 
> Eric Sunshine brought up memory leaks with the RFC, and I would think to
> have plugged all holes.

I've looked through the patches and I think they look good.  At the end
of the series all the #define tricks have been eliminated so we don't
have to worry about them possibly being left and forgotten :)


Thanks for getting the ball rolling on this.

> 
> [1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
> [2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
> [3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
> [4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
> 
> Thanks,
> Stefan
> 
> Jonathan Nieder (8):
>   pack: move prepare_packed_git_run_once to object store
>   pack: move approximate object count to object store
>   sha1_file: add repository argument to sha1_file_name
>   sha1_file: add repository argument to map_sha1_file
>   sha1_file: allow stat_sha1_file to handle arbitrary repositories
>   sha1_file: allow open_sha1_file to handle arbitrary repositories
>   sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
>   sha1_file: allow sha1_loose_object_info to handle arbitrary
>     repositories
> 
> Stefan Beller (18):
>   repository: introduce raw object store field
>   object-store: move alt_odb_list and alt_odb_tail to object store
>   object-store: free alt_odb_list
>   object-store: move packed_git and packed_git_mru to object store
>   object-store: close all packs upon clearing the object store
>   sha1_file: add raw_object_store argument to alt_odb_usable
>   sha1_file: add repository argument to link_alt_odb_entry
>   sha1_file: add repository argument to read_info_alternates
>   sha1_file: add repository argument to link_alt_odb_entries
>   sha1_file: add repository argument to prepare_alt_odb
>   sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
>   sha1_file: allow prepare_alt_odb to handle arbitrary repositories
>   sha1_file: add repository argument to stat_sha1_file
>   sha1_file: add repository argument to open_sha1_file
>   sha1_file: add repository argument to map_sha1_file_1
>   sha1_file: add repository argument to sha1_loose_object_info
>   sha1_file: allow sha1_file_name to handle arbitrary repositories
>   sha1_file: allow map_sha1_file to handle arbitrary repositories
> 
>  builtin/am.c                                    |   2 +-
>  builtin/clone.c                                 |   2 +-
>  builtin/count-objects.c                         |   6 +-
>  builtin/fetch.c                                 |   2 +-
>  builtin/fsck.c                                  |  13 ++-
>  builtin/gc.c                                    |   4 +-
>  builtin/grep.c                                  |   2 +-
>  builtin/index-pack.c                            |   1 +
>  builtin/merge.c                                 |   2 +-
>  builtin/pack-objects.c                          |  21 ++--
>  builtin/pack-redundant.c                        |   6 +-
>  builtin/receive-pack.c                          |   3 +-
>  cache.h                                         |  46 ++------
>  contrib/coccinelle/refactoring/packed_git.cocci |   7 ++
>  environment.c                                   |   5 +-
>  fast-import.c                                   |   6 +-
>  http-backend.c                                  |   6 +-
>  http-push.c                                     |   1 +
>  http-walker.c                                   |   4 +-
>  http.c                                          |   6 +-
>  mru.h                                           |   1 +
>  object-store.h                                  |  75 +++++++++++++
>  object.c                                        |  26 +++++
>  pack-bitmap.c                                   |   4 +-
>  pack-check.c                                    |   1 +
>  pack-revindex.c                                 |   1 +
>  packfile.c                                      |  64 +++++------
>  packfile.h                                      |   2 +-
>  path.c                                          |   2 +-
>  reachable.c                                     |   1 +
>  repository.c                                    |  17 ++-
>  repository.h                                    |   7 +-
>  server-info.c                                   |   6 +-
>  sha1_file.c                                     | 135 +++++++++++++-----------
>  sha1_name.c                                     |  11 +-
>  streaming.c                                     |   5 +-
>  36 files changed, 321 insertions(+), 182 deletions(-)
>  create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci
>  create mode 100644 object-store.h
> 
> -- 
> 2.16.1.73.ga2c3e9663f.dirty
> 

-- 
Brandon Williams

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> This is a real take on the first part of the recent RFC[1].

For the patches remaining in this series, The scope is about right
and the size is more manageable.  With topics already on 'master',
they have some interactions:

 - ot/mru-on-list & gs/retire-mru

   The packed_git MRU has been greatly simplified by using list API
   directly.

 - cc/sha1-file-name

   The function signature of sha1_file_name() has been updated.

I could certainly carry evil merge resolutions going forward (these
changes need to be made to object-store.h that has no mechanical
conflicts), but it may be less distracting for later readers of the
series if we rebase it on top of a more recent master.

Re: [PATCH 00/26] Moving global state into the repository object (part 1)

[Stefan Beller]

On Thu, Feb 15, 2018 at 12:42 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> This is a real take on the first part of the recent RFC[1].
>
> For the patches remaining in this series, The scope is about right
> and the size is more manageable.

ok, thanks.

>  With topics already on 'master',
> they have some interactions:
>
>  - ot/mru-on-list & gs/retire-mru
>
>    The packed_git MRU has been greatly simplified by using list API
>    directly.
>
>  - cc/sha1-file-name
>
>    The function signature of sha1_file_name() has been updated.
>
> I could certainly carry evil merge resolutions going forward (these
> changes need to be made to object-store.h that has no mechanical
> conflicts), but it may be less distracting for later readers of the
> series if we rebase it on top of a more recent master.

I was debating when to reroll this series as I want to make the
change that Duy proposed, moving the 'ignore_env' into the
object store as well.

I'll rebase on top of the latest master or these topicc while at it.

Thanks for the heads up.
Stefan

[PATCHv2 00/16] Moving global state into the repository object (part 1)

[Stefan Beller]

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.
  
Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Stefan Beller (16):
  repository: introduce raw object store field
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: allow link_alt_odb_entries to handle arbitrary object
    stores
  sha1_file: allow prepare_alt_odb to handle arbitrary object stores
  sha1_file: allow sha1_file_name to handle arbitrary object stores
  sha1_file: allow stat_sha1_file to handle arbitrary object stores
  sha1_file: allow open_sha1_file to handle arbitrary object stores
  sha1_file: allow map_sha1_file_1 to handle arbitrary object stores
  sha1_file: allow map_sha1_file to handle arbitrary object stores
  sha1_file: allow sha1_loose_object_info to handle arbitrary object
    stores

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   6 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  13 +++--
 builtin/gc.c             |   4 +-
 builtin/grep.c           |   2 +-
 builtin/index-pack.c     |   1 +
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  19 +++---
 builtin/pack-redundant.c |   6 +-
 builtin/receive-pack.c   |   3 +-
 cache.h                  |  45 ++------------
 environment.c            |   5 +-
 fast-import.c            |   6 +-
 http-backend.c           |   6 +-
 http-push.c              |   1 +
 http-walker.c            |   4 +-
 http.c                   |   6 +-
 object-store.h           |  80 +++++++++++++++++++++++++
 object.c                 |  27 +++++++++
 pack-bitmap.c            |   4 +-
 pack-check.c             |   1 +
 pack-revindex.c          |   1 +
 packfile.c               |  64 ++++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 reachable.c              |   1 +
 repository.c             |  22 +++++--
 repository.h             |   7 ++-
 server-info.c            |   6 +-
 sha1_file.c              | 123 +++++++++++++++++++++------------------
 sha1_name.c              |  11 ++--
 streaming.c              |   5 +-
 34 files changed, 314 insertions(+), 177 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog

[PATCH 01/16] repository: introduce raw object store field

[Stefan Beller]

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct as well as
'ignore_env' which is duplicated from the repository. A later refactoring
(not in this series) will hopefully get rid of that one bit, once all
the construction of repo objects goes through repo_init(). The reason for
duplication now is that the refactoring done in this series allows for
smaller scoped objects in the functions refactored, nameny passing around
the object store instead of the repository object.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 17 +++++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 19 +++++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 46 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e..ec10b062e6 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..5959d990fc
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,17 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+
+	unsigned ignore_env : 1;
+};
+#define RAW_OBJECT_STORE_INIT { NULL, 0 }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b..11d904c033 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..3b8f1b91be 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,9 +49,11 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
+	repo->objects.ignore_env = repo->ignore_env;
+
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -209,12 +218,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 02/16] object-store: move alt_odb_list and alt_odb_tail to object store

[Stefan Beller]

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 cache.h        |  4 ++--
 object-store.h |  7 ++++++-
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 6 files changed, 27 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 7a8a679d4f..908e4f092a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/cache.h b/cache.h
index 9cac7bb518..f373c30e25 100644
--- a/cache.h
+++ b/cache.h
@@ -1576,7 +1576,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -1594,7 +1594,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
diff --git a/object-store.h b/object-store.h
index 5959d990fc..a3f0d6ac15 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,8 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+#include "cache.h"
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -8,9 +10,12 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
+
 	unsigned ignore_env : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d1..216ea836ee 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 831d9e7343..1348dce68f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 03/16] object-store: free alt_odb_list

[Stefan Beller]

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 11d904c033..17b1da6d6b 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 04/16] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch to adjust callers.
  @@ @@
  - packed_git
  + the_repository->objects.packed_git

  @@ @@
  - packed_git_mru
  + the_repository->objects.packed_git_mru

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  6 ++++--
 builtin/fsck.c           |  7 +++++--
 builtin/gc.c             |  4 +++-
 builtin/index-pack.c     |  1 +
 builtin/pack-objects.c   | 19 +++++++++++--------
 builtin/pack-redundant.c |  6 ++++--
 builtin/receive-pack.c   |  1 +
 cache.h                  | 29 -----------------------------
 fast-import.c            |  6 ++++--
 http-backend.c           |  6 ++++--
 http-push.c              |  1 +
 http-walker.c            |  1 +
 http.c                   |  1 +
 object-store.h           | 36 +++++++++++++++++++++++++++++++++++-
 object.c                 |  7 +++++++
 pack-bitmap.c            |  4 +++-
 pack-check.c             |  1 +
 pack-revindex.c          |  1 +
 packfile.c               | 39 ++++++++++++++++++++-------------------
 reachable.c              |  1 +
 repository.c             |  3 +++
 server-info.c            |  6 ++++--
 sha1_name.c              |  5 +++--
 23 files changed, 118 insertions(+), 73 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 908e4f092a..2e99e02128 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0..96ff790867 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 5ebd370c56..e60cc3ed98 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 83dcbc9773..d73528b583 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1026,7 +1028,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			return want;
 	}
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1046,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2675,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2738,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2748,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2784,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3155,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3168,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index f373c30e25..018a297e10 100644
--- a/cache.h
+++ b/cache.h
@@ -1636,35 +1636,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 0913f8ab86..1ca662390f 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af82..8bb5d991bb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 31755023a4..a4a9e583c7 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/object-store.h b/object-store.h
index a3f0d6ac15..024ccc91e9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -2,6 +2,7 @@
 #define OBJECT_STORE_H
 
 #include "cache.h"
+#include "list.h"
 
 struct raw_object_store {
 	/*
@@ -10,13 +11,46 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
 	unsigned ignore_env : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
+
+#define MRU_LIST_INIT {NULL, NULL}
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+extern struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+} *packed_git;
+
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 17b1da6d6b..c76b62572a 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	while (!list_empty(&o->packed_git_mru))
+		list_del(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index ff5f62c033..bb521cf7fb 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index 216ea836ee..d41e4c83d0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +383,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/repository.c b/repository.c
index 3b8f1b91be..61b23558d0 100644
--- a/repository.c
+++ b/repository.c
@@ -144,6 +144,9 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	repo->objects.ignore_env = 1;
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

>   @@ @@
>   - packed_git_mru
>   + the_repository->objects.packed_git_mru

Regarding this...

> diff --git a/object-store.h b/object-store.h
> index a3f0d6ac15..024ccc91e9 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -2,6 +2,7 @@
>  #define OBJECT_STORE_H
>  
>  #include "cache.h"
> +#include "list.h"
>  
>  struct raw_object_store {
> ... 
> +	struct packed_git *packed_git;
> +	/*
> +	 * A most-recently-used ordered version of the packed_git list, which can
> +	 * be iterated instead of packed_git (and marked via mru_mark).
> +	 */
> +	struct list_head packed_git_mru;
> +
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  
>  	unsigned ignore_env : 1;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
> +
> +#define MRU_LIST_INIT {NULL, NULL}
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
> ...
> diff --git a/packfile.c b/packfile.c
> index 216ea836ee..d41e4c83d0 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -7,6 +7,7 @@
> -...
> -LIST_HEAD(packed_git_mru);

Given that the definition of LIST_HEAD() is

    /* Define a variable with the head and tail of the list. */
    #define LIST_HEAD(name) \
            struct list_head name = { &(name), &(name) }

doesn't the updated definition of RAW_OBJECT_STORE_INIT look fishy?

[PATCH 05/16] object-store: close all packs upon clearing the object store

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 5bdd2d7578..4762a702e3 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593..13cfaa6488 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2e..4d72efca78 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index c76b62572a..34daaf37b3 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	while (!list_empty(&o->packed_git_mru))
 		list_del(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index d41e4c83d0..511a2b0cdf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d6..6a2c57045c 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 06/16] pack: move prepare_packed_git_run_once to object store

[Stefan Beller]

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 024ccc91e9..794cb0af59 100644
--- a/object-store.h
+++ b/object-store.h
@@ -22,10 +22,16 @@ struct raw_object_store {
 	struct alternate_object_database **alt_odb_tail;
 
 	unsigned ignore_env : 1;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
 #define MRU_LIST_INIT {NULL, NULL}
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 511a2b0cdf..a8a2e7fe43 100644
--- a/packfile.c
+++ b/packfile.c
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 07/16] pack: move approximate object count to object store

[Stefan Beller]

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 794cb0af59..80a2ca6f75 100644
--- a/object-store.h
+++ b/object-store.h
@@ -23,6 +23,14 @@ struct raw_object_store {
 
 	unsigned ignore_env : 1;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -31,7 +39,7 @@ struct raw_object_store {
 };
 
 #define MRU_LIST_INIT {NULL, NULL}
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index a8a2e7fe43..693bafbc98 100644
--- a/packfile.c
+++ b/packfile.c
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 08/16] sha1_file: add raw_object_store argument to alt_odb_usable

[Stefan Beller]

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1348dce68f..a8e23bd2f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 09/16] sha1_file: allow link_alt_odb_entries to handle arbitrary object stores

[Stefan Beller]

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary object stores, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h     |  4 ++++
 sha1_file.c | 42 +++++++++++++++++++++++++-----------------
 2 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/cache.h b/cache.h
index 018a297e10..12bb42f951 100644
--- a/cache.h
+++ b/cache.h
@@ -1593,6 +1593,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 extern void prepare_alt_odb(void);
diff --git a/sha1_file.c b/sha1_file.c
index a8e23bd2f8..c4255a2da6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -389,9 +389,11 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+static void read_info_alternates(struct raw_object_store *o,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct raw_object_store *o, const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -416,7 +418,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(o, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -424,12 +426,12 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*o->alt_odb_tail = ent;
+	o->alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(o, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -464,8 +466,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+static void link_alt_odb_entries(struct raw_object_store *o, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -479,7 +481,7 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, o->objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -488,13 +490,16 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(o, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates(struct raw_object_store *o,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -506,7 +511,7 @@ static void read_info_alternates(const char * relative_base, int depth)
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(o, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -560,7 +565,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(&the_repository->objects, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -573,7 +579,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(&the_repository->objects, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -676,9 +683,10 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(&the_repository->objects, alt,
+			     PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(&the_repository->objects, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 10/16] sha1_file: allow prepare_alt_odb to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/fsck.c |  2 +-
 cache.h        |  1 -
 object-store.h |  2 ++
 packfile.c     |  2 +-
 sha1_file.c    | 23 +++++++++++------------
 sha1_name.c    |  3 ++-
 6 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 2e99e02128..c0d967d6d7 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(&the_repository->objects);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/cache.h b/cache.h
index 12bb42f951..d0c1a533c5 100644
--- a/cache.h
+++ b/cache.h
@@ -1599,7 +1599,6 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 extern int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/object-store.h b/object-store.h
index 80a2ca6f75..280e8e16ad 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,4 +67,6 @@ extern struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 } *packed_git;
 
+void prepare_alt_odb(struct raw_object_store *o);
+
 #endif /* OBJECT_STORE_H */
diff --git a/packfile.c b/packfile.c
index 693bafbc98..83b10e55c0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -890,7 +890,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index c4255a2da6..04a81e29f2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -577,7 +578,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 
 	link_alt_odb_entries(&the_repository->objects, reference,
 			     '\n', NULL, 0);
@@ -663,7 +664,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -672,21 +673,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb(struct raw_object_store *o)
 {
 	const char *alt;
 
-	if (the_repository->objects.alt_odb_tail)
+	if (o->alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(&the_repository->objects, alt,
-			     PATH_SEP, NULL, 0);
+	o->alt_odb_tail = &o->alt_odb_list;
+	link_alt_odb_entries(o, alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(&the_repository->objects, get_object_directory(), 0);
+	read_info_alternates(o, o->objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
@@ -726,7 +725,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -885,7 +884,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -916,7 +915,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..f1e6c910ba 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 11/16] sha1_file: allow sha1_file_name to handle arbitrary object stores

[Stefan Beller]

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  6 ++++++
 sha1_file.c    | 12 ++++++------
 5 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/cache.h b/cache.h
index d0c1a533c5..2495c7081d 100644
--- a/cache.h
+++ b/cache.h
@@ -961,12 +961,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 8bb5d991bb..59bc0417d2 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(&the_repository->objects, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index a4a9e583c7..058436d9d3 100644
--- a/http.c
+++ b/http.c
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(&the_repository->objects, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(&the_repository->objects, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 280e8e16ad..5f6beb9d91 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,6 +67,12 @@ extern struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 } *packed_git;
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1);
+
 void prepare_alt_odb(struct raw_object_store *o);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 04a81e29f2..d64cdbb494 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, get_object_directory());
+	strbuf_addstr(buf, o->objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
@@ -717,7 +717,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -878,7 +878,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -907,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1592,7 +1592,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(&the_repository->objects, &filename, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 12/16] sha1_file: allow stat_sha1_file to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index d64cdbb494..bd8b0331f0 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -871,22 +871,23 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+static int stat_sha1_file(struct raw_object_store *o, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&the_repository->objects, &buf, sha1);
+	sha1_file_name(o, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(&the_repository->objects);
+	prepare_alt_odb(o);
+
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -1178,7 +1179,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(&the_repository->objects, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1392,7 +1393,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(&the_repository->objects, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 13/16] sha1_file: allow open_sha1_file to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index bd8b0331f0..3356f70dd2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -900,7 +900,8 @@ static int stat_sha1_file(struct raw_object_store *o, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+static int open_sha1_file(struct raw_object_store *o,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -908,7 +909,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&the_repository->objects, &buf, sha1);
+	sha1_file_name(o, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -916,8 +917,8 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(&the_repository->objects);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(o);
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
@@ -943,7 +944,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(&the_repository->objects, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 14/16] sha1_file: allow map_sha1_file_1 to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 3356f70dd2..aeb925a895 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -934,9 +934,8 @@ static int open_sha1_file(struct raw_object_store *o,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+static void *map_sha1_file_1(struct raw_object_store *o, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -944,7 +943,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(&the_repository->objects, sha1, &path);
+		fd = open_sha1_file(o, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
@@ -965,7 +964,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(&the_repository->objects, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2195,7 +2194,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(&the_repository->objects, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 15/16] sha1_file: allow map_sha1_file to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 2 ++
 sha1_file.c    | 7 ++++---
 streaming.c    | 5 ++++-
 4 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/cache.h b/cache.h
index 2495c7081d..df29ee72e9 100644
--- a/cache.h
+++ b/cache.h
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 5f6beb9d91..30b4cfdbf5 100644
--- a/object-store.h
+++ b/object-store.h
@@ -73,6 +73,8 @@ extern struct packed_git {
  */
 void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1);
 
+void *map_sha1_file(struct raw_object_store *o, const unsigned char *sha1, unsigned long *size);
+
 void prepare_alt_odb(struct raw_object_store *o);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index aeb925a895..4cbff471a2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -962,9 +962,10 @@ static void *map_sha1_file_1(struct raw_object_store *o, const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct raw_object_store *o,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(&the_repository->objects, NULL, sha1, size);
+	return map_sha1_file_1(o, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -1186,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(&the_repository->objects, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..6888448b12 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(&the_repository->objects,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 16/16] sha1_file: allow sha1_loose_object_info to handle arbitrary object stores

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 4cbff471a2..31be57249f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1154,7 +1154,8 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
+static int sha1_loose_object_info(struct raw_object_store *o,
+				  const unsigned char *sha1,
 				  struct object_info *oi,
 				  int flags)
 {
@@ -1180,14 +1181,14 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(&the_repository->objects, sha1, &st, &path) < 0)
+		if (stat_sha1_file(o, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(&the_repository->objects, sha1, &mapsize);
+	map = map_sha1_file(o, sha1, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(&the_repository->objects, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> v2:
> * duplicated the 'ignore_env' bit into the object store as well
> * the #define trick no longer works as we do not have a "the_objectstore" global,
>   which means there is just one patch per function that is converted.
>   As this follows the same structure of the previous series, I am still confident
>   there is no hidden dependencies to globals outside the object store in these
>   converted functions.
> * rebased on top of current master, resolving the merge conflicts.
>   I think I used the list.h APIs right, but please double check.

For what it's worth, I think I prefer v1.  I put some comments on why
on patch 0 of v1 and would be interested in your thoughts on them
(e.g. as a reply to that).  I also think that even if we want to
switch to a style that passes around object_store separately from
repository, it is easier to do the migration in two steps: first get
rid of hidden dependencies on the_repository, then do the (simpler)
automatic migration from

 f(the_repository)

to

 f(the_repository->object_store)

*afterwards*.

Thoughts?

Thanks,
Jonathan

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Stefan Beller]

On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Stefan Beller wrote:
>
>> v2:
>> * duplicated the 'ignore_env' bit into the object store as well
>> * the #define trick no longer works as we do not have a "the_objectstore" global,
>>   which means there is just one patch per function that is converted.
>>   As this follows the same structure of the previous series, I am still confident
>>   there is no hidden dependencies to globals outside the object store in these
>>   converted functions.
>> * rebased on top of current master, resolving the merge conflicts.
>>   I think I used the list.h APIs right, but please double check.
>
> For what it's worth, I think I prefer v1.  I put some comments on why
> on patch 0 of v1 and would be interested in your thoughts on them
> (e.g. as a reply to that).  I also think that even if we want to
> switch to a style that passes around object_store separately from
> repository, it is easier to do the migration in two steps: first get
> rid of hidden dependencies on the_repository, then do the (simpler)
> automatic migration from
>
>  f(the_repository)
>
> to
>
>  f(the_repository->object_store)
>
> *afterwards*.
>
> Thoughts?

I would prefer to not spend more time on these conversions.
If Duy and you would come to a conclusion to either pick this
or the previous version I would be happy.

I do not see the benefit in splitting up the series even further and
do this multistep f(repo) -> f(object store), as the cost in potential
merge conflicts is too high. Note that brian just sent another object
id conversion series, also touching sha1_file.c, which I am sure will
produce merge conflicts for Junio.

For the next part 2 and onwards I'd be happy to take either this
strategy or Duys strategy as requested.

Stefan

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Hi,
> >
> > Stefan Beller wrote:
> >
> >> v2:
> >> * duplicated the 'ignore_env' bit into the object store as well
> >> * the #define trick no longer works as we do not have a "the_objectstore" global,
> >>   which means there is just one patch per function that is converted.
> >>   As this follows the same structure of the previous series, I am still confident
> >>   there is no hidden dependencies to globals outside the object store in these
> >>   converted functions.
> >> * rebased on top of current master, resolving the merge conflicts.
> >>   I think I used the list.h APIs right, but please double check.
> >
> > For what it's worth, I think I prefer v1.  I put some comments on why
> > on patch 0 of v1 and would be interested in your thoughts on them
> > (e.g. as a reply to that).  I also think that even if we want to
> > switch to a style that passes around object_store separately from
> > repository, it is easier to do the migration in two steps: first get
> > rid of hidden dependencies on the_repository, then do the (simpler)
> > automatic migration from
> >
> >  f(the_repository)
> >
> > to
> >
> >  f(the_repository->object_store)
> >
> > *afterwards*.
> >
> > Thoughts?
> 
> I would prefer to not spend more time on these conversions.
> If Duy and you would come to a conclusion to either pick this
> or the previous version I would be happy.
> 
> I do not see the benefit in splitting up the series even further and
> do this multistep f(repo) -> f(object store), as the cost in potential
> merge conflicts is too high. Note that brian just sent another object
> id conversion series, also touching sha1_file.c, which I am sure will
> produce merge conflicts for Junio.
> 
> For the next part 2 and onwards I'd be happy to take either this
> strategy or Duys strategy as requested.

I think Jonathan is trying to point out that converting to f(repo) maybe
easier than converting to f(repo->object_store) upfront which would make
it easier to write the patches (which most of them are already written)
and to review because you can use the #define trick to make some sort of
guarantees.

After we have successfully completed the migration to f(repo), then we
can revisit the subsystems which want to have a clearer abstraction
layer and make the jump to f(repo->object_store).

-- 
Brandon Williams

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 20, 2018 at 11:00 AM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> > Hi,
>> >
>> > Stefan Beller wrote:
>> >
>> >> v2:
>> >> * duplicated the 'ignore_env' bit into the object store as well
>> >> * the #define trick no longer works as we do not have a "the_objectstore" global,
>> >>   which means there is just one patch per function that is converted.
>> >>   As this follows the same structure of the previous series, I am still confident
>> >>   there is no hidden dependencies to globals outside the object store in these
>> >>   converted functions.
>> >> * rebased on top of current master, resolving the merge conflicts.
>> >>   I think I used the list.h APIs right, but please double check.
>> >
>> > For what it's worth, I think I prefer v1.  I put some comments on why
>> > on patch 0 of v1 and would be interested in your thoughts on them
>> > (e.g. as a reply to that).  I also think that even if we want to
>> > switch to a style that passes around object_store separately from
>> > repository, it is easier to do the migration in two steps: first get
>> > rid of hidden dependencies on the_repository, then do the (simpler)
>> > automatic migration from
>> >
>> >  f(the_repository)
>> >
>> > to
>> >
>> >  f(the_repository->object_store)
>> >
>> > *afterwards*.
>> >
>> > Thoughts?
>>
>> I would prefer to not spend more time on these conversions.
>> If Duy and you would come to a conclusion to either pick this
>> or the previous version I would be happy.
>>
>> I do not see the benefit in splitting up the series even further and
>> do this multistep f(repo) -> f(object store), as the cost in potential
>> merge conflicts is too high. Note that brian just sent another object
>> id conversion series, also touching sha1_file.c, which I am sure will
>> produce merge conflicts for Junio.
>>
>> For the next part 2 and onwards I'd be happy to take either this
>> strategy or Duys strategy as requested.
>
> I think Jonathan is trying to point out that converting to f(repo) maybe
> easier than converting to f(repo->object_store) upfront

I agree.

> which would make
> it easier to write the patches (which most of them are already written)

true, but for this series we also have the conversion to f(object_store)
written already.

> and to review because you can use the #define trick to make some sort of
> guarantees.

That is true, so it would be a tradeoff between reviewers and maintainers time?

> After we have successfully completed the migration to f(repo), then we
> can revisit the subsystems which want to have a clearer abstraction
> layer and make the jump to f(repo->object_store).

I would think we can take this series as-is and then move on with making
f(repo) abstractions, eventually moving to f(specialized-subsystem) as those
patches are not written yet (neither direct conversions, nor the repo trick;
the patches I already have need adaption which takes enough time on its own.)


>
> --
> Brandon Williams

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> For what it's worth, I think I prefer v1.  I put some comments on why
> on patch 0 of v1 and would be interested in your thoughts on them
> (e.g. as a reply to that).  I also think that even if we want to
> switch to a style that passes around object_store separately from
> repository, it is easier to do the migration in two steps: first get
> rid of hidden dependencies on the_repository, then do the (simpler)
> automatic migration from
>
>  f(the_repository)
>
> to
>
>  f(the_repository->object_store)
>
> *afterwards*.
>
> Thoughts?

Are we envisioning the future in which one repository has more than
one object-store (I am counting an object store and its alternates
that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a
single logical "object store")?  If not, doing f(the_repository)
migration, stopping there without f(the_repository->object_store)
may perfectly be adequate, I would think.

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 20, 2018 at 11:03 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> For what it's worth, I think I prefer v1.  I put some comments on why
>> on patch 0 of v1 and would be interested in your thoughts on them
>> (e.g. as a reply to that).  I also think that even if we want to
>> switch to a style that passes around object_store separately from
>> repository, it is easier to do the migration in two steps: first get
>> rid of hidden dependencies on the_repository, then do the (simpler)
>> automatic migration from
>>
>>  f(the_repository)
>>
>> to
>>
>>  f(the_repository->object_store)
>>
>> *afterwards*.
>>
>> Thoughts?
>
> Are we envisioning the future in which one repository has more than
> one object-store (I am counting an object store and its alternates
> that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a
> single logical "object store")?  If not, doing f(the_repository)
> migration, stopping there without f(the_repository->object_store)
> may perfectly be adequate, I would think.
>

I do not envision that, maybe Christian Couder does?
The partial clone world would be happy with just one object store
per repo, I would think.

The step to take an object store would just add expressiveness
to the code, which may help in understanding what part of the code is
related to what other part of the code, so it may be a readability gain
on its own?

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> The step to take an object store would just add expressiveness
> to the code, which may help in understanding what part of the code is
> related to what other part of the code, so it may be a readability gain
> on its own?

It certainly would allow you to have a standalone object store that
is not associated with *any* repository, but if we are not using
such a layout, I doubt it would be a readability gain to add excess
and unexercised expressiveness to the code.

Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 20, 2018 at 11:55 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> The step to take an object store would just add expressiveness
>> to the code, which may help in understanding what part of the code is
>> related to what other part of the code, so it may be a readability gain
>> on its own?
>
> It certainly would allow you to have a standalone object store that
> is not associated with *any* repository, but if we are not using
> such a layout, I doubt it would be a readability gain to add excess
> and unexercised expressiveness to the code.

So you favor v1?
Duy seems to be ok with v1 too if there is consensus that it is best
(or rather "if it makes Stefan's life hell, it's not worth doing.")[1].

I will try to resend that v1 shortly[2], as the only actual concern about the
code was where one struct was defined[3]. All other discussion was
meta-level, which direction to go.

Thanks for the clarification!
Stefan

[1] https://public-inbox.org/git/CACsJy8CPKESE8atc_eWdNVknQYp9T6ebwKwCdzLHyaFKH2BnZA@mail.gmail.com/
[2] https://public-inbox.org/git/20180213012241.187007-1-sbeller@google.com/
[3] https://public-inbox.org/git/20180213185120.GA108526@google.com/

[PATCHv3 00/27] Moving global state into the repository object (part 1)

[Stefan Beller]

v3:
* reverted back to use the repository for most of the functions
  (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as
  I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to
  object-store.h, such that inclusion of cache.h (in object-store.h)
  is not required any more.

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.

Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (2):
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (25):
  repository: introduce raw object store field
  object-store: migrate alternates struct and functions from cache.h
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to map_sha1_file
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   6 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  13 ++--
 builtin/gc.c             |   4 +-
 builtin/grep.c           |   2 +-
 builtin/index-pack.c     |   1 +
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  20 +++---
 builtin/pack-redundant.c |   6 +-
 builtin/receive-pack.c   |   3 +-
 cache.h                  |  87 -------------------------
 environment.c            |   5 +-
 fast-import.c            |   6 +-
 http-backend.c           |   6 +-
 http-push.c              |   1 +
 http-walker.c            |   4 +-
 http.c                   |   6 +-
 object-store.h           | 134 +++++++++++++++++++++++++++++++++++++++
 object.c                 |  27 ++++++++
 pack-bitmap.c            |   4 +-
 pack-check.c             |   1 +
 pack-revindex.c          |   1 +
 packfile.c               |  64 +++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 reachable.c              |   1 +
 repository.c             |  19 ++++--
 repository.h             |   7 +-
 server-info.c            |   6 +-
 sha1_file.c              | 134 +++++++++++++++++++++------------------
 sha1_name.c              |  11 ++--
 streaming.c              |   5 +-
 34 files changed, 365 insertions(+), 231 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog

[PATCH 01/27] repository: introduce raw object store field

[Stefan Beller]

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 17 +++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 42 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e..ec10b062e6 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..cf35760ceb
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b..11d904c033 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..2255ff657e 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 01/27] repository: introduce raw object store field

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> The raw object store field will contain any objects needed for
> access to objects in a given repository.
>
> This patch introduces the raw object store and populates it with the
> `objectdir`, which used to be part of the repository struct.
>
> As the struct gains members, we'll also populate the function to clear
> the memory for these members.
>
> In a later we'll introduce a struct object_parser, that will complement
> the object parsing in a repository struct: The raw object parser is the
> layer that will provide access to raw object content, while the higher
> level object parser code will parse raw objects and keeps track of
> parenthood and other object relationships using 'struct object'.
> For now only add the lower level to the repository struct.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---

Heh, I suppose that sign-off makes me not a great candidate for
reviewing this.  It's been long enough since I looked at the patch
that I feel okay trying anyway.

[...]
> --- /dev/null
> +++ b/object-store.h
> @@ -0,0 +1,15 @@
> +#ifndef OBJECT_STORE_H
> +#define OBJECT_STORE_H
> +
> +struct raw_object_store {
> +	/*
> +	 * Path to the repository's object store.
> +	 * Cannot be NULL after initialization.
> +	 */
> +	char *objectdir;
> +};
> +#define RAW_OBJECT_STORE_INIT { NULL }

Who owns and is responsible for freeing objectdir?

> +
> +void raw_object_store_clear(struct raw_object_store *o);

Oh, that answers that.

It could be worth a note in the doc comment anyway, but I don't mind
not having one.

[...]
> +
> +void raw_object_store_clear(struct raw_object_store *o)
> +{
> +	free(o->objectdir);
> +}

Should this use FREE_AND_NULL?

[...]
> --- a/repository.c
> +++ b/repository.c
[...]
> @@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
>  						    !repo->ignore_env);
>  	free(repo->commondir);
>  	repo->commondir = strbuf_detach(&sb, NULL);
> -	free(repo->objectdir);
> -	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
> +	free(repo->objects.objectdir);

Should this call raw_object_store_clear instead of calling free
directly?

> +	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
>  					    "objects", !repo->ignore_env);

Long line.  One way to break it would be

	repo->objects.objectdir =
		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
				  "objects", !repo->ignore_env);

>  	free(repo->graft_file);
>  	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
> @@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
>  {
>  	FREE_AND_NULL(repo->gitdir);
>  	FREE_AND_NULL(repo->commondir);
> -	FREE_AND_NULL(repo->objectdir);
>  	FREE_AND_NULL(repo->graft_file);
>  	FREE_AND_NULL(repo->index_file);
>  	FREE_AND_NULL(repo->worktree);
>  	FREE_AND_NULL(repo->submodule_prefix);
>  
> +	raw_object_store_clear(&repo->objects);
> +	memset(&repo->objects, 0, sizeof(repo->objects));

If raw_object_store_clear uses FREE_AND_NULL, this memset won't be necessary.

[...]
> --- a/repository.h
> +++ b/repository.h
> @@ -1,6 +1,8 @@
>  #ifndef REPOSITORY_H
>  #define REPOSITORY_H
>  
> +#include "object-store.h"
> +
>  struct config_set;
>  struct index_state;
>  struct submodule_cache;
> @@ -21,10 +23,9 @@ struct repository {
>  	char *commondir;
>  
>  	/*
> -	 * Path to the repository's object store.
> -	 * Cannot be NULL after initialization.
> +	 * Holds any information related to the object store.
>  	 */

This comment doesn't make it clear to me what the field represents.
Can it be replaced with some of the description from the commit
message?

> -	char *objectdir;
> +	struct raw_object_store objects;
>  

Thanks,
Jonathan

[PATCH 02/27] object-store: migrate alternates struct and functions from cache.h

[Stefan Beller]

Migrate the struct alternate_object_database and all its related
functions to the object store as these functions are easier found in
that header. The migration is just a verbatim copy, no need to
include the object store header at any C file, because cache.h includes
repository.h which in turn includes the object-store.h

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h        | 51 --------------------------------------------------
 object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 51 deletions(-)

diff --git a/cache.h b/cache.h
index 9cac7bb518..964873faf2 100644
--- a/cache.h
+++ b/cache.h
@@ -1576,57 +1576,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
-	struct alternate_object_database *next;
-
-	/* see alt_scratch_buf() */
-	struct strbuf scratch;
-	size_t base_len;
-
-	/*
-	 * Used to store the results of readdir(3) calls when searching
-	 * for unique abbreviated hashes.  This cache is never
-	 * invalidated, thus it's racy and not necessarily accurate.
-	 * That's fine for its purpose; don't use it for tasks requiring
-	 * greater accuracy!
-	 */
-	char loose_objects_subdir_seen[256];
-	struct oid_array loose_objects_cache;
-
-	char path[FLEX_ARRAY];
-} *alt_odb_list;
-extern void prepare_alt_odb(void);
-extern char *compute_alternate_path(const char *path, struct strbuf *err);
-typedef int alt_odb_fn(struct alternate_object_database *, void *);
-extern int foreach_alt_odb(alt_odb_fn, void*);
-
-/*
- * Allocate a "struct alternate_object_database" but do _not_ actually
- * add it to the list of alternates.
- */
-struct alternate_object_database *alloc_alt_odb(const char *dir);
-
-/*
- * Add the directory to the on-disk alternates file; the new entry will also
- * take effect in the current process.
- */
-extern void add_to_alternates_file(const char *dir);
-
-/*
- * Add the directory to the in-memory list of alternates (along with any
- * recursive alternates it points to), but do not modify the on-disk alternates
- * file.
- */
-extern void add_to_alternates_memory(const char *dir);
-
-/*
- * Returns a scratch strbuf pre-filled with the alternate object directory,
- * including a trailing slash, which can be used to access paths in the
- * alternate. Always use this over direct access to alt->scratch, as it
- * cleans up any previous use of the scratch buffer.
- */
-extern struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
-
 struct pack_window {
 	struct pack_window *next;
 	unsigned char *base;
diff --git a/object-store.h b/object-store.h
index cf35760ceb..5678aa1136 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,57 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+extern struct alternate_object_database {
+	struct alternate_object_database *next;
+
+	/* see alt_scratch_buf() */
+	struct strbuf scratch;
+	size_t base_len;
+
+	/*
+	 * Used to store the results of readdir(3) calls when searching
+	 * for unique abbreviated hashes.  This cache is never
+	 * invalidated, thus it's racy and not necessarily accurate.
+	 * That's fine for its purpose; don't use it for tasks requiring
+	 * greater accuracy!
+	 */
+	char loose_objects_subdir_seen[256];
+	struct oid_array loose_objects_cache;
+
+	char path[FLEX_ARRAY];
+} *alt_odb_list;
+void prepare_alt_odb(void);
+char *compute_alternate_path(const char *path, struct strbuf *err);
+typedef int alt_odb_fn(struct alternate_object_database *, void *);
+int foreach_alt_odb(alt_odb_fn, void*);
+
+/*
+ * Allocate a "struct alternate_object_database" but do _not_ actually
+ * add it to the list of alternates.
+ */
+struct alternate_object_database *alloc_alt_odb(const char *dir);
+
+/*
+ * Add the directory to the on-disk alternates file; the new entry will also
+ * take effect in the current process.
+ */
+void add_to_alternates_file(const char *dir);
+
+/*
+ * Add the directory to the in-memory list of alternates (along with any
+ * recursive alternates it points to), but do not modify the on-disk alternates
+ * file.
+ */
+void add_to_alternates_memory(const char *dir);
+
+/*
+ * Returns a scratch strbuf pre-filled with the alternate object directory,
+ * including a trailing slash, which can be used to access paths in the
+ * alternate. Always use this over direct access to alt->scratch, as it
+ * cleans up any previous use of the scratch buffer.
+ */
+struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> Migrate the struct alternate_object_database and all its related
> functions to the object store as these functions are easier found in
> that header. The migration is just a verbatim copy, no need to
> include the object store header at any C file, because cache.h includes
> repository.h which in turn includes the object-store.h

Nice.  Always love seeing cache.h get smaller


-- 
Brandon Williams

[PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store

[Stefan Beller]

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 object-store.h |  9 ++++++---
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 5 files changed, 25 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 7a8a679d4f..908e4f092a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/object-store.h b/object-store.h
index 5678aa1136..e78eea1dde 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,7 +1,7 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 void prepare_alt_odb(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
@@ -58,8 +58,11 @@ struct raw_object_store {
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d1..216ea836ee 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 831d9e7343..1348dce68f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> In a process with multiple repositories open, alternates should be
> associated to a single repository and not shared globally. Move
> alt_odb_list and alt_odb_tail into the_repository and adjust callers
> to reflect this.
> 
> Now that the alternative object data base is per repository, we're
> leaking its memory upon freeing a repository. The next patch plugs
> this hole.
> 
> No functional change intended.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Looks good aside from the one spacing issue (though it's not that big of
a deal).

> ---
>  builtin/fsck.c |  4 +++-
>  object-store.h |  9 ++++++---
>  packfile.c     |  3 ++-
>  sha1_file.c    | 25 ++++++++++++-------------
>  sha1_name.c    |  3 ++-
>  5 files changed, 25 insertions(+), 19 deletions(-)
> 
> diff --git a/builtin/fsck.c b/builtin/fsck.c
> index 7a8a679d4f..908e4f092a 100644
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -1,5 +1,6 @@
>  #include "builtin.h"
>  #include "cache.h"
> +#include "repository.h"
>  #include "config.h"
>  #include "commit.h"
>  #include "tree.h"
> @@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  		fsck_object_dir(get_object_directory());
>  
>  		prepare_alt_odb();
> -		for (alt = alt_odb_list; alt; alt = alt->next)
> +		for (alt = the_repository->objects.alt_odb_list;
> +				alt; alt = alt->next)

Indentation on this line looks odd.


-- 
Brandon Williams

[PATCH 04/27] object-store: free alt_odb_list

[Stefan Beller]

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 11d904c033..17b1da6d6b 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 04/27] object-store: free alt_odb_list

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> Free the memory and reset alt_odb_{list, tail} to NULL.

Good to see memory leaks being avoided (well they will be on other
repository objects)

> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  object.c | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/object.c b/object.c
> index 11d904c033..17b1da6d6b 100644
> --- a/object.c
> +++ b/object.c
> @@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
>  	}
>  }
>  
> +static void free_alt_odb(struct alternate_object_database *alt)
> +{
> +	strbuf_release(&alt->scratch);
> +	oid_array_clear(&alt->loose_objects_cache);
> +}
> +
> +static void free_alt_odbs(struct raw_object_store *o)
> +{
> +	while (o->alt_odb_list) {
> +		free_alt_odb(o->alt_odb_list);
> +		o->alt_odb_list = o->alt_odb_list->next;
> +	}
> +}
> +
>  void raw_object_store_clear(struct raw_object_store *o)
>  {
>  	free(o->objectdir);
> +
> +	free_alt_odbs(o);
> +	o->alt_odb_tail = NULL;
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

[PATCH 05/27] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch
    contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
    This semantic patch is placed in a sub directory of the coccinelle
    contrib dir, as this semantic patch is not expected to be of general
    usefulness; it is only useful during developing this series and
    merging it with other topics in flight. At a later date, just
    delete that semantic patch.

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  6 ++++--
 builtin/fsck.c           |  7 +++++--
 builtin/gc.c             |  4 +++-
 builtin/index-pack.c     |  1 +
 builtin/pack-objects.c   | 20 +++++++++++---------
 builtin/pack-redundant.c |  6 ++++--
 builtin/receive-pack.c   |  1 +
 cache.h                  | 29 ----------------------------
 fast-import.c            |  6 ++++--
 http-backend.c           |  6 ++++--
 http-push.c              |  1 +
 http-walker.c            |  1 +
 http.c                   |  1 +
 object-store.h           | 41 +++++++++++++++++++++++++++++++++++++++-
 object.c                 |  7 +++++++
 pack-bitmap.c            |  4 +++-
 pack-check.c             |  1 +
 pack-revindex.c          |  1 +
 packfile.c               | 39 +++++++++++++++++++-------------------
 reachable.c              |  1 +
 repository.c             |  2 ++
 server-info.c            |  6 ++++--
 sha1_name.c              |  5 +++--
 23 files changed, 122 insertions(+), 74 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 908e4f092a..2e99e02128 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0..96ff790867 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 5ebd370c56..e60cc3ed98 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 83dcbc9773..7f376c2aef 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1025,8 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index 964873faf2..3fd9dc52cd 100644
--- a/cache.h
+++ b/cache.h
@@ -1585,35 +1585,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 0913f8ab86..1ca662390f 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af82..8bb5d991bb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 31755023a4..a4a9e583c7 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/object-store.h b/object-store.h
index e78eea1dde..1de9e07102 100644
--- a/object-store.h
+++ b/object-store.h
@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
  */
 struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
 
+struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -59,10 +83,25 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+
+/*
+ * The mru list_head is supposed to be initialized using
+ * the LIST_HEAD macro, assigning prev/next to itself.
+ * However this doesn't work in this case as some compilers dislike
+ * that macro on member variables. Use NULL instead as that is defined
+ * and accepted, deferring the real init to prepare_packed_git_mru(). */
+#define __MRU_INIT { NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index 17b1da6d6b..c76b62572a 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	while (!list_empty(&o->packed_git_mru))
+		list_del(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index ff5f62c033..bb521cf7fb 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index 216ea836ee..d41e4c83d0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +383,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/repository.c b/repository.c
index 2255ff657e..9798f4ffca 100644
--- a/repository.c
+++ b/repository.c
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> +
> +/*
> + * The mru list_head is supposed to be initialized using
> + * the LIST_HEAD macro, assigning prev/next to itself.
> + * However this doesn't work in this case as some compilers dislike
> + * that macro on member variables. Use NULL instead as that is defined
> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
> +#define __MRU_INIT { NULL, NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }

I do not think it has to be this way to abuse two NULLs, if you
faithfully mimicked how LIST_HEAD() macro is constructed.  The
reason why it does not try to introduce

	struct list_head x = LIST_HEAD_INIT;

and instead, uses

	LIST_HEAD(x);

is because of the need for self referral.  If we learn from it, we
can do the same, i.e. instead of doing

	struct raw_object_store x = RAW_OBJECT_STORE_INIT;

we can do

	RAW_OBJECT_STORE(x);

that expands to

	struct raw_object_store x = {
		...
		{ &x.packed_git_mru, &x.packed_git_mru },
		...
	};

no?  Then we do not need such a lengthy comment that reads only like
an excuse ;-)

Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

On Wed, Feb 21, 2018 at 1:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> +
>> +/*
>> + * The mru list_head is supposed to be initialized using
>> + * the LIST_HEAD macro, assigning prev/next to itself.
>> + * However this doesn't work in this case as some compilers dislike
>> + * that macro on member variables. Use NULL instead as that is defined
>> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
>> +#define __MRU_INIT { NULL, NULL }
>> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
>
> I do not think it has to be this way to abuse two NULLs, if you
> faithfully mimicked how LIST_HEAD() macro is constructed.  The
> reason why it does not try to introduce
>
>         struct list_head x = LIST_HEAD_INIT;
>
> and instead, uses
>
>         LIST_HEAD(x);
>
> is because of the need for self referral.  If we learn from it, we
> can do the same, i.e. instead of doing
>
>         struct raw_object_store x = RAW_OBJECT_STORE_INIT;
>
> we can do
>
>         RAW_OBJECT_STORE(x);
>
> that expands to
>
>         struct raw_object_store x = {
>                 ...
>                 { &x.packed_git_mru, &x.packed_git_mru },
>                 ...
>         };
>
> no?  Then we do not need such a lengthy comment that reads only like
> an excuse ;-)

We cannot do this, because the object store is directly
embedded into the the_repository (not as a pointer),
which is a global on its own. So we'd have to do this
at the repository level, which I would not want.
I'd want to have the #defines where the structs are declared.

Any guidance on how to do that correctly with self referential
definitions without making the object store a pointer then?

Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> In a process with multiple repositories open, packfile accessors
> should be associated to a single repository and not shared globally.
> Move packed_git and packed_git_mru into the_repository and adjust
> callers to reflect this.
>
> Patch generated by
>
>  1. Moving the struct packed_git declaration to object-store.h
>     and packed_git, packed_git_mru globals to struct object_store.
>
>  2. Applying the semantic patch
>     contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
>     This semantic patch is placed in a sub directory of the coccinelle
>     contrib dir, as this semantic patch is not expected to be of general
>     usefulness; it is only useful during developing this series and
>     merging it with other topics in flight. At a later date, just
>     delete that semantic patch.

Can the semantic patch go in the commit message instead?  It is very
brief.

Actually, I don't see this semantic patch in the diffstat.  Is the
commit message stale?

>  3. Applying line wrapping fixes from "make style" to break the
>     resulting long lines.
>
>  4. Adding missing #includes of repository.h and object-store.h
>     where needed.

Is there a way to automate this step?  (I'm asking for my own
reference when writing future patches, not because of any concern
about the correctness of this one.)
>
>  5. As the packfiles are now owned by an objectstore/repository, which
>     is ephemeral unlike globals, we introduce memory leaks. So address
>     them in raw_object_store_clear().

The compound words are confusing me.  What is an
objectstore/repository?  Are these referring to particular identifiers
or something else?

Would some wording like the following work?

   5. Freeing packed_git and packed_git_mru in raw_object_store_clear
      to avoid a per-repository memory leak.  Previously they were
      global singletons, so code to free them did not exist.

[...]
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -12,6 +12,7 @@
>  #include "exec_cmd.h"
>  #include "streaming.h"
>  #include "thread-utils.h"
> +#include "object-store.h"
>  #include "packfile.h"
>  
>  static const char index_pack_usage[] =

Change from a different patch leaked into this one?

[...]
> +++ b/builtin/pack-objects.c
[...]
> @@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
>  			}
>  			want = want_found_object(exclude, p);
>  			if (!exclude && want > 0)
> -				list_move(&p->mru, &packed_git_mru);
> +				list_move(&p->mru, &the_repository->objects.packed_git_mru);

Long line.  Can "make style" catch this?

[...]
> +++ b/builtin/receive-pack.c
> @@ -7,6 +7,7 @@
>  #include "sideband.h"
>  #include "run-command.h"
>  #include "exec_cmd.h"
> +#include "object-store.h"
>  #include "commit.h"
>  #include "object.h"
>  #include "remote.h"

Another change leaked in?

[...]
> --- a/cache.h
> +++ b/cache.h
> @@ -1585,35 +1585,6 @@ struct pack_window {
>  	unsigned int inuse_cnt;
>  };
>  
> -extern struct packed_git {
[...]
> -} *packed_git;

Move detecting diff confirms that this wasn't modified.  Thanks for
creating it.

[...]
> +++ b/fast-import.c
[...]
> @@ -1110,7 +1112,7 @@ static int store_object(
>  	if (e->idx.offset) {
>  		duplicate_count_by_type[type]++;
>  		return 1;
> -	} else if (find_sha1_pack(oid.hash, packed_git)) {
> +	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {

Long line.  (I'll refrain from commenting about any further ones.)

[...]
> +++ b/http-push.c
> @@ -1,4 +1,5 @@
>  #include "cache.h"
> +#include "object-store.h"
>  #include "commit.h"
>  #include "tag.h"
>  #include "blob.h"

Stray change?

> diff --git a/http-walker.c b/http-walker.c
> index 07c2b1af82..8bb5d991bb 100644
> --- a/http-walker.c
> +++ b/http-walker.c
> @@ -4,6 +4,7 @@
>  #include "http.h"
>  #include "list.h"
>  #include "transport.h"
> +#include "object-store.h"
>  #include "packfile.h"
>  
>  struct alt_base {

Same question.

> diff --git a/http.c b/http.c
> index 31755023a4..a4a9e583c7 100644
> --- a/http.c
> +++ b/http.c
> @@ -1,6 +1,7 @@
>  #include "git-compat-util.h"
>  #include "http.h"
>  #include "config.h"
> +#include "object-store.h"
>  #include "pack.h"
>  #include "sideband.h"
>  #include "run-command.h"

Likewise.

> diff --git a/object-store.h b/object-store.h
> index e78eea1dde..1de9e07102 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
>   */
>  struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
>  
> +struct packed_git {
> +	struct packed_git *next;
> +	struct list_head mru;
> +	struct pack_window *windows;
> +	off_t pack_size;
> +	const void *index_data;
> +	size_t index_size;
> +	uint32_t num_objects;
> +	uint32_t num_bad_objects;
> +	unsigned char *bad_object_sha1;
> +	int index_version;
> +	time_t mtime;
> +	int pack_fd;
> +	unsigned pack_local:1,
> +		 pack_keep:1,
> +		 freshened:1,
> +		 do_not_close:1,
> +		 pack_promisor:1;
> +	unsigned char sha1[20];
> +	struct revindex_entry *revindex;
> +	/* something like ".git/objects/pack/xxxxx.pack" */
> +	char pack_name[FLEX_ARRAY]; /* more */
> +};
> +
>  struct raw_object_store {
>  	/*
>  	 * Path to the repository's object store.
> @@ -59,10 +83,25 @@ struct raw_object_store {
>  	 */
>  	char *objectdir;
>  
> +	struct packed_git *packed_git;
> +	/*
> +	 * A most-recently-used ordered version of the packed_git list, which can
> +	 * be iterated instead of packed_git (and marked via mru_mark).
> +	 */
> +	struct list_head packed_git_mru;

I don't understand the new part of the comment.  Can you explain here,
for me?

Is this meant as a list of related functions, an explanation of what the
field is for, or something else?

> +
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
> +
> +/*
> + * The mru list_head is supposed to be initialized using
> + * the LIST_HEAD macro, assigning prev/next to itself.
> + * However this doesn't work in this case as some compilers dislike
> + * that macro on member variables. Use NULL instead as that is defined
> + * and accepted, deferring the real init to prepare_packed_git_mru(). */

style nit: '*/' should be on its own line.

More importantly, we can avoid such an issue as described by Junio. :)

> +#define __MRU_INIT { NULL, NULL }

Identifiers with leading underscores like this are in a reserved
namespace for the language implementation --- we can't count on them
being available for our own use.

> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
[...]
> --- a/object.c
> +++ b/object.c
> @@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
>  
>  	free_alt_odbs(o);
>  	o->alt_odb_tail = NULL;
> +
> +	while (!list_empty(&o->packed_git_mru))
> +		list_del(&o->packed_git_mru);
> +	/*
> +	 * TODO: call close_all_packs once migrated to
> +	 * take an object store argument
> +	 */

Can you say more about this TODO?  Does this mean the patches are out
of order (i.e. that raw_object_store_clear leaves behind a leak until
a later patch)?

[...]
> --- a/pack-check.c
> +++ b/pack-check.c
> @@ -1,5 +1,6 @@
>  #include "cache.h"
>  #include "pack.h"
> +#include "object-store.h"
>  #include "pack-revindex.h"
>  #include "progress.h"
>  #include "packfile.h"

Another unexplained #include (I'll refrain from pointing out later
ones).

The rest looks good.

Thanks,
Jonathan

Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

>>  2. Applying the semantic patch
>>     contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
>>     This semantic patch is placed in a sub directory of the coccinelle
>>     contrib dir, as this semantic patch is not expected to be of general
>>     usefulness; it is only useful during developing this series and
>>     merging it with other topics in flight. At a later date, just
>>     delete that semantic patch.
>
> Can the semantic patch go in the commit message instead?  It is very
> brief.

done

>
> Actually, I don't see this semantic patch in the diffstat.  Is the
> commit message stale?
>
>>  3. Applying line wrapping fixes from "make style" to break the
>>     resulting long lines.
>>
>>  4. Adding missing #includes of repository.h and object-store.h
>>     where needed.
>
> Is there a way to automate this step?  (I'm asking for my own
> reference when writing future patches, not because of any concern
> about the correctness of this one.)

no, for several reasons.
(a) I don't know how to write it in coccinelle, because
(b) I have not figured out the order in which we include headers, apart from
  "cache.h goes first", the rest of the includes sometimes looks like a random
  order, because different patches add new includes at different places.
  I have the impression, that (1) some add the include after all other
  existing includes or (2) try to figure out where to add the include to make
  sense in the existing include order or (3) sort alphabetically or (4) put it
  randomly, so the chance for a merge conflict with other series in flight
  decreases.
(c) I did it in a semi automated fashion:
    while make fails:
        add another include

The mess with including repository or object-store comes from the fact that
I had v2 based on object-store, not the repository and cherry-picked this patch
over to v3. Fixed all of the includes now.

>> @@ -59,10 +83,25 @@ struct raw_object_store {
>>        */
>>       char *objectdir;
>>
>> +     struct packed_git *packed_git;
>> +     /*
>> +      * A most-recently-used ordered version of the packed_git list, which can
>> +      * be iterated instead of packed_git (and marked via mru_mark).
>> +      */
>> +     struct list_head packed_git_mru;
>
> I don't understand the new part of the comment.  Can you explain here,
> for me?

cherrypicking error, fixed.

>> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
>> +
>> +/*
>> + * The mru list_head is supposed to be initialized using
>> + * the LIST_HEAD macro, assigning prev/next to itself.
>> + * However this doesn't work in this case as some compilers dislike
>> + * that macro on member variables. Use NULL instead as that is defined
>> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
>
> style nit: '*/' should be on its own line.
>
> More importantly, we can avoid such an issue as described by Junio. :)

See reply to Junio, I am not quite sure I like that.

[PATCH 06/27] object-store: close all packs upon clearing the object store

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 5bdd2d7578..4762a702e3 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593..13cfaa6488 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2e..4d72efca78 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index c76b62572a..34daaf37b3 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	while (!list_empty(&o->packed_git_mru))
 		list_del(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index d41e4c83d0..511a2b0cdf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d6..6a2c57045c 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 06/27] object-store: close all packs upon clearing the object store

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>

Straight forward change, looks good.

> ---
>  builtin/am.c           | 2 +-
>  builtin/clone.c        | 2 +-
>  builtin/fetch.c        | 2 +-
>  builtin/merge.c        | 2 +-
>  builtin/receive-pack.c | 2 +-
>  object.c               | 6 ++----
>  packfile.c             | 4 ++--
>  packfile.h             | 2 +-
>  8 files changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/builtin/am.c b/builtin/am.c
> index 5bdd2d7578..4762a702e3 100644
> --- a/builtin/am.c
> +++ b/builtin/am.c
> @@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
>  	 */
>  	if (!state->rebasing) {
>  		am_destroy(state);
> -		close_all_packs();
> +		close_all_packs(&the_repository->objects);
>  		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
>  	}
>  }
> diff --git a/builtin/clone.c b/builtin/clone.c
> index 101c27a593..13cfaa6488 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>  	transport_disconnect(transport);
>  
>  	if (option_dissociate) {
> -		close_all_packs();
> +		close_all_packs(&the_repository->objects);
>  		dissociate_from_references();
>  	}
>  
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 8ee998ea2e..4d72efca78 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  
>  	string_list_clear(&list, 0);
>  
> -	close_all_packs();
> +	close_all_packs(&the_repository->objects);
>  
>  	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
>  	if (verbosity < 0)
> diff --git a/builtin/merge.c b/builtin/merge.c
> index 30264cfd7c..907ae44ab5 100644
> --- a/builtin/merge.c
> +++ b/builtin/merge.c
> @@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
>  			 * We ignore errors in 'gc --auto', since the
>  			 * user should see them.
>  			 */
> -			close_all_packs();
> +			close_all_packs(&the_repository->objects);
>  			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
>  		}
>  	}
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index b2eac79a6e..954fc72c7c 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  			proc.git_cmd = 1;
>  			proc.argv = argv_gc_auto;
>  
> -			close_all_packs();
> +			close_all_packs(&the_repository->objects);
>  			if (!start_command(&proc)) {
>  				if (use_sideband)
>  					copy_to_sideband(proc.err, -1, NULL);
> diff --git a/object.c b/object.c
> index c76b62572a..34daaf37b3 100644
> --- a/object.c
> +++ b/object.c
> @@ -4,6 +4,7 @@
>  #include "tree.h"
>  #include "commit.h"
>  #include "tag.h"
> +#include "packfile.h"
>  
>  static struct object **obj_hash;
>  static int nr_objs, obj_hash_size;
> @@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
>  
>  	while (!list_empty(&o->packed_git_mru))
>  		list_del(&o->packed_git_mru);
> -	/*
> -	 * TODO: call close_all_packs once migrated to
> -	 * take an object store argument
> -	 */
> +	close_all_packs(o);
>  }
> diff --git a/packfile.c b/packfile.c
> index d41e4c83d0..511a2b0cdf 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
>  	close_pack_index(p);
>  }
>  
> -void close_all_packs(void)
> +void close_all_packs(struct raw_object_store *o)
>  {
>  	struct packed_git *p;
>  
> -	for (p = the_repository->objects.packed_git; p; p = p->next)
> +	for (p = o->packed_git; p; p = p->next)
>  		if (p->do_not_close)
>  			die("BUG: want to close pack marked 'do-not-close'");
>  		else
> diff --git a/packfile.h b/packfile.h
> index a7fca598d6..6a2c57045c 100644
> --- a/packfile.h
> +++ b/packfile.h
> @@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
>  
>  extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
>  extern void close_pack_windows(struct packed_git *);
> -extern void close_all_packs(void);
> +extern void close_all_packs(struct raw_object_store *o);
>  extern void unuse_pack(struct pack_window **);
>  extern void clear_delta_base_cache(void);
>  extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

[PATCH 07/27] pack: move prepare_packed_git_run_once to object store

[Stefan Beller]

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 1de9e07102..6cecba3951 100644
--- a/object-store.h
+++ b/object-store.h
@@ -92,6 +92,12 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
 /*
@@ -101,7 +107,7 @@ struct raw_object_store {
  * that macro on member variables. Use NULL instead as that is defined
  * and accepted, deferring the real init to prepare_packed_git_mru(). */
 #define __MRU_INIT { NULL, NULL }
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 511a2b0cdf..a8a2e7fe43 100644
--- a/packfile.c
+++ b/packfile.c
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 07/27] pack: move prepare_packed_git_run_once to object store

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:10 -0800
Stefan Beller <sbeller@google.com> wrote:

> Each repository's object store can be initialized independently, so
> they must not share a run_once variable.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }

Optional: Trailing zeros are not needed in struct initializers.

[PATCH 08/27] pack: move approximate object count to object store

[Stefan Beller]

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 6cecba3951..bd1e4fcd8b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -93,6 +93,14 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -107,7 +115,7 @@ struct raw_object_store {
  * that macro on member variables. Use NULL instead as that is defined
  * and accepted, deferring the real init to prepare_packed_git_mru(). */
 #define __MRU_INIT { NULL, NULL }
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index a8a2e7fe43..693bafbc98 100644
--- a/packfile.c
+++ b/packfile.c
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 08/27] pack: move approximate object count to object store

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> The approximate_object_count() function maintains a rough count of
> objects in a repository to estimate how long object name abbreviates
> should be.  Object names are scoped to a repository and the
> appropriate length may differ by repository, so the object count
> should not be global.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  object-store.h | 10 +++++++++-
>  packfile.c     | 11 +++++------
>  2 files changed, 14 insertions(+), 7 deletions(-)
> 
> diff --git a/object-store.h b/object-store.h
> index 6cecba3951..bd1e4fcd8b 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -93,6 +93,14 @@ struct raw_object_store {
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  
> +	/*
> +	 * A fast, rough count of the number of objects in the repository.
> +	 * These two fields are not meant for direct access. Use
> +	 * approximate_object_count() instead.
> +	 */
> +	unsigned long approximate_object_count;
> +	unsigned approximate_object_count_valid : 1;

Patch looks fine and is effectively a no-op, though what is the need for
both of these variables?  Maybe it can be simplified down to just use
one?  Just musing as its out of the scope of this patch and we probably
shouldn't try to change that in this series.

> +
>  	/*
>  	 * Whether packed_git has already been populated with this repository's
>  	 * packs.
> @@ -107,7 +115,7 @@ struct raw_object_store {
>   * that macro on member variables. Use NULL instead as that is defined
>   * and accepted, deferring the real init to prepare_packed_git_mru(). */
>  #define __MRU_INIT { NULL, NULL }
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }
>  
>  void raw_object_store_clear(struct raw_object_store *o);
>  
> diff --git a/packfile.c b/packfile.c
> index a8a2e7fe43..693bafbc98 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
>  	strbuf_release(&path);
>  }
>  
> -static int approximate_object_count_valid;
> -
>  /*
>   * Give a fast, rough count of the number of objects in the repository. This
>   * ignores loose objects completely. If you have a lot of them, then either
> @@ -814,8 +812,8 @@ static int approximate_object_count_valid;
>   */
>  unsigned long approximate_object_count(void)
>  {
> -	static unsigned long count;
> -	if (!approximate_object_count_valid) {
> +	if (!the_repository->objects.approximate_object_count_valid) {
> +		unsigned long count;
>  		struct packed_git *p;
>  
>  		prepare_packed_git();
> @@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
>  				continue;
>  			count += p->num_objects;
>  		}
> +		the_repository->objects.approximate_object_count = count;
>  	}
> -	return count;
> +	return the_repository->objects.approximate_object_count;
>  }
>  
>  static void *get_next_packed_git(const void *p)
> @@ -901,7 +900,7 @@ void prepare_packed_git(void)
>  
>  void reprepare_packed_git(void)
>  {
> -	approximate_object_count_valid = 0;
> +	the_repository->objects.approximate_object_count_valid = 0;
>  	the_repository->objects.packed_git_initialized = 0;
>  	prepare_packed_git();
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

Re: [PATCH 08/27] pack: move approximate object count to object store

[Stefan Beller]

On Wed, Feb 21, 2018 at 4:47 PM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> The approximate_object_count() function maintains a rough count of
>> objects in a repository to estimate how long object name abbreviates
>> should be.  Object names are scoped to a repository and the
>> appropriate length may differ by repository, so the object count
>> should not be global.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>> ---
>>  object-store.h | 10 +++++++++-
>>  packfile.c     | 11 +++++------
>>  2 files changed, 14 insertions(+), 7 deletions(-)
>>
>> diff --git a/object-store.h b/object-store.h
>> index 6cecba3951..bd1e4fcd8b 100644
>> --- a/object-store.h
>> +++ b/object-store.h
>> @@ -93,6 +93,14 @@ struct raw_object_store {
>>       struct alternate_object_database *alt_odb_list;
>>       struct alternate_object_database **alt_odb_tail;
>>
>> +     /*
>> +      * A fast, rough count of the number of objects in the repository.
>> +      * These two fields are not meant for direct access. Use
>> +      * approximate_object_count() instead.
>> +      */
>> +     unsigned long approximate_object_count;
>> +     unsigned approximate_object_count_valid : 1;
>
> Patch looks fine and is effectively a no-op, though what is the need for
> both of these variables?  Maybe it can be simplified down to just use
> one?  Just musing as its out of the scope of this patch and we probably
> shouldn't try to change that in this series.

I agree we should. It was introduced in e323de4ad7f (sha1_file:
allow sha1_loose_object_info to handle arbitrary repositories, 2017-08-30)
and I think it was seen as a clever optimization trick back then?

Re: [PATCH 08/27] pack: move approximate object count to object store

[Jeff King]

On Fri, Feb 23, 2018 at 02:22:14PM -0800, Stefan Beller wrote:

> >> +     /*
> >> +      * A fast, rough count of the number of objects in the repository.
> >> +      * These two fields are not meant for direct access. Use
> >> +      * approximate_object_count() instead.
> >> +      */
> >> +     unsigned long approximate_object_count;
> >> +     unsigned approximate_object_count_valid : 1;
> >
> > Patch looks fine and is effectively a no-op, though what is the need for
> > both of these variables?  Maybe it can be simplified down to just use
> > one?  Just musing as its out of the scope of this patch and we probably
> > shouldn't try to change that in this series.
> 
> I agree we should. It was introduced in e323de4ad7f (sha1_file:
> allow sha1_loose_object_info to handle arbitrary repositories, 2017-08-30)
> and I think it was seen as a clever optimization trick back then?

I think you meant 8e3f52d778 (find_unique_abbrev: move logic out of
get_short_sha1(), 2016-10-03)?

Yes, it was just to avoid the dual-meaning of "0" for "not set" and a
repository with no packfiles.  It would probably be fine to get rid of
it. If you have no packfiles then you probably don't have enough objects
to worry about micro-optimizing. And anyway, the "wasted" case wouldn't
even make any syscalls (it would do a noop prepare_packed_git() and then
realize the packed_git list is empty).

-Peff

Re: [PATCH 08/27] pack: move approximate object count to object store

[Stefan Beller]

On Mon, Feb 26, 2018 at 12:55 AM, Jeff King <peff@peff.net> wrote:
> On Fri, Feb 23, 2018 at 02:22:14PM -0800, Stefan Beller wrote:
>
>> >> +     /*
>> >> +      * A fast, rough count of the number of objects in the repository.
>> >> +      * These two fields are not meant for direct access. Use
>> >> +      * approximate_object_count() instead.
>> >> +      */
>> >> +     unsigned long approximate_object_count;
>> >> +     unsigned approximate_object_count_valid : 1;
>> >
>> > Patch looks fine and is effectively a no-op, though what is the need for
>> > both of these variables?  Maybe it can be simplified down to just use
>> > one?  Just musing as its out of the scope of this patch and we probably
>> > shouldn't try to change that in this series.
>>
>> I agree we should. It was introduced in e323de4ad7f (sha1_file:
>> allow sha1_loose_object_info to handle arbitrary repositories, 2017-08-30)
>> and I think it was seen as a clever optimization trick back then?
>
> I think you meant 8e3f52d778 (find_unique_abbrev: move logic out of
> get_short_sha1(), 2016-10-03)?

Yes, copy paste error.

>
> Yes, it was just to avoid the dual-meaning of "0" for "not set" and a
> repository with no packfiles.  It would probably be fine to get rid of
> it. If you have no packfiles then you probably don't have enough objects
> to worry about micro-optimizing. And anyway, the "wasted" case wouldn't
> even make any syscalls (it would do a noop prepare_packed_git() and then
> realize the packed_git list is empty).

Good point!

>
> -Peff

[PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable

[Stefan Beller]

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1348dce68f..a8e23bd2f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:12 -0800
Stefan Beller <sbeller@google.com> wrote:

> Add a raw_object_store to alt_odb_usable to be more specific about which
> repository to act on. The choice of the repository is delegated to its
> only caller link_alt_odb_entry.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>

I checked that alt_odb_usable no longer depends on any repository-like
globals.

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

[PATCH 10/27] sha1_file: add repository argument to link_alt_odb_entry

[Stefan Beller]

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index a8e23bd2f8..aaa6276211 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -390,8 +390,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -488,7 +489,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 11/27] sha1_file: add repository argument to read_info_alternates

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index aaa6276211..6edcc3d3a7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -389,7 +389,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -430,7 +432,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -496,7 +498,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -680,7 +683,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 12/27] sha1_file: add repository argument to link_alt_odb_entries

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6edcc3d3a7..a0e9116318 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -467,8 +467,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -511,7 +515,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -565,7 +569,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -578,7 +583,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -681,7 +687,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb

[Stefan Beller]

See previous patch for explanation.

While at it, move the declaration to object-store.h,
where it should be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 object-store.h |  3 ++-
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  3 ++-
 5 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 2e99e02128..76c94e9b5a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/object-store.h b/object-store.h
index bd1e4fcd8b..514ad94287 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,7 +20,8 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-void prepare_alt_odb(void);
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+void prepare_alt_odb_the_repository(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/packfile.c b/packfile.c
index 693bafbc98..ccfe084642 100644
--- a/packfile.c
+++ b/packfile.c
@@ -890,7 +890,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index a0e9116318..c26921df4a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -581,7 +582,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -667,7 +668,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -676,7 +677,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -730,7 +731,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -889,7 +890,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -920,7 +921,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..3e490ee8f6 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> See previous patch for explanation.
>
> While at it, move the declaration to object-store.h,
> where it should be easier to find.

Which declaration?  It looks like prepare_alt_odb is already in
object-store.h.

[...]
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  	} else {
>  		fsck_object_dir(get_object_directory());
>  
> -		prepare_alt_odb();
> +		prepare_alt_odb(the_repository);

Patch 2 added a #include of "repository.h".  Good.

(I checked because with the definition of prepare_alt_odb as a macro,
the function call would compile correctly even if the_repository
weren't in scope, but we want to include what we use as a matter of
style/maintainability.)

[...]
> --- a/packfile.c
> +++ b/packfile.c
> @@ -890,7 +890,7 @@ void prepare_packed_git(void)
>  	if (the_repository->objects.packed_git_initialized)
>  		return;
>  	prepare_packed_git_one(get_object_directory(), 1);
> -	prepare_alt_odb();
> +	prepare_alt_odb(the_repository);

Also good, since patch 3 added a #include of "repository.h".

[...]
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -23,6 +23,7 @@
>  #include "sha1-lookup.h"
>  #include "bulk-checkin.h"
>  #include "repository.h"
> +#include "object-store.h"

Should this #include have been added in an earlier patch, since the
file both uses and defines prepare_alt_odb, which is declared there?

The rest looks good.

Thanks,
Jonathan

Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb

[Stefan Beller]

>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -23,6 +23,7 @@
>>  #include "sha1-lookup.h"
>>  #include "bulk-checkin.h"
>>  #include "repository.h"
>> +#include "object-store.h"
>
> Should this #include have been added in an earlier patch, since the
> file both uses and defines prepare_alt_odb, which is declared there?

This is another superflous include, dropped it. Thanks for catching.

[PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories

[Stefan Beller]

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  4 ++++
 sha1_file.c    | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/object-store.h b/object-store.h
index 514ad94287..f283fbdba9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -18,6 +18,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 #define prepare_alt_odb(r) prepare_alt_odb_##r()
diff --git a/sha1_file.c b/sha1_file.c
index c26921df4a..6e5105a252 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> Actually this also allows read_info_alternates and link_alt_odb_entry to
> handle arbitrary repositories, but link_alt_odb_entries is the most
> interesting function in this set of functions, hence the commit subject.
> 
> These functions span a strongly connected component in the function
> graph, i.e. the recursive call chain might look like
> 
>   -> link_alt_odb_entries
>     -> link_alt_odb_entry
>       -> read_info_alternates
>         -> link_alt_odb_entries
> 
> That is why we need to convert all these functions at the same time.
> 
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---

Looks good.

>  object-store.h |  4 ++++
>  sha1_file.c    | 36 ++++++++++++++++--------------------
>  2 files changed, 20 insertions(+), 20 deletions(-)
> 
> diff --git a/object-store.h b/object-store.h
> index 514ad94287..f283fbdba9 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -18,6 +18,10 @@ struct alternate_object_database {
>  	char loose_objects_subdir_seen[256];
>  	struct oid_array loose_objects_cache;
>  
> +	/*
> +	 * Path to the alternative object store. If this is a relative path,
> +	 * it is relative to the current working directory.
> +	 */

Thanks for adding the comment.

>  	char path[FLEX_ARRAY];
>  };
>  #define prepare_alt_odb(r) prepare_alt_odb_##r()
> diff --git a/sha1_file.c b/sha1_file.c
> index c26921df4a..6e5105a252 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,
>   * SHA1, an extra slash for the first level indirection, and the
>   * terminating NUL.
>   */
> -#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
> -static void read_info_alternates_the_repository(const char *relative_base,
> -						int depth);
> -#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
> -static int link_alt_odb_entry_the_repository(const char *entry,
> +static void read_info_alternates(struct repository *r,
> +				 const char *relative_base,
> +				 int depth);
> +static int link_alt_odb_entry(struct repository *r, const char *entry,
>  	const char *relative_base, int depth, const char *normalized_objdir)
>  {
>  	struct alternate_object_database *ent;
> @@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
>  	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
>  		strbuf_setlen(&pathbuf, pathbuf.len - 1);
>  
> -	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
> +	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
>  		strbuf_release(&pathbuf);
>  		return -1;
>  	}
> @@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
>  	ent = alloc_alt_odb(pathbuf.buf);
>  
>  	/* add the alternate entry */
> -	*the_repository->objects.alt_odb_tail = ent;
> -	the_repository->objects.alt_odb_tail = &(ent->next);
> +	*r->objects.alt_odb_tail = ent;
> +	r->objects.alt_odb_tail = &(ent->next);
>  	ent->next = NULL;
>  
>  	/* recursively add alternates */
> -	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
> +	read_info_alternates(r, pathbuf.buf, depth + 1);
>  
>  	strbuf_release(&pathbuf);
>  	return 0;
> @@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,
>  	return end;
>  }
>  
> -#define link_alt_odb_entries(r, a, s, rb, d) \
> -	link_alt_odb_entries_##r(a, s, rb, d)
> -static void link_alt_odb_entries_the_repository(const char *alt,
> -						int sep,
> -						const char *relative_base,
> -						int depth)
> +static void link_alt_odb_entries(struct repository *r, const char *alt,
> +				 int sep, const char *relative_base, int depth)
>  {
>  	struct strbuf objdirbuf = STRBUF_INIT;
>  	struct strbuf entry = STRBUF_INIT;
> @@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
>  		return;
>  	}
>  
> -	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
> +	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
>  	if (strbuf_normalize_path(&objdirbuf) < 0)
>  		die("unable to normalize object directory: %s",
>  		    objdirbuf.buf);
> @@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
>  		alt = parse_alt_odb_entry(alt, sep, &entry);
>  		if (!entry.len)
>  			continue;
> -		link_alt_odb_entry(the_repository, entry.buf,
> +		link_alt_odb_entry(r, entry.buf,
>  				   relative_base, depth, objdirbuf.buf);
>  	}
>  	strbuf_release(&entry);
>  	strbuf_release(&objdirbuf);
>  }
>  
> -static void read_info_alternates_the_repository(const char *relative_base,
> -						int depth)
> +static void read_info_alternates(struct repository *r,
> +				 const char *relative_base,
> +				 int depth)
>  {
>  	char *path;
>  	struct strbuf buf = STRBUF_INIT;
> @@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
>  		return;
>  	}
>  
> -	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
> +	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
>  	strbuf_release(&buf);
>  	free(path);
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

[PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 21 +++++++++++----------
 2 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/object-store.h b/object-store.h
index f283fbdba9..873b341774 100644
--- a/object-store.h
+++ b/object-store.h
@@ -24,8 +24,7 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/sha1_file.c b/sha1_file.c
index 6e5105a252..9cf3fffbeb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -673,21 +673,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+
+	if (!r->ignore_env) {
+		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+		if (!alt)
+			alt = "";
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	}
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:18 -0800
Stefan Beller <sbeller@google.com> wrote:

> -void prepare_alt_odb_the_repository(void)
> +void prepare_alt_odb(struct repository *r)
>  {
> -	const char *alt;
> -
> -	if (the_repository->objects.alt_odb_tail)
> +	if (r->objects.alt_odb_tail)
>  		return;
>  
> -	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> +
> +	if (!r->ignore_env) {
> +		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +		if (!alt)
> +			alt = "";

alt can be NULL, just like in the existing code, so these 2 lines are
unnecessary. (link_alt_odb_entries() will just do nothing in either
case.)

(I also think that the check of absence of alt should be done by the
caller of link_alt_odb_entries(), not by link_alt_odb_entries() itself,
but that is much beyond the scope of this patch set.)

>  
> -	the_repository->objects.alt_odb_tail =
> -			&the_repository->objects.alt_odb_list;
> -	link_alt_odb_entries(the_repository, alt,
> -			     PATH_SEP, NULL, 0);
> +		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> +	}
>  
> -	read_info_alternates(the_repository, get_object_directory(), 0);
> +	read_info_alternates(r, r->objects.objectdir, 0);
>  }

Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Stefan Beller]

On Wed, Feb 21, 2018 at 4:35 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
> On Tue, 20 Feb 2018 17:54:18 -0800
> Stefan Beller <sbeller@google.com> wrote:
>
>> -void prepare_alt_odb_the_repository(void)
>> +void prepare_alt_odb(struct repository *r)
>>  {
>> -     const char *alt;
>> -
>> -     if (the_repository->objects.alt_odb_tail)
>> +     if (r->objects.alt_odb_tail)
>>               return;
>>
>> -     alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>> +     r->objects.alt_odb_tail = &r->objects.alt_odb_list;
>> +
>> +     if (!r->ignore_env) {
>> +             const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>> +             if (!alt)
>> +                     alt = "";
>
> alt can be NULL, just like in the existing code, so these 2 lines are
> unnecessary. (link_alt_odb_entries() will just do nothing in either
> case.)
>
> (I also think that the check of absence of alt should be done by the
> caller of link_alt_odb_entries(), not by link_alt_odb_entries() itself,
> but that is much beyond the scope of this patch set.)
>

reverted to a literal translation of s/the_repository/r/, I forget
why I implemented this change in the first place. dropped for now.

Thanks,
Stefan

[PATCH 16/27] sha1_file: add repository argument to sha1_file_name

[Stefan Beller]

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  7 +++++++
 sha1_file.c    | 10 +++++-----
 5 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/cache.h b/cache.h
index 3fd9dc52cd..e641c23b0a 100644
--- a/cache.h
+++ b/cache.h
@@ -961,12 +961,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 8bb5d991bb..76bfdbca76 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(the_repository, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index a4a9e583c7..afe2707e86 100644
--- a/http.c
+++ b/http.c
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(the_repository, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 873b341774..4f5060e768 100644
--- a/object-store.h
+++ b/object-store.h
@@ -123,4 +123,11 @@ struct raw_object_store {
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 9cf3fffbeb..811ed9e53f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
 {
 	strbuf_addstr(buf, get_object_directory());
 	strbuf_addch(buf, '/');
@@ -720,7 +720,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -881,7 +881,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -910,7 +910,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1595,7 +1595,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name

[Brandon Williams]

On 02/20, Stefan Beller wrote:
> Add a repository argument to allow sha1_file_name callers to be more
> specific about which repository to handle. This is a small mechanical
> change; it doesn't change the implementation to handle repositories
> other than the_repository yet.
> 
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
> 
> While at it, move the declaration to object-store.h, where it should
> be easier to find.

Seems like we may want to make a sha1-file.h or an oid-file.h or
something like that at some point as that seems like a better place for
the function than in the object-store.h file?


-- 
Brandon Williams

Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name

[Stefan Beller]

On Wed, Feb 21, 2018 at 4:51 PM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> Add a repository argument to allow sha1_file_name callers to be more
>> specific about which repository to handle. This is a small mechanical
>> change; it doesn't change the implementation to handle repositories
>> other than the_repository yet.
>>
>> As with the previous commits, use a macro to catch callers passing a
>> repository other than the_repository at compile time.
>>
>> While at it, move the declaration to object-store.h, where it should
>> be easier to find.
>
> Seems like we may want to make a sha1-file.h or an oid-file.h or
> something like that at some point as that seems like a better place for
> the function than in the object-store.h file?

It depends what our long term goal is.
Do we want header and source file name to match for each function?
Or do we want a coarser set of headers, such that we have a broad
object-store.h, but that is implemented in sha1_file.c, packfile.c
for the parts of the raw_objectstore and other .c files for the higher
levels of the object store?

For now I'd just keep it in object-store.h as moving out just a couple
functions seems less consistent?

Stefan

Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name

[Brandon Williams]

On 02/23, Stefan Beller wrote:
> On Wed, Feb 21, 2018 at 4:51 PM, Brandon Williams <bmwill@google.com> wrote:
> > On 02/20, Stefan Beller wrote:
> >> Add a repository argument to allow sha1_file_name callers to be more
> >> specific about which repository to handle. This is a small mechanical
> >> change; it doesn't change the implementation to handle repositories
> >> other than the_repository yet.
> >>
> >> As with the previous commits, use a macro to catch callers passing a
> >> repository other than the_repository at compile time.
> >>
> >> While at it, move the declaration to object-store.h, where it should
> >> be easier to find.
> >
> > Seems like we may want to make a sha1-file.h or an oid-file.h or
> > something like that at some point as that seems like a better place for
> > the function than in the object-store.h file?
> 
> It depends what our long term goal is.
> Do we want header and source file name to match for each function?
> Or do we want a coarser set of headers, such that we have a broad
> object-store.h, but that is implemented in sha1_file.c, packfile.c
> for the parts of the raw_objectstore and other .c files for the higher
> levels of the object store?
> 
> For now I'd just keep it in object-store.h as moving out just a couple
> functions seems less consistent?

Fair enough :)

-- 
Brandon Williams

[PATCH 17/27] sha1_file: add repository argument to stat_sha1_file

[Stefan Beller]

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 811ed9e53f..ed297773e5 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
@@ -1181,7 +1182,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1395,7 +1396,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> Add a repository argument to allow the stat_sha1_file caller to be
> more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
>
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  sha1_file.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)

Simple and obviously correct.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

[PATCH 18/27] sha1_file: add repository argument to open_sha1_file

[Stefan Beller]

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index ed297773e5..68aff90792 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -903,7 +903,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -946,7 +948,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1

[Stefan Beller]

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 68aff90792..7fa9ed2f25 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -938,9 +938,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -969,7 +970,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2199,7 +2200,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:22 -0800
Stefan Beller <sbeller@google.com> wrote:

> Add a repository argument to allow the map_sha1_file_1 caller to be
> more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
> 
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

[PATCH 20/27] sha1_file: add repository argument to map_sha1_file

[Stefan Beller]

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 3 +++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index e641c23b0a..d338b65d75 100644
--- a/cache.h
+++ b/cache.h
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 4f5060e768..a37d74a0c5 100644
--- a/object-store.h
+++ b/object-store.h
@@ -130,4 +130,7 @@ void raw_object_store_clear(struct raw_object_store *o);
 #define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
 void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
 
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 7fa9ed2f25..0d3ff99cf2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -968,7 +968,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1192,7 +1192,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..22d27df55e 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info

[Stefan Beller]

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 0d3ff99cf2..d1f77a4c80 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1159,9 +1159,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1280,7 +1281,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info

[Jonathan Nieder]

Hi,

Stefan Beller wrote:

> Add a repository argument to allow the sha1_loose_object_info caller
> to be more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
>
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  sha1_file.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)

Simple and obviously correct.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

[PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index a37d74a0c5..146358fe33 100644
--- a/object-store.h
+++ b/object-store.h
@@ -127,8 +127,7 @@ void raw_object_store_clear(struct raw_object_store *o);
  * Put in `buf` the name of the file in the local object database that
  * would be used to store a loose object with the specified sha1.
  */
-#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
 #define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
 void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
diff --git a/sha1_file.c b/sha1_file.c
index d1f77a4c80..8fad523d85 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, get_object_directory());
+	strbuf_addstr(buf, r->objects.objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:25 -0800
Stefan Beller <sbeller@google.com> wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

> -void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
> +void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
>  {
> -	strbuf_addstr(buf, get_object_directory());
> +	strbuf_addstr(buf, r->objects.objectdir);
>  	strbuf_addch(buf, '/');
>  	fill_sha1_path(buf, sha1);
>  }

In the future, we should probably have:
 - a function to get the object store out of a repo (so that it can
   lazily initialize the object store struct if necessary)
 - when the object store is obtained, its objectdir field is guaranteed
   to be populated
 - sha1_file_name should take the object store struct, not the repo
   struct

but this is outside the scope of this patch.

Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories

[Stefan Beller]

On Wed, Feb 21, 2018 at 4:44 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
> On Tue, 20 Feb 2018 17:54:25 -0800
> Stefan Beller <sbeller@google.com> wrote:
>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>
> Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
>
>> -void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
>> +void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
>>  {
>> -     strbuf_addstr(buf, get_object_directory());
>> +     strbuf_addstr(buf, r->objects.objectdir);
>>       strbuf_addch(buf, '/');
>>       fill_sha1_path(buf, sha1);
>>  }
>
> In the future, we should probably have:
>  - a function to get the object store out of a repo (so that it can
>    lazily initialize the object store struct if necessary)
>  - when the object store is obtained, its objectdir field is guaranteed
>    to be populated
>  - sha1_file_name should take the object store struct, not the repo
>    struct

We had that in v2, but I reverted it as the consensus seemed that we prefer
the_repository to be passed around.

>
> but this is outside the scope of this patch.

[PATCH 23/27] sha1_file: allow stat_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8fad523d85..6879af8993 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,23 +874,22 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
-static int stat_sha1_file_the_repository(const unsigned char *sha1,
-					 struct stat *st, const char **path)
+static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(the_repository);
+	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6879af8993..598acc5410 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -902,9 +902,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
-static int open_sha1_file_the_repository(const unsigned char *sha1,
-					 const char **path)
+static int open_sha1_file(struct repository *r,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -912,7 +911,7 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -920,8 +919,8 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories

[Jonathan Tan]

On Tue, 20 Feb 2018 17:54:27 -0800
Stefan Beller <sbeller@google.com> wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

[PATCH 25/27] sha1_file: allow map_sha1_file_1 to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 598acc5410..522804d9f0 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -936,10 +936,8 @@ static int open_sha1_file(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
-static void *map_sha1_file_1_the_repository(const char *path,
-					    const unsigned char *sha1,
-					    unsigned long *size)
+static void *map_sha1_file_1(struct repository *r, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -947,7 +945,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(the_repository, sha1, &path);
+		fd = open_sha1_file(r, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog

[PATCH 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 5 +++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index 146358fe33..5d93677783 100644
--- a/object-store.h
+++ b/object-store.h
@@ -129,7 +129,6 @@ void raw_object_store_clear(struct raw_object_store *o);
  */
 void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
-#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 522804d9f0..a20f811a07 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -964,9 +964,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,
 	return map;
 }
 
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct repository *r,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(the_repository, NULL, sha1, size);
+	return map_sha1_file_1(r, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCH 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories

[Jonathan Nieder]

Stefan Beller wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  object-store.h | 3 +--
>  sha1_file.c    | 5 +++--
>  2 files changed, 4 insertions(+), 4 deletions(-)

Thanks: this is short and simple, making my life as a reviewer very
easy.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

[PATCH 27/27] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index a20f811a07..36282acb1a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1156,10 +1156,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
-static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
-						 struct object_info *oi,
-						 int flags)
+static int sha1_loose_object_info(struct repository *r,
+				  const unsigned char *sha1,
+				  struct object_info *oi, int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1183,14 +1182,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
+		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(the_repository, sha1, &mapsize);
+	map = map_sha1_file(r, sha1, &mapsize);
 	if (!map)
 		return -1;
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCHv3 00/27] Moving global state into the repository object (part 1)

[Stefan Beller]

On Tue, Feb 20, 2018 at 5:54 PM, Stefan Beller <sbeller@google.com> wrote:
> v3:
> * reverted back to use the repository for most of the functions
>   (including unduplicating 'ignore_env')
> * rebased on master again (I lost that state when doing v2, as
>   I did both rebase as well as conversion to object store in one go)
> * one additional patch, that moves the alternates related things to
>   object-store.h, such that inclusion of cache.h (in object-store.h)
>   is not required any more.

This is also available at
https://github.com/stefanbeller/git/tree/object-store-part1-v3

[PATCHv4 00/27] Moving global state into the repository object (part 1)

[Stefan Beller]

v4:
* addressed feedback from the single patches (mostly nits)
* rebased on latest master

v3:
* reverted back to use the repository for most of the functions
  (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as
  I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to
  object-store.h, such that inclusion of cache.h (in object-store.h)
  is not required any more.

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.

Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (2):
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (25):
  repository: introduce raw object store field
  object-store: migrate alternates struct and functions from cache.h
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to map_sha1_file
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   5 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  12 ++--
 builtin/gc.c             |   3 +-
 builtin/grep.c           |   2 +-
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  20 ++++---
 builtin/pack-redundant.c |   5 +-
 builtin/receive-pack.c   |   2 +-
 cache.h                  |  87 ---------------------------
 environment.c            |   5 +-
 fast-import.c            |   7 ++-
 http-backend.c           |   5 +-
 http-walker.c            |   3 +-
 http.c                   |   5 +-
 object-store.h           | 124 ++++++++++++++++++++++++++++++++++++++
 object.c                 |  27 +++++++++
 pack-bitmap.c            |   3 +-
 packfile.c               |  64 ++++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 repository.c             |  21 +++++--
 repository.h             |   7 ++-
 server-info.c            |   5 +-
 sha1_file.c              | 125 +++++++++++++++++++++------------------
 sha1_name.c              |  10 ++--
 streaming.c              |   5 +-
 29 files changed, 335 insertions(+), 229 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 01/27] repository: introduce raw object store field

[Stefan Beller]

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 19 ++++++++++++++-----
 repository.h   |  7 ++++---
 7 files changed, 43 insertions(+), 12 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8c..0f0c195705f 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e7..ec10b062e68 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 00000000000..cf35760ceb6
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b0..a2acdee1405 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	FREE_AND_NULL(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730d..81a42d91155 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94e..fb65e8072d5 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,9 +49,10 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-					    "objects", !repo->ignore_env);
+	raw_object_store_clear(&repo->objects);
+	repo->objects.objectdir =
+		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+				  "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -209,12 +217,13 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f5..fa73ab8e938 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to accessing the raw object content.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCHv4 01/27] repository: introduce raw object store field

[Duy Nguyen]

On Fri, Feb 23, 2018 at 04:47:28PM -0800, Stefan Beller wrote:
>  /* The main repository */
>  static struct repository the_repo = {
> -	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
> +	NULL, NULL,
> +	RAW_OBJECT_STORE_INIT,
> +	NULL, NULL, NULL,
> +	NULL, NULL, NULL,
> +	&the_index,
> +	&hash_algos[GIT_HASH_SHA1],
> +	0, 0
>  };
>  struct repository *the_repository = &the_repo;

I wonder if we should do something like this. It makes the_repo
initialization easier to read and it helps unify the init code with
submodule.

No don't reroll. If you think it's a good idea, you can do something
like this in the next series instead.

-- 8< --
diff --git a/check-racy.c b/check-racy.c
index 24b6542352..47cbb4eb6d 100644
--- a/check-racy.c
+++ b/check-racy.c
@@ -1,10 +1,12 @@
 #include "cache.h"
+#include "repository.h"
 
 int main(int ac, char **av)
 {
 	int i;
 	int dirty, clean, racy;
 
+	init_the_repository();
 	dirty = clean = racy = 0;
 	read_cache();
 	for (i = 0; i < active_nr; i++) {
diff --git a/common-main.c b/common-main.c
index 6a689007e7..a13ab981aa 100644
--- a/common-main.c
+++ b/common-main.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "exec_cmd.h"
 #include "attr.h"
+#include "repository.h"
 
 /*
  * Many parts of Git have subprograms communicate via pipe, expect the
@@ -32,6 +33,8 @@ int main(int argc, const char **argv)
 	 */
 	sanitize_stdfds();
 
+	init_the_repository();
+
 	git_setup_gettext();
 
 	attr_start();
diff --git a/object-store.h b/object-store.h
index cf35760ceb..c3253ebc59 100644
--- a/object-store.h
+++ b/object-store.h
@@ -8,8 +8,8 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
 
+void raw_object_store_init(struct raw_object_store *o);
 void raw_object_store_clear(struct raw_object_store *o);
 
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 11d904c033..8a4d01dd5f 100644
--- a/object.c
+++ b/object.c
@@ -446,6 +446,11 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+void raw_object_store_init(struct raw_object_store *o)
+{
+	memset(o, 0, sizeof(*o));
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
diff --git a/repository.c b/repository.c
index 2255ff657e..0ebcca8539 100644
--- a/repository.c
+++ b/repository.c
@@ -5,16 +5,22 @@
 #include "submodule-config.h"
 
 /* The main repository */
-static struct repository the_repo = {
-	NULL, NULL,
-	RAW_OBJECT_STORE_INIT,
-	NULL, NULL, NULL,
-	NULL, NULL, NULL,
-	&the_index,
-	&hash_algos[GIT_HASH_SHA1],
-	0, 0
-};
-struct repository *the_repository = &the_repo;
+static struct repository the_repo;
+struct repository *the_repository;
+
+static void repo_pre_init(struct repository *repo)
+{
+	memset(repo, 0, sizeof(*repo));
+	raw_object_store_init(&repo->objects);
+}
+
+void init_the_repository(void)
+{
+	the_repository = &the_repo;
+	repo_pre_init(the_repository);
+	the_repository->index = &the_index;
+	repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+}
 
 static char *git_path_from_env(const char *envvar, const char *git_dir,
 			       const char *path, int fromenv)
@@ -138,7 +144,8 @@ static int read_and_verify_repository_format(struct repository_format *format,
 int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 {
 	struct repository_format format;
-	memset(repo, 0, sizeof(*repo));
+
+	repo_pre_init(repo);
 
 	repo->ignore_env = 1;
 
diff --git a/repository.h b/repository.h
index 1f8bc7a7cf..da6a8f9af9 100644
--- a/repository.h
+++ b/repository.h
@@ -93,6 +93,7 @@ extern void repo_set_gitdir(struct repository *repo, const char *path);
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
 extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
+extern void init_the_repository(void);
 extern int repo_submodule_init(struct repository *submodule,
 			       struct repository *superproject,
 			       const char *path);
-- 8< --

Re: [PATCHv4 01/27] repository: introduce raw object store field

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..a13ab981aa 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>  
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -32,6 +33,8 @@ int main(int argc, const char **argv)
>  	 */
>  	sanitize_stdfds();
>  
> +	init_the_repository();
> +
>  	git_setup_gettext();
> ...
> +void init_the_repository(void)
> +{
> +	the_repository = &the_repo;
> +	repo_pre_init(the_repository);
> +	the_repository->index = &the_index;
> +	repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +}

I see what you did here, and I like it.

Re: [PATCHv4 01/27] repository: introduce raw object store field

[Brandon Williams]

On 02/26, Junio C Hamano wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
> 
> > diff --git a/common-main.c b/common-main.c
> > index 6a689007e7..a13ab981aa 100644
> > --- a/common-main.c
> > +++ b/common-main.c
> > @@ -1,6 +1,7 @@
> >  #include "cache.h"
> >  #include "exec_cmd.h"
> >  #include "attr.h"
> > +#include "repository.h"
> >  
> >  /*
> >   * Many parts of Git have subprograms communicate via pipe, expect the
> > @@ -32,6 +33,8 @@ int main(int argc, const char **argv)
> >  	 */
> >  	sanitize_stdfds();
> >  
> > +	init_the_repository();
> > +
> >  	git_setup_gettext();
> > ...
> > +void init_the_repository(void)
> > +{
> > +	the_repository = &the_repo;
> > +	repo_pre_init(the_repository);
> > +	the_repository->index = &the_index;
> > +	repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> > +}
> 
> I see what you did here, and I like it.

I thought this would be a good idea to do eventually but back when I
first introduced struct repository there wasn't enough to justify it
till now.  This definitely makes it much easier to read the
initialization and I prefer this over the initializer.  Thanks for
working on this :)

-- 
Brandon Williams

Re: [PATCHv4 01/27] repository: introduce raw object store field

[Stefan Beller]

On Mon, Feb 26, 2018 at 1:30 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Fri, Feb 23, 2018 at 04:47:28PM -0800, Stefan Beller wrote:
>>  /* The main repository */
>>  static struct repository the_repo = {
>> -     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
>> +     NULL, NULL,
>> +     RAW_OBJECT_STORE_INIT,
>> +     NULL, NULL, NULL,
>> +     NULL, NULL, NULL,
>> +     &the_index,
>> +     &hash_algos[GIT_HASH_SHA1],
>> +     0, 0
>>  };
>>  struct repository *the_repository = &the_repo;
>
> I wonder if we should do something like this. It makes the_repo
> initialization easier to read and it helps unify the init code with
> submodule.
>
> No don't reroll. If you think it's a good idea, you can do something
> like this in the next series instead.
>
> -- 8< --
> diff --git a/check-racy.c b/check-racy.c
> index 24b6542352..47cbb4eb6d 100644
> --- a/check-racy.c
> +++ b/check-racy.c

totally offtopic: Do we want to move this file into t/helper?

> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -32,6 +33,8 @@ int main(int argc, const char **argv)
>          */
>         sanitize_stdfds();
>
> +       init_the_repository();
> +

So this is the real deal.

> -#define RAW_OBJECT_STORE_INIT { NULL }
>
> +void raw_object_store_init(struct raw_object_store *o);
>  void raw_object_store_clear(struct raw_object_store *o);
>
>  #endif /* OBJECT_STORE_H */
> diff --git a/object.c b/object.c
> index 11d904c033..8a4d01dd5f 100644
> --- a/object.c
> +++ b/object.c
> @@ -446,6 +446,11 @@ void clear_commit_marks_all(unsigned int flags)
>         }
>  }
>
> +void raw_object_store_init(struct raw_object_store *o)
> +{
> +       memset(o, 0, sizeof(*o));
> +}

We'll have to adapt this for the list that we use for packed_git_mru,
but that should be no problem.

> +
> +static void repo_pre_init(struct repository *repo)
> +{
> +       memset(repo, 0, sizeof(*repo));
> +       raw_object_store_init(&repo->objects);
> +}
> +
> +void init_the_repository(void)
> +{
> +       the_repository = &the_repo;
> +       repo_pre_init(the_repository);
> +       the_repository->index = &the_index;
> +       repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +}
>
>  static char *git_path_from_env(const char *envvar, const char *git_dir,
>                                const char *path, int fromenv)
> @@ -138,7 +144,8 @@ static int read_and_verify_repository_format(struct repository_format *format,
>  int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
>  {
>         struct repository_format format;
> -       memset(repo, 0, sizeof(*repo));
> +
> +       repo_pre_init(repo);
>
>         repo->ignore_env = 1;
>

I like the approach. Though as you say, let's have it in the next series.

Thanks for the thoughts and guidance,
Stefan

Re: [PATCHv4 01/27] repository: introduce raw object store field

[Duy Nguyen]

On Tue, Feb 27, 2018 at 2:28 AM, Stefan Beller <sbeller@google.com> wrote:
> On Mon, Feb 26, 2018 at 1:30 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Fri, Feb 23, 2018 at 04:47:28PM -0800, Stefan Beller wrote:
>>>  /* The main repository */
>>>  static struct repository the_repo = {
>>> -     NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
>>> +     NULL, NULL,
>>> +     RAW_OBJECT_STORE_INIT,
>>> +     NULL, NULL, NULL,
>>> +     NULL, NULL, NULL,
>>> +     &the_index,
>>> +     &hash_algos[GIT_HASH_SHA1],
>>> +     0, 0
>>>  };
>>>  struct repository *the_repository = &the_repo;
>>
>> I wonder if we should do something like this. It makes the_repo
>> initialization easier to read and it helps unify the init code with
>> submodule.
>>
>> No don't reroll. If you think it's a good idea, you can do something
>> like this in the next series instead.
>>
>> -- 8< --
>> diff --git a/check-racy.c b/check-racy.c
>> index 24b6542352..47cbb4eb6d 100644
>> --- a/check-racy.c
>> +++ b/check-racy.c
>
> totally offtopic: Do we want to move this file into t/helper?

No wonder both Jeff and I missed this program (he didn't convert it to
use cmd_main, and I didn't move it to t/helper). This git-check-racy
is added in 42f774063d (Add check program "git-check-racy" -
2006-08-15) and is not part of the default build. You need to manually
update Makefile first to build it.

Right now it's broken (multiple definition of 'main'). If you add
init_the_repository() or something similar, just leave this file
untouched. Maybe I'll fix it later, separately. Or perhaps I'll move
this functionality to git-update-index if this is still worth keeping.
-- 
Duy

[PATCHv4 02/27] object-store: migrate alternates struct and functions from cache.h

[Stefan Beller]

Migrate the struct alternate_object_database and all its related
functions to the object store as these functions are easier found in
that header. The migration is just a verbatim copy, no need to
include the object store header at any C file, because cache.h includes
repository.h which in turn includes the object-store.h

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h        | 51 --------------------------------------------------
 object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 51 deletions(-)

diff --git a/cache.h b/cache.h
index 21fbcc24149..99eb2bdb893 100644
--- a/cache.h
+++ b/cache.h
@@ -1560,57 +1560,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
-	struct alternate_object_database *next;
-
-	/* see alt_scratch_buf() */
-	struct strbuf scratch;
-	size_t base_len;
-
-	/*
-	 * Used to store the results of readdir(3) calls when searching
-	 * for unique abbreviated hashes.  This cache is never
-	 * invalidated, thus it's racy and not necessarily accurate.
-	 * That's fine for its purpose; don't use it for tasks requiring
-	 * greater accuracy!
-	 */
-	char loose_objects_subdir_seen[256];
-	struct oid_array loose_objects_cache;
-
-	char path[FLEX_ARRAY];
-} *alt_odb_list;
-extern void prepare_alt_odb(void);
-extern char *compute_alternate_path(const char *path, struct strbuf *err);
-typedef int alt_odb_fn(struct alternate_object_database *, void *);
-extern int foreach_alt_odb(alt_odb_fn, void*);
-
-/*
- * Allocate a "struct alternate_object_database" but do _not_ actually
- * add it to the list of alternates.
- */
-struct alternate_object_database *alloc_alt_odb(const char *dir);
-
-/*
- * Add the directory to the on-disk alternates file; the new entry will also
- * take effect in the current process.
- */
-extern void add_to_alternates_file(const char *dir);
-
-/*
- * Add the directory to the in-memory list of alternates (along with any
- * recursive alternates it points to), but do not modify the on-disk alternates
- * file.
- */
-extern void add_to_alternates_memory(const char *dir);
-
-/*
- * Returns a scratch strbuf pre-filled with the alternate object directory,
- * including a trailing slash, which can be used to access paths in the
- * alternate. Always use this over direct access to alt->scratch, as it
- * cleans up any previous use of the scratch buffer.
- */
-extern struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
-
 struct pack_window {
 	struct pack_window *next;
 	unsigned char *base;
diff --git a/object-store.h b/object-store.h
index cf35760ceb6..5678aa1136c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,57 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+extern struct alternate_object_database {
+	struct alternate_object_database *next;
+
+	/* see alt_scratch_buf() */
+	struct strbuf scratch;
+	size_t base_len;
+
+	/*
+	 * Used to store the results of readdir(3) calls when searching
+	 * for unique abbreviated hashes.  This cache is never
+	 * invalidated, thus it's racy and not necessarily accurate.
+	 * That's fine for its purpose; don't use it for tasks requiring
+	 * greater accuracy!
+	 */
+	char loose_objects_subdir_seen[256];
+	struct oid_array loose_objects_cache;
+
+	char path[FLEX_ARRAY];
+} *alt_odb_list;
+void prepare_alt_odb(void);
+char *compute_alternate_path(const char *path, struct strbuf *err);
+typedef int alt_odb_fn(struct alternate_object_database *, void *);
+int foreach_alt_odb(alt_odb_fn, void*);
+
+/*
+ * Allocate a "struct alternate_object_database" but do _not_ actually
+ * add it to the list of alternates.
+ */
+struct alternate_object_database *alloc_alt_odb(const char *dir);
+
+/*
+ * Add the directory to the on-disk alternates file; the new entry will also
+ * take effect in the current process.
+ */
+void add_to_alternates_file(const char *dir);
+
+/*
+ * Add the directory to the in-memory list of alternates (along with any
+ * recursive alternates it points to), but do not modify the on-disk alternates
+ * file.
+ */
+void add_to_alternates_memory(const char *dir);
+
+/*
+ * Returns a scratch strbuf pre-filled with the alternate object directory,
+ * including a trailing slash, which can be used to access paths in the
+ * alternate. Always use this over direct access to alt->scratch, as it
+ * cleans up any previous use of the scratch buffer.
+ */
+struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 03/27] object-store: move alt_odb_list and alt_odb_tail to object store

[Stefan Beller]

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 object-store.h |  9 ++++++---
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 5 files changed, 25 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 9981db22637..78edcb75c45 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -722,7 +723,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+		     alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/object-store.h b/object-store.h
index 5678aa1136c..e78eea1dde3 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,7 +1,7 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 void prepare_alt_odb(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
@@ -58,8 +58,11 @@ struct raw_object_store {
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d17..216ea836ee3 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 826d7a0ae37..ac25146076e 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -340,9 +341,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -362,7 +360,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -422,8 +420,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -557,7 +555,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -655,7 +653,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -667,12 +665,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -716,7 +715,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -876,7 +875,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -906,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dde..957ce25680b 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 04/27] object-store: free alt_odb_list

[Stefan Beller]

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index a2acdee1405..920dc4463fa 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	FREE_AND_NULL(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 05/27] object-store: move packed_git and packed_git_mru to object store

[Stefan Beller]

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Apply the following semantic patch to adjust callers:
    @@ @@
    - packed_git
    + the_repository->objects.packed_git

    @@ @@
    - packed_git_mru
    + the_repository->objects.packed_git_mru

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h where needed.

 5. As the packfiles are now owned by an objectstore, which is ephemeral
    unlike globals, we introduce memory leaks. So address them in
    raw_object_store_clear(). Defer freeing packed_git to the next
    patch due to the size of this patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  5 +++--
 builtin/fsck.c           |  6 ++++--
 builtin/gc.c             |  3 ++-
 builtin/pack-objects.c   | 20 +++++++++++---------
 builtin/pack-redundant.c |  5 +++--
 cache.h                  | 29 -----------------------------
 fast-import.c            |  7 +++++--
 http-backend.c           |  5 +++--
 object-store.h           | 31 ++++++++++++++++++++++++++++++-
 object.c                 |  6 ++++++
 pack-bitmap.c            |  3 ++-
 packfile.c               | 39 ++++++++++++++++++++-------------------
 repository.c             |  4 +++-
 server-info.c            |  5 +++--
 sha1_name.c              |  5 +++--
 15 files changed, 98 insertions(+), 75 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c83..6aab8d08fcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,7 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +121,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 78edcb75c45..c3a339193c8 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -735,7 +735,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -743,7 +744,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0b..59bf5c2ea0b 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -173,7 +174,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 5c674b2843c..a64005760be 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "attr.h"
 #include "object.h"
@@ -1025,8 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru,
+					  &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa81363222..a196fae79d4 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,7 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/cache.h b/cache.h
index 99eb2bdb893..63889acb591 100644
--- a/cache.h
+++ b/cache.h
@@ -1569,35 +1569,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index 0f818cd9328..3d7002d1cd6 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,6 +154,7 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
 #include "object.h"
@@ -1110,7 +1111,8 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash,
+				  the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1307,7 +1309,8 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash,
+				  the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a3..85b6a97923f 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
 #include "object.h"
@@ -518,13 +519,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/object-store.h b/object-store.h
index e78eea1dde3..4f768465a14 100644
--- a/object-store.h
+++ b/object-store.h
@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
  */
 struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
 
+struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -59,10 +83,15 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/* A most-recently-used ordered version of the packed_git list. */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index 920dc4463fa..367441efa94 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	INIT_LIST_HEAD(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f5..bbede32589a 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -10,6 +10,7 @@
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +336,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/packfile.c b/packfile.c
index 216ea836ee3..59210deaaf7 100644
--- a/packfile.c
+++ b/packfile.c
@@ -45,8 +45,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +244,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +314,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +382,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +684,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +767,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +819,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +868,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1014,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1845,14 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru,
+				  &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/repository.c b/repository.c
index fb65e8072d5..a069b1b6404 100644
--- a/repository.c
+++ b/repository.c
@@ -7,7 +7,7 @@
 /* The main repository */
 static struct repository the_repo = {
 	NULL, NULL,
-	RAW_OBJECT_STORE_INIT,
+	RAW_OBJECT_STORE_INIT(the_repo.objects),
 	NULL, NULL, NULL,
 	NULL, NULL, NULL,
 	&the_index,
@@ -143,6 +143,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d4..c369ac27cbc 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
 #include "object.h"
 #include "commit.h"
@@ -200,7 +201,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +211,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680b..016c883b5c7 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 06/27] object-store: close all packs upon clearing the object store

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 7 +++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 6661edc162b..fc1d9b43c51 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593f..13cfaa6488a 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2ee..4d72efca781 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 92ba99a1a5e..0af0c53a632 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 75e7f18acef..8e25aae54c8 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index 367441efa94..a7c238339bd 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -468,8 +469,6 @@ void raw_object_store_clear(struct raw_object_store *o)
 	o->alt_odb_tail = NULL;
 
 	INIT_LIST_HEAD(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
+	o->packed_git = NULL;
 }
diff --git a/packfile.c b/packfile.c
index 59210deaaf7..65d9a4f6c61 100644
--- a/packfile.c
+++ b/packfile.c
@@ -310,11 +310,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d67..6a2c57045ca 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 07/27] pack: move prepare_packed_git_run_once to object store

[Stefan Beller]

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 4f768465a14..bd6441e525f 100644
--- a/object-store.h
+++ b/object-store.h
@@ -89,9 +89,15 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 65d9a4f6c61..3ee349ab1bd 100644
--- a/packfile.c
+++ b/packfile.c
@@ -883,12 +883,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -896,13 +895,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 08/27] pack: move approximate object count to object store

[Stefan Beller]

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index bd6441e525f..31ca4d3f340 100644
--- a/object-store.h
+++ b/object-store.h
@@ -90,6 +90,14 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -97,7 +105,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 3ee349ab1bd..8d7b1dd5f9c 100644
--- a/packfile.c
+++ b/packfile.c
@@ -802,8 +802,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -813,8 +811,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -824,8 +822,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -900,7 +899,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 09/27] sha1_file: add raw_object_store argument to alt_odb_usable

[Stefan Beller]

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index ac25146076e..8e6f213f9d9 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -344,7 +344,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -360,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -412,7 +414,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 10/27] sha1_file: add repository argument to link_alt_odb_entry

[Stefan Beller]

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8e6f213f9d9..e5fe0aa04ef 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -388,8 +388,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -486,7 +487,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 11/27] sha1_file: add repository argument to read_info_alternates

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index e5fe0aa04ef..f93d3b95b54 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -387,7 +387,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -428,7 +430,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -494,7 +496,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -678,7 +681,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 12/27] sha1_file: add repository argument to link_alt_odb_entries

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index f93d3b95b54..8fb56ca0362 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -465,8 +465,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -509,7 +513,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -563,7 +567,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -576,7 +581,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -679,7 +685,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 13/27] sha1_file: add repository argument to prepare_alt_odb

[Stefan Beller]

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 object-store.h |  3 ++-
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  2 +-
 5 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index c3a339193c8..cdcf2fa926c 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -722,7 +722,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 		     alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/object-store.h b/object-store.h
index 31ca4d3f340..95f328482aa 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,7 +20,8 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-void prepare_alt_odb(void);
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+void prepare_alt_odb_the_repository(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/packfile.c b/packfile.c
index 8d7b1dd5f9c..1e02fe3b0bb 100644
--- a/packfile.c
+++ b/packfile.c
@@ -889,7 +889,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index 8fb56ca0362..381eaf8221d 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -579,7 +580,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -665,7 +666,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -674,7 +675,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -728,7 +729,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -887,7 +888,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -918,7 +919,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c7..627e93f1447 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -353,7 +353,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories

[Stefan Beller]

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  4 ++++
 sha1_file.c    | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/object-store.h b/object-store.h
index 95f328482aa..08cd48ade11 100644
--- a/object-store.h
+++ b/object-store.h
@@ -18,6 +18,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 #define prepare_alt_odb(r) prepare_alt_odb_##r()
diff --git a/sha1_file.c b/sha1_file.c
index 381eaf8221d..ab42d430b8a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -388,11 +388,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -418,7 +417,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -426,12 +425,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -466,12 +465,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -485,7 +480,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -494,15 +489,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -514,7 +510,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 12 +++++-------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/object-store.h b/object-store.h
index 08cd48ade11..24b9a750aef 100644
--- a/object-store.h
+++ b/object-store.h
@@ -24,8 +24,7 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/sha1_file.c b/sha1_file.c
index ab42d430b8a..03b9bbe8bb7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -671,21 +671,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
 	const char *alt;
 
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 16/27] sha1_file: add repository argument to sha1_file_name

[Stefan Beller]

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  7 +++++++
 sha1_file.c    | 10 +++++-----
 5 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/cache.h b/cache.h
index 63889acb591..94d85a5c692 100644
--- a/cache.h
+++ b/cache.h
@@ -936,12 +936,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af826..2c33969123a 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -545,7 +546,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(the_repository, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index 31755023a4f..df9dbea59c5 100644
--- a/http.c
+++ b/http.c
@@ -2246,7 +2246,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2395,8 +2395,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(the_repository, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 24b9a750aef..c0d1292cfc2 100644
--- a/object-store.h
+++ b/object-store.h
@@ -113,4 +113,11 @@ struct raw_object_store {
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 03b9bbe8bb7..8ef2f856035 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -321,7 +321,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
 {
 	strbuf_addstr(buf, get_object_directory());
 	strbuf_addch(buf, '/');
@@ -715,7 +715,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -876,7 +876,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -905,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1591,7 +1591,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, oid->hash);
+	sha1_file_name(the_repository, &filename, oid->hash);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 17/27] sha1_file: add repository argument to stat_sha1_file

[Stefan Beller]

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8ef2f856035..b7e6e4a9bfc 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -869,8 +869,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
@@ -1176,7 +1177,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1390,7 +1391,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 18/27] sha1_file: add repository argument to open_sha1_file

[Stefan Beller]

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index b7e6e4a9bfc..7e471e0cf4b 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -898,7 +898,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -941,7 +943,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 19/27] sha1_file: add repository argument to map_sha1_file_1

[Stefan Beller]

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 7e471e0cf4b..d4af8f56814 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -933,9 +933,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -964,7 +965,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2200,7 +2201,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 20/27] sha1_file: add repository argument to map_sha1_file

[Stefan Beller]

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 3 +++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index 94d85a5c692..57173991839 100644
--- a/cache.h
+++ b/cache.h
@@ -1226,7 +1226,6 @@ extern int force_object_loose(const struct object_id *oid, time_t mtime);
 
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index c0d1292cfc2..dbd589e0083 100644
--- a/object-store.h
+++ b/object-store.h
@@ -120,4 +120,7 @@ void raw_object_store_clear(struct raw_object_store *o);
 #define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
 void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
 
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index d4af8f56814..6ec43e745d7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -963,7 +963,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1187,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd89..22d27df55eb 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 21/27] sha1_file: add repository argument to sha1_loose_object_info

[Stefan Beller]

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6ec43e745d7..831e2da3572 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1154,9 +1154,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index dbd589e0083..8b38330d256 100644
--- a/object-store.h
+++ b/object-store.h
@@ -117,8 +117,7 @@ void raw_object_store_clear(struct raw_object_store *o);
  * Put in `buf` the name of the file in the local object database that
  * would be used to store a loose object with the specified sha1.
  */
-#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
 #define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
 void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
diff --git a/sha1_file.c b/sha1_file.c
index 831e2da3572..59993a0878d 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -321,9 +321,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, get_object_directory());
+	strbuf_addstr(buf, r->objects.objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 23/27] sha1_file: allow stat_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 59993a0878d..952deb02195 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -869,23 +869,22 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
-static int stat_sha1_file_the_repository(const unsigned char *sha1,
-					 struct stat *st, const char **path)
+static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(the_repository);
+	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 952deb02195..73bbe3a3e88 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -897,9 +897,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
-static int open_sha1_file_the_repository(const unsigned char *sha1,
-					 const char **path)
+static int open_sha1_file(struct repository *r,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -907,7 +906,7 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -915,8 +914,8 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 25/27] sha1_file: allow map_sha1_file_1 to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 73bbe3a3e88..e32f1da6b69 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -931,10 +931,8 @@ static int open_sha1_file(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
-static void *map_sha1_file_1_the_repository(const char *path,
-					    const unsigned char *sha1,
-					    unsigned long *size)
+static void *map_sha1_file_1(struct repository *r, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -942,7 +940,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(the_repository, sha1, &path);
+		fd = open_sha1_file(r, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories

[Stefan Beller]

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 5 +++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index 8b38330d256..afe2f93459b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -119,7 +119,6 @@ void raw_object_store_clear(struct raw_object_store *o);
  */
 void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
-#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index e32f1da6b69..902cba42106 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -959,9 +959,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,
 	return map;
 }
 
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct repository *r,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(the_repository, NULL, sha1, size);
+	return map_sha1_file_1(r, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
-- 
2.16.1.291.g4437f3f132-goog

[PATCHv4 27/27] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories

[Stefan Beller]

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 902cba42106..dfc8deec38e 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1151,10 +1151,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
-static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
-						 struct object_info *oi,
-						 int flags)
+static int sha1_loose_object_info(struct repository *r,
+				  const unsigned char *sha1,
+				  struct object_info *oi, int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1178,14 +1177,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
+		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(the_repository, sha1, &mapsize);
+	map = map_sha1_file(r, sha1, &mapsize);
 	if (!map)
 		return -1;
 
-- 
2.16.1.291.g4437f3f132-goog

Re: [PATCHv4 00/27] Moving global state into the repository object (part 1)

[Duy Nguyen]

I notice that there are a few global state (or configuration rather)
left after this: packed_git_window_size, packed_git_limit and
delta_base_cache_limit. These can be changed by $GIT_DIR/config, but
since it's still global, a submodule repository will use the same
settings of its super repository. If $SUBMODULE/config changes any of
these, they are ignored.

The natural thing to do is move these to raw_object_store too (and
repo_submodule_init needs to check submodule's config file). But one
may argue these should be per-process instead of per-repo though. I
don't know. But I thought I should mention this.
-- 
Duy

Re: [PATCHv4 00/27] Moving global state into the repository object (part 1)

[Stefan Beller]

On Sat, Feb 24, 2018 at 7:00 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> I notice that there are a few global state (or configuration rather)
> left after this: packed_git_window_size, packed_git_limit and
> delta_base_cache_limit. These can be changed by $GIT_DIR/config, but
> since it's still global, a submodule repository will use the same
> settings of its super repository. If $SUBMODULE/config changes any of
> these, they are ignored.

That sounds all packing related, which I plan on working on next.

> The natural thing to do is move these to raw_object_store too (and
> repo_submodule_init needs to check submodule's config file). But one
> may argue these should be per-process instead of per-repo though. I
> don't know. But I thought I should mention this.

For now a process and a repository is the same as git-gc or git-repack
doesn't know about the --recurse-submodules flag, yet.
I wonder if we ever want to teach those commands the submodule
recursion, because of the issue you bring up, which settings do we apply
for a submodule? Traditionally we'd just have the command line override
the configuration, which I don't know if it is a good idea for these settings.

Thanks,
Stefan

Re: [PATCHv4 00/27] Moving global state into the repository object (part 1)

[Duy Nguyen]

On Tue, Feb 27, 2018 at 3:50 AM, Stefan Beller <sbeller@google.com> wrote:
>> The natural thing to do is move these to raw_object_store too (and
>> repo_submodule_init needs to check submodule's config file). But one
>> may argue these should be per-process instead of per-repo though. I
>> don't know. But I thought I should mention this.
>
> For now a process and a repository is the same as git-gc or git-repack

I think you're thinking about the pack writing part, but these are
configuration for pack reading (aka "object store"). If you read a
blob from a submodule (e.g. git-grep), you'll use these configurations
at some point.

There are of course more configuration for pack writing (like
zlib_compression_level) which I deliberately avoided to mention
because I don't even know where they belong.

> doesn't know about the --recurse-submodules flag, yet.
> I wonder if we ever want to teach those commands the submodule
> recursion, because of the issue you bring up, which settings do we apply
> for a submodule? Traditionally we'd just have the command line override
> the configuration, which I don't know if it is a good idea for these settings.
-- 
Duy

Re: [PATCHv4 00/27] Moving global state into the repository object (part 1)

[Jonathan Tan]

On Fri, 23 Feb 2018 16:47:27 -0800
Stefan Beller <sbeller@google.com> wrote:

> v4:
> * addressed feedback from the single patches (mostly nits)
> * rebased on latest master

The patches I looked at previously (patches 7, 15, 19, 22, and 24) look
good to me.

[PATCH 00/44] reroll nd/remove-ignore-env.. sb/object-store and sb/packfiles..

[Nguyễn Thái Ngọc Duy]

On Sat, Mar 3, 2018 at 9:54 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Thu, Mar 1, 2018 at 2:09 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Stefan Beller <sbeller@google.com> writes:
>>
>>> On Wed, Feb 28, 2018 at 9:59 AM, Junio C Hamano <gitster@pobox.com> wrote:
>>>> Duy Nguyen <pclouds@gmail.com> writes:
>>>>
>>>>> Looking at the full-series diff though, it makes me wonder if we
>>>>> should keep prepare_packed_git() private (i.e. how we initialize the
>>>>> object store, packfile included, is a private matter). How about
>>>>> something like this on top?
>>>>
>>>> Yup, that looks cleaner.
>>>
>>> I agree that it looks cleaner. So we plan on just putting
>>> it on top of that series?
>>
>> We tend to avoid "oops, that was wrong and here is a band aid on
>> top" for things that are still mushy, so it would be preferrable to
>> get it fixed inline, especially if there are more changes to the
>> other parts of the series coming.
>
> I agree with this. Stefan, if you're getting bored with rerolling
> refactor patches, I can update this series and send out v2.

Since Stefan is traveling, I take this opportunity to reroll it.
Unfortunately, I think the fix should go in 46cd557bd9 (object-store:
move packed_git and packed_git_mru to object store - 2018-02-23) where
we start removing the global "packed_git". But that's in
sb/object-store, so.. I'm rerolling all three

01/44 - 05/44: nd/remove-ignore-env-field

  This series is moved up top. After this the patch that touch
  alternate-db in sha1_file.c looks natural because no env is involved
  anymore

  I also take this opportunity to introduce a new patch 01/44 to avoid
  struct initialization that makes it hard to read and update. Later
  patches are also simplified thanks to this.

06/44 - 32/44: sb/object-store

  06/44 is updated to introduce raw_object_store_init() instead of
  RAW_OBJECT_STORE_INIT macro. This function is now used to initialize
  both main repo and submodule ones.

  10/44 (moving "packed_git") also introduces two new access wrapper
  get_packed_git() and get_packed_git_mru()

33/44 - 44/44: sb/packfiles-in-repository

  The only thing new here is 44/44 which makes prepare_packed_git()
  internal. get_packed_git() and get_packed_git_mru() introduced
  earlier will call prepare_packed_git() automatically.

The whole thing is also available at

https://github.com/pclouds/git/tree/ignore-env-object-store-packfiles

And interdiff of all three, compared to what is currently in 'pu'.
Looks pretty good in my opinon:

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index d480301763..ee6ae35244 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -121,9 +121,8 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!the_repository->objects.packed_git)
-			prepare_packed_git(the_repository);
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+
+		for (p = get_packed_git(the_repository); p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 030c7fb7a0..9911c52bc8 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -726,10 +726,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			uint32_t total = 0, count = 0;
 			struct progress *progress = NULL;
 
-			prepare_packed_git(the_repository);
-
 			if (show_progress) {
-				for (p = the_repository->objects.packed_git; p;
+				for (p = get_packed_git(the_repository); p;
 				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
@@ -738,7 +736,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = the_repository->objects.packed_git; p;
+			for (p = get_packed_git(the_repository); p;
 			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
diff --git a/builtin/gc.c b/builtin/gc.c
index 80d19c54d5..be63bec09c 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -173,8 +173,7 @@ static int too_many_packs(void)
 	if (gc_auto_pack_limit <= 0)
 		return 0;
 
-	prepare_packed_git(the_repository);
-	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (cnt = 0, p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 142ec223cb..435f091a69 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1026,7 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-	list_for_each(pos, &the_repository->objects.packed_git_mru) {
+	list_for_each(pos, get_packed_git_mru(the_repository)) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1045,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
 				list_move(&p->mru,
-					  &the_repository->objects.packed_git_mru);
+					  get_packed_git_mru(the_repository));
 			if (want != -1)
 				return want;
 		}
@@ -2674,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2738,7 +2738,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	struct packed_git *p;
 
 	p = (last_found != (void *)1) ? last_found :
-					the_repository->objects.packed_git;
+					get_packed_git(the_repository);
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2747,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = the_repository->objects.packed_git;
+			p = get_packed_git(the_repository);
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2783,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3151,10 +3151,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	if (progress && all_progress_implied)
 		progress = 2;
 
-	prepare_packed_git(the_repository);
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next)
+		for (p = get_packed_git(the_repository); p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3167,7 +3166,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = get_packed_git(the_repository); p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index 3b02f94248..02b5f0becc 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -572,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -587,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
 	while (p) {
 		add_pack(p);
@@ -630,8 +630,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)
 			break;
 	}
 
-	prepare_packed_git(the_repository);
-
 	if (load_all_packs)
 		load_all();
 	else
diff --git a/common-main.c b/common-main.c
index 6a689007e7..7d716d5a54 100644
--- a/common-main.c
+++ b/common-main.c
@@ -34,6 +34,8 @@ int main(int argc, const char **argv)
 
 	git_setup_gettext();
 
+	initialize_the_repository();
+
 	attr_start();
 
 	git_extract_argv0_path(argv[0]);
diff --git a/environment.c b/environment.c
index b2128c1188..c05705e384 100644
--- a/environment.c
+++ b/environment.c
@@ -13,8 +13,8 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
-#include "object-store.h"
 #include "argv-array.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
diff --git a/fast-import.c b/fast-import.c
index d2032e4ade..2298bfcdfd 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1112,7 +1112,7 @@ static int store_object(
 		duplicate_count_by_type[type]++;
 		return 1;
 	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+				  get_packed_git(the_repository))) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1308,7 +1308,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		truncate_pack(&checkpoint);
 
 	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+				  get_packed_git(the_repository))) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -3472,7 +3472,6 @@ int cmd_main(int argc, const char **argv)
 		rc_free[i].next = &rc_free[i + 1];
 	rc_free[cmd_save - 1].next = NULL;
 
-	prepare_packed_git(the_repository);
 	start_packfile();
 	set_die_routine(die_nicely);
 	set_checkpoint_signal();
diff --git a/http-backend.c b/http-backend.c
index defa6ba350..22d2e1668e 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -518,14 +518,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 	size_t cnt = 0;
 
 	select_getanyfile(hdr);
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/object-store.h b/object-store.h
index 9b1303549b..521f5a1755 100644
--- a/object-store.h
+++ b/object-store.h
@@ -90,9 +90,9 @@ struct raw_object_store {
 	/* Path to extra alternate object database if not NULL */
 	char *alternate_db;
 
-	struct packed_git *packed_git;
+	struct packed_git *packed_git; /* private */
 	/* A most-recently-used ordered version of the packed_git list. */
-	struct list_head packed_git_mru;
+	struct list_head packed_git_mru; /* private */
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
@@ -112,8 +112,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
-
+void raw_object_store_init(struct raw_object_store *o);
 void raw_object_store_clear(struct raw_object_store *o);
 
 /*
diff --git a/object.c b/object.c
index 5317cfc390..60ca76b285 100644
--- a/object.c
+++ b/object.c
@@ -447,6 +447,12 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+void raw_object_store_init(struct raw_object_store *o)
+{
+	memset(o, 0, sizeof(*o));
+	INIT_LIST_HEAD(&o->packed_git_mru);
+}
+
 static void free_alt_odb(struct alternate_object_database *alt)
 {
 	strbuf_release(&alt->scratch);
diff --git a/pack-bitmap.c b/pack-bitmap.c
index bcc04bc45e..2a007b5539 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -335,8 +335,7 @@ static int open_pack_bitmap(void)
 
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/packfile.c b/packfile.c
index 1a045157d0..773cd99a13 100644
--- a/packfile.c
+++ b/packfile.c
@@ -802,6 +802,7 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local
 	strbuf_release(&path);
 }
 
+static void prepare_packed_git(struct repository *r);
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -882,7 +883,7 @@ static void prepare_packed_git_mru(struct repository *r)
 		list_add_tail(&p->mru, &r->objects.packed_git_mru);
 }
 
-void prepare_packed_git(struct repository *r)
+static void prepare_packed_git(struct repository *r)
 {
 	struct alternate_object_database *alt;
 
@@ -904,6 +905,18 @@ void reprepare_packed_git(struct repository *r)
 	prepare_packed_git(r);
 }
 
+struct packed_git *get_packed_git(struct repository *r)
+{
+	prepare_packed_git(r);
+	return r->objects.packed_git;
+}
+
+struct list_head *get_packed_git_mru(struct repository *r)
+{
+	prepare_packed_git(r);
+	return &r->objects.packed_git_mru;
+}
+
 unsigned long unpack_object_header_buffer(const unsigned char *buf,
 		unsigned long len, enum object_type *type, unsigned long *sizep)
 {
diff --git a/packfile.h b/packfile.h
index 6f7b9e91d6..efda10329c 100644
--- a/packfile.h
+++ b/packfile.h
@@ -34,10 +34,12 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
 #define PACKDIR_FILE_GARBAGE 4
 extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
-extern void prepare_packed_git(struct repository *r);
 extern void reprepare_packed_git(struct repository *r);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
+struct packed_git *get_packed_git(struct repository *r);
+struct list_head *get_packed_git_mru(struct repository *r);
+
 /*
  * Give a rough count of objects in the repository. This sacrifices accuracy
  * for speed.
diff --git a/repository.c b/repository.c
index 4f44384dde..34c0a7f180 100644
--- a/repository.c
+++ b/repository.c
@@ -5,16 +5,17 @@
 #include "submodule-config.h"
 
 /* The main repository */
-static struct repository the_repo = {
-	NULL, NULL,
-	RAW_OBJECT_STORE_INIT(the_repo.objects),
-	NULL, NULL, NULL,
-	NULL, NULL, NULL,
-	&the_index,
-	&hash_algos[GIT_HASH_SHA1],
-	0
-};
-struct repository *the_repository = &the_repo;
+static struct repository the_repo;
+struct repository *the_repository;
+
+void initialize_the_repository(void)
+{
+	the_repository = &the_repo;
+
+	the_repo.index = &the_index;
+	raw_object_store_init(&the_repo.objects);
+	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
+}
 
 static void expand_base_dir(char **out, const char *in,
 			    const char *base_dir, const char *def_in)
@@ -134,12 +135,14 @@ static int read_and_verify_repository_format(struct repository_format *format,
  * Initialize 'repo' based on the provided 'gitdir'.
  * Return 0 upon success and a non-zero value upon failure.
  */
-int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
+static int repo_init(struct repository *repo,
+		     const char *gitdir,
+		     const char *worktree)
 {
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
 
-	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+	raw_object_store_init(&repo->objects);
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
diff --git a/repository.h b/repository.h
index e05a77a099..6c383c05c6 100644
--- a/repository.h
+++ b/repository.h
@@ -93,7 +93,7 @@ extern void repo_set_gitdir(struct repository *repo,
 			    const struct set_gitdir_args *optional);
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
-extern int repo_init(struct repository *repo, const char *gitdir, const char *worktree);
+extern void initialize_the_repository(void);
 extern int repo_submodule_init(struct repository *submodule,
 			       struct repository *superproject,
 			       const char *path);
diff --git a/server-info.c b/server-info.c
index f5e4d1cc89..af737acd24 100644
--- a/server-info.c
+++ b/server-info.c
@@ -200,8 +200,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdir = get_object_directory();
 	objdirlen = strlen(objdir);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -211,7 +210,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (i = 0, p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 773322ac81..f25a0970ca 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -195,8 +195,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 {
 	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	for (p = get_packed_git(the_repository); p && !ds->ambiguous;
 	     p = p->next)
 		unique_in_pack(p, ds);
 }
@@ -566,8 +565,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 {
 	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = get_packed_git(the_repository); p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.435.g8f24da2e1a

[PATCH 00/27] sb/object-store updates

[Nguyễn Thái Ngọc Duy]

I think I have addressed all comments I've received so far. What I
decided not to do, I have responded individually. One comment I did
not respond nor do is the approximate thing, which could be done
later.

Interdiff is big due to the "objects." to "objects->" conversion
(blame Brandon for his suggestion, don't blame me :D)

diff --git a/builtin/am.c b/builtin/am.c
index 4762a702e3..47beddbe24 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs(&the_repository->objects);
+		close_all_packs(the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 13cfaa6488..7df5932b85 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -27,6 +27,7 @@
 #include "connected.h"
 #include "packfile.h"
 #include "list-objects-filter-options.h"
+#include "object-store.h"
 
 /*
  * Overall FIXMEs:
@@ -1217,7 +1218,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs(&the_repository->objects);
+		close_all_packs(the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 5c7c3c6ae3..b28ff00be2 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -12,6 +12,7 @@
 #include "parse-options.h"
 #include "quote.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static unsigned long garbage;
 static off_t size_garbage;
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 4d72efca78..a39e9d7b15 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs(&the_repository->objects);
+	close_all_packs(the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/fsck.c b/builtin/fsck.c
index b0abba6e04..3ef25fab97 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -17,6 +17,7 @@
 #include "streaming.h"
 #include "decorate.h"
 #include "packfile.h"
+#include "object-store.h"
 
 #define REACHABLE 0x0001
 #define SEEN      0x0002
@@ -714,11 +715,13 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		for_each_loose_object(mark_loose_for_connectivity, NULL, 0);
 		for_each_packed_object(mark_packed_for_connectivity, NULL, 0);
 	} else {
+		struct alternate_object_database *alt_odb_list;
+
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb(the_repository);
-		for (alt = the_repository->objects.alt_odb_list;
-		     alt; alt = alt->next)
+		alt_odb_list = the_repository->objects->alt_odb_list;
+		for (alt = alt_odb_list; alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/builtin/gc.c b/builtin/gc.c
index dd30067ac1..b00238cd5d 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -21,6 +21,7 @@
 #include "argv-array.h"
 #include "commit.h"
 #include "packfile.h"
+#include "object-store.h"
 
 #define FAILED_RUN "failed to run %s"
 
diff --git a/builtin/grep.c b/builtin/grep.c
index 0f0c195705..1e9cdbdf78 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -22,6 +22,7 @@
 #include "pathspec.h"
 #include "submodule.h"
 #include "submodule-config.h"
+#include "object-store.h"
 
 static char const * const grep_usage[] = {
 	N_("git grep [<options>] [-e] <pattern> [<rev>...] [[--] <path>...]"),
@@ -432,7 +433,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objects.objectdir);
+	add_to_alternates_memory(submodule.objects->objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 5ebd370c56..1d6bc87b76 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -13,6 +13,7 @@
 #include "streaming.h"
 #include "thread-utils.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static const char index_pack_usage[] =
 "git index-pack [-v] [-o <index-file>] [--keep | --keep=<msg>] [--verify] [--strict] (<pack-file> | --stdin [--fix-thin] [<pack-file>])";
diff --git a/builtin/merge.c b/builtin/merge.c
index 907ae44ab5..96d56cbdd2 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs(&the_repository->objects);
+			close_all_packs(the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 0e5fde1d6b..223f2d9fc0 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -29,6 +29,7 @@
 #include "argv-array.h"
 #include "list.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static const char *pack_usage[] = {
 	N_("git pack-objects --stdout [<options>...] [< <ref-list> | < <object-list>]"),
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index d6d8a44959..b5b007e706 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -9,6 +9,7 @@
 #include "builtin.h"
 #include "repository.h"
 #include "packfile.h"
+#include "object-store.h"
 
 #define BLKSIZE 512
 
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index ac478b7b99..1a298a6711 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2026,7 +2026,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs(&the_repository->objects);
+			close_all_packs(the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index b1daca995f..6d8e002be7 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -16,6 +16,7 @@
 #include "revision.h"
 #include "diffcore.h"
 #include "diff.h"
+#include "object-store.h"
 
 #define OPT_QUIET (1 << 0)
 #define OPT_CACHED (1 << 1)
diff --git a/environment.c b/environment.c
index c05705e384..93c9fbb0ba 100644
--- a/environment.c
+++ b/environment.c
@@ -271,9 +271,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objects.objectdir)
+	if (!the_repository->objects->objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objects.objectdir;
+	return the_repository->objects->objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/fast-import.c b/fast-import.c
index 0dba555478..b3492fce5c 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -169,6 +169,7 @@ Format of STDIN stream:
 #include "dir.h"
 #include "run-command.h"
 #include "packfile.h"
+#include "object-store.h"
 
 #define PACK_ID_BITS 16
 #define MAX_PACK_ID ((1<<PACK_ID_BITS)-1)
diff --git a/http-backend.c b/http-backend.c
index 22921d169a..64dde78c1b 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -11,6 +11,7 @@
 #include "url.h"
 #include "argv-array.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static const char content_type[] = "Content-Type";
 static const char content_length[] = "Content-Length";
diff --git a/http-push.c b/http-push.c
index 0913f8ab86..97fe22a705 100644
--- a/http-push.c
+++ b/http-push.c
@@ -12,6 +12,7 @@
 #include "sigchain.h"
 #include "argv-array.h"
 #include "packfile.h"
+#include "object-store.h"
 
 #ifdef EXPAT_NEEDS_XMLPARSE_H
 #include <xmlparse.h>
diff --git a/http-walker.c b/http-walker.c
index 2c33969123..75d55d42a9 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -6,6 +6,7 @@
 #include "list.h"
 #include "transport.h"
 #include "packfile.h"
+#include "object-store.h"
 
 struct alt_base {
 	char *base;
diff --git a/http.c b/http.c
index df9dbea59c..4d613d5f6b 100644
--- a/http.c
+++ b/http.c
@@ -14,6 +14,7 @@
 #include "packfile.h"
 #include "protocol.h"
 #include "string-list.h"
+#include "object-store.h"
 
 static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
 static int trace_curl_data = 1;
diff --git a/object-store.h b/object-store.h
index 521f5a1755..fef33f345f 100644
--- a/object-store.h
+++ b/object-store.h
@@ -90,13 +90,19 @@ struct raw_object_store {
 	/* Path to extra alternate object database if not NULL */
 	char *alternate_db;
 
-	struct packed_git *packed_git; /* private */
-	/* A most-recently-used ordered version of the packed_git list. */
-	struct list_head packed_git_mru; /* private */
-
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * private data
+	 *
+	 * should only be accessed directly by packfile.c
+	 */
+
+	struct packed_git *packed_git;
+	/* A most-recently-used ordered version of the packed_git list. */
+	struct list_head packed_git_mru;
+
 	/*
 	 * A fast, rough count of the number of objects in the repository.
 	 * These two fields are not meant for direct access. Use
@@ -112,7 +118,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-void raw_object_store_init(struct raw_object_store *o);
+struct raw_object_store *raw_object_store_new(void);
 void raw_object_store_clear(struct raw_object_store *o);
 
 /*
diff --git a/object.c b/object.c
index 60ca76b285..4c2cf7ff5d 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static struct object **obj_hash;
@@ -447,23 +448,30 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
-void raw_object_store_init(struct raw_object_store *o)
+struct raw_object_store *raw_object_store_new(void)
 {
+	struct raw_object_store *o = xmalloc(sizeof(*o));
+
 	memset(o, 0, sizeof(*o));
 	INIT_LIST_HEAD(&o->packed_git_mru);
+	return o;
 }
 
 static void free_alt_odb(struct alternate_object_database *alt)
 {
 	strbuf_release(&alt->scratch);
 	oid_array_clear(&alt->loose_objects_cache);
+	free(alt);
 }
 
 static void free_alt_odbs(struct raw_object_store *o)
 {
 	while (o->alt_odb_list) {
+		struct alternate_object_database *next;
+
+		next = o->alt_odb_list->next;
 		free_alt_odb(o->alt_odb_list);
-		o->alt_odb_list = o->alt_odb_list->next;
+		o->alt_odb_list = next;
 	}
 }
 
diff --git a/pack-bitmap.c b/pack-bitmap.c
index abed43cdb5..22cd425788 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -11,6 +11,7 @@
 #include "pack-objects.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..2378f25999 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -3,6 +3,7 @@
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
+#include "object-store.h"
 
 struct idx_entry {
 	off_t                offset;
diff --git a/pack-revindex.c b/pack-revindex.c
index ff5f62c033..bb521cf7fb 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index a0842521e5..17170fc662 100644
--- a/packfile.c
+++ b/packfile.c
@@ -14,6 +14,7 @@
 #include "tag.h"
 #include "tree-walk.h"
 #include "tree.h"
+#include "object-store.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -244,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = the_repository->objects->packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -382,7 +383,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = the_repository->objects->packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -684,8 +685,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = the_repository->objects.packed_git;
-	the_repository->objects.packed_git = pack;
+	pack->next = the_repository->objects->packed_git;
+	the_repository->objects->packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -767,7 +768,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = the_repository->objects.packed_git; p;
+			for (p = the_repository->objects->packed_git; p;
 			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
@@ -811,20 +812,20 @@ static void prepare_packed_git_one(char *objdir, int local)
  */
 unsigned long approximate_object_count(void)
 {
-	if (!the_repository->objects.approximate_object_count_valid) {
+	if (!the_repository->objects->approximate_object_count_valid) {
 		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
 		count = 0;
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = the_repository->objects->packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
 		}
-		the_repository->objects.approximate_object_count = count;
+		the_repository->objects->approximate_object_count = count;
 	}
-	return the_repository->objects.approximate_object_count;
+	return the_repository->objects->approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -867,8 +868,8 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	the_repository->objects.packed_git = llist_mergesort(
-		the_repository->objects.packed_git, get_next_packed_git,
+	the_repository->objects->packed_git = llist_mergesort(
+		the_repository->objects->packed_git, get_next_packed_git,
 		set_next_packed_git, sort_pack);
 }
 
@@ -876,42 +877,42 @@ static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects->packed_git_mru);
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
+	for (p = the_repository->objects->packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects->packed_git_mru);
 }
 
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (the_repository->objects.packed_git_initialized)
+	if (the_repository->objects->packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	the_repository->objects.packed_git_initialized = 1;
+	the_repository->objects->packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
-	the_repository->objects.approximate_object_count_valid = 0;
-	the_repository->objects.packed_git_initialized = 0;
+	the_repository->objects->approximate_object_count_valid = 0;
+	the_repository->objects->packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
 struct packed_git *get_packed_git(struct repository *r)
 {
-	return r->objects.packed_git;
+	return r->objects->packed_git;
 }
 
 struct list_head *get_packed_git_mru(struct repository *r)
 {
-	return &r->objects.packed_git_mru;
+	return &r->objects->packed_git_mru;
 }
 
 unsigned long unpack_object_header_buffer(const unsigned char *buf,
@@ -1022,7 +1023,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = the_repository->objects->packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1853,14 +1854,14 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!the_repository->objects.packed_git)
+	if (!the_repository->objects->packed_git)
 		return 0;
 
-	list_for_each(pos, &the_repository->objects.packed_git_mru) {
+	list_for_each(pos, &the_repository->objects->packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
 			list_move(&p->mru,
-				  &the_repository->objects.packed_git_mru);
+				  &the_repository->objects->packed_git_mru);
 			return 1;
 		}
 	}
@@ -1907,7 +1908,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = the_repository->objects->packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -1938,7 +1939,7 @@ static int add_promisor_object(const struct object_id *oid,
 
 	/*
 	 * If this is a tree, commit, or tag, the objects it refers
-	 * to are also promisor objects. (Blobs refer to no objects.)
+	 * to are also promisor objects-> (Blobs refer to no objects->)
 	 */
 	if (obj->type == OBJ_TREE) {
 		struct tree *tree = (struct tree *)obj;
diff --git a/path.c b/path.c
index 81a42d9115..3308b7b958 100644
--- a/path.c
+++ b/path.c
@@ -10,6 +10,7 @@
 #include "submodule-config.h"
 #include "path.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static int get_st_mode_bits(const char *path, int *mode)
 {
@@ -382,7 +383,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects->objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/reachable.c b/reachable.c
index 88d7d679da..25cfd99d1c 100644
--- a/reachable.c
+++ b/reachable.c
@@ -11,6 +11,7 @@
 #include "list-objects.h"
 #include "packfile.h"
 #include "worktree.h"
+#include "object-store.h"
 
 struct connectivity_progress {
 	struct progress *progress;
diff --git a/repository.c b/repository.c
index 34c0a7f180..a4848c1bd0 100644
--- a/repository.c
+++ b/repository.c
@@ -13,7 +13,7 @@ void initialize_the_repository(void)
 	the_repository = &the_repo;
 
 	the_repo.index = &the_index;
-	raw_object_store_init(&the_repo.objects);
+	the_repo.objects = raw_object_store_new();
 	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
 }
 
@@ -60,10 +60,10 @@ void repo_set_gitdir(struct repository *repo,
 	free(old_gitdir);
 
 	repo_set_commondir(repo, o->commondir);
-	expand_base_dir(&repo->objects.objectdir, o->object_dir,
+	expand_base_dir(&repo->objects->objectdir, o->object_dir,
 			repo->commondir, "objects");
-	free(repo->objects.alternate_db);
-	repo->objects.alternate_db = xstrdup_or_null(o->alternate_db);
+	free(repo->objects->alternate_db);
+	repo->objects->alternate_db = xstrdup_or_null(o->alternate_db);
 	expand_base_dir(&repo->graft_file, o->graft_file,
 			repo->commondir, "info/grafts");
 	expand_base_dir(&repo->index_file, o->index_file,
@@ -142,7 +142,7 @@ static int repo_init(struct repository *repo,
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
 
-	raw_object_store_init(&repo->objects);
+	repo->objects = raw_object_store_new();
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
@@ -223,7 +223,8 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
-	raw_object_store_clear(&repo->objects);
+	raw_object_store_clear(repo->objects);
+	FREE_AND_NULL(repo->objects);
 
 	if (repo->config) {
 		git_configset_clear(repo->config);
diff --git a/repository.h b/repository.h
index 6c383c05c6..abba2c09bc 100644
--- a/repository.h
+++ b/repository.h
@@ -1,12 +1,11 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
-#include "object-store.h"
-
 struct config_set;
+struct git_hash_algo;
 struct index_state;
+struct raw_object_store;
 struct submodule_cache;
-struct git_hash_algo;
 
 struct repository {
 	/* Environment */
@@ -25,7 +24,7 @@ struct repository {
 	/*
 	 * Holds any information related to accessing the raw object content.
 	 */
-	struct raw_object_store objects;
+	struct raw_object_store *objects;
 
 	/*
 	 * Path to the repository's graft file.
diff --git a/server-info.c b/server-info.c
index 6fe64ede17..75a8b65e47 100644
--- a/server-info.c
+++ b/server-info.c
@@ -5,6 +5,7 @@
 #include "commit.h"
 #include "tag.h"
 #include "packfile.h"
+#include "object-store.h"
 
 /*
  * Create the file "path" by writing to a temporary file and renaming
diff --git a/sha1_file.c b/sha1_file.c
index 7066d4c9ce..0989bbd948 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,7 +23,6 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
-#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -31,6 +30,7 @@
 #include "quote.h"
 #include "packfile.h"
 #include "fetch-object.h"
+#include "object-store.h"
 
 const unsigned char null_sha1[GIT_MAX_RAWSZ];
 const struct object_id null_oid;
@@ -325,7 +325,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 
 void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, r->objects.objectdir);
+	strbuf_addstr(buf, r->objects->objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
@@ -419,7 +419,7 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -427,8 +427,8 @@ static int link_alt_odb_entry(struct repository *r, const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*r->objects.alt_odb_tail = ent;
-	r->objects.alt_odb_tail = &(ent->next);
+	*r->objects->alt_odb_tail = ent;
+	r->objects->alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -482,7 +482,7 @@ static void link_alt_odb_entries(struct repository *r, const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
+	strbuf_add_absolute_path(&objdirbuf, r->objects->objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -565,7 +565,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (the_repository->objects.alt_odb_tail)
+		if (the_repository->objects->alt_odb_tail)
 			link_alt_odb_entries(the_repository, reference,
 					     '\n', NULL, 0);
 	}
@@ -665,7 +665,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb(the_repository);
-	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects->alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -675,13 +675,13 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 
 void prepare_alt_odb(struct repository *r)
 {
-	if (r->objects.alt_odb_tail)
+	if (r->objects->alt_odb_tail)
 		return;
 
-	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
-	link_alt_odb_entries(r, r->objects.alternate_db, PATH_SEP, NULL, 0);
+	r->objects->alt_odb_tail = &r->objects->alt_odb_list;
+	link_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);
 
-	read_info_alternates(r, r->objects.objectdir, 0);
+	read_info_alternates(r, r->objects->objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
@@ -722,7 +722,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -882,7 +882,7 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
 
 	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -913,7 +913,7 @@ static int open_sha1_file(struct repository *r,
 	most_interesting_errno = errno;
 
 	prepare_alt_odb(r);
-	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 2065be90d2..4325f74e0c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "object-store.h"
 #include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
@@ -105,7 +106,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = the_repository->objects.alt_odb_list;
+	fakeent->next = the_repository->objects->alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
diff --git a/submodule.c b/submodule.c
index 47ddc9b273..b03e5f5045 100644
--- a/submodule.c
+++ b/submodule.c
@@ -21,6 +21,7 @@
 #include "remote.h"
 #include "worktree.h"
 #include "parse-options.h"
+#include "object-store.h"
 
 static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;
 static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP;
diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c
index 7120634b04..7314b5943e 100644
--- a/t/helper/test-ref-store.c
+++ b/t/helper/test-ref-store.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "refs.h"
 #include "worktree.h"
+#include "object-store.h"
 
 static const char *notnull(const char *arg, const char *name)
 {
diff --git a/tmp-objdir.c b/tmp-objdir.c
index b2d9280f10..fea3f55545 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -6,6 +6,7 @@
 #include "strbuf.h"
 #include "argv-array.h"
 #include "quote.h"
+#include "object-store.h"
 
 struct tmp_objdir {
 	struct strbuf path;
diff --git a/transport.c b/transport.c
index 00d48b5b56..3afc632472 100644
--- a/transport.c
+++ b/transport.c
@@ -18,6 +18,7 @@
 #include "sha1-array.h"
 #include "sigchain.h"
 #include "transport-internal.h"
+#include "object-store.h"
 
 static void set_upstreams(struct transport *transport, struct ref *refs,
 	int pretend)
-- 
2.17.0.rc0.348.gd5a49e0b6f

[PATCH 11/27] sha1_file: add repository argument to read_info_alternates

[Nguyễn Thái Ngọc Duy]

From: Stefan Beller <sbeller@google.com>

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 7c0ace646a..81ad2a84f2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -390,7 +390,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -431,7 +433,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -497,7 +499,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -678,7 +681,7 @@ void prepare_alt_odb(void)
 	link_alt_odb_entries(the_repository->objects->alternate_db,
 			     PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.17.0.rc0.348.gd5a49e0b6f