[PATCH v3 00/44] alternate refs backends

[PATCH v3 00/44] alternate refs backends

[David Turner]

This version of the patch set includes the following changes:
1. Numerous formatting and style updates.  Thanks for Michael Haggerty to
pointing these out; remaining errors are of course my own.

2. Fixed warnings.  Thanks to Michael Haggerty, Michael Blume, and
Torsten Bögershausen for pointing these out.

3. I reworded some commit messages.

4. Made tests work with LMDB turned off. Thanks to Dennis Kaarsemaker.

5. Added Jeff King's core.repositoryformatversion patch.

6. After discussion with Michael Haggerty and Junio, redesigned the
backend structure.  Now the generic code builds ref transactions, and
backends apply these transactions.  The ref transaction backend patch
changed as a result of this, and I added three more patches:

refs: make lock generic (to allow ref backends to use a generic ref_update
struct to handle their own locking)

refs: move duplicate check to common code (to simplify the implementation
of ref backends, and to simplify the next patch)

refs: always handle non-normal refs in files backend (cross-backend
transactions)


David Turner (24):
  refs: make repack_without_refs and is_branch public
  refs: move transaction functions into common code
  refs-be-files.c: add method for for_each_reftype_...
  refs-be-files.c: add do_for_each_per_worktree_ref
  refs.c: move refname_is_safe to the common code
  refs.h: document make refname_is_safe and add it to header
  refs.c: move copy_msg to the common code
  refs.c: move peel_object to the common code
  refs.c: move should_autocreate_reflog to common code
  refs.c: add ref backend init function
  refs.c: add methods for reflog
  refs.c: add method for initial ref transaction commit
  initdb: move safe_create_dir into common code
  refs.c: add method for initializing refs db
  refs-be-files.c: add method to rename refs
  refs: make lock generic
  refs: make files_log_ref_write functions public
  refs: break out a ref conflict check
  refs: move duplicate check to common code
  refs: always handle non-normal refs in files backend
  refs: allow ref backend to be set for clone
  refs: add register_refs_backend
  refs: add LMDB refs backend
  refs: tests for db backend

Jeff King (1):
  introduce "extensions" form of core.repositoryformatversion

Ronnie Sahlberg (19):
  refs.c: create a public version of verify_refname_available
  refs-be-files.c: rename refs to refs-be-files
  refs.c: add a new refs.c file to hold all common refs code
  refs.c: move update_ref to refs.c
  refs.c: move delete_ref and delete_refs to the common code
  refs.c: move read_ref_at to the common refs file
  refs.c: move the hidden refs functions to the common code
  refs.c: move dwim and friend functions to the common refs code
  refs.c: move warn_if_dangling_symref* to the common code
  refs.c: move read_ref, read_ref_full and ref_exists to the common code
  refs.c: move resolve_refdup to common
  refs.c: move check_refname_format to the common code
  refs.c: move is_branch to the common code
  refs.c: move prettify_refname to the common code
  refs.c: move ref iterators to the common code
  refs.c: move head_ref_namespaced to the common code
  refs: add a backend method structure with transaction functions
  refs-be-files.c: add methods for misc ref operations
  refs-be-files.c: add methods for the ref iterators

 .gitignore                                     |    1 +
 Documentation/config.txt                       |    7 +
 Documentation/git-clone.txt                    |    4 +
 Documentation/git-init-db.txt                  |    2 +-
 Documentation/git-init.txt                     |    6 +
 Documentation/technical/refs-be-lmdb.txt       |   50 +
 Documentation/technical/repository-version.txt |   86 +
 Makefile                                       |   13 +
 builtin/clone.c                                |   27 +-
 builtin/init-db.c                              |   47 +-
 builtin/submodule--helper.c                    |    5 +-
 cache.h                                        |   19 +
 config.c                                       |   34 +
 configure.ac                                   |   33 +
 contrib/workdir/git-new-workdir                |    3 +
 environment.c                                  |    1 +
 path.c                                         |   12 +
 refs-be-files.c                                | 3521 ++++++++++++++++
 refs-be-lmdb.c                                 | 1997 +++++++++
 refs.c                                         | 5322 +++++-------------------
 refs.h                                         |  306 +-
 setup.c                                        |   69 +-
 t/t1302-repo-version.sh                        |   38 +
 t/t1460-refs-be-db.sh                          | 1109 +++++
 t/t1470-refs-be-db-reflog.sh                   |  359 ++
 t/test-lib.sh                                  |    1 +
 test-refs-be-lmdb.c                            |   68 +
 27 files changed, 8849 insertions(+), 4291 deletions(-)
 create mode 100644 Documentation/technical/refs-be-lmdb.txt
 create mode 100644 Documentation/technical/repository-version.txt
 create mode 100644 refs-be-files.c
 create mode 100644 refs-be-lmdb.c
 create mode 100755 t/t1460-refs-be-db.sh
 create mode 100755 t/t1470-refs-be-db-reflog.sh
 create mode 100644 test-refs-be-lmdb.c

-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 01/44] refs.c: create a public version of verify_refname_available

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Create a public version of verify_refname_available that backends can
provide.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs.c | 65 ++++++++++++++++++++++++++++++-----------------------------------
 refs.h | 17 +++++++++++++++++
 2 files changed, 47 insertions(+), 35 deletions(-)

diff --git a/refs.c b/refs.c
index 132eff5..fe71ea0 100644
--- a/refs.c
+++ b/refs.c
@@ -279,7 +279,7 @@ struct ref_dir {
  * presence of an empty subdirectory does not block the creation of a
  * similarly-named reference.  (The fact that reference names with the
  * same leading components can conflict *with each other* is a
- * separate issue that is regulated by verify_refname_available().)
+ * separate issue that is regulated by verify_refname_available_dir().)
  *
  * Please note that the name field contains the fully-qualified
  * reference (or subdirectory) name.  Space could be saved by only
@@ -897,25 +897,13 @@ static int nonmatching_ref_fn(struct ref_entry *entry, void *vdata)
 /*
  * Return 0 if a reference named refname could be created without
  * conflicting with the name of an existing reference in dir.
- * Otherwise, return a negative value and write an explanation to err.
- * If extras is non-NULL, it is a list of additional refnames with
- * which refname is not allowed to conflict. If skip is non-NULL,
- * ignore potential conflicts with refs in skip (e.g., because they
- * are scheduled for deletion in the same operation). Behavior is
- * undefined if the same name is listed in both extras and skip.
- *
- * Two reference names conflict if one of them exactly matches the
- * leading components of the other; e.g., "refs/foo/bar" conflicts
- * with both "refs/foo" and with "refs/foo/bar/baz" but not with
- * "refs/foo/bar" or "refs/foo/barbados".
- *
- * extras and skip must be sorted.
+ * See verify_refname_available for details.
  */
-static int verify_refname_available(const char *refname,
-				    const struct string_list *extras,
-				    const struct string_list *skip,
-				    struct ref_dir *dir,
-				    struct strbuf *err)
+static int verify_refname_available_dir(const char *refname,
+					const struct string_list *extras,
+					const struct string_list *skip,
+					struct ref_dir *dir,
+					struct strbuf *err)
 {
 	const char *slash;
 	int pos;
@@ -2464,9 +2452,12 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 		 */
 		strbuf_git_path(&orig_ref_file, "%s", orig_refname);
 		if (remove_empty_directories(&orig_ref_file)) {
+			struct ref_dir *loose_refs;
+			loose_refs = get_loose_refs(&ref_cache);
 			last_errno = errno;
-			if (!verify_refname_available(orig_refname, extras, skip,
-						      get_loose_refs(&ref_cache), err))
+			if (!verify_refname_available_dir(orig_refname, extras,
+							  skip, loose_refs,
+							  err))
 				strbuf_addf(err, "there are still refs under '%s'",
 					    orig_refname);
 			goto error_return;
@@ -2479,8 +2470,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	if (!refname) {
 		last_errno = errno;
 		if (last_errno != ENOTDIR ||
-		    !verify_refname_available(orig_refname, extras, skip,
-					      get_loose_refs(&ref_cache), err))
+		    !verify_refname_available_dir(orig_refname, extras, skip,
+						  get_loose_refs(&ref_cache),
+						  err))
 			strbuf_addf(err, "unable to resolve reference %s: %s",
 				    orig_refname, strerror(last_errno));
 
@@ -2493,8 +2485,8 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	 * our refname.
 	 */
 	if (is_null_oid(&lock->old_oid) &&
-	    verify_refname_available(refname, extras, skip,
-				     get_packed_refs(&ref_cache), err)) {
+	    verify_refname_available_dir(refname, extras, skip,
+					 get_packed_refs(&ref_cache), err)) {
 		last_errno = ENOTDIR;
 		goto error_return;
 	}
@@ -3127,10 +3119,7 @@ static int rename_ref_available(const char *oldname, const char *newname)
 	int ret;
 
 	string_list_insert(&skip, oldname);
-	ret = !verify_refname_available(newname, NULL, &skip,
-					get_packed_refs(&ref_cache), &err)
-		&& !verify_refname_available(newname, NULL, &skip,
-					     get_loose_refs(&ref_cache), &err);
+	ret = !verify_refname_available(newname, NULL, &skip, &err);
 	if (!ret)
 		error("%s", err.buf);
 
@@ -3299,6 +3288,17 @@ static int should_autocreate_reflog(const char *refname)
 		!strcmp(refname, "HEAD");
 }
 
+int verify_refname_available(const char *newname, struct string_list *extra,
+			     struct string_list *skip, struct strbuf *err)
+{
+	struct ref_dir *packed_refs = get_packed_refs(&ref_cache);
+	struct ref_dir *loose_refs = get_loose_refs(&ref_cache);
+	return verify_refname_available_dir(newname, extra, skip,
+					    packed_refs, err) ||
+		verify_refname_available_dir(newname, extra, skip,
+					     loose_refs, err);
+}
+
 /*
  * Create a reflog for a ref.  If force_create = 0, the reflog will
  * only be created for certain refs (those for which
@@ -4334,8 +4334,6 @@ static int ref_present(const char *refname,
 int initial_ref_transaction_commit(struct ref_transaction *transaction,
 				   struct strbuf *err)
 {
-	struct ref_dir *loose_refs = get_loose_refs(&ref_cache);
-	struct ref_dir *packed_refs = get_packed_refs(&ref_cache);
 	int ret = 0, i;
 	int n = transaction->nr;
 	struct ref_update **updates = transaction->updates;
@@ -4378,10 +4376,7 @@ int initial_ref_transaction_commit(struct ref_transaction *transaction,
 			die("BUG: initial ref transaction with old_sha1 set");
 		if (verify_refname_available(update->refname,
 					     &affected_refnames, NULL,
-					     loose_refs, err) ||
-		    verify_refname_available(update->refname,
-					     &affected_refnames, NULL,
-					     packed_refs, err)) {
+					     err)) {
 			ret = TRANSACTION_NAME_CONFLICT;
 			goto cleanup;
 		}
diff --git a/refs.h b/refs.h
index 6d30c98..79ea220 100644
--- a/refs.h
+++ b/refs.h
@@ -218,6 +218,23 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st
 int pack_refs(unsigned int flags);
 
 /*
+ * Return true iff a reference named refname could be created without
+ * conflicting with the name of an existing reference.  If
+ * skip is non-NULL, ignore potential conflicts with refs in skip
+ * (e.g., because they are scheduled for deletion in the same
+ * operation).
+ *
+ * Two reference names conflict if one of them exactly matches the
+ * leading components of the other; e.g., "foo/bar" conflicts with
+ * both "foo" and with "foo/bar/baz" but not with "foo/bar" or
+ * "foo/barbados".
+ *
+ * skip must be sorted.
+ */
+int verify_refname_available(const char *newname, struct string_list *extra,
+			     struct string_list *skip, struct strbuf *err);
+
+/*
  * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
  * REF_NODEREF: act on the ref directly, instead of dereferencing
  *              symbolic references.
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 02/44] refs: make repack_without_refs and is_branch public

[David Turner]

is_branch was already non-static, but this patch declares it in the
header.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs.c |  9 +--------
 refs.h | 13 +++++++++++++
 2 files changed, 14 insertions(+), 8 deletions(-)

diff --git a/refs.c b/refs.c
index fe71ea0..d0dfdfc 100644
--- a/refs.c
+++ b/refs.c
@@ -2814,14 +2814,7 @@ int pack_refs(unsigned int flags)
 	return 0;
 }
 
-/*
- * Rewrite the packed-refs file, omitting any refs listed in
- * 'refnames'. On error, leave packed-refs unchanged, write an error
- * message to 'err', and return a nonzero value.
- *
- * The refs in 'refnames' needn't be sorted. `err` must not be NULL.
- */
-static int repack_without_refs(struct string_list *refnames, struct strbuf *err)
+int repack_without_refs(struct string_list *refnames, struct strbuf *err)
 {
 	struct ref_dir *packed;
 	struct string_list_item *refname;
diff --git a/refs.h b/refs.h
index 79ea220..729bc3c 100644
--- a/refs.h
+++ b/refs.h
@@ -218,6 +218,19 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st
 int pack_refs(unsigned int flags);
 
 /*
+ * Rewrite the packed-refs file, omitting any refs listed in
+ * 'refnames'. On error, packed-refs will be unchanged, the return
+ * value is nonzero, and a message about the error is written to the
+ * 'err' strbuf.
+ *
+ * The refs in 'refnames' needn't be sorted. `err` must not be NULL.
+ */
+extern int repack_without_refs(struct string_list *refnames,
+			       struct strbuf *err);
+
+extern int is_branch(const char *refname);
+
+/*
  * Return true iff a reference named refname could be created without
  * conflicting with the name of an existing reference.  If
  * skip is non-NULL, ignore potential conflicts with refs in skip
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 03/44] refs-be-files.c: rename refs to refs-be-files

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Rename refs.c to refs-be-files.c to indicate that this file now
holds the implementation for the files based refs backend.
A smaller portion of the code in this file is backend agnostic and will
be moved to a a new refs.c file that will hold all the common refs code
that is shared across all backends.

A second reason for first moving all the code to the new file and then
move the backend agnostic code back to refs.c instead of the other way
around is because the code that will eventually remain in this new
refs-be-files.c file is so entangled that it would then be very
difficult to break the split up into small independent patches/chunks.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 Makefile                  | 2 +-
 refs.c => refs-be-files.c | 0
 2 files changed, 1 insertion(+), 1 deletion(-)
 rename refs.c => refs-be-files.c (100%)

diff --git a/Makefile b/Makefile
index 46d0ca7..19036de 100644
--- a/Makefile
+++ b/Makefile
@@ -766,7 +766,7 @@ LIB_OBJS += quote.o
 LIB_OBJS += reachable.o
 LIB_OBJS += read-cache.o
 LIB_OBJS += reflog-walk.o
-LIB_OBJS += refs.o
+LIB_OBJS += refs-be-files.o
 LIB_OBJS += ref-filter.o
 LIB_OBJS += remote.o
 LIB_OBJS += replace_object.o
diff --git a/refs.c b/refs-be-files.c
similarity index 100%
rename from refs.c
rename to refs-be-files.c
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 04/44] refs.c: add a new refs.c file to hold all common refs code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Create a new refs.c file that will be used to hold all the refs
code that is backend agnostic and will be shared across all backends.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 Makefile | 1 +
 refs.c   | 3 +++
 2 files changed, 4 insertions(+)
 create mode 100644 refs.c

diff --git a/Makefile b/Makefile
index 19036de..43ceab0 100644
--- a/Makefile
+++ b/Makefile
@@ -767,6 +767,7 @@ LIB_OBJS += reachable.o
 LIB_OBJS += read-cache.o
 LIB_OBJS += reflog-walk.o
 LIB_OBJS += refs-be-files.o
+LIB_OBJS += refs.o
 LIB_OBJS += ref-filter.o
 LIB_OBJS += remote.o
 LIB_OBJS += replace_object.o
diff --git a/refs.c b/refs.c
new file mode 100644
index 0000000..77492ff
--- /dev/null
+++ b/refs.c
@@ -0,0 +1,3 @@
+/*
+ * Common refs code for all backends.
+ */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 05/44] refs.c: move update_ref to refs.c

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Move update_ref() to the refs.c file since this function does not
contain any backend specific code.  Move the ref classifier functions
as well, since update_ref depends on them.

Based on Ronnie Sahlberg's patch

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 117 +-------------------------------------------------------
 refs.c          | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 116 insertions(+), 116 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index d0dfdfc..7fe4931 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2675,8 +2675,6 @@ struct pack_refs_cb_data {
 	struct ref_to_prune *ref_to_prune;
 };
 
-static int is_per_worktree_ref(const char *refname);
-
 /*
  * An each_ref_entry_fn that is run over loose references only.  If
  * the loose reference can be packed, add an entry in the packed ref
@@ -2691,7 +2689,7 @@ static int pack_if_possible_fn(struct ref_entry *entry, void *cb_data)
 	int is_tag_ref = starts_with(entry->name, "refs/tags/");
 
 	/* Do not pack per-worktree refs: */
-	if (is_per_worktree_ref(entry->name))
+	if (ref_type(entry->name) == REF_TYPE_PER_WORKTREE)
 		return 0;
 
 	/* ALWAYS pack tags */
@@ -2879,77 +2877,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 	return 0;
 }
 
-static int is_per_worktree_ref(const char *refname)
-{
-	return !strcmp(refname, "HEAD") ||
-		starts_with(refname, "refs/bisect/");
-}
-
-static int is_pseudoref_syntax(const char *refname)
-{
-	const char *c;
-
-	for (c = refname; *c; c++) {
-		if (!isupper(*c) && *c != '-' && *c != '_')
-			return 0;
-	}
-
-	return 1;
-}
-
-enum ref_type ref_type(const char *refname)
-{
-	if (is_per_worktree_ref(refname))
-		return REF_TYPE_PER_WORKTREE;
-	if (is_pseudoref_syntax(refname))
-		return REF_TYPE_PSEUDOREF;
-       return REF_TYPE_NORMAL;
-}
-
-static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,
-			   const unsigned char *old_sha1, struct strbuf *err)
-{
-	const char *filename;
-	int fd;
-	static struct lock_file lock;
-	struct strbuf buf = STRBUF_INIT;
-	int ret = -1;
-
-	strbuf_addf(&buf, "%s\n", sha1_to_hex(sha1));
-
-	filename = git_path("%s", pseudoref);
-	fd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
-	if (fd < 0) {
-		strbuf_addf(err, "Could not open '%s' for writing: %s",
-			    filename, strerror(errno));
-		return -1;
-	}
-
-	if (old_sha1) {
-		unsigned char actual_old_sha1[20];
-
-		if (read_ref(pseudoref, actual_old_sha1))
-			die("could not read ref '%s'", pseudoref);
-		if (hashcmp(actual_old_sha1, old_sha1)) {
-			strbuf_addf(err, "Unexpected sha1 when writing %s", pseudoref);
-			rollback_lock_file(&lock);
-			goto done;
-		}
-	}
-
-	if (write_in_full(fd, buf.buf, buf.len) != buf.len) {
-		strbuf_addf(err, "Could not write to '%s'", filename);
-		rollback_lock_file(&lock);
-		goto done;
-	}
-
-	commit_lock_file(&lock);
-	ret = 0;
-done:
-	strbuf_release(&buf);
-	return ret;
-}
-
 static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)
 {
 	static struct lock_file lock;
@@ -4098,48 +4025,6 @@ int ref_transaction_verify(struct ref_transaction *transaction,
 				      flags, NULL, err);
 }
 
-int update_ref(const char *msg, const char *refname,
-	       const unsigned char *new_sha1, const unsigned char *old_sha1,
-	       unsigned int flags, enum action_on_err onerr)
-{
-	struct ref_transaction *t = NULL;
-	struct strbuf err = STRBUF_INIT;
-	int ret = 0;
-
-	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		ret = write_pseudoref(refname, new_sha1, old_sha1, &err);
-	} else {
-		t = ref_transaction_begin(&err);
-		if (!t ||
-		    ref_transaction_update(t, refname, new_sha1, old_sha1,
-					   flags, msg, &err) ||
-		    ref_transaction_commit(t, &err)) {
-			ret = 1;
-			ref_transaction_free(t);
-		}
-	}
-	if (ret) {
-		const char *str = "update_ref failed for ref '%s': %s";
-
-		switch (onerr) {
-		case UPDATE_REFS_MSG_ON_ERR:
-			error(str, refname, err.buf);
-			break;
-		case UPDATE_REFS_DIE_ON_ERR:
-			die(str, refname, err.buf);
-			break;
-		case UPDATE_REFS_QUIET_ON_ERR:
-			break;
-		}
-		strbuf_release(&err);
-		return 1;
-	}
-	strbuf_release(&err);
-	if (t)
-		ref_transaction_free(t);
-	return 0;
-}
-
 static int ref_update_reject_duplicates(struct string_list *refnames,
 					struct strbuf *err)
 {
diff --git a/refs.c b/refs.c
index 77492ff..2d10708 100644
--- a/refs.c
+++ b/refs.c
@@ -1,3 +1,118 @@
 /*
  * Common refs code for all backends.
  */
+#include "cache.h"
+#include "refs.h"
+#include "lockfile.h"
+
+static int is_per_worktree_ref(const char *refname)
+{
+	return !strcmp(refname, "HEAD") ||
+		starts_with(refname, "refs/bisect/");
+}
+
+static int is_pseudoref_syntax(const char *refname)
+{
+	const char *c;
+
+	for (c = refname; *c; c++) {
+		if (!isupper(*c) && *c != '-' && *c != '_')
+			return 0;
+	}
+
+	return 1;
+}
+
+enum ref_type ref_type(const char *refname)
+{
+	if (is_per_worktree_ref(refname))
+		return REF_TYPE_PER_WORKTREE;
+	if (is_pseudoref_syntax(refname))
+		return REF_TYPE_PSEUDOREF;
+       return REF_TYPE_NORMAL;
+}
+
+static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,
+			   const unsigned char *old_sha1, struct strbuf *err)
+{
+	const char *filename;
+	int fd;
+	static struct lock_file lock;
+	struct strbuf buf = STRBUF_INIT;
+	int ret = -1;
+
+	strbuf_addf(&buf, "%s\n", sha1_to_hex(sha1));
+
+	filename = git_path("%s", pseudoref);
+	fd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
+	if (fd < 0) {
+		strbuf_addf(err, "Could not open '%s' for writing: %s",
+			    filename, strerror(errno));
+		return -1;
+	}
+
+	if (old_sha1) {
+		unsigned char actual_old_sha1[20];
+		read_ref(pseudoref, actual_old_sha1);
+		if (hashcmp(actual_old_sha1, old_sha1)) {
+			strbuf_addf(err, "Unexpected sha1 when writing %s", pseudoref);
+			rollback_lock_file(&lock);
+			goto done;
+		}
+	}
+
+	if (write_in_full(fd, buf.buf, buf.len) != buf.len) {
+		strbuf_addf(err, "Could not write to '%s'", filename);
+		rollback_lock_file(&lock);
+		goto done;
+	}
+
+	commit_lock_file(&lock);
+	ret = 0;
+done:
+	strbuf_release(&buf);
+	return ret;
+}
+
+int update_ref(const char *msg, const char *refname,
+	       const unsigned char *new_sha1, const unsigned char *old_sha1,
+	       unsigned int flags, enum action_on_err onerr)
+{
+	struct ref_transaction *t = NULL;
+	struct strbuf err = STRBUF_INIT;
+	int ret = 0;
+
+	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
+		ret = write_pseudoref(refname, new_sha1, old_sha1, &err);
+	} else {
+		t = ref_transaction_begin(&err);
+		if (!t ||
+		    ref_transaction_update(t, refname, new_sha1, old_sha1,
+					   flags, msg, &err) ||
+		    ref_transaction_commit(t, &err)) {
+			ret = 1;
+			ref_transaction_free(t);
+		}
+	}
+
+	if (ret) {
+		const char *str = "update_ref failed for ref '%s': %s";
+
+		switch (onerr) {
+		case UPDATE_REFS_MSG_ON_ERR:
+			error(str, refname, err.buf);
+			break;
+		case UPDATE_REFS_DIE_ON_ERR:
+			die(str, refname, err.buf);
+			break;
+		case UPDATE_REFS_QUIET_ON_ERR:
+			break;
+		}
+		strbuf_release(&err);
+		return 1;
+	}
+	strbuf_release(&err);
+	if (t)
+		ref_transaction_free(t);
+	return 0;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 06/44] refs.c: move delete_ref and delete_refs to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Move delete_pseudoref, delete_ref() and delete_refs() to the refs.c
file since these functions do not contain any backend specific code.

Based on a patch by Ronnie Sahlberg.

Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs-be-files.c | 94 ---------------------------------------------------------
 refs.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 94 insertions(+), 94 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 7fe4931..099df75 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2877,100 +2877,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 	return 0;
 }
 
-static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)
-{
-	static struct lock_file lock;
-	const char *filename;
-
-	filename = git_path("%s", pseudoref);
-
-	if (old_sha1 && !is_null_sha1(old_sha1)) {
-		int fd;
-		unsigned char actual_old_sha1[20];
-
-		fd = hold_lock_file_for_update(&lock, filename,
-					       LOCK_DIE_ON_ERROR);
-		if (fd < 0)
-			die_errno(_("Could not open '%s' for writing"), filename);
-		if (read_ref(pseudoref, actual_old_sha1))
-			die("could not read ref '%s'", pseudoref);
-		if (hashcmp(actual_old_sha1, old_sha1)) {
-			warning("Unexpected sha1 when deleting %s", pseudoref);
-			rollback_lock_file(&lock);
-			return -1;
-		}
-
-		unlink(filename);
-		rollback_lock_file(&lock);
-	} else {
-		unlink(filename);
-	}
-
-	return 0;
-}
-
-int delete_ref(const char *refname, const unsigned char *old_sha1,
-	       unsigned int flags)
-{
-	struct ref_transaction *transaction;
-	struct strbuf err = STRBUF_INIT;
-
-	if (ref_type(refname) == REF_TYPE_PSEUDOREF)
-		return delete_pseudoref(refname, old_sha1);
-
-	transaction = ref_transaction_begin(&err);
-	if (!transaction ||
-	    ref_transaction_delete(transaction, refname, old_sha1,
-				   flags, NULL, &err) ||
-	    ref_transaction_commit(transaction, &err)) {
-		error("%s", err.buf);
-		ref_transaction_free(transaction);
-		strbuf_release(&err);
-		return 1;
-	}
-	ref_transaction_free(transaction);
-	strbuf_release(&err);
-	return 0;
-}
-
-int delete_refs(struct string_list *refnames)
-{
-	struct strbuf err = STRBUF_INIT;
-	int i, result = 0;
-
-	if (!refnames->nr)
-		return 0;
-
-	result = repack_without_refs(refnames, &err);
-	if (result) {
-		/*
-		 * If we failed to rewrite the packed-refs file, then
-		 * it is unsafe to try to remove loose refs, because
-		 * doing so might expose an obsolete packed value for
-		 * a reference that might even point at an object that
-		 * has been garbage collected.
-		 */
-		if (refnames->nr == 1)
-			error(_("could not delete reference %s: %s"),
-			      refnames->items[0].string, err.buf);
-		else
-			error(_("could not delete references: %s"), err.buf);
-
-		goto out;
-	}
-
-	for (i = 0; i < refnames->nr; i++) {
-		const char *refname = refnames->items[i].string;
-
-		if (delete_ref(refname, NULL, 0))
-			result |= error(_("could not remove reference %s"), refname);
-	}
-
-out:
-	strbuf_release(&err);
-	return result;
-}
-
 /*
  * People using contrib's git-new-workdir have .git/logs/refs ->
  * /some/other/path/.git/logs/refs, and that may live on another device.
diff --git a/refs.c b/refs.c
index 2d10708..205a899 100644
--- a/refs.c
+++ b/refs.c
@@ -116,3 +116,97 @@ int update_ref(const char *msg, const char *refname,
 		ref_transaction_free(t);
 	return 0;
 }
+
+
+static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)
+{
+	static struct lock_file lock;
+	const char *filename;
+
+	filename = git_path("%s", pseudoref);
+
+	if (old_sha1 && !is_null_sha1(old_sha1)) {
+		int fd;
+		unsigned char actual_old_sha1[20];
+
+		fd = hold_lock_file_for_update(&lock, filename,
+					       LOCK_DIE_ON_ERROR);
+		if (fd < 0)
+			die_errno(_("Could not open '%s' for writing"), filename);
+		read_ref(pseudoref, actual_old_sha1);
+		if (hashcmp(actual_old_sha1, old_sha1)) {
+			warning("Unexpected sha1 when deleting %s", pseudoref);
+			rollback_lock_file(&lock);
+			return -1;
+		}
+
+		unlink(filename);
+		rollback_lock_file(&lock);
+	} else {
+		unlink(filename);
+	}
+
+	return 0;
+}
+
+int delete_ref(const char *refname, const unsigned char *old_sha1,
+	       unsigned int flags)
+{
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
+
+	if (ref_type(refname) == REF_TYPE_PSEUDOREF)
+		return delete_pseudoref(refname, old_sha1);
+
+	transaction = ref_transaction_begin(&err);
+	if (!transaction ||
+	    ref_transaction_delete(transaction, refname, old_sha1,
+				   flags, NULL, &err) ||
+	    ref_transaction_commit(transaction, &err)) {
+		error("%s", err.buf);
+		ref_transaction_free(transaction);
+		strbuf_release(&err);
+		return 1;
+	}
+	ref_transaction_free(transaction);
+	strbuf_release(&err);
+	return 0;
+}
+
+int delete_refs(struct string_list *refnames)
+{
+	struct strbuf err = STRBUF_INIT;
+	int i, result = 0;
+
+	if (!refnames->nr)
+		return 0;
+
+	result = repack_without_refs(refnames, &err);
+	if (result) {
+		/*
+		 * If we failed to rewrite the packed-refs file, then
+		 * it is unsafe to try to remove loose refs, because
+		 * doing so might expose an obsolete packed value for
+		 * a reference that might even point at an object that
+		 * has been garbage collected.
+		 */
+		if (refnames->nr == 1)
+			error(_("could not delete reference %s: %s"),
+			      refnames->items[0].string, err.buf);
+		else
+			error(_("could not delete references: %s"), err.buf);
+
+		goto out;
+	}
+
+	for (i = 0; i < refnames->nr; i++) {
+		const char *refname = refnames->items[i].string;
+
+		if (delete_ref(refname, NULL, 0))
+			result |= error(_("could not remove reference %s"), refname);
+	}
+
+out:
+	strbuf_release(&err);
+	return result;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 07/44] refs.c: move read_ref_at to the common refs file

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Move read_ref_at() to the refs.c file since this function does not
contain any backend specific code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 118 --------------------------------------------------------
 refs.c          | 118 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 118 insertions(+), 118 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 099df75..4e7e05e 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3424,124 +3424,6 @@ int create_symref(const char *ref_target, const char *refs_heads_master,
 	return 0;
 }
 
-struct read_ref_at_cb {
-	const char *refname;
-	unsigned long at_time;
-	int cnt;
-	int reccnt;
-	unsigned char *sha1;
-	int found_it;
-
-	unsigned char osha1[20];
-	unsigned char nsha1[20];
-	int tz;
-	unsigned long date;
-	char **msg;
-	unsigned long *cutoff_time;
-	int *cutoff_tz;
-	int *cutoff_cnt;
-};
-
-static int read_ref_at_ent(unsigned char *osha1, unsigned char *nsha1,
-		const char *email, unsigned long timestamp, int tz,
-		const char *message, void *cb_data)
-{
-	struct read_ref_at_cb *cb = cb_data;
-
-	cb->reccnt++;
-	cb->tz = tz;
-	cb->date = timestamp;
-
-	if (timestamp <= cb->at_time || cb->cnt == 0) {
-		if (cb->msg)
-			*cb->msg = xstrdup(message);
-		if (cb->cutoff_time)
-			*cb->cutoff_time = timestamp;
-		if (cb->cutoff_tz)
-			*cb->cutoff_tz = tz;
-		if (cb->cutoff_cnt)
-			*cb->cutoff_cnt = cb->reccnt - 1;
-		/*
-		 * we have not yet updated cb->[n|o]sha1 so they still
-		 * hold the values for the previous record.
-		 */
-		if (!is_null_sha1(cb->osha1)) {
-			hashcpy(cb->sha1, nsha1);
-			if (hashcmp(cb->osha1, nsha1))
-				warning("Log for ref %s has gap after %s.",
-					cb->refname, show_date(cb->date, cb->tz, DATE_MODE(RFC2822)));
-		}
-		else if (cb->date == cb->at_time)
-			hashcpy(cb->sha1, nsha1);
-		else if (hashcmp(nsha1, cb->sha1))
-			warning("Log for ref %s unexpectedly ended on %s.",
-				cb->refname, show_date(cb->date, cb->tz,
-						       DATE_MODE(RFC2822)));
-		hashcpy(cb->osha1, osha1);
-		hashcpy(cb->nsha1, nsha1);
-		cb->found_it = 1;
-		return 1;
-	}
-	hashcpy(cb->osha1, osha1);
-	hashcpy(cb->nsha1, nsha1);
-	if (cb->cnt > 0)
-		cb->cnt--;
-	return 0;
-}
-
-static int read_ref_at_ent_oldest(unsigned char *osha1, unsigned char *nsha1,
-				  const char *email, unsigned long timestamp,
-				  int tz, const char *message, void *cb_data)
-{
-	struct read_ref_at_cb *cb = cb_data;
-
-	if (cb->msg)
-		*cb->msg = xstrdup(message);
-	if (cb->cutoff_time)
-		*cb->cutoff_time = timestamp;
-	if (cb->cutoff_tz)
-		*cb->cutoff_tz = tz;
-	if (cb->cutoff_cnt)
-		*cb->cutoff_cnt = cb->reccnt;
-	hashcpy(cb->sha1, osha1);
-	if (is_null_sha1(cb->sha1))
-		hashcpy(cb->sha1, nsha1);
-	/* We just want the first entry */
-	return 1;
-}
-
-int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time, int cnt,
-		unsigned char *sha1, char **msg,
-		unsigned long *cutoff_time, int *cutoff_tz, int *cutoff_cnt)
-{
-	struct read_ref_at_cb cb;
-
-	memset(&cb, 0, sizeof(cb));
-	cb.refname = refname;
-	cb.at_time = at_time;
-	cb.cnt = cnt;
-	cb.msg = msg;
-	cb.cutoff_time = cutoff_time;
-	cb.cutoff_tz = cutoff_tz;
-	cb.cutoff_cnt = cutoff_cnt;
-	cb.sha1 = sha1;
-
-	for_each_reflog_ent_reverse(refname, read_ref_at_ent, &cb);
-
-	if (!cb.reccnt) {
-		if (flags & GET_SHA1_QUIETLY)
-			exit(128);
-		else
-			die("Log for %s is empty.", refname);
-	}
-	if (cb.found_it)
-		return 0;
-
-	for_each_reflog_ent(refname, read_ref_at_ent_oldest, &cb);
-
-	return 1;
-}
-
 int reflog_exists(const char *refname)
 {
 	struct stat st;
diff --git a/refs.c b/refs.c
index 205a899..5a8ef0c 100644
--- a/refs.c
+++ b/refs.c
@@ -210,3 +210,121 @@ out:
 	strbuf_release(&err);
 	return result;
 }
+
+struct read_ref_at_cb {
+	const char *refname;
+	unsigned long at_time;
+	int cnt;
+	int reccnt;
+	unsigned char *sha1;
+	int found_it;
+
+	unsigned char osha1[20];
+	unsigned char nsha1[20];
+	int tz;
+	unsigned long date;
+	char **msg;
+	unsigned long *cutoff_time;
+	int *cutoff_tz;
+	int *cutoff_cnt;
+};
+
+static int read_ref_at_ent(unsigned char *osha1, unsigned char *nsha1,
+		const char *email, unsigned long timestamp, int tz,
+		const char *message, void *cb_data)
+{
+	struct read_ref_at_cb *cb = cb_data;
+
+	cb->reccnt++;
+	cb->tz = tz;
+	cb->date = timestamp;
+
+	if (timestamp <= cb->at_time || cb->cnt == 0) {
+		if (cb->msg)
+			*cb->msg = xstrdup(message);
+		if (cb->cutoff_time)
+			*cb->cutoff_time = timestamp;
+		if (cb->cutoff_tz)
+			*cb->cutoff_tz = tz;
+		if (cb->cutoff_cnt)
+			*cb->cutoff_cnt = cb->reccnt - 1;
+		/*
+		 * we have not yet updated cb->[n|o]sha1 so they still
+		 * hold the values for the previous record.
+		 */
+		if (!is_null_sha1(cb->osha1)) {
+			hashcpy(cb->sha1, nsha1);
+			if (hashcmp(cb->osha1, nsha1))
+				warning("Log for ref %s has gap after %s.",
+					cb->refname, show_date(cb->date, cb->tz, DATE_MODE(RFC2822)));
+		}
+		else if (cb->date == cb->at_time)
+			hashcpy(cb->sha1, nsha1);
+		else if (hashcmp(nsha1, cb->sha1))
+			warning("Log for ref %s unexpectedly ended on %s.",
+				cb->refname, show_date(cb->date, cb->tz,
+						       DATE_MODE(RFC2822)));
+		hashcpy(cb->osha1, osha1);
+		hashcpy(cb->nsha1, nsha1);
+		cb->found_it = 1;
+		return 1;
+	}
+	hashcpy(cb->osha1, osha1);
+	hashcpy(cb->nsha1, nsha1);
+	if (cb->cnt > 0)
+		cb->cnt--;
+	return 0;
+}
+
+static int read_ref_at_ent_oldest(unsigned char *osha1, unsigned char *nsha1,
+				  const char *email, unsigned long timestamp,
+				  int tz, const char *message, void *cb_data)
+{
+	struct read_ref_at_cb *cb = cb_data;
+
+	if (cb->msg)
+		*cb->msg = xstrdup(message);
+	if (cb->cutoff_time)
+		*cb->cutoff_time = timestamp;
+	if (cb->cutoff_tz)
+		*cb->cutoff_tz = tz;
+	if (cb->cutoff_cnt)
+		*cb->cutoff_cnt = cb->reccnt;
+	hashcpy(cb->sha1, osha1);
+	if (is_null_sha1(cb->sha1))
+		hashcpy(cb->sha1, nsha1);
+	/* We just want the first entry */
+	return 1;
+}
+
+int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time, int cnt,
+		unsigned char *sha1, char **msg,
+		unsigned long *cutoff_time, int *cutoff_tz, int *cutoff_cnt)
+{
+	struct read_ref_at_cb cb;
+
+	memset(&cb, 0, sizeof(cb));
+	cb.refname = refname;
+	cb.at_time = at_time;
+	cb.cnt = cnt;
+	cb.msg = msg;
+	cb.cutoff_time = cutoff_time;
+	cb.cutoff_tz = cutoff_tz;
+	cb.cutoff_cnt = cutoff_cnt;
+	cb.sha1 = sha1;
+
+	for_each_reflog_ent_reverse(refname, read_ref_at_ent, &cb);
+
+	if (!cb.reccnt) {
+		if (flags & GET_SHA1_QUIETLY)
+			exit(128);
+		else
+			die("Log for %s is empty.", refname);
+	}
+	if (cb.found_it)
+		return 0;
+
+	for_each_reflog_ent(refname, read_ref_at_ent_oldest, &cb);
+
+	return 1;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 08/44] refs.c: move the hidden refs functions to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Move the hidden refs functions to the refs.c file since these
functions do not contain any backend specific code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 51 ---------------------------------------------------
 refs.c          | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 51 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 4e7e05e..d110bf8 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -4169,57 +4169,6 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 	return xstrdup(refname);
 }
 
-static struct string_list *hide_refs;
-
-int parse_hide_refs_config(const char *var, const char *value, const char *section)
-{
-	if (!strcmp("transfer.hiderefs", var) ||
-	    /* NEEDSWORK: use parse_config_key() once both are merged */
-	    (starts_with(var, section) && var[strlen(section)] == '.' &&
-	     !strcmp(var + strlen(section), ".hiderefs"))) {
-		char *ref;
-		int len;
-
-		if (!value)
-			return config_error_nonbool(var);
-		ref = xstrdup(value);
-		len = strlen(ref);
-		while (len && ref[len - 1] == '/')
-			ref[--len] = '\0';
-		if (!hide_refs) {
-			hide_refs = xcalloc(1, sizeof(*hide_refs));
-			hide_refs->strdup_strings = 1;
-		}
-		string_list_append(hide_refs, ref);
-	}
-	return 0;
-}
-
-int ref_is_hidden(const char *refname)
-{
-	int i;
-
-	if (!hide_refs)
-		return 0;
-	for (i = hide_refs->nr - 1; i >= 0; i--) {
-		const char *match = hide_refs->items[i].string;
-		int neg = 0;
-		int len;
-
-		if (*match == '!') {
-			neg = 1;
-			match++;
-		}
-
-		if (!starts_with(refname, match))
-			continue;
-		len = strlen(match);
-		if (!refname[len] || refname[len] == '/')
-			return !neg;
-	}
-	return 0;
-}
-
 struct expire_reflog_cb {
 	unsigned int flags;
 	reflog_expiry_should_prune_fn *should_prune_fn;
diff --git a/refs.c b/refs.c
index 5a8ef0c..6b2fc39 100644
--- a/refs.c
+++ b/refs.c
@@ -328,3 +328,53 @@ int read_ref_at(const char *refname, unsigned int flags, unsigned long at_time,
 
 	return 1;
 }
+
+static struct string_list *hide_refs;
+
+int parse_hide_refs_config(const char *var, const char *value, const char *section)
+{
+	if (!strcmp("transfer.hiderefs", var) ||
+	    /* NEEDSWORK: use parse_config_key() once both are merged */
+	    (starts_with(var, section) && var[strlen(section)] == '.' &&
+	     !strcmp(var + strlen(section), ".hiderefs"))) {
+		char *ref;
+		int len;
+
+		if (!value)
+			return config_error_nonbool(var);
+		ref = xstrdup(value);
+		len = strlen(ref);
+		while (len && ref[len - 1] == '/')
+			ref[--len] = '\0';
+		if (!hide_refs) {
+			hide_refs = xcalloc(1, sizeof(*hide_refs));
+			hide_refs->strdup_strings = 1;
+		}
+		string_list_append(hide_refs, ref);
+	}
+	return 0;
+}
+
+int ref_is_hidden(const char *refname)
+{
+	int i;
+
+	if (!hide_refs)
+		return 0;
+	for (i = hide_refs->nr - 1; i >= 0; i--) {
+		const char *match = hide_refs->items[i].string;
+		int neg = 0;
+		int len;
+		if (*match == '!') {
+			neg = 1;
+			match++;
+		}
+
+		if (!starts_with(refname, match))
+			continue;
+		len = strlen(match);
+		if (!refname[len] || refname[len] == '/')
+			return !neg;
+	}
+	return 0;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 09/44] refs.c: move dwim and friend functions to the common refs code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

These functions do not contain any backend specific code so we move
them to the common code and share across all backends.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 203 --------------------------------------------------------
 refs.c          | 203 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 203 insertions(+), 203 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index d110bf8..37772a7 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2247,30 +2247,6 @@ const char *prettify_refname(const char *name)
 		0);
 }
 
-static const char *ref_rev_parse_rules[] = {
-	"%.*s",
-	"refs/%.*s",
-	"refs/tags/%.*s",
-	"refs/heads/%.*s",
-	"refs/remotes/%.*s",
-	"refs/remotes/%.*s/HEAD",
-	NULL
-};
-
-int refname_match(const char *abbrev_name, const char *full_name)
-{
-	const char **p;
-	const int abbrev_name_len = strlen(abbrev_name);
-
-	for (p = ref_rev_parse_rules; *p; p++) {
-		if (!strcmp(full_name, mkpath(*p, abbrev_name_len, abbrev_name))) {
-			return 1;
-		}
-	}
-
-	return 0;
-}
-
 static void unlock_ref(struct ref_lock *lock)
 {
 	/* Do not free lock->lk -- atexit() still looks at them */
@@ -2323,92 +2299,6 @@ static int remove_empty_directories(struct strbuf *path)
 }
 
 /*
- * *string and *len will only be substituted, and *string returned (for
- * later free()ing) if the string passed in is a magic short-hand form
- * to name a branch.
- */
-static char *substitute_branch_name(const char **string, int *len)
-{
-	struct strbuf buf = STRBUF_INIT;
-	int ret = interpret_branch_name(*string, *len, &buf);
-
-	if (ret == *len) {
-		size_t size;
-		*string = strbuf_detach(&buf, &size);
-		*len = size;
-		return (char *)*string;
-	}
-
-	return NULL;
-}
-
-int dwim_ref(const char *str, int len, unsigned char *sha1, char **ref)
-{
-	char *last_branch = substitute_branch_name(&str, &len);
-	const char **p, *r;
-	int refs_found = 0;
-
-	*ref = NULL;
-	for (p = ref_rev_parse_rules; *p; p++) {
-		char fullref[PATH_MAX];
-		unsigned char sha1_from_ref[20];
-		unsigned char *this_result;
-		int flag;
-
-		this_result = refs_found ? sha1_from_ref : sha1;
-		mksnpath(fullref, sizeof(fullref), *p, len, str);
-		r = resolve_ref_unsafe(fullref, RESOLVE_REF_READING,
-				       this_result, &flag);
-		if (r) {
-			if (!refs_found++)
-				*ref = xstrdup(r);
-			if (!warn_ambiguous_refs)
-				break;
-		} else if ((flag & REF_ISSYMREF) && strcmp(fullref, "HEAD")) {
-			warning("ignoring dangling symref %s.", fullref);
-		} else if ((flag & REF_ISBROKEN) && strchr(fullref, '/')) {
-			warning("ignoring broken ref %s.", fullref);
-		}
-	}
-	free(last_branch);
-	return refs_found;
-}
-
-int dwim_log(const char *str, int len, unsigned char *sha1, char **log)
-{
-	char *last_branch = substitute_branch_name(&str, &len);
-	const char **p;
-	int logs_found = 0;
-
-	*log = NULL;
-	for (p = ref_rev_parse_rules; *p; p++) {
-		unsigned char hash[20];
-		char path[PATH_MAX];
-		const char *ref, *it;
-
-		mksnpath(path, sizeof(path), *p, len, str);
-		ref = resolve_ref_unsafe(path, RESOLVE_REF_READING,
-					 hash, NULL);
-		if (!ref)
-			continue;
-		if (reflog_exists(path))
-			it = path;
-		else if (strcmp(ref, path) && reflog_exists(ref))
-			it = ref;
-		else
-			continue;
-		if (!logs_found++) {
-			*log = xstrdup(it);
-			hashcpy(sha1, hash);
-		}
-		if (!warn_ambiguous_refs)
-			break;
-	}
-	free(last_branch);
-	return logs_found;
-}
-
-/*
  * Locks a ref returning the lock on success and NULL on failure.
  * On failure errno is set to something meaningful.
  */
@@ -4076,99 +3966,6 @@ cleanup:
 	return ret;
 }
 
-char *shorten_unambiguous_ref(const char *refname, int strict)
-{
-	int i;
-	static char **scanf_fmts;
-	static int nr_rules;
-	char *short_name;
-
-	if (!nr_rules) {
-		/*
-		 * Pre-generate scanf formats from ref_rev_parse_rules[].
-		 * Generate a format suitable for scanf from a
-		 * ref_rev_parse_rules rule by interpolating "%s" at the
-		 * location of the "%.*s".
-		 */
-		size_t total_len = 0;
-		size_t offset = 0;
-
-		/* the rule list is NULL terminated, count them first */
-		for (nr_rules = 0; ref_rev_parse_rules[nr_rules]; nr_rules++)
-			/* -2 for strlen("%.*s") - strlen("%s"); +1 for NUL */
-			total_len += strlen(ref_rev_parse_rules[nr_rules]) - 2 + 1;
-
-		scanf_fmts = xmalloc(nr_rules * sizeof(char *) + total_len);
-
-		offset = 0;
-		for (i = 0; i < nr_rules; i++) {
-			assert(offset < total_len);
-			scanf_fmts[i] = (char *)&scanf_fmts[nr_rules] + offset;
-			offset += snprintf(scanf_fmts[i], total_len - offset,
-					   ref_rev_parse_rules[i], 2, "%s") + 1;
-		}
-	}
-
-	/* bail out if there are no rules */
-	if (!nr_rules)
-		return xstrdup(refname);
-
-	/* buffer for scanf result, at most refname must fit */
-	short_name = xstrdup(refname);
-
-	/* skip first rule, it will always match */
-	for (i = nr_rules - 1; i > 0 ; --i) {
-		int j;
-		int rules_to_fail = i;
-		int short_name_len;
-
-		if (1 != sscanf(refname, scanf_fmts[i], short_name))
-			continue;
-
-		short_name_len = strlen(short_name);
-
-		/*
-		 * in strict mode, all (except the matched one) rules
-		 * must fail to resolve to a valid non-ambiguous ref
-		 */
-		if (strict)
-			rules_to_fail = nr_rules;
-
-		/*
-		 * check if the short name resolves to a valid ref,
-		 * but use only rules prior to the matched one
-		 */
-		for (j = 0; j < rules_to_fail; j++) {
-			const char *rule = ref_rev_parse_rules[j];
-			char refname[PATH_MAX];
-
-			/* skip matched rule */
-			if (i == j)
-				continue;
-
-			/*
-			 * the short name is ambiguous, if it resolves
-			 * (with this previous rule) to a valid ref
-			 * read_ref() returns 0 on success
-			 */
-			mksnpath(refname, sizeof(refname),
-				 rule, short_name_len, short_name);
-			if (ref_exists(refname))
-				break;
-		}
-
-		/*
-		 * short name is non-ambiguous if all previous rules
-		 * haven't resolved to a valid ref
-		 */
-		if (j == rules_to_fail)
-			return short_name;
-	}
-
-	free(short_name);
-	return xstrdup(refname);
-}
-
 struct expire_reflog_cb {
 	unsigned int flags;
 	reflog_expiry_should_prune_fn *should_prune_fn;
diff --git a/refs.c b/refs.c
index 6b2fc39..310b7f5 100644
--- a/refs.c
+++ b/refs.c
@@ -378,3 +378,206 @@ int ref_is_hidden(const char *refname)
 	}
 	return 0;
 }
+
+static const char *ref_rev_parse_rules[] = {
+	"%.*s",
+	"refs/%.*s",
+	"refs/tags/%.*s",
+	"refs/heads/%.*s",
+	"refs/remotes/%.*s",
+	"refs/remotes/%.*s/HEAD",
+	NULL
+};
+
+int refname_match(const char *abbrev_name, const char *full_name)
+{
+	const char **p;
+	const int abbrev_name_len = strlen(abbrev_name);
+
+	for (p = ref_rev_parse_rules; *p; p++) {
+		if (!strcmp(full_name, mkpath(*p, abbrev_name_len, abbrev_name))) {
+			return 1;
+		}
+	}
+
+	return 0;
+}
+
+/*
+ * *string and *len will only be substituted, and *string returned (for
+ * later free()ing) if the string passed in is a magic short-hand form
+ * to name a branch.
+ */
+static char *substitute_branch_name(const char **string, int *len)
+{
+	struct strbuf buf = STRBUF_INIT;
+	int ret = interpret_branch_name(*string, *len, &buf);
+
+	if (ret == *len) {
+		size_t size;
+		*string = strbuf_detach(&buf, &size);
+		*len = size;
+		return (char *)*string;
+	}
+
+	return NULL;
+}
+
+int dwim_ref(const char *str, int len, unsigned char *sha1, char **ref)
+{
+	char *last_branch = substitute_branch_name(&str, &len);
+	const char **p, *r;
+	int refs_found = 0;
+
+	*ref = NULL;
+	for (p = ref_rev_parse_rules; *p; p++) {
+		char fullref[PATH_MAX];
+		unsigned char sha1_from_ref[20];
+		unsigned char *this_result;
+		int flag;
+
+		this_result = refs_found ? sha1_from_ref : sha1;
+		mksnpath(fullref, sizeof(fullref), *p, len, str);
+		r = resolve_ref_unsafe(fullref, RESOLVE_REF_READING,
+				       this_result, &flag);
+		if (r) {
+			if (!refs_found++)
+				*ref = xstrdup(r);
+			if (!warn_ambiguous_refs)
+				break;
+		} else if ((flag & REF_ISSYMREF) && strcmp(fullref, "HEAD")) {
+			warning("ignoring dangling symref %s.", fullref);
+		} else if ((flag & REF_ISBROKEN) && strchr(fullref, '/')) {
+			warning("ignoring broken ref %s.", fullref);
+		}
+	}
+	free(last_branch);
+	return refs_found;
+}
+
+int dwim_log(const char *str, int len, unsigned char *sha1, char **log)
+{
+	char *last_branch = substitute_branch_name(&str, &len);
+	const char **p;
+	int logs_found = 0;
+
+	*log = NULL;
+	for (p = ref_rev_parse_rules; *p; p++) {
+		unsigned char hash[20];
+		char path[PATH_MAX];
+		const char *ref, *it;
+
+		mksnpath(path, sizeof(path), *p, len, str);
+		ref = resolve_ref_unsafe(path, RESOLVE_REF_READING,
+					 hash, NULL);
+		if (!ref)
+			continue;
+		if (reflog_exists(path))
+			it = path;
+		else if (strcmp(ref, path) && reflog_exists(ref))
+			it = ref;
+		else
+			continue;
+		if (!logs_found++) {
+			*log = xstrdup(it);
+			hashcpy(sha1, hash);
+		}
+		if (!warn_ambiguous_refs)
+			break;
+	}
+	free(last_branch);
+	return logs_found;
+}
+
+char *shorten_unambiguous_ref(const char *refname, int strict)
+{
+	int i;
+	static char **scanf_fmts;
+	static int nr_rules;
+	char *short_name;
+
+	if (!nr_rules) {
+		/*
+		 * Pre-generate scanf formats from ref_rev_parse_rules[].
+		 * Generate a format suitable for scanf from a
+		 * ref_rev_parse_rules rule by interpolating "%s" at the
+		 * location of the "%.*s".
+		 */
+		size_t total_len = 0;
+		size_t offset = 0;
+
+		/* the rule list is NULL terminated, count them first */
+		for (nr_rules = 0; ref_rev_parse_rules[nr_rules]; nr_rules++)
+			/* -2 for strlen("%.*s") - strlen("%s"); +1 for NUL */
+			total_len += strlen(ref_rev_parse_rules[nr_rules]) - 2 + 1;
+
+		scanf_fmts = xmalloc(nr_rules * sizeof(char *) + total_len);
+
+		offset = 0;
+		for (i = 0; i < nr_rules; i++) {
+			assert(offset < total_len);
+			scanf_fmts[i] = (char *)&scanf_fmts[nr_rules] + offset;
+			offset += snprintf(scanf_fmts[i], total_len - offset,
+					   ref_rev_parse_rules[i], 2, "%s") + 1;
+		}
+	}
+
+	/* bail out if there are no rules */
+	if (!nr_rules)
+		return xstrdup(refname);
+
+	/* buffer for scanf result, at most refname must fit */
+	short_name = xstrdup(refname);
+
+	/* skip first rule, it will always match */
+	for (i = nr_rules - 1; i > 0 ; --i) {
+		int j;
+		int rules_to_fail = i;
+		int short_name_len;
+
+		if (1 != sscanf(refname, scanf_fmts[i], short_name))
+			continue;
+
+		short_name_len = strlen(short_name);
+
+		/*
+		 * in strict mode, all (except the matched one) rules
+		 * must fail to resolve to a valid non-ambiguous ref
+		 */
+		if (strict)
+			rules_to_fail = nr_rules;
+
+		/*
+		 * check if the short name resolves to a valid ref,
+		 * but use only rules prior to the matched one
+		 */
+		for (j = 0; j < rules_to_fail; j++) {
+			const char *rule = ref_rev_parse_rules[j];
+			char refname[PATH_MAX];
+
+			/* skip matched rule */
+			if (i == j)
+				continue;
+
+			/*
+			 * the short name is ambiguous, if it resolves
+			 * (with this previous rule) to a valid ref
+			 * read_ref() returns 0 on success
+			 */
+			mksnpath(refname, sizeof(refname),
+				 rule, short_name_len, short_name);
+			if (ref_exists(refname))
+				break;
+		}
+
+		/*
+		 * short name is non-ambiguous if all previous rules
+		 * haven't resolved to a valid ref
+		 */
+		if (j == rules_to_fail)
+			return short_name;
+	}
+
+	free(short_name);
+	return xstrdup(refname);
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 10/44] refs.c: move warn_if_dangling_symref* to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

These functions do not use any backend specific code so we move
them to the common code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 52 ----------------------------------------------------
 refs.c          | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+), 52 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 37772a7..7d82864 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1944,58 +1944,6 @@ int peel_ref(const char *refname, unsigned char *sha1)
 	return peel_object(base, sha1);
 }
 
-struct warn_if_dangling_data {
-	FILE *fp;
-	const char *refname;
-	const struct string_list *refnames;
-	const char *msg_fmt;
-};
-
-static int warn_if_dangling_symref(const char *refname, const struct object_id *oid,
-				   int flags, void *cb_data)
-{
-	struct warn_if_dangling_data *d = cb_data;
-	const char *resolves_to;
-	struct object_id junk;
-
-	if (!(flags & REF_ISSYMREF))
-		return 0;
-
-	resolves_to = resolve_ref_unsafe(refname, 0, junk.hash, NULL);
-	if (!resolves_to
-	    || (d->refname
-		? strcmp(resolves_to, d->refname)
-		: !string_list_has_string(d->refnames, resolves_to))) {
-		return 0;
-	}
-
-	fprintf(d->fp, d->msg_fmt, refname);
-	fputc('\n', d->fp);
-	return 0;
-}
-
-void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)
-{
-	struct warn_if_dangling_data data;
-
-	data.fp = fp;
-	data.refname = refname;
-	data.refnames = NULL;
-	data.msg_fmt = msg_fmt;
-	for_each_rawref(warn_if_dangling_symref, &data);
-}
-
-void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_list *refnames)
-{
-	struct warn_if_dangling_data data;
-
-	data.fp = fp;
-	data.refname = NULL;
-	data.refnames = refnames;
-	data.msg_fmt = msg_fmt;
-	for_each_rawref(warn_if_dangling_symref, &data);
-}
-
 /*
  * Call fn for each reference in the specified ref_cache, omitting
  * references not in the containing_dir of base.  fn is called for all
diff --git a/refs.c b/refs.c
index 310b7f5..24d5e28 100644
--- a/refs.c
+++ b/refs.c
@@ -581,3 +581,55 @@ char *shorten_unambiguous_ref(const char *refname, int strict)
 	free(short_name);
 	return xstrdup(refname);
 }
+
+struct warn_if_dangling_data {
+	FILE *fp;
+	const char *refname;
+	const struct string_list *refnames;
+	const char *msg_fmt;
+};
+
+static int warn_if_dangling_symref(const char *refname, const struct object_id *oid,
+				   int flags, void *cb_data)
+{
+	struct warn_if_dangling_data *d = cb_data;
+	const char *resolves_to;
+	struct object_id junk;
+
+	if (!(flags & REF_ISSYMREF))
+		return 0;
+
+	resolves_to = resolve_ref_unsafe(refname, 0, junk.hash, NULL);
+	if (!resolves_to
+	    || (d->refname
+		? strcmp(resolves_to, d->refname)
+		: !string_list_has_string(d->refnames, resolves_to))) {
+		return 0;
+	}
+
+	fprintf(d->fp, d->msg_fmt, refname);
+	fputc('\n', d->fp);
+	return 0;
+}
+
+void warn_dangling_symref(FILE *fp, const char *msg_fmt, const char *refname)
+{
+	struct warn_if_dangling_data data;
+
+	data.fp = fp;
+	data.refname = refname;
+	data.refnames = NULL;
+	data.msg_fmt = msg_fmt;
+	for_each_rawref(warn_if_dangling_symref, &data);
+}
+
+void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_list *refnames)
+{
+	struct warn_if_dangling_data data;
+
+	data.fp = fp;
+	data.refname = NULL;
+	data.refnames = refnames;
+	data.msg_fmt = msg_fmt;
+	for_each_rawref(warn_if_dangling_symref, &data);
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 11/44] refs.c: move read_ref, read_ref_full and ref_exists to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

These functions do not depend on the backend implementation so we
move them to the common code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 18 ------------------
 refs.c          | 18 ++++++++++++++++++
 2 files changed, 18 insertions(+), 18 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 7d82864..bd0dbb5 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1791,24 +1791,6 @@ struct ref_filter {
 	void *cb_data;
 };
 
-int read_ref_full(const char *refname, int resolve_flags, unsigned char *sha1, int *flags)
-{
-	if (resolve_ref_unsafe(refname, resolve_flags, sha1, flags))
-		return 0;
-	return -1;
-}
-
-int read_ref(const char *refname, unsigned char *sha1)
-{
-	return read_ref_full(refname, RESOLVE_REF_READING, sha1, NULL);
-}
-
-int ref_exists(const char *refname)
-{
-	unsigned char sha1[20];
-	return !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);
-}
-
 static int filter_refs(const char *refname, const struct object_id *oid,
 			   int flags, void *data)
 {
diff --git a/refs.c b/refs.c
index 24d5e28..242f66d 100644
--- a/refs.c
+++ b/refs.c
@@ -633,3 +633,21 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct string_li
 	data.msg_fmt = msg_fmt;
 	for_each_rawref(warn_if_dangling_symref, &data);
 }
+
+int read_ref_full(const char *refname, int resolve_flags, unsigned char *sha1, int *flags)
+{
+	if (resolve_ref_unsafe(refname, resolve_flags, sha1, flags))
+		return 0;
+	return -1;
+}
+
+int read_ref(const char *refname, unsigned char *sha1)
+{
+	return read_ref_full(refname, RESOLVE_REF_READING, sha1, NULL);
+}
+
+int ref_exists(const char *refname)
+{
+	unsigned char sha1[20];
+	return !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 12/44] refs.c: move resolve_refdup to common

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

This function can be shared across all refs backends so move it
to the common code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 7 -------
 refs.c          | 7 +++++++
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index bd0dbb5..045bf09 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1777,13 +1777,6 @@ const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 	return ret;
 }
 
-char *resolve_refdup(const char *refname, int resolve_flags,
-		     unsigned char *sha1, int *flags)
-{
-	return xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,
-						  sha1, flags));
-}
-
 /* The argument to filter_refs */
 struct ref_filter {
 	const char *pattern;
diff --git a/refs.c b/refs.c
index 242f66d..039e5c0 100644
--- a/refs.c
+++ b/refs.c
@@ -651,3 +651,10 @@ int ref_exists(const char *refname)
 	unsigned char sha1[20];
 	return !!resolve_ref_unsafe(refname, RESOLVE_REF_READING, sha1, NULL);
 }
+
+char *resolve_refdup(const char *refname, int resolve_flags,
+		     unsigned char *sha1, int *flags)
+{
+	return xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,
+						  sha1, flags));
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 13/44] refs.c: move check_refname_format to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

This function does not contain any backend specific code so we
move it to the common code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 109 --------------------------------------------------------
 refs.c          | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 109 insertions(+), 109 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 045bf09..9c57dcc 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -14,27 +14,6 @@ struct ref_lock {
 };
 
 /*
- * How to handle various characters in refnames:
- * 0: An acceptable character for refs
- * 1: End-of-component
- * 2: ., look for a preceding . to reject .. in refs
- * 3: {, look for a preceding @ to reject @{ in refs
- * 4: A bad character: ASCII control characters, and
- *    ":", "?", "[", "\", "^", "~", SP, or TAB
- * 5: *, reject unless REFNAME_REFSPEC_PATTERN is set
- */
-static unsigned char refname_disposition[256] = {
-	1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
-	4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
-	4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 2, 1,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 4,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 4, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
-	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 4, 4
-};
-
-/*
  * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken
  * refs (i.e., because the reference is about to be deleted anyway).
  */
@@ -69,94 +48,6 @@ static unsigned char refname_disposition[256] = {
  * value to ref_update::flags
  */
 
-/*
- * Try to read one refname component from the front of refname.
- * Return the length of the component found, or -1 if the component is
- * not legal.  It is legal if it is something reasonable to have under
- * ".git/refs/"; We do not like it if:
- *
- * - any path component of it begins with ".", or
- * - it has double dots "..", or
- * - it has ASCII control characters, or
- * - it has ":", "?", "[", "\", "^", "~", SP, or TAB anywhere, or
- * - it has "*" anywhere unless REFNAME_REFSPEC_PATTERN is set, or
- * - it ends with a "/", or
- * - it ends with ".lock", or
- * - it contains a "@{" portion
- */
-static int check_refname_component(const char *refname, int *flags)
-{
-	const char *cp;
-	char last = '\0';
-
-	for (cp = refname; ; cp++) {
-		int ch = *cp & 255;
-		unsigned char disp = refname_disposition[ch];
-		switch (disp) {
-		case 1:
-			goto out;
-		case 2:
-			if (last == '.')
-				return -1; /* Refname contains "..". */
-			break;
-		case 3:
-			if (last == '@')
-				return -1; /* Refname contains "@{". */
-			break;
-		case 4:
-			return -1;
-		case 5:
-			if (!(*flags & REFNAME_REFSPEC_PATTERN))
-				return -1; /* refspec can't be a pattern */
-
-			/*
-			 * Unset the pattern flag so that we only accept
-			 * a single asterisk for one side of refspec.
-			 */
-			*flags &= ~ REFNAME_REFSPEC_PATTERN;
-			break;
-		}
-		last = ch;
-	}
-out:
-	if (cp == refname)
-		return 0; /* Component has zero length. */
-	if (refname[0] == '.')
-		return -1; /* Component starts with '.'. */
-	if (cp - refname >= LOCK_SUFFIX_LEN &&
-	    !memcmp(cp - LOCK_SUFFIX_LEN, LOCK_SUFFIX, LOCK_SUFFIX_LEN))
-		return -1; /* Refname ends with ".lock". */
-	return cp - refname;
-}
-
-int check_refname_format(const char *refname, int flags)
-{
-	int component_len, component_count = 0;
-
-	if (!strcmp(refname, "@"))
-		/* Refname is a single character '@'. */
-		return -1;
-
-	while (1) {
-		/* We are at the start of a path component. */
-		component_len = check_refname_component(refname, &flags);
-		if (component_len <= 0)
-			return -1;
-
-		component_count++;
-		if (refname[component_len] == '\0')
-			break;
-		/* Skip to next component. */
-		refname += component_len + 1;
-	}
-
-	if (refname[component_len - 1] == '.')
-		return -1; /* Refname ends with '.'. */
-	if (!(flags & REFNAME_ALLOW_ONELEVEL) && component_count < 2)
-		return -1; /* Refname has only one component. */
-	return 0;
-}
-
 struct ref_entry;
 
 /*
diff --git a/refs.c b/refs.c
index 039e5c0..e9cc2d4 100644
--- a/refs.c
+++ b/refs.c
@@ -658,3 +658,112 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 	return xstrdup_or_null(resolve_ref_unsafe(refname, resolve_flags,
 						  sha1, flags));
 }
+
+/*
+ * How to handle various characters in refnames:
+ * 0: An acceptable character for refs
+ * 1: End-of-component
+ * 2: ., look for a preceding . to reject .. in refs
+ * 3: {, look for a preceding @ to reject @{ in refs
+ * 4: A bad character: ASCII control characters, and
+ *    ":", "?", "[", "\", "^", "~", SP, or TAB
+ * 5: *, reject unless REFNAME_REFSPEC_PATTERN is set
+ */
+static unsigned char refname_disposition[256] = {
+	1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+	4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+	4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 2, 1,
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 4,
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 4, 0, 4, 0,
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 4, 4
+};
+
+/*
+ * Try to read one refname component from the front of refname.
+ * Return the length of the component found, or -1 if the component is
+ * not legal.  It is legal if it is something reasonable to have under
+ * ".git/refs/"; We do not like it if:
+ *
+ * - any path component of it begins with ".", or
+ * - it has double dots "..", or
+ * - it has ASCII control characters, or
+ * - it has ":", "?", "[", "\", "^", "~", SP, or TAB anywhere, or
+ * - it has "*" anywhere unless REFNAME_REFSPEC_PATTERN is set, or
+ * - it ends with a "/", or
+ * - it ends with ".lock", or
+ * - it contains a "@{" portion
+ */
+static int check_refname_component(const char *refname, int *flags)
+{
+	const char *cp;
+	char last = '\0';
+
+	for (cp = refname; ; cp++) {
+		int ch = *cp & 255;
+		unsigned char disp = refname_disposition[ch];
+		switch (disp) {
+		case 1:
+			goto out;
+		case 2:
+			if (last == '.')
+				return -1; /* Refname contains "..". */
+			break;
+		case 3:
+			if (last == '@')
+				return -1; /* Refname contains "@{". */
+			break;
+		case 4:
+			return -1;
+		case 5:
+			if (!(*flags & REFNAME_REFSPEC_PATTERN))
+				return -1; /* refspec can't be a pattern */
+
+			/*
+			 * Unset the pattern flag so that we only accept
+			 * a single asterisk for one side of refspec.
+			 */
+			*flags &= ~ REFNAME_REFSPEC_PATTERN;
+			break;
+		}
+		last = ch;
+	}
+out:
+	if (cp == refname)
+		return 0; /* Component has zero length. */
+	if (refname[0] == '.')
+		return -1; /* Component starts with '.'. */
+	if (cp - refname >= LOCK_SUFFIX_LEN &&
+	    !memcmp(cp - LOCK_SUFFIX_LEN, LOCK_SUFFIX, LOCK_SUFFIX_LEN))
+		return -1; /* Refname ends with ".lock". */
+	return cp - refname;
+}
+
+int check_refname_format(const char *refname, int flags)
+{
+	int component_len, component_count = 0;
+
+	if (!strcmp(refname, "@"))
+		/* Refname is a single character '@'. */
+		return -1;
+
+	while (1) {
+		/* We are at the start of a path component. */
+		component_len = check_refname_component(refname, &flags);
+		if (component_len <= 0)
+			return -1;
+
+		component_count++;
+		if (refname[component_len] == '\0')
+			break;
+		/* Skip to next component. */
+		refname += component_len + 1;
+	}
+
+	if (refname[component_len - 1] == '.')
+		return -1; /* Refname ends with '.'. */
+	if (!(flags & REFNAME_ALLOW_ONELEVEL) && component_count < 2)
+		return -1; /* Refname has only one component. */
+	return 0;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 14/44] refs.c: move is_branch to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 5 -----
 refs.c          | 5 +++++
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 9c57dcc..1214d9e 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2957,11 +2957,6 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,
 	return ret;
 }
 
-int is_branch(const char *refname)
-{
-	return !strcmp(refname, "HEAD") || starts_with(refname, "refs/heads/");
-}
-
 /*
  * Write sha1 into the open lockfile, then close the lockfile. On
  * errors, rollback the lockfile, fill in *err and
diff --git a/refs.c b/refs.c
index e9cc2d4..bc8750c 100644
--- a/refs.c
+++ b/refs.c
@@ -767,3 +767,8 @@ int check_refname_format(const char *refname, int flags)
 		return -1; /* Refname has only one component. */
 	return 0;
 }
+
+int is_branch(const char *refname)
+{
+	return !strcmp(refname, "HEAD") || starts_with(refname, "refs/heads/");
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 15/44] refs.c: move prettify_refname to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs-be-files.c | 9 ---------
 refs.c          | 9 +++++++++
 2 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 1214d9e..ecf2b33 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2052,15 +2052,6 @@ int for_each_rawref(each_ref_fn fn, void *cb_data)
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
 }
 
-const char *prettify_refname(const char *name)
-{
-	return name + (
-		starts_with(name, "refs/heads/") ? 11 :
-		starts_with(name, "refs/tags/") ? 10 :
-		starts_with(name, "refs/remotes/") ? 13 :
-		0);
-}
-
 static void unlock_ref(struct ref_lock *lock)
 {
 	/* Do not free lock->lk -- atexit() still looks at them */
diff --git a/refs.c b/refs.c
index bc8750c..44ee4f4 100644
--- a/refs.c
+++ b/refs.c
@@ -772,3 +772,12 @@ int is_branch(const char *refname)
 {
 	return !strcmp(refname, "HEAD") || starts_with(refname, "refs/heads/");
 }
+
+const char *prettify_refname(const char *name)
+{
+	return name + (
+		starts_with(name, "refs/heads/") ? 11 :
+		starts_with(name, "refs/tags/") ? 10 :
+		starts_with(name, "refs/remotes/") ? 13 :
+		0);
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 16/44] refs.c: move ref iterators to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs-be-files.c | 82 ---------------------------------------------------------
 refs.c          | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 81 insertions(+), 82 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index ecf2b33..46fa43c 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1668,23 +1668,6 @@ const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 	return ret;
 }
 
-/* The argument to filter_refs */
-struct ref_filter {
-	const char *pattern;
-	each_ref_fn *fn;
-	void *cb_data;
-};
-
-static int filter_refs(const char *refname, const struct object_id *oid,
-			   int flags, void *data)
-{
-	struct ref_filter *filter = (struct ref_filter *)data;
-
-	if (wildmatch(filter->pattern, refname, 0, NULL))
-		return 0;
-	return filter->fn(refname, oid, flags, filter->cb_data);
-}
-
 enum peel_status {
 	/* object was peeled successfully: */
 	PEEL_PEELED = 0,
@@ -1950,37 +1933,6 @@ int for_each_ref_in_submodule(const char *submodule, const char *prefix,
 {
 	return do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);
 }
-
-int for_each_tag_ref(each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in("refs/tags/", fn, cb_data);
-}
-
-int for_each_tag_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in_submodule(submodule, "refs/tags/", fn, cb_data);
-}
-
-int for_each_branch_ref(each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in("refs/heads/", fn, cb_data);
-}
-
-int for_each_branch_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in_submodule(submodule, "refs/heads/", fn, cb_data);
-}
-
-int for_each_remote_ref(each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in("refs/remotes/", fn, cb_data);
-}
-
-int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
-{
-	return for_each_ref_in_submodule(submodule, "refs/remotes/", fn, cb_data);
-}
-
 int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, git_replace_ref_base, fn,
@@ -2012,40 +1964,6 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	return ret;
 }
 
-int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
-	const char *prefix, void *cb_data)
-{
-	struct strbuf real_pattern = STRBUF_INIT;
-	struct ref_filter filter;
-	int ret;
-
-	if (!prefix && !starts_with(pattern, "refs/"))
-		strbuf_addstr(&real_pattern, "refs/");
-	else if (prefix)
-		strbuf_addstr(&real_pattern, prefix);
-	strbuf_addstr(&real_pattern, pattern);
-
-	if (!has_glob_specials(pattern)) {
-		/* Append implied '/' '*' if not present. */
-		strbuf_complete(&real_pattern, '/');
-		/* No need to check for '*', there is none. */
-		strbuf_addch(&real_pattern, '*');
-	}
-
-	filter.pattern = real_pattern.buf;
-	filter.fn = fn;
-	filter.cb_data = cb_data;
-	ret = for_each_ref(filter_refs, &filter);
-
-	strbuf_release(&real_pattern);
-	return ret;
-}
-
-int for_each_glob_ref(each_ref_fn fn, const char *pattern, void *cb_data)
-{
-	return for_each_glob_ref_in(fn, pattern, NULL, cb_data);
-}
-
 int for_each_rawref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, "", fn, 0,
diff --git a/refs.c b/refs.c
index 44ee4f4..7714dad 100644
--- a/refs.c
+++ b/refs.c
@@ -781,3 +781,84 @@ const char *prettify_refname(const char *name)
 		starts_with(name, "refs/remotes/") ? 13 :
 		0);
 }
+
+/* The argument to filter_refs */
+struct ref_filter {
+	const char *pattern;
+	each_ref_fn *fn;
+	void *cb_data;
+};
+
+static int filter_refs(const char *refname, const struct object_id *oid, int flags,
+		       void *data)
+{
+	struct ref_filter *filter = (struct ref_filter *)data;
+	if (wildmatch(filter->pattern, refname, 0, NULL))
+		return 0;
+	return filter->fn(refname, oid, flags, filter->cb_data);
+}
+
+int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
+	const char *prefix, void *cb_data)
+{
+	struct strbuf real_pattern = STRBUF_INIT;
+	struct ref_filter filter;
+	int ret;
+
+	if (!prefix && !starts_with(pattern, "refs/"))
+		strbuf_addstr(&real_pattern, "refs/");
+	else if (prefix)
+		strbuf_addstr(&real_pattern, prefix);
+	strbuf_addstr(&real_pattern, pattern);
+
+	if (!has_glob_specials(pattern)) {
+		/* Append implied '/' '*' if not present. */
+		if (real_pattern.buf[real_pattern.len - 1] != '/')
+			strbuf_addch(&real_pattern, '/');
+		/* No need to check for '*', there is none. */
+		strbuf_addch(&real_pattern, '*');
+	}
+
+	filter.pattern = real_pattern.buf;
+	filter.fn = fn;
+	filter.cb_data = cb_data;
+	ret = for_each_ref(filter_refs, &filter);
+
+	strbuf_release(&real_pattern);
+	return ret;
+}
+
+int for_each_glob_ref(each_ref_fn fn, const char *pattern, void *cb_data)
+{
+	return for_each_glob_ref_in(fn, pattern, NULL, cb_data);
+}
+
+int for_each_tag_ref(each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in("refs/tags/", fn, cb_data);
+}
+
+int for_each_tag_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in_submodule(submodule, "refs/tags/", fn, cb_data);
+}
+
+int for_each_branch_ref(each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in("refs/heads/", fn, cb_data);
+}
+
+int for_each_branch_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in_submodule(submodule, "refs/heads/", fn, cb_data);
+}
+
+int for_each_remote_ref(each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in("refs/remotes/", fn, cb_data);
+}
+
+int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	return for_each_ref_in_submodule(submodule, "refs/remotes/", fn, cb_data);
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 17/44] refs.c: move head_ref_namespaced to the common code

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs-be-files.c | 15 ---------------
 refs.c          | 15 +++++++++++++++
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 46fa43c..34ed220 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1939,21 +1939,6 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 			       strlen(git_replace_ref_base), 0, cb_data);
 }
 
-int head_ref_namespaced(each_ref_fn fn, void *cb_data)
-{
-	struct strbuf buf = STRBUF_INIT;
-	int ret = 0;
-	struct object_id oid;
-	int flag;
-
-	strbuf_addf(&buf, "%sHEAD", get_git_namespace());
-	if (!read_ref_full(buf.buf, RESOLVE_REF_READING, oid.hash, &flag))
-		ret = fn(buf.buf, &oid, flag, cb_data);
-	strbuf_release(&buf);
-
-	return ret;
-}
-
 int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
diff --git a/refs.c b/refs.c
index 7714dad..0f4e19a 100644
--- a/refs.c
+++ b/refs.c
@@ -862,3 +862,18 @@ int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *c
 {
 	return for_each_ref_in_submodule(submodule, "refs/remotes/", fn, cb_data);
 }
+
+int head_ref_namespaced(each_ref_fn fn, void *cb_data)
+{
+	struct strbuf buf = STRBUF_INIT;
+	int ret = 0;
+	struct object_id oid;
+	int flag;
+
+	strbuf_addf(&buf, "%sHEAD", get_git_namespace());
+	if (!read_ref_full(buf.buf, RESOLVE_REF_READING, oid.hash, &flag))
+		ret = fn(buf.buf, &oid, flag, cb_data);
+	strbuf_release(&buf);
+
+	return ret;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 18/44] refs: move transaction functions into common code

[David Turner]

The common ref code will build up a ref transaction.  Backends will
then commit it.  So the transaction creation and update functions should
be in the common code.  We also need to move the ref structs into
the common code so that alternate backends can access them.

Later, we will modify struct ref_update to support alternate backends.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 198 --------------------------------------------------------
 refs.c          | 108 +++++++++++++++++++++++++++++++
 refs.h          |  91 +++++++++++++++++++++++++-
 3 files changed, 198 insertions(+), 199 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 34ed220..1308955 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -13,41 +13,6 @@ struct ref_lock {
 	struct object_id old_oid;
 };
 
-/*
- * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken
- * refs (i.e., because the reference is about to be deleted anyway).
- */
-#define REF_DELETING	0x02
-
-/*
- * Used as a flag in ref_update::flags when a loose ref is being
- * pruned.
- */
-#define REF_ISPRUNING	0x04
-
-/*
- * Used as a flag in ref_update::flags when the reference should be
- * updated to new_sha1.
- */
-#define REF_HAVE_NEW	0x08
-
-/*
- * Used as a flag in ref_update::flags when old_sha1 should be
- * checked.
- */
-#define REF_HAVE_OLD	0x10
-
-/*
- * Used as a flag in ref_update::flags when the lockfile needs to be
- * committed.
- */
-#define REF_NEEDS_COMMIT 0x20
-
-/*
- * 0x40 is REF_FORCE_CREATE_REFLOG, so skip it if you're adding a
- * value to ref_update::flags
- */
-
 struct ref_entry;
 
 /*
@@ -3243,169 +3208,6 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)
 	return retval;
 }
 
-/**
- * Information needed for a single ref update. Set new_sha1 to the new
- * value or to null_sha1 to delete the ref. To check the old value
- * while the ref is locked, set (flags & REF_HAVE_OLD) and set
- * old_sha1 to the old value, or to null_sha1 to ensure the ref does
- * not exist before update.
- */
-struct ref_update {
-	/*
-	 * If (flags & REF_HAVE_NEW), set the reference to this value:
-	 */
-	unsigned char new_sha1[20];
-	/*
-	 * If (flags & REF_HAVE_OLD), check that the reference
-	 * previously had this value:
-	 */
-	unsigned char old_sha1[20];
-	/*
-	 * One or more of REF_HAVE_NEW, REF_HAVE_OLD, REF_NODEREF,
-	 * REF_DELETING, and REF_ISPRUNING:
-	 */
-	unsigned int flags;
-	struct ref_lock *lock;
-	int type;
-	char *msg;
-	const char refname[FLEX_ARRAY];
-};
-
-/*
- * Transaction states.
- * OPEN:   The transaction is in a valid state and can accept new updates.
- *         An OPEN transaction can be committed.
- * CLOSED: A closed transaction is no longer active and no other operations
- *         than free can be used on it in this state.
- *         A transaction can either become closed by successfully committing
- *         an active transaction or if there is a failure while building
- *         the transaction thus rendering it failed/inactive.
- */
-enum ref_transaction_state {
-	REF_TRANSACTION_OPEN   = 0,
-	REF_TRANSACTION_CLOSED = 1
-};
-
-/*
- * Data structure for holding a reference transaction, which can
- * consist of checks and updates to multiple references, carried out
- * as atomically as possible.  This structure is opaque to callers.
- */
-struct ref_transaction {
-	struct ref_update **updates;
-	size_t alloc;
-	size_t nr;
-	enum ref_transaction_state state;
-};
-
-struct ref_transaction *ref_transaction_begin(struct strbuf *err)
-{
-	assert(err);
-
-	return xcalloc(1, sizeof(struct ref_transaction));
-}
-
-void ref_transaction_free(struct ref_transaction *transaction)
-{
-	int i;
-
-	if (!transaction)
-		return;
-
-	for (i = 0; i < transaction->nr; i++) {
-		free(transaction->updates[i]->msg);
-		free(transaction->updates[i]);
-	}
-	free(transaction->updates);
-	free(transaction);
-}
-
-static struct ref_update *add_update(struct ref_transaction *transaction,
-				     const char *refname)
-{
-	size_t len = strlen(refname) + 1;
-	struct ref_update *update = xcalloc(1, sizeof(*update) + len);
-
-	memcpy((char *)update->refname, refname, len); /* includes NUL */
-	ALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);
-	transaction->updates[transaction->nr++] = update;
-	return update;
-}
-
-int ref_transaction_update(struct ref_transaction *transaction,
-			   const char *refname,
-			   const unsigned char *new_sha1,
-			   const unsigned char *old_sha1,
-			   unsigned int flags, const char *msg,
-			   struct strbuf *err)
-{
-	struct ref_update *update;
-
-	assert(err);
-
-	if (transaction->state != REF_TRANSACTION_OPEN)
-		die("BUG: update called for transaction that is not open");
-
-	if (new_sha1 && !is_null_sha1(new_sha1) &&
-	    check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
-		strbuf_addf(err, "refusing to update ref with bad name %s",
-			    refname);
-		return -1;
-	}
-
-	update = add_update(transaction, refname);
-	if (new_sha1) {
-		hashcpy(update->new_sha1, new_sha1);
-		flags |= REF_HAVE_NEW;
-	}
-	if (old_sha1) {
-		hashcpy(update->old_sha1, old_sha1);
-		flags |= REF_HAVE_OLD;
-	}
-	update->flags = flags;
-	if (msg)
-		update->msg = xstrdup(msg);
-	return 0;
-}
-
-int ref_transaction_create(struct ref_transaction *transaction,
-			   const char *refname,
-			   const unsigned char *new_sha1,
-			   unsigned int flags, const char *msg,
-			   struct strbuf *err)
-{
-	if (!new_sha1 || is_null_sha1(new_sha1))
-		die("BUG: create called without valid new_sha1");
-	return ref_transaction_update(transaction, refname, new_sha1,
-				      null_sha1, flags, msg, err);
-}
-
-int ref_transaction_delete(struct ref_transaction *transaction,
-			   const char *refname,
-			   const unsigned char *old_sha1,
-			   unsigned int flags, const char *msg,
-			   struct strbuf *err)
-{
-	if (old_sha1 && is_null_sha1(old_sha1))
-		die("BUG: delete called with old_sha1 set to zeros");
-	return ref_transaction_update(transaction, refname,
-				      null_sha1, old_sha1,
-				      flags, msg, err);
-}
-
-int ref_transaction_verify(struct ref_transaction *transaction,
-			   const char *refname,
-			   const unsigned char *old_sha1,
-			   unsigned int flags,
-			   struct strbuf *err)
-{
-	if (!old_sha1)
-		die("BUG: verify called with old_sha1 set to NULL");
-	return ref_transaction_update(transaction, refname,
-				      NULL, old_sha1,
-				      flags, NULL, err);
-}
-
 static int ref_update_reject_duplicates(struct string_list *refnames,
 					struct strbuf *err)
 {
diff --git a/refs.c b/refs.c
index 0f4e19a..25ad3b5 100644
--- a/refs.c
+++ b/refs.c
@@ -877,3 +877,111 @@ int head_ref_namespaced(each_ref_fn fn, void *cb_data)
 
 	return ret;
 }
+
+struct ref_transaction *ref_transaction_begin(struct strbuf *err)
+{
+	assert(err);
+
+	return xcalloc(1, sizeof(struct ref_transaction));
+}
+
+void ref_transaction_free(struct ref_transaction *transaction)
+{
+	int i;
+
+	if (!transaction)
+		return;
+
+	for (i = 0; i < transaction->nr; i++) {
+		free(transaction->updates[i]->msg);
+		free(transaction->updates[i]);
+	}
+	free(transaction->updates);
+	free(transaction);
+}
+
+static struct ref_update *add_update(struct ref_transaction *transaction,
+				     const char *refname)
+{
+	size_t len = strlen(refname) + 1;
+	struct ref_update *update = xcalloc(1, sizeof(*update) + len);
+
+	memcpy((char *)update->refname, refname, len); /* includes NUL */
+	ALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);
+	transaction->updates[transaction->nr++] = update;
+	return update;
+}
+
+int ref_transaction_update(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *new_sha1,
+			   const unsigned char *old_sha1,
+			   unsigned int flags, const char *msg,
+			   struct strbuf *err)
+{
+	struct ref_update *update;
+
+	assert(err);
+
+	if (transaction->state != REF_TRANSACTION_OPEN)
+		die("BUG: update called for transaction that is not open");
+
+	if (new_sha1 && !is_null_sha1(new_sha1) &&
+	    check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
+		strbuf_addf(err, "refusing to update ref with bad name %s",
+			    refname);
+		return -1;
+	}
+
+	update = add_update(transaction, refname);
+	if (new_sha1) {
+		hashcpy(update->new_sha1, new_sha1);
+		flags |= REF_HAVE_NEW;
+	}
+	if (old_sha1) {
+		hashcpy(update->old_sha1, old_sha1);
+		flags |= REF_HAVE_OLD;
+	}
+	update->flags = flags;
+	if (msg)
+		update->msg = xstrdup(msg);
+	return 0;
+}
+
+int ref_transaction_create(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *new_sha1,
+			   unsigned int flags, const char *msg,
+			   struct strbuf *err)
+{
+	if (!new_sha1 || is_null_sha1(new_sha1))
+		die("BUG: create called without valid new_sha1");
+	return ref_transaction_update(transaction, refname, new_sha1,
+				      null_sha1, flags, msg, err);
+}
+
+int ref_transaction_delete(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *old_sha1,
+			   unsigned int flags, const char *msg,
+			   struct strbuf *err)
+{
+	if (old_sha1 && is_null_sha1(old_sha1))
+		die("BUG: delete called with old_sha1 set to zeros");
+	return ref_transaction_update(transaction, refname,
+				      null_sha1, old_sha1,
+				      flags, msg, err);
+}
+
+int ref_transaction_verify(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *old_sha1,
+			   unsigned int flags,
+			   struct strbuf *err)
+{
+	if (!old_sha1)
+		die("BUG: verify called with old_sha1 set to NULL");
+	return ref_transaction_update(transaction, refname,
+				      NULL, old_sha1,
+				      flags, NULL, err);
+}
diff --git a/refs.h b/refs.h
index 729bc3c..4940ae9 100644
--- a/refs.h
+++ b/refs.h
@@ -94,6 +94,84 @@ extern int dwim_ref(const char *str, int len, unsigned char *sha1, char **ref);
 extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);
 
 /*
+ * Transaction states.
+ * OPEN:   The transaction is in a valid state and can accept new updates.
+ *         An OPEN transaction can be committed.
+ * CLOSED: A closed transaction is no longer active and no other operations
+ *         than free can be used on it in this state.
+ *         A transaction can either become closed by successfully committing
+ *         an active transaction or if there is a failure while building
+ *         the transaction thus rendering it failed/inactive.
+ */
+enum ref_transaction_state {
+	REF_TRANSACTION_OPEN   = 0,
+	REF_TRANSACTION_CLOSED = 1
+};
+
+/*
+ * Flag passed to lock_ref_sha1_basic() telling it to tolerate broken
+ * refs (i.e., because the reference is about to be deleted anyway).
+ */
+#define REF_DELETING	0x02
+
+/*
+ * Used as a flag in ref_update::flags when a loose ref is being
+ * pruned.
+ */
+#define REF_ISPRUNING	0x04
+
+/*
+ * Used as a flag in ref_update::flags when the reference should be
+ * updated to new_sha1.
+ */
+#define REF_HAVE_NEW	0x08
+
+/*
+ * Used as a flag in ref_update::flags when old_sha1 should be
+ * checked.
+ */
+#define REF_HAVE_OLD	0x10
+
+/*
+ * Used as a flag in ref_update::flags when the lockfile needs to be
+ * committed.
+ */
+#define REF_NEEDS_COMMIT 0x20
+
+/*
+ * 0x40 is REF_FORCE_CREATE_REFLOG, so skip it if you're adding a
+ * value to ref_update::flags
+ */
+
+/**
+ * Information needed for a single ref update. Set new_sha1 to the new
+ * value or to null_sha1 to delete the ref. To check the old value
+ * while the ref is locked, set (flags & REF_HAVE_OLD) and set
+ * old_sha1 to the old value, or to null_sha1 to ensure the ref does
+ * not exist before update.
+ */
+struct ref_update {
+	/*
+	 * If (flags & REF_HAVE_NEW), set the reference to this value:
+	 */
+	unsigned char new_sha1[20];
+	/*
+	 * If (flags & REF_HAVE_OLD), check that the reference
+	 * previously had this value:
+	 */
+	unsigned char old_sha1[20];
+	/*
+	 * One or more of REF_HAVE_NEW, REF_HAVE_OLD, REF_NODEREF,
+	 * REF_DELETING, and REF_ISPRUNING:
+	 */
+	unsigned int flags;
+	struct ref_lock *lock;
+	int type;
+	char *msg;
+	const char refname[FLEX_ARRAY];
+};
+
+/*
  * A ref_transaction represents a collection of ref updates
  * that should succeed or fail together.
  *
@@ -125,7 +203,18 @@ extern int dwim_log(const char *str, int len, unsigned char *sha1, char **ref);
  * The message is appended to err without first clearing err.
  * err will not be '\n' terminated.
  */
-struct ref_transaction;
+
+/*
+ * Data structure for holding a reference transaction, which can
+ * consist of checks and updates to multiple references, carried out
+ * as atomically as possible.  This structure is opaque to callers.
+ */
+struct ref_transaction {
+	struct ref_update **updates;
+	size_t alloc;
+	size_t nr;
+	enum ref_transaction_state state;
+};
 
 /*
  * Bit values set in the flags argument passed to each_ref_fn():
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 19/44] refs: add a backend method structure with transaction functions

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Add a ref structure for backend methods. Start by adding a method pointer
for the transaction commit function.

Add a function set_refs_backend to switch between backends. The files
based backend is the default.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 10 ++++++++--
 refs.c          | 30 ++++++++++++++++++++++++++++++
 refs.h          | 15 +++++++++++++++
 3 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 1308955..3050f1d 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3225,8 +3225,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-int ref_transaction_commit(struct ref_transaction *transaction,
-			   struct strbuf *err)
+static int files_transaction_commit(struct ref_transaction *transaction,
+				    struct strbuf *err)
 {
 	int ret = 0, i;
 	int n = transaction->nr;
@@ -3612,3 +3612,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
 	unlock_ref(lock);
 	return -1;
 }
+
+struct ref_be refs_be_files = {
+	NULL,
+	"files",
+	files_transaction_commit,
+};
diff --git a/refs.c b/refs.c
index 25ad3b5..f930fe0 100644
--- a/refs.c
+++ b/refs.c
@@ -4,6 +4,29 @@
 #include "cache.h"
 #include "refs.h"
 #include "lockfile.h"
+/*
+ * We always have a files backend and it is the default.
+ */
+struct ref_be *the_refs_backend = &refs_be_files;
+/*
+ * List of all available backends
+ */
+struct ref_be *refs_backends = &refs_be_files;
+
+/*
+ * This function is used to switch to an alternate backend.
+ */
+int set_refs_backend(const char *name)
+{
+	struct ref_be *be;
+
+	for (be = refs_backends; be; be = be->next)
+		if (!strcmp(be->name, name)) {
+			the_refs_backend = be;
+			return 0;
+		}
+	return 1;
+}
 
 static int is_per_worktree_ref(const char *refname)
 {
@@ -985,3 +1008,10 @@ int ref_transaction_verify(struct ref_transaction *transaction,
 				      NULL, old_sha1,
 				      flags, NULL, err);
 }
+
+/* backend functions */
+int ref_transaction_commit(struct ref_transaction *transaction,
+			   struct strbuf *err)
+{
+	return the_refs_backend->transaction_commit(transaction, err);
+}
diff --git a/refs.h b/refs.h
index 4940ae9..419abf4 100644
--- a/refs.h
+++ b/refs.h
@@ -619,4 +619,19 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 			 reflog_expiry_cleanup_fn cleanup_fn,
 			 void *policy_cb_data);
 
+/* refs backends */
+typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
+				      struct strbuf *err);
+typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
+
+struct ref_be {
+	struct ref_be *next;
+	const char *name;
+	ref_transaction_commit_fn *transaction_commit;
+};
+
+
+extern struct ref_be refs_be_files;
+int set_refs_backend(const char *name);
+
 #endif /* REFS_H */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 19/44] refs-be-files.c: add a backend method structure with transaction functions

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Add a ref structure for backend methods. Start by adding a method pointer
for the transaction commit function.

Add a function set_refs_backend to switch between backends. The files
based backend is the default.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 10 ++++++++--
 refs.c          | 30 ++++++++++++++++++++++++++++++
 refs.h          | 15 +++++++++++++++
 3 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 1308955..3050f1d 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3225,8 +3225,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
 	return 0;
 }
 
-int ref_transaction_commit(struct ref_transaction *transaction,
-			   struct strbuf *err)
+static int files_transaction_commit(struct ref_transaction *transaction,
+				    struct strbuf *err)
 {
 	int ret = 0, i;
 	int n = transaction->nr;
@@ -3612,3 +3612,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
 	unlock_ref(lock);
 	return -1;
 }
+
+struct ref_be refs_be_files = {
+	NULL,
+	"files",
+	files_transaction_commit,
+};
diff --git a/refs.c b/refs.c
index 25ad3b5..f930fe0 100644
--- a/refs.c
+++ b/refs.c
@@ -4,6 +4,29 @@
 #include "cache.h"
 #include "refs.h"
 #include "lockfile.h"
+/*
+ * We always have a files backend and it is the default.
+ */
+struct ref_be *the_refs_backend = &refs_be_files;
+/*
+ * List of all available backends
+ */
+struct ref_be *refs_backends = &refs_be_files;
+
+/*
+ * This function is used to switch to an alternate backend.
+ */
+int set_refs_backend(const char *name)
+{
+	struct ref_be *be;
+
+	for (be = refs_backends; be; be = be->next)
+		if (!strcmp(be->name, name)) {
+			the_refs_backend = be;
+			return 0;
+		}
+	return 1;
+}
 
 static int is_per_worktree_ref(const char *refname)
 {
@@ -985,3 +1008,10 @@ int ref_transaction_verify(struct ref_transaction *transaction,
 				      NULL, old_sha1,
 				      flags, NULL, err);
 }
+
+/* backend functions */
+int ref_transaction_commit(struct ref_transaction *transaction,
+			   struct strbuf *err)
+{
+	return the_refs_backend->transaction_commit(transaction, err);
+}
diff --git a/refs.h b/refs.h
index 4940ae9..419abf4 100644
--- a/refs.h
+++ b/refs.h
@@ -619,4 +619,19 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 			 reflog_expiry_cleanup_fn cleanup_fn,
 			 void *policy_cb_data);
 
+/* refs backends */
+typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
+				      struct strbuf *err);
+typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
+
+struct ref_be {
+	struct ref_be *next;
+	const char *name;
+	ref_transaction_commit_fn *transaction_commit;
+};
+
+
+extern struct ref_be refs_be_files;
+int set_refs_backend(const char *name);
+
 #endif /* REFS_H */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 20/44] refs-be-files.c: add methods for misc ref operations

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Add ref backend methods for:
resolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,
create_symref, resolve_gitlink_ref.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twitter.com>
---
 builtin/init-db.c |  1 +
 cache.h           |  7 +++++++
 refs-be-files.c   | 36 +++++++++++++++++++++++++++---------
 refs.c            | 47 +++++++++++++++++++++++++++++++++++++++++++++++
 refs.h            | 18 ++++++++++++++++++
 5 files changed, 100 insertions(+), 9 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index cf6a3c8..313d13f 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -8,6 +8,7 @@
 #include "builtin.h"
 #include "exec_cmd.h"
 #include "parse-options.h"
+#include "refs.h"
 
 #ifndef DEFAULT_GIT_TEMPLATE_DIR
 #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
diff --git a/cache.h b/cache.h
index 497d296..1c45b62 100644
--- a/cache.h
+++ b/cache.h
@@ -1104,6 +1104,13 @@ extern char *oid_to_hex(const struct object_id *oid);	/* same static buffer as s
 extern int interpret_branch_name(const char *str, int len, struct strbuf *);
 extern int get_sha1_mb(const char *str, unsigned char *sha1);
 
+/*
+ * Return true iff abbrev_name is a possible abbreviation for
+ * full_name according to the rules defined by ref_rev_parse_rules in
+ * refs.c.
+ */
+extern int refname_match(const char *abbrev_name, const char *full_name);
+
 extern int validate_headref(const char *ref);
 
 extern int base_name_compare(const char *name1, int len1, int mode1, const char *name2, int len2, int mode2);
diff --git a/refs-be-files.c b/refs-be-files.c
index 3050f1d..116d72d 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1386,7 +1386,8 @@ static int resolve_gitlink_ref_recursive(struct ref_cache *refs,
 	return resolve_gitlink_ref_recursive(refs, p, sha1, recursion+1);
 }
 
-int resolve_gitlink_ref(const char *path, const char *refname, unsigned char *sha1)
+static int files_resolve_gitlink_ref(const char *path, const char *refname,
+				     unsigned char *sha1)
 {
 	int len = strlen(path), retval;
 	char *submodule;
@@ -1618,8 +1619,10 @@ static const char *resolve_ref_1(const char *refname,
 	}
 }
 
-const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
-			       unsigned char *sha1, int *flags)
+static const char *files_resolve_ref_unsafe(const char *refname,
+					    int resolve_flags,
+					    unsigned char *sha1,
+					    int *flags)
 {
 	static struct strbuf sb_refname = STRBUF_INIT;
 	struct strbuf sb_contents = STRBUF_INIT;
@@ -1721,7 +1724,7 @@ static enum peel_status peel_entry(struct ref_entry *entry, int repeel)
 	return status;
 }
 
-int peel_ref(const char *refname, unsigned char *sha1)
+static int files_peel_ref(const char *refname, unsigned char *sha1)
 {
 	int flag;
 	unsigned char base[20];
@@ -2355,7 +2358,7 @@ static void prune_refs(struct ref_to_prune *r)
 	}
 }
 
-int pack_refs(unsigned int flags)
+static int files_pack_refs(unsigned int flags)
 {
 	struct pack_refs_cb_data cbdata;
 
@@ -2677,8 +2680,16 @@ static int should_autocreate_reflog(const char *refname)
 		!strcmp(refname, "HEAD");
 }
 
-int verify_refname_available(const char *newname, struct string_list *extra,
-			     struct string_list *skip, struct strbuf *err)
+/*
+ * Create a reflog for a ref.  If force_create = 0, the reflog will
+ * only be created for certain refs (those for which
+ * should_autocreate_reflog returns non-zero.  Otherwise, create it
+ * regardless of the ref name.  Fill in *err and return -1 on failure.
+ */
+static int files_verify_refname_available(const char *newname,
+					  struct string_list *extra,
+					  struct string_list *skip,
+					  struct strbuf *err)
 {
 	struct ref_dir *packed_refs = get_packed_refs(&ref_cache);
 	struct ref_dir *loose_refs = get_loose_refs(&ref_cache);
@@ -2913,8 +2924,9 @@ static int commit_ref_update(struct ref_lock *lock,
 	return 0;
 }
 
-int create_symref(const char *ref_target, const char *refs_heads_master,
-		  const char *logmsg)
+static int files_create_symref(const char *ref_target,
+			       const char *refs_heads_master,
+			       const char *logmsg)
 {
 	char *lockpath = NULL;
 	char ref[1000];
@@ -3617,4 +3629,10 @@ struct ref_be refs_be_files = {
 	NULL,
 	"files",
 	files_transaction_commit,
+	files_resolve_ref_unsafe,
+	files_verify_refname_available,
+	files_pack_refs,
+	files_peel_ref,
+	files_create_symref,
+	files_resolve_gitlink_ref,
 };
diff --git a/refs.c b/refs.c
index f930fe0..61d90f5 100644
--- a/refs.c
+++ b/refs.c
@@ -1015,3 +1015,50 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 {
 	return the_refs_backend->transaction_commit(transaction, err);
 }
+
+const char *resolve_ref_unsafe(const char *ref, int resolve_flags,
+			       unsigned char *sha1, int *flags)
+{
+	return the_refs_backend->resolve_ref_unsafe(ref, resolve_flags, sha1,
+						    flags);
+}
+
+int verify_refname_available(const char *refname, struct string_list *extra,
+			     struct string_list *skip, struct strbuf *err)
+{
+	return the_refs_backend->verify_refname_available(refname, extra, skip, err);
+}
+
+int pack_refs(unsigned int flags)
+{
+	return the_refs_backend->pack_refs(flags);
+}
+
+int peel_ref(const char *refname, unsigned char *sha1)
+{
+	return the_refs_backend->peel_ref(refname, sha1);
+}
+
+int create_symref(const char *ref_target, const char *refs_heads_master,
+		  const char *logmsg)
+{
+	struct strbuf err = STRBUF_INIT;
+	struct ref_transaction *transaction = ref_transaction_begin(&err);
+	if (!transaction)
+		return -1;
+
+	if (the_refs_backend->create_symref(transaction, ref_target,
+					    refs_heads_master, logmsg))
+		return -1;
+
+	if (ref_transaction_commit(transaction, &err))
+		return -1;
+
+	return 0;
+}
+
+int resolve_gitlink_ref(const char *path, const char *refname,
+			unsigned char *sha1)
+{
+	return the_refs_backend->resolve_gitlink_ref(path, refname, sha1);
+}
diff --git a/refs.h b/refs.h
index 419abf4..ee06a20 100644
--- a/refs.h
+++ b/refs.h
@@ -623,11 +623,29 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
 				      struct strbuf *err);
 typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
+typedef const char *resolve_ref_unsafe_fn(const char *ref,
+					  int resolve_flags,
+					  unsigned char *sha1, int *flags);
+typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);
+typedef int pack_refs_fn(unsigned int flags);
+typedef int peel_ref_fn(const char *refname, unsigned char *sha1);
+typedef int create_symref_fn(struct ref_transaction *transaction,
+			     const char *ref_target,
+			     const char *refs_heads_master,
+			     const char *logmsg);
+typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,
+				   unsigned char *sha1);
 
 struct ref_be {
 	struct ref_be *next;
 	const char *name;
 	ref_transaction_commit_fn *transaction_commit;
+	resolve_ref_unsafe_fn *resolve_ref_unsafe;
+	verify_refname_available_fn *verify_refname_available;
+	pack_refs_fn *pack_refs;
+	peel_ref_fn *peel_ref;
+	create_symref_fn *create_symref;
+	resolve_gitlink_ref_fn *resolve_gitlink_ref;
 };
 
 
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 21/44] refs-be-files.c: add methods for the ref iterators

[David Turner]

From: Ronnie Sahlberg <sahlberg@google.com>

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 41 ++++++++++++++++++++++++----------
 refs.c          | 69 ++++++++++++++++++++++++++++++++++++++++++++++-----------
 refs.h          | 34 +++++++++++++++++++++++++---
 3 files changed, 116 insertions(+), 28 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 116d72d..2e2399e 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1862,32 +1862,36 @@ static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)
 	return 0;
 }
 
-int head_ref(each_ref_fn fn, void *cb_data)
+static int files_head_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_head_ref(NULL, fn, cb_data);
 }
 
-int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+static int files_head_ref_submodule(const char *submodule, each_ref_fn fn,
+				    void *cb_data)
 {
 	return do_head_ref(submodule, fn, cb_data);
 }
 
-int for_each_ref(each_ref_fn fn, void *cb_data)
+static int files_for_each_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, "", fn, 0, 0, cb_data);
 }
 
-int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+static int files_for_each_ref_submodule(const char *submodule, each_ref_fn fn,
+					void *cb_data)
 {
 	return do_for_each_ref(get_ref_cache(submodule), "", fn, 0, 0, cb_data);
 }
 
-int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
+static int files_for_each_ref_in(const char *prefix, each_ref_fn fn,
+				 void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, prefix, fn, strlen(prefix), 0, cb_data);
 }
 
-int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)
+static int files_for_each_fullref_in(const char *prefix, each_ref_fn fn,
+				     void *cb_data, unsigned int broken)
 {
 	unsigned int flag = 0;
 
@@ -1896,18 +1900,21 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig
 	return do_for_each_ref(&ref_cache, prefix, fn, 0, flag, cb_data);
 }
 
-int for_each_ref_in_submodule(const char *submodule, const char *prefix,
-		each_ref_fn fn, void *cb_data)
+static int files_for_each_ref_in_submodule(const char *submodule,
+					   const char *prefix,
+					   each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(get_ref_cache(submodule), prefix, fn, strlen(prefix), 0, cb_data);
+	return do_for_each_ref(get_ref_cache(submodule), prefix, fn,
+			       strlen(prefix), 0, cb_data);
 }
-int for_each_replace_ref(each_ref_fn fn, void *cb_data)
+
+static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, git_replace_ref_base, fn,
 			       strlen(git_replace_ref_base), 0, cb_data);
 }
 
-int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
@@ -1917,7 +1924,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	return ret;
 }
 
-int for_each_rawref(each_ref_fn fn, void *cb_data)
+static int files_for_each_rawref(each_ref_fn fn, void *cb_data)
 {
 	return do_for_each_ref(&ref_cache, "", fn, 0,
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
@@ -3635,4 +3642,14 @@ struct ref_be refs_be_files = {
 	files_peel_ref,
 	files_create_symref,
 	files_resolve_gitlink_ref,
+	files_head_ref,
+	files_head_ref_submodule,
+	files_for_each_ref,
+	files_for_each_ref_submodule,
+	files_for_each_ref_in,
+	files_for_each_fullref_in,
+	files_for_each_ref_in_submodule,
+	files_for_each_rawref,
+	files_for_each_namespaced_ref,
+	files_for_each_replace_ref,
 };
diff --git a/refs.c b/refs.c
index 61d90f5..657ab73 100644
--- a/refs.c
+++ b/refs.c
@@ -1042,19 +1042,8 @@ int peel_ref(const char *refname, unsigned char *sha1)
 int create_symref(const char *ref_target, const char *refs_heads_master,
 		  const char *logmsg)
 {
-	struct strbuf err = STRBUF_INIT;
-	struct ref_transaction *transaction = ref_transaction_begin(&err);
-	if (!transaction)
-		return -1;
-
-	if (the_refs_backend->create_symref(transaction, ref_target,
-					    refs_heads_master, logmsg))
-		return -1;
-
-	if (ref_transaction_commit(transaction, &err))
-		return -1;
-
-	return 0;
+	return the_refs_backend->create_symref(ref_target, refs_heads_master,
+					       logmsg);
 }
 
 int resolve_gitlink_ref(const char *path, const char *refname,
@@ -1062,3 +1051,57 @@ int resolve_gitlink_ref(const char *path, const char *refname,
 {
 	return the_refs_backend->resolve_gitlink_ref(path, refname, sha1);
 }
+
+int head_ref(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->head_ref(fn, cb_data);
+}
+
+int head_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->head_ref_submodule(submodule, fn, cb_data);
+}
+
+int for_each_ref(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_ref(fn, cb_data);
+}
+
+int for_each_ref_submodule(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_ref_submodule(submodule, fn, cb_data);
+}
+
+int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_ref_in(prefix, fn, cb_data);
+}
+
+int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data,
+			unsigned int broken)
+{
+	return the_refs_backend->for_each_fullref_in(prefix, fn, cb_data,
+						     broken);
+}
+
+int for_each_ref_in_submodule(const char *submodule, const char *prefix,
+			      each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_ref_in_submodule(submodule, prefix,
+							   fn, cb_data);
+}
+
+int for_each_rawref(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_rawref(fn, cb_data);
+}
+
+int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_namespaced_ref(fn, cb_data);
+}
+
+int for_each_replace_ref(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_replace_ref(fn, cb_data);
+}
diff --git a/refs.h b/refs.h
index ee06a20..6dec0da 100644
--- a/refs.h
+++ b/refs.h
@@ -626,15 +626,33 @@ typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
 typedef const char *resolve_ref_unsafe_fn(const char *ref,
 					  int resolve_flags,
 					  unsigned char *sha1, int *flags);
-typedef int verify_refname_available_fn(const char *refname, struct string_list *extra, struct string_list *skip, struct strbuf *err);
+typedef int verify_refname_available_fn(const char *refname,
+					struct string_list *extra,
+					struct string_list *skip,
+					struct strbuf *err);
 typedef int pack_refs_fn(unsigned int flags);
 typedef int peel_ref_fn(const char *refname, unsigned char *sha1);
-typedef int create_symref_fn(struct ref_transaction *transaction,
-			     const char *ref_target,
+typedef int create_symref_fn(const char *ref_target,
 			     const char *refs_heads_master,
 			     const char *logmsg);
 typedef int resolve_gitlink_ref_fn(const char *path, const char *refname,
 				   unsigned char *sha1);
+typedef int head_ref_fn(each_ref_fn fn, void *cb_data);
+typedef int head_ref_submodule_fn(const char *submodule, each_ref_fn fn,
+				  void *cb_data);
+typedef int for_each_ref_fn(each_ref_fn fn, void *cb_data);
+typedef int for_each_ref_submodule_fn(const char *submodule, each_ref_fn fn,
+				      void *cb_data);
+typedef int for_each_ref_in_fn(const char *prefix, each_ref_fn fn,
+			       void *cb_data);
+typedef int for_each_fullref_in_fn(const char *prefix, each_ref_fn fn,
+				   void *cb_data, unsigned int broken);
+typedef int for_each_ref_in_submodule_fn(const char *submodule,
+					 const char *prefix,
+					 each_ref_fn fn, void *cb_data);
+typedef int for_each_rawref_fn(each_ref_fn fn, void *cb_data);
+typedef int for_each_namespaced_ref_fn(each_ref_fn fn, void *cb_data);
+typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);
 
 struct ref_be {
 	struct ref_be *next;
@@ -646,6 +664,16 @@ struct ref_be {
 	peel_ref_fn *peel_ref;
 	create_symref_fn *create_symref;
 	resolve_gitlink_ref_fn *resolve_gitlink_ref;
+	head_ref_fn *head_ref;
+	head_ref_submodule_fn *head_ref_submodule;
+	for_each_ref_fn *for_each_ref;
+	for_each_ref_submodule_fn *for_each_ref_submodule;
+	for_each_ref_in_fn *for_each_ref_in;
+	for_each_fullref_in_fn *for_each_fullref_in;
+	for_each_ref_in_submodule_fn *for_each_ref_in_submodule;
+	for_each_rawref_fn *for_each_rawref;
+	for_each_namespaced_ref_fn *for_each_namespaced_ref;
+	for_each_replace_ref_fn *for_each_replace_ref;
 };
 
 
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 22/44] refs-be-files.c: add method for for_each_reftype_...

[David Turner]

Add method for for_each_reftype_fullpath.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 11 +++++++++++
 refs.c          |  7 +++++++
 refs.h          |  3 +++
 3 files changed, 21 insertions(+)

diff --git a/refs-be-files.c b/refs-be-files.c
index 2e2399e..c99319f 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1914,6 +1914,16 @@ static int files_for_each_replace_ref(each_ref_fn fn, void *cb_data)
 			       strlen(git_replace_ref_base), 0, cb_data);
 }
 
+static int files_for_each_reftype_fullpath(each_ref_fn fn, char *type,
+					   unsigned int broken, void *cb_data)
+{
+	unsigned int flag = 0;
+
+	if (broken)
+		flag = DO_FOR_EACH_INCLUDE_BROKEN;
+	return do_for_each_ref(&ref_cache, type, fn, 0, flag, cb_data);
+}
+
 static int files_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
@@ -3652,4 +3662,5 @@ struct ref_be refs_be_files = {
 	files_for_each_rawref,
 	files_for_each_namespaced_ref,
 	files_for_each_replace_ref,
+	files_for_each_reftype_fullpath,
 };
diff --git a/refs.c b/refs.c
index 657ab73..e627d74 100644
--- a/refs.c
+++ b/refs.c
@@ -1105,3 +1105,10 @@ int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
 	return the_refs_backend->for_each_replace_ref(fn, cb_data);
 }
+
+int for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken,
+			      void *cb_data)
+{
+	return the_refs_backend->for_each_reftype_fullpath(fn, type, broken,
+							   cb_data);
+}
diff --git a/refs.h b/refs.h
index 6dec0da..491aa1d 100644
--- a/refs.h
+++ b/refs.h
@@ -653,6 +653,8 @@ typedef int for_each_ref_in_submodule_fn(const char *submodule,
 typedef int for_each_rawref_fn(each_ref_fn fn, void *cb_data);
 typedef int for_each_namespaced_ref_fn(each_ref_fn fn, void *cb_data);
 typedef int for_each_replace_ref_fn(each_ref_fn fn, void *cb_data);
+typedef int for_each_reftype_fullpath_fn(each_ref_fn fn, char *type,
+					 unsigned int broken, void *cb_data);
 
 struct ref_be {
 	struct ref_be *next;
@@ -674,6 +676,7 @@ struct ref_be {
 	for_each_rawref_fn *for_each_rawref;
 	for_each_namespaced_ref_fn *for_each_namespaced_ref;
 	for_each_replace_ref_fn *for_each_replace_ref;
+	for_each_reftype_fullpath_fn *for_each_reftype_fullpath;
 };
 
 
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 23/44] refs-be-files.c: add do_for_each_per_worktree_ref

[David Turner]

Alternate refs backends might still use files to store per-worktree
refs.  So the files backend's ref-loading infrastructure should be
available to those backends, just for use on per-worktree refs.  Add
do_for_each_per_worktree_ref, which iterates over per-worktree refs.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 15 ++++++++++++---
 refs.h          | 11 +++++++++++
 2 files changed, 23 insertions(+), 3 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index c99319f..9548fb3 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -552,9 +552,6 @@ static void sort_ref_dir(struct ref_dir *dir)
 	dir->sorted = dir->nr = i;
 }
 
-/* Include broken references in a do_for_each_ref*() iteration: */
-#define DO_FOR_EACH_INCLUDE_BROKEN 0x01
-
 /*
  * Return true iff the reference described by entry can be resolved to
  * an object in the database.  Emit a warning if the referred-to
@@ -602,6 +599,10 @@ static int do_one_ref(struct ref_entry *entry, void *cb_data)
 	struct ref_entry *old_current_ref;
 	int retval;
 
+	if (data->flags & DO_FOR_EACH_PER_WORKTREE_ONLY &&
+	    ref_type(entry->name) != REF_TYPE_PER_WORKTREE)
+		return 0;
+
 	if (!starts_with(entry->name, data->base))
 		return 0;
 
@@ -1844,6 +1845,14 @@ static int do_for_each_ref(struct ref_cache *refs, const char *base,
 	return do_for_each_entry(refs, base, do_one_ref, &data);
 }
 
+int do_for_each_per_worktree_ref(const char *submodule, const char *base,
+				 each_ref_fn fn, int trim, int flags,
+				 void *cb_data)
+{
+	return do_for_each_ref(get_ref_cache(submodule), base, fn, trim,
+			       flags | DO_FOR_EACH_PER_WORKTREE_ONLY, cb_data);
+}
+
 static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)
 {
 	struct object_id oid;
diff --git a/refs.h b/refs.h
index 491aa1d..e2ba725 100644
--- a/refs.h
+++ b/refs.h
@@ -241,6 +241,12 @@ struct ref_transaction {
  */
 #define REF_BAD_NAME 0x08
 
+/* Include broken references in a do_for_each_ref*() iteration */
+#define DO_FOR_EACH_INCLUDE_BROKEN 0x01
+
+/* Only include per-worktree refs in a do_for_each_ref*() iteration */
+#define DO_FOR_EACH_PER_WORKTREE_ONLY 0x02
+
 /*
  * The signature for the callback function for the for_each_*()
  * functions below.  The memory pointed to by the refname and sha1
@@ -284,6 +290,11 @@ extern int for_each_namespaced_ref(each_ref_fn fn, void *cb_data);
 /* can be used to learn about broken ref and symref */
 extern int for_each_rawref(each_ref_fn fn, void *cb_data);
 
+
+int do_for_each_per_worktree_ref(const char *submodule, const char *base,
+				 each_ref_fn fn, int trim, int flags,
+				 void *cb_data);
+
 static inline const char *has_glob_specials(const char *pattern)
 {
 	return strpbrk(pattern, "?*[");
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 24/44] refs.c: move refname_is_safe to the common code

[David Turner]

This function does not contain any backend specific code so we
move it to the common code.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 33 ---------------------------------
 refs.c          | 24 ++++++++++++++++++++++++
 refs.h          |  2 ++
 3 files changed, 26 insertions(+), 33 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 9548fb3..346429e 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -196,39 +196,6 @@ static struct ref_dir *get_ref_dir(struct ref_entry *entry)
 	return dir;
 }
 
-/*
- * Check if a refname is safe.
- * For refs that start with "refs/" we consider it safe as long they do
- * not try to resolve to outside of refs/.
- *
- * For all other refs we only consider them safe iff they only contain
- * upper case characters and '_' (like "HEAD" AND "MERGE_HEAD", and not like
- * "config").
- */
-static int refname_is_safe(const char *refname)
-{
-	if (starts_with(refname, "refs/")) {
-		char *buf;
-		int result;
-
-		buf = xmalloc(strlen(refname) + 1);
-		/*
-		 * Does the refname try to escape refs/?
-		 * For example: refs/foo/../bar is safe but refs/foo/../../bar
-		 * is not.
-		 */
-		result = !normalize_path_copy(buf, refname + strlen("refs/"));
-		free(buf);
-		return result;
-	}
-	while (*refname) {
-		if (!isupper(*refname) && *refname != '_')
-			return 0;
-		refname++;
-	}
-	return 1;
-}
-
 static struct ref_entry *create_ref_entry(const char *refname,
 					  const unsigned char *sha1, int flag,
 					  int check_name)
diff --git a/refs.c b/refs.c
index e627d74..2515f6e 100644
--- a/refs.c
+++ b/refs.c
@@ -1009,6 +1009,30 @@ int ref_transaction_verify(struct ref_transaction *transaction,
 				      flags, NULL, err);
 }
 
+int refname_is_safe(const char *refname)
+{
+	if (starts_with(refname, "refs/")) {
+		char *buf;
+		int result;
+
+		buf = xmalloc(strlen(refname) + 1);
+		/*
+		 * Does the refname try to escape refs/?
+		 * For example: refs/foo/../bar is safe but refs/foo/../../bar
+		 * is not.
+		 */
+		result = !normalize_path_copy(buf, refname + strlen("refs/"));
+		free(buf);
+		return result;
+	}
+	while (*refname) {
+		if (!isupper(*refname) && *refname != '_')
+			return 0;
+		refname++;
+	}
+	return 1;
+}
+
 /* backend functions */
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err)
diff --git a/refs.h b/refs.h
index e2ba725..fc8a748 100644
--- a/refs.h
+++ b/refs.h
@@ -415,6 +415,8 @@ extern int for_each_reflog(each_ref_fn, void *);
  */
 extern int check_refname_format(const char *refname, int flags);
 
+extern int refname_is_safe(const char *refname);
+
 extern const char *prettify_refname(const char *refname);
 
 extern char *shorten_unambiguous_ref(const char *refname, int strict);
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 25/44] refs.h: document make refname_is_safe and add it to header

[David Turner]

This function might be used by other refs backends

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/refs.h b/refs.h
index fc8a748..7a936e2 100644
--- a/refs.h
+++ b/refs.h
@@ -348,6 +348,17 @@ int verify_refname_available(const char *newname, struct string_list *extra,
 			     struct string_list *skip, struct strbuf *err);
 
 /*
+ * Check if a refname is safe.
+ * For refs that start with "refs/" we consider it safe as long they do
+ * not try to resolve to outside of refs/.
+ *
+ * For all other refs we only consider them safe iff they only contain
+ * upper case characters and '_' (like "HEAD" AND "MERGE_HEAD", and not like
+ * "config").
+ */
+int refname_is_safe(const char *refname);
+
+/*
  * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
  * REF_NODEREF: act on the ref directly, instead of dereferencing
  *              symbolic references.
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 26/44] refs.c: move copy_msg to the common code

[David Turner]

Rename copy_msg to copy_reflog_msg and make it public.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 28 +---------------------------
 refs.c          | 26 ++++++++++++++++++++++++++
 refs.h          |  2 ++
 3 files changed, 29 insertions(+), 27 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 346429e..830b5a1 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2637,32 +2637,6 @@ static int commit_ref(struct ref_lock *lock)
 	return 0;
 }
 
-/*
- * copy the reflog message msg to buf, which has been allocated sufficiently
- * large, while cleaning up the whitespaces.  Especially, convert LF to space,
- * because reflog file is one line per entry.
- */
-static int copy_msg(char *buf, const char *msg)
-{
-	char *cp = buf;
-	char c;
-	int wasspace = 1;
-
-	*cp++ = '\t';
-	while ((c = *msg++)) {
-		if (wasspace && isspace(c))
-			continue;
-		wasspace = isspace(c);
-		if (wasspace)
-			c = ' ';
-		*cp++ = c;
-	}
-	while (buf < cp && isspace(cp[-1]))
-		cp--;
-	*cp++ = '\n';
-	return cp - buf;
-}
-
 static int should_autocreate_reflog(const char *refname)
 {
 	if (!log_all_ref_updates)
@@ -2765,7 +2739,7 @@ static int log_ref_write_fd(int fd, const unsigned char *old_sha1,
 			sha1_to_hex(new_sha1),
 			committer);
 	if (msglen)
-		len += copy_msg(logrec + len - 1, msg) - 1;
+		len += copy_reflog_msg(logrec + len - 1, msg) - 1;
 
 	written = len <= maxlen ? write_in_full(fd, logrec, len) : -1;
 	free(logrec);
diff --git a/refs.c b/refs.c
index 2515f6e..bd8c71b 100644
--- a/refs.c
+++ b/refs.c
@@ -886,6 +886,32 @@ int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *c
 	return for_each_ref_in_submodule(submodule, "refs/remotes/", fn, cb_data);
 }
 
+/*
+ * copy the reflog message msg to buf, which has been allocated sufficiently
+ * large, while cleaning up the whitespaces.  Especially, convert LF to space,
+ * because reflog file is one line per entry.
+ */
+int copy_reflog_msg(char *buf, const char *msg)
+{
+	char *cp = buf;
+	char c;
+	int wasspace = 1;
+
+	*cp++ = '\t';
+	while ((c = *msg++)) {
+		if (wasspace && isspace(c))
+			continue;
+		wasspace = isspace(c);
+		if (wasspace)
+			c = ' ';
+		*cp++ = c;
+	}
+	while (buf < cp && isspace(cp[-1]))
+		cp--;
+	*cp++ = '\n';
+	return cp - buf;
+}
+
 int head_ref_namespaced(each_ref_fn fn, void *cb_data)
 {
 	struct strbuf buf = STRBUF_INIT;
diff --git a/refs.h b/refs.h
index 7a936e2..3da5d09 100644
--- a/refs.h
+++ b/refs.h
@@ -597,6 +597,8 @@ enum ref_type {
 
 enum ref_type ref_type(const char *refname);
 
+int copy_reflog_msg(char *buf, const char *msg);
+
 enum expire_reflog_flags {
 	EXPIRE_REFLOGS_DRY_RUN = 1 << 0,
 	EXPIRE_REFLOGS_UPDATE_REF = 1 << 1,
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 27/44] refs.c: move peel_object to the common code

[David Turner]

This function does not contain any backend specific code so we
move it to the common code.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 53 -----------------------------------------------------
 refs.c          | 31 +++++++++++++++++++++++++++++++
 refs.h          | 27 +++++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 53 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 830b5a1..bf2fd7a 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -1604,59 +1604,6 @@ static const char *files_resolve_ref_unsafe(const char *refname,
 	return ret;
 }
 
-enum peel_status {
-	/* object was peeled successfully: */
-	PEEL_PEELED = 0,
-
-	/*
-	 * object cannot be peeled because the named object (or an
-	 * object referred to by a tag in the peel chain), does not
-	 * exist.
-	 */
-	PEEL_INVALID = -1,
-
-	/* object cannot be peeled because it is not a tag: */
-	PEEL_NON_TAG = -2,
-
-	/* ref_entry contains no peeled value because it is a symref: */
-	PEEL_IS_SYMREF = -3,
-
-	/*
-	 * ref_entry cannot be peeled because it is broken (i.e., the
-	 * symbolic reference cannot even be resolved to an object
-	 * name):
-	 */
-	PEEL_BROKEN = -4
-};
-
-/*
- * Peel the named object; i.e., if the object is a tag, resolve the
- * tag recursively until a non-tag is found.  If successful, store the
- * result to sha1 and return PEEL_PEELED.  If the object is not a tag
- * or is not valid, return PEEL_NON_TAG or PEEL_INVALID, respectively,
- * and leave sha1 unchanged.
- */
-static enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
-{
-	struct object *o = lookup_unknown_object(name);
-
-	if (o->type == OBJ_NONE) {
-		int type = sha1_object_info(name, NULL);
-		if (type < 0 || !object_as_type(o, type, 0))
-			return PEEL_INVALID;
-	}
-
-	if (o->type != OBJ_TAG)
-		return PEEL_NON_TAG;
-
-	o = deref_tag_noverify(o);
-	if (!o)
-		return PEEL_INVALID;
-
-	hashcpy(sha1, o->sha1);
-	return PEEL_PEELED;
-}
-
 /*
  * Peel the entry (if possible) and return its new peel_status.  If
  * repeel is true, re-peel the entry even if there is an old peeled
diff --git a/refs.c b/refs.c
index bd8c71b..99b31f6 100644
--- a/refs.c
+++ b/refs.c
@@ -4,6 +4,9 @@
 #include "cache.h"
 #include "refs.h"
 #include "lockfile.h"
+#include "object.h"
+#include "tag.h"
+
 /*
  * We always have a files backend and it is the default.
  */
@@ -1059,6 +1062,34 @@ int refname_is_safe(const char *refname)
 	return 1;
 }
 
+/*
+ * Peel the named object; i.e., if the object is a tag, resolve the
+ * tag recursively until a non-tag is found.  If successful, store the
+ * result to sha1 and return PEEL_PEELED.  If the object is not a tag
+ * or is not valid, return PEEL_NON_TAG or PEEL_INVALID, respectively,
+ * and leave sha1 unchanged.
+ */
+enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
+{
+	struct object *o = lookup_unknown_object(name);
+
+	if (o->type == OBJ_NONE) {
+		int type = sha1_object_info(name, NULL);
+		if (type < 0 || !object_as_type(o, type, 0))
+			return PEEL_INVALID;
+	}
+
+	if (o->type != OBJ_TAG)
+		return PEEL_NON_TAG;
+
+	o = deref_tag_noverify(o);
+	if (!o)
+		return PEEL_INVALID;
+
+	hashcpy(sha1, o->sha1);
+	return PEEL_PEELED;
+}
+
 /* backend functions */
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err)
diff --git a/refs.h b/refs.h
index 3da5d09..636f959 100644
--- a/refs.h
+++ b/refs.h
@@ -76,6 +76,33 @@ extern int is_branch(const char *refname);
  */
 extern int peel_ref(const char *refname, unsigned char *sha1);
 
+enum peel_status {
+	/* object was peeled successfully: */
+	PEEL_PEELED = 0,
+
+	/*
+	 * object cannot be peeled because the named object (or an
+	 * object referred to by a tag in the peel chain), does not
+	 * exist.
+	 */
+	PEEL_INVALID = -1,
+
+	/* object cannot be peeled because it is not a tag: */
+	PEEL_NON_TAG = -2,
+
+	/* ref_entry contains no peeled value because it is a symref: */
+	PEEL_IS_SYMREF = -3,
+
+	/*
+	 * ref_entry cannot be peeled because it is broken (i.e., the
+	 * symbolic reference cannot even be resolved to an object
+	 * name):
+	 */
+	PEEL_BROKEN = -4
+};
+
+enum peel_status peel_object(const unsigned char *name, unsigned char *sha1);
+
 /**
  * Resolve refname in the nested "gitlink" repository that is located
  * at path.  If the resolution is successful, return 0 and set sha1 to
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 28/44] refs.c: move should_autocreate_reflog to common code

[David Turner]

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 10 ----------
 refs.c          | 10 ++++++++++
 refs.h          |  2 ++
 3 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index bf2fd7a..73111e7 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2584,16 +2584,6 @@ static int commit_ref(struct ref_lock *lock)
 	return 0;
 }
 
-static int should_autocreate_reflog(const char *refname)
-{
-	if (!log_all_ref_updates)
-		return 0;
-	return starts_with(refname, "refs/heads/") ||
-		starts_with(refname, "refs/remotes/") ||
-		starts_with(refname, "refs/notes/") ||
-		!strcmp(refname, "HEAD");
-}
-
 /*
  * Create a reflog for a ref.  If force_create = 0, the reflog will
  * only be created for certain refs (those for which
diff --git a/refs.c b/refs.c
index 99b31f6..1de8529 100644
--- a/refs.c
+++ b/refs.c
@@ -685,6 +685,16 @@ char *resolve_refdup(const char *refname, int resolve_flags,
 						  sha1, flags));
 }
 
+int should_autocreate_reflog(const char *refname)
+{
+	if (!log_all_ref_updates)
+		return 0;
+	return starts_with(refname, "refs/heads/") ||
+		starts_with(refname, "refs/remotes/") ||
+		starts_with(refname, "refs/notes/") ||
+		!strcmp(refname, "HEAD");
+}
+
 /*
  * How to handle various characters in refnames:
  * 0: An acceptable character for refs
diff --git a/refs.h b/refs.h
index 636f959..6d284f5 100644
--- a/refs.h
+++ b/refs.h
@@ -58,6 +58,8 @@ extern const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 extern char *resolve_refdup(const char *refname, int resolve_flags,
 			    unsigned char *sha1, int *flags);
 
+extern int should_autocreate_reflog(const char *refname);
+
 extern int read_ref_full(const char *refname, int resolve_flags,
 			 unsigned char *sha1, int *flags);
 extern int read_ref(const char *refname, unsigned char *sha1);
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 29/44] refs.c: add ref backend init function

[David Turner]

The file backend doesn't need this function, but other backends might.

Signed-off-by: David Turner <dturner@twopensource.com>
Signed-off-by: Ronnie Sahlberg <rsahlberg@google.com>
---
 refs-be-files.c | 6 ++++++
 refs.c          | 3 ++-
 refs.h          | 4 +++-
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 73111e7..f123459 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3307,6 +3307,11 @@ static int ref_present(const char *refname,
 
 	return string_list_has_string(affected_refnames, refname);
 }
+ 
+void files_init_backend(void *data)
+{
+	/* do nothing */
+}
 
 int initial_ref_transaction_commit(struct ref_transaction *transaction,
 				   struct strbuf *err)
@@ -3532,6 +3537,7 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
 struct ref_be refs_be_files = {
 	NULL,
 	"files",
+	files_init_backend,
 	files_transaction_commit,
 	files_resolve_ref_unsafe,
 	files_verify_refname_available,
diff --git a/refs.c b/refs.c
index 1de8529..6370ace 100644
--- a/refs.c
+++ b/refs.c
@@ -19,13 +19,14 @@ struct ref_be *refs_backends = &refs_be_files;
 /*
  * This function is used to switch to an alternate backend.
  */
-int set_refs_backend(const char *name)
+int set_refs_backend(const char *name, void *init_data)
 {
 	struct ref_be *be;
 
 	for (be = refs_backends; be; be = be->next)
 		if (!strcmp(be->name, name)) {
 			the_refs_backend = be;
+			be->init_backend(init_data);
 			return 0;
 		}
 	return 1;
diff --git a/refs.h b/refs.h
index 6d284f5..e76553a 100644
--- a/refs.h
+++ b/refs.h
@@ -675,6 +675,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 			 void *policy_cb_data);
 
 /* refs backends */
+typedef void ref_backend_init_fn(void *data);
 typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
 				      struct strbuf *err);
 typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
@@ -714,6 +715,7 @@ typedef int for_each_reftype_fullpath_fn(each_ref_fn fn, char *type,
 struct ref_be {
 	struct ref_be *next;
 	const char *name;
+	ref_backend_init_fn *init_backend;
 	ref_transaction_commit_fn *transaction_commit;
 	resolve_ref_unsafe_fn *resolve_ref_unsafe;
 	verify_refname_available_fn *verify_refname_available;
@@ -736,6 +738,6 @@ struct ref_be {
 
 
 extern struct ref_be refs_be_files;
-int set_refs_backend(const char *name);
+int set_refs_backend(const char *name, void *init_data);
 
 #endif /* REFS_H */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 30/44] refs.c: add methods for reflog

[David Turner]

In the file-based backend, the reflog piggybacks on the ref lock.
Since other backends won't have the same sort of ref lock, ref backends
must also handle reflogs.

Signed-off-by: Ronnie Sahlberg <rsahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 35 +++++++++++++++++++++++------------
 refs.c          | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 refs.h          | 28 ++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+), 12 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index f123459..104e90f 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2650,7 +2650,8 @@ static int log_ref_setup(const char *refname, struct strbuf *logfile, struct str
 }
 
 
-int safe_create_reflog(const char *refname, int force_create, struct strbuf *err)
+static int files_create_reflog(const char *refname, int force_create,
+			       struct strbuf *err)
 {
 	int ret;
 	struct strbuf sb = STRBUF_INIT;
@@ -2898,7 +2899,7 @@ static int files_create_symref(const char *ref_target,
 	return 0;
 }
 
-int reflog_exists(const char *refname)
+static int files_reflog_exists(const char *refname)
 {
 	struct stat st;
 
@@ -2906,7 +2907,7 @@ int reflog_exists(const char *refname)
 		S_ISREG(st.st_mode);
 }
 
-int delete_reflog(const char *refname)
+static int files_delete_reflog(const char *refname)
 {
 	return remove_path(git_path("logs/%s", refname));
 }
@@ -2950,7 +2951,9 @@ static char *find_beginning_of_line(char *bob, char *scan)
 	return scan;
 }
 
-int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void *cb_data)
+static int files_for_each_reflog_ent_reverse(const char *refname,
+					     each_reflog_ent_fn fn,
+					     void *cb_data)
 {
 	struct strbuf sb = STRBUF_INIT;
 	FILE *logfp;
@@ -3052,7 +3055,8 @@ int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn, void
 	return ret;
 }
 
-int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn, void *cb_data)
+static int files_for_each_reflog_ent(const char *refname,
+				     each_reflog_ent_fn fn, void *cb_data)
 {
 	FILE *logfp;
 	struct strbuf sb = STRBUF_INIT;
@@ -3114,7 +3118,7 @@ static int do_for_each_reflog(struct strbuf *name, each_ref_fn fn, void *cb_data
 	return retval;
 }
 
-int for_each_reflog(each_ref_fn fn, void *cb_data)
+static int files_for_each_reflog(each_ref_fn fn, void *cb_data)
 {
 	int retval;
 	struct strbuf name;
@@ -3429,12 +3433,12 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,
 	return 0;
 }
 
-int reflog_expire(const char *refname, const unsigned char *sha1,
-		 unsigned int flags,
-		 reflog_expiry_prepare_fn prepare_fn,
-		 reflog_expiry_should_prune_fn should_prune_fn,
-		 reflog_expiry_cleanup_fn cleanup_fn,
-		 void *policy_cb_data)
+static int files_reflog_expire(const char *refname, const unsigned char *sha1,
+			       unsigned int flags,
+			       reflog_expiry_prepare_fn prepare_fn,
+			       reflog_expiry_should_prune_fn should_prune_fn,
+			       reflog_expiry_cleanup_fn cleanup_fn,
+			       void *policy_cb_data)
 {
 	static struct lock_file reflog_lock;
 	struct expire_reflog_cb cb;
@@ -3539,6 +3543,13 @@ struct ref_be refs_be_files = {
 	"files",
 	files_init_backend,
 	files_transaction_commit,
+	files_for_each_reflog_ent,
+	files_for_each_reflog_ent_reverse,
+	files_for_each_reflog,
+	files_reflog_exists,
+	files_create_reflog,
+	files_delete_reflog,
+	files_reflog_expire,
 	files_resolve_ref_unsafe,
 	files_verify_refname_available,
 	files_pack_refs,
diff --git a/refs.c b/refs.c
index 6370ace..304fead 100644
--- a/refs.c
+++ b/refs.c
@@ -1204,3 +1204,49 @@ int for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken,
 	return the_refs_backend->for_each_reftype_fullpath(fn, type, broken,
 							   cb_data);
 }
+
+int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,
+				void *cb_data)
+{
+	return the_refs_backend->for_each_reflog_ent_reverse(refname, fn,
+							     cb_data);
+}
+
+int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,
+			void *cb_data)
+{
+	return the_refs_backend->for_each_reflog_ent(refname, fn, cb_data);
+}
+
+int for_each_reflog(each_ref_fn fn, void *cb_data)
+{
+	return the_refs_backend->for_each_reflog(fn, cb_data);
+}
+
+int reflog_exists(const char *refname)
+{
+	return the_refs_backend->reflog_exists(refname);
+}
+
+int safe_create_reflog(const char *refname, int force_create,
+		       struct strbuf *err)
+{
+	return the_refs_backend->create_reflog(refname, force_create, err);
+}
+
+int delete_reflog(const char *refname)
+{
+	return the_refs_backend->delete_reflog(refname);
+}
+
+int reflog_expire(const char *refname, const unsigned char *sha1,
+		  unsigned int flags,
+		  reflog_expiry_prepare_fn prepare_fn,
+		  reflog_expiry_should_prune_fn should_prune_fn,
+		  reflog_expiry_cleanup_fn cleanup_fn,
+		  void *policy_cb_data)
+{
+	return the_refs_backend->reflog_expire(refname, sha1, flags,
+					       prepare_fn, should_prune_fn,
+					       cleanup_fn, policy_cb_data);
+}
diff --git a/refs.h b/refs.h
index e76553a..5888877 100644
--- a/refs.h
+++ b/refs.h
@@ -679,6 +679,27 @@ typedef void ref_backend_init_fn(void *data);
 typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
 				      struct strbuf *err);
 typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
+
+/* reflog functions */
+typedef int for_each_reflog_ent_fn(const char *refname,
+				   each_reflog_ent_fn fn,
+				   void *cb_data);
+typedef int for_each_reflog_ent_reverse_fn(const char *refname,
+					   each_reflog_ent_fn fn,
+					   void *cb_data);
+typedef int for_each_reflog_fn(each_ref_fn fn, void *cb_data);
+typedef int reflog_exists_fn(const char *refname);
+typedef int create_reflog_fn(const char *refname, int force_create,
+			     struct strbuf *err);
+typedef int delete_reflog_fn(const char *refname);
+typedef int reflog_expire_fn(const char *refname, const unsigned char *sha1,
+			     unsigned int flags,
+			     reflog_expiry_prepare_fn prepare_fn,
+			     reflog_expiry_should_prune_fn should_prune_fn,
+			     reflog_expiry_cleanup_fn cleanup_fn,
+			     void *policy_cb_data);
+
+/* resolution functions */
 typedef const char *resolve_ref_unsafe_fn(const char *ref,
 					  int resolve_flags,
 					  unsigned char *sha1, int *flags);
@@ -717,6 +738,13 @@ struct ref_be {
 	const char *name;
 	ref_backend_init_fn *init_backend;
 	ref_transaction_commit_fn *transaction_commit;
+	for_each_reflog_ent_fn *for_each_reflog_ent;
+	for_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;
+	for_each_reflog_fn *for_each_reflog;
+	reflog_exists_fn *reflog_exists;
+	create_reflog_fn *create_reflog;
+	delete_reflog_fn *delete_reflog;
+	reflog_expire_fn *reflog_expire;
 	resolve_ref_unsafe_fn *resolve_ref_unsafe;
 	verify_refname_available_fn *verify_refname_available;
 	pack_refs_fn *pack_refs;
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 31/44] refs.c: add method for initial ref transaction commit

[David Turner]

Signed-off-by: Ronnie Sahlberg <rsahlberg@google.com>
Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 5 +++--
 refs.c          | 6 ++++++
 refs.h          | 1 +
 3 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 104e90f..7560a55 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3317,8 +3317,8 @@ void files_init_backend(void *data)
 	/* do nothing */
 }
 
-int initial_ref_transaction_commit(struct ref_transaction *transaction,
-				   struct strbuf *err)
+static int files_initial_transaction_commit(struct ref_transaction *transaction,
+					    struct strbuf *err)
 {
 	int ret = 0, i;
 	int n = transaction->nr;
@@ -3543,6 +3543,7 @@ struct ref_be refs_be_files = {
 	"files",
 	files_init_backend,
 	files_transaction_commit,
+	files_initial_transaction_commit,
 	files_for_each_reflog_ent,
 	files_for_each_reflog_ent_reverse,
 	files_for_each_reflog,
diff --git a/refs.c b/refs.c
index 304fead..6e41ed8 100644
--- a/refs.c
+++ b/refs.c
@@ -1250,3 +1250,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
 					       prepare_fn, should_prune_fn,
 					       cleanup_fn, policy_cb_data);
 }
+
+int initial_ref_transaction_commit(struct ref_transaction *transaction,
+				   struct strbuf *err)
+{
+	return the_refs_backend->initial_transaction_commit(transaction, err);
+}
diff --git a/refs.h b/refs.h
index 5888877..e3136ee 100644
--- a/refs.h
+++ b/refs.h
@@ -738,6 +738,7 @@ struct ref_be {
 	const char *name;
 	ref_backend_init_fn *init_backend;
 	ref_transaction_commit_fn *transaction_commit;
+	ref_transaction_commit_fn *initial_transaction_commit;
 	for_each_reflog_ent_fn *for_each_reflog_ent;
 	for_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;
 	for_each_reflog_fn *for_each_reflog;
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 32/44] initdb: move safe_create_dir into common code

[David Turner]

In a moment, we'll create initdb functions for ref backends, and code
from initdb that calls this function needs to move into the files
backend.  So this function needs to be public.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 builtin/init-db.c | 12 ------------
 cache.h           |  5 +++++
 path.c            | 12 ++++++++++++
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index 313d13f..f6f7259 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -25,18 +25,6 @@ static int init_shared_repository = -1;
 static const char *init_db_template_dir;
 static const char *git_link;
 
-static void safe_create_dir(const char *dir, int share)
-{
-	if (mkdir(dir, 0777) < 0) {
-		if (errno != EEXIST) {
-			perror(dir);
-			exit(1);
-		}
-	}
-	else if (share && adjust_shared_perm(dir))
-		die(_("Could not make %s writable by group"), dir);
-}
-
 static void copy_templates_1(struct strbuf *path, struct strbuf *template,
 			     DIR *dir)
 {
diff --git a/cache.h b/cache.h
index 1c45b62..cc817dc 100644
--- a/cache.h
+++ b/cache.h
@@ -1749,4 +1749,9 @@ void stat_validity_update(struct stat_validity *sv, int fd);
 int versioncmp(const char *s1, const char *s2);
 void sleep_millisec(int millisec);
 
+/*
+ * Create a directory and (if share is nonzero) adjust its permissions
+ * according to the shared_repository setting.
+ */
+void safe_create_dir(const char *dir, int share);
 #endif /* CACHE_H */
diff --git a/path.c b/path.c
index 48bd252..9ab2710 100644
--- a/path.c
+++ b/path.c
@@ -740,6 +740,18 @@ int adjust_shared_perm(const char *path)
 	return 0;
 }
 
+void safe_create_dir(const char *dir, int share)
+{
+	if (mkdir(dir, 0777) < 0) {
+		if (errno != EEXIST) {
+			perror(dir);
+			exit(1);
+		}
+	}
+	else if (share && adjust_shared_perm(dir))
+		die(_("Could not make %s writable by group"), dir);
+}
+
 static int have_same_root(const char *path1, const char *path2)
 {
 	int is_abs1, is_abs2;
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 33/44] refs.c: add method for initializing refs db

[David Turner]

Alternate refs backends might not need refs/heads and so on, so we make
ref db initialization part of the backend.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 builtin/init-db.c | 14 ++++----------
 refs-be-files.c   | 17 +++++++++++++++++
 refs.c            |  5 +++++
 refs.h            |  4 ++++
 4 files changed, 30 insertions(+), 10 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index f6f7259..2705739 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -178,13 +178,7 @@ static int create_default_files(const char *template_path)
 	char junk[2];
 	int reinit;
 	int filemode;
-
-	/*
-	 * Create .git/refs/{heads,tags}
-	 */
-	safe_create_dir(git_path_buf(&buf, "refs"), 1);
-	safe_create_dir(git_path_buf(&buf, "refs/heads"), 1);
-	safe_create_dir(git_path_buf(&buf, "refs/tags"), 1);
+	struct strbuf err = STRBUF_INIT;
 
 	/* Just look for `init.templatedir` */
 	git_config(git_init_db_config, NULL);
@@ -208,11 +202,11 @@ static int create_default_files(const char *template_path)
 	 */
 	if (shared_repository) {
 		adjust_shared_perm(get_git_dir());
-		adjust_shared_perm(git_path_buf(&buf, "refs"));
-		adjust_shared_perm(git_path_buf(&buf, "refs/heads"));
-		adjust_shared_perm(git_path_buf(&buf, "refs/tags"));
 	}
 
+	if (refs_initdb(&err, shared_repository))
+		die("failed to set up refs db: %s", err.buf);
+
 	/*
 	 * Create the default symlink from ".git/HEAD" to the "master"
 	 * branch, if it does not exist yet.
diff --git a/refs-be-files.c b/refs-be-files.c
index 7560a55..5f2602c 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3538,10 +3538,27 @@ static int files_reflog_expire(const char *refname, const unsigned char *sha1,
 	return -1;
 }
 
+static int files_initdb(struct strbuf *err, int shared)
+{
+	/*
+	 * Create .git/refs/{heads,tags}
+	 */
+	safe_create_dir(git_path("refs"), 1);
+	safe_create_dir(git_path("refs/heads"), 1);
+	safe_create_dir(git_path("refs/tags"), 1);
+	if (shared) {
+		adjust_shared_perm(git_path("refs"));
+		adjust_shared_perm(git_path("refs/heads"));
+		adjust_shared_perm(git_path("refs/tags"));
+	}
+	return 0;
+}
+
 struct ref_be refs_be_files = {
 	NULL,
 	"files",
 	files_init_backend,
+	files_initdb,
 	files_transaction_commit,
 	files_initial_transaction_commit,
 	files_for_each_reflog_ent,
diff --git a/refs.c b/refs.c
index 6e41ed8..5fc8eb7 100644
--- a/refs.c
+++ b/refs.c
@@ -1102,6 +1102,11 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
 }
 
 /* backend functions */
+int refs_initdb(struct strbuf *err, int shared)
+{
+	return the_refs_backend->initdb(err, shared);
+}
+
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err)
 {
diff --git a/refs.h b/refs.h
index e3136ee..ce71d33 100644
--- a/refs.h
+++ b/refs.h
@@ -68,6 +68,8 @@ extern int ref_exists(const char *refname);
 
 extern int is_branch(const char *refname);
 
+extern int refs_initdb(struct strbuf *err, int shared);
+
 /*
  * If refname is a non-symbolic reference that refers to a tag object,
  * and the tag can be (recursively) dereferenced to a non-tag object,
@@ -676,6 +678,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 
 /* refs backends */
 typedef void ref_backend_init_fn(void *data);
+typedef int ref_backend_initdb_fn(struct strbuf *err, int shared);
 typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
 				      struct strbuf *err);
 typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
@@ -737,6 +740,7 @@ struct ref_be {
 	struct ref_be *next;
 	const char *name;
 	ref_backend_init_fn *init_backend;
+	ref_backend_initdb_fn *initdb;
 	ref_transaction_commit_fn *transaction_commit;
 	ref_transaction_commit_fn *initial_transaction_commit;
 	for_each_reflog_ent_fn *for_each_reflog_ent;
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 34/44] refs-be-files.c: add method to rename refs

[David Turner]

We also have to make rename_ref_available public, since alternate
backends for rename_ref will need it.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 20 +++-----------------
 refs.c          | 21 +++++++++++++++++++++
 refs.h          |  7 ++++++-
 3 files changed, 30 insertions(+), 18 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 5f2602c..4e6f032 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2444,29 +2444,14 @@ out:
 	return ret;
 }
 
-static int rename_ref_available(const char *oldname, const char *newname)
-{
-	struct string_list skip = STRING_LIST_INIT_NODUP;
-	struct strbuf err = STRBUF_INIT;
-	int ret;
-
-	string_list_insert(&skip, oldname);
-	ret = !verify_refname_available(newname, NULL, &skip, &err);
-	if (!ret)
-		error("%s", err.buf);
-
-	string_list_clear(&skip, 0);
-	strbuf_release(&err);
-	return ret;
-}
-
 static int write_ref_to_lockfile(struct ref_lock *lock,
 				 const unsigned char *sha1, struct strbuf *err);
 static int commit_ref_update(struct ref_lock *lock,
 			     const unsigned char *sha1, const char *logmsg,
 			     int flags, struct strbuf *err);
 
-int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)
+static int files_rename_ref(const char *oldrefname, const char *newrefname,
+			    const char *logmsg)
 {
 	unsigned char sha1[20], orig_sha1[20];
 	int flag = 0, logmoved = 0;
@@ -3561,6 +3546,7 @@ struct ref_be refs_be_files = {
 	files_initdb,
 	files_transaction_commit,
 	files_initial_transaction_commit,
+	files_rename_ref,
 	files_for_each_reflog_ent,
 	files_for_each_reflog_ent_reverse,
 	files_for_each_reflog,
diff --git a/refs.c b/refs.c
index 5fc8eb7..5a3125d 100644
--- a/refs.c
+++ b/refs.c
@@ -1113,6 +1113,11 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	return the_refs_backend->transaction_commit(transaction, err);
 }
 
+int rename_ref(const char *oldref, const char *newref, const char *logmsg)
+{
+	return the_refs_backend->rename_ref(oldref, newref, logmsg);
+}
+
 const char *resolve_ref_unsafe(const char *ref, int resolve_flags,
 			       unsigned char *sha1, int *flags)
 {
@@ -1126,6 +1131,22 @@ int verify_refname_available(const char *refname, struct string_list *extra,
 	return the_refs_backend->verify_refname_available(refname, extra, skip, err);
 }
 
+int rename_ref_available(const char *oldname, const char *newname)
+{
+	struct string_list skip = STRING_LIST_INIT_NODUP;
+	struct strbuf err = STRBUF_INIT;
+	int ret;
+
+	string_list_insert(&skip, oldname);
+	ret = !verify_refname_available(newname, NULL, &skip, &err);
+	if (!ret)
+		error("%s", err.buf);
+
+	string_list_clear(&skip, 0);
+	strbuf_release(&err);
+	return ret;
+}
+
 int pack_refs(unsigned int flags)
 {
 	return the_refs_backend->pack_refs(flags);
diff --git a/refs.h b/refs.h
index ce71d33..87d5801 100644
--- a/refs.h
+++ b/refs.h
@@ -464,10 +464,13 @@ extern const char *prettify_refname(const char *refname);
 extern char *shorten_unambiguous_ref(const char *refname, int strict);
 
 /** rename ref, return 0 on success **/
-extern int rename_ref(const char *oldref, const char *newref, const char *logmsg);
+extern int rename_ref(const char *oldref, const char *newref,
+		      const char *logmsg);
 
 extern int create_symref(const char *ref, const char *refs_heads_master, const char *logmsg);
 
+int rename_ref_available(const char *oldname, const char *newname);
+
 enum action_on_err {
 	UPDATE_REFS_MSG_ON_ERR,
 	UPDATE_REFS_DIE_ON_ERR,
@@ -703,6 +706,7 @@ typedef int reflog_expire_fn(const char *refname, const unsigned char *sha1,
 			     void *policy_cb_data);
 
 /* resolution functions */
+typedef int rename_ref_fn(const char *oldref, const char *newref, const char *logmsg);
 typedef const char *resolve_ref_unsafe_fn(const char *ref,
 					  int resolve_flags,
 					  unsigned char *sha1, int *flags);
@@ -743,6 +747,7 @@ struct ref_be {
 	ref_backend_initdb_fn *initdb;
 	ref_transaction_commit_fn *transaction_commit;
 	ref_transaction_commit_fn *initial_transaction_commit;
+	rename_ref_fn *rename_ref;
 	for_each_reflog_ent_fn *for_each_reflog_ent;
 	for_each_reflog_ent_reverse_fn *for_each_reflog_ent_reverse;
 	for_each_reflog_fn *for_each_reflog;
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 35/44] refs: make lock generic

[David Turner]

Instead of using a files-backend-specific struct ref_lock, the generic
ref_transaction struct should provide a void pointer that backends can use
for their own lock data.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 29 ++++++++++++++++-------------
 refs.h          |  2 +-
 2 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 4e6f032..798a995 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3167,11 +3167,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 	 */
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
+		struct ref_lock *lock;
 
 		if ((update->flags & REF_HAVE_NEW) &&
 		    is_null_sha1(update->new_sha1))
 			update->flags |= REF_DELETING;
-		update->lock = lock_ref_sha1_basic(
+		lock = lock_ref_sha1_basic(
 				update->refname,
 				((update->flags & REF_HAVE_OLD) ?
 				 update->old_sha1 : NULL),
@@ -3179,7 +3180,8 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 				update->flags,
 				&update->type,
 				err);
-		if (!update->lock) {
+		update->backend_data = lock;
+		if (!lock) {
 			char *reason;
 
 			ret = (errno == ENOTDIR)
@@ -3197,12 +3199,12 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 						  (update->flags & REF_NODEREF));
 
 			if (!overwriting_symref &&
-			    !hashcmp(update->lock->old_oid.hash, update->new_sha1)) {
+			    !hashcmp(lock->old_oid.hash, update->new_sha1)) {
 				/*
 				 * The reference already has the desired
 				 * value, so we don't need to write it.
 				 */
-			} else if (write_ref_to_lockfile(update->lock,
+			} else if (write_ref_to_lockfile(lock,
 							 update->new_sha1,
 							 err)) {
 				char *write_err = strbuf_detach(err, NULL);
@@ -3211,7 +3213,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 				 * The lock was freed upon failure of
 				 * write_ref_to_lockfile():
 				 */
-				update->lock = NULL;
+				update->backend_data = NULL;
 				strbuf_addf(err,
 					    "cannot update the ref '%s': %s",
 					    update->refname, write_err);
@@ -3227,7 +3229,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 			 * We didn't have to write anything to the lockfile.
 			 * Close it to free up the file descriptor:
 			 */
-			if (close_ref(update->lock)) {
+			if (close_ref(lock)) {
 				strbuf_addf(err, "Couldn't close %s.lock",
 					    update->refname);
 				goto cleanup;
@@ -3240,16 +3242,16 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 		struct ref_update *update = updates[i];
 
 		if (update->flags & REF_NEEDS_COMMIT) {
-			if (commit_ref_update(update->lock,
+			if (commit_ref_update(update->backend_data,
 					      update->new_sha1, update->msg,
 					      update->flags, err)) {
 				/* freed by commit_ref_update(): */
-				update->lock = NULL;
+				update->backend_data = NULL;
 				ret = TRANSACTION_GENERIC_ERROR;
 				goto cleanup;
 			} else {
 				/* freed by commit_ref_update(): */
-				update->lock = NULL;
+				update->backend_data = NULL;
 			}
 		}
 	}
@@ -3257,16 +3259,17 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 	/* Perform deletes now that updates are safely completed */
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
+		struct ref_lock *lock = update->backend_data;
 
 		if (update->flags & REF_DELETING) {
-			if (delete_ref_loose(update->lock, update->type, err)) {
+			if (delete_ref_loose(lock, update->type, err)) {
 				ret = TRANSACTION_GENERIC_ERROR;
 				goto cleanup;
 			}
 
 			if (!(update->flags & REF_ISPRUNING))
 				string_list_append(&refs_to_delete,
-						   update->lock->ref_name);
+						   lock->ref_name);
 		}
 	}
 
@@ -3282,8 +3285,8 @@ cleanup:
 	transaction->state = REF_TRANSACTION_CLOSED;
 
 	for (i = 0; i < n; i++)
-		if (updates[i]->lock)
-			unlock_ref(updates[i]->lock);
+		if (updates[i]->backend_data)
+			unlock_ref(updates[i]->backend_data);
 	string_list_clear(&refs_to_delete, 0);
 	string_list_clear(&affected_refnames, 0);
 	return ret;
diff --git a/refs.h b/refs.h
index 87d5801..4d8b3bb 100644
--- a/refs.h
+++ b/refs.h
@@ -196,7 +196,7 @@ struct ref_update {
 	 * REF_DELETING, and REF_ISPRUNING:
 	 */
 	unsigned int flags;
-	struct ref_lock *lock;
+	void *backend_data;
 	int type;
 	char *msg;
 	const char refname[FLEX_ARRAY];
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 36/44] refs: make files_log_ref_write functions public

[David Turner]

Because HEAD and stash are per-worktree, refs.c needs to go through
the files backend to write these refs.

In this patch, we make one files backend internal functions
public. Later, we will use this to handle reflog updates for
per-worktree symbolic refs (HEAD).

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 8 ++++++++
 refs.h          | 5 +++++
 2 files changed, 13 insertions(+)

diff --git a/refs-be-files.c b/refs-be-files.c
index 798a995..8d1ffce 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -2710,6 +2710,14 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,
 			 const unsigned char *new_sha1, const char *msg,
 			 int flags, struct strbuf *err)
 {
+	return files_log_ref_write(refname, old_sha1, new_sha1, msg, flags,
+				   err);
+}
+
+int files_log_ref_write(const char *refname, const unsigned char *old_sha1,
+			const unsigned char *new_sha1, const char *msg,
+			int flags, struct strbuf *err)
+{
 	struct strbuf sb = STRBUF_INIT;
 	int ret = log_ref_write_1(refname, old_sha1, new_sha1, msg, &sb, flags,
 				  err);
diff --git a/refs.h b/refs.h
index 4d8b3bb..3aad3b8 100644
--- a/refs.h
+++ b/refs.h
@@ -633,6 +633,11 @@ enum ref_type ref_type(const char *refname);
 
 int copy_reflog_msg(char *buf, const char *msg);
 
+int files_log_ref_write(const char *refname, const unsigned char *old_sha1,
+			const unsigned char *new_sha1, const char *msg,
+			int flags, struct strbuf *err);
+
+
 enum expire_reflog_flags {
 	EXPIRE_REFLOGS_DRY_RUN = 1 << 0,
 	EXPIRE_REFLOGS_UPDATE_REF = 1 << 1,
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 37/44] refs: break out a ref conflict check

[David Turner]

Create new function verify_no_descendants, to hold one of the ref
conflict checks used in verify_refname_available.  Multiple backends
will need this function, so it goes in the common code.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 33 ++++++++-------------------------
 refs.c          | 22 ++++++++++++++++++++++
 refs.h          |  7 +++++++
 3 files changed, 37 insertions(+), 25 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index 8d1ffce..dd2c42e 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -730,6 +730,7 @@ static int verify_refname_available_dir(const char *refname,
 					struct strbuf *err)
 {
 	const char *slash;
+	const char *extra_refname;
 	int pos;
 	struct strbuf dirname = STRBUF_INIT;
 	int ret = -1;
@@ -835,33 +836,15 @@ static int verify_refname_available_dir(const char *refname,
 		}
 	}
 
-	if (extras) {
-		/*
-		 * Check for entries in extras that start with
-		 * "$refname/". We do that by looking for the place
-		 * where "$refname/" would be inserted in extras. If
-		 * there is an entry at that position that starts with
-		 * "$refname/" and is not in skip, then we have a
-		 * conflict.
-		 */
-		for (pos = string_list_find_insert_index(extras, dirname.buf, 0);
-		     pos < extras->nr; pos++) {
-			const char *extra_refname = extras->items[pos].string;
-
-			if (!starts_with(extra_refname, dirname.buf))
-				break;
-
-			if (!skip || !string_list_has_string(skip, extra_refname)) {
-				strbuf_addf(err, "cannot process '%s' and '%s' at the same time",
-					    refname, extra_refname);
-				goto cleanup;
-			}
-		}
+	extra_refname = find_descendant_ref(dirname.buf, extras, skip);
+	if (extra_refname) {
+		strbuf_addf(err,
+			    "cannot process '%s' and '%s' at the same time",
+			    refname, extra_refname);
+	} else {
+		ret = 0;
 	}
 
-	/* No conflicts were found */
-	ret = 0;
-
 cleanup:
 	strbuf_release(&dirname);
 	return ret;
diff --git a/refs.c b/refs.c
index 5a3125d..3ae0274 100644
--- a/refs.c
+++ b/refs.c
@@ -1101,6 +1101,28 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
 	return PEEL_PEELED;
 }
 
+const char *find_descendant_ref(const char *refname,
+				const struct string_list *extras,
+				const struct string_list *skip)
+{
+	int pos;
+	if (!extras)
+		return NULL;
+
+	/* Look for the place where "$refname/" would be inserted in extras. */
+	for (pos = string_list_find_insert_index(extras, refname, 0);
+	     pos < extras->nr; pos++) {
+		const char *extra_refname = extras->items[pos].string;
+
+		if (!starts_with(extra_refname, refname))
+			break;
+
+		if (!skip || !string_list_has_string(skip, extra_refname))
+			return extra_refname;
+	}
+	return NULL;
+}
+
 /* backend functions */
 int refs_initdb(struct strbuf *err, int shared)
 {
diff --git a/refs.h b/refs.h
index 3aad3b8..f8becea 100644
--- a/refs.h
+++ b/refs.h
@@ -637,6 +637,13 @@ int files_log_ref_write(const char *refname, const unsigned char *old_sha1,
 			const unsigned char *new_sha1, const char *msg,
 			int flags, struct strbuf *err);
 
+/*
+ * Check for entries in extras that start with "$refname/", ignoring
+ * those in skip. If there is such an entry, then we have a conflict.
+ */
+const char *find_descendant_ref(const char *refname,
+				const struct string_list *extras,
+				const struct string_list *skip);
 
 enum expire_reflog_flags {
 	EXPIRE_REFLOGS_DRY_RUN = 1 << 0,
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 38/44] refs: move duplicate check to common code

[David Turner]

The check for duplicate refnames in a transaction is needed for
all backends, so move it to the common code.

ref_transaction_commit_fn gains a new argument, the sorted
string_list of affected refnames.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs-be-files.c | 57 ++++-----------------------------------------
 refs.c          | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 refs.h          |  1 +
 3 files changed, 75 insertions(+), 54 deletions(-)

diff --git a/refs-be-files.c b/refs-be-files.c
index dd2c42e..b89bd5c 100644
--- a/refs-be-files.c
+++ b/refs-be-files.c
@@ -3104,24 +3104,8 @@ static int files_for_each_reflog(each_ref_fn fn, void *cb_data)
 	return retval;
 }
 
-static int ref_update_reject_duplicates(struct string_list *refnames,
-					struct strbuf *err)
-{
-	int i, n = refnames->nr;
-
-	assert(err);
-
-	for (i = 1; i < n; i++)
-		if (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {
-			strbuf_addf(err,
-				    "Multiple updates for ref '%s' not allowed.",
-				    refnames->items[i].string);
-			return 1;
-		}
-	return 0;
-}
-
 static int files_transaction_commit(struct ref_transaction *transaction,
+				    struct string_list *affected_refnames,
 				    struct strbuf *err)
 {
 	int ret = 0, i;
@@ -3129,26 +3113,6 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 	struct ref_update **updates = transaction->updates;
 	struct string_list refs_to_delete = STRING_LIST_INIT_NODUP;
 	struct string_list_item *ref_to_delete;
-	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
-
-	assert(err);
-
-	if (transaction->state != REF_TRANSACTION_OPEN)
-		die("BUG: commit called for transaction that is not open");
-
-	if (!n) {
-		transaction->state = REF_TRANSACTION_CLOSED;
-		return 0;
-	}
-
-	/* Fail if a refname appears more than once in the transaction: */
-	for (i = 0; i < n; i++)
-		string_list_append(&affected_refnames, updates[i]->refname);
-	string_list_sort(&affected_refnames);
-	if (ref_update_reject_duplicates(&affected_refnames, err)) {
-		ret = TRANSACTION_GENERIC_ERROR;
-		goto cleanup;
-	}
 
 	/*
 	 * Acquire all locks, verify old values if provided, check
@@ -3167,7 +3131,7 @@ static int files_transaction_commit(struct ref_transaction *transaction,
 				update->refname,
 				((update->flags & REF_HAVE_OLD) ?
 				 update->old_sha1 : NULL),
-				&affected_refnames, NULL,
+				affected_refnames, NULL,
 				update->flags,
 				&update->type,
 				err);
@@ -3279,7 +3243,6 @@ cleanup:
 		if (updates[i]->backend_data)
 			unlock_ref(updates[i]->backend_data);
 	string_list_clear(&refs_to_delete, 0);
-	string_list_clear(&affected_refnames, 0);
 	return ret;
 }
 
@@ -3297,27 +3260,18 @@ void files_init_backend(void *data)
 }
 
 static int files_initial_transaction_commit(struct ref_transaction *transaction,
+					    struct string_list *affected_refnames,
 					    struct strbuf *err)
 {
 	int ret = 0, i;
 	int n = transaction->nr;
 	struct ref_update **updates = transaction->updates;
-	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
 
 	assert(err);
 
 	if (transaction->state != REF_TRANSACTION_OPEN)
 		die("BUG: commit called for transaction that is not open");
 
-	/* Fail if a refname appears more than once in the transaction: */
-	for (i = 0; i < n; i++)
-		string_list_append(&affected_refnames, updates[i]->refname);
-	string_list_sort(&affected_refnames);
-	if (ref_update_reject_duplicates(&affected_refnames, err)) {
-		ret = TRANSACTION_GENERIC_ERROR;
-		goto cleanup;
-	}
-
 	/*
 	 * It's really undefined to call this function in an active
 	 * repository or when there are existing references: we are
@@ -3330,7 +3284,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,
 	 * so here we really only check that none of the references
 	 * that we are creating already exists.
 	 */
-	if (for_each_rawref(ref_present, &affected_refnames))
+	if (for_each_rawref(ref_present, affected_refnames))
 		die("BUG: initial ref transaction called with existing refs");
 
 	for (i = 0; i < n; i++) {
@@ -3340,7 +3294,7 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,
 		    !is_null_sha1(update->old_sha1))
 			die("BUG: initial ref transaction with old_sha1 set");
 		if (verify_refname_available(update->refname,
-					     &affected_refnames, NULL,
+					     affected_refnames, NULL,
 					     err)) {
 			ret = TRANSACTION_NAME_CONFLICT;
 			goto cleanup;
@@ -3371,7 +3325,6 @@ static int files_initial_transaction_commit(struct ref_transaction *transaction,
 
 cleanup:
 	transaction->state = REF_TRANSACTION_CLOSED;
-	string_list_clear(&affected_refnames, 0);
 	return ret;
 }
 
diff --git a/refs.c b/refs.c
index 3ae0274..b00f910 100644
--- a/refs.c
+++ b/refs.c
@@ -1123,6 +1123,37 @@ const char *find_descendant_ref(const char *refname,
 	return NULL;
 }
 
+/*
+ * Return 1 if there are any duplicate refnames in the updates in
+ * `transaction`, and fill in err with an appropriate error message.
+ * Fill in `refnames` with the refnames from the transaction.
+ */
+
+static int ref_update_reject_duplicates(struct ref_transaction *transaction,
+					struct string_list *refnames,
+					struct strbuf *err)
+{
+	int i, n = transaction->nr;
+	struct ref_update **updates;
+
+	assert(err);
+
+	updates = transaction->updates;
+	/* Fail if a refname appears more than once in the transaction: */
+	for (i = 0; i < n; i++)
+		string_list_append(refnames, updates[i]->refname);
+	string_list_sort(refnames);
+
+	for (i = 1; i < n; i++)
+		if (!strcmp(refnames->items[i - 1].string, refnames->items[i].string)) {
+			strbuf_addf(err,
+				    "Multiple updates for ref '%s' not allowed.",
+				    refnames->items[i].string);
+			return 1;
+		}
+	return 0;
+}
+
 /* backend functions */
 int refs_initdb(struct strbuf *err, int shared)
 {
@@ -1132,7 +1163,29 @@ int refs_initdb(struct strbuf *err, int shared)
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err)
 {
-	return the_refs_backend->transaction_commit(transaction, err);
+	int ret = -1;
+	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
+
+	assert(err);
+
+	if (transaction->state != REF_TRANSACTION_OPEN)
+		die("BUG: commit called for transaction that is not open");
+
+	if (!transaction->nr) {
+		transaction->state = REF_TRANSACTION_CLOSED;
+		return 0;
+	}
+
+	if (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {
+		ret = TRANSACTION_GENERIC_ERROR;
+		goto done;
+	}
+
+	ret = the_refs_backend->transaction_commit(transaction,
+						   &affected_refnames, err);
+done:
+	string_list_clear(&affected_refnames, 0);
+	return ret;
 }
 
 int rename_ref(const char *oldref, const char *newref, const char *logmsg)
@@ -1302,5 +1355,19 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
 int initial_ref_transaction_commit(struct ref_transaction *transaction,
 				   struct strbuf *err)
 {
-	return the_refs_backend->initial_transaction_commit(transaction, err);
+
+	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
+	int ret;
+
+	if (ref_update_reject_duplicates(transaction,
+					 &affected_refnames, err)) {
+		ret = TRANSACTION_GENERIC_ERROR;
+		goto done;
+	}
+	ret = the_refs_backend->initial_transaction_commit(transaction,
+							   &affected_refnames,
+							   err);
+done:
+	string_list_clear(&affected_refnames, 0);
+	return ret;
 }
diff --git a/refs.h b/refs.h
index f8becea..a9fdbf0 100644
--- a/refs.h
+++ b/refs.h
@@ -695,6 +695,7 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 typedef void ref_backend_init_fn(void *data);
 typedef int ref_backend_initdb_fn(struct strbuf *err, int shared);
 typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
+				      struct string_list *affected_refnames,
 				      struct strbuf *err);
 typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
 
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 39/44] refs: always handle non-normal refs in files backend

[David Turner]

Always handle non-normal (per-worktree or pseudo) refs in the files
backend instead of alternate backends.

Sometimes a ref transaction will update both a per-worktree ref and a
normal ref.  For instance, an ordinary commit might update
refs/heads/master and HEAD (or at least HEAD's reflog).

We handle three cases here:

1. updates to normal refs continue to go through the chosen backend

2. updates to non-normal refs with REF_NODEREF or to non-symbolic refs
are moved to a separate files backend transaction.

3. updates to symbolic refs are dereferenced to their base ref.  The
update to the base ref then goes through the ordinary backend, while
the files backend is directly called to update the symref's reflog.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs.c | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 139 insertions(+), 2 deletions(-)

diff --git a/refs.c b/refs.c
index b00f910..025cc08 100644
--- a/refs.c
+++ b/refs.c
@@ -7,6 +7,11 @@
 #include "object.h"
 #include "tag.h"
 
+const char split_transaction_fail_warning[] =
+	"A ref transaction was split across two refs backends.  Part of the "
+	"transaction succeeded, but then the update to the per-worktree refs "
+	"failed.  Your repository may be in an inconsistent state.";
+
 /*
  * We always have a files backend and it is the default.
  */
@@ -963,6 +968,13 @@ void ref_transaction_free(struct ref_transaction *transaction)
 	free(transaction);
 }
 
+static void add_update_obj(struct ref_transaction *transaction,
+			   struct ref_update *update)
+{
+	ALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);
+	transaction->updates[transaction->nr++] = update;
+}
+
 static struct ref_update *add_update(struct ref_transaction *transaction,
 				     const char *refname)
 {
@@ -970,8 +982,7 @@ static struct ref_update *add_update(struct ref_transaction *transaction,
 	struct ref_update *update = xcalloc(1, sizeof(*update) + len);
 
 	memcpy((char *)update->refname, refname, len); /* includes NUL */
-	ALLOC_GROW(transaction->updates, transaction->nr + 1, transaction->alloc);
-	transaction->updates[transaction->nr++] = update;
+	add_update_obj(transaction, update);
 	return update;
 }
 
@@ -1160,11 +1171,87 @@ int refs_initdb(struct strbuf *err, int shared)
 	return the_refs_backend->initdb(err, shared);
 }
 
+/*
+ * Special case for non-normal refs.  For symbolic-refs when
+ * REF_NODEREF is not turned on, we dereference them here and replace
+ * updates to the symbolic refs with updates to the underlying ref.
+ * Then we do our own reflogging for the symbolic ref.
+ *
+ * We move other non-normal ref updates with into a specially-created
+ * files-backend transaction
+ */
+static int move_abnormal_ref_updates(struct ref_transaction *transaction,
+				     struct ref_transaction *files_transaction,
+				     struct string_list *symrefs)
+{
+	int i;
+
+	for (i = 0; i < transaction->nr; i++) {
+		struct ref_update *update = transaction->updates[i];
+		const char *resolved;
+		int flags = 0;
+		unsigned char sha1[20];
+
+		if (ref_type(update->refname) == REF_TYPE_NORMAL)
+			continue;
+
+		resolved = resolve_ref_unsafe(update->refname, 0, sha1, &flags);
+
+		if (update->flags & REF_NODEREF || !(flags & REF_ISSYMREF)) {
+			int last;
+
+			add_update_obj(files_transaction, update);
+			/*
+			 * Replace this transaction with the
+			 * last transaction, removing it from
+			 * the list of backend transactions
+			 */
+			last = --transaction->nr;
+			transaction->updates[i] = transaction->updates[last];
+			continue;
+		}
+
+		if (resolved) {
+			struct ref_update *new_update;
+			struct string_list_item *item;
+
+			if (ref_type(resolved) != REF_TYPE_NORMAL)
+				die("Non-normal symbolic ref `%s` points to non-normal ref `%s`", update->refname, resolved);
+
+			new_update = xmalloc(sizeof(*new_update) +
+					     strlen(resolved) + 1);
+			memcpy(new_update, update, sizeof(*update));
+
+			if (update->flags & REF_HAVE_OLD &&
+			    hashcmp(sha1, update->old_sha1)) {
+				/* consistency check failed */
+				free(new_update);
+				return -1;
+			} else {
+				hashcpy(update->old_sha1, sha1);
+			}
+
+			strcpy((char *)new_update->refname, resolved);
+			transaction->updates[i] = new_update;
+
+			item = string_list_append(symrefs, update->refname);
+			item->util = new_update;
+			free(update);
+		}
+	}
+
+	return 0;
+}
+
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err)
 {
 	int ret = -1;
 	struct string_list affected_refnames = STRING_LIST_INIT_NODUP;
+	struct string_list files_affected_refnames = STRING_LIST_INIT_NODUP;
+	struct string_list symrefs = STRING_LIST_INIT_DUP;
+	struct string_list_item *item;
+	struct ref_transaction *files_transaction = NULL;
 
 	assert(err);
 
@@ -1176,6 +1263,26 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 		return 0;
 	}
 
+	if (the_refs_backend != &refs_be_files) {
+		files_transaction = ref_transaction_begin(err);
+		if (!files_transaction)
+			die("%s", err->buf);
+
+		ret = move_abnormal_ref_updates(transaction, files_transaction,
+						&symrefs);
+		if (ret)
+			goto done;
+
+		/* files backend commit */
+		if (ref_update_reject_duplicates(files_transaction,
+						 &files_affected_refnames,
+						 err)) {
+			ret = TRANSACTION_GENERIC_ERROR;
+			goto done;
+		}
+	}
+
+	/* main backend commit */
 	if (ref_update_reject_duplicates(transaction, &affected_refnames, err)) {
 		ret = TRANSACTION_GENERIC_ERROR;
 		goto done;
@@ -1183,8 +1290,35 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 
 	ret = the_refs_backend->transaction_commit(transaction,
 						   &affected_refnames, err);
+	if (ret)
+		goto done;
+
+	if (the_refs_backend != &refs_be_files) {
+		ret = refs_be_files.transaction_commit(files_transaction,
+						       &files_affected_refnames,
+						       err);
+		if (ret) {
+			warning(split_transaction_fail_warning);
+			goto done;
+		}
+
+		/* reflogging for dereferenced symbolic refs */
+		for_each_string_list_item(item, &symrefs) {
+			struct ref_update *update = item->util;
+			if (files_log_ref_write(item->string, update->old_sha1,
+						update->new_sha1,
+						update->msg, update->flags, err))
+				warning("failed to log ref update for symref %s",
+					item->string);
+		}
+	}
+
 done:
 	string_list_clear(&affected_refnames, 0);
+	string_list_clear(&files_affected_refnames, 0);
+	if (files_transaction)
+		ref_transaction_free(files_transaction);
+	string_list_clear(&symrefs, 0);
 	return ret;
 }
 
@@ -1235,6 +1369,9 @@ int peel_ref(const char *refname, unsigned char *sha1)
 int create_symref(const char *ref_target, const char *refs_heads_master,
 		  const char *logmsg)
 {
+	if (ref_type(ref_target) != REF_TYPE_NORMAL)
+		return refs_be_files.create_symref(ref_target, refs_heads_master,
+						   logmsg);
 	return the_refs_backend->create_symref(ref_target, refs_heads_master,
 					       logmsg);
 }
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 40/44] refs: allow ref backend to be set for clone

[David Turner]

Add a new option, --refs-backend-type, to allow the ref backend type to
be set on new clones.

Submodules must use the same ref backend as the parent repository, so
we also pass the --refs-backend-type option option when cloning
submodules.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 Documentation/git-clone.txt |  4 ++++
 builtin/clone.c             | 27 +++++++++++++++++++++++++--
 builtin/submodule--helper.c |  5 ++++-
 cache.h                     |  1 +
 refs.c                      |  2 ++
 5 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index f1f2a3f..d7a4cb0 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -14,6 +14,7 @@ SYNOPSIS
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--dissociate] [--separate-git-dir <git dir>]
 	  [--depth <depth>] [--[no-]single-branch]
+	  [--refs-backend-type=<name>]
 	  [--recursive | --recurse-submodules] [--] <repository>
 	  [<directory>]
 
@@ -216,6 +217,9 @@ objects from the source repository into a pack in the cloned repository.
 	The result is Git repository can be separated from working
 	tree.
 
+--refs-backend-type=<name>::
+	Type of refs backend. Default is to use the original files based
+	backend.
 
 <repository>::
 	The (possibly remote) repository to clone from.  See the
diff --git a/builtin/clone.c b/builtin/clone.c
index 3e14491..d489a87 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -92,11 +92,13 @@ static struct option builtin_clone_options[] = {
 		   N_("separate git dir from working tree")),
 	OPT_STRING_LIST('c', "config", &option_config, N_("key=value"),
 			N_("set config inside the new repository")),
+	OPT_STRING(0, "refs-backend-type", &refs_backend_type,
+		   N_("name"), N_("name of backend type to use")),
 	OPT_END()
 };
 
 static const char *argv_submodule[] = {
-	"submodule", "update", "--init", "--recursive", NULL
+	"submodule", "update", "--init", "--recursive", NULL, NULL
 };
 
 static const char *get_repo_path_1(struct strbuf *path, int *is_bundle)
@@ -724,8 +726,24 @@ static int checkout(void)
 	err |= run_hook_le(NULL, "post-checkout", sha1_to_hex(null_sha1),
 			   sha1_to_hex(sha1), "1", NULL);
 
-	if (!err && option_recursive)
+	if (!err && option_recursive) {
+		const char **backend_arg = argv_submodule;
+		char *new_backend_arg = NULL;
+		if (refs_backend_type) {
+			while (*backend_arg)
+				++backend_arg;
+
+			new_backend_arg = xmalloc(21 + strlen(refs_backend_type));
+			sprintf(new_backend_arg, "--refs-backend-type=%s",
+				refs_backend_type);
+			*backend_arg = new_backend_arg;
+		}
 		err = run_command_v_opt(argv_submodule, RUN_GIT_CMD);
+		if (refs_backend_type) {
+			free(new_backend_arg);
+			*backend_arg = NULL;
+		}
+	}
 
 	return err;
 }
@@ -744,6 +762,11 @@ static void write_config(struct string_list *config)
 					       write_one_config, NULL) < 0)
 			die("unable to write parameters to config file");
 	}
+
+	if (refs_backend_type &&
+	    write_one_config("core.refs-backend-type",
+			     refs_backend_type, NULL) < 0)
+			die("unable to write backend parameter to config file");
 }
 
 static void write_refspec_config(const char *src_ref_prefix,
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index f4c3eff..5c9ca4e 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -140,7 +140,10 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url
 		argv_array_pushl(&cp.args, "--reference", reference, NULL);
 	if (gitdir && *gitdir)
 		argv_array_pushl(&cp.args, "--separate-git-dir", gitdir, NULL);
-
+	if (refs_backend_type && *refs_backend_type) {
+		argv_array_push(&cp.args, "--refs-backend-type");
+		argv_array_push(&cp.args, refs_backend_type);
+	}
 	argv_array_push(&cp.args, url);
 	argv_array_push(&cp.args, path);
 
diff --git a/cache.h b/cache.h
index cc817dc..692cfd3 100644
--- a/cache.h
+++ b/cache.h
@@ -695,6 +695,7 @@ enum object_creation_mode {
 extern enum object_creation_mode object_creation_mode;
 
 extern char *notes_ref_name;
+extern const char *refs_backend_type;
 
 extern int grafts_replace_parents;
 
diff --git a/refs.c b/refs.c
index 025cc08..901558d 100644
--- a/refs.c
+++ b/refs.c
@@ -21,6 +21,8 @@ struct ref_be *the_refs_backend = &refs_be_files;
  */
 struct ref_be *refs_backends = &refs_be_files;
 
+const char *refs_backend_type;
+
 /*
  * This function is used to switch to an alternate backend.
  */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 41/44] refs: add register_refs_backend

[David Turner]

Add register_refs_backend, to allow refs backends to be registered.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 refs.c | 6 ++++++
 refs.h | 1 +
 2 files changed, 7 insertions(+)

diff --git a/refs.c b/refs.c
index 901558d..534d12e 100644
--- a/refs.c
+++ b/refs.c
@@ -23,6 +23,12 @@ struct ref_be *refs_backends = &refs_be_files;
 
 const char *refs_backend_type;
 
+void register_refs_backend(struct ref_be *be)
+{
+	be->next = refs_backends;
+	refs_backends = be;
+}
+
 /*
  * This function is used to switch to an alternate backend.
  */
diff --git a/refs.h b/refs.h
index a9fdbf0..45615e8 100644
--- a/refs.h
+++ b/refs.h
@@ -790,5 +790,6 @@ struct ref_be {
 
 extern struct ref_be refs_be_files;
 int set_refs_backend(const char *name, void *init_data);
+void register_refs_backend(struct ref_be *be);
 
 #endif /* REFS_H */
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 42/44] introduce "extensions" form of core.repositoryformatversion

[David Turner]

From: Jeff King <peff@peff.net>

Normally we try to avoid bumps of the whole-repository
core.repositoryformatversion field. However, it is
unavoidable if we want to safely change certain aspects of
git in a backwards-incompatible way (e.g., modifying the set
of ref tips that we must traverse to generate a list of
unreachable, safe-to-prune objects).

If we were to bump the repository version for every such
change, then any implementation understanding version `X`
would also have to understand `X-1`, `X-2`, and so forth,
even though the incompatibilities may be in orthogonal parts
of the system, and there is otherwise no reason we cannot
implement one without the other (or more importantly, that
the user cannot choose to use one feature without the other,
weighing the tradeoff in compatibility only for that
particular feature).

This patch documents the existing repositoryformatversion
strategy and introduces a new format, "1", which lets a
repository specify that it must run with an arbitrary set of
extensions. This can be used, for example:

 - to inform git that the objects should not be pruned based
   only on the reachability of the ref tips (e.g, because it
   has "clone --shared" children)

 - that the refs are stored in a format besides the usual
   "refs" and "packed-refs" directories

Because we bump to format "1", and because format "1"
requires that a running git knows about any extensions
mentioned, we know that older versions of the code will not
do something dangerous when confronted with these new
formats.

For example, if the user chooses to use database storage for
refs, they may set the "extensions.refbackend" config to
"db". Older versions of git will not understand format "1"
and bail. Versions of git which understand "1" but do not
know about "refbackend", or which know about "refbackend"
but not about the "db" backend, will refuse to run. This is
annoying, of course, but much better than the alternative of
claiming that there are no refs in the repository, or
writing to a location that other implementations will not
read.

Note that we are only defining the rules for format 1 here.
We do not ever write format 1 ourselves; it is a tool that
is meant to be used by users and future extensions to
provide safety with older implementations.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/technical/repository-version.txt | 81 ++++++++++++++++++++++++++
 cache.h                                        |  6 ++
 setup.c                                        | 37 +++++++++++-
 t/t1302-repo-version.sh                        | 38 ++++++++++++
 4 files changed, 159 insertions(+), 3 deletions(-)
 create mode 100644 Documentation/technical/repository-version.txt

diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt
new file mode 100644
index 0000000..3d7106d
--- /dev/null
+++ b/Documentation/technical/repository-version.txt
@@ -0,0 +1,81 @@
+Git Repository Format Versions
+==============================
+
+Every git repository is marked with a numeric version in the
+`core.repositoryformatversion` key of its `config` file. This version
+specifies the rules for operating on the on-disk repository data. An
+implementation of git which does not understand a particular version
+advertised by an on-disk repository MUST NOT operate on that repository;
+doing so risks not only producing wrong results, but actually losing
+data.
+
+Because of this rule, version bumps should be kept to an absolute
+minimum. Instead, we generally prefer these strategies:
+
+  - bumping format version numbers of individual data files (e.g.,
+    index, packfiles, etc). This restricts the incompatibilities only to
+    those files.
+
+  - introducing new data that gracefully degrades when used by older
+    clients (e.g., pack bitmap files are ignored by older clients, which
+    simply do not take advantage of the optimization they provide).
+
+A whole-repository format version bump should only be part of a change
+that cannot be independently versioned. For instance, if one were to
+change the reachability rules for objects, or the rules for locking
+refs, that would require a bump of the repository format version.
+
+Note that this applies only to accessing the repository's disk contents
+directly. An older client which understands only format `0` may still
+connect via `git://` to a repository using format `1`, as long as the
+server process understands format `1`.
+
+The preferred strategy for rolling out a version bump (whether whole
+repository or for a single file) is to teach git to read the new format,
+and allow writing the new format with a config switch or command line
+option (for experimentation or for those who do not care about backwards
+compatibility with older gits). Then after a long period to allow the
+reading capability to become common, we may switch to writing the new
+format by default.
+
+The currently defined format versions are:
+
+Version `0`
+-----------
+
+This is the format defined by the initial version of git, including but
+not limited to the format of the repository directory, the repository
+configuration file, and the object and ref storage. Specifying the
+complete behavior of git is beyond the scope of this document.
+
+Version `1`
+-----------
+
+This format is identical to version `0`, with the following exceptions:
+
+  1. When reading the `core.repositoryformatversion` variable, a git
+     implementation which supports version 1 MUST also read any
+     configuration keys found in the `extensions` section of the
+     configuration file.
+
+  2. If a version-1 repository specifies any `extensions.*` keys that
+     the running git has not implemented, the operation MUST NOT
+     proceed. Similarly, if the value of any known key is not understood
+     by the implementation, the operation MUST NOT proceed.
+
+Note that if no extensions are specified in the config file, then
+`core.repositoryformatversion` SHOULD be set to `0` (setting it to `1`
+provides no benefit, and makes the repository incompatible with older
+implementations of git).
+
+This document will serve as the master list for extensions. Any
+implementation wishing to define a new extension should make a note of
+it here, in order to claim the name.
+
+The defined extensions are:
+
+`noop`
+~~~~~~
+
+This extension does not change git's behavior at all. It is useful only
+for testing format-1 compatibility.
diff --git a/cache.h b/cache.h
index 692cfd3..edfe7fa 100644
--- a/cache.h
+++ b/cache.h
@@ -699,7 +699,13 @@ extern const char *refs_backend_type;
 
 extern int grafts_replace_parents;
 
+/*
+ * GIT_REPO_VERSION is the version we write by default. The
+ * _READ variant is the highest number we know how to
+ * handle.
+ */
 #define GIT_REPO_VERSION 0
+#define GIT_REPO_VERSION_READ 1
 extern int repository_format_version;
 extern int check_repository_format(void);
 
diff --git a/setup.c b/setup.c
index b264471..7433eed 100644
--- a/setup.c
+++ b/setup.c
@@ -5,6 +5,7 @@
 static int inside_git_dir = -1;
 static int inside_work_tree = -1;
 static int work_tree_config_is_bogus;
+static struct string_list unknown_extensions = STRING_LIST_INIT_DUP;
 
 /*
  * The input parameter must contain an absolute path, and it must already be
@@ -356,10 +357,23 @@ void setup_work_tree(void)
 
 static int check_repo_format(const char *var, const char *value, void *cb)
 {
+	const char *ext;
+
 	if (strcmp(var, "core.repositoryformatversion") == 0)
 		repository_format_version = git_config_int(var, value);
 	else if (strcmp(var, "core.sharedrepository") == 0)
 		shared_repository = git_config_perm(var, value);
+	else if (skip_prefix(var, "extensions.", &ext)) {
+		/*
+		 * record any known extensions here; otherwise,
+		 * we fall through to recording it as unknown, and
+		 * check_repository_format will complain
+		 */
+		if (!strcmp(ext, "noop"))
+			;
+		else
+			string_list_append(&unknown_extensions, ext);
+	}
 	return 0;
 }
 
@@ -370,6 +384,8 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)
 	config_fn_t fn;
 	int ret = 0;
 
+	string_list_clear(&unknown_extensions, 0);
+
 	if (get_common_dir(&sb, gitdir))
 		fn = check_repo_format;
 	else
@@ -387,16 +403,31 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)
 	 * is a good one.
 	 */
 	git_config_early(fn, NULL, repo_config);
-	if (GIT_REPO_VERSION < repository_format_version) {
+	if (GIT_REPO_VERSION_READ < repository_format_version) {
 		if (!nongit_ok)
 			die ("Expected git repo version <= %d, found %d",
-			     GIT_REPO_VERSION, repository_format_version);
+			     GIT_REPO_VERSION_READ, repository_format_version);
 		warning("Expected git repo version <= %d, found %d",
-			GIT_REPO_VERSION, repository_format_version);
+			GIT_REPO_VERSION_READ, repository_format_version);
 		warning("Please upgrade Git");
 		*nongit_ok = -1;
 		ret = -1;
 	}
+
+	if (repository_format_version >= 1 && unknown_extensions.nr) {
+		int i;
+
+		if (!nongit_ok)
+			die("unknown repository extension: %s",
+			    unknown_extensions.items[0].string);
+
+		for (i = 0; i < unknown_extensions.nr; i++)
+			warning("unknown repository extension: %s",
+				unknown_extensions.items[i].string);
+		*nongit_ok = -1;
+		ret = -1;
+	}
+
 	strbuf_release(&sb);
 	return ret;
 }
diff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh
index 0d9388a..8dd6fd7 100755
--- a/t/t1302-repo-version.sh
+++ b/t/t1302-repo-version.sh
@@ -67,4 +67,42 @@ test_expect_success 'gitdir required mode' '
 	)
 '
 
+check_allow () {
+	git rev-parse --git-dir >actual &&
+	echo .git >expect &&
+	test_cmp expect actual
+}
+
+check_abort () {
+	test_must_fail git rev-parse --git-dir
+}
+
+# avoid git-config, since it cannot be trusted to run
+# in a repository with a broken version
+mkconfig () {
+	echo '[core]' &&
+	echo "repositoryformatversion = $1" &&
+	shift &&
+
+	if test $# -gt 0; then
+		echo '[extensions]' &&
+		for i in "$@"; do
+			echo "$i"
+		done
+	fi
+}
+
+while read outcome version extensions; do
+	test_expect_success "$outcome version=$version $extensions" "
+		mkconfig $version $extensions >.git/config &&
+		check_${outcome}
+	"
+done <<\EOF
+allow 0
+allow 1
+allow 1 noop
+abort 1 no-such-extension
+allow 0 no-such-extension
+EOF
+
 test_done
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 43/44] refs: add LMDB refs backend

[David Turner]

Add a database backend for refs using LMDB.  This backend runs git
for-each-ref about 30% faster than the files backend with fully-packed
refs on a repo with ~120k refs.  It's also about 4x faster than using
fully-unpacked refs.  In addition, and perhaps more importantly , it
avoids case-conflict issues on OS X.

LMDB has a few features that make it suitable for usage in git:

1. It is relatively lightweight; it requires only one header file, and
the library code takes under 64k at runtime.

2. It is well-tested: it's been used in OpenLDAP for years.

3. It's very fast.  LMDB's benchmarks show that it is among
the fastest key-value stores.

4. It has a relatively simple concurrency story; readers don't
block writers and writers don't block readers.

Ronnie Sahlberg's original version of this patchset used tdb.  The
major disadvantage of tdb is that tdb is hard to build on OS X.  It's
also not in homebrew.  So lmdb seemed simpler.

To test this backend's correctness, I hacked test-lib.sh and
test-lib-functions.sh to run all tests under the refs backend. Dozens
of tests use manual ref/reflog reading/writing, or create submodules
without passing --refs-backend-type to git init.  If those tests are
changed to use the update-ref machinery or test-refs-be-db (or, in the
case of packed-refs, corrupt refs, and dumb fetch tests, are skipped),
the only remaining failing tests are the git-new-workdir tests and the
gitweb tests.

Signed-off-by: David Turner <dturner@twopensource.com>
---
 .gitignore                                     |    1 +
 Documentation/config.txt                       |    7 +
 Documentation/git-clone.txt                    |    2 +-
 Documentation/git-init-db.txt                  |    2 +-
 Documentation/git-init.txt                     |    6 +
 Documentation/technical/refs-be-lmdb.txt       |   50 +
 Documentation/technical/repository-version.txt |    5 +
 Makefile                                       |   12 +
 builtin/clone.c                                |    2 +-
 builtin/init-db.c                              |   20 +-
 config.c                                       |   34 +
 configure.ac                                   |   33 +
 contrib/workdir/git-new-workdir                |    3 +
 environment.c                                  |    1 +
 refs-be-lmdb.c                                 | 1997 ++++++++++++++++++++++++
 refs.h                                         |   10 +
 setup.c                                        |   32 +-
 test-refs-be-lmdb.c                            |   68 +
 18 files changed, 2275 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/technical/refs-be-lmdb.txt
 create mode 100644 refs-be-lmdb.c
 create mode 100644 test-refs-be-lmdb.c

diff --git a/.gitignore b/.gitignore
index 1c2f832..7decc2f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -199,6 +199,7 @@
 /test-path-utils
 /test-prio-queue
 /test-read-cache
+/test-refs-be-lmdb
 /test-regex
 /test-revision-walking
 /test-run-command
diff --git a/Documentation/config.txt b/Documentation/config.txt
index ab57276..eccfdb6 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -496,6 +496,13 @@ core.repositoryFormatVersion::
 	Internal variable identifying the repository format and layout
 	version.
 
+core.refsBackendType::
+	Type of refs backend. Default is to use the original files
+	based backend. Set to 'lmdb' to activate the lmdb database
+	backend.  If you use the lmdb backend,
+	core.repositoryFormatVersion must be set to 1, and
+	extensions.refBackend must be set to 'lmdb'.
+
 core.sharedRepository::
 	When 'group' (or 'true'), the repository is made shareable between
 	several users in a group (making sure all the files and objects are
diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index d7a4cb0..6a4cf28 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -219,7 +219,7 @@ objects from the source repository into a pack in the cloned repository.
 
 --refs-backend-type=<name>::
 	Type of refs backend. Default is to use the original files based
-	backend.
+	backend. Set to "lmdb" to activate the lmdb database backend.
 
 <repository>::
 	The (possibly remote) repository to clone from.  See the
diff --git a/Documentation/git-init-db.txt b/Documentation/git-init-db.txt
index 648a6cd..72fbd71 100644
--- a/Documentation/git-init-db.txt
+++ b/Documentation/git-init-db.txt
@@ -9,7 +9,7 @@ git-init-db - Creates an empty Git repository
 SYNOPSIS
 --------
 [verse]
-'git init-db' [-q | --quiet] [--bare] [--template=<template_directory>] [--separate-git-dir <git dir>] [--shared[=<permissions>]]
+'git init-db' [-q | --quiet] [--bare] [--template=<template_directory>] [--separate-git-dir <git dir>] [--shared[=<permissions>]] [--refs-backend-type=<name>]
 
 
 DESCRIPTION
diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt
index 8174d27..d352788 100644
--- a/Documentation/git-init.txt
+++ b/Documentation/git-init.txt
@@ -12,6 +12,7 @@ SYNOPSIS
 'git init' [-q | --quiet] [--bare] [--template=<template_directory>]
 	  [--separate-git-dir <git dir>]
 	  [--shared[=<permissions>]] [directory]
+	  [--refs-backend-type=<name>]
 
 
 DESCRIPTION
@@ -113,6 +114,11 @@ does not exist, it will be created.
 
 --
 
+--refs-backend-type=<name>::
+Type of refs backend. Default is to use the original "files" backend,
+which stores ref data in files in .git/refs and .git/packed-refs.  Set
+to "lmdb" to activate the lmdb database backend.
+
 TEMPLATE DIRECTORY
 ------------------
 
diff --git a/Documentation/technical/refs-be-lmdb.txt b/Documentation/technical/refs-be-lmdb.txt
new file mode 100644
index 0000000..c497ffc
--- /dev/null
+++ b/Documentation/technical/refs-be-lmdb.txt
@@ -0,0 +1,50 @@
+Notes on the LMDB refs backend
+==============================
+
+Design:
+------
+
+Refs and reflogs are stored in a lmdb database in .git/refdb.  All
+keys and values are \0-terminated.
+
+Keys for refs are the name of the ref (e.g. refs/heads/master).
+Values are the value of the ref, in hex
+(e.g. 61f23eb0f81357c19fa91e2b8c6f3906c3a8f9b0).
+
+All per-worktree refs (refs/bisect/* and HEAD) are store using
+the traditional files-based backend.
+
+Reflogs are stored as a series of database entries.
+
+For non-empty reflogs, there is one entry per logged ref
+update.  The key format is logs/[refname]\0[timestamp].  The timestamp
+is a 64-bit unsigned integer number of nanoseconds since 1/1/1970.
+This means that reflog entries are chronologically ordered.  Because
+LMDB is a btree database, we can efficiently iterate over these keys.
+
+For an empty reflog, there is a "header" entry to show that a reflog
+exists.  The header has the same format as an ordinary reflog, but with
+a timeztamp of all zeros and an empty value.
+
+Reflog values are in the same format as the original files-based
+reflog.
+
+Weaknesses:
+-----------
+
+The reflog format is somewhat inefficient: a binary format could store
+reflog date/time information in somewhat less space.
+
+The rsync and file:// transports don't work yet, because they
+don't use the refs API.
+
+git new-workdir is incompatible with the lmdb backend.  Fortunately,
+git new-workdir is deprecated, and worktrees work fine.
+
+LMDB locks the entire database for write.  Any other writer waits
+until the first writer is done before beginning.  Readers do not wait
+for writers, and writers do not wait for readers.  The underlying
+scheme is approximately MVCC; each reader's queries see the state of
+the database as-of the time that the reader acquired its read lock.
+This is not too far off from the files backend, which loads all refs
+into memory when one is requested.
diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt
index 3d7106d..04d457d 100644
--- a/Documentation/technical/repository-version.txt
+++ b/Documentation/technical/repository-version.txt
@@ -79,3 +79,8 @@ The defined extensions are:
 
 This extension does not change git's behavior at all. It is useful only
 for testing format-1 compatibility.
+
+`refbackend`
+
+This extension allows the user of alternate ref backends.  The only
+defined value is `lmdb`.
diff --git a/Makefile b/Makefile
index 43ceab0..e96f84b 100644
--- a/Makefile
+++ b/Makefile
@@ -1036,6 +1036,17 @@ ifdef USE_LIBPCRE
 	EXTLIBS += -lpcre
 endif
 
+ifdef USE_LIBLMDB
+	BASIC_CFLAGS += -DUSE_LIBLMDB
+	ifdef LIBLMDBDIR
+		BASIC_CFLAGS += -I$(LIBLMDBDIR)/include
+		EXTLIBS += -L$(LIBLMDBDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBLMDBDIR)/$(lib)
+	endif
+	EXTLIBS += -llmdb
+	LIB_OBJS += refs-be-lmdb.o
+	TEST_PROGRAMS_NEED_X += test-refs-be-lmdb
+endif
+
 ifdef HAVE_ALLOCA_H
 	BASIC_CFLAGS += -DHAVE_ALLOCA_H
 endif
@@ -2122,6 +2133,7 @@ GIT-BUILD-OPTIONS: FORCE
 	@echo NO_CURL=\''$(subst ','\'',$(subst ','\'',$(NO_CURL)))'\' >>$@+
 	@echo NO_EXPAT=\''$(subst ','\'',$(subst ','\'',$(NO_EXPAT)))'\' >>$@+
 	@echo USE_LIBPCRE=\''$(subst ','\'',$(subst ','\'',$(USE_LIBPCRE)))'\' >>$@+
+	@echo USE_LIBLMDB=\''$(subst ','\'',$(subst ','\'',$(USE_LIBLMDB)))'\' >>$@+
 	@echo NO_PERL=\''$(subst ','\'',$(subst ','\'',$(NO_PERL)))'\' >>$@+
 	@echo NO_PYTHON=\''$(subst ','\'',$(subst ','\'',$(NO_PYTHON)))'\' >>$@+
 	@echo NO_UNIX_SOCKETS=\''$(subst ','\'',$(subst ','\'',$(NO_UNIX_SOCKETS)))'\' >>$@+
diff --git a/builtin/clone.c b/builtin/clone.c
index d489a87..c0f482c 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -764,7 +764,7 @@ static void write_config(struct string_list *config)
 	}
 
 	if (refs_backend_type &&
-	    write_one_config("core.refs-backend-type",
+	    write_one_config("core.refsBackendType",
 			     refs_backend_type, NULL) < 0)
 			die("unable to write backend parameter to config file");
 }
diff --git a/builtin/init-db.c b/builtin/init-db.c
index 2705739..0483e2d 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -179,6 +179,7 @@ static int create_default_files(const char *template_path)
 	int reinit;
 	int filemode;
 	struct strbuf err = STRBUF_INIT;
+	int repo_version = 0;
 
 	/* Just look for `init.templatedir` */
 	git_config(git_init_db_config, NULL);
@@ -204,6 +205,21 @@ static int create_default_files(const char *template_path)
 		adjust_shared_perm(get_git_dir());
 	}
 
+	if (refs_backend_type) {
+		struct refdb_config_data config_data = {NULL};
+		git_config_set("core.refsBackendType", refs_backend_type);
+		config_data.refs_backend_type = refs_backend_type;
+		config_data.refs_base = get_git_dir();
+#ifdef USE_LIBLMDB
+		register_refs_backend(&refs_be_lmdb);
+#endif
+		set_refs_backend(refs_backend_type, &config_data);
+		if (!strcmp(refs_backend_type, "lmdb")) {
+			git_config_set("extensions.refbackend", "lmdb");
+			repo_version = 1;
+		}
+	}
+
 	if (refs_initdb(&err, shared_repository))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -221,7 +237,7 @@ static int create_default_files(const char *template_path)
 
 	/* This forces creation of new config file */
 	xsnprintf(repo_version_string, sizeof(repo_version_string),
-		  "%d", GIT_REPO_VERSION);
+		  "%d", repo_version);
 	git_config_set("core.repositoryformatversion", repo_version_string);
 
 	/* Check filemode trustability */
@@ -469,6 +485,8 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)
 		OPT_BIT('q', "quiet", &flags, N_("be quiet"), INIT_DB_QUIET),
 		OPT_STRING(0, "separate-git-dir", &real_git_dir, N_("gitdir"),
 			   N_("separate git dir from working tree")),
+		OPT_STRING(0, "refs-backend-type", &refs_backend_type,
+			   N_("name"), N_("name of backend type to use")),
 		OPT_END()
 	};
 
diff --git a/config.c b/config.c
index 248a21a..c17c7b9 100644
--- a/config.c
+++ b/config.c
@@ -10,6 +10,7 @@
 #include "exec_cmd.h"
 #include "strbuf.h"
 #include "quote.h"
+#include "refs.h"
 #include "hashmap.h"
 #include "string-list.h"
 #include "utf8.h"
@@ -1207,6 +1208,39 @@ int git_config_early(config_fn_t fn, void *data, const char *repo_config)
 	}
 
 	if (repo_config && !access_or_die(repo_config, R_OK, 0)) {
+		struct refdb_config_data refdb_data = {NULL};
+		char *repo_config_copy;
+
+		/*
+		 * make sure we always read the backend config from the
+		 * core section on startup
+		 */
+		ret += git_config_from_file(refdb_config, repo_config,
+					    &refdb_data);
+
+		repo_config_copy = xstrdup(repo_config);
+		refdb_data.refs_base = strdup(dirname(repo_config_copy));
+		free(repo_config_copy);
+
+		if (!refdb_data.refs_backend_type)
+			refdb_data.refs_backend_type = "";
+
+		if ((!*refdb_data.refs_backend_type) ||
+		    (!strcmp(refdb_data.refs_backend_type, "files"))) {
+			/* default backend, nothing to do */
+		} else if (!strcmp(refdb_data.refs_backend_type, "lmdb")) {
+
+#ifdef USE_LIBLMDB
+			refs_backend_type = refdb_data.refs_backend_type;
+			register_refs_backend(&refs_be_lmdb);
+			set_refs_backend(refs_backend_type, &refdb_data);
+#else
+			die("Git was not built with USE_LIBLMDB, so the db refs backend is not available");
+#endif
+		} else {
+			die("Unknown ref backend type '%s'", refdb_data.refs_backend_type);
+		}
+
 		ret += git_config_from_file(fn, repo_config, data);
 		found += 1;
 	}
diff --git a/configure.ac b/configure.ac
index 63b7244..ea8e93d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -271,6 +271,24 @@ AS_HELP_STRING([],           [ARG can be also prefix for libpcre library and hea
         dnl it yet.
 	GIT_CONF_SUBST([LIBPCREDIR])
     fi)
+
+USE_LIBLMDB=YesPlease
+AC_ARG_WITH(liblmdb,
+AS_HELP_STRING([--with-liblmdb],[support lmdb (default is YES])
+AS_HELP_STRING([],           [ARG can be also prefix for liblmdb library and headers]),
+    if test "$withval" = "no"; then
+	USE_LIBLMDB=
+    elif test "$withval" = "yes"; then
+	USE_LIBLMDB=YesPlease
+    else
+	USE_LIBLMDB=YesPlease
+	LIBLMDBDIR=$withval
+	AC_MSG_NOTICE([Setting LIBLMDBDIR to $LIBLMDBDIR])
+        dnl USE_LIBLMDB can still be modified below, so don't substitute
+        dnl it yet.
+	GIT_CONF_SUBST([LIBLMDBDIR])
+    fi)
+
 #
 # Define HAVE_ALLOCA_H if you have working alloca(3) defined in that header.
 AC_FUNC_ALLOCA
@@ -510,6 +528,21 @@ GIT_CONF_SUBST([USE_LIBPCRE])
 
 fi
 
+if test -n "$USE_LIBLMDB"; then
+
+GIT_STASH_FLAGS($LIBLMDBDIR)
+
+AC_CHECK_LIB([lmdb], [mdb_env_open],
+[USE_LIBLMDB=YesPlease],
+[USE_LIBLMDB=])
+
+GIT_UNSTASH_FLAGS($LIBLMDBDIR)
+
+GIT_CONF_SUBST([USE_LIBLMDB])
+
+fi
+
+
 #
 # Define NO_CURL if you do not have libcurl installed.  git-http-pull and
 # git-http-push are not built, and you cannot use http:// and https://
diff --git a/contrib/workdir/git-new-workdir b/contrib/workdir/git-new-workdir
index 888c34a..66b7ecf 100755
--- a/contrib/workdir/git-new-workdir
+++ b/contrib/workdir/git-new-workdir
@@ -28,6 +28,9 @@ git_dir=$(cd "$orig_git" 2>/dev/null &&
   git rev-parse --git-dir 2>/dev/null) ||
   die "Not a git repository: \"$orig_git\""
 
+
+test "$(git config core.refsbackendtype)" = "lmdb" && die "git-new-workdir is incompatible with the refs lmdb backend"
+
 case "$git_dir" in
 .git)
 	git_dir="$orig_git/.git"
diff --git a/environment.c b/environment.c
index c5b65f5..6a2f639 100644
--- a/environment.c
+++ b/environment.c
@@ -65,6 +65,7 @@ int merge_log_config = -1;
 int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
 struct startup_info *startup_info;
 unsigned long pack_size_limit_cfg;
+const char *refs_backend_type;
 
 #ifndef PROTECT_HFS_DEFAULT
 #define PROTECT_HFS_DEFAULT 0
diff --git a/refs-be-lmdb.c b/refs-be-lmdb.c
new file mode 100644
index 0000000..3c41206
--- /dev/null
+++ b/refs-be-lmdb.c
@@ -0,0 +1,1997 @@
+/*
+ * This file implements a lmdb backend for refs.
+ *
+ * The design of this backend relies on lmdb's write lock -- that is, any
+ * write transaction blocks all other writers.  Thus, as soon as a ref
+ * transaction is opened, we know that any values we read won't
+ * change out from under us, and we have a fully-consistent view of the
+ * database.
+ *
+ * We store the content of refs including the trailing \0 so that
+ * standard C string functions can handle them.  Just like struct
+ * strbuf.
+ */
+#include "cache.h"
+#include <lmdb.h>
+#include "object.h"
+#include "refs.h"
+#include "tag.h"
+#include "lockfile.h"
+
+static struct trace_key db_trace = TRACE_KEY_INIT(LMDB);
+
+static MDB_env *env;
+
+static char *db_path;
+
+static struct ref_be *files_be;
+
+struct lmdb_transaction {
+	MDB_txn *txn;
+	MDB_dbi dbi;
+	MDB_cursor *cursor;
+	const char *submodule;
+	int flags;
+};
+
+struct lmdb_transaction transaction;
+
+static char *get_refdb_path(const char *base)
+{
+	struct strbuf path_buf = STRBUF_INIT;
+	strbuf_addf(&path_buf, "%s/refdb", base);
+	return strbuf_detach(&path_buf, NULL);
+}
+
+static int in_write_transaction(void)
+{
+	return transaction.txn && !(transaction.flags & MDB_RDONLY);
+}
+
+static void init_env(MDB_env **env, const char *path)
+{
+	int ret;
+	if (*env)
+		return;
+
+	ret = mdb_env_create(env);
+	if (ret)
+		die("mdb_env_create failed: %s", mdb_strerror(ret));
+	ret = mdb_env_set_maxreaders(*env, 1000);
+	if (ret)
+		die("BUG: mdb_env_set_maxreaders failed: %s", mdb_strerror(ret));
+	ret = mdb_env_set_mapsize(*env, (1<<30));
+	if (ret)
+		die("BUG: mdb_set_mapsize failed: %s", mdb_strerror(ret));
+	ret = mdb_env_open(*env, path, 0 , 0664);
+	if (ret)
+		die("BUG: mdb_env_open (%s) failed: %s", path,
+		    mdb_strerror(ret));
+}
+
+static int lmdb_initdb(struct strbuf *err, int shared)
+{
+	/*
+	 * To create a db, all we need to do is make a directory for
+	 * it to live in; lmdb will do the rest.
+	 */
+
+	assert(db_path);
+	if (mkdir(db_path, 0775) && errno != EEXIST) {
+		strbuf_addf(err, "%s", strerror(errno));
+		return -1;
+	}
+	return files_be->initdb(err, shared);
+}
+
+static void lmdb_init_backend(void *cbdata)
+{
+	struct refdb_config_data *data = (struct refdb_config_data *)cbdata;
+
+	if (db_path)
+		return;
+
+	db_path = xstrdup(real_path(get_refdb_path(data->refs_base)));
+
+	files_be = &refs_be_files;
+	files_be->init_backend(NULL);
+	trace_printf_key(&db_trace, "Init backend\n");
+}
+
+static void mdb_cursor_open_or_die(struct lmdb_transaction *transaction,
+				   MDB_cursor **cursor)
+{
+	int ret = mdb_cursor_open(transaction->txn, transaction->dbi, cursor);
+	if (ret)
+		die("mdb_cursor_open failed: %s", mdb_strerror(ret));
+}
+
+static void submodule_path(struct strbuf *sb, const char *submodule,
+			   const char *refname)
+{
+	if (submodule)
+		strbuf_git_path_submodule(sb, submodule, "%s", refname);
+	else
+		strbuf_git_path(sb, "%s", refname);
+}
+
+static int read_per_worktree_ref(const char *submodule, const char *refname,
+				 struct MDB_val *val)
+{
+	static struct strbuf sb = STRBUF_INIT;
+	static struct strbuf path = STRBUF_INIT;
+
+	submodule_path(&path, submodule, refname);
+
+	if (strbuf_read_file(&sb, path.buf, 200) < 0) {
+		strbuf_release(&sb);
+		strbuf_release(&path);
+		if (errno == ENOENT)
+			return MDB_NOTFOUND;
+		return -1;
+	}
+	strbuf_setlen(&sb, sb.len - 1);
+
+	val->mv_data = sb.buf;
+	val->mv_size = sb.len + 1;
+
+	strbuf_detach(&sb, NULL);
+	strbuf_release(&path);
+	return 0;
+}
+
+static void write_per_worktree_ref(const char *submodule, const char *refname,
+				   MDB_val *val)
+{
+	static struct lock_file lock;
+	int fd;
+	int len = val->mv_size - 1;
+	struct strbuf path = STRBUF_INIT;
+
+	submodule_path(&path, submodule, refname);
+	safe_create_leading_directories(path.buf);
+
+	fd = hold_lock_file_for_update(&lock, path.buf, LOCK_DIE_ON_ERROR);
+	strbuf_release(&path);
+
+	if (write_in_full(fd, val->mv_data, len) != len ||
+	    write_in_full(fd, "\n", 1) != 1)
+		die_errno("failed to write new HEAD");
+
+	if (commit_lock_file(&lock))
+		die_errno("failed to write new HEAD");
+}
+
+static int del_per_worktree_ref(const char *submodule, const char *refname,
+				MDB_val *val)
+{
+	struct strbuf path = STRBUF_INIT;
+	int result;
+
+	/*
+	 * Returning deleted ref data is not yet implemented, but no
+	 * callers need it.
+	 */
+	assert(val == NULL);
+
+	submodule_path(&path, submodule, refname);
+
+	result = unlink(path.buf);
+	strbuf_release(&path);
+	if (result && errno != ENOENT)
+		return 1;
+
+	return 0;
+}
+
+static int mdb_get_or_die(struct lmdb_transaction *transaction, MDB_val *key,
+			  MDB_val *val)
+{
+	int ret;
+
+	if (ref_type(key->mv_data) != REF_TYPE_NORMAL)
+		return read_per_worktree_ref(transaction->submodule,
+					     key->mv_data, val);
+
+	ret = mdb_get(transaction->txn, transaction->dbi, key, val);
+	if (ret) {
+		if (ret != MDB_NOTFOUND)
+			die("mdb_get failed: %s", mdb_strerror(ret));
+		return ret;
+	}
+	return 0;
+}
+
+static int mdb_del_or_die(struct lmdb_transaction *transaction, MDB_val *key,
+			  MDB_val *val)
+{
+	int ret;
+
+	if (ref_type(key->mv_data) != REF_TYPE_NORMAL)
+		die("BUG: this backend should only try to delete normal refs");
+
+	ret = mdb_del(transaction->txn, transaction->dbi, key, val);
+	if (ret) {
+		if (ret != MDB_NOTFOUND)
+			die("mdb_del failed: %s", mdb_strerror(ret));
+		return ret;
+	}
+	return 0;
+}
+
+static void mdb_put_or_die(struct lmdb_transaction *transaction, MDB_val *key,
+			   MDB_val *val, int mode)
+{
+	int ret;
+
+	if (ref_type(key->mv_data) != REF_TYPE_NORMAL)
+		die("BUG: this backend should only try to write normal refs");
+
+	ret = mdb_put(transaction->txn, transaction->dbi, key, val, mode);
+	if (ret) {
+		if (ret == MDB_BAD_VALSIZE)
+			die("Ref name %s too long (max size is %d)",
+			    (const char *)key->mv_data,
+			    mdb_env_get_maxkeysize(env));
+		else
+			die("mdb_put failed: (%s -> %s) %s",
+			    (const char *)key->mv_data,
+			    (const char *)val->mv_data, mdb_strerror(ret));
+	}
+}
+
+static int mdb_cursor_get_or_die(MDB_cursor *cursor, MDB_val *key, MDB_val *val, int mode)
+{
+	int ret;
+
+	ret = mdb_cursor_get(cursor, key, val, mode);
+	if (ret) {
+		if (ret != MDB_NOTFOUND)
+			die("mdb_cursor_get failed: %s", mdb_strerror(ret));
+		return ret;
+	}
+	assert(((char *)val->mv_data)[val->mv_size - 1] == 0);
+	return 0;
+}
+
+static int mdb_cursor_del_or_die(MDB_cursor *cursor, int flags)
+{
+	int ret = mdb_cursor_del(cursor, flags);
+	if (ret) {
+		if (ret != MDB_NOTFOUND)
+			die("mdb_cursor_del failed: %s", mdb_strerror(ret));
+		return ret;
+	}
+	return 0;
+}
+
+/*
+ * Begin a transaction. Because only one transaction per thread is
+ * permitted, we use a global transaction object.  If a read-write
+ * transaction is presently already in-progress, and a read-only
+ * transaction is requested, the read-write transaction will be
+ * returned instead.  If a read-write transaction is requested and a
+ * read-only transaction is open, the read-only transaction will be
+ * closed.
+ *
+ * It is a bug to request a read-write transaction during another
+ * read-write transaction.
+ *
+ * As a result, it is unsafe to retain read-only transactions past the
+ * point where a read-write transaction might be needed.  For
+ * instance, any call that has callbacks outside this module must
+ * conclude all of its reads from the database before calling those
+ * callbacks, or must reacquire the transaction after its callbacks
+ * are completed.
+ */
+int lmdb_transaction_begin_flags(struct strbuf *err, unsigned int flags)
+{
+	int ret;
+	MDB_txn *txn;
+	static size_t last_txnid = 0;
+	int force_restart = 0;
+	MDB_envinfo stat;
+
+	init_env(&env, db_path);
+
+	/*
+	 * Since each transaction sees a consistent view of the db,
+	 * downstream processes that write the db won't be seen in
+	 * this transaction.  We can check if the last transaction id
+	 * has changed since this read transaction was started, and if
+	 * so, we want to reopen the transaction.
+	 */
+
+	mdb_env_info(env, &stat);
+	if (stat.me_last_txnid != last_txnid) {
+		force_restart = 1;
+		last_txnid = stat.me_last_txnid;
+	}
+
+	if (!transaction.txn) {
+		ret = mdb_txn_begin(env, NULL, flags, &txn);
+		if (ret) {
+			strbuf_addf(err, "mdb_txn_begin failed: %s",
+				    mdb_strerror(ret));
+			return -1;
+		}
+		ret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);
+		if (ret) {
+			strbuf_addf(err, "mdb_txn_open failed: %s",
+				    mdb_strerror(ret));
+			return -1;
+		}
+		transaction.txn = txn;
+		transaction.flags = flags;
+		return 0;
+	}
+
+	if (transaction.flags == flags && !(flags & MDB_RDONLY))
+		die("BUG: rw transaction started during another rw txn");
+
+	if (force_restart || (transaction.flags != flags && transaction.flags & MDB_RDONLY)) {
+		/*
+		 * RO -> RW, or forced restart due to possible changes
+		 * from downstream processes.
+		 */
+		mdb_txn_abort(transaction.txn);
+		ret = mdb_txn_begin(env, NULL, flags, &txn);
+		if (ret) {
+			strbuf_addf(err, "restarting txn: mdb_txn_begin failed: %s",
+				    mdb_strerror(ret));
+			return -1;
+		}
+		ret = mdb_dbi_open(txn, NULL, 0, &transaction.dbi);
+		if (ret) {
+			strbuf_addf(err, "mdb_txn_open failed: %s",
+				    mdb_strerror(ret));
+			return -1;
+		}
+		transaction.txn = txn;
+		transaction.flags = flags;
+	}
+	/* RW -> RO just keeps the RW txn */
+	return 0;
+}
+
+static struct lmdb_transaction *lmdb_transaction_begin_flags_or_die(int flags)
+{
+	struct strbuf err = STRBUF_INIT;
+	if (lmdb_transaction_begin_flags(&err, flags))
+		die("%s", err.buf);
+	return &transaction;
+}
+
+#define MAXDEPTH 5
+
+static const char *parse_ref_data(struct lmdb_transaction *transaction,
+				  const char *refname, const char *ref_data,
+				  unsigned char *sha1, int resolve_flags,
+				  int *flags, int bad_name)
+{
+	int depth = MAXDEPTH;
+	const char *buf;
+	static struct strbuf refname_buffer = STRBUF_INIT;
+	static struct strbuf refdata_buffer = STRBUF_INIT;
+	MDB_val key, val;
+
+	for (;;) {
+		if (--depth < 0)
+			return NULL;
+
+		if (!starts_with(ref_data, "ref:")) {
+			if (get_sha1_hex(ref_data, sha1) ||
+			    (ref_data[40] != '\0' && !isspace(ref_data[40]))) {
+				if (flags)
+					*flags |= REF_ISBROKEN;
+				errno = EINVAL;
+				return NULL;
+			}
+
+			if (bad_name) {
+				hashclr(sha1);
+				if (flags)
+					*flags |= REF_ISBROKEN;
+			} else if (is_null_sha1(sha1)) {
+				if (flags)
+					*flags |= REF_ISBROKEN;
+			}
+			return refname;
+		}
+		if (flags)
+			*flags |= REF_ISSYMREF;
+		buf = ref_data + 4;
+		while (isspace(*buf))
+			buf++;
+		strbuf_reset(&refname_buffer);
+		strbuf_addstr(&refname_buffer, buf);
+		refname = refname_buffer.buf;
+		if (resolve_flags & RESOLVE_REF_NO_RECURSE) {
+			hashclr(sha1);
+			return refname;
+		}
+		if (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {
+			if (flags)
+				*flags |= REF_ISBROKEN;
+
+			if (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||
+			    !refname_is_safe(buf)) {
+				errno = EINVAL;
+				return NULL;
+			}
+			bad_name = 1;
+		}
+
+		key.mv_data = (char *)refname;
+		key.mv_size = strlen(refname) + 1;
+		if (mdb_get_or_die(transaction, &key, &val)) {
+			hashclr(sha1);
+			if (bad_name) {
+				if (flags)
+					*flags |= REF_ISBROKEN;
+			}
+			if (resolve_flags & RESOLVE_REF_READING)
+				return NULL;
+
+			return refname;
+		}
+		strbuf_reset(&refdata_buffer);
+		strbuf_add(&refdata_buffer, val.mv_data, val.mv_size);
+		ref_data = refdata_buffer.buf;
+	}
+	return refname;
+}
+
+static int verify_refname_available_txn(struct lmdb_transaction *transaction,
+					const char *refname,
+					struct string_list *extras,
+					struct string_list *skip,
+					struct strbuf *err)
+{
+	MDB_cursor *cursor;
+	MDB_val key;
+	MDB_val val;
+	int mdb_ret;
+	size_t refname_len;
+	char *search_key;
+	const char *extra_refname;
+	int ret = 1;
+	size_t i;
+
+	mdb_cursor_open_or_die(transaction, &cursor);
+
+	refname_len = strlen(refname) + 2;
+	key.mv_size = refname_len;
+	search_key = xmalloc(refname_len);
+	memcpy(search_key, refname, refname_len - 2);
+	search_key[refname_len - 2] = '/';
+	search_key[refname_len - 1] = 0;
+	key.mv_data = search_key;
+
+	/* Check for subdirs of refname: we start at refname/ */
+	mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);
+
+	while (!mdb_ret) {
+		if (starts_with(key.mv_data, refname) &&
+		    ((char*)key.mv_data)[refname_len - 2] == '/') {
+			if (skip && string_list_has_string(skip, key.mv_data))
+				goto next;
+
+			strbuf_addf(err, "'%s' exists; cannot create '%s'", (char *)key.mv_data, refname);
+			goto done;
+		}
+		break;
+	next:
+		mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);
+	}
+
+	/* Check for parent dirs of refname. */
+	for (i = 0; i < refname_len - 2; i++) {
+		if (search_key[i] == '/') {
+			search_key[i] = 0;
+			if (skip && string_list_has_string(skip, search_key)) {
+				search_key[i] = '/';
+				continue;
+			}
+
+			if (extras && string_list_has_string(extras, search_key)) {
+				strbuf_addf(err, "cannot process '%s' and '%s' at the same time",
+					    refname, search_key);
+				goto done;
+			}
+
+			key.mv_data = search_key;
+			key.mv_size = i + 1;
+			if (!mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET)) {
+				strbuf_addf(err, "'%s' exists; cannot create '%s'", (char *)key.mv_data, refname);
+				goto done;
+			}
+			search_key[i] = '/';
+		}
+	}
+
+	extra_refname = find_descendant_ref(refname, extras, skip);
+	if (extra_refname) {
+		strbuf_addf(err,
+			    "cannot process '%s' and '%s' at the same time",
+			    refname, extra_refname);
+		ret = 1;
+	} else {
+		ret = 0;
+	}
+done:
+	mdb_cursor_close(cursor);
+	free(search_key);
+	return ret;
+}
+
+static const char *resolve_ref_unsafe_txn(struct lmdb_transaction *transaction,
+					  const char *refname,
+					  int resolve_flags,
+					  unsigned char *sha1,
+					  int *flags)
+{
+	int bad_name = 0;
+	char *ref_data;
+	struct MDB_val key, val;
+	struct strbuf err = STRBUF_INIT;
+
+	val.mv_size = 0;
+	val.mv_data = NULL;
+
+	if (flags)
+		*flags = 0;
+
+	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
+		if (flags)
+			*flags |= REF_BAD_NAME;
+
+		if (!(resolve_flags & RESOLVE_REF_ALLOW_BAD_NAME) ||
+		    !refname_is_safe(refname)) {
+			errno = EINVAL;
+			return NULL;
+		}
+		/*
+		 * dwim_ref() uses REF_ISBROKEN to distinguish between
+		 * missing refs and refs that were present but invalid,
+		 * to complain about the latter to stderr.
+		 *
+		 * We don't know whether the ref exists, so don't set
+		 * REF_ISBROKEN yet.
+		 */
+		bad_name = 1;
+	}
+
+	key.mv_data = (void *)refname;
+	key.mv_size = strlen(refname) + 1;
+	if (mdb_get_or_die(transaction, &key, &val)) {
+		if (bad_name) {
+			hashclr(sha1);
+			if (flags)
+				*flags |= REF_ISBROKEN;
+		}
+
+		if (resolve_flags & RESOLVE_REF_READING)
+			return NULL;
+
+		if (verify_refname_available_txn(transaction, refname, NULL, NULL, &err)) {
+			error("%s", err.buf);
+			strbuf_release(&err);
+			return NULL;
+		}
+
+		hashclr(sha1);
+		return refname;
+	}
+
+	ref_data = val.mv_data;
+	assert(ref_data[val.mv_size - 1] == 0);
+
+	return parse_ref_data(transaction, refname, ref_data, sha1, resolve_flags,
+			      flags, bad_name);
+}
+
+static const char *lmdb_resolve_ref_unsafe(const char *refname, int resolve_flags,
+					   unsigned char *sha1, int *flags)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return resolve_ref_unsafe_txn(&transaction, refname,
+				      resolve_flags, sha1, flags);
+}
+
+static void write_u64(char *buf, uint64_t number)
+{
+	int i;
+
+	for (i = 0; i < 8; i++)
+		buf[i] = (number >> (i * 8)) & 0xff;
+}
+
+static int show_one_reflog_ent(struct strbuf *sb, each_reflog_ent_fn fn, void *cb_data)
+{
+	unsigned char osha1[20], nsha1[20];
+	char *email_end, *message;
+	unsigned long timestamp;
+	int tz;
+
+	/* old (raw) new (raw) name <email> SP time TAB msg LF */
+	if (sb->len < 41 || sb->buf[sb->len - 1] != '\n' ||
+	    !(email_end = strchr(sb->buf + 40, '>')) ||
+	    email_end[1] != ' ' ||
+	    !(timestamp = strtoul(email_end + 2, &message, 10)) ||
+	    !message || message[0] != ' ' ||
+	    (message[1] != '+' && message[1] != '-') ||
+	    !isdigit(message[2]) || !isdigit(message[3]) ||
+	    !isdigit(message[4]) || !isdigit(message[5]))
+		return 0; /* corrupt? */
+
+	hashcpy(osha1, (const unsigned char *)sb->buf);
+	hashcpy(nsha1, (const unsigned char *)sb->buf + 20);
+
+	email_end[1] = '\0';
+	tz = strtol(message + 1, NULL, 10);
+	if (message[6] != '\t')
+		message += 6;
+	else
+		message += 7;
+	return fn(osha1, nsha1, sb->buf + 40, timestamp, tz, message, cb_data);
+}
+
+static void format_reflog_entry(struct strbuf *buf,
+				const unsigned char *old_sha1,
+				const unsigned char *new_sha1,
+				const char *committer, const char *msg)
+{
+	int len;
+	int msglen;
+
+	assert(buf->len == 0);
+	strbuf_add(buf, old_sha1, 20);
+	strbuf_add(buf, new_sha1, 20);
+	strbuf_addstr(buf, committer);
+	strbuf_addch(buf, '\n');
+
+	len = buf->len;
+	msglen = msg ? strlen(msg) : 0;
+	if (msglen) {
+		int copied;
+		strbuf_grow(buf, msglen + 1);
+		copied = copy_reflog_msg(buf->buf + 40 + strlen(committer), msg) - 1;
+		buf->len = len + copied;
+		buf->buf[buf->len] = 0;
+	}
+}
+
+static int log_ref_write(const char *refname,
+			 const unsigned char *old_sha1,
+			 const unsigned char *new_sha1,
+			 const char *msg,
+			 int flags,
+			 struct strbuf *err)
+{
+	MDB_val key, val;
+	uint64_t now = getnanotime();
+	int result;
+	char *log_key;
+	int refname_len;
+	MDB_cursor *cursor;
+	struct strbuf buf = STRBUF_INIT;
+	const char *timestamp;
+
+	if (log_all_ref_updates < 0)
+		log_all_ref_updates = !is_bare_repository();
+
+	/* it is assumed that we are in a ref transaction here */
+	assert(transaction.txn);
+
+	result = safe_create_reflog(refname, flags & REF_FORCE_CREATE_REFLOG, err);
+	if (result)
+		return result;
+
+	/* "logs/" + refname + \0 + 8-byte timestamp for sorting and expiry. */
+	refname_len = strlen(refname);
+	key.mv_size = refname_len + 14;
+	log_key = xcalloc(1, key.mv_size);
+	sprintf(log_key, "logs/%s", refname);
+	key.mv_data = log_key;
+
+	mdb_cursor_open_or_die(&transaction, &cursor);
+
+	/* check that a reflog exists */
+	if (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||
+	    strcmp(key.mv_data, log_key)) {
+		free(log_key);
+		mdb_cursor_close(cursor);
+		return 0;
+	}
+
+	/* Is this a header?  We only need the header for empty reflogs */
+	timestamp = (const char *)key.mv_data + refname_len + 6;
+	if (ntohll(*(uint64_t *)timestamp) == 0)
+		mdb_cursor_del_or_die(cursor, 0);
+
+	key.mv_data = log_key;
+
+	write_u64((char *)key.mv_data + refname_len + 6, htonll(now));
+
+	format_reflog_entry(&buf, old_sha1, new_sha1,
+			    git_committer_info(0), msg);
+	assert(buf.len >= 42);
+	val.mv_data = buf.buf;
+	val.mv_size = buf.len + 1;
+
+	mdb_put_or_die(&transaction, &key, &val, 0);
+
+	strbuf_release(&buf);
+	free(log_key);
+	mdb_cursor_close(cursor);
+	return 0;
+}
+
+static int lmdb_verify_refname_available(const char *refname,
+					 struct string_list *extras,
+					 struct string_list *skip,
+					 struct strbuf *err)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return verify_refname_available_txn(&transaction, refname, extras, skip, err);
+}
+
+/*
+ * Attempt to resolve `refname` to `old_sha1` (if old_sha1 is
+ * non-null).  The return value is a pointer to a newly-allocated
+ * string containing the next ref name that this resolves to.  So if
+ * HEAD is a symbolic ref to refs/heads/example, which is itself a
+ * symbolic ref to refs/heads/foo, return refs/heads/example,
+ * and fill in resolved_sha1 with the sha of refs/heads/foo.
+ */
+static char *check_ref(MDB_txn *txn, const char *refname,
+		       const unsigned char *old_sha1,
+		       unsigned char *resolved_sha1, int flags,
+		       int *type_p)
+{
+	int mustexist = (old_sha1 && !is_null_sha1(old_sha1));
+	int resolve_flags = 0;
+	int type;
+	char *resolved_refname;
+
+	if (mustexist)
+		resolve_flags |= RESOLVE_REF_READING;
+	if (flags & REF_DELETING) {
+		resolve_flags |= RESOLVE_REF_ALLOW_BAD_NAME;
+		if (flags & REF_NODEREF)
+			resolve_flags |= RESOLVE_REF_NO_RECURSE;
+	}
+
+	/*
+	 * The first time we resolve the refname, we're just trying to
+	 * see if there is any ref at all by this name, even if it is
+	 * a broken symref.
+	 */
+	refname = resolve_ref_unsafe(refname, resolve_flags,
+				     resolved_sha1, &type);
+	if (type_p)
+	    *type_p = type;
+
+	if (!refname)
+		return NULL;
+
+	/*
+	 * Need to copy refname here because the resolve_ref_unsafe
+	 * returns a pointer to a static buffer that could get mangled
+	 * by the second call.
+	 */
+	resolved_refname = strdup(refname);
+
+	if (old_sha1) {
+		if (flags & REF_NODEREF) {
+			resolve_flags &= ~RESOLVE_REF_NO_RECURSE;
+
+			resolve_ref_unsafe(refname, resolve_flags,
+					   resolved_sha1, &type);
+		}
+		if (hashcmp(old_sha1, resolved_sha1)) {
+			error("ref %s is at %s but expected %s", refname,
+			      sha1_to_hex(resolved_sha1), sha1_to_hex(old_sha1));
+
+			return NULL;
+		}
+	}
+	return resolved_refname;
+}
+
+static int mdb_transaction_commit(struct lmdb_transaction *transaction,
+				  struct strbuf *err)
+{
+	int result;
+
+	result = mdb_txn_commit(transaction->txn);
+	if (result && err)
+		strbuf_addstr(err, mdb_strerror(result));
+
+	transaction->txn = NULL;
+	return result;
+}
+
+static int lmdb_delete_reflog(const char *refname)
+{
+	MDB_val key, val;
+	char *log_path;
+	int len;
+	MDB_cursor *cursor;
+	int ret = 0;
+	int mdb_ret;
+	struct strbuf err = STRBUF_INIT;
+	int in_transaction;
+
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return files_be->delete_reflog(refname);
+
+	in_transaction = in_write_transaction();
+
+	len = strlen(refname) + 6;
+	log_path = xmalloc(len);
+	sprintf(log_path, "logs/%s", refname);
+
+	key.mv_data = log_path;
+	key.mv_size = len;
+
+	if (!in_transaction)
+		lmdb_transaction_begin_flags_or_die(0);
+
+	mdb_cursor_open_or_die(&transaction, &cursor);
+
+	mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);
+
+	while (!mdb_ret) {
+		if (key.mv_size < len)
+			break;
+
+		if (!starts_with(key.mv_data, log_path) || ((char*)key.mv_data)[len - 1] != 0)
+			break;
+
+		mdb_cursor_del_or_die(cursor, 0);
+		mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);
+	}
+
+	free(log_path);
+	mdb_cursor_close(cursor);
+	transaction.cursor = NULL;
+
+	if (!in_transaction && mdb_transaction_commit(&transaction, &err)) {
+		warning("%s", err.buf);
+		ret = 01;
+	}
+	strbuf_release(&err);
+	return ret;
+}
+
+#define REF_NO_REFLOG 0x8000
+
+static int lmdb_transaction_update(struct lmdb_transaction *transaction,
+				   const char *refname,
+				   const unsigned char *new_sha1,
+				   const unsigned char *old_sha1,
+				   unsigned int flags, const char *msg,
+				   struct strbuf *err)
+{
+	const char *orig_refname = refname;
+	MDB_val key, val;
+	unsigned char resolved_sha1[20];
+	int type;
+	int ret = -1;
+
+	if ((flags & REF_HAVE_NEW) && is_null_sha1(new_sha1))
+		flags |= REF_DELETING;
+
+	if (new_sha1 && !is_null_sha1(new_sha1) &&
+	    check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
+		strbuf_addf(err, "refusing to update ref with bad name %s",
+			    refname);
+		return TRANSACTION_GENERIC_ERROR;
+	}
+
+	refname = check_ref(transaction->txn, orig_refname, old_sha1,
+			    resolved_sha1, flags, &type);
+	if (refname == NULL) {
+		strbuf_addf(err, "cannot lock the ref '%s'", orig_refname);
+		return TRANSACTION_GENERIC_ERROR;
+	}
+
+	if (!(flags & REF_DELETING) && is_null_sha1(resolved_sha1) &&
+	    verify_refname_available_txn(transaction, refname, NULL, NULL, err))
+		return TRANSACTION_NAME_CONFLICT;
+
+	if (flags & REF_NODEREF) {
+		free((void *)refname);
+		refname = orig_refname;
+	}
+
+	key.mv_size = strlen(refname) + 1;
+	key.mv_data = (void *)refname;
+
+	if ((flags & REF_HAVE_NEW) && !is_null_sha1(new_sha1)) {
+		int overwriting_symref = ((type & REF_ISSYMREF) &&
+					  (flags & REF_NODEREF));
+
+		struct object *o = parse_object(new_sha1);
+		if (!o) {
+			strbuf_addf(err,
+				    "Trying to write ref %s with nonexistent object %s",
+				    refname, sha1_to_hex(new_sha1));
+			goto done;
+		}
+		if (o->type != OBJ_COMMIT && is_branch(refname)) {
+			strbuf_addf(err,
+				    "Trying to write non-commit object %s to branch %s",
+				    sha1_to_hex(new_sha1), refname);
+			goto done;
+		}
+
+		if (!overwriting_symref
+		    && !hashcmp(resolved_sha1, new_sha1)) {
+			/*
+			 * The reference already has the desired
+			 * value, so we don't need to write it.
+			 */
+			flags |= REF_NO_REFLOG;
+		} else {
+			val.mv_size = 41;
+			if (new_sha1)
+				val.mv_data = sha1_to_hex(new_sha1);
+			else
+				val.mv_data = sha1_to_hex(null_sha1);
+			mdb_put_or_die(transaction, &key, &val, 0);
+		}
+	}
+
+	if (flags & REF_DELETING) {
+		if (mdb_del_or_die(transaction, &key, NULL)) {
+			if (old_sha1 && !is_null_sha1(old_sha1)) {
+				strbuf_addf(err, "No such ref %s", refname);
+				ret = TRANSACTION_GENERIC_ERROR;
+				goto done;
+			}
+		}
+		lmdb_delete_reflog(orig_refname);
+	} else if (!(flags & REF_NO_REFLOG)) {
+		if (!new_sha1)
+			new_sha1 = null_sha1;
+		if (log_ref_write(orig_refname, resolved_sha1,
+				  new_sha1, msg, flags, err) < 0)
+			goto done;
+		if (strcmp (refname, orig_refname) &&
+		    log_ref_write(refname, resolved_sha1,
+				  new_sha1, msg, flags, err) < 0)
+			goto done;
+	}
+
+	ret = 0;
+done:
+	if (refname != orig_refname)
+		free((void *) refname);
+	return ret;
+}
+
+static int lmdb_transaction_commit(struct ref_transaction *ref_transaction,
+				   struct string_list *affected_refnames,
+				   struct strbuf *err)
+{
+	int ret = 0, i;
+	int n = ref_transaction->nr;
+	struct lmdb_transaction *lmdb_transaction;
+	struct ref_update **updates = ref_transaction->updates;
+
+	/*
+	 * We might already be in a write transaction, because some
+	 * lmdb backend functionality is implemented in terms of
+	 * (other stuff) + ref_transaction_commit
+	 */
+	if (in_write_transaction())
+		lmdb_transaction = &transaction;
+	else
+		lmdb_transaction = lmdb_transaction_begin_flags_or_die(0);
+
+	for (i = 0; i < n; i++) {
+		struct ref_update *update = updates[i];
+
+		if (lmdb_transaction_update(lmdb_transaction,
+					    update->refname,
+					    update->new_sha1,
+					    (update->flags & REF_HAVE_OLD) ?
+					     update->old_sha1 : NULL,
+					    update->flags,
+					    update->msg,
+					    err)) {
+			mdb_txn_abort(lmdb_transaction->txn);
+			ret = -1;
+			goto cleanup;
+		}
+
+	}
+	ret = mdb_transaction_commit(lmdb_transaction, err);
+
+cleanup:
+	ref_transaction->state = REF_TRANSACTION_CLOSED;
+	return ret;
+}
+
+static int rename_reflog_ent(unsigned char *osha1, unsigned char *nsha1,
+			     const char *email, unsigned long timestamp, int tz,
+			     const char *message, void *cb_data)
+{
+
+	const char *newrefname = cb_data;
+	MDB_val key, new_key, val;
+
+	assert(transaction.cursor);
+
+	if (mdb_cursor_get_or_die(transaction.cursor, &key, &val, MDB_GET_CURRENT))
+		die("renaming ref: mdb_cursor_get failed to get current");
+
+	new_key.mv_size = strlen(newrefname) + 5 + 1 + 8;
+	new_key.mv_data = xmalloc(new_key.mv_size);
+	strcpy(new_key.mv_data, "logs/");
+	strcpy((char *)new_key.mv_data + 5, newrefname);
+	memcpy((char *)new_key.mv_data + new_key.mv_size - 8,
+	       (const char *)key.mv_data + key.mv_size - 8, 8);
+	mdb_put_or_die(&transaction, &new_key, &val, 0);
+	mdb_cursor_del_or_die(transaction.cursor, 0);
+	free(new_key.mv_data);
+	return 0;
+}
+
+static int lmdb_rename_ref(const char *oldref, const char *newref, const char *logmsg)
+{
+	unsigned char orig_sha1[20];
+	int flag = 0;
+	int log = reflog_exists(oldref);
+	const char *symref = NULL;
+	struct strbuf err = STRBUF_INIT;
+	struct ref_transaction *ref_transaction;
+
+	if (!strcmp(oldref, newref))
+		return 0;
+
+	lmdb_transaction_begin_flags_or_die(0);
+
+	ref_transaction = ref_transaction_begin(&err);
+	if (!ref_transaction)
+		die("%s", err.buf);
+
+	symref = resolve_ref_unsafe(oldref, RESOLVE_REF_READING,
+				    orig_sha1, &flag);
+	if (flag & REF_ISSYMREF) {
+		error("refname %s is a symbolic ref, renaming it is not supported",
+		      oldref);
+		goto fail;
+	}
+	if (!symref) {
+		mdb_txn_abort(transaction.txn);
+		error("refname %s not found", oldref);
+		goto fail;
+	}
+	if (!rename_ref_available(oldref, newref))
+		goto fail;
+
+	/* Copy the reflog from the old to the new */
+	if (log) {
+		struct strbuf old_log_sentinel = STRBUF_INIT;
+		MDB_val key;
+		int log_all;
+
+		log_all = log_all_ref_updates;
+		log_all_ref_updates = 1;
+		if (safe_create_reflog(newref, 0, &err)) {
+			error("can't create reflog for %s: %s", newref, err.buf);
+			strbuf_release(&err);
+			goto fail;
+		}
+		log_all_ref_updates = log_all;
+
+		for_each_reflog_ent(oldref, rename_reflog_ent, (void *)newref);
+		strbuf_addf(&old_log_sentinel, "logs/%sxxxxxxxx", oldref);
+		memset(old_log_sentinel.buf + old_log_sentinel.len - 8, 0, 8);
+
+		key.mv_size = old_log_sentinel.len;
+		key.mv_data = old_log_sentinel.buf;
+
+		/* It's OK if the old reflog is missing */
+		mdb_del_or_die(&transaction, &key, NULL);
+		strbuf_release(&old_log_sentinel);
+	}
+
+	if (ref_transaction_delete(ref_transaction, oldref,
+				   orig_sha1, REF_NODEREF, NULL, &err)) {
+		error("unable to delete old %s", oldref);
+		goto fail;
+	}
+
+	if (ref_transaction_update(ref_transaction, newref, orig_sha1, NULL,
+				    0, logmsg, &err)) {
+		error("%s", err.buf);
+		goto fail;
+	}
+
+	if (ref_transaction_commit(ref_transaction, &err)) {
+		error("%s", err.buf);
+		goto fail;
+	}
+
+	return 0;
+
+fail:
+	ref_transaction_free(ref_transaction);
+	strbuf_release(&err);
+	mdb_txn_abort(transaction.txn);
+	return 1;
+}
+
+static int lmdb_for_each_reflog_ent_order(const char *refname,
+					  each_reflog_ent_fn fn,
+					  void *cb_data, int reverse)
+{
+	MDB_val key, val;
+	char *search_key;
+	char *log_path;
+	int len;
+	MDB_cursor *cursor;
+	int ret = 0;
+	struct strbuf sb = STRBUF_INIT;
+	enum MDB_cursor_op direction = reverse ? MDB_PREV : MDB_NEXT;
+	uint64_t zero = 0ULL;
+
+	len = strlen(refname) + 6;
+	log_path = xmalloc(len);
+	search_key = xmalloc(len + 1);
+	sprintf(log_path, "logs/%s", refname);
+	strcpy(search_key, log_path);
+
+	if (reverse) {
+		/*
+		 * For a reverse search, start at the key
+		 * lexicographically after the searched-for key.
+		 * That's the one with \001 appended to the key.
+		 */
+
+		search_key[len - 1] = 1;
+		search_key[len] = 0;
+		key.mv_size = len + 1;
+	} else {
+		key.mv_size = len;
+	}
+
+	key.mv_data = search_key;
+
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+
+	mdb_cursor_open_or_die(&transaction, &cursor);
+
+	transaction.cursor = cursor;
+
+	/*
+	 * MDB's cursor API requires that the first mdb_cursor_get be
+	 * called with MDB_SET_RANGE.  For reverse searches, this will
+	 * give us the entry one-past the entry we're looking for, so
+	 * we should jump back using MDB_PREV.
+	 */
+	mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);
+	if (direction == MDB_PREV)
+		mdb_cursor_get_or_die(cursor, &key, &val, direction);
+
+	do {
+		if (key.mv_size < len)
+			break;
+
+		if (!starts_with(key.mv_data, log_path) || ((char *)key.mv_data)[len - 1] != 0)
+			break;
+
+		if (!memcmp(&zero, ((char *)key.mv_data) + key.mv_size - 8, 8))
+			continue;
+
+		assert(val.mv_size != 0);
+
+		strbuf_add(&sb, val.mv_data, val.mv_size - 1);
+		ret = show_one_reflog_ent(&sb, fn, cb_data);
+		if (ret)
+			break;
+
+		strbuf_reset(&sb);
+	} while (!mdb_cursor_get_or_die(cursor, &key, &val, direction));
+
+	strbuf_release(&sb);
+	free(log_path);
+	free(search_key);
+	mdb_cursor_close(cursor);
+	return ret;
+}
+
+static int lmdb_for_each_reflog_ent(const char *refname,
+				    each_reflog_ent_fn fn,
+				    void *cb_data)
+{
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return files_be->for_each_reflog_ent(refname, fn, cb_data);
+	return lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 0);
+}
+
+static int lmdb_for_each_reflog_ent_reverse(const char *refname,
+					    each_reflog_ent_fn fn,
+					    void *cb_data)
+{
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return files_be->for_each_reflog_ent_reverse(refname, fn, cb_data);
+	return lmdb_for_each_reflog_ent_order(refname, fn, cb_data, 1);
+}
+
+static int lmdb_reflog_exists(const char *refname)
+{
+	MDB_val key, val;
+	char *log_path;
+	int len;
+	MDB_cursor *cursor;
+	int ret = 1;
+
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return files_be->reflog_exists(refname);
+
+	len = strlen(refname) + 6;
+	log_path = xmalloc(len);
+	sprintf(log_path, "logs/%s", refname);
+
+	key.mv_data = log_path;
+	key.mv_size = len;
+
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	mdb_cursor_open_or_die(&transaction, &cursor);
+
+	if (mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE) ||
+	    !starts_with(key.mv_data, log_path))
+		ret = 0;
+
+	free(log_path);
+	mdb_cursor_close(cursor);
+
+	return ret;
+}
+
+struct wrapped_each_ref_fn {
+	each_ref_fn *fn;
+	void *cb_data;
+};
+
+static int check_reflog(const char *refname,
+			const struct object_id *oid, int flags, void *cb_data)
+{
+	struct wrapped_each_ref_fn *wrapped = cb_data;
+
+	if (reflog_exists(refname))
+		return wrapped->fn(refname, oid, 0, wrapped->cb_data);
+
+	return 0;
+}
+
+static int lmdb_for_each_reflog(each_ref_fn fn, void *cb_data)
+{
+	struct wrapped_each_ref_fn wrapped = {fn, cb_data};
+	int result = head_ref(fn, cb_data);
+	if (result)
+		return result;
+	return for_each_ref(check_reflog, &wrapped);
+}
+
+static int lmdb_create_reflog(const char *refname, int force_create, struct strbuf *err)
+{
+	/*
+	 * We mark that there is a reflog by creating a key of the
+	 * form logs/$refname followed by nine \0 (one for
+	 * string-termination, 8 in lieu of a timestamp), with an empty
+	 * value.
+	 */
+
+	int in_transaction = in_write_transaction();
+	MDB_val key, val;
+
+	if (!force_create && !should_autocreate_reflog(refname))
+		return 0;
+
+	if (!in_transaction)
+		lmdb_transaction_begin_flags_or_die(0);
+
+	key.mv_size = strlen(refname) + 5 + 1 + 8;
+	key.mv_data = xcalloc(1, key.mv_size);
+	sprintf((char *)key.mv_data, "logs/%s", refname);
+	val.mv_size = 0;
+	val.mv_data = NULL;
+	mdb_put_or_die(&transaction, &key, &val, 0);
+
+	free(key.mv_data);
+	if (!in_transaction)
+		return mdb_transaction_commit(&transaction, err);
+	return 0;
+}
+
+struct expire_reflog_cb {
+	unsigned int flags;
+	reflog_expiry_should_prune_fn *should_prune_fn;
+	void *policy_cb;
+	unsigned char last_kept_sha1[20];
+};
+
+static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,
+			     const char *email, unsigned long timestamp, int tz,
+			     const char *message, void *cb_data)
+{
+	struct expire_reflog_cb *cb = cb_data;
+	struct expire_reflog_policy_cb *policy_cb = cb->policy_cb;
+
+	if (cb->flags & EXPIRE_REFLOGS_REWRITE)
+		osha1 = cb->last_kept_sha1;
+
+	if ((*cb->should_prune_fn)(osha1, nsha1, email, timestamp, tz,
+				   message, policy_cb)) {
+		if (cb->flags & EXPIRE_REFLOGS_DRY_RUN)
+			printf("would prune %s", message);
+		else {
+			if (cb->flags & EXPIRE_REFLOGS_VERBOSE)
+				printf("prune %s", message);
+
+			mdb_cursor_del_or_die(transaction.cursor, 0);
+		}
+	} else {
+		hashcpy(cb->last_kept_sha1, nsha1);
+		if (cb->flags & EXPIRE_REFLOGS_VERBOSE)
+			printf("keep %s", message);
+	}
+	return 0;
+}
+
+static int write_ref(const char *refname, const unsigned char *sha1)
+{
+	struct strbuf err = STRBUF_INIT;
+	struct ref_transaction *transaction;
+
+	transaction = ref_transaction_begin(&err);
+	if (!transaction) {
+		error("%s", err.buf);
+		strbuf_release(&err);
+		return -1;
+	}
+
+	if (ref_transaction_update(transaction, refname, sha1, NULL,
+				   REF_NO_REFLOG, NULL, &err)) {
+		error("%s", err.buf);
+		strbuf_release(&err);
+		return -1;
+	}
+
+	if (ref_transaction_commit(transaction, &err)) {
+		error("%s", err.buf);
+		strbuf_release(&err);
+		return -1;
+	}
+
+	return 0;
+}
+
+int lmdb_reflog_expire(const char *refname, const unsigned char *sha1,
+		       unsigned int flags,
+		       reflog_expiry_prepare_fn prepare_fn,
+		       reflog_expiry_should_prune_fn should_prune_fn,
+		       reflog_expiry_cleanup_fn cleanup_fn,
+		       void *policy_cb_data)
+{
+	struct expire_reflog_cb cb;
+	int dry_run = flags & EXPIRE_REFLOGS_DRY_RUN;
+	int status = 0;
+	struct strbuf err = STRBUF_INIT;
+	unsigned char resolved_sha1[20];
+	int type;
+	char *resolved;
+
+	if (ref_type(refname) != REF_TYPE_NORMAL)
+		return files_be->reflog_expire(refname, sha1, flags, prepare_fn,
+					       should_prune_fn, cleanup_fn,
+					       policy_cb_data);
+
+	memset(&cb, 0, sizeof(cb));
+	cb.flags = flags;
+	cb.policy_cb = policy_cb_data;
+	cb.should_prune_fn = should_prune_fn;
+
+	lmdb_transaction_begin_flags_or_die(dry_run ? MDB_RDONLY : 0);
+
+	resolved = check_ref(transaction.txn, refname, sha1,
+			     resolved_sha1, 0, &type);
+	if (!resolved)
+		die("Failed to resolve %s", refname);
+	free(resolved);
+
+	(*prepare_fn)(refname, sha1, cb.policy_cb);
+	lmdb_for_each_reflog_ent(refname, expire_reflog_ent, &cb);
+	(*cleanup_fn)(cb.policy_cb);
+
+	if (!dry_run) {
+		/*
+		 * It doesn't make sense to adjust a reference pointed
+		 * to by a symbolic ref based on expiring entries in
+		 * the symbolic reference's reflog. Nor can we update
+		 * a reference if there are no remaining reflog
+		 * entries.
+		 */
+		int update = (flags & EXPIRE_REFLOGS_UPDATE_REF) &&
+			!(type & REF_ISSYMREF) &&
+			!is_null_sha1(cb.last_kept_sha1);
+
+		if (mdb_transaction_commit(&transaction, &err)) {
+			status |= error("couldn't write logs/%s: %s", refname,
+					err.buf);
+			strbuf_release(&err);
+		} else if (update &&
+			   write_ref(refname, cb.last_kept_sha1)) {
+			status |= error("couldn't set %s",
+					refname);
+		}
+	}
+	return status;
+}
+
+static int lmdb_pack_refs(unsigned int flags)
+{
+	/* This concept does not exist in this backend. */
+	return 0;
+}
+
+static int lmdb_peel_ref(const char *refname, unsigned char *sha1)
+{
+	int flag;
+	unsigned char base[20];
+
+	if (read_ref_full(refname, RESOLVE_REF_READING, base, &flag))
+		return -1;
+
+	return peel_object(base, sha1);
+}
+
+static int lmdb_create_symref(const char *ref_target,
+			      const char *refs_heads_master,
+			      const char *logmsg)
+{
+
+	struct strbuf err = STRBUF_INIT;
+	unsigned char old_sha1[20], new_sha1[20];
+	MDB_val key, val;
+	char *valdata;
+	int ret = 0;
+	int in_transaction;
+
+	in_transaction = in_write_transaction();
+
+	if (logmsg && read_ref(ref_target, old_sha1))
+		hashclr(old_sha1);
+
+	key.mv_size = strlen(ref_target) + 1;
+	key.mv_data = xstrdup(ref_target);
+
+	val.mv_size = strlen(refs_heads_master) + 1 + 5;
+	valdata = xmalloc(val.mv_size);
+	sprintf(valdata, "ref: %s", refs_heads_master);
+	val.mv_data = valdata;
+
+	if (!in_transaction)
+		lmdb_transaction_begin_flags_or_die(0);
+
+	mdb_put_or_die(&transaction, &key, &val, 0);
+
+	if (logmsg && !read_ref(refs_heads_master, new_sha1) &&
+	    log_ref_write(ref_target, old_sha1, new_sha1, logmsg, 0, &err)) {
+		error("create_symref: log_ref_write failed: %s", err.buf);
+		ret = -1;
+		goto done;
+	}
+
+	if (!in_transaction && mdb_transaction_commit(&transaction, &err)) {
+		error("create_symref: commit failed: %s", err.buf);
+		ret = -1;
+	}
+
+done:
+	strbuf_release(&err);
+	free(key.mv_data);
+	free(valdata);
+
+	return ret;
+}
+
+MDB_env *submodule_txn_begin(struct lmdb_transaction *transaction)
+{
+	int ret;
+	MDB_env *submodule_env = NULL;
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_git_path_submodule(&path, transaction->submodule, "refdb");
+
+	if (!is_directory(path.buf))
+		goto done;
+
+	mkdir(path.buf, 0775);
+
+	init_env(&submodule_env, path.buf);
+
+	ret = mdb_txn_begin(submodule_env, NULL, MDB_RDONLY, &transaction->txn);
+	if (ret) {
+		die("mdb_txn_begin failed: %s", mdb_strerror(ret));
+
+	}
+	ret = mdb_dbi_open(transaction->txn, NULL, 0, &transaction->dbi);
+	if (ret) {
+		die("mdb_txn_open failed: %s", mdb_strerror(ret));
+	}
+
+done:
+	strbuf_release(&path);
+	return submodule_env;
+}
+
+static int lmdb_resolve_gitlink_ref(const char *submodule, const char *refname,
+				    unsigned char *sha1)
+{
+	struct lmdb_transaction transaction;
+	MDB_env *submodule_env;
+	int result;
+
+	transaction.txn = NULL;
+	transaction.submodule = submodule;
+	submodule_env = submodule_txn_begin(&transaction);
+	if (!submodule_env)
+		return -1;
+	result = !resolve_ref_unsafe_txn(&transaction, refname,
+					 RESOLVE_REF_READING, sha1, NULL);
+
+	mdb_txn_abort(transaction.txn);
+	mdb_env_close(submodule_env);
+	return result ? -1 : 0;
+}
+
+static int do_head_ref(const char *submodule, each_ref_fn fn, void *cb_data)
+{
+	struct object_id oid;
+	int flag;
+
+	if (submodule) {
+		if (resolve_gitlink_ref(submodule, "HEAD", oid.hash) == 0)
+			return fn("HEAD", &oid, 0, cb_data);
+
+		return 0;
+	}
+
+	if (!read_ref_full("HEAD", RESOLVE_REF_READING, oid.hash, &flag))
+		return fn("HEAD", &oid, flag, cb_data);
+
+	return 0;
+}
+
+static int lmdb_head_ref(each_ref_fn fn, void *cb_data)
+{
+	return do_head_ref(NULL, fn, cb_data);
+}
+
+static int lmdb_head_ref_submodule(const char *submodule, each_ref_fn fn,
+				   void *cb_data)
+{
+	return do_head_ref(submodule, fn, cb_data);
+}
+
+/*
+ * Call fn for each reference for which the refname begins with base.
+ * If trim is non-zero, then trim that many characters off the
+ * beginning of each refname before passing the refname to fn.  flags
+ * can be DO_FOR_EACH_INCLUDE_BROKEN to include broken references in
+ * the iteration.  If fn ever returns a non-zero value, stop the
+ * iteration and return that value; otherwise, return 0.
+ */
+static int do_for_each_ref(struct lmdb_transaction *transaction,
+			   const char *base, each_ref_fn fn, int trim,
+			   int flags, void *cb_data)
+{
+
+	MDB_val key, val;
+	MDB_cursor *cursor;
+	int baselen;
+	char *search_key;
+	int retval;
+	int mdb_ret;
+
+	retval = do_for_each_per_worktree_ref(transaction->submodule, base, fn,
+					      trim, flags, cb_data);
+	if (retval)
+		return retval;
+
+	if (ref_paranoia < 0)
+		ref_paranoia = git_env_bool("GIT_REF_PARANOIA", 0);
+	if (ref_paranoia)
+		flags |= DO_FOR_EACH_INCLUDE_BROKEN;
+
+	if (!base || !*base) {
+		base = "refs/";
+		trim = 0;
+	}
+
+	baselen = strlen(base);
+	search_key = xmalloc(baselen + 1);
+	strcpy(search_key, base);
+	key.mv_size = baselen + 1;
+	key.mv_data = search_key;
+
+	mdb_cursor_open_or_die(transaction, &cursor);
+
+	mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);
+
+	while (!mdb_ret) {
+		struct object_id oid;
+		int parsed_flags = 0;
+
+		if (memcmp(key.mv_data, base, baselen))
+			break;
+
+		parse_ref_data(transaction, (const char *)key.mv_data + (trim ? baselen : 0),
+			       val.mv_data, oid.hash, 0, &parsed_flags, 0);
+
+		if (flags & DO_FOR_EACH_INCLUDE_BROKEN ||
+		    (!(parsed_flags & REF_ISBROKEN) &&
+		     has_sha1_file(oid.hash))) {
+			retval = fn((const char *)key.mv_data + (trim ? baselen : 0), &oid, parsed_flags, cb_data);
+			if (retval)
+				break;
+		}
+
+		mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);
+	}
+
+	mdb_cursor_close(cursor);
+	free(search_key);
+
+	return retval;
+}
+
+static int lmdb_for_each_ref(each_ref_fn fn, void *cb_data)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, "", fn, 0, 0, cb_data);
+}
+
+static int lmdb_for_each_ref_submodule(const char *submodule, each_ref_fn fn,
+				       void *cb_data)
+{
+	struct lmdb_transaction transaction;
+	MDB_env *submodule_env;
+	int result;
+
+	if (!submodule)
+		return for_each_ref(fn, cb_data);
+
+	transaction.txn = NULL;
+	transaction.submodule = submodule;
+
+	submodule_env = submodule_txn_begin(&transaction);
+	if (!submodule_env)
+		return 0;
+	result = do_for_each_ref(&transaction, "", fn, 0, 0, cb_data);
+	mdb_txn_abort(transaction.txn);
+	mdb_env_close(submodule_env);
+	return result;
+}
+
+static int lmdb_for_each_ref_in(const char *prefix, each_ref_fn fn,
+				 void *cb_data)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, prefix, fn, strlen(prefix),
+			       0, cb_data);
+}
+
+static int lmdb_for_each_fullref_in(const char *prefix, each_ref_fn fn,
+				    void *cb_data, unsigned int broken)
+{
+	unsigned int flag = 0;
+
+	if (broken)
+		flag = DO_FOR_EACH_INCLUDE_BROKEN;
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, prefix, fn, 0, flag, cb_data);
+}
+
+static int lmdb_for_each_ref_in_submodule(const char *submodule,
+					  const char *prefix,
+					  each_ref_fn fn, void *cb_data)
+{
+	struct lmdb_transaction transaction = {NULL};
+	MDB_env *submodule_env;
+	int result;
+
+	if (!submodule)
+		return for_each_ref_in(prefix, fn, cb_data);
+
+	transaction.submodule = submodule;
+	submodule_env = submodule_txn_begin(&transaction);
+	if (!submodule_env)
+		return 0;
+	result = do_for_each_ref(&transaction, prefix, fn,
+				 strlen(prefix), 0, cb_data);
+	mdb_txn_abort(transaction.txn);
+	mdb_env_close(submodule_env);
+	return result;
+}
+
+static int lmdb_for_each_replace_ref(each_ref_fn fn, void *cb_data)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, git_replace_ref_base, fn,
+			       strlen(git_replace_ref_base), 0, cb_data);
+}
+
+static int lmdb_for_each_reftype_fullpath(each_ref_fn fn, char *type, unsigned int broken, void *cb_data)
+{
+	unsigned int flag = 0;
+
+	if (broken)
+		flag = DO_FOR_EACH_INCLUDE_BROKEN;
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, type, fn, 0, flag, cb_data);
+}
+
+static int lmdb_for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
+{
+	struct strbuf buf = STRBUF_INIT;
+	int ret;
+
+	strbuf_addf(&buf, "%srefs/", get_git_namespace());
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	ret = do_for_each_ref(&transaction, buf.buf, fn, 0, 0, cb_data);
+	strbuf_release(&buf);
+	return ret;
+}
+
+static int lmdb_for_each_rawref(each_ref_fn fn, void *cb_data)
+{
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	return do_for_each_ref(&transaction, "", fn, 0,
+			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
+}
+
+/* For testing only! */
+int test_refdb_raw_read(const char *key)
+{
+	MDB_val key_val, val;
+	char *keydup;
+	int ret;
+
+	lmdb_transaction_begin_flags_or_die(MDB_RDONLY);
+	keydup = xstrdup(key);
+	key_val.mv_data = keydup;
+	key_val.mv_size = strlen(key) + 1;
+
+	ret = mdb_get_or_die(&transaction, &key_val, &val);
+	free(keydup);
+	switch (ret) {
+	case 0:
+		printf("%s\n", (char *)val.mv_data);
+		return 0;
+	case MDB_NOTFOUND:
+		fprintf(stderr, "%s not found\n", key);
+		return 1;
+	default:
+		return 2;
+	}
+}
+
+/* For testing only! */
+void test_refdb_raw_write(const char *key, const char *value)
+{
+	MDB_val key_val, val;
+	char *keydup, *valdup;
+
+	if (ref_type(key) != REF_TYPE_NORMAL) {
+		val.mv_data = (void *)value;
+		val.mv_size = strlen(value) + 1;
+		write_per_worktree_ref(NULL, key, &val);
+		return;
+	}
+
+	lmdb_transaction_begin_flags_or_die(0);
+
+	keydup = xstrdup(key);
+	key_val.mv_data = keydup;
+	key_val.mv_size = strlen(key) + 1;
+
+	valdup = xstrdup(value);
+	val.mv_data = valdup;
+	val.mv_size = strlen(value) + 1;
+
+	mdb_put_or_die(&transaction, &key_val, &val, 0);
+	assert(mdb_transaction_commit(&transaction, NULL) == 0);
+
+	free(keydup);
+	free(valdup);
+}
+
+/* For testing only! */
+int test_refdb_raw_delete(const char *key)
+{
+	MDB_val key_val;
+	char *keydup;
+	int ret;
+
+	if (ref_type(key) != REF_TYPE_NORMAL)
+		return del_per_worktree_ref(NULL, key, NULL);
+
+	lmdb_transaction_begin_flags_or_die(0);
+	keydup = xstrdup(key);
+	key_val.mv_data = keydup;
+	key_val.mv_size = strlen(key) + 1;
+
+	ret = mdb_del_or_die(&transaction, &key_val, NULL);
+
+	assert(mdb_transaction_commit(&transaction, NULL) == 0);
+
+	free(keydup);
+	return ret;
+}
+
+static int print_raw_reflog_ent(unsigned char *osha1, unsigned char *nsha1,
+				const char *email, unsigned long timestamp,
+				int tz, const char *message, void *cb_data)
+{
+	int *any = cb_data;
+	*any = 1;
+
+	if (*message != '\n')
+		printf("%s %s %s %lu %+05d\t%s", sha1_to_hex(osha1),
+		       sha1_to_hex(nsha1),
+		       email, timestamp, tz, message);
+	else
+		printf("%s %s %s %lu %+05d\n", sha1_to_hex(osha1),
+		       sha1_to_hex(nsha1),
+		       email, timestamp, tz);
+	return 0;
+}
+
+/* For testing only! */
+int test_refdb_raw_reflog(const char *refname)
+{
+	int any = 0;
+
+	for_each_reflog_ent(refname, print_raw_reflog_ent, &any);
+
+	return !any;
+}
+
+/* For testing only! */
+void test_refdb_raw_delete_reflog(char *refname)
+{
+	MDB_val key, val;
+	int mdb_ret;
+	char *search_key;
+	MDB_cursor *cursor;
+	int len;
+
+	if (refname) {
+		len = strlen(refname) + 5 + 1; /* logs/ + 0*/
+		search_key = xmalloc(len);
+		sprintf(search_key, "logs/%s", refname);
+	} else {
+		len = 6; /* logs/ + 0*/
+		search_key = xstrdup("logs/");
+	}
+	key.mv_data = search_key;
+	key.mv_size = len;
+
+	lmdb_transaction_begin_flags_or_die(0);
+
+	mdb_cursor_open_or_die(&transaction, &cursor);
+
+	mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_SET_RANGE);
+	while (!mdb_ret) {
+		if (!starts_with(key.mv_data, search_key))
+			break;
+		if (refname && ((char *)val.mv_data)[len - 1] == 0)
+			break;
+
+		mdb_cursor_del_or_die(cursor, 0);
+		mdb_ret = mdb_cursor_get_or_die(cursor, &key, &val, MDB_NEXT);
+	}
+
+	free(search_key);
+	mdb_cursor_close(cursor);
+
+	assert(mdb_transaction_commit(&transaction, NULL) == 0);
+	return;
+}
+
+static void format_lmdb_reflog_ent(struct strbuf *dst, struct strbuf *src)
+{
+	unsigned char osha1[20], nsha1[20];
+	const char *msg;
+
+	get_sha1_hex(src->buf, osha1);
+	get_sha1_hex(src->buf + 41, nsha1);
+
+	msg = strchr(src->buf + 82, '\n');
+	assert(msg);
+	msg += 1;
+
+	format_reflog_entry(dst, osha1, nsha1, src->buf + 82, msg);
+}
+
+/* For testing only! */
+void test_refdb_raw_append_reflog(const char *refname)
+{
+	struct strbuf input = STRBUF_INIT;
+	uint64_t now = getnanotime();
+	MDB_val key, val;
+
+	key.mv_size = strlen(refname) + 14;
+	key.mv_data = xcalloc(1, key.mv_size);
+	sprintf(key.mv_data, "logs/%s", refname);
+
+	lmdb_transaction_begin_flags_or_die(0);
+
+	/* We do not remove the header here, because this is just for
+	 * tests, so it's OK to be a bit inefficient */
+
+	while (strbuf_getwholeline(&input, stdin, '\n') != EOF) {
+		struct strbuf sb = STRBUF_INIT;
+
+		/* "logs/" + \0 + 8-byte timestamp for sorting and expiry */
+		write_u64((char *)key.mv_data + key.mv_size - 8, htonll(now++));
+
+		/*
+		 * Convert the input from files-reflog format to
+		 * lmdb-reflog-format
+		 */
+
+		format_lmdb_reflog_ent(&sb, &input);
+		val.mv_data = sb.buf;
+		val.mv_size = sb.len + 1;
+		mdb_put_or_die(&transaction, &key, &val, 0);
+		strbuf_reset(&sb);
+		input.len = 0;
+	}
+
+	assert(mdb_transaction_commit(&transaction, NULL) == 0);
+	free(key.mv_data);
+}
+
+struct ref_be refs_be_lmdb = {
+	NULL,
+	"lmdb",
+	lmdb_init_backend,
+	lmdb_initdb,
+	lmdb_transaction_commit,
+	lmdb_transaction_commit, /* initial commit */
+	lmdb_rename_ref,
+	lmdb_for_each_reflog_ent,
+	lmdb_for_each_reflog_ent_reverse,
+	lmdb_for_each_reflog,
+	lmdb_reflog_exists,
+	lmdb_create_reflog,
+	lmdb_delete_reflog,
+	lmdb_reflog_expire,
+	lmdb_resolve_ref_unsafe,
+	lmdb_verify_refname_available,
+	lmdb_pack_refs,
+	lmdb_peel_ref,
+	lmdb_create_symref,
+	lmdb_resolve_gitlink_ref,
+	lmdb_head_ref,
+	lmdb_head_ref_submodule,
+	lmdb_for_each_ref,
+	lmdb_for_each_ref_submodule,
+	lmdb_for_each_ref_in,
+	lmdb_for_each_fullref_in,
+	lmdb_for_each_ref_in_submodule,
+	lmdb_for_each_rawref,
+	lmdb_for_each_namespaced_ref,
+	lmdb_for_each_replace_ref,
+	lmdb_for_each_reftype_fullpath,
+};
diff --git a/refs.h b/refs.h
index 45615e8..8d85a6e 100644
--- a/refs.h
+++ b/refs.h
@@ -691,6 +691,15 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
 			 reflog_expiry_cleanup_fn cleanup_fn,
 			 void *policy_cb_data);
 
+/*
+ * Read the refdb configuration data out of the config file
+ */
+struct refdb_config_data {
+	const char *refs_backend_type;
+	const char *refs_base;
+};
+int refdb_config(const char *var, const char *value, void *ptr);
+
 /* refs backends */
 typedef void ref_backend_init_fn(void *data);
 typedef int ref_backend_initdb_fn(struct strbuf *err, int shared);
@@ -789,6 +798,7 @@ struct ref_be {
 
 
 extern struct ref_be refs_be_files;
+extern struct ref_be refs_be_lmdb;
 int set_refs_backend(const char *name, void *init_data);
 void register_refs_backend(struct ref_be *be);
 
diff --git a/setup.c b/setup.c
index 7433eed..c8ed245 100644
--- a/setup.c
+++ b/setup.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "dir.h"
+#include "refs.h"
 #include "string-list.h"
 
 static int inside_git_dir = -1;
@@ -263,16 +264,26 @@ int get_common_dir_noenv(struct strbuf *sb, const char *gitdir)
 	return ret;
 }
 
+int refdb_config(const char *var, const char *value, void *ptr)
+{
+       struct refdb_config_data *cdata = ptr;
+
+       if (!strcmp(var, "core.refsbackendtype"))
+	       cdata->refs_backend_type = strdup((char *)value);
+       return 0;
+}
+
 /*
  * Test if it looks like we're at a git directory.
  * We want to see:
  *
  *  - either an objects/ directory _or_ the proper
  *    GIT_OBJECT_DIRECTORY environment variable
- *  - a refs/ directory
- *  - either a HEAD symlink or a HEAD file that is formatted as
- *    a proper "ref:", or a regular file HEAD that has a properly
- *    formatted sha1 object name.
+ *  - a refdb/ directory or
+ *    - a refs/ directory
+ *    - either a HEAD symlink or a HEAD file that is formatted as
+ *      a proper "ref:", or a regular file HEAD that has a properly
+ *      formatted sha1 object name.
  */
 int is_git_directory(const char *suspect)
 {
@@ -303,8 +314,13 @@ int is_git_directory(const char *suspect)
 
 	strbuf_setlen(&path, len);
 	strbuf_addstr(&path, "/refs");
-	if (access(path.buf, X_OK))
-		goto done;
+
+	if (access(path.buf, X_OK)) {
+		strbuf_setlen(&path, len);
+		strbuf_addstr(&path, "/refdb");
+		if (access(path.buf, X_OK))
+			goto done;
+	}
 
 	ret = 1;
 done:
@@ -371,6 +387,10 @@ static int check_repo_format(const char *var, const char *value, void *cb)
 		 */
 		if (!strcmp(ext, "noop"))
 			;
+#ifdef USE_LIBLMDB
+		else if (!strcmp(ext, "refbackend") && !strcmp(value, "lmdb"))
+			;
+#endif
 		else
 			string_list_append(&unknown_extensions, ext);
 	}
diff --git a/test-refs-be-lmdb.c b/test-refs-be-lmdb.c
new file mode 100644
index 0000000..bb875ff
--- /dev/null
+++ b/test-refs-be-lmdb.c
@@ -0,0 +1,68 @@
+#include "cache.h"
+#include "string-list.h"
+#include "parse-options.h"
+#include "refs.h"
+
+static const char * const test_refs_be_lmdb_usage[] = {
+	"git test-refs-be-lmdb <key>",
+	"git test-refs-be-lmdb <key> <value>",
+	NULL,
+};
+
+int test_refdb_raw_read(const char *key);
+void test_refdb_raw_write(const char *key, const char *value);
+int test_refdb_raw_reflog(const char *refname);
+int test_refdb_raw_delete(const char *key);
+void test_refdb_raw_delete_reflog(const char *refname);
+void test_refdb_raw_append_reflog(const char *refname);
+
+int main(int argc, const char **argv)
+{
+	const char *delete = NULL;
+	const char *reflog = NULL;
+	const char *append_reflog = NULL;
+	int delete_missing_error = 0;
+	int clear_reflog = 0;
+	struct refdb_config_data config_data = {NULL};
+
+	struct option options[] = {
+		OPT_STRING('d', NULL, &delete, "branch", "delete refdb entry"),
+		OPT_STRING('l', NULL, &reflog, "branch", "show reflog"),
+		OPT_STRING('a', NULL, &append_reflog, "branch", "append to reflog"),
+		OPT_BOOL('c', NULL, &clear_reflog, "delete reflog. If a branch is provided, the reflog for that branch will be deleted; else all reflogs will be deleted."),
+		OPT_BOOL('x', NULL, &delete_missing_error,
+			 "deleting a missing key is an error"),
+		OPT_END(),
+	};
+
+	argc = parse_options(argc, argv, "", options, test_refs_be_lmdb_usage,
+			     0);
+
+	if (!append_reflog && !clear_reflog && !delete && !reflog && argc != 1 && argc != 2)
+		usage_with_options(test_refs_be_lmdb_usage,
+				   options);
+
+	git_config(git_default_config, NULL);
+
+	config_data.refs_backend_type = "lmdb";
+	config_data.refs_base = get_git_dir();
+
+	register_refs_backend(&refs_be_lmdb);
+	set_refs_backend("lmdb", &config_data);
+
+	if (clear_reflog) {
+		test_refdb_raw_delete_reflog(argv[0]);
+	} else if (append_reflog) {
+		test_refdb_raw_append_reflog(append_reflog);
+	} else if (reflog) {
+		return test_refdb_raw_reflog(reflog);
+	} else if (delete) {
+		if (test_refdb_raw_delete(delete) && delete_missing_error)
+			return 1;
+	} else if (argc == 1) {
+		return test_refdb_raw_read(argv[0]);
+	} else {
+		test_refdb_raw_write(argv[0], argv[1]);
+	}
+	return 0;
+}
-- 
2.4.2.644.g97b850b-twtrsrc

[PATCH v3 44/44] refs: tests for db backend

[David Turner]

Add tests for the database backend.

Signed-off-by: David Turner <dturner@twopensource.com>
Helped-by: Dennis Kaarsemaker <dennis@kaarsemaker.net>
---
 t/t1460-refs-be-db.sh        | 1109 ++++++++++++++++++++++++++++++++++++++++++
 t/t1470-refs-be-db-reflog.sh |  359 ++++++++++++++
 t/test-lib.sh                |    1 +
 3 files changed, 1469 insertions(+)
 create mode 100755 t/t1460-refs-be-db.sh
 create mode 100755 t/t1470-refs-be-db-reflog.sh

diff --git a/t/t1460-refs-be-db.sh b/t/t1460-refs-be-db.sh
new file mode 100755
index 0000000..56d2d2b
--- /dev/null
+++ b/t/t1460-refs-be-db.sh
@@ -0,0 +1,1109 @@
+#!/bin/sh
+#
+# Copyright (c) 2015 Twitter, Inc
+# Copyright (c) 2006 Shawn Pearce
+# This test is based on t1400-update-ref.sh
+#
+
+test_description='Test lmdb refs backend'
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+if ! test_have_prereq LMDB
+then
+	skip_all="Skipping lmdb refs backend tests, lmdb backend not built"
+	test_done
+fi
+
+raw_ref() {
+	test-refs-be-lmdb "$1"
+}
+
+delete_ref() {
+	test-refs-be-lmdb -d "$1"
+}
+
+write_ref() {
+	test-refs-be-lmdb "$1" "$2"
+}
+
+raw_reflog() {
+	test-refs-be-lmdb -l "$1"
+}
+
+delete_all_reflogs() {
+	test-refs-be-lmdb -c
+}
+
+append_reflog() {
+	test-refs-be-lmdb -a "$1"
+}
+
+Z=$_z40
+
+test_expect_success setup '
+	git init --refs-backend-type=lmdb &&
+	for name in A B C D E F
+	do
+		test_tick &&
+		T=$(git write-tree) &&
+		sha1=$(echo $name | git commit-tree $T) &&
+		eval $name=$sha1
+	done
+'
+
+m=refs/heads/master
+n_dir=refs/heads/gu
+n=$n_dir/fixes
+
+test_expect_success \
+	"create $m" \
+	"git update-ref $m $A &&
+	 test $A"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"create $m" \
+	"git update-ref $m $B $A &&
+	 test $B"' = $(raw_ref '"$m"')'
+test_expect_success "fail to delete $m with stale ref" '
+	test_must_fail git update-ref -d $m $A &&
+	test $B = "$(raw_ref $m)"
+'
+test_expect_success "delete $m" '
+	git update-ref -d $m $B &&
+	! raw_ref $m
+'
+delete_ref $m
+
+test_expect_success "delete $m without oldvalue verification" "
+	git update-ref $m $A &&
+	test $A = \$(raw_ref $m) &&
+	git update-ref -d $m &&
+	! raw_ref $m
+"
+delete_ref $m
+
+test_expect_success \
+	"fail to create $n" \
+	"git update-ref $n_dir $A &&
+	 test_must_fail git update-ref $n $A >out 2>err"
+
+delete_ref $n_dir
+rm -f out err
+
+test_expect_success \
+	"create $m (by HEAD)" \
+	"git update-ref HEAD $A &&
+	 test $A"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"create $m (by HEAD)" \
+	"git update-ref HEAD $B $A &&
+	 test $B"' = $(raw_ref '"$m"')'
+test_expect_success "fail to delete $m (by HEAD) with stale ref" '
+	test_must_fail git update-ref -d HEAD $A &&
+	test $B = $(raw_ref '"$m"')
+'
+test_expect_success "delete $m (by HEAD)" '
+	git update-ref -d HEAD $B &&
+	! raw_ref $m
+'
+delete_ref $m
+
+test_expect_success \
+	"create $m (by HEAD)" \
+	"git update-ref HEAD $A &&
+	 test $A"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"pack refs" \
+	"git pack-refs --all"
+test_expect_success \
+	"move $m (by HEAD)" \
+	"git update-ref HEAD $B $A &&
+	 test $B"' = $(raw_ref '"$m"')'
+test_expect_success "delete $m (by HEAD) should remove both packed and loose $m" '
+	git update-ref -d HEAD $B &&
+	! raw_ref $m
+'
+delete_ref $m
+
+OLD_HEAD=$(raw_ref HEAD)
+test_expect_success "delete symref without dereference" '
+	git update-ref --no-deref -d HEAD &&
+	! raw_ref HEAD
+'
+write_ref HEAD "$OLD_HEAD"
+
+test_expect_success "delete symref without dereference when the referred ref is packed" '
+	echo foo >foo.c &&
+	git add foo.c &&
+	git commit -m foo &&
+	git pack-refs --all &&
+	git update-ref --no-deref -d HEAD &&
+	! raw_ref HEAD
+'
+write_ref HEAD "$OLD_HEAD"
+delete_ref $m
+
+test_expect_success 'update-ref -d is not confused by self-reference' '
+	git symbolic-ref refs/heads/self refs/heads/self &&
+	test_when_finished "delete_ref refs/heads/self" &&
+	test_must_fail git update-ref -d refs/heads/self
+'
+
+test_expect_success 'update-ref --no-deref -d can delete self-reference' '
+	git symbolic-ref refs/heads/self refs/heads/self &&
+	test_when_finished "delete_ref refs/heads/self" &&
+	git update-ref --no-deref -d refs/heads/self
+'
+
+test_expect_success 'update-ref --no-deref -d can delete reference to bad ref' '
+	test-refs-be-lmdb refs/heads/bad "" &&
+	test_when_finished "delete_ref refs/heads/bad" &&
+	git symbolic-ref refs/heads/ref-to-bad refs/heads/bad &&
+	test_when_finished "delete_ref refs/heads/ref-to-bad" &&
+	raw_ref refs/heads/ref-to-bad &&
+	git update-ref --no-deref -d refs/heads/ref-to-bad &&
+	! raw_ref refs/heads/ref-to-bad
+'
+
+test_expect_success '(not) create HEAD with old sha1' "
+	test_must_fail git update-ref HEAD $A $B
+"
+test_expect_success "(not) prior created .git/$m" "
+	! raw_ref $m
+"
+delete_ref $m
+
+test_expect_success \
+	"create HEAD" \
+	"git update-ref HEAD $A"
+test_expect_success '(not) change HEAD with wrong SHA1' "
+	test_must_fail git update-ref HEAD $B $Z
+"
+test_expect_success "(not) changed .git/$m" "
+	! test $B"' = $(raw_ref '"$m"')
+'
+
+: a repository with working tree always has reflog these days...
+delete_all_reflogs
+: | append_reflog $m
+delete_ref $m
+
+test_expect_success \
+	"create $m (logged by touch)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:30" \
+	 git update-ref HEAD '"$A"' -m "Initial Creation" &&
+	 test '"$A"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"update $m (logged by touch)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:31" \
+	 git update-ref HEAD'" $B $A "'-m "Switch" &&
+	 test '"$B"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"set $m (logged by touch)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:41" \
+	 git update-ref HEAD'" $A &&
+	 test $A"' = $(raw_ref '"$m"')'
+
+cat >expect <<EOF
+$Z $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150200 +0000	Initial Creation
+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150260 +0000	Switch
+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000
+EOF
+test_expect_success \
+	"verifying $m's log" \
+	"raw_reflog $m >actual &&
+	 test_cmp expect actual"
+delete_ref $m
+delete_all_reflogs
+: | append_reflog $m
+rm -f actual expect
+
+test_expect_success \
+	'enable core.logAllRefUpdates' \
+	'git config core.logAllRefUpdates true &&
+	 test true = $(git config --bool --get core.logAllRefUpdates)'
+
+test_expect_success \
+	"create $m (logged by config)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:32" \
+	 git update-ref HEAD'" $A "'-m "Initial Creation" &&
+	 test '"$A"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"update $m (logged by config)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:33" \
+	 git update-ref HEAD'" $B $A "'-m "Switch" &&
+	 test '"$B"' = $(raw_ref '"$m"')'
+test_expect_success \
+	"set $m (logged by config)" \
+	'GIT_COMMITTER_DATE="2005-05-26 23:43" \
+	 git update-ref HEAD '"$A &&
+	 test $A"' = $(raw_ref '"$m"')'
+
+cat >expect <<EOF
+$Z $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150320 +0000	Initial Creation
+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 +0000	Switch
+$B $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 +0000
+EOF
+test_expect_success \
+	"verifying $m's log" \
+	'raw_reflog $m >actual &&
+	test_cmp expect actual'
+delete_ref $m
+rm -f expect
+
+git update-ref $m $D
+git reflog expire --expire=all $m
+
+append_reflog $m <<EOF
+0000000000000000000000000000000000000000 $C $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150320 -0500
+$C $A $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150350 -0500
+$A $B $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150380 -0500
+$F $Z $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150680 -0500
+$Z $E $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150980 -0500
+EOF
+
+ed="Thu, 26 May 2005 18:32:00 -0500"
+gd="Thu, 26 May 2005 18:33:00 -0500"
+ld="Thu, 26 May 2005 18:43:00 -0500"
+test_expect_success \
+	'Query "master@{May 25 2005}" (before history)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{May 25 2005}" >o 2>e &&
+	 test '"$C"' = $(cat o) &&
+	 test "warning: Log for '\'master\'' only goes back to $ed." = "$(cat e)"'
+test_expect_success \
+	"Query master@{2005-05-25} (before history)" \
+	'rm -f o e &&
+	 git rev-parse --verify master@{2005-05-25} >o 2>e &&
+	 test '"$C"' = $(cat o) &&
+	 echo test "warning: Log for '\'master\'' only goes back to $ed." = "$(cat e)"'
+test_expect_success \
+	'Query "master@{May 26 2005 23:31:59}" (1 second before history)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{May 26 2005 23:31:59}" >o 2>e &&
+	 test '"$C"' = $(cat o) &&
+	 test "warning: Log for '\''master'\'' only goes back to $ed." = "$(cat e)"'
+test_expect_success \
+	'Query "master@{May 26 2005 23:32:00}" (exactly history start)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{May 26 2005 23:32:00}" >o 2>e &&
+	 test '"$C"' = $(cat o) &&
+	 test "" = "$(cat e)"'
+test_expect_success \
+	'Query "master@{May 26 2005 23:32:30}" (first non-creation change)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{May 26 2005 23:32:30}" >o 2>e &&
+	 test '"$A"' = $(cat o) &&
+	 test "" = "$(cat e)"'
+test_expect_success \
+	'Query "master@{2005-05-26 23:33:01}" (middle of history with gap)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{2005-05-26 23:33:01}" >o 2>e &&
+	 test '"$B"' = $(cat o) &&
+	 test "warning: Log for ref '"$m has gap after $gd"'." = "$(cat e)"'
+test_expect_success \
+	'Query "master@{2005-05-26 23:38:00}" (middle of history)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{2005-05-26 23:38:00}" >o 2>e &&
+	 test '"$Z"' = $(cat o) &&
+	 test "" = "$(cat e)"'
+test_expect_success \
+	'Query "master@{2005-05-26 23:43:00}" (exact end of history)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{2005-05-26 23:43:00}" >o 2>e &&
+	 test '"$E"' = $(cat o) &&
+	 test "" = "$(cat e)"'
+test_expect_success \
+	'Query "master@{2005-05-28}" (past end of history)' \
+	'rm -f o e &&
+	 git rev-parse --verify "master@{2005-05-28}" >o 2>e &&
+	 test '"$D"' = $(cat o) &&
+	 test "warning: Log for ref '"$m unexpectedly ended on $ld"'." = "$(cat e)"'
+
+
+git reflog expire --expire=all $m
+delete_ref $m
+
+test_expect_success \
+    'creating initial files' \
+    'echo TEST >F &&
+     git add F &&
+	 GIT_AUTHOR_DATE="2005-05-26 23:30" \
+	 GIT_COMMITTER_DATE="2005-05-26 23:30" git commit -m add -a &&
+	 h_TEST=$(git rev-parse --verify HEAD) &&
+	 echo The other day this did not work. >M &&
+	 echo And then Bob told me how to fix it. >>M &&
+	 echo OTHER >F &&
+	 GIT_AUTHOR_DATE="2005-05-26 23:41" \
+	 GIT_COMMITTER_DATE="2005-05-26 23:41" git commit -F M -a &&
+	 h_OTHER=$(git rev-parse --verify HEAD) &&
+	 GIT_AUTHOR_DATE="2005-05-26 23:44" \
+	 GIT_COMMITTER_DATE="2005-05-26 23:44" git commit --amend &&
+	 h_FIXED=$(git rev-parse --verify HEAD) &&
+	 echo Merged initial commit and a later commit. >M &&
+	 echo $h_TEST >.git/MERGE_HEAD &&
+	 GIT_AUTHOR_DATE="2005-05-26 23:45" \
+	 GIT_COMMITTER_DATE="2005-05-26 23:45" git commit -F M &&
+	 h_MERGED=$(git rev-parse --verify HEAD) &&
+	 rm -f M'
+
+cat >expect <<EOF
+$Z $h_TEST $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150200 +0000	commit (initial): add
+$h_TEST $h_OTHER $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117150860 +0000	commit: The other day this did not work.
+$h_OTHER $h_FIXED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151040 +0000	commit (amend): The other day this did not work.
+$h_FIXED $h_MERGED $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL> 1117151100 +0000	commit (merge): Merged initial commit and a later commit.
+EOF
+test_expect_success \
+	'git commit logged updates' \
+	"raw_reflog $m >actual &&
+	test_cmp expect actual"
+unset h_TEST h_OTHER h_FIXED h_MERGED
+
+test_expect_success \
+	'git cat-file blob master:F (expect OTHER)' \
+	'test OTHER = $(git cat-file blob master:F)'
+test_expect_success \
+	'git cat-file blob master@{2005-05-26 23:30}:F (expect TEST)' \
+	'test TEST = $(git cat-file blob "master@{2005-05-26 23:30}:F")'
+test_expect_success \
+	'git cat-file blob master@{2005-05-26 23:42}:F (expect OTHER)' \
+	'test OTHER = $(git cat-file blob "master@{2005-05-26 23:42}:F")'
+
+a=refs/heads/a
+b=refs/heads/b
+c=refs/heads/c
+E='""'
+F='%s\0'
+pws='path with space'
+
+test_expect_success 'stdin test setup' '
+	echo "$pws" >"$pws" &&
+	git add -- "$pws" &&
+	git commit -m "$pws"
+'
+
+test_expect_success '-z fails without --stdin' '
+	test_must_fail git update-ref -z $m $m $m 2>err &&
+	grep "usage: git update-ref" err
+'
+
+test_expect_success 'stdin works with no input' '
+	>stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse --verify -q $m
+'
+
+test_expect_success 'stdin fails on empty line' '
+	echo "" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: empty command in input" err
+'
+
+test_expect_success 'stdin fails on only whitespace' '
+	echo " " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: whitespace before command:  " err
+'
+
+test_expect_success 'stdin fails on leading whitespace' '
+	echo " create $a $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: whitespace before command:  create $a $m" err
+'
+
+test_expect_success 'stdin fails on unknown command' '
+	echo "unknown $a" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: unknown command: unknown $a" err
+'
+
+test_expect_success 'stdin fails on unbalanced quotes' '
+	echo "create $a \"master" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: badly quoted argument: \\\"master" err
+'
+
+test_expect_success 'stdin fails on invalid escape' '
+	echo "create $a \"ma\zter\"" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: badly quoted argument: \\\"ma\\\\zter\\\"" err
+'
+
+test_expect_success 'stdin fails on junk after quoted argument' '
+	echo "create \"$a\"master" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: unexpected character after quoted argument: \\\"$a\\\"master" err
+'
+
+test_expect_success 'stdin fails create with no ref' '
+	echo "create " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: create: missing <ref>" err
+'
+
+test_expect_success 'stdin fails create with no new value' '
+	echo "create $a" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: create $a: missing <newvalue>" err
+'
+
+test_expect_success 'stdin fails create with too many arguments' '
+	echo "create $a $m $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: create $a: extra input:  $m" err
+'
+
+test_expect_success 'stdin fails update with no ref' '
+	echo "update " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: update: missing <ref>" err
+'
+
+test_expect_success 'stdin fails update with no new value' '
+	echo "update $a" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: update $a: missing <newvalue>" err
+'
+
+test_expect_success 'stdin fails update with too many arguments' '
+	echo "update $a $m $m $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: update $a: extra input:  $m" err
+'
+
+test_expect_success 'stdin fails delete with no ref' '
+	echo "delete " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: delete: missing <ref>" err
+'
+
+test_expect_success 'stdin fails delete with too many arguments' '
+	echo "delete $a $m $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: delete $a: extra input:  $m" err
+'
+
+test_expect_success 'stdin fails verify with too many arguments' '
+	echo "verify $a $m $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: verify $a: extra input:  $m" err
+'
+
+test_expect_success 'stdin fails option with unknown name' '
+	echo "option unknown" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: option unknown: unknown" err
+'
+
+test_expect_success 'stdin fails with duplicate refs' '
+	cat >stdin <<-EOF &&
+	create $a $m
+	create $b $m
+	create $a $m
+	EOF
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: Multiple updates for ref '"'"'$a'"'"' not allowed." err
+'
+
+test_expect_success 'stdin create ref works' '
+	echo "create $a $m" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin succeeds with quoted argument' '
+	git update-ref -d $a &&
+	echo "create $a \"$m\"" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin succeeds with escaped character' '
+	git update-ref -d $a &&
+	echo "create $a \"ma\\163ter\"" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin update ref creates with zero old value' '
+	echo "update $b $m $Z" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	git update-ref -d $b
+'
+
+test_expect_success 'stdin update ref creates with empty old value' '
+	echo "update $b $m $E" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin create ref works with path with space to blob' '
+	echo "create refs/blobs/pws \"$m:$pws\"" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse "$m:$pws" >expect &&
+	git rev-parse refs/blobs/pws >actual &&
+	test_cmp expect actual &&
+	git update-ref -d refs/blobs/pws
+'
+
+test_expect_success 'stdin update ref fails with wrong old value' '
+	echo "update $c $m $m~1" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$c'"'"'" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin update ref fails with bad old value' '
+	echo "update $c $m does-not-exist" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: update $c: invalid <oldvalue>: does-not-exist" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin create ref fails with bad new value' '
+	echo "create $c does-not-exist" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: create $c: invalid <newvalue>: does-not-exist" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin create ref fails with zero new value' '
+	echo "create $c " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: create $c: zero <newvalue>" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin update ref works with right old value' '
+	echo "update $b $m~1 $m" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin delete ref fails with wrong old value' '
+	echo "delete $a $m~1" >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$a'"'"'" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin delete ref fails with zero old value' '
+	echo "delete $a " >stdin &&
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: delete $a: zero <oldvalue>" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin update symref works option no-deref' '
+	git symbolic-ref refs/TESTSYMREF $b &&
+	cat >stdin <<-EOF &&
+	option no-deref
+	update refs/TESTSYMREF $a $b
+	EOF
+	git update-ref --stdin <stdin &&
+	git rev-parse refs/TESTSYMREF >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin delete symref works option no-deref' '
+	git symbolic-ref refs/TESTSYMREF $b &&
+	cat >stdin <<-EOF &&
+	option no-deref
+	delete refs/TESTSYMREF $b
+	EOF
+	git update-ref --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/TESTSYMREF &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin delete ref works with right old value' '
+	echo "delete $b $m~1" >stdin &&
+	git update-ref --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q $b
+'
+
+test_expect_success 'stdin update/create/verify combination works' '
+	cat >stdin <<-EOF &&
+	update $a $m
+	create $b $m
+	verify $c
+	EOF
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin verify succeeds for correct value' '
+	git rev-parse $m >expect &&
+	echo "verify $m $m" >stdin &&
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin verify succeeds for missing reference' '
+	echo "verify refs/heads/missing $Z" >stdin &&
+	git update-ref --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/heads/missing
+'
+
+test_expect_success 'stdin verify treats no value as missing' '
+	echo "verify refs/heads/missing" >stdin &&
+	git update-ref --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/heads/missing
+'
+
+test_expect_success 'stdin verify fails for wrong value' '
+	git rev-parse $m >expect &&
+	echo "verify $m $m~1" >stdin &&
+	test_must_fail git update-ref --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin verify fails for mistaken null value' '
+	git rev-parse $m >expect &&
+	echo "verify $m $Z" >stdin &&
+	test_must_fail git update-ref --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin verify fails for mistaken empty value' '
+	M=$(git rev-parse $m) &&
+	test_when_finished "git update-ref $m $M" &&
+	git rev-parse $m >expect &&
+	echo "verify $m" >stdin &&
+	test_must_fail git update-ref --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin update refs works with identity updates' '
+	cat >stdin <<-EOF &&
+	update $a $m $m
+	update $b $m $m
+	update $c $Z $E
+	EOF
+	git update-ref --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin update refs fails with wrong old value' '
+	git update-ref $c $m &&
+	cat >stdin <<-EOF &&
+	update $a $m $m
+	update $b $m $m
+	update $c  ''
+	EOF
+	test_must_fail git update-ref --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$c'"'"'" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	git rev-parse $c >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin delete refs works with packed and loose refs' '
+	git pack-refs --all &&
+	git update-ref $c $m~1 &&
+	cat >stdin <<-EOF &&
+	delete $a $m
+	update $b $Z $m
+	update $c $E $m~1
+	EOF
+	git update-ref --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q $a &&
+	test_must_fail git rev-parse --verify -q $b &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z works on empty input' '
+	>stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse --verify -q $m
+'
+
+test_expect_success 'stdin -z fails on empty line' '
+	echo "" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: whitespace before command: " err
+'
+
+test_expect_success 'stdin -z fails on empty command' '
+	printf $F "" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: empty command in input" err
+'
+
+test_expect_success 'stdin -z fails on only whitespace' '
+	printf $F " " >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: whitespace before command:  " err
+'
+
+test_expect_success 'stdin -z fails on leading whitespace' '
+	printf $F " create $a" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: whitespace before command:  create $a" err
+'
+
+test_expect_success 'stdin -z fails on unknown command' '
+	printf $F "unknown $a" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: unknown command: unknown $a" err
+'
+
+test_expect_success 'stdin -z fails create with no ref' '
+	printf $F "create " >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: create: missing <ref>" err
+'
+
+test_expect_success 'stdin -z fails create with no new value' '
+	printf $F "create $a" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: create $a: unexpected end of input when reading <newvalue>" err
+'
+
+test_expect_success 'stdin -z fails create with too many arguments' '
+	printf $F "create $a" "$m" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: unknown command: $m" err
+'
+
+test_expect_success 'stdin -z fails update with no ref' '
+	printf $F "update " >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: update: missing <ref>" err
+'
+
+test_expect_success 'stdin -z fails update with too few args' '
+	printf $F "update $a" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: update $a: unexpected end of input when reading <oldvalue>" err
+'
+
+test_expect_success 'stdin -z emits warning with empty new value' '
+	git update-ref $a $m &&
+	printf $F "update $a" "" "" >stdin &&
+	git update-ref -z --stdin <stdin 2>err &&
+	grep "warning: update $a: missing <newvalue>, treating as zero" err &&
+	test_must_fail git rev-parse --verify -q $a
+'
+
+test_expect_success 'stdin -z fails update with no new value' '
+	printf $F "update $a" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: update $a: unexpected end of input when reading <newvalue>" err
+'
+
+test_expect_success 'stdin -z fails update with no old value' '
+	printf $F "update $a" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: update $a: unexpected end of input when reading <oldvalue>" err
+'
+
+test_expect_success 'stdin -z fails update with too many arguments' '
+	printf $F "update $m" "$m" "$m" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: unknown command: $m" err
+'
+
+test_expect_success 'stdin -z fails delete with no ref' '
+	printf $F "delete " >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: delete: missing <ref>" err
+'
+
+test_expect_success 'stdin -z fails delete with no old value' '
+	printf $F "delete $a" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: delete $a: unexpected end of input when reading <oldvalue>" err
+'
+
+test_expect_success 'stdin -z fails delete with too many arguments' '
+	printf $F "delete $m" "$m" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: unknown command: $m" err
+'
+
+test_expect_success 'stdin -z fails verify with too many arguments' '
+	printf $F "verify $m" "$m" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: unknown command: $m" err
+'
+
+test_expect_success 'stdin -z fails verify with no old value' '
+	printf $F "verify $a" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: verify $a: unexpected end of input when reading <oldvalue>" err
+'
+
+test_expect_success 'stdin -z fails option with unknown name' '
+	printf $F "option unknown" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: option unknown: unknown" err
+'
+
+test_expect_success 'stdin -z fails with duplicate refs' '
+	printf $F "create $a" "$m" "create $b" "$m" "create $a" "$m" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: Multiple updates for ref '"'"'$a'"'"' not allowed." err
+'
+
+test_expect_success 'stdin -z create ref works' '
+	printf $F "create $a" "$m" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z update ref creates with zero old value' '
+	printf $F "update $b" "$m" "$Z" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	git update-ref -d $b
+'
+
+test_expect_success 'stdin -z update ref creates with empty old value' '
+	printf $F "update $b" "$m" "" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z create ref works with path with space to blob' '
+	printf $F "create refs/blobs/pws" "$m:$pws" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse "$m:$pws" >expect &&
+	git rev-parse refs/blobs/pws >actual &&
+	test_cmp expect actual &&
+	git update-ref -d refs/blobs/pws
+'
+
+test_expect_success 'stdin -z update ref fails with wrong old value' '
+	printf $F "update $c" "$m" "$m~1" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$c'"'"'" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z update ref fails with bad old value' '
+	printf $F "update $c" "$m" "does-not-exist" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: update $c: invalid <oldvalue>: does-not-exist" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z create ref fails when ref exists' '
+	git update-ref $c $m &&
+	git rev-parse "$c" >expect &&
+	printf $F "create $c" "$m~1" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$c'"'"'" err &&
+	git rev-parse "$c" >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z create ref fails with bad new value' '
+	git update-ref -d "$c" &&
+	printf $F "create $c" "does-not-exist" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: create $c: invalid <newvalue>: does-not-exist" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z create ref fails with empty new value' '
+	printf $F "create $c" "" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: create $c: missing <newvalue>" err &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z update ref works with right old value' '
+	printf $F "update $b" "$m~1" "$m" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z delete ref fails with wrong old value' '
+	printf $F "delete $a" "$m~1" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$a'"'"'" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z delete ref fails with zero old value' '
+	printf $F "delete $a" "$Z" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: delete $a: zero <oldvalue>" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z update symref works option no-deref' '
+	git symbolic-ref refs/TESTSYMREF $b &&
+	printf $F "option no-deref" "update refs/TESTSYMREF" "$a" "$b" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse refs/TESTSYMREF >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z delete symref works option no-deref' '
+	git symbolic-ref refs/TESTSYMREF $b &&
+	printf $F "option no-deref" "delete refs/TESTSYMREF" "$b" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/TESTSYMREF &&
+	git rev-parse $m~1 >expect &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z delete ref works with right old value' '
+	printf $F "delete $b" "$m~1" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q $b
+'
+
+test_expect_success 'stdin -z update/create/verify combination works' '
+	printf $F "update $a" "$m" "" "create $b" "$m" "verify $c" "" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z verify succeeds for correct value' '
+	git rev-parse $m >expect &&
+	printf $F "verify $m" "$m" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z verify succeeds for missing reference' '
+	printf $F "verify refs/heads/missing" "$Z" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/heads/missing
+'
+
+test_expect_success 'stdin -z verify treats no value as missing' '
+	printf $F "verify refs/heads/missing" "" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q refs/heads/missing
+'
+
+test_expect_success 'stdin -z verify fails for wrong value' '
+	git rev-parse $m >expect &&
+	printf $F "verify $m" "$m~1" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z verify fails for mistaken null value' '
+	git rev-parse $m >expect &&
+	printf $F "verify $m" "$Z" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z verify fails for mistaken empty value' '
+	M=$(git rev-parse $m) &&
+	test_when_finished "git update-ref $m $M" &&
+	git rev-parse $m >expect &&
+	printf $F "verify $m" "" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin &&
+	git rev-parse $m >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z update refs works with identity updates' '
+	printf $F "update $a" "$m" "$m" "update $b" "$m" "$m" "update $c" "$Z" "" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_expect_success 'stdin -z update refs fails with wrong old value' '
+	git update-ref $c $m &&
+	printf $F "update $a" "$m" "$m" "update $b" "$m" "$m" "update $c" "$m" "$Z" >stdin &&
+	test_must_fail git update-ref -z --stdin <stdin 2>err &&
+	grep "fatal: cannot lock the ref '"'"'$c'"'"'" err &&
+	git rev-parse $m >expect &&
+	git rev-parse $a >actual &&
+	test_cmp expect actual &&
+	git rev-parse $b >actual &&
+	test_cmp expect actual &&
+	git rev-parse $c >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stdin -z delete refs works with packed and loose refs' '
+	git pack-refs --all &&
+	git update-ref $c $m~1 &&
+	printf $F "delete $a" "$m" "update $b" "$Z" "$m" "update $c" "" "$m~1" >stdin &&
+	git update-ref -z --stdin <stdin &&
+	test_must_fail git rev-parse --verify -q $a &&
+	test_must_fail git rev-parse --verify -q $b &&
+	test_must_fail git rev-parse --verify -q $c
+'
+
+test_done
diff --git a/t/t1470-refs-be-db-reflog.sh b/t/t1470-refs-be-db-reflog.sh
new file mode 100755
index 0000000..2c6b867
--- /dev/null
+++ b/t/t1470-refs-be-db-reflog.sh
@@ -0,0 +1,359 @@
+#!/bin/sh
+#
+# Copyright (c) 2015 Twitter, Inc
+# Copyright (c) 2007 Junio C Hamano
+#
+
+test_description='Test prune and reflog expiration'
+TEST_NO_CREATE_REPO=1
+. ./test-lib.sh
+
+if ! test_have_prereq LMDB
+then
+	skip_all="Skipping lmdb refs backend tests, lmdb backend not built"
+	test_done
+fi
+
+raw_reflog() {
+	cat .git/logs/$1 2>/dev/null || test-refs-be-lmdb -l "$1"
+}
+
+append_reflog() {
+	test-refs-be-lmdb -a "$1"
+}
+
+check_have () {
+	gaah= &&
+	for N in "$@"
+	do
+		eval "o=\$$N" && git cat-file -t $o || {
+			echo Gaah $N
+			gaah=$N
+			break
+		}
+	done &&
+	test -z "$gaah"
+}
+
+check_fsck () {
+	output=$(git fsck --full)
+	case "$1" in
+	'')
+		test -z "$output" ;;
+	*)
+		echo "$output" | grep "$1" ;;
+	esac
+}
+
+corrupt () {
+	aa=${1%??????????????????????????????????????} zz=${1#??}
+	mv .git/objects/$aa/$zz .git/$aa$zz
+}
+
+recover () {
+	aa=${1%??????????????????????????????????????} zz=${1#??}
+	mkdir -p .git/objects/$aa
+	mv .git/$aa$zz .git/objects/$aa/$zz
+}
+
+check_dont_have () {
+	gaah= &&
+	for N in "$@"
+	do
+		eval "o=\$$N"
+		git cat-file -t $o && {
+			echo Gaah $N
+			gaah=$N
+			break
+		}
+	done
+	test -z "$gaah"
+}
+
+test_expect_success setup '
+	git init --refs-backend-type=lmdb &&
+	mkdir -p A/B &&
+	echo rat >C &&
+	echo ox >A/D &&
+	echo tiger >A/B/E &&
+	git add . &&
+
+	test_tick && git commit -m rabbit &&
+	H=`git rev-parse --verify HEAD` &&
+	A=`git rev-parse --verify HEAD:A` &&
+	B=`git rev-parse --verify HEAD:A/B` &&
+	C=`git rev-parse --verify HEAD:C` &&
+	D=`git rev-parse --verify HEAD:A/D` &&
+	E=`git rev-parse --verify HEAD:A/B/E` &&
+	check_fsck &&
+
+	test_chmod +x C &&
+	git add C &&
+	test_tick && git commit -m dragon &&
+	L=`git rev-parse --verify HEAD` &&
+	check_fsck &&
+
+	rm -f C A/B/E &&
+	echo snake >F &&
+	echo horse >A/G &&
+	git add F A/G &&
+	test_tick && git commit -a -m sheep &&
+	F=`git rev-parse --verify HEAD:F` &&
+	G=`git rev-parse --verify HEAD:A/G` &&
+	I=`git rev-parse --verify HEAD:A` &&
+	J=`git rev-parse --verify HEAD` &&
+	check_fsck &&
+
+	rm -f A/G &&
+	test_tick && git commit -a -m monkey &&
+	K=`git rev-parse --verify HEAD` &&
+	check_fsck &&
+
+	check_have A B C D E F G H I J K L &&
+
+	git prune &&
+
+	check_have A B C D E F G H I J K L &&
+
+	check_fsck &&
+
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 4 reflog
+'
+
+test_expect_success rewind '
+	test_tick && git reset --hard HEAD~2 &&
+	test -f C &&
+	test -f A/B/E &&
+	! test -f F &&
+	! test -f A/G &&
+
+	check_have A B C D E F G H I J K L &&
+
+	git prune &&
+
+	check_have A B C D E F G H I J K L &&
+
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 5 reflog
+'
+
+test_expect_success 'corrupt and check' '
+
+	corrupt $F &&
+	check_fsck "missing blob $F"
+
+'
+
+test_expect_success 'reflog expire --dry-run should not touch reflog' '
+
+	git reflog expire --dry-run \
+		--expire=$(($test_tick - 10000)) \
+		--expire-unreachable=$(($test_tick - 10000)) \
+		--stale-fix \
+		--all &&
+
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 5 reflog &&
+
+	check_fsck "missing blob $F"
+'
+
+test_expect_success 'reflog expire' '
+
+	git reflog expire --verbose \
+		--expire=$(($test_tick - 10000)) \
+		--expire-unreachable=$(($test_tick - 10000)) \
+		--stale-fix \
+		--all &&
+
+	echo git reflog expire --verbose \
+		--expire=$(($test_tick - 10000)) \
+		--expire-unreachable=$(($test_tick - 10000)) \
+		--stale-fix \
+		--all &&
+
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 2 reflog &&
+
+	check_fsck "dangling commit $K"
+'
+
+test_expect_success 'prune and fsck' '
+
+	git prune &&
+	check_fsck &&
+
+	check_have A B C D E H L &&
+	check_dont_have F G I J K
+
+'
+
+test_expect_success 'recover and check' '
+
+	recover $F &&
+	check_fsck "dangling blob $F"
+
+'
+
+test_expect_success 'delete' '
+	echo 1 > C &&
+	test_tick &&
+	git commit -m rat C &&
+
+	echo 2 > C &&
+	test_tick &&
+	git commit -m ox C &&
+
+	echo 3 > C &&
+	test_tick &&
+	git commit -m tiger C &&
+
+	HEAD_entry_count=$(git reflog | wc -l) &&
+	master_entry_count=$(git reflog show master | wc -l) &&
+
+	test $HEAD_entry_count = 5 &&
+	test $master_entry_count = 5 &&
+
+
+	git reflog delete master@{1} &&
+	git reflog show master > output &&
+	test $(($master_entry_count - 1)) = $(wc -l < output) &&
+	test $HEAD_entry_count = $(git reflog | wc -l) &&
+	! grep ox < output &&
+
+	master_entry_count=$(wc -l < output) &&
+
+	git reflog delete HEAD@{1} &&
+	test $(($HEAD_entry_count -1)) = $(git reflog | wc -l) &&
+	test $master_entry_count = $(git reflog show master | wc -l) &&
+
+	HEAD_entry_count=$(git reflog | wc -l) &&
+
+	git reflog delete master@{07.04.2005.15:15:00.-0700} &&
+	git reflog show master > output &&
+	test $(($master_entry_count - 1)) = $(wc -l < output) &&
+	! grep dragon < output
+
+'
+
+test_expect_success 'rewind2' '
+
+	test_tick && git reset --hard HEAD~2 &&
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 4 reflog
+'
+
+test_expect_success '--expire=never' '
+
+	git reflog expire --verbose \
+		--expire=never \
+		--expire-unreachable=never \
+		--all &&
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 4 reflog
+'
+
+test_expect_success 'gc.reflogexpire=never' '
+
+	git config gc.reflogexpire never &&
+	git config gc.reflogexpireunreachable never &&
+	git reflog expire --verbose --all &&
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 4 reflog
+'
+
+test_expect_success 'gc.reflogexpire=false' '
+
+	git config gc.reflogexpire false &&
+	git config gc.reflogexpireunreachable false &&
+	git reflog expire --verbose --all &&
+	raw_reflog refs/heads/master >reflog &&
+	test_when_finished rm -f reflog &&
+	test_line_count = 4 reflog &&
+
+	git config --unset gc.reflogexpire &&
+	git config --unset gc.reflogexpireunreachable
+
+'
+
+test_expect_success 'checkout should not delete log for packed ref' '
+	test $(git reflog master | wc -l) = 4 &&
+	git branch foo &&
+	git pack-refs --all &&
+	git checkout foo &&
+	test $(git reflog master | wc -l) = 4
+'
+
+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs on)' '
+	test_when_finished "git branch -d one || git branch -d one/two" &&
+
+	git branch one/two master &&
+	echo "one/two@{0} branch: Created from master" >expect &&
+	git log -g --format="%gd %gs" one/two >actual &&
+	test_cmp expect actual &&
+	git branch -d one/two &&
+
+	# now logs/refs/heads/one is a stale directory, but
+	# we should move it out of the way to create "one" reflog
+	git branch one master &&
+	echo "one@{0} branch: Created from master" >expect &&
+	git log -g --format="%gd %gs" one >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'stale dirs do not cause d/f conflicts (reflogs off)' '
+	test_when_finished "git branch -d one || git branch -d one/two" &&
+
+	git branch one/two master &&
+	echo "one/two@{0} branch: Created from master" >expect &&
+	git log -g --format="%gd %gs" one/two >actual &&
+	test_cmp expect actual &&
+	git branch -d one/two &&
+
+	# same as before, but we only create a reflog for "one" if
+	# it already exists, which it does not
+	git -c core.logallrefupdates=false branch one master &&
+	: >expect &&
+	git log -g --format="%gd %gs" one >actual &&
+	test_cmp expect actual
+'
+
+# Triggering the bug detected by this test requires a newline to fall
+# exactly BUFSIZ-1 bytes from the end of the file. We don't know
+# what that value is, since it's platform dependent. However, if
+# we choose some value N, we also catch any D which divides N evenly
+# (since we will read backwards in chunks of D). So we choose 8K,
+# which catches glibc (with an 8K BUFSIZ) and *BSD (1K).
+#
+# Each line is 114 characters, so we need 75 to still have a few before the
+# last 8K. The 89-character padding on the final entry lines up our
+# newline exactly.
+test_expect_success 'parsing reverse reflogs at BUFSIZ boundaries' '
+	git checkout -b reflogskip &&
+	z38=00000000000000000000000000000000000000 &&
+	ident="abc <xyz> 0000000001 +0000" &&
+	for i in $(test_seq 1 75); do
+		printf "$z38%02d $z38%02d %s\t" $i $(($i+1)) "$ident" &&
+		if test $i = 75; then
+			for j in $(test_seq 1 89); do
+				printf X
+			done
+		else
+			printf X
+		fi &&
+		printf "\n"
+	done | append_reflog refs/heads/reflogskip &&
+	git rev-parse reflogskip@{73} >actual &&
+	echo ${z38}03 >expect &&
+	test_cmp expect actual
+'
+
+test_done
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 16c4d7b..c6ba541 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -958,6 +958,7 @@ test -z "$NO_PERL" && test_set_prereq PERL
 test -z "$NO_PYTHON" && test_set_prereq PYTHON
 test -n "$USE_LIBPCRE" && test_set_prereq LIBPCRE
 test -z "$NO_GETTEXT" && test_set_prereq GETTEXT
+test -n "$USE_LIBLMDB" && test_set_prereq LMDB
 
 # Can we rely on git's output in the C locale?
 if test -n "$GETTEXT_POISON"
-- 
2.4.2.644.g97b850b-twtrsrc

Re: [PATCH v3 19/44] refs-be-files.c: add a backend method structure with transaction functions

[David Turner]

Please disregard this one; I rewrorded the commit message and forgot to
delete this patch.  

On Mon, 2015-10-12 at 17:51 -0400, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Add a ref structure for backend methods. Start by adding a method pointer
> for the transaction commit function.
> 
> Add a function set_refs_backend to switch between backends. The files
> based backend is the default.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 10 ++++++++--
>  refs.c          | 30 ++++++++++++++++++++++++++++++
>  refs.h          | 15 +++++++++++++++
>  3 files changed, 53 insertions(+), 2 deletions(-)
> 
> diff --git a/refs-be-files.c b/refs-be-files.c
> index 1308955..3050f1d 100644
> --- a/refs-be-files.c
> +++ b/refs-be-files.c
> @@ -3225,8 +3225,8 @@ static int ref_update_reject_duplicates(struct string_list *refnames,
>  	return 0;
>  }
>  
> -int ref_transaction_commit(struct ref_transaction *transaction,
> -			   struct strbuf *err)
> +static int files_transaction_commit(struct ref_transaction *transaction,
> +				    struct strbuf *err)
>  {
>  	int ret = 0, i;
>  	int n = transaction->nr;
> @@ -3612,3 +3612,9 @@ int reflog_expire(const char *refname, const unsigned char *sha1,
>  	unlock_ref(lock);
>  	return -1;
>  }
> +
> +struct ref_be refs_be_files = {
> +	NULL,
> +	"files",
> +	files_transaction_commit,
> +};
> diff --git a/refs.c b/refs.c
> index 25ad3b5..f930fe0 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -4,6 +4,29 @@
>  #include "cache.h"
>  #include "refs.h"
>  #include "lockfile.h"
> +/*
> + * We always have a files backend and it is the default.
> + */
> +struct ref_be *the_refs_backend = &refs_be_files;
> +/*
> + * List of all available backends
> + */
> +struct ref_be *refs_backends = &refs_be_files;
> +
> +/*
> + * This function is used to switch to an alternate backend.
> + */
> +int set_refs_backend(const char *name)
> +{
> +	struct ref_be *be;
> +
> +	for (be = refs_backends; be; be = be->next)
> +		if (!strcmp(be->name, name)) {
> +			the_refs_backend = be;
> +			return 0;
> +		}
> +	return 1;
> +}
>  
>  static int is_per_worktree_ref(const char *refname)
>  {
> @@ -985,3 +1008,10 @@ int ref_transaction_verify(struct ref_transaction *transaction,
>  				      NULL, old_sha1,
>  				      flags, NULL, err);
>  }
> +
> +/* backend functions */
> +int ref_transaction_commit(struct ref_transaction *transaction,
> +			   struct strbuf *err)
> +{
> +	return the_refs_backend->transaction_commit(transaction, err);
> +}
> diff --git a/refs.h b/refs.h
> index 4940ae9..419abf4 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -619,4 +619,19 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
>  			 reflog_expiry_cleanup_fn cleanup_fn,
>  			 void *policy_cb_data);
>  
> +/* refs backends */
> +typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
> +				      struct strbuf *err);
> +typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
> +
> +struct ref_be {
> +	struct ref_be *next;
> +	const char *name;
> +	ref_transaction_commit_fn *transaction_commit;
> +};
> +
> +
> +extern struct ref_be refs_be_files;
> +int set_refs_backend(const char *name);
> +
>  #endif /* REFS_H */

Re: [PATCH v3 01/44] refs.c: create a public version of verify_refname_available

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Create a public version of verify_refname_available that backends can
> provide.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs.c | 65 ++++++++++++++++++++++++++++++-----------------------------------
>  refs.h | 17 +++++++++++++++++
>  2 files changed, 47 insertions(+), 35 deletions(-)
> 
> diff --git a/refs.c b/refs.c
> index 132eff5..fe71ea0 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -279,7 +279,7 @@ struct ref_dir {
>   * presence of an empty subdirectory does not block the creation of a
>   * similarly-named reference.  (The fact that reference names with the
>   * same leading components can conflict *with each other* is a
> - * separate issue that is regulated by verify_refname_available().)
> + * separate issue that is regulated by verify_refname_available_dir().)
>   *
>   * Please note that the name field contains the fully-qualified
>   * reference (or subdirectory) name.  Space could be saved by only
> @@ -897,25 +897,13 @@ static int nonmatching_ref_fn(struct ref_entry *entry, void *vdata)
>  /*
>   * Return 0 if a reference named refname could be created without
>   * conflicting with the name of an existing reference in dir.
> - * Otherwise, return a negative value and write an explanation to err.
> - * If extras is non-NULL, it is a list of additional refnames with
> - * which refname is not allowed to conflict. If skip is non-NULL,
> - * ignore potential conflicts with refs in skip (e.g., because they
> - * are scheduled for deletion in the same operation). Behavior is
> - * undefined if the same name is listed in both extras and skip.
> - *
> - * Two reference names conflict if one of them exactly matches the
> - * leading components of the other; e.g., "refs/foo/bar" conflicts
> - * with both "refs/foo" and with "refs/foo/bar/baz" but not with
> - * "refs/foo/bar" or "refs/foo/barbados".
> - *
> - * extras and skip must be sorted.
> + * See verify_refname_available for details.
>   */
> -static int verify_refname_available(const char *refname,
> -				    const struct string_list *extras,
> -				    const struct string_list *skip,
> -				    struct ref_dir *dir,
> -				    struct strbuf *err)
> +static int verify_refname_available_dir(const char *refname,
> +					const struct string_list *extras,
> +					const struct string_list *skip,
> +					struct ref_dir *dir,
> +					struct strbuf *err)
>  {
>  	const char *slash;
>  	int pos;
> @@ -2464,9 +2452,12 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
>  		 */
>  		strbuf_git_path(&orig_ref_file, "%s", orig_refname);
>  		if (remove_empty_directories(&orig_ref_file)) {
> +			struct ref_dir *loose_refs;
> +			loose_refs = get_loose_refs(&ref_cache);

It's OK to initialize `loose_refs` on the line where it is declared.

>  			last_errno = errno;
> -			if (!verify_refname_available(orig_refname, extras, skip,
> -						      get_loose_refs(&ref_cache), err))
> +			if (!verify_refname_available_dir(orig_refname, extras,
> +							  skip, loose_refs,
> +							  err))
>  				strbuf_addf(err, "there are still refs under '%s'",
>  					    orig_refname);
>  			goto error_return;
> [...]
> diff --git a/refs.h b/refs.h
> index 6d30c98..79ea220 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -218,6 +218,23 @@ extern void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, const struct st
>  int pack_refs(unsigned int flags);
>  
>  /*
> + * Return true iff a reference named refname could be created without
> + * conflicting with the name of an existing reference.  If
> + * skip is non-NULL, ignore potential conflicts with refs in skip
> + * (e.g., because they are scheduled for deletion in the same
> + * operation).
> + *
> + * Two reference names conflict if one of them exactly matches the
> + * leading components of the other; e.g., "foo/bar" conflicts with
> + * both "foo" and with "foo/bar/baz" but not with "foo/bar" or
> + * "foo/barbados".
> + *
> + * skip must be sorted.
> + */
> +int verify_refname_available(const char *newname, struct string_list *extra,
> +			     struct string_list *skip, struct strbuf *err);

This function has an `extra` parameter but the description of this
parameter disappeared as the docstring was moved from refs.c to here.

> +
> +/*
>   * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
>   * REF_NODEREF: act on the ref directly, instead of dereferencing
>   *              symbolic references.
> 

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 02/44] refs: make repack_without_refs and is_branch public

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> is_branch was already non-static, but this patch declares it in the
> header.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
> [...]

It seems odd that repack_without_refs() should be made public (and
ultimately end up in refs.h) given that it intrinsically only has to do
with file-based references. But I will read on...

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 05/44] refs.c: move update_ref to refs.c

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Move update_ref() to the refs.c file since this function does not
> contain any backend specific code.  Move the ref classifier functions
> as well, since update_ref depends on them.
> 
> Based on Ronnie Sahlberg's patch
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 117 +-------------------------------------------------------
>  refs.c          | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 116 insertions(+), 116 deletions(-)
> 
> diff --git a/refs-be-files.c b/refs-be-files.c
> index d0dfdfc..7fe4931 100644
> --- a/refs-be-files.c
> +++ b/refs-be-files.c
> [...]
> @@ -2879,77 +2877,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
> [...]
> -static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,
> -			   const unsigned char *old_sha1, struct strbuf *err)
> -{
> -	const char *filename;
> -	int fd;
> -	static struct lock_file lock;
> -	struct strbuf buf = STRBUF_INIT;
> -	int ret = -1;
> -
> -	strbuf_addf(&buf, "%s\n", sha1_to_hex(sha1));
> -
> -	filename = git_path("%s", pseudoref);
> -	fd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
> -	if (fd < 0) {
> -		strbuf_addf(err, "Could not open '%s' for writing: %s",
> -			    filename, strerror(errno));
> -		return -1;
> -	}
> -
> -	if (old_sha1) {
> -		unsigned char actual_old_sha1[20];
> -
> -		if (read_ref(pseudoref, actual_old_sha1))
> -			die("could not read ref '%s'", pseudoref);
> -		if (hashcmp(actual_old_sha1, old_sha1)) {
> -			strbuf_addf(err, "Unexpected sha1 when writing %s", pseudoref);
> -			rollback_lock_file(&lock);
> -			goto done;
> -		}
> -	}
> -
> -	if (write_in_full(fd, buf.buf, buf.len) != buf.len) {
> -		strbuf_addf(err, "Could not write to '%s'", filename);
> -		rollback_lock_file(&lock);
> -		goto done;
> -	}
> -
> -	commit_lock_file(&lock);
> -	ret = 0;
> -done:
> -	strbuf_release(&buf);
> -	return ret;
> -}
> -
> [...]
> diff --git a/refs.c b/refs.c
> index 77492ff..2d10708 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1,3 +1,118 @@
> [...]
> +static int write_pseudoref(const char *pseudoref, const unsigned char *sha1,
> +			   const unsigned char *old_sha1, struct strbuf *err)
> +{
> +	const char *filename;
> +	int fd;
> +	static struct lock_file lock;
> +	struct strbuf buf = STRBUF_INIT;
> +	int ret = -1;
> +
> +	strbuf_addf(&buf, "%s\n", sha1_to_hex(sha1));
> +
> +	filename = git_path("%s", pseudoref);
> +	fd = hold_lock_file_for_update(&lock, filename, LOCK_DIE_ON_ERROR);
> +	if (fd < 0) {
> +		strbuf_addf(err, "Could not open '%s' for writing: %s",
> +			    filename, strerror(errno));
> +		return -1;
> +	}
> +
> +	if (old_sha1) {
> +		unsigned char actual_old_sha1[20];
> +		read_ref(pseudoref, actual_old_sha1);

The original read

		if (read_ref(pseudoref, actual_old_sha1))
			die("could not read ref '%s'", pseudoref);

This seems like an important test. What happened to it?

If its removal was intentional, it deserves a careful explanation (and
should probably be done as a separate commit). If it was an accident,
please consider how this accident arose and try to think about whether
similar accidents might have happened elsewhere in this series.

> [...]

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 06/44] refs.c: move delete_ref and delete_refs to the common code

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Move delete_pseudoref, delete_ref() and delete_refs() to the refs.c
> file since these functions do not contain any backend specific code.
> 
> Based on a patch by Ronnie Sahlberg.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs-be-files.c | 94 ---------------------------------------------------------
>  refs.c          | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 94 insertions(+), 94 deletions(-)
> 
> diff --git a/refs-be-files.c b/refs-be-files.c
> index 7fe4931..099df75 100644
> --- a/refs-be-files.c
> +++ b/refs-be-files.c
> @@ -2877,100 +2877,6 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>  	return 0;
>  }
>  
> -static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)
> -{
> -	static struct lock_file lock;
> -	const char *filename;
> -
> -	filename = git_path("%s", pseudoref);
> -
> -	if (old_sha1 && !is_null_sha1(old_sha1)) {
> -		int fd;
> -		unsigned char actual_old_sha1[20];
> -
> -		fd = hold_lock_file_for_update(&lock, filename,
> -					       LOCK_DIE_ON_ERROR);
> -		if (fd < 0)
> -			die_errno(_("Could not open '%s' for writing"), filename);
> -		if (read_ref(pseudoref, actual_old_sha1))
> -			die("could not read ref '%s'", pseudoref);
> -		if (hashcmp(actual_old_sha1, old_sha1)) {
> -			warning("Unexpected sha1 when deleting %s", pseudoref);
> -			rollback_lock_file(&lock);
> -			return -1;
> -		}
> -
> -		unlink(filename);
> -		rollback_lock_file(&lock);
> -	} else {
> -		unlink(filename);
> -	}
> -
> -	return 0;
> -}
> -
> [...]
> diff --git a/refs.c b/refs.c
> index 2d10708..205a899 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -116,3 +116,97 @@ int update_ref(const char *msg, const char *refname,
>  		ref_transaction_free(t);
>  	return 0;
>  }
> +
> +
> +static int delete_pseudoref(const char *pseudoref, const unsigned char *old_sha1)
> +{
> +	static struct lock_file lock;
> +	const char *filename;
> +
> +	filename = git_path("%s", pseudoref);
> +
> +	if (old_sha1 && !is_null_sha1(old_sha1)) {
> +		int fd;
> +		unsigned char actual_old_sha1[20];
> +
> +		fd = hold_lock_file_for_update(&lock, filename,
> +					       LOCK_DIE_ON_ERROR);
> +		if (fd < 0)
> +			die_errno(_("Could not open '%s' for writing"), filename);
> +		read_ref(pseudoref, actual_old_sha1);

The same test vanishes here as in the previous patch. The original was

		if (read_ref(pseudoref, actual_old_sha1))
			die("could not read ref '%s'", pseudoref);

> +		if (hashcmp(actual_old_sha1, old_sha1)) {
> +			warning("Unexpected sha1 when deleting %s", pseudoref);
> +			rollback_lock_file(&lock);
> +			return -1;
> +		}
> +
> +		unlink(filename);
> +		rollback_lock_file(&lock);
> +	} else {
> +		unlink(filename);
> +	}
> +
> +	return 0;
> +}
> +
> [...]

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 16/44] refs.c: move ref iterators to the common code

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs-be-files.c | 82 ---------------------------------------------------------
>  refs.c          | 81 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 81 insertions(+), 82 deletions(-)
> 
> diff --git a/refs-be-files.c b/refs-be-files.c
> index ecf2b33..46fa43c 100644
> --- a/refs-be-files.c
> +++ b/refs-be-files.c
> [...]
> @@ -2012,40 +1964,6 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
>  	return ret;
>  }
>  
> -int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
> -	const char *prefix, void *cb_data)
> -{
> -	struct strbuf real_pattern = STRBUF_INIT;
> -	struct ref_filter filter;
> -	int ret;
> -
> -	if (!prefix && !starts_with(pattern, "refs/"))
> -		strbuf_addstr(&real_pattern, "refs/");
> -	else if (prefix)
> -		strbuf_addstr(&real_pattern, prefix);
> -	strbuf_addstr(&real_pattern, pattern);
> -
> -	if (!has_glob_specials(pattern)) {
> -		/* Append implied '/' '*' if not present. */
> -		strbuf_complete(&real_pattern, '/');
> -		/* No need to check for '*', there is none. */
> -		strbuf_addch(&real_pattern, '*');
> -	}
> -
> -	filter.pattern = real_pattern.buf;
> -	filter.fn = fn;
> -	filter.cb_data = cb_data;
> -	ret = for_each_ref(filter_refs, &filter);
> -
> -	strbuf_release(&real_pattern);
> -	return ret;
> -}
> -
> [...]
> diff --git a/refs.c b/refs.c
> index 44ee4f4..7714dad 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -781,3 +781,84 @@ const char *prettify_refname(const char *name)
>  		starts_with(name, "refs/remotes/") ? 13 :
>  		0);
>  }
> [...]
> +int for_each_glob_ref_in(each_ref_fn fn, const char *pattern,
> +	const char *prefix, void *cb_data)
> +{
> +	struct strbuf real_pattern = STRBUF_INIT;
> +	struct ref_filter filter;
> +	int ret;
> +
> +	if (!prefix && !starts_with(pattern, "refs/"))
> +		strbuf_addstr(&real_pattern, "refs/");
> +	else if (prefix)
> +		strbuf_addstr(&real_pattern, prefix);
> +	strbuf_addstr(&real_pattern, pattern);
> +
> +	if (!has_glob_specials(pattern)) {
> +		/* Append implied '/' '*' if not present. */
> +		if (real_pattern.buf[real_pattern.len - 1] != '/')
> +			strbuf_addch(&real_pattern, '/');

The original has

		strbuf_complete(&real_pattern, '/');

in place of the above two lines. I think you made a mistake when
rebasing on top of

    00b6c178c3 use strbuf_complete to conditionally append slash

> +		/* No need to check for '*', there is none. */
> +		strbuf_addch(&real_pattern, '*');
> +	}
> +
> +	filter.pattern = real_pattern.buf;
> +	filter.fn = fn;
> +	filter.cb_data = cb_data;
> +	ret = for_each_ref(filter_refs, &filter);
> +
> +	strbuf_release(&real_pattern);
> +	return ret;
> +}
> +
> [...]

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 18/44] refs: move transaction functions into common code

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> The common ref code will build up a ref transaction.  Backends will
> then commit it.  So the transaction creation and update functions should
> be in the common code.  We also need to move the ref structs into
> the common code so that alternate backends can access them.
> 
> Later, we will modify struct ref_update to support alternate backends.

This patch leaks internal implementation details into the public refs
interface. I want to make sure that code elsewhere in Git treats these
structures as opaque. That is why I suggested creating an extra module
to hold "protected" code for the reference backend "class hierarchy" [1].

For a sketch of what my suggestion would look like, see the two commits
at the tip of branch "refs-be-common" on my GitHub repo [2].

Michael

[1] http://article.gmane.org/gmane.comp.version-control.git/279049
[2] https://github.com/mhagger/git

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 02/44] refs: make repack_without_refs and is_branch public

[Michael Haggerty]

On 10/13/2015 04:39 AM, Michael Haggerty wrote:
> On 10/12/2015 11:51 PM, David Turner wrote:
>> is_branch was already non-static, but this patch declares it in the
>> header.
>>
>> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
>> Signed-off-by: David Turner <dturner@twopensource.com>
>> ---
>> [...]
> 
> It seems odd that repack_without_refs() should be made public (and
> ultimately end up in refs.h) given that it intrinsically only has to do
> with file-based references. But I will read on...

I think the reason you needed to do this was because you wanted to move
delete_refs() to the common code. It is true that delete_ref() can be
moved to the common code. And most of the code in delete_refs() is just
a loop calling delete_ref(). But delete_refs() also does the very
files-specific optimization of calling repack_without_refs() before the
loop. *That* call shouldn't be in the common code.

So my suggestion is that you write a common_delete_refs() function that
only includes the loop over delete_ref(), and a files_delete_refs()
function that is basically

    {
            result = repack_without_refs(refnames, &err);
            if (result) {
                    ...report error...
                    return result;
            }
            return common_delete_refs(...);
    }

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 25/44] refs.h: document make refname_is_safe and add it to header

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> This function might be used by other refs backends
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs.h | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/refs.h b/refs.h
> index fc8a748..7a936e2 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -348,6 +348,17 @@ int verify_refname_available(const char *newname, struct string_list *extra,
>  			     struct string_list *skip, struct strbuf *err);
>  
>  /*
> + * Check if a refname is safe.
> + * For refs that start with "refs/" we consider it safe as long they do
> + * not try to resolve to outside of refs/.
> + *
> + * For all other refs we only consider them safe iff they only contain
> + * upper case characters and '_' (like "HEAD" AND "MERGE_HEAD", and not like
> + * "config").
> + */
> +int refname_is_safe(const char *refname);
> +
> +/*
>   * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
>   * REF_NODEREF: act on the ref directly, instead of dereferencing
>   *              symbolic references.
> 

The previous commit deleted this comment from where it previously
appeared in refs-be-files.c. It would make more sense to squash this
commit onto that one so it's clear that you are moving the comment
rather than creating a new comment out of thin air.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 20/44] refs-be-files.c: add methods for misc ref operations

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Add ref backend methods for:
> resolve_ref_unsafe, verify_refname_available, pack_refs, peel_ref,
> create_symref, resolve_gitlink_ref.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twitter.com>
> ---
>  builtin/init-db.c |  1 +
>  cache.h           |  7 +++++++
>  refs-be-files.c   | 36 +++++++++++++++++++++++++++---------
>  refs.c            | 47 +++++++++++++++++++++++++++++++++++++++++++++++
>  refs.h            | 18 ++++++++++++++++++
>  5 files changed, 100 insertions(+), 9 deletions(-)

This commit doesn't build:

refs-be-files.c:3636:2: error: initialization from incompatible pointer
type [-Werror]
  files_create_symref,
  ^
refs-be-files.c:3636:2: error: (near initialization for
‘refs_be_files.create_symref’) [-Werror]

Apparently one of the hunks from the subsequent commit is needed in this
commit.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 19/44] refs-be-files.c: add a backend method structure with transaction functions

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> From: Ronnie Sahlberg <sahlberg@google.com>
> 
> Add a ref structure for backend methods. Start by adding a method pointer
> for the transaction commit function.
> 
> Add a function set_refs_backend to switch between backends. The files
> based backend is the default.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 10 ++++++++--
>  refs.c          | 30 ++++++++++++++++++++++++++++++
>  refs.h          | 15 +++++++++++++++
>  3 files changed, 53 insertions(+), 2 deletions(-)
> 
> [...]
> diff --git a/refs.h b/refs.h
> index 4940ae9..419abf4 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -619,4 +619,19 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
>  			 reflog_expiry_cleanup_fn cleanup_fn,
>  			 void *policy_cb_data);
>  
> +/* refs backends */
> +typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
> +				      struct strbuf *err);
> +typedef void ref_transaction_free_fn(struct ref_transaction *transaction);

The ref_transaction_free_fn typedef isn't used anywhere.

> +struct ref_be {
> +	struct ref_be *next;
> +	const char *name;
> +	ref_transaction_commit_fn *transaction_commit;
> +};
> +
> +
> +extern struct ref_be refs_be_files;

I don't think that refs_be_files is needed in the public interface.

> +int set_refs_backend(const char *name);
> +
>  #endif /* REFS_H */

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 25/44] refs.h: document make refname_is_safe and add it to header

[Michael Haggerty]

On 10/13/2015 09:33 AM, Michael Haggerty wrote:
> On 10/12/2015 11:51 PM, David Turner wrote:
>> This function might be used by other refs backends
>>
>> Signed-off-by: David Turner <dturner@twopensource.com>
>> ---
>>  refs.h | 11 +++++++++++
>>  1 file changed, 11 insertions(+)
>>
>> diff --git a/refs.h b/refs.h
>> index fc8a748..7a936e2 100644
>> --- a/refs.h
>> +++ b/refs.h
>> @@ -348,6 +348,17 @@ int verify_refname_available(const char *newname, struct string_list *extra,
>>  			     struct string_list *skip, struct strbuf *err);
>>  
>>  /*
>> + * Check if a refname is safe.
>> + * For refs that start with "refs/" we consider it safe as long they do
>> + * not try to resolve to outside of refs/.
>> + *
>> + * For all other refs we only consider them safe iff they only contain
>> + * upper case characters and '_' (like "HEAD" AND "MERGE_HEAD", and not like
>> + * "config").
>> + */
>> +int refname_is_safe(const char *refname);
>> +
>> +/*
>>   * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
>>   * REF_NODEREF: act on the ref directly, instead of dereferencing
>>   *              symbolic references.
>>
> 
> The previous commit deleted this comment from where it previously
> appeared in refs-be-files.c. It would make more sense to squash this
> commit onto that one so it's clear that you are moving the comment
> rather than creating a new comment out of thin air.

Also, after this commit the prototype for this function appears twice in
refs.h.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 26/44] refs.c: move copy_msg to the common code

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> Rename copy_msg to copy_reflog_msg and make it public.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 28 +---------------------------
>  refs.c          | 26 ++++++++++++++++++++++++++
>  refs.h          |  2 ++
>  3 files changed, 29 insertions(+), 27 deletions(-)
> 
> [...]
> diff --git a/refs.c b/refs.c
> index 2515f6e..bd8c71b 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -886,6 +886,32 @@ int for_each_remote_ref_submodule(const char *submodule, each_ref_fn fn, void *c
>  	return for_each_ref_in_submodule(submodule, "refs/remotes/", fn, cb_data);
>  }
>  
> +/*
> + * copy the reflog message msg to buf, which has been allocated sufficiently
> + * large, while cleaning up the whitespaces.  Especially, convert LF to space,
> + * because reflog file is one line per entry.
> + */

Please put the docstring by the declaration in the header file.

> [...]
> diff --git a/refs.h b/refs.h
> index 7a936e2..3da5d09 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -597,6 +597,8 @@ enum ref_type {
>  
>  enum ref_type ref_type(const char *refname);
>  
> +int copy_reflog_msg(char *buf, const char *msg);
> +
>  enum expire_reflog_flags {
>  	EXPIRE_REFLOGS_DRY_RUN = 1 << 0,
>  	EXPIRE_REFLOGS_UPDATE_REF = 1 << 1,
> 

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 27/44] refs.c: move peel_object to the common code

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> This function does not contain any backend specific code so we
> move it to the common code.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 53 -----------------------------------------------------
>  refs.c          | 31 +++++++++++++++++++++++++++++++
>  refs.h          | 27 +++++++++++++++++++++++++++
>  3 files changed, 58 insertions(+), 53 deletions(-)
> 
> [...]
> diff --git a/refs.c b/refs.c
> index bd8c71b..99b31f6 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -4,6 +4,9 @@
>  #include "cache.h"
>  #include "refs.h"
>  #include "lockfile.h"
> +#include "object.h"
> +#include "tag.h"
> +
>  /*
>   * We always have a files backend and it is the default.
>   */
> @@ -1059,6 +1062,34 @@ int refname_is_safe(const char *refname)
>  	return 1;
>  }
>  
> +/*
> + * Peel the named object; i.e., if the object is a tag, resolve the
> + * tag recursively until a non-tag is found.  If successful, store the
> + * result to sha1 and return PEEL_PEELED.  If the object is not a tag
> + * or is not valid, return PEEL_NON_TAG or PEEL_INVALID, respectively,
> + * and leave sha1 unchanged.
> + */

Please move the docstring to the header file.

> [...]
> diff --git a/refs.h b/refs.h
> index 3da5d09..636f959 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -76,6 +76,33 @@ extern int is_branch(const char *refname);
>   */
>  extern int peel_ref(const char *refname, unsigned char *sha1);
>  
> +enum peel_status {
> +	/* object was peeled successfully: */
> +	PEEL_PEELED = 0,
> +
> +	/*
> +	 * object cannot be peeled because the named object (or an
> +	 * object referred to by a tag in the peel chain), does not
> +	 * exist.
> +	 */
> +	PEEL_INVALID = -1,
> +
> +	/* object cannot be peeled because it is not a tag: */
> +	PEEL_NON_TAG = -2,
> +
> +	/* ref_entry contains no peeled value because it is a symref: */
> +	PEEL_IS_SYMREF = -3,
> +
> +	/*
> +	 * ref_entry cannot be peeled because it is broken (i.e., the
> +	 * symbolic reference cannot even be resolved to an object
> +	 * name):
> +	 */
> +	PEEL_BROKEN = -4
> +};
> +
> +enum peel_status peel_object(const unsigned char *name, unsigned char *sha1);
> +
>  /**
>   * Resolve refname in the nested "gitlink" repository that is located
>   * at path.  If the resolution is successful, return 0 and set sha1 to
> 

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 37/44] refs: break out a ref conflict check

[Michael Haggerty]

On 10/12/2015 11:51 PM, David Turner wrote:
> Create new function verify_no_descendants, to hold one of the ref
> conflict checks used in verify_refname_available.  Multiple backends
> will need this function, so it goes in the common code.
> 
> Signed-off-by: David Turner <dturner@twopensource.com>
> ---
>  refs-be-files.c | 33 ++++++++-------------------------
>  refs.c          | 22 ++++++++++++++++++++++
>  refs.h          |  7 +++++++
>  3 files changed, 37 insertions(+), 25 deletions(-)
> 
> [...]
> diff --git a/refs.c b/refs.c
> index 5a3125d..3ae0274 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1101,6 +1101,28 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
>  	return PEEL_PEELED;
>  }
>  
> +const char *find_descendant_ref(const char *refname,
> +				const struct string_list *extras,
> +				const struct string_list *skip)
> +{
> +	int pos;
> +	if (!extras)
> +		return NULL;
> +
> +	/* Look for the place where "$refname/" would be inserted in extras. */

The comment above is misleading. See my note at the bottom of this email
for an explanation.

> +	for (pos = string_list_find_insert_index(extras, refname, 0);
> +	     pos < extras->nr; pos++) {
> +		const char *extra_refname = extras->items[pos].string;
> +
> +		if (!starts_with(extra_refname, refname))
> +			break;
> +
> +		if (!skip || !string_list_has_string(skip, extra_refname))
> +			return extra_refname;
> +	}
> +	return NULL;
> +}
> +
>  /* backend functions */
>  int refs_initdb(struct strbuf *err, int shared)
>  {
> diff --git a/refs.h b/refs.h
> index 3aad3b8..f8becea 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -637,6 +637,13 @@ int files_log_ref_write(const char *refname, const unsigned char *old_sha1,
>  			const unsigned char *new_sha1, const char *msg,
>  			int flags, struct strbuf *err);
>  
> +/*
> + * Check for entries in extras that start with "$refname/", ignoring
> + * those in skip. If there is such an entry, then we have a conflict.
> + */
> +const char *find_descendant_ref(const char *refname,
> +				const struct string_list *extras,
> +				const struct string_list *skip);

The documentation is is not correct. As written, the function needs not
the refname as argument but the refname followed by '/'. Without the
trailing slash, "refs/heads/foo" would be blocked by "refs/heads/foobar".

>From the point of view of simplicity, it would be nice if this function
could take a refname (without the trailing slash) as argument. But then
it would basically be forced to allocate a new string, copy refname to
it, append a '/', then free the string again when it's done.
Alternatively, it could accept its refname argument in a strbuf and
temporarily append the '/'.

But neither one of these alternatives is so elegant, so maybe it's OK if
this function is documented to take a "directory name, including the
trailing '/'" as argument and rename the argument (e.g., to "dirname").

Please also document that "skip" and "extras" can be NULL, but if not
NULL they need to be sorted lists.

> [...]

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 18/44] refs: move transaction functions into common code

[Michael Haggerty]

I reviewed the patches up to here pretty carefully, and aside from the
comments I already sent, they look good.

I like the new approach where the ref_transaction-building code is
shared across backends.

It seems to me that a good breaking point for the first batch of patches
would be here, just before you start creating the VTABLE in commit
[19/44]. The patches before this point all have to do with moving code
around and a little bit of light refactoring. They cause a lot of code
churn that will soon conflict with other people's work (e.g., [1]), but
I think they are pretty uncontroversial.

After this you start making a few important design decisions that
*could* be controversial. Therefore, by making a cut, you can maximize
the chance that the earlier patches can be merged to master relatively
quickly, after which the cross section for future conflicts will be much
smaller.

Ideally, you would include a few later patches in the "pre-VTABLE" patch
series. It looks like the following patches also mostly have to do with
moving code around, and would fit logically with the "pre-VTABLE" changes:

[24] refs.c: move refname_is_safe to the common code
[25] refs.h: document make refname_is_safe and add it to header
[26] refs.c: move copy_msg to the common code
[27] refs.c: move peel_object to the common code
[28] refs.c: move should_autocreate_reflog to common code
[32] initdb: move safe_create_dir into common code
[36] refs: make files_log_ref_write functions public
[37] refs: break out a ref conflict check

I tried rebasing those commits on top of your patch 18 and it wasn't too
bad. The result is on branch "refs-backend-pre-vtable" on my GitHub repo
[2], including my suggested changes to those eight patches (which
therefore became seven because I squashed the first two together).

Michael

[1] http://article.gmane.org/gmane.comp.version-control.git/279286
[2] https://github.com/mhagger/git

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 19/44] refs-be-files.c: add a backend method structure with transaction functions

[David Turner]

On Tue, 2015-10-13 at 09:56 +0200, Michael Haggerty wrote:
> On 10/12/2015 11:51 PM, David Turner wrote:
> > From: Ronnie Sahlberg <sahlberg@google.com>
> > 
> > Add a ref structure for backend methods. Start by adding a method pointer
> > for the transaction commit function.
> > 
> > Add a function set_refs_backend to switch between backends. The files
> > based backend is the default.
> > 
> > Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> > Signed-off-by: David Turner <dturner@twopensource.com>
> > ---
> >  refs-be-files.c | 10 ++++++++--
> >  refs.c          | 30 ++++++++++++++++++++++++++++++
> >  refs.h          | 15 +++++++++++++++
> >  3 files changed, 53 insertions(+), 2 deletions(-)
> > 
> > [...]
> > diff --git a/refs.h b/refs.h
> > index 4940ae9..419abf4 100644
> > --- a/refs.h
> > +++ b/refs.h
> > @@ -619,4 +619,19 @@ extern int reflog_expire(const char *refname, const unsigned char *sha1,
> >  			 reflog_expiry_cleanup_fn cleanup_fn,
> >  			 void *policy_cb_data);
> >  
> > +/* refs backends */
> > +typedef int ref_transaction_commit_fn(struct ref_transaction *transaction,
> > +				      struct strbuf *err);
> > +typedef void ref_transaction_free_fn(struct ref_transaction *transaction);
> 
> The ref_transaction_free_fn typedef isn't used anywhere.

Will fix, thanks.

> > +struct ref_be {
> > +	struct ref_be *next;
> > +	const char *name;
> > +	ref_transaction_commit_fn *transaction_commit;
> > +};
> > +
> > +
> > +extern struct ref_be refs_be_files;
> 
> I don't think that refs_be_files is needed in the public interface.

We use refs_be_lmdb in a few other places to set up the lmdb backend, so
I thought I would put refs_be_files in refs.h too.  But I can remove
refs_be_files and just stick it in the places it's needed.

Re: [PATCH v3 25/44] refs.h: document make refname_is_safe and add it to header

[David Turner]

On Tue, 2015-10-13 at 11:18 +0200, Michael Haggerty wrote:
> On 10/13/2015 09:33 AM, Michael Haggerty wrote:
> > On 10/12/2015 11:51 PM, David Turner wrote:
> >> This function might be used by other refs backends
> >>
> >> Signed-off-by: David Turner <dturner@twopensource.com>
> >> ---
> >>  refs.h | 11 +++++++++++
> >>  1 file changed, 11 insertions(+)
> >>
> >> diff --git a/refs.h b/refs.h
> >> index fc8a748..7a936e2 100644
> >> --- a/refs.h
> >> +++ b/refs.h
> >> @@ -348,6 +348,17 @@ int verify_refname_available(const char *newname, struct string_list *extra,
> >>  			     struct string_list *skip, struct strbuf *err);
> >>  
> >>  /*
> >> + * Check if a refname is safe.
> >> + * For refs that start with "refs/" we consider it safe as long they do
> >> + * not try to resolve to outside of refs/.
> >> + *
> >> + * For all other refs we only consider them safe iff they only contain
> >> + * upper case characters and '_' (like "HEAD" AND "MERGE_HEAD", and not like
> >> + * "config").
> >> + */
> >> +int refname_is_safe(const char *refname);
> >> +
> >> +/*
> >>   * Flags controlling ref_transaction_update(), ref_transaction_create(), etc.
> >>   * REF_NODEREF: act on the ref directly, instead of dereferencing
> >>   *              symbolic references.
> >>
> > 
> > The previous commit deleted this comment from where it previously
> > appeared in refs-be-files.c. It would make more sense to squash this
> > commit onto that one so it's clear that you are moving the comment
> > rather than creating a new comment out of thin air.
> 
> Also, after this commit the prototype for this function appears twice in
> refs.h.

Will squash and fix.

Re: [PATCH v3 37/44] refs: break out a ref conflict check

[David Turner]

On Tue, 2015-10-13 at 12:25 +0200, Michael Haggerty wrote:
> On 10/12/2015 11:51 PM, David Turner wrote:
> > Create new function verify_no_descendants, to hold one of the ref
> > conflict checks used in verify_refname_available.  Multiple backends
> > will need this function, so it goes in the common code.
> > 
> > Signed-off-by: David Turner <dturner@twopensource.com>
> > ---
> >  refs-be-files.c | 33 ++++++++-------------------------
> >  refs.c          | 22 ++++++++++++++++++++++
> >  refs.h          |  7 +++++++
> >  3 files changed, 37 insertions(+), 25 deletions(-)
> > 
> > [...]
> > diff --git a/refs.c b/refs.c
> > index 5a3125d..3ae0274 100644
> > --- a/refs.c
> > +++ b/refs.c
> > @@ -1101,6 +1101,28 @@ enum peel_status peel_object(const unsigned char *name, unsigned char *sha1)
> >  	return PEEL_PEELED;
> >  }
> >  
> > +const char *find_descendant_ref(const char *refname,
> > +				const struct string_list *extras,
> > +				const struct string_list *skip)
> > +{
> > +	int pos;
> > +	if (!extras)
> > +		return NULL;
> > +
> > +	/* Look for the place where "$refname/" would be inserted in extras. */
> 
> The comment above is misleading. See my note at the bottom of this email
> for an explanation.
> 
> > +	for (pos = string_list_find_insert_index(extras, refname, 0);
> > +	     pos < extras->nr; pos++) {
> > +		const char *extra_refname = extras->items[pos].string;
> > +
> > +		if (!starts_with(extra_refname, refname))
> > +			break;
> > +
> > +		if (!skip || !string_list_has_string(skip, extra_refname))
> > +			return extra_refname;
> > +	}
> > +	return NULL;
> > +}
> > +
> >  /* backend functions */
> >  int refs_initdb(struct strbuf *err, int shared)
> >  {
> > diff --git a/refs.h b/refs.h
> > index 3aad3b8..f8becea 100644
> > --- a/refs.h
> > +++ b/refs.h
> > @@ -637,6 +637,13 @@ int files_log_ref_write(const char *refname, const unsigned char *old_sha1,
> >  			const unsigned char *new_sha1, const char *msg,
> >  			int flags, struct strbuf *err);
> >  
> > +/*
> > + * Check for entries in extras that start with "$refname/", ignoring
> > + * those in skip. If there is such an entry, then we have a conflict.
> > + */
> > +const char *find_descendant_ref(const char *refname,
> > +				const struct string_list *extras,
> > +				const struct string_list *skip);
> 
> The documentation is is not correct. As written, the function needs not
> the refname as argument but the refname followed by '/'. Without the
> trailing slash, "refs/heads/foo" would be blocked by "refs/heads/foobar".
> 
> From the point of view of simplicity, it would be nice if this function
> could take a refname (without the trailing slash) as argument. But then
> it would basically be forced to allocate a new string, copy refname to
> it, append a '/', then free the string again when it's done.
> Alternatively, it could accept its refname argument in a strbuf and
> temporarily append the '/'.
> 
> But neither one of these alternatives is so elegant, so maybe it's OK if
> this function is documented to take a "directory name, including the
> trailing '/'" as argument and rename the argument (e.g., to "dirname").
> 
> Please also document that "skip" and "extras" can be NULL, but if not
> NULL they need to be sorted lists.

I think `extras` may not be NULL.  But I've otherwise fixed this.
Thanks.

Re: [PATCH v3 05/44] refs.c: move update_ref to refs.c

[David Turner]

On Tue, 2015-10-13 at 05:41 +0200, Michael Haggerty wrote:

> If its removal was intentional, it deserves a careful explanation (and
> should probably be done as a separate commit). If it was an accident,
> please consider how this accident arose and try to think about whether
> similar accidents might have happened elsewhere in this series.

This was an accident. I think it must have happened when I
forward-ported Ronnie's changes over my change that introduced that
check.  Usually, when there were conflicts during this process
(indicating that the moved code had changed in the meantime), I did the
move by copy-pasting the code (rather than by choosing the old version).
Apparently, I missed this one.

Will fix.

> > [...]
> 
> Michael
> 

Re: [PATCH v3 19/44] refs-be-files.c: add a backend method structure with transaction functions

[Michael Haggerty]

On 10/13/2015 08:28 PM, David Turner wrote:
> On Tue, 2015-10-13 at 09:56 +0200, Michael Haggerty wrote:
>> On 10/12/2015 11:51 PM, David Turner wrote:
>>> [...]
>>> +extern struct ref_be refs_be_files;
>>
>> I don't think that refs_be_files is needed in the public interface.
> 
> We use refs_be_lmdb in a few other places to set up the lmdb backend, so
> I thought I would put refs_be_files in refs.h too.  But I can remove
> refs_be_files and just stick it in the places it's needed.

It's OK then. It doesn't hurt to leave it for consistency's sake.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v3 02/44] refs: make repack_without_refs and is_branch public

[David Turner]

On Tue, 2015-10-13 at 09:23 +0200, Michael Haggerty wrote:
> On 10/13/2015 04:39 AM, Michael Haggerty wrote:
> > On 10/12/2015 11:51 PM, David Turner wrote:
> >> is_branch was already non-static, but this patch declares it in the
> >> header.
> >>
> >> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> >> Signed-off-by: David Turner <dturner@twopensource.com>
> >> ---
> >> [...]
> > 
> > It seems odd that repack_without_refs() should be made public (and
> > ultimately end up in refs.h) given that it intrinsically only has to do
> > with file-based references. But I will read on...
> 
> I think the reason you needed to do this was because you wanted to move
> delete_refs() to the common code. It is true that delete_ref() can be
> moved to the common code. And most of the code in delete_refs() is just
> a loop calling delete_ref(). But delete_refs() also does the very
> files-specific optimization of calling repack_without_refs() before the
> loop. *That* call shouldn't be in the common code.
> 
> So my suggestion is that you write a common_delete_refs() function that
> only includes the loop over delete_ref(), and a files_delete_refs()
> function that is basically
> 
>     {
>             result = repack_without_refs(refnames, &err);
>             if (result) {
>                     ...report error...
>                     return result;
>             }
>             return common_delete_refs(...);
>     }

OK, I can do that.  That will have to be part of the backends work, so
I'll exclude it from the refs-backend-pre-vtable patch set.

Re: [PATCH v3 18/44] refs: move transaction functions into common code

[David Turner]

On Tue, 2015-10-13 at 13:29 +0200, Michael Haggerty wrote:
> I reviewed the patches up to here pretty carefully, and aside from the
> comments I already sent, they look good.
> 
> I like the new approach where the ref_transaction-building code is
> shared across backends.
> 
> It seems to me that a good breaking point for the first batch of patches
> would be here, just before you start creating the VTABLE in commit
> [19/44]. The patches before this point all have to do with moving code
> around and a little bit of light refactoring. They cause a lot of code
> churn that will soon conflict with other people's work (e.g., [1]), but
> I think they are pretty uncontroversial.
> 
> After this you start making a few important design decisions that
> *could* be controversial. Therefore, by making a cut, you can maximize
> the chance that the earlier patches can be merged to master relatively
> quickly, after which the cross section for future conflicts will be much
> smaller.
> 
> Ideally, you would include a few later patches in the "pre-VTABLE" patch
> series. It looks like the following patches also mostly have to do with
> moving code around, and would fit logically with the "pre-VTABLE" changes:
> 
> [24] refs.c: move refname_is_safe to the common code
> [25] refs.h: document make refname_is_safe and add it to header
> [26] refs.c: move copy_msg to the common code
> [27] refs.c: move peel_object to the common code
> [28] refs.c: move should_autocreate_reflog to common code
> [32] initdb: move safe_create_dir into common code
> [36] refs: make files_log_ref_write functions public
> [37] refs: break out a ref conflict check
> 
> I tried rebasing those commits on top of your patch 18 and it wasn't too
> bad. The result is on branch "refs-backend-pre-vtable" on my GitHub repo
> [2], including my suggested changes to those eight patches (which
> therefore became seven because I squashed the first two together).

Thanks.  I started from that, and made the changes that you suggested 
in reviews that were not yet in there.

I also added Jeff's extension patch, since it seems uncontroversial to
me, and since we'll need it sooner or later anyway.

I put the result on my github at:
https://github.com/dturner-tw/git
on the refs-backend-pre-vtable branch 

Re: [PATCH v3 18/44] refs: move transaction functions into common code

[David Turner]

On Tue, 2015-10-13 at 18:21 -0400, David Turner wrote:
> On Tue, 2015-10-13 at 13:29 +0200, Michael Haggerty wrote:
> > I reviewed the patches up to here pretty carefully, and aside from the
> > comments I already sent, they look good.
> > 
> > I like the new approach where the ref_transaction-building code is
> > shared across backends.
> > 
> > It seems to me that a good breaking point for the first batch of patches
> > would be here, just before you start creating the VTABLE in commit
> > [19/44]. The patches before this point all have to do with moving code
> > around and a little bit of light refactoring. They cause a lot of code
> > churn that will soon conflict with other people's work (e.g., [1]), but
> > I think they are pretty uncontroversial.
> > 
> > After this you start making a few important design decisions that
> > *could* be controversial. Therefore, by making a cut, you can maximize
> > the chance that the earlier patches can be merged to master relatively
> > quickly, after which the cross section for future conflicts will be much
> > smaller.
> > 
> > Ideally, you would include a few later patches in the "pre-VTABLE" patch
> > series. It looks like the following patches also mostly have to do with
> > moving code around, and would fit logically with the "pre-VTABLE" changes:
> > 
> > [24] refs.c: move refname_is_safe to the common code
> > [25] refs.h: document make refname_is_safe and add it to header
> > [26] refs.c: move copy_msg to the common code
> > [27] refs.c: move peel_object to the common code
> > [28] refs.c: move should_autocreate_reflog to common code
> > [32] initdb: move safe_create_dir into common code
> > [36] refs: make files_log_ref_write functions public
> > [37] refs: break out a ref conflict check
> > 
> > I tried rebasing those commits on top of your patch 18 and it wasn't too
> > bad. The result is on branch "refs-backend-pre-vtable" on my GitHub repo
> > [2], including my suggested changes to those eight patches (which
> > therefore became seven because I squashed the first two together).
> 
> Thanks.  I started from that, and made the changes that you suggested 
> in reviews that were not yet in there.
> 
> I also added Jeff's extension patch, since it seems uncontroversial to
> me, and since we'll need it sooner or later anyway.
> 
> I put the result on my github at:
> https://github.com/dturner-tw/git
> on the refs-backend-pre-vtable branch 

While rebasing the rest of the series on this, I noticed an issue in one
of the patches, which I have now fixed.  I re-pushed.

Re: [PATCH v3 05/44] refs.c: move update_ref to refs.c

[David Turner]

On Tue, 2015-10-13 at 05:41 +0200, Michael Haggerty wrote:

> The original read
> 
> 		if (read_ref(pseudoref, actual_old_sha1))
> 			die("could not read ref '%s'", pseudoref);
> 
> This seems like an important test. What happened to it?
> 
> If its removal was intentional, it deserves a careful explanation (and
> should probably be done as a separate commit). If it was an accident,
> please consider how this accident arose and try to think about whether
> similar accidents might have happened elsewhere in this series.

I went ahead and manually rechecked all of the patches to ensure that
nothing else was screwed up.  While doing so, I found two or three
spurious whitespace changes (moving newlines around), which I fixed (in
refs-backend-pre-vtable, which I'll send to the list tomorrow-ish).