[PATCH v8 00/44] Use ref transactions for all ref updates

[PATCH v8 00/44] Use ref transactions for all ref updates

[Ronnie Sahlberg]

This patch series can also be found at
https://github.com/rsahlberg/git/tree/ref-transactions


This patch series is based on next and expands on the transaction API. It
converts all ref updates, inside refs.c as well as external, to use the
transaction API for updates. This makes most of the ref updates to become
atomic when there are failures locking or writing to a ref.

This version completes the work to convert all ref updates to use transactions.
Now that all updates are through transactions I will start working on
cleaning up the reading of refs and to create an api for managing reflogs but
all that will go in a different patch series.

Version 8:
 - Updates after review by JN
 - Improve commit messages
 - Add a patch that adds an err argument to repack_without_refs
 - Add a patch that adds an err argument to delete_loose_ref
 - Better document that a _update/_delete/_create failure means the whole
   transaction has failed and needs rollback.

Version 7:
 - Updated commit messages per JNs review comments.
 - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not
   exposed through refs.h

Version 6:
 - Convert all updates in refs.c to use transactions too.

Version 5:
 - Reword commit messages for having _create/_delete/_update returning
   success/failure. There are no conditions yet that return an error from
   these failures but there will be in the future. So we still check the
   return from these functions in the callers in preparation for this.
 - Don't leak memory by just passing a strbuf_detach() pointer to functions.
   Use <obj>.buf and explicitely strbuf_release the data afterwards.
 - Remove the function update_ref_lock.
 - Remove the function update_ref_write.
 - Track transaction status and die(BUG:) if we call _create/_delete/_update/
   _commit for a transaction that is not OPEN.

Version 4:
 - Rename patch series from "Use ref transactions from most callers" to
   "Use ref transactions for all ref updates".
 - Convert all external ref writes to use transactions and make write_ref_sha1
   and lock_ref_sha1 static functions.
 - Change the ref commit and free handling so we no longer pass pointer to
   pointer to _commit. _commit no longer frees the transaction. The caller
   MUST call _free itself.
 - Change _commit to take a strbuf pointer instead of a char* for error
   reporting back to the caller.
 - Re-add the walker patch after fixing it.

Version 3:
 - Remove the walker patch for now. Walker needs more complex solution
   so defer it until the basics are done.
 - Remove the onerr argument to ref_transaction_commit(). All callers
   that need to die() on error now have to do this explicitely.
 - Pass an error string from ref_transaction_commit() back to the callers
   so that they can craft a nice error message upon failures.
 - Make ref_transaction_rollback() accept NULL as argument.
 - Change ref_transaction_commit() to take a pointer to pointer argument for
   the transaction and have it clear the callers pointer to NULL when
   invoked. This allows for much nicer handling of transaction rollback on
   failure.
Version 2:
 - Add a patch to ref_transaction_commit to make it honor onerr even if the
   error triggered in ref_Transaction_commit itself rather than in a call
   to other functions (that already honor onerr).
 - Add a patch to make the update_ref() helper function use transactions
   internally.
 - Change ref_transaction_update to die() instead of error() if we pass
   if a NULL old_sha1 but have have_old == true.
 - Change ref_transaction_create to die() instead of error() if new_sha1
   is false but we pass it a null_sha1.
 - Change ref_transaction_delete die() instead of error() if we pass
   if a NULL old_sha1 but have have_old == true.
 - Change several places to do  if(!transaction || ref_transaction_update()
   || ref_Transaction_commit()) die(generic-message) instead of checking each
   step separately and having a different message for each failure.
   Most users are likely not interested in what step of the transaction
   failed and only whether it failed or not.
 - Change commit.c to only pass a pointer to ref_transaction_update
   iff current_head is non-NULL.
   The previous patch used to compute a garbage pointer for
   current_head->object.sha1 and relied on the fact that ref_transaction_update
   would not try to dereference this pointer if !!current_head was 0.
 - Updated commit message for the walker_fetch change to try to justify why
   the change in locking semantics should not be harmful.



Ronnie Sahlberg (44):
  refs.c: constify the sha arguments for
    ref_transaction_create|delete|update
  refs.c: allow passing NULL to ref_transaction_free
  refs.c: add a strbuf argument to ref_transaction_commit for error
    logging
  refs.c: add an err argument to repack_without_refs
  refs.c: make ref_update_reject_duplicates take a strbuf argument for
    errors
  refs.c: add an err argument ro delete_loose_ref
  refs.c: make update_ref_write update a strbuf on failure
  update-ref.c: log transaction error from the update_ref
  refs.c: remove the onerr argument to ref_transaction_commit
  refs.c: change ref_transaction_update() to do error checking and
    return status
  refs.c: change ref_transaction_create to do error checking and return
    status
  refs.c: ref_transaction_delete to check for error and return status
  tag.c: use ref transactions when doing updates
  replace.c: use the ref transaction functions for updates
  commit.c: use ref transactions for updates
  sequencer.c: use ref transactions for all ref updates
  fast-import.c: change update_branch to use ref transactions
  branch.c: use ref transaction for all ref updates
  refs.c: change update_ref to use a transaction
  refs.c: free the transaction before returning when number of updates
    is 0
  refs.c: ref_transaction_commit should not free the transaction
  fetch.c: clear errno before calling functions that might set it
  fetch.c: change s_update_ref to use a ref transaction
  fetch.c: use a single ref transaction for all ref updates
  receive-pack.c: use a reference transaction for updating the refs
  fast-import.c: use a ref transaction when dumping tags
  walker.c: use ref transaction for ref updates
  refs.c: make write_ref_sha1 static
  refs.c: make lock_ref_sha1 static
  refs.c: add transaction.status and track OPEN/CLOSED/ERROR
  refs.c: remove the update_ref_lock function
  refs.c: remove the update_ref_write function
  refs.c: remove lock_ref_sha1
  refs.c: make prune_ref use a transaction to delete the ref
  refs.c: make delete_ref use a transaction
  refs.c: pass the ref log message to _create/delete/update instead of
    _commit
  refs.c: pass NULL as *flags to read_ref_full
  refs.c: pack all refs before we start to rename a ref
  refs.c: move the check for valid refname to lock_ref_sha1_basic
  refs.c: call lock_ref_sha1_basic directly from commit
  refs.c: add a new flag for transaction delete for refs we know are
    packed only
  refs.c: pass a skip list to name_conflict_fn
  refs.c: make rename_ref use a transaction
  refs.c: remove forward declaration of write_ref_sha1

 branch.c               |  31 ++--
 builtin/commit.c       |  24 ++-
 builtin/fetch.c        |  29 ++--
 builtin/receive-pack.c |  20 +--
 builtin/replace.c      |  15 +-
 builtin/tag.c          |  15 +-
 builtin/update-ref.c   |  32 ++--
 fast-import.c          |  39 +++--
 refs.c                 | 443 ++++++++++++++++++++++++++++---------------------
 refs.h                 |  54 +++---
 sequencer.c            |  24 ++-
 t/t3200-branch.sh      |   2 +-
 walker.c               |  51 +++---
 13 files changed, 449 insertions(+), 330 deletions(-)

-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 01/44] refs.c: constify the sha arguments for ref_transaction_create|delete|update

[Ronnie Sahlberg]

ref_transaction_create|delete|update has no need to modify the sha1
arguments passed to it so it should use const unsigned char* instead
of unsigned char*.

Some functions, such as fast_forward_to(), already have its old/new
sha1 arguments as consts. This function will at some point need to
use ref_transaction_update() in which case this change is required.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 7 ++++---
 refs.h | 7 ++++---
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/refs.c b/refs.c
index 6898263..2114748 100644
--- a/refs.c
+++ b/refs.c
@@ -3338,7 +3338,8 @@ static struct ref_update *add_update(struct ref_transaction *transaction,
 
 void ref_transaction_update(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *new_sha1, unsigned char *old_sha1,
+			    const unsigned char *new_sha1,
+			    const unsigned char *old_sha1,
 			    int flags, int have_old)
 {
 	struct ref_update *update = add_update(transaction, refname);
@@ -3352,7 +3353,7 @@ void ref_transaction_update(struct ref_transaction *transaction,
 
 void ref_transaction_create(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *new_sha1,
+			    const unsigned char *new_sha1,
 			    int flags)
 {
 	struct ref_update *update = add_update(transaction, refname);
@@ -3366,7 +3367,7 @@ void ref_transaction_create(struct ref_transaction *transaction,
 
 void ref_transaction_delete(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *old_sha1,
+			    const unsigned char *old_sha1,
 			    int flags, int have_old)
 {
 	struct ref_update *update = add_update(transaction, refname);
diff --git a/refs.h b/refs.h
index 09ff483..6d97700 100644
--- a/refs.h
+++ b/refs.h
@@ -245,7 +245,8 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
  */
 void ref_transaction_update(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *new_sha1, unsigned char *old_sha1,
+			    const unsigned char *new_sha1,
+			    const unsigned char *old_sha1,
 			    int flags, int have_old);
 
 /*
@@ -256,7 +257,7 @@ void ref_transaction_update(struct ref_transaction *transaction,
  */
 void ref_transaction_create(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *new_sha1,
+			    const unsigned char *new_sha1,
 			    int flags);
 
 /*
@@ -266,7 +267,7 @@ void ref_transaction_create(struct ref_transaction *transaction,
  */
 void ref_transaction_delete(struct ref_transaction *transaction,
 			    const char *refname,
-			    unsigned char *old_sha1,
+			    const unsigned char *old_sha1,
 			    int flags, int have_old);
 
 /*
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free

[Ronnie Sahlberg]

Allow ref_transaction_free to be called with NULL and as a result allow
ref_transaction_rollback to be called for a NULL transaction.

This allows us to write code that will
  if ( (!transaction ||
        ref_transaction_update(...))  ||
      (ref_transaction_commit(...) && !(transaction = NULL)) {
          ref_transaction_rollback(transaction);
          ...
  }

In this case transaction is reset to NULL IFF ref_transaction_commit() was
invoked and thus the rollback becomes ref_transaction_rollback(NULL) which
is safe. IF the conditional triggered prior to ref_transaction_commit()
then transaction is untouched and then ref_transaction_rollback(transaction)
will rollback the failed transaction.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/refs.c b/refs.c
index 2114748..88d73c8 100644
--- a/refs.c
+++ b/refs.c
@@ -3312,6 +3312,9 @@ static void ref_transaction_free(struct ref_transaction *transaction)
 {
 	int i;
 
+	if (!transaction)
+		return;
+
 	for (i = 0; i < transaction->nr; i++)
 		free(transaction->updates[i]);
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 03/44] refs.c: add a strbuf argument to ref_transaction_commit for error logging

[Ronnie Sahlberg]

Add a strbuf argument to _commit so that we can pass an error string back to
the caller. So that we can do error logging from the caller instead of from
_commit.

Longer term plan is to first convert all callers to use onerr==QUIET_ON_ERR
and craft any log messages from the callers themselves and finally remove the
onerr argument completely.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c | 2 +-
 refs.c               | 6 +++++-
 refs.h               | 5 ++++-
 3 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 405267f..aaa06aa 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -367,7 +367,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 		if (end_null)
 			line_termination = '\0';
 		update_refs_stdin();
-		ret = ref_transaction_commit(transaction, msg,
+		ret = ref_transaction_commit(transaction, msg, NULL,
 					     UPDATE_REFS_DIE_ON_ERR);
 		return ret;
 	}
diff --git a/refs.c b/refs.c
index 88d73c8..1a7f9d9 100644
--- a/refs.c
+++ b/refs.c
@@ -3423,7 +3423,8 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,
 }
 
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, enum action_on_err onerr)
+			   const char *msg, struct strbuf *err,
+			   enum action_on_err onerr)
 {
 	int ret = 0, delnum = 0, i;
 	const char **delnames;
@@ -3452,6 +3453,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 					       update->flags,
 					       &update->type, onerr);
 		if (!update->lock) {
+			if (err)
+				strbuf_addf(err, "Cannot lock the ref '%s'.",
+					    update->refname);
 			ret = 1;
 			goto cleanup;
 		}
diff --git a/refs.h b/refs.h
index 6d97700..25ae110 100644
--- a/refs.h
+++ b/refs.h
@@ -274,9 +274,12 @@ void ref_transaction_delete(struct ref_transaction *transaction,
  * Commit all of the changes that have been queued in transaction, as
  * atomically as possible.  Return a nonzero value if there is a
  * problem.  The ref_transaction is freed by this function.
+ * If err is non-NULL we will add an error string to it to explain why
+ * the transaction failed. The string does not end in newline.
  */
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, enum action_on_err onerr);
+			   const char *msg, struct strbuf *err,
+			   enum action_on_err onerr);
 
 /** Lock a ref and then write its file */
 int update_ref(const char *action, const char *refname,
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 04/44] refs.c: add an err argument to repack_without_refs

[Ronnie Sahlberg]

Update repack_without_refs to take an err argument and update it if there
is a failure. Pass the err variable from ref_transaction_commit to this
function so that callers can print a meaningful error message if _commit
fails due to a problem in repack_without_refs.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/refs.c b/refs.c
index 1a7f9d9..c2b720e 100644
--- a/refs.c
+++ b/refs.c
@@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
 	return 0;
 }
 
-static int repack_without_refs(const char **refnames, int n)
+static int repack_without_refs(const char **refnames, int n, struct strbuf *err)
 {
 	struct ref_dir *packed;
 	struct string_list refs_to_delete = STRING_LIST_INIT_DUP;
 	struct string_list_item *ref_to_delete;
-	int i, removed = 0;
+	int i, ret, removed = 0;
 
 	/* Look for a packed ref */
 	for (i = 0; i < n; i++)
@@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)
 
 	if (lock_packed_refs(0)) {
 		unable_to_lock_error(git_path("packed-refs"), errno);
+		if (err)
+			strbuf_addf(err, "cannot delete '%s' from packed refs",
+				    refnames[i]);
 		return error("cannot delete '%s' from packed refs", refnames[i]);
 	}
 	packed = get_packed_refs(&ref_cache);
@@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)
 	}
 
 	/* Write what remains */
-	return commit_packed_refs();
+	ret = commit_packed_refs();
+	if (ret && err)
+		strbuf_addf(err, "unable to overwrite old ref-pack file");
+	return ret;
 }
 
 static int repack_without_ref(const char *refname)
 {
-	return repack_without_refs(&refname, 1);
+	return repack_without_refs(&refname, 1, NULL);
 }
 
 static int delete_ref_loose(struct ref_lock *lock, int flag)
@@ -3486,7 +3492,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 		}
 	}
 
-	ret |= repack_without_refs(delnames, delnum);
+	ret |= repack_without_refs(delnames, delnum, err);
 	for (i = 0; i < delnum; i++)
 		unlink_or_warn(git_path("logs/%s", delnames[i]));
 	clear_loose_ref_cache(&ref_cache);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 05/44] refs.c: make ref_update_reject_duplicates take a strbuf argument for errors

[Ronnie Sahlberg]

Make ref_update_reject_duplicates return any error that occurs through a
new strbuf argument. This means that when a transaction commit fails in
this function we will now be able to pass a helpful error message back to the
caller.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index c2b720e..072e3e9 100644
--- a/refs.c
+++ b/refs.c
@@ -3408,6 +3408,7 @@ static int ref_update_compare(const void *r1, const void *r2)
 }
 
 static int ref_update_reject_duplicates(struct ref_update **updates, int n,
+					struct strbuf *err,
 					enum action_on_err onerr)
 {
 	int i;
@@ -3415,6 +3416,9 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,
 		if (!strcmp(updates[i - 1]->refname, updates[i]->refname)) {
 			const char *str =
 				"Multiple updates for ref '%s' not allowed.";
+			if (err)
+				strbuf_addf(err, str, updates[i]->refname);
+
 			switch (onerr) {
 			case UPDATE_REFS_MSG_ON_ERR:
 				error(str, updates[i]->refname); break;
@@ -3445,7 +3449,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 
 	/* Copy, sort, and reject duplicate refs */
 	qsort(updates, n, sizeof(*updates), ref_update_compare);
-	ret = ref_update_reject_duplicates(updates, n, onerr);
+	ret = ref_update_reject_duplicates(updates, n, err, onerr);
 	if (ret)
 		goto cleanup;
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref

[Ronnie Sahlberg]

Add an err argument to delete_loose_ref so that we can pass a descriptive
error string back to the caller. Pass the err argument from transaction
commit to this function so that transaction users will have a nice error
string if the transaction failed due to delete_loose_ref failing.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)

diff --git a/refs.c b/refs.c
index 072e3e9..a150520 100644
--- a/refs.c
+++ b/refs.c
@@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)
 	return repack_without_refs(&refname, 1, NULL);
 }
 
-static int delete_ref_loose(struct ref_lock *lock, int flag)
+static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 {
 	if (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {
 		/* loose */
-		int err, i = strlen(lock->lk->filename) - 5; /* .lock */
+		int res, i = strlen(lock->lk->filename) - 5; /* .lock */
 
 		lock->lk->filename[i] = 0;
-		err = unlink_or_warn(lock->lk->filename);
+		res = unlink_or_warn(lock->lk->filename);
 		lock->lk->filename[i] = '.';
-		if (err && errno != ENOENT)
+		if (res && errno != ENOENT) {
+			if (err)
+				strbuf_addf(err,
+					    "failed to delete loose ref '%s'",
+					    lock->lk->filename);
 			return 1;
+		}
 	}
 	return 0;
 }
@@ -2507,7 +2512,7 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
 	lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);
 	if (!lock)
 		return 1;
-	ret |= delete_ref_loose(lock, flag);
+	ret |= delete_ref_loose(lock, flag, NULL);
 
 	/* removing the loose one could have resurrected an earlier
 	 * packed one.  Also, if it was not loose we need to repack
@@ -3492,7 +3497,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 
 		if (update->lock) {
 			delnames[delnum++] = update->lock->ref_name;
-			ret |= delete_ref_loose(update->lock, update->type);
+			ret |= delete_ref_loose(update->lock, update->type,
+						err);
 		}
 	}
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 07/44] refs.c: make update_ref_write update a strbuf on failure

[Ronnie Sahlberg]

Change update_ref_write to also update an error strbuf on failure.
This makes the error available to ref_transaction_commit callers if the
transaction failed due to update_ref_sha1/write_ref_sha1 failures.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/refs.c b/refs.c
index a150520..18dff1b 100644
--- a/refs.c
+++ b/refs.c
@@ -3273,10 +3273,13 @@ static struct ref_lock *update_ref_lock(const char *refname,
 
 static int update_ref_write(const char *action, const char *refname,
 			    const unsigned char *sha1, struct ref_lock *lock,
-			    enum action_on_err onerr)
+			    struct strbuf *err, enum action_on_err onerr)
 {
 	if (write_ref_sha1(lock, sha1, action) < 0) {
 		const char *str = "Cannot update the ref '%s'.";
+		if (err)
+			strbuf_addf(err, str, refname);
+
 		switch (onerr) {
 		case UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;
 		case UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;
@@ -3402,7 +3405,7 @@ int update_ref(const char *action, const char *refname,
 	lock = update_ref_lock(refname, oldval, flags, NULL, onerr);
 	if (!lock)
 		return 1;
-	return update_ref_write(action, refname, sha1, lock, onerr);
+	return update_ref_write(action, refname, sha1, lock, NULL, onerr);
 }
 
 static int ref_update_compare(const void *r1, const void *r2)
@@ -3484,7 +3487,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 			ret = update_ref_write(msg,
 					       update->refname,
 					       update->new_sha1,
-					       update->lock, onerr);
+					       update->lock, err, onerr);
 			update->lock = NULL; /* freed by update_ref_write */
 			if (ret)
 				goto cleanup;
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 08/44] update-ref.c: log transaction error from the update_ref

[Ronnie Sahlberg]

Call ref_transaction_commit with QUIET_ON_ERR and use the strbuf that is
returned to print a log message if/after the transaction fails.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index aaa06aa..207e24d 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -342,6 +342,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 	const char *refname, *oldval, *msg = NULL;
 	unsigned char sha1[20], oldsha1[20];
 	int delete = 0, no_deref = 0, read_stdin = 0, end_null = 0, flags = 0;
+	struct strbuf err = STRBUF_INIT;
 	struct option options[] = {
 		OPT_STRING( 'm', NULL, &msg, N_("reason"), N_("reason of the update")),
 		OPT_BOOL('d', NULL, &delete, N_("delete the reference")),
@@ -359,17 +360,16 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 		die("Refusing to perform update with empty message.");
 
 	if (read_stdin) {
-		int ret;
 		transaction = ref_transaction_begin();
-
 		if (delete || no_deref || argc > 0)
 			usage_with_options(git_update_ref_usage, options);
 		if (end_null)
 			line_termination = '\0';
 		update_refs_stdin();
-		ret = ref_transaction_commit(transaction, msg, NULL,
-					     UPDATE_REFS_DIE_ON_ERR);
-		return ret;
+		if (ref_transaction_commit(transaction, msg, &err,
+					   UPDATE_REFS_QUIET_ON_ERR))
+			die("%s", err.buf);
+		return 0;
 	}
 
 	if (end_null)
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 09/44] refs.c: remove the onerr argument to ref_transaction_commit

[Ronnie Sahlberg]

Since all callers now use QUIET_ON_ERR we no longer need to provide an onerr
argument any more. Remove the onerr argument from the ref_transaction_commit
signature.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c |  3 +--
 refs.c               | 22 +++++++---------------
 refs.h               |  3 +--
 3 files changed, 9 insertions(+), 19 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 207e24d..2bef2a0 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -366,8 +366,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 		if (end_null)
 			line_termination = '\0';
 		update_refs_stdin();
-		if (ref_transaction_commit(transaction, msg, &err,
-					   UPDATE_REFS_QUIET_ON_ERR))
+		if (ref_transaction_commit(transaction, msg, &err))
 			die("%s", err.buf);
 		return 0;
 	}
diff --git a/refs.c b/refs.c
index 18dff1b..bc21060 100644
--- a/refs.c
+++ b/refs.c
@@ -3416,8 +3416,7 @@ static int ref_update_compare(const void *r1, const void *r2)
 }
 
 static int ref_update_reject_duplicates(struct ref_update **updates, int n,
-					struct strbuf *err,
-					enum action_on_err onerr)
+					struct strbuf *err)
 {
 	int i;
 	for (i = 1; i < n; i++)
@@ -3427,22 +3426,13 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,
 			if (err)
 				strbuf_addf(err, str, updates[i]->refname);
 
-			switch (onerr) {
-			case UPDATE_REFS_MSG_ON_ERR:
-				error(str, updates[i]->refname); break;
-			case UPDATE_REFS_DIE_ON_ERR:
-				die(str, updates[i]->refname); break;
-			case UPDATE_REFS_QUIET_ON_ERR:
-				break;
-			}
 			return 1;
 		}
 	return 0;
 }
 
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, struct strbuf *err,
-			   enum action_on_err onerr)
+			   const char *msg, struct strbuf *err)
 {
 	int ret = 0, delnum = 0, i;
 	const char **delnames;
@@ -3457,7 +3447,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 
 	/* Copy, sort, and reject duplicate refs */
 	qsort(updates, n, sizeof(*updates), ref_update_compare);
-	ret = ref_update_reject_duplicates(updates, n, err, onerr);
+	ret = ref_update_reject_duplicates(updates, n, err);
 	if (ret)
 		goto cleanup;
 
@@ -3469,7 +3459,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 					       (update->have_old ?
 						update->old_sha1 : NULL),
 					       update->flags,
-					       &update->type, onerr);
+					       &update->type,
+					       UPDATE_REFS_QUIET_ON_ERR);
 		if (!update->lock) {
 			if (err)
 				strbuf_addf(err, "Cannot lock the ref '%s'.",
@@ -3487,7 +3478,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 			ret = update_ref_write(msg,
 					       update->refname,
 					       update->new_sha1,
-					       update->lock, err, onerr);
+					       update->lock, err,
+					       UPDATE_REFS_QUIET_ON_ERR);
 			update->lock = NULL; /* freed by update_ref_write */
 			if (ret)
 				goto cleanup;
diff --git a/refs.h b/refs.h
index 25ae110..555ee59 100644
--- a/refs.h
+++ b/refs.h
@@ -278,8 +278,7 @@ void ref_transaction_delete(struct ref_transaction *transaction,
  * the transaction failed. The string does not end in newline.
  */
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, struct strbuf *err,
-			   enum action_on_err onerr);
+			   const char *msg, struct strbuf *err);
 
 /** Lock a ref and then write its file */
 int update_ref(const char *action, const char *refname,
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status

[Ronnie Sahlberg]

Update ref_transaction_update() do some basic error checking and return
non-zero on error. Update all callers to check ref_transaction_update() for
error. There are currently no conditions in _update that will return error but
there will be in the future.

Also check for BUGs during update and die(BUG:...) if we are calling
_update with have_old but the old_sha1 pointer is NULL.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c | 10 ++++++----
 refs.c               |  9 +++++++--
 refs.h               | 13 ++++++++-----
 3 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 2bef2a0..59c4d6b 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)
 	if (*next != line_termination)
 		die("update %s: extra input: %s", refname, next);
 
-	ref_transaction_update(transaction, refname, new_sha1, old_sha1,
-			       update_flags, have_old);
+	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
+				   update_flags, have_old))
+		die("update %s: failed", refname);
 
 	update_flags = 0;
 	free(refname);
@@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)
 	if (*next != line_termination)
 		die("verify %s: extra input: %s", refname, next);
 
-	ref_transaction_update(transaction, refname, new_sha1, old_sha1,
-			       update_flags, have_old);
+	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
+				   update_flags, have_old))
+		die("failed transaction update for %s", refname);
 
 	update_flags = 0;
 	free(refname);
diff --git a/refs.c b/refs.c
index bc21060..e8c2345 100644
--- a/refs.c
+++ b/refs.c
@@ -3353,19 +3353,24 @@ static struct ref_update *add_update(struct ref_transaction *transaction,
 	return update;
 }
 
-void ref_transaction_update(struct ref_transaction *transaction,
+int ref_transaction_update(struct ref_transaction *transaction,
 			    const char *refname,
 			    const unsigned char *new_sha1,
 			    const unsigned char *old_sha1,
 			    int flags, int have_old)
 {
-	struct ref_update *update = add_update(transaction, refname);
+	struct ref_update *update;
+
+	if (have_old && !old_sha1)
+		die("BUG: have_old is true but old_sha1 is NULL");
 
+	update = add_update(transaction, refname);
 	hashcpy(update->new_sha1, new_sha1);
 	update->flags = flags;
 	update->have_old = have_old;
 	if (have_old)
 		hashcpy(update->old_sha1, old_sha1);
+	return 0;
 }
 
 void ref_transaction_create(struct ref_transaction *transaction,
diff --git a/refs.h b/refs.h
index 555ee59..fd8d731 100644
--- a/refs.h
+++ b/refs.h
@@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
  * be deleted.  If have_old is true, then old_sha1 holds the value
  * that the reference should have had before the update, or zeros if
  * it must not have existed beforehand.
+ * Function returns 0 on success and non-zero on failure. A failure to update
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
-void ref_transaction_update(struct ref_transaction *transaction,
-			    const char *refname,
-			    const unsigned char *new_sha1,
-			    const unsigned char *old_sha1,
-			    int flags, int have_old);
+int ref_transaction_update(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *new_sha1,
+			   const unsigned char *old_sha1,
+			   int flags, int have_old);
 
 /*
  * Add a reference creation to transaction.  new_sha1 is the value
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status

[Ronnie Sahlberg]

Do basic error checking in ref_transaction_create() and make it return
non-zero on error. Update all callers to check the result of
ref_transaction_create(). There are currently no conditions in _create that
will return error but there will be in the future.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c |  4 +++-
 refs.c               | 17 +++++++++++------
 refs.h               | 11 +++++++----
 3 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 59c4d6b..0924502 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -225,7 +225,9 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)
 	if (*next != line_termination)
 		die("create %s: extra input: %s", refname, next);
 
-	ref_transaction_create(transaction, refname, new_sha1, update_flags);
+	if (ref_transaction_create(transaction, refname, new_sha1,
+				   update_flags))
+		die("cannot create ref '%s'", refname);
 
 	update_flags = 0;
 	free(refname);
diff --git a/refs.c b/refs.c
index e8c2345..f38d3d6 100644
--- a/refs.c
+++ b/refs.c
@@ -3373,18 +3373,23 @@ int ref_transaction_update(struct ref_transaction *transaction,
 	return 0;
 }
 
-void ref_transaction_create(struct ref_transaction *transaction,
-			    const char *refname,
-			    const unsigned char *new_sha1,
-			    int flags)
+int ref_transaction_create(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *new_sha1,
+			   int flags)
 {
-	struct ref_update *update = add_update(transaction, refname);
+	struct ref_update *update;
+
+	if (!new_sha1 || is_null_sha1(new_sha1))
+		die("BUG: create ref with null new_sha1");
+
+	update = add_update(transaction, refname);
 
-	assert(!is_null_sha1(new_sha1));
 	hashcpy(update->new_sha1, new_sha1);
 	hashclr(update->old_sha1);
 	update->flags = flags;
 	update->have_old = 1;
+	return 0;
 }
 
 void ref_transaction_delete(struct ref_transaction *transaction,
diff --git a/refs.h b/refs.h
index fd8d731..e3bdd57 100644
--- a/refs.h
+++ b/refs.h
@@ -257,11 +257,14 @@ int ref_transaction_update(struct ref_transaction *transaction,
  * that the reference should have after the update; it must not be the
  * null SHA-1.  It is verified that the reference does not exist
  * already.
+ * Function returns 0 on success and non-zero on failure. A failure to create
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
-void ref_transaction_create(struct ref_transaction *transaction,
-			    const char *refname,
-			    const unsigned char *new_sha1,
-			    int flags);
+int ref_transaction_create(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *new_sha1,
+			   int flags);
 
 /*
  * Add a reference deletion to transaction.  If have_old is true, then
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status

[Ronnie Sahlberg]

Change ref_transaction_delete() to do basic error checking and return
non-zero of error. Update all callers to check the return for
ref_transaction_delete(). There are currently no conditions in _delete that
will return error but there will be in the future.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/update-ref.c |  5 +++--
 refs.c               | 15 ++++++++++-----
 refs.h               | 11 +++++++----
 3 files changed, 20 insertions(+), 11 deletions(-)

diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 0924502..2a51df1 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -257,8 +257,9 @@ static const char *parse_cmd_delete(struct strbuf *input, const char *next)
 	if (*next != line_termination)
 		die("delete %s: extra input: %s", refname, next);
 
-	ref_transaction_delete(transaction, refname, old_sha1,
-			       update_flags, have_old);
+	if (ref_transaction_delete(transaction, refname, old_sha1,
+				   update_flags, have_old))
+		die("failed transaction delete for %s", refname);
 
 	update_flags = 0;
 	free(refname);
diff --git a/refs.c b/refs.c
index f38d3d6..6e5e940 100644
--- a/refs.c
+++ b/refs.c
@@ -3392,19 +3392,24 @@ int ref_transaction_create(struct ref_transaction *transaction,
 	return 0;
 }
 
-void ref_transaction_delete(struct ref_transaction *transaction,
-			    const char *refname,
-			    const unsigned char *old_sha1,
-			    int flags, int have_old)
+int ref_transaction_delete(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *old_sha1,
+			   int flags, int have_old)
 {
-	struct ref_update *update = add_update(transaction, refname);
+	struct ref_update *update;
 
+	if (have_old && !old_sha1)
+		die("BUG: have_old is true but old_sha1 is NULL");
+
+	update = add_update(transaction, refname);
 	update->flags = flags;
 	update->have_old = have_old;
 	if (have_old) {
 		assert(!is_null_sha1(old_sha1));
 		hashcpy(update->old_sha1, old_sha1);
 	}
+	return 0;
 }
 
 int update_ref(const char *action, const char *refname,
diff --git a/refs.h b/refs.h
index e3bdd57..cba2f3d 100644
--- a/refs.h
+++ b/refs.h
@@ -270,11 +270,14 @@ int ref_transaction_create(struct ref_transaction *transaction,
  * Add a reference deletion to transaction.  If have_old is true, then
  * old_sha1 holds the value that the reference should have had before
  * the update (which must not be the null SHA-1).
+ * Function returns 0 on success and non-zero on failure. A failure to delete
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
-void ref_transaction_delete(struct ref_transaction *transaction,
-			    const char *refname,
-			    const unsigned char *old_sha1,
-			    int flags, int have_old);
+int ref_transaction_delete(struct ref_transaction *transaction,
+			   const char *refname,
+			   const unsigned char *old_sha1,
+			   int flags, int have_old);
 
 /*
  * Commit all of the changes that have been queued in transaction, as
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 13/44] tag.c: use ref transactions when doing updates

[Ronnie Sahlberg]

Change tag.c to use ref transactions for all ref updates.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/tag.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/builtin/tag.c b/builtin/tag.c
index c6e8a71..bf2a5c3 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -548,7 +548,6 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	struct strbuf ref = STRBUF_INIT;
 	unsigned char object[20], prev[20];
 	const char *object_ref, *tag;
-	struct ref_lock *lock;
 	struct create_tag_options opt;
 	char *cleanup_arg = NULL;
 	int annotate = 0, force = 0, lines = -1;
@@ -556,6 +555,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	const char *msgfile = NULL, *keyid = NULL;
 	struct msg_arg msg = { 0, STRBUF_INIT };
 	struct commit_list *with_commit = NULL;
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
 	struct option options[] = {
 		OPT_CMDMODE('l', "list", &cmdmode, N_("list tag names"), 'l'),
 		{ OPTION_INTEGER, 'n', NULL, &lines, N_("n"),
@@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	if (annotate)
 		create_tag(object, tag, &buf, &opt, prev, object);
 
-	lock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);
-	if (!lock)
-		die(_("%s: cannot lock the ref"), ref.buf);
-	if (write_ref_sha1(lock, object, NULL) < 0)
-		die(_("%s: cannot update the ref"), ref.buf);
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_update(transaction, ref.buf, object, prev,
+				   0, !is_null_sha1(prev)) ||
+	    ref_transaction_commit(transaction, NULL, &err))
+		die(_("%s: cannot update the ref: %s"), ref.buf, err.buf);
 	if (force && !is_null_sha1(prev) && hashcmp(prev, object))
 		printf(_("Updated tag '%s' (was %s)\n"), tag, find_unique_abbrev(prev, DEFAULT_ABBREV));
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 14/44] replace.c: use the ref transaction functions for updates

[Ronnie Sahlberg]

Update replace.c to use ref transactions for updates.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/replace.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)

diff --git a/builtin/replace.c b/builtin/replace.c
index 3da1bae..ee34d5d 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -133,7 +133,8 @@ static int replace_object_sha1(const char *object_ref,
 	unsigned char prev[20];
 	enum object_type obj_type, repl_type;
 	char ref[PATH_MAX];
-	struct ref_lock *lock;
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
 
 	if (snprintf(ref, sizeof(ref),
 		     "refs/replace/%s",
@@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,
 	else if (!force)
 		die("replace ref '%s' already exists", ref);
 
-	lock = lock_any_ref_for_update(ref, prev, 0, NULL);
-	if (!lock)
-		die("%s: cannot lock the ref", ref);
-	if (write_ref_sha1(lock, repl, NULL) < 0)
-		die("%s: cannot update the ref", ref);
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_update(transaction, ref, repl, prev,
+				   0, !is_null_sha1(prev)) ||
+	    ref_transaction_commit(transaction, NULL, &err))
+		die("%s: failed to replace ref: %s", ref, err.buf);
 
 	return 0;
 }
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 15/44] commit.c: use ref transactions for updates

[Ronnie Sahlberg]

Change commit.c to use ref transactions for all ref updates.
Make sure we pass a NULL pointer to ref_transaction_update if have_old
is false.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/commit.c | 23 ++++++++++-------------
 1 file changed, 10 insertions(+), 13 deletions(-)

diff --git a/builtin/commit.c b/builtin/commit.c
index d28505a..592f019 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1581,11 +1581,12 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 	const char *index_file, *reflog_msg;
 	char *nl;
 	unsigned char sha1[20];
-	struct ref_lock *ref_lock;
 	struct commit_list *parents = NULL, **pptr = &parents;
 	struct stat statbuf;
 	struct commit *current_head = NULL;
 	struct commit_extra_header *extra = NULL;
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
 
 	if (argc == 2 && !strcmp(argv[1], "-h"))
 		usage_with_options(builtin_commit_usage, builtin_commit_options);
@@ -1707,16 +1708,6 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 	strbuf_release(&author_ident);
 	free_commit_extra_headers(extra);
 
-	ref_lock = lock_any_ref_for_update("HEAD",
-					   !current_head
-					   ? NULL
-					   : current_head->object.sha1,
-					   0, NULL);
-	if (!ref_lock) {
-		rollback_index_files();
-		die(_("cannot lock HEAD ref"));
-	}
-
 	nl = strchr(sb.buf, '\n');
 	if (nl)
 		strbuf_setlen(&sb, nl + 1 - sb.buf);
@@ -1725,9 +1716,15 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 	strbuf_insert(&sb, 0, reflog_msg, strlen(reflog_msg));
 	strbuf_insert(&sb, strlen(reflog_msg), ": ", 2);
 
-	if (write_ref_sha1(ref_lock, sha1, sb.buf) < 0) {
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_update(transaction, "HEAD", sha1,
+				   current_head ?
+				   current_head->object.sha1 : NULL,
+				   0, !!current_head) ||
+	    ref_transaction_commit(transaction, sb.buf, &err)) {
 		rollback_index_files();
-		die(_("cannot update HEAD ref"));
+		die("%s", err.buf);
 	}
 
 	unlink(git_path("CHERRY_PICK_HEAD"));
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 16/44] sequencer.c: use ref transactions for all ref updates

[Ronnie Sahlberg]

Change to use ref transactions for all updates to refs.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 sequencer.c | 24 ++++++++++++++++--------
 1 file changed, 16 insertions(+), 8 deletions(-)

diff --git a/sequencer.c b/sequencer.c
index 0a80c58..9282a12 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -272,23 +272,31 @@ static int error_dirty_index(struct replay_opts *opts)
 static int fast_forward_to(const unsigned char *to, const unsigned char *from,
 			int unborn, struct replay_opts *opts)
 {
-	struct ref_lock *ref_lock;
+	struct ref_transaction *transaction;
 	struct strbuf sb = STRBUF_INIT;
-	int ret;
+	struct strbuf err = STRBUF_INIT;
 
 	read_cache();
 	if (checkout_fast_forward(from, to, 1))
 		exit(1); /* the callee should have complained already */
-	ref_lock = lock_any_ref_for_update("HEAD", unborn ? null_sha1 : from,
-					   0, NULL);
-	if (!ref_lock)
-		return error(_("Failed to lock HEAD during fast_forward_to"));
 
 	strbuf_addf(&sb, "%s: fast-forward", action_name(opts));
-	ret = write_ref_sha1(ref_lock, to, sb.buf);
+
+	transaction = ref_transaction_begin();
+	if ((!transaction ||
+	    ref_transaction_update(transaction, "HEAD", to, from,
+				   0, !unborn)) ||
+	    (ref_transaction_commit(transaction, sb.buf, &err) &&
+	     !(transaction = NULL))) {
+		ref_transaction_rollback(transaction);
+		error(_("HEAD: Could not fast-forward: %s\n"), err.buf);
+		strbuf_release(&sb);
+		strbuf_release(&err);
+		return -1;
+	}
 
 	strbuf_release(&sb);
-	return ret;
+	return 0;
 }
 
 static int do_recursive_merge(struct commit *base, struct commit *next,
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions

[Ronnie Sahlberg]

Change update_branch() to use ref transactions for updates.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 fast-import.c | 22 ++++++++++++++--------
 1 file changed, 14 insertions(+), 8 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index 6707a66..79d219b 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1679,39 +1679,45 @@ found_entry:
 static int update_branch(struct branch *b)
 {
 	static const char *msg = "fast-import";
-	struct ref_lock *lock;
+	struct ref_transaction *transaction;
 	unsigned char old_sha1[20];
+	struct strbuf err = STRBUF_INIT;
 
 	if (read_ref(b->name, old_sha1))
 		hashclr(old_sha1);
+
 	if (is_null_sha1(b->sha1)) {
 		if (b->delete)
 			delete_ref(b->name, old_sha1, 0);
 		return 0;
 	}
-	lock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);
-	if (!lock)
-		return error("Unable to lock %s", b->name);
 	if (!force_update && !is_null_sha1(old_sha1)) {
 		struct commit *old_cmit, *new_cmit;
 
 		old_cmit = lookup_commit_reference_gently(old_sha1, 0);
 		new_cmit = lookup_commit_reference_gently(b->sha1, 0);
 		if (!old_cmit || !new_cmit) {
-			unlock_ref(lock);
 			return error("Branch %s is missing commits.", b->name);
 		}
 
 		if (!in_merge_bases(old_cmit, new_cmit)) {
-			unlock_ref(lock);
 			warning("Not updating %s"
 				" (new tip %s does not contain %s)",
 				b->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));
 			return -1;
 		}
 	}
-	if (write_ref_sha1(lock, b->sha1, msg) < 0)
-		return error("Unable to update %s", b->name);
+	transaction = ref_transaction_begin();
+	if ((!transaction ||
+	    ref_transaction_update(transaction, b->name, b->sha1, old_sha1,
+				   0, 1)) ||
+	    (ref_transaction_commit(transaction, msg, &err) &&
+	     !(transaction = NULL))) {
+		ref_transaction_rollback(transaction);
+		error("Unable to update branch %s: %s", b->name, err.buf);
+		strbuf_release(&err);
+		return -1;
+	}
 	return 0;
 }
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 18/44] branch.c: use ref transaction for all ref updates

[Ronnie Sahlberg]

Change create_branch to use a ref transaction when creating the new branch.
ref_transaction_create will check that the ref does not already exist and fail
otherwise meaning that we no longer need to keep a lock on the ref during the
setup_tracking. This simplifies the code since we can now do the transaction
in one single step.

If the forcing flag is false then use ref_transaction_create since this will
fail if the ref already exist. Otherwise use ref_transaction_update.

This also fixes a race condition in the old code where two concurrent
create_branch could race since the lock_any_ref_for_update/write_ref_sha1
did not protect against the ref already existsing. I.e. one thread could end up
overwriting a branch even if the forcing flag is false.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 branch.c | 30 ++++++++++++++++--------------
 1 file changed, 16 insertions(+), 14 deletions(-)

diff --git a/branch.c b/branch.c
index 660097b..2aa5c82 100644
--- a/branch.c
+++ b/branch.c
@@ -226,7 +226,6 @@ void create_branch(const char *head,
 		   int force, int reflog, int clobber_head,
 		   int quiet, enum branch_track track)
 {
-	struct ref_lock *lock = NULL;
 	struct commit *commit;
 	unsigned char sha1[20];
 	char *real_ref, msg[PATH_MAX + 20];
@@ -285,15 +284,6 @@ void create_branch(const char *head,
 		die(_("Not a valid branch point: '%s'."), start_name);
 	hashcpy(sha1, commit->object.sha1);
 
-	if (!dont_change_ref) {
-		lock = lock_any_ref_for_update(ref.buf, NULL, 0, NULL);
-		if (!lock)
-			die_errno(_("Failed to lock ref for update"));
-	}
-
-	if (reflog)
-		log_all_ref_updates = 1;
-
 	if (forcing)
 		snprintf(msg, sizeof msg, "branch: Reset to %s",
 			 start_name);
@@ -301,13 +291,25 @@ void create_branch(const char *head,
 		snprintf(msg, sizeof msg, "branch: Created from %s",
 			 start_name);
 
+	if (reflog)
+		log_all_ref_updates = 1;
+
+	if (!dont_change_ref) {
+		struct ref_transaction *transaction;
+		struct strbuf err = STRBUF_INIT;
+
+		transaction = ref_transaction_begin();
+		if (!transaction ||
+			ref_transaction_update(transaction, ref.buf, sha1,
+					       null_sha1, 0, !forcing) ||
+			ref_transaction_commit(transaction, msg, &err))
+				die_errno(_("%s: failed to write ref: %s"),
+					  ref.buf, err.buf);
+	}
+
 	if (real_ref && track)
 		setup_tracking(ref.buf + 11, real_ref, track, quiet);
 
-	if (!dont_change_ref)
-		if (write_ref_sha1(lock, sha1, msg) < 0)
-			die_errno(_("Failed to write ref"));
-
 	strbuf_release(&ref);
 	free(real_ref);
 }
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 19/44] refs.c: change update_ref to use a transaction

[Ronnie Sahlberg]

Change the update_ref helper function to use a ref transaction internally.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/refs.c b/refs.c
index 6e5e940..0476892 100644
--- a/refs.c
+++ b/refs.c
@@ -3416,11 +3416,28 @@ int update_ref(const char *action, const char *refname,
 	       const unsigned char *sha1, const unsigned char *oldval,
 	       int flags, enum action_on_err onerr)
 {
-	struct ref_lock *lock;
-	lock = update_ref_lock(refname, oldval, flags, NULL, onerr);
-	if (!lock)
+	struct ref_transaction *t;
+	struct strbuf err = STRBUF_INIT;
+
+	t = ref_transaction_begin();
+	if ((!t ||
+	    ref_transaction_update(t, refname, sha1, oldval, flags,
+				   !!oldval)) ||
+	    (ref_transaction_commit(t, action, &err) && !(t = NULL))) {
+		const char *str = "update_ref failed for ref '%s': %s";
+
+		ref_transaction_rollback(t);
+		switch (onerr) {
+		case UPDATE_REFS_MSG_ON_ERR:
+			error(str, refname, err.buf); break;
+		case UPDATE_REFS_DIE_ON_ERR:
+			die(str, refname, err.buf); break;
+		case UPDATE_REFS_QUIET_ON_ERR: break;
+		}
+		strbuf_release(&err);
 		return 1;
-	return update_ref_write(action, refname, sha1, lock, NULL, onerr);
+	}
+	return 0;
 }
 
 static int ref_update_compare(const void *r1, const void *r2)
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 20/44] refs.c: free the transaction before returning when number of updates is 0

[Ronnie Sahlberg]

We have to free the transaction before returning in the early check for
'return early if number of updates == 0' or else the following code would
create a memory leak with the transaction never being freed :
   t = ref_transaction_begin()
   ref_transaction_commit(t)

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 0476892..4a37f87 100644
--- a/refs.c
+++ b/refs.c
@@ -3471,8 +3471,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	int n = transaction->nr;
 	struct ref_update **updates = transaction->updates;
 
-	if (!n)
+	if (!n) {
+		ref_transaction_free(transaction);
 		return 0;
+	}
 
 	/* Allocate work space */
 	delnames = xmalloc(sizeof(*delnames) * n);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction

[Ronnie Sahlberg]

Change ref_transaction_commit so that it does not free the transaction.
Instead require that a caller will end a transaction by either calling
ref_transaction_rollback or ref_transaction_free.

By having the transaction object remaining valid after _commit returns allows
us to write much nicer code and still be able to call ref_transaction_rollback
safely. Instead of this horribleness
	t = ref_transaction_begin();
	if ((!t ||
	    ref_transaction_update(t, refname, sha1, oldval, flags,
				   !!oldval)) ||
	    (ref_transaction_commit(t, action, &err) && !(t = NULL))) {
		ref_transaction_rollback(t);

we can now just do the much nicer
	t = ref_transaction_begin();
	if (!t ||
	    ref_transaction_update(t, refname, sha1, oldval, flags,
				   !!oldval) ||
	    ref_transaction_commit(&t, action, &err)) {
		ref_transaction_rollback(t);
		... die/return ...

	ref_transaction_free(transaction);

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 branch.c             |  1 +
 builtin/commit.c     |  1 +
 builtin/replace.c    |  1 +
 builtin/tag.c        |  1 +
 builtin/update-ref.c |  1 +
 fast-import.c        |  8 ++++----
 refs.c               | 14 ++++++--------
 refs.h               | 14 +++++++++-----
 sequencer.c          |  8 ++++----
 9 files changed, 28 insertions(+), 21 deletions(-)

diff --git a/branch.c b/branch.c
index 2aa5c82..8e58908 100644
--- a/branch.c
+++ b/branch.c
@@ -305,6 +305,7 @@ void create_branch(const char *head,
 			ref_transaction_commit(transaction, msg, &err))
 				die_errno(_("%s: failed to write ref: %s"),
 					  ref.buf, err.buf);
+		ref_transaction_free(transaction);
 	}
 
 	if (real_ref && track)
diff --git a/builtin/commit.c b/builtin/commit.c
index 592f019..16fadbb 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1726,6 +1726,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 		rollback_index_files();
 		die("%s", err.buf);
 	}
+	ref_transaction_free(transaction);
 
 	unlink(git_path("CHERRY_PICK_HEAD"));
 	unlink(git_path("REVERT_HEAD"));
diff --git a/builtin/replace.c b/builtin/replace.c
index ee34d5d..91354a7 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -164,6 +164,7 @@ static int replace_object_sha1(const char *object_ref,
 	    ref_transaction_commit(transaction, NULL, &err))
 		die("%s: failed to replace ref: %s", ref, err.buf);
 
+	ref_transaction_free(transaction);
 	return 0;
 }
 
diff --git a/builtin/tag.c b/builtin/tag.c
index bf2a5c3..fbd2989 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -708,6 +708,7 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 				   0, !is_null_sha1(prev)) ||
 	    ref_transaction_commit(transaction, NULL, &err))
 		die(_("%s: cannot update the ref: %s"), ref.buf, err.buf);
+	ref_transaction_free(transaction);
 	if (force && !is_null_sha1(prev) && hashcmp(prev, object))
 		printf(_("Updated tag '%s' (was %s)\n"), tag, find_unique_abbrev(prev, DEFAULT_ABBREV));
 
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index 2a51df1..b5f4731 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -373,6 +373,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 		update_refs_stdin();
 		if (ref_transaction_commit(transaction, msg, &err))
 			die("%s", err.buf);
+		ref_transaction_free(transaction);
 		return 0;
 	}
 
diff --git a/fast-import.c b/fast-import.c
index 79d219b..3e356da 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1708,16 +1708,16 @@ static int update_branch(struct branch *b)
 		}
 	}
 	transaction = ref_transaction_begin();
-	if ((!transaction ||
+	if (!transaction ||
 	    ref_transaction_update(transaction, b->name, b->sha1, old_sha1,
-				   0, 1)) ||
-	    (ref_transaction_commit(transaction, msg, &err) &&
-	     !(transaction = NULL))) {
+				   0, 1) ||
+	    ref_transaction_commit(transaction, msg, &err)) {
 		ref_transaction_rollback(transaction);
 		error("Unable to update branch %s: %s", b->name, err.buf);
 		strbuf_release(&err);
 		return -1;
 	}
+	ref_transaction_free(transaction);
 	return 0;
 }
 
diff --git a/refs.c b/refs.c
index 4a37f87..82a8d4e 100644
--- a/refs.c
+++ b/refs.c
@@ -3322,7 +3322,7 @@ struct ref_transaction *ref_transaction_begin(void)
 	return xcalloc(1, sizeof(struct ref_transaction));
 }
 
-static void ref_transaction_free(struct ref_transaction *transaction)
+void ref_transaction_free(struct ref_transaction *transaction)
 {
 	int i;
 
@@ -3420,10 +3420,10 @@ int update_ref(const char *action, const char *refname,
 	struct strbuf err = STRBUF_INIT;
 
 	t = ref_transaction_begin();
-	if ((!t ||
+	if (!t ||
 	    ref_transaction_update(t, refname, sha1, oldval, flags,
-				   !!oldval)) ||
-	    (ref_transaction_commit(t, action, &err) && !(t = NULL))) {
+				   !!oldval) ||
+	    ref_transaction_commit(t, action, &err)) {
 		const char *str = "update_ref failed for ref '%s': %s";
 
 		ref_transaction_rollback(t);
@@ -3437,6 +3437,7 @@ int update_ref(const char *action, const char *refname,
 		strbuf_release(&err);
 		return 1;
 	}
+	ref_transaction_free(t);
 	return 0;
 }
 
@@ -3471,10 +3472,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	int n = transaction->nr;
 	struct ref_update **updates = transaction->updates;
 
-	if (!n) {
-		ref_transaction_free(transaction);
+	if (!n)
 		return 0;
-	}
 
 	/* Allocate work space */
 	delnames = xmalloc(sizeof(*delnames) * n);
@@ -3541,7 +3540,6 @@ cleanup:
 		if (updates[i]->lock)
 			unlock_ref(updates[i]->lock);
 	free(delnames);
-	ref_transaction_free(transaction);
 	return ret;
 }
 
diff --git a/refs.h b/refs.h
index cba2f3d..b94e1ac 100644
--- a/refs.h
+++ b/refs.h
@@ -216,8 +216,8 @@ enum action_on_err {
 
 /*
  * Begin a reference transaction.  The reference transaction must
- * eventually be commited using ref_transaction_commit() or rolled
- * back using ref_transaction_rollback().
+ * eventually be freed by either calling ref_transaction_rollback()
+ * or ref_transaction_free().
  */
 struct ref_transaction *ref_transaction_begin(void);
 
@@ -282,13 +282,17 @@ int ref_transaction_delete(struct ref_transaction *transaction,
 /*
  * Commit all of the changes that have been queued in transaction, as
  * atomically as possible.  Return a nonzero value if there is a
- * problem.  The ref_transaction is freed by this function.
- * If err is non-NULL we will add an error string to it to explain why
- * the transaction failed. The string does not end in newline.
+ * problem. If err is non-NULL we will add an error string to it to explain
+ * why the transaction failed. The string does not end in newline.
  */
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   const char *msg, struct strbuf *err);
 
+/*
+ * Free an existing transaction.
+ */
+void ref_transaction_free(struct ref_transaction *transaction);
+
 /** Lock a ref and then write its file */
 int update_ref(const char *action, const char *refname,
 		const unsigned char *sha1, const unsigned char *oldval,
diff --git a/sequencer.c b/sequencer.c
index 9282a12..0cfdaf0 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -283,17 +283,17 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,
 	strbuf_addf(&sb, "%s: fast-forward", action_name(opts));
 
 	transaction = ref_transaction_begin();
-	if ((!transaction ||
+	if (!transaction ||
 	    ref_transaction_update(transaction, "HEAD", to, from,
-				   0, !unborn)) ||
-	    (ref_transaction_commit(transaction, sb.buf, &err) &&
-	     !(transaction = NULL))) {
+				   0, !unborn) ||
+	    ref_transaction_commit(transaction, sb.buf, &err)) {
 		ref_transaction_rollback(transaction);
 		error(_("HEAD: Could not fast-forward: %s\n"), err.buf);
 		strbuf_release(&sb);
 		strbuf_release(&err);
 		return -1;
 	}
+	ref_transaction_free(transaction);
 
 	strbuf_release(&sb);
 	return 0;
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it

[Ronnie Sahlberg]

In s_update_ref there are two calls that when they fail we return an error
based on the errno value. In particular we want to return a specific error
if ENOTDIR happened. Both these functions do have failure modes where they
may return an error without updating errno, in which case a previous and
unrelated ENOTDIR may cause us to return the wrong error. Clear errno before
calling any functions if we check errno afterwards.

Also skip initializing a static variable to 0. Statics live in .bss and
are all automatically initialized to 0.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/fetch.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 55f457c..a93c893 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -44,7 +44,7 @@ static struct transport *gtransport;
 static struct transport *gsecondary;
 static const char *submodule_prefix = "";
 static const char *recurse_submodules_default;
-static int shown_url = 0;
+static int shown_url;
 
 static int option_parse_recurse_submodules(const struct option *opt,
 				   const char *arg, int unset)
@@ -382,6 +382,8 @@ static int s_update_ref(const char *action,
 	if (!rla)
 		rla = default_rla.buf;
 	snprintf(msg, sizeof(msg), "%s: %s", rla, action);
+
+	errno = 0;
 	lock = lock_any_ref_for_update(ref->name,
 				       check_old ? ref->old_sha1 : NULL,
 				       0, NULL);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction

[Ronnie Sahlberg]

Change s_update_ref to use a ref transaction for the ref update.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/fetch.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index a93c893..b41d7b7 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -375,7 +375,7 @@ static int s_update_ref(const char *action,
 {
 	char msg[1024];
 	char *rla = getenv("GIT_REFLOG_ACTION");
-	static struct ref_lock *lock;
+	struct ref_transaction *transaction;
 
 	if (dry_run)
 		return 0;
@@ -384,15 +384,16 @@ static int s_update_ref(const char *action,
 	snprintf(msg, sizeof(msg), "%s: %s", rla, action);
 
 	errno = 0;
-	lock = lock_any_ref_for_update(ref->name,
-				       check_old ? ref->old_sha1 : NULL,
-				       0, NULL);
-	if (!lock)
-		return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
-					  STORE_REF_ERROR_OTHER;
-	if (write_ref_sha1(lock, ref->new_sha1, msg) < 0)
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_update(transaction, ref->name, ref->new_sha1,
+				   ref->old_sha1, 0, check_old) ||
+	    ref_transaction_commit(transaction, msg, NULL)) {
+		ref_transaction_rollback(transaction);
 		return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
 					  STORE_REF_ERROR_OTHER;
+	}
+	ref_transaction_free(transaction);
 	return 0;
 }
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates

[Ronnie Sahlberg]

Change store_updated_refs to use a single ref transaction for all refs that
are updated during the fetch. This makes the fetch more atomic when update
failures occur.

Since ref update failures will now no longer occur in the code path for
updating a single ref in s_update_ref, we no longer have as detailed error
message logging the exact reference and the ref log action as in the old cod
Instead since we fail the entire transaction we log a much more generic
message. But since we commit the transaction using MSG_ON_ERR we will log
an error containing the ref name if either locking of writing the ref would
so the regression in the log message is minor.

This will also change the order in which errors are checked for and logged
which may alter which error will be logged if there are multiple errors
occuring during a fetch.

For example, assume we have a fetch for two refs that both would fail.
Where the first ref would fail with ENOTDIR due to a directory in the ref
path not existing, and the second ref in the fetch would fail due to
the check in update_logical_ref():
        if (current_branch &&
            !strcmp(ref->name, current_branch->name) &&
            !(update_head_ok || is_bare_repository()) &&
            !is_null_sha1(ref->old_sha1)) {
                /*
                 * If this is the head, and it's not okay to update
                 * the head, and the old value of the head isn't empty...
                 */

In the old code since we would update the refs one ref at a time we would
first fail the ENOTDIR and then fail the second update of HEAD as well.
But since the first ref failed with ENOTDIR we would eventually fail the who
fetch with STORE_REF_ERROR_DF_CONFLICT

In the new code, since we defer committing the transaction until all refs
have been processed, we would now detect that the second ref was bad and
rollback the transaction before we would even try start writing the update t
disk and thus we would not return STORE_REF_ERROR_DF_CONFLICT for this case.

I think this new behaviour is more correct, since if there was a problem
we would not even try to commit the transaction but need to highlight this
change in how/what errors are reported.
This change in what error is returned only occurs if there are multiple
refs that fail to update and only some, but not all, of them fail due to
ENOTDIR.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/fetch.c | 34 ++++++++++++++++------------------
 1 file changed, 16 insertions(+), 18 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index b41d7b7..5dbd0f0 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -45,6 +45,7 @@ static struct transport *gsecondary;
 static const char *submodule_prefix = "";
 static const char *recurse_submodules_default;
 static int shown_url;
+struct ref_transaction *transaction;
 
 static int option_parse_recurse_submodules(const struct option *opt,
 				   const char *arg, int unset)
@@ -373,27 +374,13 @@ static int s_update_ref(const char *action,
 			struct ref *ref,
 			int check_old)
 {
-	char msg[1024];
-	char *rla = getenv("GIT_REFLOG_ACTION");
-	struct ref_transaction *transaction;
-
 	if (dry_run)
 		return 0;
-	if (!rla)
-		rla = default_rla.buf;
-	snprintf(msg, sizeof(msg), "%s: %s", rla, action);
 
-	errno = 0;
-	transaction = ref_transaction_begin();
-	if (!transaction ||
-	    ref_transaction_update(transaction, ref->name, ref->new_sha1,
-				   ref->old_sha1, 0, check_old) ||
-	    ref_transaction_commit(transaction, msg, NULL)) {
-		ref_transaction_rollback(transaction);
-		return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
-					  STORE_REF_ERROR_OTHER;
-	}
-	ref_transaction_free(transaction);
+	if (ref_transaction_update(transaction, ref->name, ref->new_sha1,
+			       ref->old_sha1, 0, check_old))
+		return STORE_REF_ERROR_OTHER;
+
 	return 0;
 }
 
@@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 		goto abort;
 	}
 
+	errno = 0;
+	transaction = ref_transaction_begin();
+	if (!transaction) {
+		rc = error(_("cannot start ref transaction\n"));
+		goto abort;
+	}
+
 	/*
 	 * We do a pass for each fetch_head_status type in their enum order, so
 	 * merged entries are written before not-for-merge. That lets readers
@@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 			}
 		}
 	}
+	if (ref_transaction_commit(transaction, "fetch_ref transaction", NULL))
+		rc |= errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
+		  STORE_REF_ERROR_OTHER;
+	ref_transaction_free(transaction);
 
 	if (rc & STORE_REF_ERROR_DF_CONFLICT)
 		error(_("some local refs could not be updated; try running\n"
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs

[Ronnie Sahlberg]

Wrap all the ref updates inside a transaction to make the update atomic.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 builtin/receive-pack.c | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index c323081..d580176 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -46,6 +46,8 @@ static void *head_name_to_free;
 static int sent_capabilities;
 static int shallow_update;
 static const char *alt_shallow_file;
+struct strbuf err = STRBUF_INIT;
+struct ref_transaction *transaction;
 
 static enum deny_action parse_deny_action(const char *var, const char *value)
 {
@@ -475,7 +477,6 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 	const char *namespaced_name;
 	unsigned char *old_sha1 = cmd->old_sha1;
 	unsigned char *new_sha1 = cmd->new_sha1;
-	struct ref_lock *lock;
 
 	/* only refs/... are allowed */
 	if (!starts_with(name, "refs/") || check_refname_format(name + 5, 0)) {
@@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 		    update_shallow_ref(cmd, si))
 			return "shallow error";
 
-		lock = lock_any_ref_for_update(namespaced_name, old_sha1,
-					       0, NULL);
-		if (!lock) {
-			rp_error("failed to lock %s", name);
-			return "failed to lock";
-		}
-		if (write_ref_sha1(lock, new_sha1, "push")) {
-			return "failed to write"; /* error() already called */
-		}
+		if (ref_transaction_update(transaction, namespaced_name,
+					   new_sha1, old_sha1, 0, 1))
+			return "failed to update";
 		return NULL; /* good */
 	}
 }
@@ -812,6 +807,7 @@ static void execute_commands(struct command *commands,
 	head_name = head_name_to_free = resolve_refdup("HEAD", sha1, 0, NULL);
 
 	checked_connectivity = 1;
+	transaction = ref_transaction_begin();
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		if (cmd->error_string)
 			continue;
@@ -827,6 +823,10 @@ static void execute_commands(struct command *commands,
 			checked_connectivity = 0;
 		}
 	}
+	if (ref_transaction_commit(transaction, "push", &err))
+		error("%s", err.buf);
+	ref_transaction_free(transaction);
+	strbuf_release(&err);
 
 	if (shallow_update && !checked_connectivity)
 		error("BUG: run 'git fsck' for safety.\n"
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags

[Ronnie Sahlberg]

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 fast-import.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/fast-import.c b/fast-import.c
index 3e356da..5587cf6 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1736,15 +1736,22 @@ static void dump_tags(void)
 {
 	static const char *msg = "fast-import";
 	struct tag *t;
-	struct ref_lock *lock;
 	char ref_name[PATH_MAX];
+	struct strbuf err = STRBUF_INIT;
+	struct ref_transaction *transaction;
 
+	transaction = ref_transaction_begin();
 	for (t = first_tag; t; t = t->next_tag) {
-		sprintf(ref_name, "tags/%s", t->name);
-		lock = lock_ref_sha1(ref_name, NULL);
-		if (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)
-			failure |= error("Unable to update %s", ref_name);
+		sprintf(ref_name, "refs/tags/%s", t->name);
+
+		if (ref_transaction_update(transaction, ref_name, t->sha1,
+					   NULL, 0, 0))
+			failure |= error("Unable to update %s", err.buf);
 	}
+	if (failure || ref_transaction_commit(transaction, msg, &err))
+		failure |= error("Unable to update %s", err.buf);
+	ref_transaction_free(transaction);
+	strbuf_release(&err);
 }
 
 static void dump_marks_helper(FILE *f,
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 27/44] walker.c: use ref transaction for ref updates

[Ronnie Sahlberg]

Switch to using ref transactions in walker_fetch(). As part of the refactoring
to use ref transactions we also fix a potential memory leak where in the
original code if write_ref_sha1() would fail we would end up returning from
the function without free()ing the msg string.

This changes the locking slightly for walker_fetch. Previously the code would
lock all refs before writing them but now we do not lock the refs until the
commit stage. There is thus a very short window where changes could be done
locally during the fetch which would be overwritten when the fetch completes
and commits its transaction. But this window should be reasonably short.
Even if this race does trigger, since both the old code and the new code
just overwrites the refs to the new values without checking or comparing
them with the previous value, this is not too dissimilar to a similar scenario
where you first do a ref change locally and then later do a fetch that
overwrites the local change. With this in mind I do not see the change in
locking semantics to be critical.

Note that this function is only called when fetching from a remote HTTP
repository onto the local (most of the time single-user) repository which
likely means that the type of collissions that the previous locking would
protect against and cause the fetch to fail for to be even more rare.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 walker.c | 51 ++++++++++++++++++++++++++-------------------------
 1 file changed, 26 insertions(+), 25 deletions(-)

diff --git a/walker.c b/walker.c
index 1dd86b8..6044ccf 100644
--- a/walker.c
+++ b/walker.c
@@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)
 int walker_fetch(struct walker *walker, int targets, char **target,
 		 const char **write_ref, const char *write_ref_log_details)
 {
-	struct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));
+	char ref_name[PATH_MAX];
+	struct strbuf err = STRBUF_INIT;
+	struct ref_transaction *transaction;
 	unsigned char *sha1 = xmalloc(targets * 20);
-	char *msg;
-	int ret;
+	char *msg = NULL;
 	int i;
 
 	save_commit_buffer = 0;
 
-	for (i = 0; i < targets; i++) {
-		if (!write_ref || !write_ref[i])
-			continue;
-
-		lock[i] = lock_ref_sha1(write_ref[i], NULL);
-		if (!lock[i]) {
-			error("Can't lock ref %s", write_ref[i]);
-			goto unlock_and_fail;
-		}
-	}
+	transaction = ref_transaction_begin();
+	if (!transaction)
+		return -1;
 
 	if (!walker->get_recover)
 		for_each_ref(mark_complete, NULL);
@@ -276,14 +270,14 @@ int walker_fetch(struct walker *walker, int targets, char **target,
 	for (i = 0; i < targets; i++) {
 		if (interpret_target(walker, target[i], &sha1[20 * i])) {
 			error("Could not interpret response from server '%s' as something to pull", target[i]);
-			goto unlock_and_fail;
+			goto rollback_and_fail;
 		}
 		if (process(walker, lookup_unknown_object(&sha1[20 * i])))
-			goto unlock_and_fail;
+			goto rollback_and_fail;
 	}
 
 	if (loop(walker))
-		goto unlock_and_fail;
+		goto rollback_and_fail;
 
 	if (write_ref_log_details) {
 		msg = xmalloc(strlen(write_ref_log_details) + 12);
@@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,
 	for (i = 0; i < targets; i++) {
 		if (!write_ref || !write_ref[i])
 			continue;
-		ret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : "fetch (unknown)");
-		lock[i] = NULL;
-		if (ret)
-			goto unlock_and_fail;
+		sprintf(ref_name, "refs/%s", write_ref[i]);
+		if (ref_transaction_update(transaction, ref_name,
+					   &sha1[20 * i], NULL,
+					   0, 0))
+			goto rollback_and_fail;
+	}
+
+	if (ref_transaction_commit(transaction, msg ? msg : "fetch (unknown)",
+				   &err)) {
+		error("%s", err.buf);
+		goto rollback_and_fail;
 	}
-	free(msg);
 
+	free(msg);
 	return 0;
 
-unlock_and_fail:
-	for (i = 0; i < targets; i++)
-		if (lock[i])
-			unlock_ref(lock[i]);
+rollback_and_fail:
+	free(msg);
+	strbuf_release(&err);
+	ref_transaction_free(transaction);
 
 	return -1;
 }
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 28/44] refs.c: make write_ref_sha1 static

[Ronnie Sahlberg]

No external users call write_ref_sha1 any more so lets declare it static.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 4 +++-
 refs.h | 3 ---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/refs.c b/refs.c
index 82a8d4e..e5729ad 100644
--- a/refs.c
+++ b/refs.c
@@ -251,6 +251,8 @@ struct ref_entry {
 };
 
 static void read_loose_refs(const char *dirname, struct ref_dir *dir);
+static int write_ref_sha1(struct ref_lock *lock,
+			  const unsigned char *sha1, const char *logmsg);
 
 static struct ref_dir *get_ref_dir(struct ref_entry *entry)
 {
@@ -2805,7 +2807,7 @@ static int is_branch(const char *refname)
 	return !strcmp(refname, "HEAD") || starts_with(refname, "refs/heads/");
 }
 
-int write_ref_sha1(struct ref_lock *lock,
+static int write_ref_sha1(struct ref_lock *lock,
 	const unsigned char *sha1, const char *logmsg)
 {
 	static char term = '\n';
diff --git a/refs.h b/refs.h
index b94e1ac..796e396 100644
--- a/refs.h
+++ b/refs.h
@@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);
 /** Release any lock taken but not written. **/
 extern void unlock_ref(struct ref_lock *lock);
 
-/** Writes sha1 into the ref specified by the lock. **/
-extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);
-
 /** Setup reflog before using. **/
 int log_ref_setup(const char *refname, char *logfile, int bufsize);
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 29/44] refs.c: make lock_ref_sha1 static

[Ronnie Sahlberg]

No external callers reference lock_ref_sha1 any more so lets declare it static.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 2 +-
 refs.h | 3 ---
 2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/refs.c b/refs.c
index e5729ad..600f9b3 100644
--- a/refs.c
+++ b/refs.c
@@ -2126,7 +2126,7 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	return NULL;
 }
 
-struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)
+static struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)
 {
 	char refpath[PATH_MAX];
 	if (check_refname_format(refname, 0))
diff --git a/refs.h b/refs.h
index 796e396..221632c 100644
--- a/refs.h
+++ b/refs.h
@@ -132,9 +132,6 @@ extern int ref_exists(const char *);
  */
 extern int peel_ref(const char *refname, unsigned char *sha1);
 
-/** Locks a "refs/" ref returning the lock on success and NULL on failure. **/
-extern struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1);
-
 /** Locks any ref (for 'HEAD' type refs). */
 #define REF_NODEREF	0x01
 extern struct ref_lock *lock_any_ref_for_update(const char *refname,
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR

[Ronnie Sahlberg]

Track the status of a transaction in a new status field. Check the field for
sanity, i.e. that status must be OPEN when _commit/_create/_delete or
_update is called or else die(BUG:...)

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 600f9b3..f11f832 100644
--- a/refs.c
+++ b/refs.c
@@ -3308,6 +3308,12 @@ struct ref_update {
 	const char refname[FLEX_ARRAY];
 };
 
+enum ref_transaction_status {
+	REF_TRANSACTION_OPEN   = 0,
+	REF_TRANSACTION_CLOSED = 1,
+	REF_TRANSACTION_ERROR  = 2,
+};
+
 /*
  * Data structure for holding a reference transaction, which can
  * consist of checks and updates to multiple references, carried out
@@ -3317,6 +3323,7 @@ struct ref_transaction {
 	struct ref_update **updates;
 	size_t alloc;
 	size_t nr;
+	enum ref_transaction_status status;
 };
 
 struct ref_transaction *ref_transaction_begin(void)
@@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)
 
 void ref_transaction_rollback(struct ref_transaction *transaction)
 {
+	if (!transaction)
+		return;
+
+	transaction->status = REF_TRANSACTION_ERROR;
+
 	ref_transaction_free(transaction);
 }
 
@@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
 	if (have_old && !old_sha1)
 		die("BUG: have_old is true but old_sha1 is NULL");
 
+	if (transaction->status != REF_TRANSACTION_OPEN)
+		die("BUG: update on transaction that is not open");
+
 	update = add_update(transaction, refname);
 	hashcpy(update->new_sha1, new_sha1);
 	update->flags = flags;
@@ -3385,6 +3400,9 @@ int ref_transaction_create(struct ref_transaction *transaction,
 	if (!new_sha1 || is_null_sha1(new_sha1))
 		die("BUG: create ref with null new_sha1");
 
+	if (transaction->status != REF_TRANSACTION_OPEN)
+		die("BUG: create on transaction that is not open");
+
 	update = add_update(transaction, refname);
 
 	hashcpy(update->new_sha1, new_sha1);
@@ -3404,6 +3422,9 @@ int ref_transaction_delete(struct ref_transaction *transaction,
 	if (have_old && !old_sha1)
 		die("BUG: have_old is true but old_sha1 is NULL");
 
+	if (transaction->status != REF_TRANSACTION_OPEN)
+		die("BUG: delete on transaction that is not open");
+
 	update = add_update(transaction, refname);
 	update->flags = flags;
 	update->have_old = have_old;
@@ -3474,8 +3495,13 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	int n = transaction->nr;
 	struct ref_update **updates = transaction->updates;
 
-	if (!n)
+	if (transaction->status != REF_TRANSACTION_OPEN)
+		die("BUG: commit on transaction that is not open");
+
+	if (!n) {
+		transaction->status = REF_TRANSACTION_CLOSED;
 		return 0;
+	}
 
 	/* Allocate work space */
 	delnames = xmalloc(sizeof(*delnames) * n);
@@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	clear_loose_ref_cache(&ref_cache);
 
 cleanup:
+	transaction->status = ret ? REF_TRANSACTION_ERROR
+	  : REF_TRANSACTION_CLOSED;
+
 	for (i = 0; i < n; i++)
 		if (updates[i]->lock)
 			unlock_ref(updates[i]->lock);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 31/44] refs.c: remove the update_ref_lock function

[Ronnie Sahlberg]

Since we now only call update_ref_lock with onerr==QUIET_ON_ERR we no longer
need this function and can replace it with just calling lock_any_ref_for_update
directly.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 30 ++++++------------------------
 1 file changed, 6 insertions(+), 24 deletions(-)

diff --git a/refs.c b/refs.c
index f11f832..07377ad 100644
--- a/refs.c
+++ b/refs.c
@@ -3255,24 +3255,6 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)
 	return retval;
 }
 
-static struct ref_lock *update_ref_lock(const char *refname,
-					const unsigned char *oldval,
-					int flags, int *type_p,
-					enum action_on_err onerr)
-{
-	struct ref_lock *lock;
-	lock = lock_any_ref_for_update(refname, oldval, flags, type_p);
-	if (!lock) {
-		const char *str = "Cannot lock the ref '%s'.";
-		switch (onerr) {
-		case UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;
-		case UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;
-		case UPDATE_REFS_QUIET_ON_ERR: break;
-		}
-	}
-	return lock;
-}
-
 static int update_ref_write(const char *action, const char *refname,
 			    const unsigned char *sha1, struct ref_lock *lock,
 			    struct strbuf *err, enum action_on_err onerr)
@@ -3516,12 +3498,12 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
 
-		update->lock = update_ref_lock(update->refname,
-					       (update->have_old ?
-						update->old_sha1 : NULL),
-					       update->flags,
-					       &update->type,
-					       UPDATE_REFS_QUIET_ON_ERR);
+		update->lock = lock_any_ref_for_update(update->refname,
+						       (update->have_old ?
+							update->old_sha1 :
+							NULL),
+						       update->flags,
+						       &update->type);
 		if (!update->lock) {
 			if (err)
 				strbuf_addf(err, "Cannot lock the ref '%s'.",
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 32/44] refs.c: remove the update_ref_write function

[Ronnie Sahlberg]

Since we only call update_ref_write from a single place and we only call it
with onerr==QUIET_ON_ERR we can just as well get rid of it and just call
write_ref_sha1 directly.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 35 +++++++++--------------------------
 1 file changed, 9 insertions(+), 26 deletions(-)

diff --git a/refs.c b/refs.c
index 07377ad..7aa5512 100644
--- a/refs.c
+++ b/refs.c
@@ -3255,25 +3255,6 @@ int for_each_reflog(each_ref_fn fn, void *cb_data)
 	return retval;
 }
 
-static int update_ref_write(const char *action, const char *refname,
-			    const unsigned char *sha1, struct ref_lock *lock,
-			    struct strbuf *err, enum action_on_err onerr)
-{
-	if (write_ref_sha1(lock, sha1, action) < 0) {
-		const char *str = "Cannot update the ref '%s'.";
-		if (err)
-			strbuf_addf(err, str, refname);
-
-		switch (onerr) {
-		case UPDATE_REFS_MSG_ON_ERR: error(str, refname); break;
-		case UPDATE_REFS_DIE_ON_ERR: die(str, refname); break;
-		case UPDATE_REFS_QUIET_ON_ERR: break;
-		}
-		return 1;
-	}
-	return 0;
-}
-
 /**
  * Information needed for a single ref update.  Set new_sha1 to the
  * new value or to zero to delete the ref.  To check the old value
@@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 		struct ref_update *update = updates[i];
 
 		if (!is_null_sha1(update->new_sha1)) {
-			ret = update_ref_write(msg,
-					       update->refname,
-					       update->new_sha1,
-					       update->lock, err,
-					       UPDATE_REFS_QUIET_ON_ERR);
-			update->lock = NULL; /* freed by update_ref_write */
-			if (ret)
+			ret = write_ref_sha1(update->lock, update->new_sha1,
+					     msg);
+			update->lock = NULL; /* freed by write_ref_sha1 */
+			if (ret) {
+				const char *str = "Cannot update the ref '%s'.";
+
+				if (err)
+					strbuf_addf(err, str, update->refname);
 				goto cleanup;
+			}
 		}
 	}
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 33/44] refs.c: remove lock_ref_sha1

[Ronnie Sahlberg]

lock_ref_sha1 was only called from one place in refc.c and only provided
a check that the refname was sane before adding back the initial "refs/"
part of the ref path name, the initial "refs/" that this caller had already
stripped off before calling lock_ref_sha1.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/refs.c b/refs.c
index 7aa5512..f1ef940 100644
--- a/refs.c
+++ b/refs.c
@@ -2126,15 +2126,6 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	return NULL;
 }
 
-static struct ref_lock *lock_ref_sha1(const char *refname, const unsigned char *old_sha1)
-{
-	char refpath[PATH_MAX];
-	if (check_refname_format(refname, 0))
-		return NULL;
-	strcpy(refpath, mkpath("refs/%s", refname));
-	return lock_ref_sha1_basic(refpath, old_sha1, 0, NULL);
-}
-
 struct ref_lock *lock_any_ref_for_update(const char *refname,
 					 const unsigned char *old_sha1,
 					 int flags, int *type_p)
@@ -2335,8 +2326,12 @@ static void try_remove_empty_parents(char *name)
 /* make sure nobody touched the ref, and unlink */
 static void prune_ref(struct ref_to_prune *r)
 {
-	struct ref_lock *lock = lock_ref_sha1(r->name + 5, r->sha1);
+	struct ref_lock *lock;
+
+	if (check_refname_format(r->name + 5, 0))
+		return;
 
+	lock = lock_ref_sha1_basic(r->name, r->sha1, 0, NULL);
 	if (lock) {
 		unlink_or_warn(git_path("%s", r->name));
 		unlock_ref(lock);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref

[Ronnie Sahlberg]

Change prune_ref to delete the ref using a ref transaction. To do this we also
need to add a new flag REF_ISPRUNING that will tell the transaction that we
do not want to delete this ref from the packed refs. This flag is private to
refs.c and not exposed to external callers.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/refs.c b/refs.c
index f1ef940..88ab012 100644
--- a/refs.c
+++ b/refs.c
@@ -29,6 +29,11 @@ static inline int bad_ref_char(int ch)
 	return 0;
 }
 
+/** Used as a flag to ref_transaction_delete when a loose ref is beeing
+ *  pruned.
+ */
+#define REF_ISPRUNING	0x0100
+
 /*
  * Try to read one refname component from the front of refname.  Return
  * the length of the component found, or -1 if the component is not
@@ -2326,17 +2331,24 @@ static void try_remove_empty_parents(char *name)
 /* make sure nobody touched the ref, and unlink */
 static void prune_ref(struct ref_to_prune *r)
 {
-	struct ref_lock *lock;
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
 
 	if (check_refname_format(r->name + 5, 0))
 		return;
 
-	lock = lock_ref_sha1_basic(r->name, r->sha1, 0, NULL);
-	if (lock) {
-		unlink_or_warn(git_path("%s", r->name));
-		unlock_ref(lock);
-		try_remove_empty_parents(r->name);
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_delete(transaction, r->name, r->sha1,
+				   REF_ISPRUNING, 1) ||
+	    ref_transaction_commit(transaction, NULL, &err)) {
+		ref_transaction_rollback(transaction);
+		warning("prune_ref: %s", err.buf);
+		strbuf_release(&err);
+		return;
 	}
+	ref_transaction_free(transaction);
+	try_remove_empty_parents(r->name);
 }
 
 static void prune_refs(struct ref_to_prune *r)
@@ -3512,9 +3524,10 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 		struct ref_update *update = updates[i];
 
 		if (update->lock) {
-			delnames[delnum++] = update->lock->ref_name;
 			ret |= delete_ref_loose(update->lock, update->type,
 						err);
+			if (!(update->flags & REF_ISPRUNING))
+				delnames[delnum++] = update->lock->ref_name;
 		}
 	}
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 35/44] refs.c: make delete_ref use a transaction

[Ronnie Sahlberg]

Change delete_ref to use a ref transaction for the deletion. At the same time
since we no longer have any callers of repack_without_ref we can now delete
this function.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 31 ++++++++++---------------------
 1 file changed, 10 insertions(+), 21 deletions(-)

diff --git a/refs.c b/refs.c
index 88ab012..e7bd95d 100644
--- a/refs.c
+++ b/refs.c
@@ -2488,11 +2488,6 @@ static int repack_without_refs(const char **refnames, int n, struct strbuf *err)
 	return ret;
 }
 
-static int repack_without_ref(const char *refname)
-{
-	return repack_without_refs(&refname, 1, NULL);
-}
-
 static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 {
 	if (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {
@@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 
 int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
 {
-	struct ref_lock *lock;
-	int ret = 0, flag = 0;
+	struct ref_transaction *transaction;
 
-	lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);
-	if (!lock)
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_delete(transaction, refname, sha1, delopt,
+				   sha1 && !is_null_sha1(sha1)) ||
+	    ref_transaction_commit(transaction, NULL, NULL)) {
+		ref_transaction_rollback(transaction);
 		return 1;
-	ret |= delete_ref_loose(lock, flag, NULL);
-
-	/* removing the loose one could have resurrected an earlier
-	 * packed one.  Also, if it was not loose we need to repack
-	 * without it.
-	 */
-	ret |= repack_without_ref(lock->ref_name);
-
-	unlink_or_warn(git_path("logs/%s", lock->ref_name));
-	clear_loose_ref_cache(&ref_cache);
-	unlock_ref(lock);
-	return ret;
+	}
+	ref_transaction_free(transaction);
+	return 0;
 }
 
 /*
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit

[Ronnie Sahlberg]

Change the reference transactions so that we pass the reflog message
through to the create/delete/update function instead of the commit message.
This allows for individual messages for each change in a multi ref
transaction.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 branch.c               |  6 +++---
 builtin/commit.c       |  4 ++--
 builtin/fetch.c        | 10 ++++++++--
 builtin/receive-pack.c |  4 ++--
 builtin/replace.c      |  4 ++--
 builtin/tag.c          |  4 ++--
 builtin/update-ref.c   | 13 +++++++------
 fast-import.c          |  8 ++++----
 refs.c                 | 34 +++++++++++++++++++++-------------
 refs.h                 |  8 ++++----
 sequencer.c            |  4 ++--
 walker.c               |  6 +++---
 12 files changed, 60 insertions(+), 45 deletions(-)

diff --git a/branch.c b/branch.c
index 8e58908..74d55e7 100644
--- a/branch.c
+++ b/branch.c
@@ -300,9 +300,9 @@ void create_branch(const char *head,
 
 		transaction = ref_transaction_begin();
 		if (!transaction ||
-			ref_transaction_update(transaction, ref.buf, sha1,
-					       null_sha1, 0, !forcing) ||
-			ref_transaction_commit(transaction, msg, &err))
+		    ref_transaction_update(transaction, ref.buf, sha1,
+					   null_sha1, 0, !forcing, msg) ||
+		    ref_transaction_commit(transaction, &err))
 				die_errno(_("%s: failed to write ref: %s"),
 					  ref.buf, err.buf);
 		ref_transaction_free(transaction);
diff --git a/builtin/commit.c b/builtin/commit.c
index 16fadbb..799a59f 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1721,8 +1721,8 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 	    ref_transaction_update(transaction, "HEAD", sha1,
 				   current_head ?
 				   current_head->object.sha1 : NULL,
-				   0, !!current_head) ||
-	    ref_transaction_commit(transaction, sb.buf, &err)) {
+				   0, !!current_head, sb.buf) ||
+	    ref_transaction_commit(transaction, &err)) {
 		rollback_index_files();
 		die("%s", err.buf);
 	}
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 5dbd0f0..3a849b0 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -374,11 +374,17 @@ static int s_update_ref(const char *action,
 			struct ref *ref,
 			int check_old)
 {
+	char msg[1024];
+	char *rla = getenv("GIT_REFLOG_ACTION");
+
 	if (dry_run)
 		return 0;
+	if (!rla)
+		rla = default_rla.buf;
+	snprintf(msg, sizeof(msg), "%s: %s", rla, action);
 
 	if (ref_transaction_update(transaction, ref->name, ref->new_sha1,
-			       ref->old_sha1, 0, check_old))
+				   ref->old_sha1, 0, check_old, msg))
 		return STORE_REF_ERROR_OTHER;
 
 	return 0;
@@ -670,7 +676,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 			}
 		}
 	}
-	if (ref_transaction_commit(transaction, "fetch_ref transaction", NULL))
+	if (ref_transaction_commit(transaction, NULL))
 		rc |= errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
 		  STORE_REF_ERROR_OTHER;
 	ref_transaction_free(transaction);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index d580176..991c659 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -582,7 +582,7 @@ static const char *update(struct command *cmd, struct shallow_info *si)
 			return "shallow error";
 
 		if (ref_transaction_update(transaction, namespaced_name,
-					   new_sha1, old_sha1, 0, 1))
+					   new_sha1, old_sha1, 0, 1, "push"))
 			return "failed to update";
 		return NULL; /* good */
 	}
@@ -823,7 +823,7 @@ static void execute_commands(struct command *commands,
 			checked_connectivity = 0;
 		}
 	}
-	if (ref_transaction_commit(transaction, "push", &err))
+	if (ref_transaction_commit(transaction, &err))
 		error("%s", err.buf);
 	ref_transaction_free(transaction);
 	strbuf_release(&err);
diff --git a/builtin/replace.c b/builtin/replace.c
index 91354a7..0c46b3a 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -160,8 +160,8 @@ static int replace_object_sha1(const char *object_ref,
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_update(transaction, ref, repl, prev,
-				   0, !is_null_sha1(prev)) ||
-	    ref_transaction_commit(transaction, NULL, &err))
+				   0, !is_null_sha1(prev), NULL) ||
+	    ref_transaction_commit(transaction, &err))
 		die("%s: failed to replace ref: %s", ref, err.buf);
 
 	ref_transaction_free(transaction);
diff --git a/builtin/tag.c b/builtin/tag.c
index fbd2989..2cc260f 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -705,8 +705,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_update(transaction, ref.buf, object, prev,
-				   0, !is_null_sha1(prev)) ||
-	    ref_transaction_commit(transaction, NULL, &err))
+				   0, !is_null_sha1(prev), NULL) ||
+	    ref_transaction_commit(transaction, &err))
 		die(_("%s: cannot update the ref: %s"), ref.buf, err.buf);
 	ref_transaction_free(transaction);
 	if (force && !is_null_sha1(prev) && hashcmp(prev, object))
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index b5f4731..bd7e96f 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -16,6 +16,7 @@ static struct ref_transaction *transaction;
 
 static char line_termination = '\n';
 static int update_flags;
+static const char *msg;
 
 /*
  * Parse one whitespace- or NUL-terminated, possibly C-quoted argument
@@ -198,7 +199,7 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)
 		die("update %s: extra input: %s", refname, next);
 
 	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
-				   update_flags, have_old))
+				   update_flags, have_old, msg))
 		die("update %s: failed", refname);
 
 	update_flags = 0;
@@ -226,7 +227,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)
 		die("create %s: extra input: %s", refname, next);
 
 	if (ref_transaction_create(transaction, refname, new_sha1,
-				   update_flags))
+				   update_flags, msg))
 		die("cannot create ref '%s'", refname);
 
 	update_flags = 0;
@@ -258,7 +259,7 @@ static const char *parse_cmd_delete(struct strbuf *input, const char *next)
 		die("delete %s: extra input: %s", refname, next);
 
 	if (ref_transaction_delete(transaction, refname, old_sha1,
-				   update_flags, have_old))
+				   update_flags, have_old, msg))
 		die("failed transaction delete for %s", refname);
 
 	update_flags = 0;
@@ -291,7 +292,7 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)
 		die("verify %s: extra input: %s", refname, next);
 
 	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
-				   update_flags, have_old))
+				   update_flags, have_old, msg))
 		die("failed transaction update for %s", refname);
 
 	update_flags = 0;
@@ -344,7 +345,7 @@ static void update_refs_stdin(void)
 
 int cmd_update_ref(int argc, const char **argv, const char *prefix)
 {
-	const char *refname, *oldval, *msg = NULL;
+	const char *refname, *oldval;
 	unsigned char sha1[20], oldsha1[20];
 	int delete = 0, no_deref = 0, read_stdin = 0, end_null = 0, flags = 0;
 	struct strbuf err = STRBUF_INIT;
@@ -371,7 +372,7 @@ int cmd_update_ref(int argc, const char **argv, const char *prefix)
 		if (end_null)
 			line_termination = '\0';
 		update_refs_stdin();
-		if (ref_transaction_commit(transaction, msg, &err))
+		if (ref_transaction_commit(transaction, &err))
 			die("%s", err.buf);
 		ref_transaction_free(transaction);
 		return 0;
diff --git a/fast-import.c b/fast-import.c
index 5587cf6..099e71b 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1710,8 +1710,8 @@ static int update_branch(struct branch *b)
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_update(transaction, b->name, b->sha1, old_sha1,
-				   0, 1) ||
-	    ref_transaction_commit(transaction, msg, &err)) {
+				   0, 1, msg) ||
+	    ref_transaction_commit(transaction, &err)) {
 		ref_transaction_rollback(transaction);
 		error("Unable to update branch %s: %s", b->name, err.buf);
 		strbuf_release(&err);
@@ -1745,10 +1745,10 @@ static void dump_tags(void)
 		sprintf(ref_name, "refs/tags/%s", t->name);
 
 		if (ref_transaction_update(transaction, ref_name, t->sha1,
-					   NULL, 0, 0))
+					   NULL, 0, 0, msg))
 			failure |= error("Unable to update %s", err.buf);
 	}
-	if (failure || ref_transaction_commit(transaction, msg, &err))
+	if (failure || ref_transaction_commit(transaction, &err))
 		failure |= error("Unable to update %s", err.buf);
 	ref_transaction_free(transaction);
 	strbuf_release(&err);
diff --git a/refs.c b/refs.c
index e7bd95d..1977640 100644
--- a/refs.c
+++ b/refs.c
@@ -2340,8 +2340,8 @@ static void prune_ref(struct ref_to_prune *r)
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_delete(transaction, r->name, r->sha1,
-				   REF_ISPRUNING, 1) ||
-	    ref_transaction_commit(transaction, NULL, &err)) {
+				   REF_ISPRUNING, 1, NULL) ||
+	    ref_transaction_commit(transaction, &err)) {
 		ref_transaction_rollback(transaction);
 		warning("prune_ref: %s", err.buf);
 		strbuf_release(&err);
@@ -2515,8 +2515,8 @@ int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_delete(transaction, refname, sha1, delopt,
-				   sha1 && !is_null_sha1(sha1)) ||
-	    ref_transaction_commit(transaction, NULL, NULL)) {
+				   sha1 && !is_null_sha1(sha1), NULL) ||
+	    ref_transaction_commit(transaction, NULL)) {
 		ref_transaction_rollback(transaction);
 		return 1;
 	}
@@ -3264,6 +3264,7 @@ struct ref_update {
 	int have_old; /* 1 if old_sha1 is valid, 0 otherwise */
 	struct ref_lock *lock;
 	int type;
+	const char *msg;
 	const char refname[FLEX_ARRAY];
 };
 
@@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)
 	if (!transaction)
 		return;
 
-	for (i = 0; i < transaction->nr; i++)
+	for (i = 0; i < transaction->nr; i++) {
+	  free((char *)transaction->updates[i]->msg);
 		free(transaction->updates[i]);
-
+	}
 	free(transaction->updates);
 	free(transaction);
 }
@@ -3330,7 +3332,7 @@ int ref_transaction_update(struct ref_transaction *transaction,
 			    const char *refname,
 			    const unsigned char *new_sha1,
 			    const unsigned char *old_sha1,
-			    int flags, int have_old)
+			   int flags, int have_old, const char *msg)
 {
 	struct ref_update *update;
 
@@ -3346,13 +3348,15 @@ int ref_transaction_update(struct ref_transaction *transaction,
 	update->have_old = have_old;
 	if (have_old)
 		hashcpy(update->old_sha1, old_sha1);
+	if (msg)
+		update->msg = xstrdup(msg);
 	return 0;
 }
 
 int ref_transaction_create(struct ref_transaction *transaction,
 			   const char *refname,
 			   const unsigned char *new_sha1,
-			   int flags)
+			   int flags, const char *msg)
 {
 	struct ref_update *update;
 
@@ -3368,13 +3372,15 @@ int ref_transaction_create(struct ref_transaction *transaction,
 	hashclr(update->old_sha1);
 	update->flags = flags;
 	update->have_old = 1;
+	if (msg)
+		update->msg = xstrdup(msg);
 	return 0;
 }
 
 int ref_transaction_delete(struct ref_transaction *transaction,
 			   const char *refname,
 			   const unsigned char *old_sha1,
-			   int flags, int have_old)
+			   int flags, int have_old, const char *msg)
 {
 	struct ref_update *update;
 
@@ -3391,6 +3397,8 @@ int ref_transaction_delete(struct ref_transaction *transaction,
 		assert(!is_null_sha1(old_sha1));
 		hashcpy(update->old_sha1, old_sha1);
 	}
+	if (msg)
+		update->msg = xstrdup(msg);
 	return 0;
 }
 
@@ -3404,8 +3412,8 @@ int update_ref(const char *action, const char *refname,
 	t = ref_transaction_begin();
 	if (!t ||
 	    ref_transaction_update(t, refname, sha1, oldval, flags,
-				   !!oldval) ||
-	    ref_transaction_commit(t, action, &err)) {
+				   !!oldval, action) ||
+	    ref_transaction_commit(t, &err)) {
 		const char *str = "update_ref failed for ref '%s': %s";
 
 		ref_transaction_rollback(t);
@@ -3447,7 +3455,7 @@ static int ref_update_reject_duplicates(struct ref_update **updates, int n,
 }
 
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, struct strbuf *err)
+			   struct strbuf *err)
 {
 	int ret = 0, delnum = 0, i;
 	const char **delnames;
@@ -3496,7 +3504,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 
 		if (!is_null_sha1(update->new_sha1)) {
 			ret = write_ref_sha1(update->lock, update->new_sha1,
-					     msg);
+					     update->msg);
 			update->lock = NULL; /* freed by write_ref_sha1 */
 			if (ret) {
 				const char *str = "Cannot update the ref '%s'.";
diff --git a/refs.h b/refs.h
index 221632c..0e6d416 100644
--- a/refs.h
+++ b/refs.h
@@ -244,7 +244,7 @@ int ref_transaction_update(struct ref_transaction *transaction,
 			   const char *refname,
 			   const unsigned char *new_sha1,
 			   const unsigned char *old_sha1,
-			   int flags, int have_old);
+			   int flags, int have_old, const char *msg);
 
 /*
  * Add a reference creation to transaction.  new_sha1 is the value
@@ -258,7 +258,7 @@ int ref_transaction_update(struct ref_transaction *transaction,
 int ref_transaction_create(struct ref_transaction *transaction,
 			   const char *refname,
 			   const unsigned char *new_sha1,
-			   int flags);
+			   int flags, const char *msg);
 
 /*
  * Add a reference deletion to transaction.  If have_old is true, then
@@ -271,7 +271,7 @@ int ref_transaction_create(struct ref_transaction *transaction,
 int ref_transaction_delete(struct ref_transaction *transaction,
 			   const char *refname,
 			   const unsigned char *old_sha1,
-			   int flags, int have_old);
+			   int flags, int have_old, const char *msg);
 
 /*
  * Commit all of the changes that have been queued in transaction, as
@@ -280,7 +280,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,
  * why the transaction failed. The string does not end in newline.
  */
 int ref_transaction_commit(struct ref_transaction *transaction,
-			   const char *msg, struct strbuf *err);
+			   struct strbuf *err);
 
 /*
  * Free an existing transaction.
diff --git a/sequencer.c b/sequencer.c
index 0cfdaf0..3a0ee09 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -285,8 +285,8 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,
 	transaction = ref_transaction_begin();
 	if (!transaction ||
 	    ref_transaction_update(transaction, "HEAD", to, from,
-				   0, !unborn) ||
-	    ref_transaction_commit(transaction, sb.buf, &err)) {
+				   0, !unborn, sb.buf) ||
+	    ref_transaction_commit(transaction, &err)) {
 		ref_transaction_rollback(transaction);
 		error(_("HEAD: Could not fast-forward: %s\n"), err.buf);
 		strbuf_release(&sb);
diff --git a/walker.c b/walker.c
index 6044ccf..c2a1266 100644
--- a/walker.c
+++ b/walker.c
@@ -291,12 +291,12 @@ int walker_fetch(struct walker *walker, int targets, char **target,
 		sprintf(ref_name, "refs/%s", write_ref[i]);
 		if (ref_transaction_update(transaction, ref_name,
 					   &sha1[20 * i], NULL,
-					   0, 0))
+					   0, 0,
+					   msg ? msg : "fetch (unknown)"))
 			goto rollback_and_fail;
 	}
 
-	if (ref_transaction_commit(transaction, msg ? msg : "fetch (unknown)",
-				   &err)) {
+	if (ref_transaction_commit(transaction, &err)) {
 		error("%s", err.buf);
 		goto rollback_and_fail;
 	}
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 37/44] refs.c: pass NULL as *flags to read_ref_full

[Ronnie Sahlberg]

We call read_ref_full with a pointer to flags from rename_ref but since
we never actually use the returned flags we can just pass NULL here instead.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 1977640..8105ee6 100644
--- a/refs.c
+++ b/refs.c
@@ -2612,7 +2612,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 		goto rollback;
 	}
 
-	if (!read_ref_full(newrefname, sha1, 1, &flag) &&
+	if (!read_ref_full(newrefname, sha1, 1, NULL) &&
 	    delete_ref(newrefname, sha1, REF_NODEREF)) {
 		if (errno==EISDIR) {
 			if (remove_empty_directories(git_path("%s", newrefname))) {
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref

[Ronnie Sahlberg]

This means that most loose refs will no longer be present after the rename
which triggered a test failure since it assumes the file for an unrelated
ref would still be present after the rename.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c            | 3 +++
 t/t3200-branch.sh | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/refs.c b/refs.c
index 8105ee6..79d95db 100644
--- a/refs.c
+++ b/refs.c
@@ -2607,6 +2607,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 		return error("unable to move logfile logs/%s to "TMP_RENAMED_LOG": %s",
 			oldrefname, strerror(errno));
 
+	if (pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE))
+		return error("unable to pack refs");
+
 	if (delete_ref(oldrefname, orig_sha1, REF_NODEREF)) {
 		error("unable to delete old %s", oldrefname);
 		goto rollback;
diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh
index ac31b71..fafd38a 100755
--- a/t/t3200-branch.sh
+++ b/t/t3200-branch.sh
@@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '
 	git symbolic-ref refs/heads/master2 refs/heads/master &&
 	test_must_fail git branch -m master2 master3 &&
 	git symbolic-ref refs/heads/master2 &&
-	test_path_is_file .git/refs/heads/master &&
+	test_path_is_missing .git/refs/heads/master &&
 	test_path_is_missing .git/refs/heads/master3
 '
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic

[Ronnie Sahlberg]

Move the check for check_refname_format from lock_any_ref_for_update
to lock_ref_sha1_basic. At some later stage we will get rid of
lock_any_ref_for_update completely.

This leaves lock_any_ref_for_updates as a no-op wrapper which could be removed.
But this wrapper is also called from an external caller and we will soon
make changes to the signature to lock_ref_sha1_basic that we do not want to
expose to that caller.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/refs.c b/refs.c
index 79d95db..53fdbd1 100644
--- a/refs.c
+++ b/refs.c
@@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	int missing = 0;
 	int attempts_remaining = 3;
 
+	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
+		return NULL;
+
 	lock = xcalloc(1, sizeof(struct ref_lock));
 	lock->lock_fd = -1;
 
@@ -2135,8 +2138,6 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,
 					 const unsigned char *old_sha1,
 					 int flags, int *type_p)
 {
-	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
-		return NULL;
 	return lock_ref_sha1_basic(refname, old_sha1, flags, type_p);
 }
 
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 40/44] refs.c: call lock_ref_sha1_basic directly from commit

[Ronnie Sahlberg]

Skip using the lock_any_ref_for_update wrapper and call lock_ref_sha1_basic
directly from the commit function.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/refs.c b/refs.c
index 53fdbd1..69ad51d 100644
--- a/refs.c
+++ b/refs.c
@@ -3487,12 +3487,12 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
 
-		update->lock = lock_any_ref_for_update(update->refname,
-						       (update->have_old ?
-							update->old_sha1 :
-							NULL),
-						       update->flags,
-						       &update->type);
+		update->lock = lock_ref_sha1_basic(update->refname,
+						   (update->have_old ?
+						    update->old_sha1 :
+						    NULL),
+						   update->flags,
+						   &update->type);
 		if (!update->lock) {
 			if (err)
 				strbuf_addf(err, "Cannot lock the ref '%s'.",
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Ronnie Sahlberg]

Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.
This flag indicates that the ref does not exist as a loose ref andf only as
a packed ref. If this is the case we then change the commit code so that
we skip taking out a lock file and we skip calling delete_ref_loose.
Check for this flag and die(BUG:...) if used with _update or _create.

At the start of the transaction, before we even start locking any refs,
we add all such REF_ISPACKONLY refs to delnames so that we have a list of
all pack only refs that we will be deleting during this transaction.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/refs.c b/refs.c
index 69ad51d..f458ff9 100644
--- a/refs.c
+++ b/refs.c
@@ -33,6 +33,10 @@ static inline int bad_ref_char(int ch)
  *  pruned.
  */
 #define REF_ISPRUNING	0x0100
+/** Deletion of a ref that only exists as a packed ref in which case we do not
+ *  need to lock the loose ref during the transaction.
+ */
+#define REF_ISPACKONLY	0x0200
 
 /*
  * Try to read one refname component from the front of refname.  Return
@@ -3346,6 +3350,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
 	if (transaction->status != REF_TRANSACTION_OPEN)
 		die("BUG: update on transaction that is not open");
 
+	if (flags & REF_ISPACKONLY)
+		die("BUG: REF_ISPACKONLY can not be used with updates");
+
 	update = add_update(transaction, refname);
 	hashcpy(update->new_sha1, new_sha1);
 	update->flags = flags;
@@ -3370,6 +3377,9 @@ int ref_transaction_create(struct ref_transaction *transaction,
 	if (transaction->status != REF_TRANSACTION_OPEN)
 		die("BUG: create on transaction that is not open");
 
+	if (flags & REF_ISPACKONLY)
+		die("BUG: REF_ISPACKONLY can not be used with creates");
+
 	update = add_update(transaction, refname);
 
 	hashcpy(update->new_sha1, new_sha1);
@@ -3483,10 +3493,20 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	if (ret)
 		goto cleanup;
 
+	for (i = 0; i < n; i++) {
+		struct ref_update *update = updates[i];
+
+		if (update->flags & REF_ISPACKONLY)
+			delnames[delnum++] = update->refname;
+	}
+
 	/* Acquire all locks while verifying old values */
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
 
+		if (update->flags & REF_ISPACKONLY)
+			continue;
+
 		update->lock = lock_ref_sha1_basic(update->refname,
 						   (update->have_old ?
 						    update->old_sha1 :
@@ -3524,6 +3544,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 	for (i = 0; i < n; i++) {
 		struct ref_update *update = updates[i];
 
+		if (update->flags & REF_ISPACKONLY)
+			continue;
+
 		if (update->lock) {
 			ret |= delete_ref_loose(update->lock, update->type,
 						err);
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn

[Ronnie Sahlberg]

Allow passing a list of refs to skip checking to name_conflict_fn.
There are some conditions where we want to allow a temporary conflict and skip
checking those refs. For example if we have a transaction that
1, guarantees that m is a packed refs and there is no loose ref for m
2, the transaction will delete m from the packed ref
3, the transaction will create conflicting m/m

For this case we want to be able to lock and create m/m since we know that the
conflict is only transient. I.e. the conflict will be automatically resolved
by the transaction when it deletes m.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 43 +++++++++++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 10 deletions(-)

diff --git a/refs.c b/refs.c
index f458ff9..c940509 100644
--- a/refs.c
+++ b/refs.c
@@ -798,11 +798,19 @@ struct name_conflict_cb {
 	const char *refname;
 	const char *oldrefname;
 	const char *conflicting_refname;
+	const char **skip;
+	int skipnum;
 };
 
 static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
 {
 	struct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;
+	int i;
+	for(i = 0; i < data->skipnum; i++) {
+		if (!strcmp(entry->name, data->skip[i])) {
+			return 0;
+		}
+	}
 	if (data->oldrefname && !strcmp(data->oldrefname, entry->name))
 		return 0;
 	if (names_conflict(data->refname, entry->name)) {
@@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
  * conflicting with the name of an existing reference in dir.  If
  * oldrefname is non-NULL, ignore potential conflicts with oldrefname
  * (e.g., because oldrefname is scheduled for deletion in the same
- * operation).
+ * operation). skip contains a list of refs we want to skip checking for
+ * conflicts with. Refs may be skipped due to us knowing that it will
+ * be deleted later during a transaction that deletes one reference and then
+ * creates a new conflicting reference. For example a rename from m to m/m.
  */
 static int is_refname_available(const char *refname, const char *oldrefname,
-				struct ref_dir *dir)
+				struct ref_dir *dir,
+				const char **skip, int skipnum)
 {
 	struct name_conflict_cb data;
 	data.refname = refname;
 	data.oldrefname = oldrefname;
 	data.conflicting_refname = NULL;
+	data.skip = skip;
+	data.skipnum = skipnum;
 
 	sort_ref_dir(dir);
 	if (do_for_each_entry_in_dir(dir, 0, name_conflict_fn, &data)) {
@@ -2037,7 +2051,8 @@ int dwim_log(const char *str, int len, unsigned char *sha1, char **log)
 
 static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 					    const unsigned char *old_sha1,
-					    int flags, int *type_p)
+					    int flags, int *type_p,
+					    const char **skip, int skipnum)
 {
 	char *ref_file;
 	const char *orig_refname = refname;
@@ -2084,7 +2099,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
 	 * name is a proper prefix of our refname.
 	 */
 	if (missing &&
-	     !is_refname_available(refname, NULL, get_packed_refs(&ref_cache))) {
+	     !is_refname_available(refname, NULL,
+				   get_packed_refs(&ref_cache),
+				   skip, skipnum)) {
 		last_errno = ENOTDIR;
 		goto error_return;
 	}
@@ -2142,7 +2159,7 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,
 					 const unsigned char *old_sha1,
 					 int flags, int *type_p)
 {
-	return lock_ref_sha1_basic(refname, old_sha1, flags, type_p);
+	return lock_ref_sha1_basic(refname, old_sha1, flags, type_p, NULL, 0);
 }
 
 /*
@@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 	int log = !lstat(git_path("logs/%s", oldrefname), &loginfo);
 	const char *symref = NULL;
 
+	if (!strcmp(oldrefname, newrefname))
+		return 0;
+
 	if (log && S_ISLNK(loginfo.st_mode))
 		return error("reflog for %s is a symlink", oldrefname);
 
@@ -2602,10 +2622,12 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 	if (!symref)
 		return error("refname %s not found", oldrefname);
 
-	if (!is_refname_available(newrefname, oldrefname, get_packed_refs(&ref_cache)))
+	if (!is_refname_available(newrefname, oldrefname,
+				  get_packed_refs(&ref_cache), NULL, 0))
 		return 1;
 
-	if (!is_refname_available(newrefname, oldrefname, get_loose_refs(&ref_cache)))
+	if (!is_refname_available(newrefname, oldrefname,
+				  get_loose_refs(&ref_cache), NULL, 0))
 		return 1;
 
 	if (log && rename(git_path("logs/%s", oldrefname), git_path(TMP_RENAMED_LOG)))
@@ -2638,7 +2660,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 
 	logmoved = log;
 
-	lock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL);
+	lock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL, NULL, 0);
 	if (!lock) {
 		error("unable to lock %s for update", newrefname);
 		goto rollback;
@@ -2653,7 +2675,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 	return 0;
 
  rollback:
-	lock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL);
+	lock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL, NULL, 0);
 	if (!lock) {
 		error("unable to lock %s for rollback", oldrefname);
 		goto rollbacklog;
@@ -3512,7 +3534,8 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 						    update->old_sha1 :
 						    NULL),
 						   update->flags,
-						   &update->type);
+						   &update->type,
+						   delnames, delnum);
 		if (!update->lock) {
 			if (err)
 				strbuf_addf(err, "Cannot lock the ref '%s'.",
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 43/44] refs.c: make rename_ref use a transaction

[Ronnie Sahlberg]

Change rename_ref to use a single transaction to perform the ref rename.

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 73 ++++++++++++++++++------------------------------------------------
 1 file changed, 20 insertions(+), 53 deletions(-)

diff --git a/refs.c b/refs.c
index c940509..336df40 100644
--- a/refs.c
+++ b/refs.c
@@ -2602,9 +2602,10 @@ static int rename_tmp_log(const char *newrefname)
 
 int rename_ref(const char *oldrefname, const char *newrefname, const char *logmsg)
 {
-	unsigned char sha1[20], orig_sha1[20];
-	int flag = 0, logmoved = 0;
-	struct ref_lock *lock;
+	unsigned char sha1[20];
+	int flag = 0;
+	struct ref_transaction *transaction;
+	struct strbuf err = STRBUF_INIT;
 	struct stat loginfo;
 	int log = !lstat(git_path("logs/%s", oldrefname), &loginfo);
 	const char *symref = NULL;
@@ -2615,7 +2616,7 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 	if (log && S_ISLNK(loginfo.st_mode))
 		return error("reflog for %s is a symlink", oldrefname);
 
-	symref = resolve_ref_unsafe(oldrefname, orig_sha1, 1, &flag);
+	symref = resolve_ref_unsafe(oldrefname, sha1, 1, &flag);
 	if (flag & REF_ISSYMREF)
 		return error("refname %s is a symbolic ref, renaming it is not supported",
 			oldrefname);
@@ -2637,62 +2638,28 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
 	if (pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE))
 		return error("unable to pack refs");
 
-	if (delete_ref(oldrefname, orig_sha1, REF_NODEREF)) {
-		error("unable to delete old %s", oldrefname);
-		goto rollback;
-	}
-
-	if (!read_ref_full(newrefname, sha1, 1, NULL) &&
-	    delete_ref(newrefname, sha1, REF_NODEREF)) {
-		if (errno==EISDIR) {
-			if (remove_empty_directories(git_path("%s", newrefname))) {
-				error("Directory not empty: %s", newrefname);
-				goto rollback;
-			}
-		} else {
-			error("unable to delete existing %s", newrefname);
-			goto rollback;
-		}
+	transaction = ref_transaction_begin();
+	if (!transaction ||
+	    ref_transaction_delete(transaction, oldrefname, sha1,
+				   REF_NODEREF | REF_ISPACKONLY,
+				   1, NULL) ||
+	    ref_transaction_update(transaction, newrefname, sha1,
+				   NULL, 0, 0, logmsg) ||
+	    ref_transaction_commit(transaction, &err)) {
+		ref_transaction_rollback(transaction);
+		error("rename_ref failed: %s", err.buf);
+		strbuf_release(&err);
+		goto rollbacklog;
 	}
+	ref_transaction_free(transaction);
 
 	if (log && rename_tmp_log(newrefname))
-		goto rollback;
-
-	logmoved = log;
-
-	lock = lock_ref_sha1_basic(newrefname, NULL, 0, NULL, NULL, 0);
-	if (!lock) {
-		error("unable to lock %s for update", newrefname);
-		goto rollback;
-	}
-	lock->force_write = 1;
-	hashcpy(lock->old_sha1, orig_sha1);
-	if (write_ref_sha1(lock, orig_sha1, logmsg)) {
-		error("unable to write current sha1 into %s", newrefname);
-		goto rollback;
-	}
-
-	return 0;
-
- rollback:
-	lock = lock_ref_sha1_basic(oldrefname, NULL, 0, NULL, NULL, 0);
-	if (!lock) {
-		error("unable to lock %s for rollback", oldrefname);
 		goto rollbacklog;
-	}
 
-	lock->force_write = 1;
-	flag = log_all_ref_updates;
-	log_all_ref_updates = 0;
-	if (write_ref_sha1(lock, orig_sha1, NULL))
-		error("unable to write current sha1 into %s", oldrefname);
-	log_all_ref_updates = flag;
+	return 0;
 
  rollbacklog:
-	if (logmoved && rename(git_path("logs/%s", newrefname), git_path("logs/%s", oldrefname)))
-		error("unable to restore logfile %s from %s: %s",
-			oldrefname, newrefname, strerror(errno));
-	if (!logmoved && log &&
+	if (log &&
 	    rename(git_path(TMP_RENAMED_LOG), git_path("logs/%s", oldrefname)))
 		error("unable to restore logfile %s from "TMP_RENAMED_LOG": %s",
 			oldrefname, strerror(errno));
-- 
2.0.0.rc3.477.g0f8edf7

[PATCH v8 44/44] refs.c: remove forward declaration of write_ref_sha1

[Ronnie Sahlberg]

Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
---
 refs.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/refs.c b/refs.c
index 336df40..ed93b75 100644
--- a/refs.c
+++ b/refs.c
@@ -260,8 +260,6 @@ struct ref_entry {
 };
 
 static void read_loose_refs(const char *dirname, struct ref_dir *dir);
-static int write_ref_sha1(struct ref_lock *lock,
-			  const unsigned char *sha1, const char *logmsg);
 
 static struct ref_dir *get_ref_dir(struct ref_entry *entry)
 {
-- 
2.0.0.rc3.477.g0f8edf7

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This patch series is based on next and expands on the transaction API.

Thanks.  Will pick up in v8 where I left off with v6.

Applies with just one minor conflict on top of a merge of
mh/ref-transaction, rs/ref-update-check-errors-early, and
rs/reflog-exists.  Here's an interdiff against version 6 for those
following along.

> Version 8:
>  - Updates after review by JN
>  - Improve commit messages
>  - Add a patch that adds an err argument to repack_without_refs
>  - Add a patch that adds an err argument to delete_loose_ref
>  - Better document that a _update/_delete/_create failure means the whole
>    transaction has failed and needs rollback.
>
> Version 7:
>  - Updated commit messages per JNs review comments.
>  - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not
>    exposed through refs.h

diff --git a/builtin/commit.c b/builtin/commit.c
index 0f4e1fc..07ccc97 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1684,7 +1684,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
 				   0, !!current_head, sb.buf) ||
 	    ref_transaction_commit(transaction, &err)) {
 		rollback_index_files();
-		die(_("HEAD: cannot update ref: %s"), err.buf);
+		die("%s", err.buf);
 	}
 	ref_transaction_free(transaction);
 
diff --git a/builtin/replace.c b/builtin/replace.c
index 47c360c..952b589 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -163,7 +163,7 @@ static int replace_object(const char *object_ref, const char *replace_ref,
 	    ref_transaction_update(transaction, ref, repl, prev,
 				   0, !is_null_sha1(prev), NULL) ||
 	    ref_transaction_commit(transaction, &err))
-		die(_("%s: failed to replace ref: %s"), ref, err.buf);
+		die("%s: failed to replace ref: %s", ref, err.buf);
 
 	ref_transaction_free(transaction);
 	return 0;
diff --git a/builtin/update-ref.c b/builtin/update-ref.c
index c5aff92..bd7e96f 100644
--- a/builtin/update-ref.c
+++ b/builtin/update-ref.c
@@ -228,7 +228,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)
 
 	if (ref_transaction_create(transaction, refname, new_sha1,
 				   update_flags, msg))
-		die("failed transaction create for %s", refname);
+		die("cannot create ref '%s'", refname);
 
 	update_flags = 0;
 	free(refname);
diff --git a/refs.c b/refs.c
index 302a2b3..ed93b75 100644
--- a/refs.c
+++ b/refs.c
@@ -29,6 +29,15 @@ static inline int bad_ref_char(int ch)
 	return 0;
 }
 
+/** Used as a flag to ref_transaction_delete when a loose ref is beeing
+ *  pruned.
+ */
+#define REF_ISPRUNING	0x0100
+/** Deletion of a ref that only exists as a packed ref in which case we do not
+ *  need to lock the loose ref during the transaction.
+ */
+#define REF_ISPACKONLY	0x0200
+
 /*
  * Try to read one refname component from the front of refname.  Return
  * the length of the component found, or -1 if the component is not
@@ -2447,12 +2456,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
 	return 0;
 }
 
-static int repack_without_refs(const char **refnames, int n)
+static int repack_without_refs(const char **refnames, int n, struct strbuf *err)
 {
 	struct ref_dir *packed;
 	struct string_list refs_to_delete = STRING_LIST_INIT_DUP;
 	struct string_list_item *ref_to_delete;
-	int i, removed = 0;
+	int i, ret, removed = 0;
 
 	/* Look for a packed ref */
 	for (i = 0; i < n; i++)
@@ -2465,6 +2474,9 @@ static int repack_without_refs(const char **refnames, int n)
 
 	if (lock_packed_refs(0)) {
 		unable_to_lock_error(git_path("packed-refs"), errno);
+		if (err)
+			strbuf_addf(err, "cannot delete '%s' from packed refs",
+				    refnames[i]);
 		return error("cannot delete '%s' from packed refs", refnames[i]);
 	}
 	packed = get_packed_refs(&ref_cache);
@@ -2490,20 +2502,28 @@ static int repack_without_refs(const char **refnames, int n)
 	}
 
 	/* Write what remains */
-	return commit_packed_refs();
+	ret = commit_packed_refs();
+	if (ret && err)
+		strbuf_addf(err, "unable to overwrite old ref-pack file");
+	return ret;
 }
 
-static int delete_ref_loose(struct ref_lock *lock, int flag)
+static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
 {
 	if (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {
 		/* loose */
-		int err, i = strlen(lock->lk->filename) - 5; /* .lock */
+		int res, i = strlen(lock->lk->filename) - 5; /* .lock */
 
 		lock->lk->filename[i] = 0;
-		err = unlink_or_warn(lock->lk->filename);
+		res = unlink_or_warn(lock->lk->filename);
 		lock->lk->filename[i] = '.';
-		if (err && errno != ENOENT)
+		if (res && errno != ENOENT) {
+			if (err)
+				strbuf_addf(err,
+					    "failed to delete loose ref '%s'",
+					    lock->lk->filename);
 			return 1;
+		}
 	}
 	return 0;
 }
@@ -3483,7 +3503,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 						   delnames, delnum);
 		if (!update->lock) {
 			if (err)
-				strbuf_addf(err ,"Cannot lock the ref '%s'.",
+				strbuf_addf(err, "Cannot lock the ref '%s'.",
 					    update->refname);
 			ret = 1;
 			goto cleanup;
@@ -3516,13 +3536,14 @@ int ref_transaction_commit(struct ref_transaction *transaction,
 			continue;
 
 		if (update->lock) {
-			ret |= delete_ref_loose(update->lock, update->type);
+			ret |= delete_ref_loose(update->lock, update->type,
+						err);
 			if (!(update->flags & REF_ISPRUNING))
 				delnames[delnum++] = update->lock->ref_name;
 		}
 	}
 
-	ret |= repack_without_refs(delnames, delnum);
+	ret |= repack_without_refs(delnames, delnum, err);
 	for (i = 0; i < delnum; i++)
 		unlink_or_warn(git_path("logs/%s", delnames[i]));
 	clear_loose_ref_cache(&ref_cache);
diff --git a/refs.h b/refs.h
index 03b4a7e..0e6d416 100644
--- a/refs.h
+++ b/refs.h
@@ -134,10 +134,6 @@ extern int peel_ref(const char *refname, unsigned char *sha1);
 
 /** Locks any ref (for 'HEAD' type refs). */
 #define REF_NODEREF	0x01
-/** Deleting a loose ref during prune */
-#define REF_ISPRUNING	0x02
-/** Deletion of a ref that only exists as a packed ref */
-#define REF_ISPACKONLY	0x04
 extern struct ref_lock *lock_any_ref_for_update(const char *refname,
 						const unsigned char *old_sha1,
 						int flags, int *type_p);
@@ -240,6 +236,9 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
  * be deleted.  If have_old is true, then old_sha1 holds the value
  * that the reference should have had before the update, or zeros if
  * it must not have existed beforehand.
+ * Function returns 0 on success and non-zero on failure. A failure to update
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
 int ref_transaction_update(struct ref_transaction *transaction,
 			   const char *refname,
@@ -252,6 +251,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
  * that the reference should have after the update; it must not be the
  * null SHA-1.  It is verified that the reference does not exist
  * already.
+ * Function returns 0 on success and non-zero on failure. A failure to create
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
 int ref_transaction_create(struct ref_transaction *transaction,
 			   const char *refname,
@@ -262,6 +264,9 @@ int ref_transaction_create(struct ref_transaction *transaction,
  * Add a reference deletion to transaction.  If have_old is true, then
  * old_sha1 holds the value that the reference should have had before
  * the update (which must not be the null SHA-1).
+ * Function returns 0 on success and non-zero on failure. A failure to delete
+ * means that the transaction as a whole has failed and will need to be
+ * rolled back.
  */
 int ref_transaction_delete(struct ref_transaction *transaction,
 			   const char *refname,
@@ -272,7 +277,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,
  * Commit all of the changes that have been queued in transaction, as
  * atomically as possible.  Return a nonzero value if there is a
  * problem. If err is non-NULL we will add an error string to it to explain
- * why the transaction failed.
+ * why the transaction failed. The string does not end in newline.
  */
 int ref_transaction_commit(struct ref_transaction *transaction,
 			   struct strbuf *err);

Re: [PATCH v8 01/44] refs.c: constify the sha arguments for ref_transaction_create|delete|update

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> ref_transaction_create|delete|update has no need to modify the sha1
> arguments passed to it so it should use const unsigned char* instead
> of unsigned char*.

Obviously good.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Allow ref_transaction_free to be called with NULL and as a result allow
> ref_transaction_rollback to be called for a NULL transaction.
>
> This allows us to write code that will
>   if ( (!transaction ||
>         ref_transaction_update(...))  ||
>       (ref_transaction_commit(...) && !(transaction = NULL)) {
>           ref_transaction_rollback(transaction);
>           ...
>   }
>
> In this case transaction is reset to NULL IFF ref_transaction_commit() was
> invoked and thus the rollback becomes ref_transaction_rollback(NULL) which
> is safe. IF the conditional triggered prior to ref_transaction_commit()
> then transaction is untouched and then ref_transaction_rollback(transaction)
> will rollback the failed transaction.

I still think these last two paragraphs confuse more than enlighten
here.  There's plenty of time to explain them in the patch that uses
that code.

I'd just say something like

	Allow ref_transaction_free(NULL) and hence ref_transaction_rollback(NULL)
	as no-ops.

	This makes ref_transaction_rollback easier to use and more similar to
	plain 'free'.

And maybe:

	In particular, it lets us rollback unconditionally as part of cleanup
	code after setting 'transaction = NULL' if a transaction has been
	committed or rolled back already.

Thanks,
Jonathan

Re: [PATCH v8 02/44] refs.c: allow passing NULL to ref_transaction_free

[Ronnie Sahlberg]

Thanks.

I used your improved text in the commit message.

On Thu, May 15, 2014 at 11:15 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Allow ref_transaction_free to be called with NULL and as a result allow
>> ref_transaction_rollback to be called for a NULL transaction.
>>
>> This allows us to write code that will
>>   if ( (!transaction ||
>>         ref_transaction_update(...))  ||
>>       (ref_transaction_commit(...) && !(transaction = NULL)) {
>>           ref_transaction_rollback(transaction);
>>           ...
>>   }
>>
>> In this case transaction is reset to NULL IFF ref_transaction_commit() was
>> invoked and thus the rollback becomes ref_transaction_rollback(NULL) which
>> is safe. IF the conditional triggered prior to ref_transaction_commit()
>> then transaction is untouched and then ref_transaction_rollback(transaction)
>> will rollback the failed transaction.
>
> I still think these last two paragraphs confuse more than enlighten
> here.  There's plenty of time to explain them in the patch that uses
> that code.
>
> I'd just say something like
>
>         Allow ref_transaction_free(NULL) and hence ref_transaction_rollback(NULL)
>         as no-ops.
>
>         This makes ref_transaction_rollback easier to use and more similar to
>         plain 'free'.
>
> And maybe:
>
>         In particular, it lets us rollback unconditionally as part of cleanup
>         code after setting 'transaction = NULL' if a transaction has been
>         committed or rolled back already.
>
> Thanks,
> Jonathan

Re: [PATCH v8 04/44] refs.c: add an err argument to repack_without_refs

[Jonathan Nieder]

Ronnie Sahlberg wrote:
> --- a/refs.c
> +++ b/refs.c
> @@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
>  	return 0;
>  }
>  
> -static int repack_without_refs(const char **refnames, int n)
> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)

Should this also get an onerr flag to suppress the message to stderr,
or unconditionally suppress the message to stderr when err != NULL?

[...]
> @@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)
>  
>  	if (lock_packed_refs(0)) {
>  		unable_to_lock_error(git_path("packed-refs"), errno);
> +		if (err)
> +			strbuf_addf(err, "cannot delete '%s' from packed refs",
> +				    refnames[i]);

unable_to_lock_error is able to come up with a message with more
detail (path so the sysadmin can hunt down the problem even if this
was run e.g. from a cronjob where the path is not obvious, errno
hinting at the nature of the problem).

[...]
> @@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)
>  	}
>  
>  	/* Write what remains */
> -	return commit_packed_refs();
> +	ret = commit_packed_refs();
> +	if (ret && err)
> +		strbuf_addf(err, "unable to overwrite old ref-pack file");

After commit_lock_file sets errno, amazingly no one clobbers it
until we get to this point.  The only calls in between are to
free().

It would be nice to make that more explicit in commit_packed_refs:

	int save_errno;

	...
	if (commit_lock_file(packed_ref_cache->lock)) {
		save_errno = errno;
		error = -1;
	}

	packed_ref_cache->lock = NULL;
	release_packed_ref_cache(packed_ref_cache);

	errno = save_errno;
	return error;

Even without that, this message could include strerror(errno).

> +	return ret;
>  }

Thanks,
Jonathan

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> Ronnie Sahlberg wrote:
>
>> This patch series is based on next and expands on the transaction API.
>
> Thanks.  Will pick up in v8 where I left off with v6.
>
> Applies with just one minor conflict on top of a merge of
> mh/ref-transaction, rs/ref-update-check-errors-early, and
> rs/reflog-exists.  Here's an interdiff against version 6 for those
> following along.

The interdiffs look mostly sensible.  Thanks.

>> Version 8:
>>  - Updates after review by JN
>>  - Improve commit messages
>>  - Add a patch that adds an err argument to repack_without_refs
>>  - Add a patch that adds an err argument to delete_loose_ref
>>  - Better document that a _update/_delete/_create failure means the whole
>>    transaction has failed and needs rollback.
>>
>> Version 7:
>>  - Updated commit messages per JNs review comments.
>>  - Changed REF_ISPRUNING and REF_ISPACKONLY to be private flags and not
>>    exposed through refs.h
>
> diff --git a/builtin/commit.c b/builtin/commit.c
> index 0f4e1fc..07ccc97 100644
> --- a/builtin/commit.c
> +++ b/builtin/commit.c
> @@ -1684,7 +1684,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix)
>  				   0, !!current_head, sb.buf) ||
>  	    ref_transaction_commit(transaction, &err)) {
>  		rollback_index_files();
> -		die(_("HEAD: cannot update ref: %s"), err.buf);
> +		die("%s", err.buf);
>  	}
>  	ref_transaction_free(transaction);
>  
> diff --git a/builtin/replace.c b/builtin/replace.c
> index 47c360c..952b589 100644
> --- a/builtin/replace.c
> +++ b/builtin/replace.c
> @@ -163,7 +163,7 @@ static int replace_object(const char *object_ref, const char *replace_ref,
>  	    ref_transaction_update(transaction, ref, repl, prev,
>  				   0, !is_null_sha1(prev), NULL) ||
>  	    ref_transaction_commit(transaction, &err))
> -		die(_("%s: failed to replace ref: %s"), ref, err.buf);
> +		die("%s: failed to replace ref: %s", ref, err.buf);
>  
>  	ref_transaction_free(transaction);
>  	return 0;
> diff --git a/builtin/update-ref.c b/builtin/update-ref.c
> index c5aff92..bd7e96f 100644
> --- a/builtin/update-ref.c
> +++ b/builtin/update-ref.c
> @@ -228,7 +228,7 @@ static const char *parse_cmd_create(struct strbuf *input, const char *next)
>  
>  	if (ref_transaction_create(transaction, refname, new_sha1,
>  				   update_flags, msg))
> -		die("failed transaction create for %s", refname);
> +		die("cannot create ref '%s'", refname);
>  
>  	update_flags = 0;
>  	free(refname);
> diff --git a/refs.c b/refs.c
> index 302a2b3..ed93b75 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -29,6 +29,15 @@ static inline int bad_ref_char(int ch)
>  	return 0;
>  }
>  
> +/** Used as a flag to ref_transaction_delete when a loose ref is beeing
> + *  pruned.
> + */
> +#define REF_ISPRUNING	0x0100
> +/** Deletion of a ref that only exists as a packed ref in which case we do not
> + *  need to lock the loose ref during the transaction.
> + */
> +#define REF_ISPACKONLY	0x0200
> +
>  /*
>   * Try to read one refname component from the front of refname.  Return
>   * the length of the component found, or -1 if the component is not
> @@ -2447,12 +2456,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
>  	return 0;
>  }
>  
> -static int repack_without_refs(const char **refnames, int n)
> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)
>  {
>  	struct ref_dir *packed;
>  	struct string_list refs_to_delete = STRING_LIST_INIT_DUP;
>  	struct string_list_item *ref_to_delete;
> -	int i, removed = 0;
> +	int i, ret, removed = 0;
>  
>  	/* Look for a packed ref */
>  	for (i = 0; i < n; i++)
> @@ -2465,6 +2474,9 @@ static int repack_without_refs(const char **refnames, int n)
>  
>  	if (lock_packed_refs(0)) {
>  		unable_to_lock_error(git_path("packed-refs"), errno);
> +		if (err)
> +			strbuf_addf(err, "cannot delete '%s' from packed refs",
> +				    refnames[i]);
>  		return error("cannot delete '%s' from packed refs", refnames[i]);
>  	}
>  	packed = get_packed_refs(&ref_cache);
> @@ -2490,20 +2502,28 @@ static int repack_without_refs(const char **refnames, int n)
>  	}
>  
>  	/* Write what remains */
> -	return commit_packed_refs();
> +	ret = commit_packed_refs();
> +	if (ret && err)
> +		strbuf_addf(err, "unable to overwrite old ref-pack file");
> +	return ret;
>  }
>  
> -static int delete_ref_loose(struct ref_lock *lock, int flag)
> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>  {
>  	if (!(flag & REF_ISPACKED) || flag & REF_ISSYMREF) {
>  		/* loose */
> -		int err, i = strlen(lock->lk->filename) - 5; /* .lock */
> +		int res, i = strlen(lock->lk->filename) - 5; /* .lock */
>  
>  		lock->lk->filename[i] = 0;
> -		err = unlink_or_warn(lock->lk->filename);
> +		res = unlink_or_warn(lock->lk->filename);
>  		lock->lk->filename[i] = '.';
> -		if (err && errno != ENOENT)
> +		if (res && errno != ENOENT) {
> +			if (err)
> +				strbuf_addf(err,
> +					    "failed to delete loose ref '%s'",
> +					    lock->lk->filename);
>  			return 1;
> +		}
>  	}
>  	return 0;
>  }
> @@ -3483,7 +3503,7 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>  						   delnames, delnum);
>  		if (!update->lock) {
>  			if (err)
> -				strbuf_addf(err ,"Cannot lock the ref '%s'.",
> +				strbuf_addf(err, "Cannot lock the ref '%s'.",
>  					    update->refname);
>  			ret = 1;
>  			goto cleanup;
> @@ -3516,13 +3536,14 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>  			continue;
>  
>  		if (update->lock) {
> -			ret |= delete_ref_loose(update->lock, update->type);
> +			ret |= delete_ref_loose(update->lock, update->type,
> +						err);
>  			if (!(update->flags & REF_ISPRUNING))
>  				delnames[delnum++] = update->lock->ref_name;
>  		}
>  	}
>  
> -	ret |= repack_without_refs(delnames, delnum);
> +	ret |= repack_without_refs(delnames, delnum, err);
>  	for (i = 0; i < delnum; i++)
>  		unlink_or_warn(git_path("logs/%s", delnames[i]));
>  	clear_loose_ref_cache(&ref_cache);
> diff --git a/refs.h b/refs.h
> index 03b4a7e..0e6d416 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -134,10 +134,6 @@ extern int peel_ref(const char *refname, unsigned char *sha1);
>  
>  /** Locks any ref (for 'HEAD' type refs). */
>  #define REF_NODEREF	0x01
> -/** Deleting a loose ref during prune */
> -#define REF_ISPRUNING	0x02
> -/** Deletion of a ref that only exists as a packed ref */
> -#define REF_ISPACKONLY	0x04
>  extern struct ref_lock *lock_any_ref_for_update(const char *refname,
>  						const unsigned char *old_sha1,
>  						int flags, int *type_p);
> @@ -240,6 +236,9 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
>   * be deleted.  If have_old is true, then old_sha1 holds the value
>   * that the reference should have had before the update, or zeros if
>   * it must not have existed beforehand.
> + * Function returns 0 on success and non-zero on failure. A failure to update
> + * means that the transaction as a whole has failed and will need to be
> + * rolled back.
>   */
>  int ref_transaction_update(struct ref_transaction *transaction,
>  			   const char *refname,
> @@ -252,6 +251,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
>   * that the reference should have after the update; it must not be the
>   * null SHA-1.  It is verified that the reference does not exist
>   * already.
> + * Function returns 0 on success and non-zero on failure. A failure to create
> + * means that the transaction as a whole has failed and will need to be
> + * rolled back.
>   */
>  int ref_transaction_create(struct ref_transaction *transaction,
>  			   const char *refname,
> @@ -262,6 +264,9 @@ int ref_transaction_create(struct ref_transaction *transaction,
>   * Add a reference deletion to transaction.  If have_old is true, then
>   * old_sha1 holds the value that the reference should have had before
>   * the update (which must not be the null SHA-1).
> + * Function returns 0 on success and non-zero on failure. A failure to delete
> + * means that the transaction as a whole has failed and will need to be
> + * rolled back.
>   */
>  int ref_transaction_delete(struct ref_transaction *transaction,
>  			   const char *refname,
> @@ -272,7 +277,7 @@ int ref_transaction_delete(struct ref_transaction *transaction,
>   * Commit all of the changes that have been queued in transaction, as
>   * atomically as possible.  Return a nonzero value if there is a
>   * problem. If err is non-NULL we will add an error string to it to explain
> - * why the transaction failed.
> + * why the transaction failed. The string does not end in newline.
>   */
>  int ref_transaction_commit(struct ref_transaction *transaction,
>  			   struct strbuf *err);

Re: [PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> [Subject: refs.c: add an err argument ro delete_loose_ref]

s/ro/to/
s/delete_loose_ref/delete_ref_loose/

> --- a/refs.c
> +++ b/refs.c
> @@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)
>  	return repack_without_refs(&refname, 1, NULL);
>  }
>  
> -static int delete_ref_loose(struct ref_lock *lock, int flag)
> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)

Should this get an onerr flag to suppress the message to stderr
or unconditionally suppress it when err != NULL?

[...]
>  		lock->lk->filename[i] = 0;
> -		err = unlink_or_warn(lock->lk->filename);
> +		res = unlink_or_warn(lock->lk->filename);

It seems like in the new error handling scheme there should be a new
variant on wrapper.c's warn_if_unremovable:

	static int add_err_if_unremovable(const char *op, const char *file, struct strbuf *err, int rc)
	{
		int err = errno;
		if (rc < 0 && err != ENOENT) {
			strbuf_addf(err, "unable to %s %s: %s",
				    op, file, strerror(errno));
			errno = err;
		}
		return rc;
	}

	static int unlink_or_err(const char *file, struct strbuf *err)
	{
		return add_err_if_unremovable("unlink", file, err, unlink(file));
	}

		res = unlink_or_err(lock->lk->filename, err);

Re: [PATCH v8 08/44] update-ref.c: log transaction error from the update_ref

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Call ref_transaction_commit with QUIET_ON_ERR and use the strbuf that is
> returned to print a log message if/after the transaction fails.
>
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>

All error paths in _commit add to the error string now, so the only
effect should be to add the missing error message when commiting
packed-refs fails (thanks for fixing that) and to change some one-line
errors to two-line.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Update ref_transaction_update() do some basic error checking and return
> non-zero on error. Update all callers to check ref_transaction_update() for
> error. There are currently no conditions in _update that will return error but
> there will be in the future.

Probably worth passing a 'struct strbuf *err' argument.  Then callers
can do

		die("%s", err.buf);

and the error message can say which ref and whether we were trying to
create a ref, or delete one, or whatever.

> --- a/builtin/update-ref.c
> +++ b/builtin/update-ref.c
> @@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)
>  	if (*next != line_termination)
>  		die("update %s: extra input: %s", refname, next);
>  
> -	ref_transaction_update(transaction, refname, new_sha1, old_sha1,
> -			       update_flags, have_old);
> +	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
> +				   update_flags, have_old))
> +		die("update %s: failed", refname);

This could say

		die("update %s: %s", refname, err.buf);

to give context about which command it was trying to execute.

[...]
> @@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)
>  	if (*next != line_termination)
>  		die("verify %s: extra input: %s", refname, next);
>  
> -	ref_transaction_update(transaction, refname, new_sha1, old_sha1,
> -			       update_flags, have_old);
> +	if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
> +				   update_flags, have_old))
> +		die("failed transaction update for %s", refname);

And this could say

		die("verify %s: %s", refname, err.buf);

[...]
> --- a/refs.h
> +++ b/refs.h
> @@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
>   * be deleted.  If have_old is true, then old_sha1 holds the value
>   * that the reference should have had before the update, or zeros if
>   * it must not have existed beforehand.
> + * Function returns 0 on success and non-zero on failure. A failure to update
> + * means that the transaction as a whole has failed and will need to be
> + * rolled back.
> + */

Thanks for this documentation.

Re: [PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Do basic error checking in ref_transaction_create() and make it return
> non-zero on error.

Same thoughts as _update().  Basic idea is good but would be nice to
have a 'struct strbuf *err' parameter.

Thanks,
Jonathan

Re: [PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change ref_transaction_delete() to do basic error checking and return
> non-zero of error.

Likewise: a 'struct strbuf *err' would make nicer error messages
possible.

Re: [PATCH v8 06/44] refs.c: add an err argument ro delete_loose_ref

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 12:04 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> [Subject: refs.c: add an err argument ro delete_loose_ref]
>
> s/ro/to/
> s/delete_loose_ref/delete_ref_loose/
>
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2484,17 +2484,22 @@ static int repack_without_ref(const char *refname)
>>       return repack_without_refs(&refname, 1, NULL);
>>  }
>>
>> -static int delete_ref_loose(struct ref_lock *lock, int flag)
>> +static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>
> Should this get an onerr flag to suppress the message to stderr
> or unconditionally suppress it when err != NULL?
>

Fixed.
I added a new function unlink_or_err that will update err if non-NULL
and unse warning() otherwise.


> [...]
>>               lock->lk->filename[i] = 0;
>> -             err = unlink_or_warn(lock->lk->filename);
>> +             res = unlink_or_warn(lock->lk->filename);
>
> It seems like in the new error handling scheme there should be a new
> variant on wrapper.c's warn_if_unremovable:
>
>         static int add_err_if_unremovable(const char *op, const char *file, struct strbuf *err, int rc)
>         {
>                 int err = errno;
>                 if (rc < 0 && err != ENOENT) {
>                         strbuf_addf(err, "unable to %s %s: %s",
>                                     op, file, strerror(errno));
>                         errno = err;
>                 }
>                 return rc;
>         }
>
>         static int unlink_or_err(const char *file, struct strbuf *err)
>         {
>                 return add_err_if_unremovable("unlink", file, err, unlink(file));
>         }
>
>                 res = unlink_or_err(lock->lk->filename, err);

Re: [PATCH v8 13/44] tag.c: use ref transactions when doing updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/builtin/tag.c
> +++ b/builtin/tag.c
> @@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
>  	if (annotate)
>  		create_tag(object, tag, &buf, &opt, prev, object);
>  
> -	lock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);
> -	if (!lock)
> -		die(_("%s: cannot lock the ref"), ref.buf);
> -	if (write_ref_sha1(lock, object, NULL) < 0)
> -		die(_("%s: cannot update the ref"), ref.buf);
> +	transaction = ref_transaction_begin();
> +	if (!transaction ||
> +	    ref_transaction_update(transaction, ref.buf, object, prev,
> +				   0, !is_null_sha1(prev)) ||
> +	    ref_transaction_commit(transaction, NULL, &err))
> +		die(_("%s: cannot update the ref: %s"), ref.buf, err.buf);

The error string says what ref it was trying to update, so

		die("%s", err.buf);

should be enough.  (E.g.,

	fatal: refs/tags/v1.0: cannot lock the ref

would become

	fatal: Cannot lock the ref 'refs/tags/v1.0'.

instead of

	fatal: refs/tags/v1.0: cannot update the ref: Cannot lock the ref 'refs/tags/v1.0'.

.)

Thanks,
Jonathan

Re: [PATCH v8 14/44] replace.c: use the ref transaction functions for updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/builtin/replace.c
> +++ b/builtin/replace.c
[...]
> @@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,
>  	else if (!force)
>  		die("replace ref '%s' already exists", ref);
>  
> -	lock = lock_any_ref_for_update(ref, prev, 0, NULL);
> -	if (!lock)
> -		die("%s: cannot lock the ref", ref);
> -	if (write_ref_sha1(lock, repl, NULL) < 0)
> -		die("%s: cannot update the ref", ref);
> +	transaction = ref_transaction_begin();
> +	if (!transaction ||
> +	    ref_transaction_update(transaction, ref, repl, prev,
> +				   0, !is_null_sha1(prev)) ||
> +	    ref_transaction_commit(transaction, NULL, &err))
> +		die("%s: failed to replace ref: %s", ref, err.buf);

This would write

	fatal: refs/replace/09c779943364d893c190066c385e6112af421db3: failed to replace ref: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.

Perhaps something like

	$ git replace foo bar
	fatal: replace foo: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.

would make sense (die("replace %s: %s", object_ref, err.buf)).  Plain

	$ git replace foo bar
	fatal: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.

also seems fine (die("%s", err.buf)).

Thanks,
Jonathan

Re: [PATCH v8 15/44] commit.c: use ref transactions for updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change commit.c to use ref transactions for all ref updates.
> Make sure we pass a NULL pointer to ref_transaction_update if have_old
> is false.

Once ref_transaction_begin() and ref_transaction_update() set err,
this will always have a reasonable message to print in die().

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/fast-import.c
> +++ b/fast-import.c
> @@ -1679,39 +1679,45 @@ found_entry:
>  static int update_branch(struct branch *b)
>  {
>  	static const char *msg = "fast-import";
> -	struct ref_lock *lock;
> +	struct ref_transaction *transaction;
>  	unsigned char old_sha1[20];
> +	struct strbuf err = STRBUF_INIT;
>  
>  	if (read_ref(b->name, old_sha1))
>  		hashclr(old_sha1);
> +
>  	if (is_null_sha1(b->sha1)) {
>  		if (b->delete)
>  			delete_ref(b->name, old_sha1, 0);
>  		return 0;
>  	}
> -	lock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);
> -	if (!lock)
> -		return error("Unable to lock %s", b->name);
>  	if (!force_update && !is_null_sha1(old_sha1)) {
>  		struct commit *old_cmit, *new_cmit;
>  
>  		old_cmit = lookup_commit_reference_gently(old_sha1, 0);
>  		new_cmit = lookup_commit_reference_gently(b->sha1, 0);
>  		if (!old_cmit || !new_cmit) {
> -			unlock_ref(lock);
>  			return error("Branch %s is missing commits.", b->name);
>  		}

(style) Now that there's only one line in the "if" body, we can
drop the braces.

>  
>  		if (!in_merge_bases(old_cmit, new_cmit)) {
> -			unlock_ref(lock);
>  			warning("Not updating %s"
>  				" (new tip %s does not contain %s)",
>  				b->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));
>  			return -1;

(not about this patch, feel free to ignore) This could
return warning("...")

>  	}
> -	if (write_ref_sha1(lock, b->sha1, msg) < 0)
> -		return error("Unable to update %s", b->name);
> +	transaction = ref_transaction_begin();
> +	if ((!transaction ||
> +	    ref_transaction_update(transaction, b->name, b->sha1, old_sha1,
> +				   0, 1)) ||

Would be more idiomatic to drop a layer of parentheses:

	if (!transaction ||
	    ref_transaction_update(...) ||

> +	    (ref_transaction_commit(transaction, msg, &err) &&
> +	     !(transaction = NULL))) {

Would be clearer if ref_transaction_commit didn't take care of the
rollback (or in other words if patch 21 were earlier in the series).

> +		ref_transaction_rollback(transaction);
> +		error("Unable to update branch %s: %s", b->name, err.buf);
> +		strbuf_release(&err);
> +		return -1;
> +	}

Example old message:

	error: Unable to lock refs/heads/master

New message:

	error: Unable to update branch refs/heads/master: Cannot lock the ref 'refs/heads/master'.

So 'error("%s", err.buf)' would probably work better.

The only call site is dump_branches:

	for (i = 0; i < branch_table_sz; i++) {
		for (b = branch_table[i]; b; b = b->table_next_branch)
			failure |= update_branch(b);
	}

Should these happen in a single transaction?  I haven't thought
through whether it would be a good idea, if it should be optional, or
what.

Anyway, that would be a bigger behavior change, but it's interesting
to think about.

Thanks,
Jonathan

Re: [PATCH v8 16/44] sequencer.c: use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/sequencer.c
> +++ b/sequencer.c
> @@ -272,23 +272,31 @@ static int fast_forward_to(const unsigned char *to, const unsigned char *from,
[...]
> +		error(_("HEAD: Could not fast-forward: %s\n"), err.buf);

As with 'git commit', since err.buf already mentions that HEAD was the
problematic ref I think it has enough information for the user to
diagnose the problem and figure out what to do next.

		error("%s", err.buf);

Re: [PATCH v8 12/44] refs.c: ref_transaction_delete to check for error and return status

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 12:51 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Change ref_transaction_delete() to do basic error checking and return
>> non-zero of error.
>
> Likewise: a 'struct strbuf *err' would make nicer error messages
> possible.

Done.

Re: [PATCH v8 11/44] refs.c: change ref_transaction_create to do error checking and return status

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 12:44 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Do basic error checking in ref_transaction_create() and make it return
>> non-zero on error.
>
> Same thoughts as _update().  Basic idea is good but would be nice to
> have a 'struct strbuf *err' parameter.

Done.

>
> Thanks,
> Jonathan

Re: [PATCH v8 10/44] refs.c: change ref_transaction_update() to do error checking and return status

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 12:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Update ref_transaction_update() do some basic error checking and return
>> non-zero on error. Update all callers to check ref_transaction_update() for
>> error. There are currently no conditions in _update that will return error but
>> there will be in the future.
>
> Probably worth passing a 'struct strbuf *err' argument.  Then callers
> can do
>
>                 die("%s", err.buf);
>

Done.

> and the error message can say which ref and whether we were trying to
> create a ref, or delete one, or whatever.
>
>> --- a/builtin/update-ref.c
>> +++ b/builtin/update-ref.c
>> @@ -197,8 +197,9 @@ static const char *parse_cmd_update(struct strbuf *input, const char *next)
>>       if (*next != line_termination)
>>               die("update %s: extra input: %s", refname, next);
>>
>> -     ref_transaction_update(transaction, refname, new_sha1, old_sha1,
>> -                            update_flags, have_old);
>> +     if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
>> +                                update_flags, have_old))
>> +             die("update %s: failed", refname);
>
> This could say
>
>                 die("update %s: %s", refname, err.buf);

Done.

>
> to give context about which command it was trying to execute.
>
> [...]
>> @@ -286,8 +287,9 @@ static const char *parse_cmd_verify(struct strbuf *input, const char *next)
>>       if (*next != line_termination)
>>               die("verify %s: extra input: %s", refname, next);
>>
>> -     ref_transaction_update(transaction, refname, new_sha1, old_sha1,
>> -                            update_flags, have_old);
>> +     if (ref_transaction_update(transaction, refname, new_sha1, old_sha1,
>> +                                update_flags, have_old))
>> +             die("failed transaction update for %s", refname);
>
> And this could say
>
>                 die("verify %s: %s", refname, err.buf);

Done.

>
> [...]
>> --- a/refs.h
>> +++ b/refs.h
>> @@ -242,12 +242,15 @@ void ref_transaction_rollback(struct ref_transaction *transaction);
>>   * be deleted.  If have_old is true, then old_sha1 holds the value
>>   * that the reference should have had before the update, or zeros if
>>   * it must not have existed beforehand.
>> + * Function returns 0 on success and non-zero on failure. A failure to update
>> + * means that the transaction as a whole has failed and will need to be
>> + * rolled back.
>> + */
>
> Thanks for this documentation.

Re: [PATCH v8 17/44] fast-import.c: change update_branch to use ref transactions

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 2:47 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/fast-import.c
>> +++ b/fast-import.c
>> @@ -1679,39 +1679,45 @@ found_entry:
>>  static int update_branch(struct branch *b)
>>  {
>>       static const char *msg = "fast-import";
>> -     struct ref_lock *lock;
>> +     struct ref_transaction *transaction;
>>       unsigned char old_sha1[20];
>> +     struct strbuf err = STRBUF_INIT;
>>
>>       if (read_ref(b->name, old_sha1))
>>               hashclr(old_sha1);
>> +
>>       if (is_null_sha1(b->sha1)) {
>>               if (b->delete)
>>                       delete_ref(b->name, old_sha1, 0);
>>               return 0;
>>       }
>> -     lock = lock_any_ref_for_update(b->name, old_sha1, 0, NULL);
>> -     if (!lock)
>> -             return error("Unable to lock %s", b->name);
>>       if (!force_update && !is_null_sha1(old_sha1)) {
>>               struct commit *old_cmit, *new_cmit;
>>
>>               old_cmit = lookup_commit_reference_gently(old_sha1, 0);
>>               new_cmit = lookup_commit_reference_gently(b->sha1, 0);
>>               if (!old_cmit || !new_cmit) {
>> -                     unlock_ref(lock);
>>                       return error("Branch %s is missing commits.", b->name);
>>               }
>
> (style) Now that there's only one line in the "if" body, we can
> drop the braces.
>

Done.

>>
>>               if (!in_merge_bases(old_cmit, new_cmit)) {
>> -                     unlock_ref(lock);
>>                       warning("Not updating %s"
>>                               " (new tip %s does not contain %s)",
>>                               b->name, sha1_to_hex(b->sha1), sha1_to_hex(old_sha1));
>>                       return -1;
>
> (not about this patch, feel free to ignore) This could
> return warning("...")

Ignored.

>
>>       }
>> -     if (write_ref_sha1(lock, b->sha1, msg) < 0)
>> -             return error("Unable to update %s", b->name);
>> +     transaction = ref_transaction_begin();
>> +     if ((!transaction ||
>> +         ref_transaction_update(transaction, b->name, b->sha1, old_sha1,
>> +                                0, 1)) ||
>
> Would be more idiomatic to drop a layer of parentheses:
>
>         if (!transaction ||
>             ref_transaction_update(...) ||
>
>> +         (ref_transaction_commit(transaction, msg, &err) &&
>> +          !(transaction = NULL))) {
>

Done.


> Would be clearer if ref_transaction_commit didn't take care of the
> rollback (or in other words if patch 21 were earlier in the series).

True, but first making it correct, but ugly, here makes one appreciate
patch 21 more. That patch makes it correct and not ugly :-)
Leaving as is.

>
>> +             ref_transaction_rollback(transaction);
>> +             error("Unable to update branch %s: %s", b->name, err.buf);
>> +             strbuf_release(&err);
>> +             return -1;
>> +     }
>
> Example old message:
>
>         error: Unable to lock refs/heads/master
>
> New message:
>
>         error: Unable to update branch refs/heads/master: Cannot lock the ref 'refs/heads/master'.
>
> So 'error("%s", err.buf)' would probably work better.

Done.

>
> The only call site is dump_branches:
>
>         for (i = 0; i < branch_table_sz; i++) {
>                 for (b = branch_table[i]; b; b = b->table_next_branch)
>                         failure |= update_branch(b);
>         }
>
> Should these happen in a single transaction?  I haven't thought
> through whether it would be a good idea, if it should be optional, or
> what.
>
> Anyway, that would be a bigger behavior change, but it's interesting
> to think about.

I will try to add it to the next patch series.

>
> Thanks,
> Jonathan

Re: [PATCH v8 13/44] tag.c: use ref transactions when doing updates

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 2:11 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/builtin/tag.c
>> +++ b/builtin/tag.c
>> @@ -701,11 +702,12 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
>>       if (annotate)
>>               create_tag(object, tag, &buf, &opt, prev, object);
>>
>> -     lock = lock_any_ref_for_update(ref.buf, prev, 0, NULL);
>> -     if (!lock)
>> -             die(_("%s: cannot lock the ref"), ref.buf);
>> -     if (write_ref_sha1(lock, object, NULL) < 0)
>> -             die(_("%s: cannot update the ref"), ref.buf);
>> +     transaction = ref_transaction_begin();
>> +     if (!transaction ||
>> +         ref_transaction_update(transaction, ref.buf, object, prev,
>> +                                0, !is_null_sha1(prev)) ||
>> +         ref_transaction_commit(transaction, NULL, &err))
>> +             die(_("%s: cannot update the ref: %s"), ref.buf, err.buf);
>
> The error string says what ref it was trying to update, so
>
>                 die("%s", err.buf);
>
> should be enough.  (E.g.,
>
>         fatal: refs/tags/v1.0: cannot lock the ref
>
> would become
>
>         fatal: Cannot lock the ref 'refs/tags/v1.0'.
>
> instead of
>
>         fatal: refs/tags/v1.0: cannot update the ref: Cannot lock the ref 'refs/tags/v1.0'.
>
> .)
>

Done.

Thanks!

Re: [PATCH v8 14/44] replace.c: use the ref transaction functions for updates

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 2:18 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/builtin/replace.c
>> +++ b/builtin/replace.c
> [...]
>> @@ -156,11 +157,12 @@ static int replace_object_sha1(const char *object_ref,
>>       else if (!force)
>>               die("replace ref '%s' already exists", ref);
>>
>> -     lock = lock_any_ref_for_update(ref, prev, 0, NULL);
>> -     if (!lock)
>> -             die("%s: cannot lock the ref", ref);
>> -     if (write_ref_sha1(lock, repl, NULL) < 0)
>> -             die("%s: cannot update the ref", ref);
>> +     transaction = ref_transaction_begin();
>> +     if (!transaction ||
>> +         ref_transaction_update(transaction, ref, repl, prev,
>> +                                0, !is_null_sha1(prev)) ||
>> +         ref_transaction_commit(transaction, NULL, &err))
>> +             die("%s: failed to replace ref: %s", ref, err.buf);
>
> This would write
>
>         fatal: refs/replace/09c779943364d893c190066c385e6112af421db3: failed to replace ref: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.
>
> Perhaps something like
>
>         $ git replace foo bar
>         fatal: replace foo: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.
>
> would make sense (die("replace %s: %s", object_ref, err.buf)).  Plain
>
>         $ git replace foo bar
>         fatal: Cannot lock the ref 'refs/replace/09c779943364d893c190066c385e6112af421db3'.
>
> also seems fine (die("%s", err.buf)).

Done.

I used : die("%s", err.buf)
since this is consistent with most of the other if-transaction-failed-die()

>
> Thanks,
> Jonathan

Re: [PATCH v8 18/44] branch.c: use ref transaction for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change create_branch to use a ref transaction when creating the new branch.
> ref_transaction_create will check that the ref does not already exist and fail
> otherwise meaning that we no longer need to keep a lock on the ref during the
> setup_tracking. This simplifies the code since we can now do the transaction
> in one single step.

Previously the ref lock was also held during setup_tracking, which
sets up configuration for the branch in .git/config.  So in principle
they were one transaction and this patch would change that.

The race:

	checkout -B master origin/master
	--------------------------------
	                        update-ref -d refs/heads/master
	                        -------------------------------
	                                                checkout -B master other/master
	                                                -------------------------------
	create ref
	                        delete ref
	                                                create ref

	configure tracking                              configure tracking

Since setup_tracking is not a single transaction, if the two processes
are lucky enough to not try to write to .git/config at the same time
then the resulting configuration could have branch.master.merge set
by the first checkout -b and branch.master.remote set by the second.

But trying to "checkout -b" the same branch in two terminals
concurrently is a somewhat insane thing to do, so I don't mind
breaking it.

[...]
> This also fixes a race condition in the old code where two concurrent
> create_branch could race since the lock_any_ref_for_update/write_ref_sha1
> did not protect against the ref already existsing. I.e. one thread could end up

s/existsing/existing/

> overwriting a branch even if the forcing flag is false.

Good catch.

> --- a/branch.c
> +++ b/branch.c
[...]
> @@ -301,13 +291,25 @@ void create_branch(const char *head,
[...]
> +		if (!transaction ||
> +			ref_transaction_update(transaction, ref.buf, sha1,
> +					       null_sha1, 0, !forcing) ||
> +			ref_transaction_commit(transaction, msg, &err))
> +				die_errno(_("%s: failed to write ref: %s"),
> +					  ref.buf, err.buf);

errno is not guaranteed valid here.  The usual

				die("%s", err.buf);

should do the trick.

With the changes mentioned above (commit message typofix, die()
message),
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks.

Re: [PATCH v8 04/44] refs.c: add an err argument to repack_without_refs

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 11:38 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2427,12 +2427,12 @@ static int curate_packed_ref_fn(struct ref_entry *entry, void *cb_data)
>>       return 0;
>>  }
>>
>> -static int repack_without_refs(const char **refnames, int n)
>> +static int repack_without_refs(const char **refnames, int n, struct strbuf *err)
>
> Should this also get an onerr flag to suppress the message to stderr,
> or unconditionally suppress the message to stderr when err != NULL?

Unconditionally suppress the message I think.  And is what I did.

>
> [...]
>> @@ -2445,6 +2445,9 @@ static int repack_without_refs(const char **refnames, int n)
>>
>>       if (lock_packed_refs(0)) {
>>               unable_to_lock_error(git_path("packed-refs"), errno);
>> +             if (err)
>> +                     strbuf_addf(err, "cannot delete '%s' from packed refs",
>> +                                 refnames[i]);
>
> unable_to_lock_error is able to come up with a message with more
> detail (path so the sysadmin can hunt down the problem even if this
> was run e.g. from a cronjob where the path is not obvious, errno
> hinting at the nature of the problem).

I changed it so that if err is non-NULL we call a new function that updates the
strbuf with a nice message explaining the failure and then return 1
without writing anything to stderr.

For the NULL case I left it to continue logging two lines of errors
"Unable to create '%s.lock': %s.\n\n"
"cannot delete '%s' from packed refs"

That second line should probably go away since the error at this stage
is related to locking the packed
refs file itself and has nothing really to do with a particular ref
such as refnames[i]



>
> [...]
>> @@ -2470,12 +2473,15 @@ static int repack_without_refs(const char **refnames, int n)
>>       }
>>
>>       /* Write what remains */
>> -     return commit_packed_refs();
>> +     ret = commit_packed_refs();
>> +     if (ret && err)
>> +             strbuf_addf(err, "unable to overwrite old ref-pack file");
>
> After commit_lock_file sets errno, amazingly no one clobbers it
> until we get to this point.  The only calls in between are to
> free().
>
> It would be nice to make that more explicit in commit_packed_refs:
>
>         int save_errno;
>
>         ...
>         if (commit_lock_file(packed_ref_cache->lock)) {
>                 save_errno = errno;
>                 error = -1;
>         }
>
>         packed_ref_cache->lock = NULL;
>         release_packed_ref_cache(packed_ref_cache);
>
>         errno = save_errno;
>         return error;
>
> Even without that, this message could include strerror(errno).

Done and Done.

Thanks.

Re: [PATCH v8 19/44] refs.c: change update_ref to use a transaction

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change the update_ref helper function to use a ref transaction internally.
> 
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 25 +++++++++++++++++++++----
>  1 file changed, 21 insertions(+), 4 deletions(-)
> 
> diff --git a/refs.c b/refs.c
> index 6e5e940..0476892 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -3416,11 +3416,28 @@ int update_ref(const char *action, const char *refname,
>  	       const unsigned char *sha1, const unsigned char *oldval,
>  	       int flags, enum action_on_err onerr)
>  {
> -	struct ref_lock *lock;
> -	lock = update_ref_lock(refname, oldval, flags, NULL, onerr);
> -	if (!lock)
> +	struct ref_transaction *t;
> +	struct strbuf err = STRBUF_INIT;
> +
> +	t = ref_transaction_begin();
> +	if ((!t ||
> +	    ref_transaction_update(t, refname, sha1, oldval, flags,
> +				   !!oldval)) ||
> +	    (ref_transaction_commit(t, action, &err) && !(t = NULL))) {
> +		const char *str = "update_ref failed for ref '%s': %s";
> +
> +		ref_transaction_rollback(t);
> +		switch (onerr) {
> +		case UPDATE_REFS_MSG_ON_ERR:
> +			error(str, refname, err.buf); break;
> +		case UPDATE_REFS_DIE_ON_ERR:
> +			die(str, refname, err.buf); break;
> +		case UPDATE_REFS_QUIET_ON_ERR: break;
> +		}

The error string already contains the refname, so it would make sense to
do

		case UPDATE_REFS_MSG_ON_ERR:
			error("%s", err.buf); break;

etc here.

It occurs to me now that if ref_transaction_begin fails, presumably
the error string would not contain the refname.  I want to say that
the refname wouldn't help much in diagnosing or recovering from the
problem in that case so it would still be okay to just do error("%s",
err.buf).  What do you think?

Thanks,
Jonathan

Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change ref_transaction_commit so that it does not free the transaction.
> Instead require that a caller will end a transaction by either calling
> ref_transaction_rollback or ref_transaction_free.

Can I always use ref_transaction_rollback instead of ref_transaction_free?
It would be convenient if my cleanup code could always call _rollback
instead of having to do something different for success and errors.

Another way to ask the question: what is it valid to do with a
transaction after commiting?

Thanks,
Jonathan

Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction

[Ronnie Sahlberg]

On Thu, May 15, 2014 at 5:20 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Change ref_transaction_commit so that it does not free the transaction.
>> Instead require that a caller will end a transaction by either calling
>> ref_transaction_rollback or ref_transaction_free.
>
> Can I always use ref_transaction_rollback instead of ref_transaction_free?
> It would be convenient if my cleanup code could always call _rollback
> instead of having to do something different for success and errors.

Currently, yes.

But it might make sense to change these so rollback only clears the
updates that are in flight from the transaction and
free only frees the transaction itself iff there are no updates in flight.

I.e. the success and error would then differ like this :
...
   if (transaction_commit()) {
       transaction_rollback()
       transaction_free()
       return error("some error")
   }
   transaction_free()



>
> Another way to ask the question: what is it valid to do with a
> transaction after commiting?

Right now the only valid thing to do is either rollback or free. But
we could allow other things too :


re-usable transactions.
---------------------------------
I don't know if this is a good reason or not, but one reason we might
want to keep
two different names could be if we want to start allowing to re-use
transactions.
For example for cases/backends where transaction_begin() might be very
expensive.

For that case I would imagine we could allow to do things such as

t = transaction_begin()
...
/* first transaction */
transaction_update(...)
transaction_commit(...)
     if transaction failed   transaction_rollback(...)

/* second transaction,  first transaction cleared all updates in
flight either through commit or through rollback */
transaction_update()
transaction_commit()
     if transaction failed   transaction_rollback(...)
...
transaction_free()


(In order to do something like this we would still need to do some
changes so that rollback will only free the updates that were in
flight but not free the transaction itself.)


>
> Thanks,
> Jonathan

Re: [PATCH v8 21/44] refs.c: ref_transaction_commit should not free the transaction

[Ronnie Sahlberg]

On Fri, May 16, 2014 at 8:02 AM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> On Thu, May 15, 2014 at 5:20 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Ronnie Sahlberg wrote:
>>
>>> Change ref_transaction_commit so that it does not free the transaction.
>>> Instead require that a caller will end a transaction by either calling
>>> ref_transaction_rollback or ref_transaction_free.
>>
>> Can I always use ref_transaction_rollback instead of ref_transaction_free?
>> It would be convenient if my cleanup code could always call _rollback
>> instead of having to do something different for success and errors.
>
> Currently, yes.
>
> But it might make sense to change these so rollback only clears the
> updates that are in flight from the transaction and
> free only frees the transaction itself iff there are no updates in flight.
>
> I.e. the success and error would then differ like this :
> ...
>    if (transaction_commit()) {
>        transaction_rollback()
>        transaction_free()
>        return error("some error")
>    }
>    transaction_free()
>

But we do not really need rollback right now. If / when we decide to
need re-startable/re-usable transactions we can add it back when
needed.

Let me update the patch series and remove transaction_rollback and
replace all calls to it with calls to transaction_free instead.


>
>
>>
>> Another way to ask the question: what is it valid to do with a
>> transaction after commiting?
>
> Right now the only valid thing to do is either rollback or free. But
> we could allow other things too :
>
>
> re-usable transactions.
> ---------------------------------
> I don't know if this is a good reason or not, but one reason we might
> want to keep
> two different names could be if we want to start allowing to re-use
> transactions.
> For example for cases/backends where transaction_begin() might be very
> expensive.
>
> For that case I would imagine we could allow to do things such as
>
> t = transaction_begin()
> ...
> /* first transaction */
> transaction_update(...)
> transaction_commit(...)
>      if transaction failed   transaction_rollback(...)
>
> /* second transaction,  first transaction cleared all updates in
> flight either through commit or through rollback */
> transaction_update()
> transaction_commit()
>      if transaction failed   transaction_rollback(...)
> ...
> transaction_free()
>
>
> (In order to do something like this we would still need to do some
> changes so that rollback will only free the updates that were in
> flight but not free the transaction itself.)
>
>
>>
>> Thanks,
>> Jonathan

Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it

[Jonathan Nieder]

(cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case
 from long ago)
Ronnie Sahlberg wrote:

> In s_update_ref there are two calls that when they fail we return an error
> based on the errno value. In particular we want to return a specific error
> if ENOTDIR happened. Both these functions do have failure modes where they
> may return an error without updating errno

That's a bug.  Any function for which errno is supposed to be
meaningful when it returns an error should always set errno.
Otherwise errno may be set to ENOTDIR within the function by an
unrelated system call.

Functions that fail and lead callers to check errno, based on a quick
search with 'git grep -e 'errno *[!=]=':

 fopen
 lstat
 builtin/apply.c::try_create_file (= mkdir, symlink, or open)
 rmdir
 open
 mkdir
 unlink
 lock_any_ref_for_update 
 write_ref_sha1
 strtol
 kill
 odb_pack_keep
 opendir
 fgets
 read
 poll
 pread
 strtoimax
 strtoumax
 git_parse_int
 git_parse_int64
 git_parse_ulong
 write_in_full
 credential-cache.c::send_request
 fstat
 run_command_v_opt
 git.c::run_argv
 readlink
 resolve_ref_unsafe
 hold_lock_file_for_update
 unlink_or_warn
 rename
 execvp
 waitpid
 execv_git_cmd
 execv_shell_cmd
 sane_execvp
 stat
 read_object
 git_mkstemp_mode
 create_tmpfile
 start_command
 xwrite
 iconv
 fast_export_ls
 fast_export_ls_rev
 delete_ref

lock_any_ref_for_update has failure paths
 * check_ref_format [!]
 * lock_ref_sha1_basic

lock_ref_sha1_basic has failure paths
 * remove_empty_directories
 * resolve_ref_unsafe
 * safe_create_leading_directories
 * verify_lock

remove_empty_directories has one failure path
 * remove_dir_recursively
but could be more explicit about the need to preserve errno.

errno from remove_dir_recursively is meaningful.

resolve_ref_unsafe has failure paths
 * check_refname_format [!]
 * too much recursion [!]
 * lstat, readlink, open
 * handle_missing_loose_ref
 * read_in_full, but errno gets clobbered
 * invalid ref [!]
 * invalid symref [!]

verify_lock has failure paths
 * read_ref_full, but errno gets clobbered
 * old_sha1 didn't match [!]
 
write_ref_sha1 has failure paths
 * !lock [!]
 * missing object [!]
 * non-commit object [!]
 * write_in_full, close_ref, but errno gets clobbered
 * log_ref_write
 * commit_ref

log_ref_write has failure paths
 * log_ref_setup
 * write_in_full, close.  errno gets clobbered

commit_ref just calls commit_lock_file.

log_ref_setup has failure paths
 * safe_create_leading_directories, but errno gets clobbered
 * remove_empty_directories, but errno gets clobbered
 * open, but errno gets clobbered

safe_create_leading_directories doesn't set errno for SCLD_EXISTS
but otherwise its errno is useful.

That should cover the cases affecting the code path in fetch.c you
mentioned (I haven't checked the others).

How about something like this?

It's also tempting to teach vreportf and vwritef to save errno, which
would handle some of these cases automatically.

diff --git i/refs.c w/refs.c
index 82a8d4e..f98acf0 100644
--- i/refs.c
+++ w/refs.c
@@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
 	if (flag)
 		*flag = 0;
 
-	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
+	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
+		errno = EINVAL;
 		return NULL;
+	}
 
 	for (;;) {
 		char path[PATH_MAX];
@@ -1342,8 +1344,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
 		char *buf;
 		int fd;
 
-		if (--depth < 0)
+		if (--depth < 0) {
+			errno = ELOOP;
 			return NULL;
+		}
 
 		git_snpath(path, sizeof(path), "%s", refname);
 
@@ -1405,9 +1409,13 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
 				return NULL;
 		}
 		len = read_in_full(fd, buffer, sizeof(buffer)-1);
-		close(fd);
-		if (len < 0)
+		if (len < 0) {
+			int save_errno = errno;
+			close(fd);
+			errno = save_errno;
 			return NULL;
+		}
+		close(fd);
 		while (len && isspace(buffer[len-1]))
 			len--;
 		buffer[len] = '\0';
@@ -1424,6 +1432,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
 			    (buffer[40] != '\0' && !isspace(buffer[40]))) {
 				if (flag)
 					*flag |= REF_ISBROKEN;
+				errno = EINVAL;
 				return NULL;
 			}
 			return refname;
@@ -1436,6 +1445,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
 		if (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {
 			if (flag)
 				*flag |= REF_ISBROKEN;
+			errno = EINVAL;
 			return NULL;
 		}
 		refname = strcpy(refname_buffer, buf);
@@ -1908,14 +1918,17 @@ static struct ref_lock *verify_lock(struct ref_lock *lock,
 	const unsigned char *old_sha1, int mustexist)
 {
 	if (read_ref_full(lock->ref_name, lock->old_sha1, mustexist, NULL)) {
+		int save_errno = errno;
 		error("Can't verify ref %s", lock->ref_name);
 		unlock_ref(lock);
+		errno = save_errno;
 		return NULL;
 	}
 	if (hashcmp(lock->old_sha1, old_sha1)) {
 		error("Ref %s is at %s but expected %s", lock->ref_name,
 			sha1_to_hex(lock->old_sha1), sha1_to_hex(old_sha1));
 		unlock_ref(lock);
+		errno = EBUSY;
 		return NULL;
 	}
 	return lock;
@@ -1928,15 +1941,17 @@ static int remove_empty_directories(const char *file)
 	 * only empty directories), remove them.
 	 */
 	struct strbuf path;
-	int result;
+	int result, save_errno;
 
 	strbuf_init(&path, 20);
 	strbuf_addstr(&path, file);
 
 	result = remove_dir_recursively(&path, REMOVE_DIR_EMPTY_ONLY);
+	save_errno = errno;
 
 	strbuf_release(&path);
 
+	errno = save_errno;
 	return result;
 }
 
@@ -2137,8 +2152,10 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,
 					 const unsigned char *old_sha1,
 					 int flags, int *type_p)
 {
-	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
+	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
+		errno = EINVAL;
 		return NULL;
+	}
 	return lock_ref_sha1_basic(refname, old_sha1, flags, type_p);
 }
 
@@ -2734,9 +2751,12 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)
 	     starts_with(refname, "refs/remotes/") ||
 	     starts_with(refname, "refs/notes/") ||
 	     !strcmp(refname, "HEAD"))) {
-		if (safe_create_leading_directories(logfile) < 0)
-			return error("unable to create directory for %s",
-				     logfile);
+		if (safe_create_leading_directories(logfile) < 0) {
+			int save_errno = errno;
+			error("unable to create directory for %s", logfile);
+			errno = save_errno;
+			return -1;
+		}
 		oflags |= O_CREAT;
 	}
 
@@ -2747,15 +2767,21 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)
 
 		if ((oflags & O_CREAT) && errno == EISDIR) {
 			if (remove_empty_directories(logfile)) {
-				return error("There are still logs under '%s'",
-					     logfile);
+				int save_errno = errno;
+				error("There are still logs under '%s'", logfile);
+				errno = save_errno;
+				return -1;
 			}
 			logfd = open(logfile, oflags, 0666);
 		}
 
-		if (logfd < 0)
-			return error("Unable to append to %s: %s",
+		if (logfd < 0) {
+			int save_errno = errno;
+			error("Unable to append to %s: %s",
 				     logfile, strerror(errno));
+			errno = save_errno;
+			return -1;
+		}
 	}
 
 	adjust_shared_perm(logfile);
@@ -2795,8 +2821,19 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,
 		len += copy_msg(logrec + len - 1, msg) - 1;
 	written = len <= maxlen ? write_in_full(logfd, logrec, len) : -1;
 	free(logrec);
-	if (close(logfd) != 0 || written != len)
-		return error("Unable to append to %s", log_file);
+	if (written != len) {
+		int save_errno = errno;
+		close(logfd);
+		error("Unable to append to %s", log_file);
+		errno = save_errno;
+		return -1;
+	}
+	if (close(logfd)) {
+		int save_errno = errno;
+		error("Unable to append to %s", log_file);
+		errno = save_errno;
+		return -1;
+	}
 	return 0;
 }
 
@@ -2811,8 +2848,10 @@ int write_ref_sha1(struct ref_lock *lock,
 	static char term = '\n';
 	struct object *o;
 
-	if (!lock)
+	if (!lock) {
+		errno = EINVAL;
 		return -1;
+	}
 	if (!lock->force_write && !hashcmp(lock->old_sha1, sha1)) {
 		unlock_ref(lock);
 		return 0;
@@ -2822,26 +2861,32 @@ int write_ref_sha1(struct ref_lock *lock,
 		error("Trying to write ref %s with nonexistent object %s",
 			lock->ref_name, sha1_to_hex(sha1));
 		unlock_ref(lock);
+		errno = EINVAL;
 		return -1;
 	}
 	if (o->type != OBJ_COMMIT && is_branch(lock->ref_name)) {
 		error("Trying to write non-commit object %s to branch %s",
 			sha1_to_hex(sha1), lock->ref_name);
 		unlock_ref(lock);
+		errno = EINVAL;
 		return -1;
 	}
 	if (write_in_full(lock->lock_fd, sha1_to_hex(sha1), 40) != 40 ||
-	    write_in_full(lock->lock_fd, &term, 1) != 1
-		|| close_ref(lock) < 0) {
+	    write_in_full(lock->lock_fd, &term, 1) != 1 ||
+	    close_ref(lock) < 0) {
+		int save_errno = errno;
 		error("Couldn't write %s", lock->lk->filename);
 		unlock_ref(lock);
+		errno = save_errno;
 		return -1;
 	}
 	clear_loose_ref_cache(&ref_cache);
 	if (log_ref_write(lock->ref_name, lock->old_sha1, sha1, logmsg) < 0 ||
 	    (strcmp(lock->ref_name, lock->orig_ref_name) &&
 	     log_ref_write(lock->orig_ref_name, lock->old_sha1, sha1, logmsg) < 0)) {
+		int save_errno = errno;
 		unlock_ref(lock);
+		errno = save_errno;
 		return -1;
 	}
 	if (strcmp(lock->orig_ref_name, "HEAD") != 0) {
@@ -2866,8 +2911,10 @@ int write_ref_sha1(struct ref_lock *lock,
 			log_ref_write("HEAD", lock->old_sha1, sha1, logmsg);
 	}
 	if (commit_ref(lock)) {
+		int save_errno = errno;
 		error("Couldn't set %s", lock->ref_name);
 		unlock_ref(lock);
+		errno = save_errno;
 		return -1;
 	}
 	unlock_ref(lock);
diff --git i/sha1_file.c w/sha1_file.c
index 3e9f55f..a7bbba7 100644
--- i/sha1_file.c
+++ w/sha1_file.c
@@ -136,8 +136,10 @@ enum scld_error safe_create_leading_directories(char *path)
 		*slash = '\0';
 		if (!stat(path, &st)) {
 			/* path exists */
-			if (!S_ISDIR(st.st_mode))
+			if (!S_ISDIR(st.st_mode)) {
+				errno = EEXIST;
 				ret = SCLD_EXISTS;
+			}
 		} else if (mkdir(path, 0777)) {
 			if (errno == EEXIST &&
 			    !stat(path, &st) && S_ISDIR(st.st_mode))

Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction

[Jonathan Nieder]

(+cc: peff for STORE_REF_ERROR_DF_CONFLICT expertise)
Ronnie Sahlberg wrote:

> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -375,7 +375,7 @@ static int s_update_ref(const char *action,
[...]
> +	transaction = ref_transaction_begin();
> +	if (!transaction ||
> +	    ref_transaction_update(transaction, ref->name, ref->new_sha1,
> +				   ref->old_sha1, 0, check_old) ||
> +	    ref_transaction_commit(transaction, msg, NULL)) {
> +		ref_transaction_rollback(transaction);
>  		return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
>  					  STORE_REF_ERROR_OTHER;
> +	}

I'd rather not rely on errno here (see the previous patch for why).
Is there some other way to distinguish the case where a ref couldn't
be created because there was a prefix of that ref in the way?

For example, maybe ref_transaction_commit could return a different
negative integer in this case.

Thanks,
Jonathan

Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it

[Ronnie Sahlberg]

errno is hairy :-(


You probably also want something like this :
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 4603cb6..55e7dd8 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -426,7 +426,7 @@ static int update_local_ref(struct ref *ref,
                            _("! %-*s %-*s -> %s  (can't fetch in current branch
                            TRANSPORT_SUMMARY(_("[rejected]")),
                            REFCOL_WIDTH, remote, pretty_ref);
-               return 1;
+               return STORE_REF_ERROR_OTHER;
        }

        if (!is_null_sha1(ref->old_sha1) &&
@@ -513,7 +513,7 @@ static int update_local_ref(struct ref *ref,
                            TRANSPORT_SUMMARY(_("[rejected]")),
                            REFCOL_WIDTH, remote, pretty_ref,
                            _("(non-fast-forward)"));
-               return 1;
+               return STORE_REF_ERROR_OTHER;
        }
 }


To make it more clear that this function returns a specific error and
not just a generic not-zero.


On Fri, May 16, 2014 at 11:33 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> (cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case
>  from long ago)
> Ronnie Sahlberg wrote:
>
>> In s_update_ref there are two calls that when they fail we return an error
>> based on the errno value. In particular we want to return a specific error
>> if ENOTDIR happened. Both these functions do have failure modes where they
>> may return an error without updating errno
>
> That's a bug.  Any function for which errno is supposed to be
> meaningful when it returns an error should always set errno.
> Otherwise errno may be set to ENOTDIR within the function by an
> unrelated system call.
>
> Functions that fail and lead callers to check errno, based on a quick
> search with 'git grep -e 'errno *[!=]=':
>
>  fopen
>  lstat
>  builtin/apply.c::try_create_file (= mkdir, symlink, or open)
>  rmdir
>  open
>  mkdir
>  unlink
>  lock_any_ref_for_update
>  write_ref_sha1
>  strtol
>  kill
>  odb_pack_keep
>  opendir
>  fgets
>  read
>  poll
>  pread
>  strtoimax
>  strtoumax
>  git_parse_int
>  git_parse_int64
>  git_parse_ulong
>  write_in_full
>  credential-cache.c::send_request
>  fstat
>  run_command_v_opt
>  git.c::run_argv
>  readlink
>  resolve_ref_unsafe
>  hold_lock_file_for_update
>  unlink_or_warn
>  rename
>  execvp
>  waitpid
>  execv_git_cmd
>  execv_shell_cmd
>  sane_execvp
>  stat
>  read_object
>  git_mkstemp_mode
>  create_tmpfile
>  start_command
>  xwrite
>  iconv
>  fast_export_ls
>  fast_export_ls_rev
>  delete_ref
>
> lock_any_ref_for_update has failure paths
>  * check_ref_format [!]
>  * lock_ref_sha1_basic
>
> lock_ref_sha1_basic has failure paths
>  * remove_empty_directories
>  * resolve_ref_unsafe
>  * safe_create_leading_directories
>  * verify_lock
>
> remove_empty_directories has one failure path
>  * remove_dir_recursively
> but could be more explicit about the need to preserve errno.
>
> errno from remove_dir_recursively is meaningful.
>
> resolve_ref_unsafe has failure paths
>  * check_refname_format [!]
>  * too much recursion [!]
>  * lstat, readlink, open
>  * handle_missing_loose_ref
>  * read_in_full, but errno gets clobbered
>  * invalid ref [!]
>  * invalid symref [!]
>
> verify_lock has failure paths
>  * read_ref_full, but errno gets clobbered
>  * old_sha1 didn't match [!]
>
> write_ref_sha1 has failure paths
>  * !lock [!]
>  * missing object [!]
>  * non-commit object [!]
>  * write_in_full, close_ref, but errno gets clobbered
>  * log_ref_write
>  * commit_ref
>
> log_ref_write has failure paths
>  * log_ref_setup
>  * write_in_full, close.  errno gets clobbered
>
> commit_ref just calls commit_lock_file.
>
> log_ref_setup has failure paths
>  * safe_create_leading_directories, but errno gets clobbered
>  * remove_empty_directories, but errno gets clobbered
>  * open, but errno gets clobbered
>
> safe_create_leading_directories doesn't set errno for SCLD_EXISTS
> but otherwise its errno is useful.
>
> That should cover the cases affecting the code path in fetch.c you
> mentioned (I haven't checked the others).
>
> How about something like this?
>
> It's also tempting to teach vreportf and vwritef to save errno, which
> would handle some of these cases automatically.
>
> diff --git i/refs.c w/refs.c
> index 82a8d4e..f98acf0 100644
> --- i/refs.c
> +++ w/refs.c
> @@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>         if (flag)
>                 *flag = 0;
>
> -       if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
> +       if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
> +               errno = EINVAL;
>                 return NULL;
> +       }
>
>         for (;;) {
>                 char path[PATH_MAX];
> @@ -1342,8 +1344,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>                 char *buf;
>                 int fd;
>
> -               if (--depth < 0)
> +               if (--depth < 0) {
> +                       errno = ELOOP;
>                         return NULL;
> +               }
>
>                 git_snpath(path, sizeof(path), "%s", refname);
>
> @@ -1405,9 +1409,13 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>                                 return NULL;
>                 }
>                 len = read_in_full(fd, buffer, sizeof(buffer)-1);
> -               close(fd);
> -               if (len < 0)
> +               if (len < 0) {
> +                       int save_errno = errno;
> +                       close(fd);
> +                       errno = save_errno;
>                         return NULL;
> +               }
> +               close(fd);
>                 while (len && isspace(buffer[len-1]))
>                         len--;
>                 buffer[len] = '\0';
> @@ -1424,6 +1432,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>                             (buffer[40] != '\0' && !isspace(buffer[40]))) {
>                                 if (flag)
>                                         *flag |= REF_ISBROKEN;
> +                               errno = EINVAL;
>                                 return NULL;
>                         }
>                         return refname;
> @@ -1436,6 +1445,7 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>                 if (check_refname_format(buf, REFNAME_ALLOW_ONELEVEL)) {
>                         if (flag)
>                                 *flag |= REF_ISBROKEN;
> +                       errno = EINVAL;
>                         return NULL;
>                 }
>                 refname = strcpy(refname_buffer, buf);
> @@ -1908,14 +1918,17 @@ static struct ref_lock *verify_lock(struct ref_lock *lock,
>         const unsigned char *old_sha1, int mustexist)
>  {
>         if (read_ref_full(lock->ref_name, lock->old_sha1, mustexist, NULL)) {
> +               int save_errno = errno;
>                 error("Can't verify ref %s", lock->ref_name);
>                 unlock_ref(lock);
> +               errno = save_errno;
>                 return NULL;
>         }
>         if (hashcmp(lock->old_sha1, old_sha1)) {
>                 error("Ref %s is at %s but expected %s", lock->ref_name,
>                         sha1_to_hex(lock->old_sha1), sha1_to_hex(old_sha1));
>                 unlock_ref(lock);
> +               errno = EBUSY;
>                 return NULL;
>         }
>         return lock;
> @@ -1928,15 +1941,17 @@ static int remove_empty_directories(const char *file)
>          * only empty directories), remove them.
>          */
>         struct strbuf path;
> -       int result;
> +       int result, save_errno;
>
>         strbuf_init(&path, 20);
>         strbuf_addstr(&path, file);
>
>         result = remove_dir_recursively(&path, REMOVE_DIR_EMPTY_ONLY);
> +       save_errno = errno;
>
>         strbuf_release(&path);
>
> +       errno = save_errno;
>         return result;
>  }
>
> @@ -2137,8 +2152,10 @@ struct ref_lock *lock_any_ref_for_update(const char *refname,
>                                          const unsigned char *old_sha1,
>                                          int flags, int *type_p)
>  {
> -       if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
> +       if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
> +               errno = EINVAL;
>                 return NULL;
> +       }
>         return lock_ref_sha1_basic(refname, old_sha1, flags, type_p);
>  }
>
> @@ -2734,9 +2751,12 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)
>              starts_with(refname, "refs/remotes/") ||
>              starts_with(refname, "refs/notes/") ||
>              !strcmp(refname, "HEAD"))) {
> -               if (safe_create_leading_directories(logfile) < 0)
> -                       return error("unable to create directory for %s",
> -                                    logfile);
> +               if (safe_create_leading_directories(logfile) < 0) {
> +                       int save_errno = errno;
> +                       error("unable to create directory for %s", logfile);
> +                       errno = save_errno;
> +                       return -1;
> +               }
>                 oflags |= O_CREAT;
>         }
>
> @@ -2747,15 +2767,21 @@ int log_ref_setup(const char *refname, char *logfile, int bufsize)
>
>                 if ((oflags & O_CREAT) && errno == EISDIR) {
>                         if (remove_empty_directories(logfile)) {
> -                               return error("There are still logs under '%s'",
> -                                            logfile);
> +                               int save_errno = errno;
> +                               error("There are still logs under '%s'", logfile);
> +                               errno = save_errno;
> +                               return -1;
>                         }
>                         logfd = open(logfile, oflags, 0666);
>                 }
>
> -               if (logfd < 0)
> -                       return error("Unable to append to %s: %s",
> +               if (logfd < 0) {
> +                       int save_errno = errno;
> +                       error("Unable to append to %s: %s",
>                                      logfile, strerror(errno));
> +                       errno = save_errno;
> +                       return -1;
> +               }
>         }
>
>         adjust_shared_perm(logfile);
> @@ -2795,8 +2821,19 @@ static int log_ref_write(const char *refname, const unsigned char *old_sha1,
>                 len += copy_msg(logrec + len - 1, msg) - 1;
>         written = len <= maxlen ? write_in_full(logfd, logrec, len) : -1;
>         free(logrec);
> -       if (close(logfd) != 0 || written != len)
> -               return error("Unable to append to %s", log_file);
> +       if (written != len) {
> +               int save_errno = errno;
> +               close(logfd);
> +               error("Unable to append to %s", log_file);
> +               errno = save_errno;
> +               return -1;
> +       }
> +       if (close(logfd)) {
> +               int save_errno = errno;
> +               error("Unable to append to %s", log_file);
> +               errno = save_errno;
> +               return -1;
> +       }
>         return 0;
>  }
>
> @@ -2811,8 +2848,10 @@ int write_ref_sha1(struct ref_lock *lock,
>         static char term = '\n';
>         struct object *o;
>
> -       if (!lock)
> +       if (!lock) {
> +               errno = EINVAL;
>                 return -1;
> +       }
>         if (!lock->force_write && !hashcmp(lock->old_sha1, sha1)) {
>                 unlock_ref(lock);
>                 return 0;
> @@ -2822,26 +2861,32 @@ int write_ref_sha1(struct ref_lock *lock,
>                 error("Trying to write ref %s with nonexistent object %s",
>                         lock->ref_name, sha1_to_hex(sha1));
>                 unlock_ref(lock);
> +               errno = EINVAL;
>                 return -1;
>         }
>         if (o->type != OBJ_COMMIT && is_branch(lock->ref_name)) {
>                 error("Trying to write non-commit object %s to branch %s",
>                         sha1_to_hex(sha1), lock->ref_name);
>                 unlock_ref(lock);
> +               errno = EINVAL;
>                 return -1;
>         }
>         if (write_in_full(lock->lock_fd, sha1_to_hex(sha1), 40) != 40 ||
> -           write_in_full(lock->lock_fd, &term, 1) != 1
> -               || close_ref(lock) < 0) {
> +           write_in_full(lock->lock_fd, &term, 1) != 1 ||
> +           close_ref(lock) < 0) {
> +               int save_errno = errno;
>                 error("Couldn't write %s", lock->lk->filename);
>                 unlock_ref(lock);
> +               errno = save_errno;
>                 return -1;
>         }
>         clear_loose_ref_cache(&ref_cache);
>         if (log_ref_write(lock->ref_name, lock->old_sha1, sha1, logmsg) < 0 ||
>             (strcmp(lock->ref_name, lock->orig_ref_name) &&
>              log_ref_write(lock->orig_ref_name, lock->old_sha1, sha1, logmsg) < 0)) {
> +               int save_errno = errno;
>                 unlock_ref(lock);
> +               errno = save_errno;
>                 return -1;
>         }
>         if (strcmp(lock->orig_ref_name, "HEAD") != 0) {
> @@ -2866,8 +2911,10 @@ int write_ref_sha1(struct ref_lock *lock,
>                         log_ref_write("HEAD", lock->old_sha1, sha1, logmsg);
>         }
>         if (commit_ref(lock)) {
> +               int save_errno = errno;
>                 error("Couldn't set %s", lock->ref_name);
>                 unlock_ref(lock);
> +               errno = save_errno;
>                 return -1;
>         }
>         unlock_ref(lock);
> diff --git i/sha1_file.c w/sha1_file.c
> index 3e9f55f..a7bbba7 100644
> --- i/sha1_file.c
> +++ w/sha1_file.c
> @@ -136,8 +136,10 @@ enum scld_error safe_create_leading_directories(char *path)
>                 *slash = '\0';
>                 if (!stat(path, &st)) {
>                         /* path exists */
> -                       if (!S_ISDIR(st.st_mode))
> +                       if (!S_ISDIR(st.st_mode)) {
> +                               errno = EEXIST;
>                                 ret = SCLD_EXISTS;
> +                       }
>                 } else if (mkdir(path, 0777)) {
>                         if (errno == EEXIST &&
>                             !stat(path, &st) && S_ISDIR(st.st_mode))

Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction

[Ronnie Sahlberg]

On Fri, May 16, 2014 at 12:12 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> (+cc: peff for STORE_REF_ERROR_DF_CONFLICT expertise)
> Ronnie Sahlberg wrote:
>
>> --- a/builtin/fetch.c
>> +++ b/builtin/fetch.c
>> @@ -375,7 +375,7 @@ static int s_update_ref(const char *action,
> [...]
>> +     transaction = ref_transaction_begin();
>> +     if (!transaction ||
>> +         ref_transaction_update(transaction, ref->name, ref->new_sha1,
>> +                                ref->old_sha1, 0, check_old) ||
>> +         ref_transaction_commit(transaction, msg, NULL)) {
>> +             ref_transaction_rollback(transaction);
>>               return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
>>                                         STORE_REF_ERROR_OTHER;
>> +     }
>
> I'd rather not rely on errno here (see the previous patch for why).
> Is there some other way to distinguish the case where a ref couldn't
> be created because there was a prefix of that ref in the way?
>
> For example, maybe ref_transaction_commit could return a different
> negative integer in this case.

I have changed it to make transaction_commit will return a special
error if there is a name conflict
and fetch.c now uses that instead of looking at errno to decide if to
print the "try pruning" message.


>
> Thanks,
> Jonathan

Re: [PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change store_updated_refs to use a single ref transaction for all refs that
> are updated during the fetch. This makes the fetch more atomic when update
> failures occur.

Fun.

[...]
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
[...]
> @@ -373,27 +374,13 @@ static int s_update_ref(const char *action,
>  			struct ref *ref,
>  			int check_old)
>  {
> -	char msg[1024];
> -	char *rla = getenv("GIT_REFLOG_ACTION");
> -	struct ref_transaction *transaction;

Previously fetch respected GIT_REFLOG_ACTION, and this is dropping
that support.  Intended?

[...]
> +	if (ref_transaction_update(transaction, ref->name, ref->new_sha1,
> +			       ref->old_sha1, 0, check_old))
> +		return STORE_REF_ERROR_OTHER;

Should pass a strbuf in to get a message back.

[...]
> +
>  	return 0;
>  }
>  
> @@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
>  		goto abort;
>  	}
>  
> +	errno = 0;
> +	transaction = ref_transaction_begin();
> +	if (!transaction) {
> +		rc = error(_("cannot start ref transaction\n"));
> +		goto abort;

error() appends a newline on its own, so the message shouldn't
end with newline.

More importantly, this message isn't helpful without more detail about
why _transaction_begin() failed.  The user doesn't know what

	error: cannot start ref transaction

means.

	error: cannot connect to mysqld for ref storage: [etc]

would tell what they need to know.  That is:

	transaction = ref_transaction_begin(err);
	if (!transaction) {
		rc = error("%s", err.buf);
		goto abort;
	}

errno is not used here, so no need to set errno = 0.

[...]
> @@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
>  			}
>  		}
>  	}
> +	if ((ret = ref_transaction_commit(transaction, NULL)))
> +		rc |= ret == -2 ? STORE_REF_ERROR_DF_CONFLICT :
> +				STORE_REF_ERROR_OTHER;

(I cheated and stole the newer code from your repo.)

Yay!  Style nit: git avoids assignments in "if" conditionals.

	ret = ref_tr...
	if (ret == -2)
		rc |= ...
	else if (ret)
		rc |= ...

Does _update need the too as futureproofing for backends that can
detect the name conflict sooner?

Thanks,
Jonathan

Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
[...]
> @@ -384,15 +384,16 @@ static int s_update_ref(const char *action,
>  	snprintf(msg, sizeof(msg), "%s: %s", rla, action);
>  
>  	errno = 0;
> -	lock = lock_any_ref_for_update(ref->name,
> -				       check_old ? ref->old_sha1 : NULL,
> -				       0, NULL);
> -	if (!lock)
> -		return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
> -					  STORE_REF_ERROR_OTHER;
> -	if (write_ref_sha1(lock, ref->new_sha1, msg) < 0)
> +	transaction = ref_transaction_begin();
> +	if (!transaction ||
> +	    ref_transaction_update(transaction, ref->name, ref->new_sha1,
> +				   ref->old_sha1, 0, check_old) ||
> +	    ref_transaction_commit(transaction, msg, NULL)) {

Since 'err' is NULL, does that mean there's no message shown to the
user on error?

Re: [PATCH v8 22/44] fetch.c: clear errno before calling functions that might set it

[Jeff King]

On Fri, May 16, 2014 at 11:33:48AM -0700, Jonathan Nieder wrote:

> (cc-ing peff, who may remember this STORE_REF_ERROR_DF_CONFLICT case
>  from long ago)

No, but I have very good tools for searching the list archive. ;)

> > In s_update_ref there are two calls that when they fail we return an error
> > based on the errno value. In particular we want to return a specific error
> > if ENOTDIR happened. Both these functions do have failure modes where they
> > may return an error without updating errno
> 
> That's a bug.  Any function for which errno is supposed to be
> meaningful when it returns an error should always set errno.
> Otherwise errno may be set to ENOTDIR within the function by an
> unrelated system call.

I'd agree. It's OK for a function not to set errno if it is
_successful_. But if setting errno on error is part of the interface for
lock_any_ref_for_update, it should do so consistently.

We have not been very good about documenting which functions use errno,
or using it consistently (e.g., in my original patch, we are explicitly
converting errno into a numeric return value!), so I can definitely see
how it is confusing.

> That should cover the cases affecting the code path in fetch.c you
> mentioned (I haven't checked the others).
> 
> How about something like this?
> [...]

Yeah, I think this is a better direction.

> diff --git i/refs.c w/refs.c
> index 82a8d4e..f98acf0 100644
> --- i/refs.c
> +++ w/refs.c
> @@ -1333,8 +1333,10 @@ const char *resolve_ref_unsafe(const char *refname, unsigned char *sha1, int rea
>  	if (flag)
>  		*flag = 0;
>  
> -	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
> +	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL)) {
> +		errno = EINVAL;
>  		return NULL;
> +	}

check_refname_format comes up in a few places. I wonder if it should set
EINVAL itself.

Other than that, I just skimmed through your list. All looked
reasonable to me, though I was not thorough enough to be sure we got
call cases (OTOH, we can add them later on top).

-Peff

Re: [PATCH v8 24/44] fetch.c: use a single ref transaction for all ref updates

[Ronnie Sahlberg]

On Fri, May 16, 2014 at 3:52 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Change store_updated_refs to use a single ref transaction for all refs that
>> are updated during the fetch. This makes the fetch more atomic when update
>> failures occur.
>
> Fun.
>
> [...]
>> --- a/builtin/fetch.c
>> +++ b/builtin/fetch.c
> [...]
>> @@ -373,27 +374,13 @@ static int s_update_ref(const char *action,
>>                       struct ref *ref,
>>                       int check_old)
>>  {
>> -     char msg[1024];
>> -     char *rla = getenv("GIT_REFLOG_ACTION");
>> -     struct ref_transaction *transaction;
>
> Previously fetch respected GIT_REFLOG_ACTION, and this is dropping
> that support.  Intended?

I think it was added back later in the series when
ref_transaction_update started taking a "msg" argument.
I have reordered the patches in the ref-transactions so that the fetch
updates come after we add msg support to _update.

Please see current builtin/fetch.c in my ref-transactions branch.


>
> [...]
>> +     if (ref_transaction_update(transaction, ref->name, ref->new_sha1,
>> +                            ref->old_sha1, 0, check_old))
>> +             return STORE_REF_ERROR_OTHER;
>
> Should pass a strbuf in to get a message back.

I added err strbuf to both update and commit and now also error("%s",
err.buf) on failure.


>
> [...]
>> +
>>       return 0;
>>  }
>>
>> @@ -565,6 +552,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
>>               goto abort;
>>       }
>>
>> +     errno = 0;
>> +     transaction = ref_transaction_begin();
>> +     if (!transaction) {
>> +             rc = error(_("cannot start ref transaction\n"));
>> +             goto abort;
>
> error() appends a newline on its own, so the message shouldn't
> end with newline.
>
> More importantly, this message isn't helpful without more detail about
> why _transaction_begin() failed.  The user doesn't know what
>
>         error: cannot start ref transaction
>
> means.
>
>         error: cannot connect to mysqld for ref storage: [etc]
>
> would tell what they need to know.  That is:
>
>         transaction = ref_transaction_begin(err);
>         if (!transaction) {
>                 rc = error("%s", err.buf);
>                 goto abort;
>         }
>

I add this in the next patch series. Right now ref_transaction_begin
can never return failure back to the caller.


> errno is not used here, so no need to set errno = 0.

removed.

>
> [...]
>> @@ -676,6 +670,10 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
>>                       }
>>               }
>>       }
>> +     if ((ret = ref_transaction_commit(transaction, NULL)))
>> +             rc |= ret == -2 ? STORE_REF_ERROR_DF_CONFLICT :
>> +                             STORE_REF_ERROR_OTHER;
>
> (I cheated and stole the newer code from your repo.)
>
> Yay!  Style nit: git avoids assignments in "if" conditionals.
>
>         ret = ref_tr...
>         if (ret == -2)
>                 rc |= ...
>         else if (ret)
>                 rc |= ...
>
> Does _update need the too as futureproofing for backends that can
> detect the name conflict sooner?

Yes it will. I will update the next series to do this. If _update
fails with name conflict it will return -2.
Also it will store the status in the transaction so that a later call
to _commit will also return -2.



>
> Thanks,
> Jonathan

Re: [PATCH v8 23/44] fetch.c: change s_update_ref to use a ref transaction

[Ronnie Sahlberg]

On Fri, May 16, 2014 at 3:54 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/builtin/fetch.c
>> +++ b/builtin/fetch.c
> [...]
>> @@ -384,15 +384,16 @@ static int s_update_ref(const char *action,
>>       snprintf(msg, sizeof(msg), "%s: %s", rla, action);
>>
>>       errno = 0;
>> -     lock = lock_any_ref_for_update(ref->name,
>> -                                    check_old ? ref->old_sha1 : NULL,
>> -                                    0, NULL);
>> -     if (!lock)
>> -             return errno == ENOTDIR ? STORE_REF_ERROR_DF_CONFLICT :
>> -                                       STORE_REF_ERROR_OTHER;
>> -     if (write_ref_sha1(lock, ref->new_sha1, msg) < 0)
>> +     transaction = ref_transaction_begin();
>> +     if (!transaction ||
>> +         ref_transaction_update(transaction, ref->name, ref->new_sha1,
>> +                                ref->old_sha1, 0, check_old) ||
>> +         ref_transaction_commit(transaction, msg, NULL)) {
>
> Since 'err' is NULL, does that mean there's no message shown to the
> user on error?

Yes.  Updated in the ref-transactions branch.

Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Wrap all the ref updates inside a transaction to make the update atomic.

Interesting.

[...]
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -46,6 +46,8 @@ static void *head_name_to_free;
>  static int sent_capabilities;
>  static int shallow_update;
>  static const char *alt_shallow_file;
> +static struct strbuf err = STRBUF_INIT;

I think it would be cleaner for err to be local.  It isn't used for
communication between functions.

[...]
> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)
>  		    update_shallow_ref(cmd, si))
>  			return "shallow error";
>  
> -		lock = lock_any_ref_for_update(namespaced_name, old_sha1,
> -					       0, NULL);
> -		if (!lock) {
> -			rp_error("failed to lock %s", name);
> -			return "failed to lock";
> -		}
> -		if (write_ref_sha1(lock, new_sha1, "push")) {
> -			return "failed to write"; /* error() already called */
> -		}
> +		if (ref_transaction_update(transaction, namespaced_name,
> +					   new_sha1, old_sha1, 0, 1, &err))
> +			return "failed to update";

The original used rp_error to send an error message immediately via
sideband.  This drops that --- intended?

The old error string shown on the push status line was was "failed to
lock" or "failed to write" which makes it clear that the cause is
contention or database problems or filesystem problems, respectively.
After this change it would say "failed to update" which is about as
clear as "failed".

Would it be safe to send err.buf as-is over the wire, or does it
contain information or formatting that wouldn't be suitable for the
client?  (I haven't thought this through completely yet.)  Is there
some easy way to distinguish between failure to lock and failure to
write?  Or is there some one-size-fits-all error for this case?

When the transaction fails, we need to make sure that all ref updates
emit 'ng' and not 'ok' in receive-pack.c::report (see the example at
the end of Documentation/technical/pack-protocol.txt for what this
means).  What error string should they use?  Is there some way to make
it clear to the user which ref was the culprit?

What should happen when checks outside the ref transaction system
cause a ref update to fail?  I'm thinking of

 * per-ref 'update' hook (see githooks(5))
 * fast-forward check
 * ref creation/deletion checks
 * attempt to push to the currently checked out branch

I think the natural thing to do would be to put each ref update in its
own transaction to start so the semantics do not change right away.
If there are obvious answers to all these questions, then a separate
patch could combine all these into a single transaction; or if there
are no obvious answers, we could make the single-transaction-per-push
semantics optional (using a configuration variable or protocol
capability or something) to make it possible to experiment.

Hope that helps,
Jonathan

Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs

[Ronnie Sahlberg]

On Tue, May 20, 2014 at 12:42 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Wrap all the ref updates inside a transaction to make the update atomic.
>
> Interesting.
>
> [...]
>> --- a/builtin/receive-pack.c
>> +++ b/builtin/receive-pack.c
>> @@ -46,6 +46,8 @@ static void *head_name_to_free;
>>  static int sent_capabilities;
>>  static int shallow_update;
>>  static const char *alt_shallow_file;
>> +static struct strbuf err = STRBUF_INIT;
>
> I think it would be cleaner for err to be local.  It isn't used for
> communication between functions.

Done.

>
> [...]
>> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)
>>                   update_shallow_ref(cmd, si))
>>                       return "shallow error";
>>
>> -             lock = lock_any_ref_for_update(namespaced_name, old_sha1,
>> -                                            0, NULL);
>> -             if (!lock) {
>> -                     rp_error("failed to lock %s", name);
>> -                     return "failed to lock";
>> -             }
>> -             if (write_ref_sha1(lock, new_sha1, "push")) {
>> -                     return "failed to write"; /* error() already called */
>> -             }
>> +             if (ref_transaction_update(transaction, namespaced_name,
>> +                                        new_sha1, old_sha1, 0, 1, &err))
>> +                     return "failed to update";
>
> The original used rp_error to send an error message immediately via
> sideband.  This drops that --- intended?

Oops. I misread it as a normal error()

>
> The old error string shown on the push status line was was "failed to
> lock" or "failed to write" which makes it clear that the cause is
> contention or database problems or filesystem problems, respectively.
> After this change it would say "failed to update" which is about as
> clear as "failed".

I changed it to return xstrdup(err.buf) which should be as detailed as
we can get.

>
> Would it be safe to send err.buf as-is over the wire, or does it
> contain information or formatting that wouldn't be suitable for the
> client?  (I haven't thought this through completely yet.)  Is there
> some easy way to distinguish between failure to lock and failure to
> write?  Or is there some one-size-fits-all error for this case?

I think err.buf is what we want.

>
> When the transaction fails, we need to make sure that all ref updates
> emit 'ng' and not 'ok' in receive-pack.c::report (see the example at
> the end of Documentation/technical/pack-protocol.txt for what this
> means).  What error string should they use?  Is there some way to make
> it clear to the user which ref was the culprit?
>
> What should happen when checks outside the ref transaction system
> cause a ref update to fail?  I'm thinking of
>
>  * per-ref 'update' hook (see githooks(5))
>  * fast-forward check
>  * ref creation/deletion checks
>  * attempt to push to the currently checked out branch
>
> I think the natural thing to do would be to put each ref update in its
> own transaction to start so the semantics do not change right away.
> If there are obvious answers to all these questions, then a separate
> patch could combine all these into a single transaction; or if there
> are no obvious answers, we could make the single-transaction-per-push
> semantics optional (using a configuration variable or protocol
> capability or something) to make it possible to experiment.

I changed it to use one transaction per ref for now.

Please see the update ref-transactions branch.

Thanks!

Re: [PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> [Subject: fast-import.c: use a ref transaction when dumping tags]

This seems like an odd thing to do: either it would make sense to have
a single transaction for all imported refs so all fail or succeed
together, or there would be separate transactions for each ref.

That said, I don't mind, particularly if it's a step on the way to
using a single transaction for everything being dumped.

[...]
> --- a/fast-import.c
> +++ b/fast-import.c
> @@ -1736,15 +1736,22 @@ static void dump_tags(void)
>  {
>  	static const char *msg = "fast-import";
>  	struct tag *t;
> -	struct ref_lock *lock;
>  	char ref_name[PATH_MAX];
> +	struct strbuf err = STRBUF_INIT;
> +	struct ref_transaction *transaction;
>  
> +	transaction = ref_transaction_begin();
>  	for (t = first_tag; t; t = t->next_tag) {
> -		sprintf(ref_name, "tags/%s", t->name);
> +		sprintf(ref_name, "refs/tags/%s", t->name);

Can this overflow the buffer?

> -		lock = lock_ref_sha1(ref_name, NULL);
> -		if (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)
> -			failure |= error("Unable to update %s", ref_name);
> +
> +		if (ref_transaction_update(transaction, ref_name, t->sha1,
> +					   NULL, 0, 0, &err)) {
> +			failure |= 1;
> +			break;
> +		}
>  	}
> +	if (failure || ref_transaction_commit(transaction, msg, &err))
> +		failure |= error("Unable to update %s", err.buf);

This 'if (failure || ...) failure |=' idiom seems strange.
Is err.buf always set if failure is nonzero?

Elsewhere failure is 0 or -1 but this introduces the possibility for
it to be temporarily 1.

dump_branches could have caused failure to be -1 before dump_tags
is called.  I think the intent is

		if (ref_transaction_update(...)) {
			failure |= error(...);
			goto out;
		}
	}
	if (ref_transaction_commit(...))
		failure |= error(...);
 out:
	ref_transaction_free(transaction);
	...

Thanks,
Jonathan

Re: [PATCH v8 26/44] fast-import.c: use a ref transaction when dumping tags

[Ronnie Sahlberg]

On Tue, May 20, 2014 at 1:38 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> [Subject: fast-import.c: use a ref transaction when dumping tags]
>
> This seems like an odd thing to do: either it would make sense to have
> a single transaction for all imported refs so all fail or succeed
> together, or there would be separate transactions for each ref.
>
> That said, I don't mind, particularly if it's a step on the way to
> using a single transaction for everything being dumped.

For now they are two transactions but I will merge them into a single one later.

>
> [...]
>> --- a/fast-import.c
>> +++ b/fast-import.c
>> @@ -1736,15 +1736,22 @@ static void dump_tags(void)
>>  {
>>       static const char *msg = "fast-import";
>>       struct tag *t;
>> -     struct ref_lock *lock;
>>       char ref_name[PATH_MAX];
>> +     struct strbuf err = STRBUF_INIT;
>> +     struct ref_transaction *transaction;
>>
>> +     transaction = ref_transaction_begin();
>>       for (t = first_tag; t; t = t->next_tag) {
>> -             sprintf(ref_name, "tags/%s", t->name);
>> +             sprintf(ref_name, "refs/tags/%s", t->name);
>
> Can this overflow the buffer?

Changed to snprint. (We probably should do this everywhere.)

>
>> -             lock = lock_ref_sha1(ref_name, NULL);
>> -             if (!lock || write_ref_sha1(lock, t->sha1, msg) < 0)
>> -                     failure |= error("Unable to update %s", ref_name);
>> +
>> +             if (ref_transaction_update(transaction, ref_name, t->sha1,
>> +                                        NULL, 0, 0, &err)) {
>> +                     failure |= 1;
>> +                     break;
>> +             }
>>       }
>> +     if (failure || ref_transaction_commit(transaction, msg, &err))
>> +             failure |= error("Unable to update %s", err.buf);
>
> This 'if (failure || ...) failure |=' idiom seems strange.
> Is err.buf always set if failure is nonzero?
>
> Elsewhere failure is 0 or -1 but this introduces the possibility for
> it to be temporarily 1.
>
> dump_branches could have caused failure to be -1 before dump_tags
> is called.  I think the intent is
>
>                 if (ref_transaction_update(...)) {
>                         failure |= error(...);
>                         goto out;
>                 }
>         }
>         if (ref_transaction_commit(...))
>                 failure |= error(...);
>  out:
>         ref_transaction_free(transaction);
>         ...
>

Actually, since the recent change to make _commit fail if the
transaction has an error, what we want is something like this :

    ...
        if (ref_transaction_update(transaction, ref_name, t->sha1,
NULL, 0, 0, &err))
            break;
    }
    if (ref_transaction_commit(transaction, msg, &err))
        failure |= error("%s", err.buf);
    ...


Please see updated ref-transactions branch.


Thanks.

> Thanks,
> Jonathan

Re: [PATCH v8 27/44] walker.c: use ref transaction for ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This changes the locking slightly for walker_fetch. Previously the code would
> lock all refs before writing them but now we do not lock the refs until the
> commit stage. There is thus a very short window where changes could be done
> locally during the fetch which would be overwritten when the fetch completes
> and commits its transaction. But this window should be reasonably short.
> Even if this race does trigger, since both the old code and the new code
> just overwrites the refs to the new values without checking or comparing
> them with the previous value, this is not too dissimilar to a similar scenario
> where you first do a ref change locally and then later do a fetch that
> overwrites the local change. With this in mind I do not see the change in
> locking semantics to be critical.

Sounds scary.  The usual approach is

	old_sha1 = ...
	... various checks ...

	transaction = transaction_begin(&err)
	transaction_update(transaction, refname, new_sha1, old_sha1, ...);
	transaction_commit(transaction, &err);

which is not racy because _update checks against old_sha1.

If I understand correctly, you are saying 'have_old' is false here so
we don't have the usual protection.  If the "... various checks ..."
section shown above is empty, that should be fine and there is no
actual change in semantics.  If the "... various checks ..." section
shown above is nonempty then it could be a problem.

[...]
> --- a/walker.c
> +++ b/walker.c
> @@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)
>  int walker_fetch(struct walker *walker, int targets, char **target,
>  		 const char **write_ref, const char *write_ref_log_details)
>  {
> -	struct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));
> +	char ref_name[PATH_MAX];

We tend to prefer strbuf instead of fixed-size buffers in new code.

[...]
> -	char *msg;
> +	char *msg = NULL;

Needed?  The existing code seems to set msg = NULL in the
!write_ref_log_details case already.

[...]
> @@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,
>  	for (i = 0; i < targets; i++) {
>  		if (!write_ref || !write_ref[i])
>  			continue;
> -		ret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : "fetch (unknown)");

Okay, so before this patch we do:

	for each target in write_ref:
		lock it (with no particular expectation about where it
		points)

Then

	unless http-fetch was passed --recover:
		mark the objects pointed to by current refs as COMPLETE

Then we do HTTP GETs to grab the objects we need from a "dumb" HTTP
server.  The COMPLETE objects tell us about objects we don't have to
bother trying to get.

When we're done, we come up with a reflog entry and write out refs
pointing to the requested commits.

This code has two callers:

	git-remote-http (aka remote-curl.c::fetch_dumb)
	git-http-fetch (aka http-fetch.c)

The codepath in git-remote-http gets wide use, though it's diminishing
as more people switch to "smart" http.  It doesn't 't use the "write
out some refs" feature.  It just wants the objects and then takes care
of writing refs on its own.

Perhaps it's worth avoiding beginning a transaction in the first place
in the !write_ref case.

The git-http-fetch command is a piece of plumbing that used to be used
by 'git clone' and 'git fetch' in the olden days when they were shell
scripts.  I doubt anyone uses it.  As you noticed, it doesn't have any
way to specify anything about the expected old values of the refs it
writes to.  So this change doesn't introduce any race there.

> +		sprintf(ref_name, "refs/%s", write_ref[i]);
> +		if (ref_transaction_update(transaction, ref_name,
> +					   &sha1[20 * i], NULL,
> +					   0, 0))
> +			goto rollback_and_fail;
> +	}

Looks good.

> +
> +	if (ref_transaction_commit(transaction, msg ? msg : "fetch (unknown)",
> +				   &err)) {
> +		error("%s", err.buf);
> +		goto rollback_and_fail;
>  	}

Also looks good.

Thanks,
Jonathan

Re: [PATCH v8 28/44] refs.c: make write_ref_sha1 static

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> No external users call write_ref_sha1 any more so lets declare it static.

Yay!

[...]
> +++ b/refs.c
> @@ -251,6 +251,8 @@ struct ref_entry {
[...]
>  static void read_loose_refs(const char *dirname, struct ref_dir *dir);
> +static int write_ref_sha1(struct ref_lock *lock,
> +			  const unsigned char *sha1, const char *logmsg);

Is this forward declaration needed?

[...]
> --- a/refs.h
> +++ b/refs.h
> @@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);
>  /** Release any lock taken but not written. **/
>  extern void unlock_ref(struct ref_lock *lock);
>  
> -/** Writes sha1 into the ref specified by the lock. **/
> -extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);

(nit) Would be nice to keep the documentation comment.

Thanks,
Jonathan

Re: [PATCH v8 29/44] refs.c: make lock_ref_sha1 static

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> No external callers reference lock_ref_sha1 any more so lets declare it static.
>
> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 2 +-
>  refs.h | 3 ---
>  2 files changed, 1 insertion(+), 4 deletions(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 28/44] refs.c: make write_ref_sha1 static

[Ronnie Sahlberg]

Thanks!

On Tue, May 20, 2014 at 5:51 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> No external users call write_ref_sha1 any more so lets declare it static.
>
> Yay!
>
> [...]
>> +++ b/refs.c
>> @@ -251,6 +251,8 @@ struct ref_entry {
> [...]
>>  static void read_loose_refs(const char *dirname, struct ref_dir *dir);
>> +static int write_ref_sha1(struct ref_lock *lock,
>> +                       const unsigned char *sha1, const char *logmsg);
>
> Is this forward declaration needed?
>

No. Removed.
(I could have sworn I had a patch to remove this forward declaration.)

> [...]
>> --- a/refs.h
>> +++ b/refs.h
>> @@ -150,9 +150,6 @@ extern int commit_ref(struct ref_lock *lock);
>>  /** Release any lock taken but not written. **/
>>  extern void unlock_ref(struct ref_lock *lock);
>>
>> -/** Writes sha1 into the ref specified by the lock. **/
>> -extern int write_ref_sha1(struct ref_lock *lock, const unsigned char *sha1, const char *msg);
>
> (nit) Would be nice to keep the documentation comment.

Moved this comment to refs.c

Please see ref-transactions branch
Thanks!

>
> Thanks,
> Jonathan

Re: [PATCH v8 27/44] walker.c: use ref transaction for ref updates

[Ronnie Sahlberg]

On Tue, May 20, 2014 at 5:46 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> This changes the locking slightly for walker_fetch. Previously the code would
>> lock all refs before writing them but now we do not lock the refs until the
>> commit stage. There is thus a very short window where changes could be done
>> locally during the fetch which would be overwritten when the fetch completes
>> and commits its transaction. But this window should be reasonably short.
>> Even if this race does trigger, since both the old code and the new code
>> just overwrites the refs to the new values without checking or comparing
>> them with the previous value, this is not too dissimilar to a similar scenario
>> where you first do a ref change locally and then later do a fetch that
>> overwrites the local change. With this in mind I do not see the change in
>> locking semantics to be critical.
>
> Sounds scary.  The usual approach is
>
>         old_sha1 = ...
>         ... various checks ...
>
>         transaction = transaction_begin(&err)
>         transaction_update(transaction, refname, new_sha1, old_sha1, ...);
>         transaction_commit(transaction, &err);
>
> which is not racy because _update checks against old_sha1.
>
> If I understand correctly, you are saying 'have_old' is false here so
> we don't have the usual protection.  If the "... various checks ..."
> section shown above is empty, that should be fine and there is no
> actual change in semantics.  If the "... various checks ..." section
> shown above is nonempty then it could be a problem.

Yeah, we don't have the old sha1 so this function is already imo a bit dodgy.
The transaction system in this series can not really handle this with
the exact same semantics as the old code
but we do gain the ability to do things such as

transaction_begin
... lock all refs...
...do other stuff.
transaction_update for all refs
transaction_commit

in the next patch series when the transactions start supporting
multiple updates for the same ref.
However that support comes reasonably late in the next series and I
would want to avoid growing this series much more.


This is the patch in the next series that restore the lock-everything-first.
It relies on that we will do all locking in _update already and not
wait until _commit as well as that it allows multiple updates to the
same ref as long as all updates, except the first, are of the form
have_old==1 and old_sha1==new_sha1-of-previous-update.


diff --git a/walker.c b/walker.c
index 94d4988..dd8c11e 100644
--- a/walker.c
+++ b/walker.c
@@ -255,18 +255,31 @@ int walker_fetch(struct walker *walker, int targets, char
        struct strbuf err = STRBUF_INIT;
        struct ref_transaction *transaction;
        unsigned char *sha1 = xmalloc(targets * 20);
+       unsigned char transaction_sha1[20];
        char *msg;
        int i;

        save_commit_buffer = 0;

        if (write_ref) {
+               memset(transaction_sha1, 0xff, 20);
                transaction = transaction_begin(&err);
                if (!transaction) {
                        error("%s", err.buf);
                        strbuf_release(&err);
                        return -1;
                }
+               /* Lock all refs by updating to a temporary sha1 */
+               for (i = 0; i < targets; i++) {
+                       if (!write_ref[i])
+                               continue;
+                       strbuf_reset(&ref_name);
+                       strbuf_addf(&ref_name, "refs/%s", write_ref[i]);
+                       transaction_update_sha1(transaction, ref_name.buf,
+                                               transaction_sha1, NULL, 0, 0,
+                                               msg ? msg : "fetch (unknown)",
+                                               &err);
+               }
        }
        if (!walker->get_recover)
                for_each_ref(mark_complete, NULL);
@@ -295,7 +308,8 @@ int walker_fetch(struct walker *walker, int targets, char **
                strbuf_reset(&ref_name);
                strbuf_addf(&ref_name, "refs/%s", write_ref[i]);
                if (transaction_update_sha1(transaction, ref_name.buf,
-                                           &sha1[20 * i], NULL, 0, 0,
+                                           &sha1[20 * i], transaction_sha1,
+                                           0, 1,
                                            msg ? msg : "fetch (unknown)",
                                            &err))
                        break;



>
> [...]
>> --- a/walker.c
>> +++ b/walker.c
>> @@ -251,24 +251,18 @@ void walker_targets_free(int targets, char **target, const char **write_ref)
>>  int walker_fetch(struct walker *walker, int targets, char **target,
>>                const char **write_ref, const char *write_ref_log_details)
>>  {
>> -     struct ref_lock **lock = xcalloc(targets, sizeof(struct ref_lock *));
>> +     char ref_name[PATH_MAX];
>
> We tend to prefer strbuf instead of fixed-size buffers in new code.

Changed.

>
> [...]
>> -     char *msg;
>> +     char *msg = NULL;
>
> Needed?  The existing code seems to set msg = NULL in the
> !write_ref_log_details case already.

Fixed.

Thanks.


>
> [...]
>> @@ -294,19 +288,26 @@ int walker_fetch(struct walker *walker, int targets, char **target,
>>       for (i = 0; i < targets; i++) {
>>               if (!write_ref || !write_ref[i])
>>                       continue;
>> -             ret = write_ref_sha1(lock[i], &sha1[20 * i], msg ? msg : "fetch (unknown)");
>
> Okay, so before this patch we do:
>
>         for each target in write_ref:
>                 lock it (with no particular expectation about where it
>                 points)
>
> Then
>
>         unless http-fetch was passed --recover:
>                 mark the objects pointed to by current refs as COMPLETE
>
> Then we do HTTP GETs to grab the objects we need from a "dumb" HTTP
> server.  The COMPLETE objects tell us about objects we don't have to
> bother trying to get.
>
> When we're done, we come up with a reflog entry and write out refs
> pointing to the requested commits.
>
> This code has two callers:
>
>         git-remote-http (aka remote-curl.c::fetch_dumb)
>         git-http-fetch (aka http-fetch.c)
>
> The codepath in git-remote-http gets wide use, though it's diminishing
> as more people switch to "smart" http.  It doesn't 't use the "write
> out some refs" feature.  It just wants the objects and then takes care
> of writing refs on its own.
>
> Perhaps it's worth avoiding beginning a transaction in the first place
> in the !write_ref case.
>
> The git-http-fetch command is a piece of plumbing that used to be used
> by 'git clone' and 'git fetch' in the olden days when they were shell
> scripts.  I doubt anyone uses it.  As you noticed, it doesn't have any
> way to specify anything about the expected old values of the refs it
> writes to.  So this change doesn't introduce any race there.
>
>> +             sprintf(ref_name, "refs/%s", write_ref[i]);
>> +             if (ref_transaction_update(transaction, ref_name,
>> +                                        &sha1[20 * i], NULL,
>> +                                        0, 0))
>> +                     goto rollback_and_fail;
>> +     }
>
> Looks good.
>
>> +
>> +     if (ref_transaction_commit(transaction, msg ? msg : "fetch (unknown)",
>> +                                &err)) {
>> +             error("%s", err.buf);
>> +             goto rollback_and_fail;
>>       }
>
> Also looks good.
>
> Thanks,
> Jonathan

Re: [PATCH v8 25/44] receive-pack.c: use a reference transaction for updating the refs

[Ronnie Sahlberg]

On Tue, May 20, 2014 at 1:37 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> On Tue, May 20, 2014 at 12:42 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Ronnie Sahlberg wrote:
>>
>>> Wrap all the ref updates inside a transaction to make the update atomic.
>>
>> Interesting.
>>
>> [...]
>>> --- a/builtin/receive-pack.c
>>> +++ b/builtin/receive-pack.c
>>> @@ -46,6 +46,8 @@ static void *head_name_to_free;
>>>  static int sent_capabilities;
>>>  static int shallow_update;
>>>  static const char *alt_shallow_file;
>>> +static struct strbuf err = STRBUF_INIT;
>>
>> I think it would be cleaner for err to be local.  It isn't used for
>> communication between functions.
>
> Done.
>
>>
>> [...]
>>> @@ -580,15 +581,9 @@ static const char *update(struct command *cmd, struct shallow_info *si)
>>>                   update_shallow_ref(cmd, si))
>>>                       return "shallow error";
>>>
>>> -             lock = lock_any_ref_for_update(namespaced_name, old_sha1,
>>> -                                            0, NULL);
>>> -             if (!lock) {
>>> -                     rp_error("failed to lock %s", name);
>>> -                     return "failed to lock";
>>> -             }
>>> -             if (write_ref_sha1(lock, new_sha1, "push")) {
>>> -                     return "failed to write"; /* error() already called */
>>> -             }
>>> +             if (ref_transaction_update(transaction, namespaced_name,
>>> +                                        new_sha1, old_sha1, 0, 1, &err))
>>> +                     return "failed to update";
>>
>> The original used rp_error to send an error message immediately via
>> sideband.  This drops that --- intended?
>
> Oops. I misread it as a normal error()
>
>>
>> The old error string shown on the push status line was was "failed to
>> lock" or "failed to write" which makes it clear that the cause is
>> contention or database problems or filesystem problems, respectively.
>> After this change it would say "failed to update" which is about as
>> clear as "failed".
>
> I changed it to return xstrdup(err.buf) which should be as detailed as
> we can get.
>
>>
>> Would it be safe to send err.buf as-is over the wire, or does it
>> contain information or formatting that wouldn't be suitable for the
>> client?  (I haven't thought this through completely yet.)  Is there
>> some easy way to distinguish between failure to lock and failure to
>> write?  Or is there some one-size-fits-all error for this case?
>
> I think err.buf is what we want.
>
>>
>> When the transaction fails, we need to make sure that all ref updates
>> emit 'ng' and not 'ok' in receive-pack.c::report (see the example at
>> the end of Documentation/technical/pack-protocol.txt for what this
>> means).  What error string should they use?  Is there some way to make
>> it clear to the user which ref was the culprit?
>>
>> What should happen when checks outside the ref transaction system
>> cause a ref update to fail?  I'm thinking of
>>
>>  * per-ref 'update' hook (see githooks(5))
>>  * fast-forward check
>>  * ref creation/deletion checks
>>  * attempt to push to the currently checked out branch
>>
>> I think the natural thing to do would be to put each ref update in its
>> own transaction to start so the semantics do not change right away.
>> If there are obvious answers to all these questions, then a separate
>> patch could combine all these into a single transaction; or if there
>> are no obvious answers, we could make the single-transaction-per-push
>> semantics optional (using a configuration variable or protocol
>> capability or something) to make it possible to experiment.
>
> I changed it to use one transaction per ref for now.
>
> Please see the update ref-transactions branch.

I have added support for atomic pushes towards the end of the -next series.
https://github.com/rsahlberg/git/tree/ref-transactions-next

It is activated by a new --atomic-push argument to send-pack and is
then negotiated by a new option in the wire protocol.


>
> Thanks!

Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/refs.c
> +++ b/refs.c
> @@ -3308,6 +3308,12 @@ struct ref_update {
>  	const char refname[FLEX_ARRAY];
>  };
>  
> +enum ref_transaction_status {
> +	REF_TRANSACTION_OPEN   = 0,
> +	REF_TRANSACTION_CLOSED = 1,
> +	REF_TRANSACTION_ERROR  = 2,

What is the difference between _TRANSACTION_CLOSED and
_TRANSACTION_ERROR?

[...]
> @@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)
>  
>  void ref_transaction_rollback(struct ref_transaction *transaction)
>  {
> +	if (!transaction)
> +		return;
> +
> +	transaction->status = REF_TRANSACTION_ERROR;
> +
>  	ref_transaction_free(transaction);

Once the transaction is freed, transaction->status is not reachable any
more so no one can tell that you've set it to _ERROR.  What is the
intended effect?

[...]
> @@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
>  	if (have_old && !old_sha1)
>  		die("BUG: have_old is true but old_sha1 is NULL");
>  
> +	if (transaction->status != REF_TRANSACTION_OPEN)
> +		die("BUG: update on transaction that is not open");

Ok.

[...]
> @@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>  	clear_loose_ref_cache(&ref_cache);
>  
>  cleanup:
> +	transaction->status = ret ? REF_TRANSACTION_ERROR
> +	  : REF_TRANSACTION_CLOSED;

Nit: odd use of whitespace.

Overall thoughts: I like the idea of enforcing the API more strictly
("after an error, the only permitted operations are...").  The details
leave me a little confused because I don't think anything is
distinguishing between _CLOSED and _ERROR.  Maybe the enum only needs
two states.

Thanks,
Jonathan

Re: [PATCH v8 31/44] refs.c: remove the update_ref_lock function

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 30 ++++++------------------------
>  1 file changed, 6 insertions(+), 24 deletions(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 32/44] refs.c: remove the update_ref_write function

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> +++ b/refs.c
[...]
> @@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>  		struct ref_update *update = updates[i];
>  
>  		if (!is_null_sha1(update->new_sha1)) {
> -			ret = update_ref_write(msg,
> -					       update->refname,
> -					       update->new_sha1,
> -					       update->lock, err,
> -					       UPDATE_REFS_QUIET_ON_ERR);
> +			ret = write_ref_sha1(update->lock, update->new_sha1,
> +					     msg);

This changes the return value on error from 1 to -1.  That seems like a
good change.  It's probably worth mentioning in the commit message.

Re: [PATCH v8 33/44] refs.c: remove lock_ref_sha1

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 15 +++++----------
>  1 file changed, 5 insertions(+), 10 deletions(-)

Heh. :)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR

[Ronnie Sahlberg]

Please pull my ref-transactions branch.

On Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -3308,6 +3308,12 @@ struct ref_update {
>>       const char refname[FLEX_ARRAY];
>>  };
>>
>> +enum ref_transaction_status {
>> +     REF_TRANSACTION_OPEN   = 0,
>> +     REF_TRANSACTION_CLOSED = 1,
>> +     REF_TRANSACTION_ERROR  = 2,
>
> What is the difference between _TRANSACTION_CLOSED and
> _TRANSACTION_ERROR?

Closed is a transaction that has been committed successfully, and
which we can not do any more updates onto.
Error is a transaction that has failed, and which we can not do any
more updates onto.

The distinction could be useful if in the future we add support to
reuse a transaction

>
> [...]
>> @@ -3340,6 +3347,11 @@ void ref_transaction_free(struct ref_transaction *transaction)
>>
>>  void ref_transaction_rollback(struct ref_transaction *transaction)
>>  {
>> +     if (!transaction)
>> +             return;
>> +
>> +     transaction->status = REF_TRANSACTION_ERROR;
>> +
>>       ref_transaction_free(transaction);
>
> Once the transaction is freed, transaction->status is not reachable any
> more so no one can tell that you've set it to _ERROR.  What is the
> intended effect?

ref_transaction_rollback is no more. It has been removed.

>
> [...]
>> @@ -3366,6 +3378,9 @@ int ref_transaction_update(struct ref_transaction *transaction,
>>       if (have_old && !old_sha1)
>>               die("BUG: have_old is true but old_sha1 is NULL");
>>
>> +     if (transaction->status != REF_TRANSACTION_OPEN)
>> +             die("BUG: update on transaction that is not open");
>
> Ok.
>
> [...]
>> @@ -3538,6 +3564,9 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>>       clear_loose_ref_cache(&ref_cache);
>>
>>  cleanup:
>> +     transaction->status = ret ? REF_TRANSACTION_ERROR
>> +       : REF_TRANSACTION_CLOSED;
>
> Nit: odd use of whitespace.

fixed in ref-transactions.


>
> Overall thoughts: I like the idea of enforcing the API more strictly
> ("after an error, the only permitted operations are...").  The details
> leave me a little confused because I don't think anything is
> distinguishing between _CLOSED and _ERROR.  Maybe the enum only needs
> two states.

A buggy caller might do :

transaction_begin()
transaction_update()
transaction_commit()  (A)
transaction_update() (B)
transaction_commit() (C)

After A the transaction in no longer open and until we decide we want
to add support for re-usable transactions (which may or may not be a
good idea) we need to make sure that both B and C fails.
Since the transaction in A completed successfully we can't really mark
is as ERROR, instead we flag it as CLOSED.



>
> Thanks,
> Jonathan

Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Please pull my ref-transactions branch.

I'm at bd5736cb (2014-05-21 13:46) now.

> On Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Ronnie Sahlberg wrote:

>>> --- a/refs.c
>>> +++ b/refs.c
>>> @@ -3308,6 +3308,12 @@ struct ref_update {
>>>       const char refname[FLEX_ARRAY];
>>>  };
>>>
>>> +enum ref_transaction_status {
>>> +     REF_TRANSACTION_OPEN   = 0,
>>> +     REF_TRANSACTION_CLOSED = 1,
>>> +     REF_TRANSACTION_ERROR  = 2,
>>
>> What is the difference between _TRANSACTION_CLOSED and
>> _TRANSACTION_ERROR?
>
> Closed is a transaction that has been committed successfully, and
> which we can not do any more updates onto.
> Error is a transaction that has failed, and which we can not do any
> more updates onto.

That means that both mean the caller cannot do any more updates,
right?  Why not call them both _CLOSED?

> The distinction could be useful if in the future we add support to
> reuse a transaction

If the distinction becomes useful in the future, wouldn't that
be the moment to add a new state?

I don't mean that there has to be a big useful distinction.  E.g.,
maybe the idea is that when using a debugger it can be useful to see
the difference between _ERROR and _CLOSED or something, and I think
that would be fine as long as the intended meaning is documented
(e.g., using comments).  My only complaint is that it's hard to
understand the code and keep track of which status should be used in a
given place unless there is some distinction between them.

[...]
> ref_transaction_rollback is no more. It has been removed.

Thanks.  Sorry I missed that last time.

Jonathan

Re: [PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change prune_ref to delete the ref using a ref transaction. To do this we also
> need to add a new flag REF_ISPRUNING that will tell the transaction that we
> do not want to delete this ref from the packed refs.

Interesting.  Since the flag is per ref update, it even would allow
deleting some refs and pruning others in the same transaction.  Makes
sense.

Looks like this doesn't batch up multiple ref-prunings into a single
transaction.  Makes sense (it would just make the period while refs
are locked longer without having any real benefit).

[...]
> +#define REF_ISPRUNING	0x0100

Can this conflict with bit values declared elsewhere some day?  It
would be more comfortable if refs.h also had a note about bits >=
0x100 being reserved for private use.

The rest of the patch looks good.

Thanks,
Jonathan

Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/refs.c
> +++ b/refs.c
[...]
> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>  
>  int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
>  {
> -	struct ref_lock *lock;
> -	int ret = 0, flag = 0;
> +	struct ref_transaction *transaction;
>  
[...]
> -	lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);
> +	if (!transaction ||
> +	    ref_transaction_delete(transaction, refname, sha1, delopt,
> +				   sha1 && !is_null_sha1(sha1), &err) ||

What should happen when is_null_sha1(sha1)?  In that case the
caller has asked us to check that the ref doesn't exist before
deleting it, which doesn't make a lot of sense, but the old
code did exactly that if I read it correctly.  The new code
seems to disable verification instead.

Do we know that no callers call delete_ref with such an argument?
Would it make sense to just die with a "BUG:" message to make
the contract more clear?

[...]
> -	unlink_or_warn(git_path("logs/%s", lock->ref_name));

When does the reflog get deleted in the new code?

Thanks,
Jonathan

Re: [PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Change the reference transactions so that we pass the reflog message
> through to the create/delete/update function instead of the commit message.

Nice.

[...]
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -673,7 +673,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
> 			}
> 		}
> 	}
> -
> 	if (rc & STORE_REF_ERROR_DF_CONFLICT)
> 		error(_("some local refs could not be updated; try running\n"
> 		      " 'git remote prune %s' to remove any old, conflicting "

Stray whitespace change?

[...]
> --- a/refs.c
> +++ b/refs.c
[...]
> @@ -3264,6 +3264,7 @@ struct ref_update {
>  	int have_old; /* 1 if old_sha1 is valid, 0 otherwise */
>  	struct ref_lock *lock;
>  	int type;
> +	const char *msg;
>  	const char refname[FLEX_ARRAY];

Should be 'char *msg' since we own the memory (or perhaps a strbuf).

[...]
> @@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)
>  	if (!transaction)
>  		return;
>  
> -	for (i = 0; i < transaction->nr; i++)
> +	for (i = 0; i < transaction->nr; i++) {
> +	  free((char *)transaction->updates[i]->msg);
>  		free(transaction->updates[i]);

Whitespace?

No need to cast.

The rest of the patch looks good.

Thanks,
Jonathan

Re: [PATCH v8 37/44] refs.c: pass NULL as *flags to read_ref_full

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This means that most loose refs will no longer be present after the rename

Is this to handle the "git branch -m foo/bar foo" case or for some other
purpose?

[...]
> --- a/t/t3200-branch.sh
> +++ b/t/t3200-branch.sh
> @@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '
>  	git symbolic-ref refs/heads/master2 refs/heads/master &&
>  	test_must_fail git branch -m master2 master3 &&
>  	git symbolic-ref refs/heads/master2 &&
> -	test_path_is_file .git/refs/heads/master &&
> +	test_path_is_missing .git/refs/heads/master &&
>  	test_path_is_missing .git/refs/heads/master3

It's kind of silly that this test is mucking about in the .git directory
at all.  Shouldn't the check be something like

	git rev-parse --verify refs/heads/master &&
	test_must_fail git symbolic-ref refs/heads/master3 &&
	test_must_fail git rev-parse refs/heads/master3

?

Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic

[Jonathan Nieder]

Ronnie Sahlberg wrote:
> --- a/refs.c
> +++ b/refs.c
> @@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
>  	int missing = 0;
>  	int attempts_remaining = 3;
>  
> +	if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
> +		return NULL;

Are we sure that no callers are locking a poorly named ref
as preparation for repairing it?

To see what works already in that vein, I tried:

	$ git rev-parse HEAD >.git/refs/heads/foo..bar
	$ git branch -m foo..bar something-saner
	fatal: Invalid branch name: 'foo..bar'

"git branch -m" has an explicit codepath ("recovery = 1;") to handle
this case, but it looks like it was not well tested and regressed in
v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has
the right format, 2011-09-15).

Is what the recovery codepath of branch -m does misguided?  One
school of thought would be that people with malformed refs are
responsible for recovering using low-level tools like "mv" and "vi"
instead of normal git commands since normal git commands should never
have created such a bad situation.  Another school of thought would
assert that

 * git commands can have bugs
 * the format checked by check_refname_format() keeps getting stricter
   over time

which means it's nice when people can recover with 'update-ref -d'
or 'branch -m'.  It's not obvious to me what the right thing to do
here is (maybe a special flag to be attached to a ref update during
recovery?).

Hope that helps,
Jonathan

Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 4:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/refs.c
>> +++ b/refs.c
> [...]
>> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>>
>>  int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
>>  {
>> -     struct ref_lock *lock;
>> -     int ret = 0, flag = 0;
>> +     struct ref_transaction *transaction;
>>
> [...]
>> -     lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);
>> +     if (!transaction ||
>> +         ref_transaction_delete(transaction, refname, sha1, delopt,
>> +                                sha1 && !is_null_sha1(sha1), &err) ||
>
> What should happen when is_null_sha1(sha1)?  In that case the
> caller has asked us to check that the ref doesn't exist before
> deleting it, which doesn't make a lot of sense, but the old
> code did exactly that if I read it correctly.  The new code
> seems to disable verification instead.
>
> Do we know that no callers call delete_ref with such an argument?
> Would it make sense to just die with a "BUG:" message to make
> the contract more clear?

There are no callers that pass in null_sha1 explicitely and all tests pass.
I have changed it to a die("BUG... to make it more explicit as you suggested.

>
> [...]
>> -     unlink_or_warn(git_path("logs/%s", lock->ref_name));
>
> When does the reflog get deleted in the new code?

It is deleted towards the end of ref_transaction_commit, after the ref
itself has been deleted.

Thanks!

ronnie sahlberg

Re: [PATCH v8 36/44] refs.c: pass the ref log message to _create/delete/update instead of _commit

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 4:47 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Change the reference transactions so that we pass the reflog message
>> through to the create/delete/update function instead of the commit message.
>
> Nice.
>
> [...]
>> --- a/builtin/fetch.c
>> +++ b/builtin/fetch.c
>> @@ -673,7 +673,6 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
>>                       }
>>               }
>>       }
>> -
>>       if (rc & STORE_REF_ERROR_DF_CONFLICT)
>>               error(_("some local refs could not be updated; try running\n"
>>                     " 'git remote prune %s' to remove any old, conflicting "
>
> Stray whitespace change?

Fixed.

>
> [...]
>> --- a/refs.c
>> +++ b/refs.c
> [...]
>> @@ -3264,6 +3264,7 @@ struct ref_update {
>>       int have_old; /* 1 if old_sha1 is valid, 0 otherwise */
>>       struct ref_lock *lock;
>>       int type;
>> +     const char *msg;
>>       const char refname[FLEX_ARRAY];
>
> Should be 'char *msg' since we own the memory (or perhaps a strbuf).

I changed it to char *msg, but it still might/should be const. We own
the memory but we are not supposed to modify the content
(after we have copied what the caller gave us).

Yes, we should change it into a strbuf at some stage.

>
> [...]
>> @@ -3297,9 +3298,10 @@ void ref_transaction_free(struct ref_transaction *transaction)
>>       if (!transaction)
>>               return;
>>
>> -     for (i = 0; i < transaction->nr; i++)
>> +     for (i = 0; i < transaction->nr; i++) {
>> +       free((char *)transaction->updates[i]->msg);
>>               free(transaction->updates[i]);
>
> Whitespace?

Fixed.

>
> No need to cast.

Done.

Thanks!
ronnie sahlberg

Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> This means that most loose refs will no longer be present after the rename
>
> Is this to handle the "git branch -m foo/bar foo" case or for some other
> purpose?

Yes. That is the main reason.

>
> [...]
>> --- a/t/t3200-branch.sh
>> +++ b/t/t3200-branch.sh
>> @@ -289,7 +289,7 @@ test_expect_success 'renaming a symref is not allowed' '
>>       git symbolic-ref refs/heads/master2 refs/heads/master &&
>>       test_must_fail git branch -m master2 master3 &&
>>       git symbolic-ref refs/heads/master2 &&
>> -     test_path_is_file .git/refs/heads/master &&
>> +     test_path_is_missing .git/refs/heads/master &&
>>       test_path_is_missing .git/refs/heads/master3
>
> It's kind of silly that this test is mucking about in the .git directory
> at all.  Shouldn't the check be something like
>
>         git rev-parse --verify refs/heads/master &&
>         test_must_fail git symbolic-ref refs/heads/master3 &&
>         test_must_fail git rev-parse refs/heads/master3
>
> ?

Thanks. I updated the test with your change.

There is a whole bunch of tests that are like and access files directly..
Testing if a path exists or not, or checking that a reflog file
contains x number of lines. etc.

All these tests will be updated to not access the files directly once
I start mucking around with a TDB based refs backend.

Re: [PATCH v8 35/44] refs.c: make delete_ref use a transaction

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 8:32 AM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> On Wed, May 21, 2014 at 4:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Ronnie Sahlberg wrote:
>>
>>> --- a/refs.c
>>> +++ b/refs.c
>> [...]
>>> @@ -2515,24 +2510,18 @@ static int delete_ref_loose(struct ref_lock *lock, int flag, struct strbuf *err)
>>>
>>>  int delete_ref(const char *refname, const unsigned char *sha1, int delopt)
>>>  {
>>> -     struct ref_lock *lock;
>>> -     int ret = 0, flag = 0;
>>> +     struct ref_transaction *transaction;
>>>
>> [...]
>>> -     lock = lock_ref_sha1_basic(refname, sha1, delopt, &flag);
>>> +     if (!transaction ||
>>> +         ref_transaction_delete(transaction, refname, sha1, delopt,
>>> +                                sha1 && !is_null_sha1(sha1), &err) ||
>>
>> What should happen when is_null_sha1(sha1)?  In that case the
>> caller has asked us to check that the ref doesn't exist before
>> deleting it, which doesn't make a lot of sense, but the old
>> code did exactly that if I read it correctly.  The new code
>> seems to disable verification instead.
>>
>> Do we know that no callers call delete_ref with such an argument?
>> Would it make sense to just die with a "BUG:" message to make
>> the contract more clear?
>
> There are no callers that pass in null_sha1 explicitely and all tests pass.
> I have changed it to a die("BUG... to make it more explicit as you suggested.

Strike that.

While there are no callers I can see that deliberately send null_sha1
it can happen because resolve_ref_unsafe(reading=0) calls
handle_missing_loose_ref which will clear sha1
if the ref is missing.

This can happen for either symrefs or refs that are soft links that
point to a nonexisting ref.



>
>>
>> [...]
>>> -     unlink_or_warn(git_path("logs/%s", lock->ref_name));
>>
>> When does the reflog get deleted in the new code?
>
> It is deleted towards the end of ref_transaction_commit, after the ref
> itself has been deleted.
>
> Thanks!
>
> ronnie sahlberg

Re: [PATCH v8 32/44] refs.c: remove the update_ref_write function

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 3:07 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> +++ b/refs.c
> [...]
>> @@ -3518,14 +3499,16 @@ int ref_transaction_commit(struct ref_transaction *transaction,
>>               struct ref_update *update = updates[i];
>>
>>               if (!is_null_sha1(update->new_sha1)) {
>> -                     ret = update_ref_write(msg,
>> -                                            update->refname,
>> -                                            update->new_sha1,
>> -                                            update->lock, err,
>> -                                            UPDATE_REFS_QUIET_ON_ERR);
>> +                     ret = write_ref_sha1(update->lock, update->new_sha1,
>> +                                          msg);
>
> This changes the return value on error from 1 to -1.  That seems like a
> good change.  It's probably worth mentioning in the commit message.

Done.

Re: [PATCH v8 34/44] refs.c: make prune_ref use a transaction to delete the ref

[Ronnie Sahlberg]

Added a comment that any flags >=0x100 are reserved for internal use.


On Wed, May 21, 2014 at 4:01 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Change prune_ref to delete the ref using a ref transaction. To do this we also
>> need to add a new flag REF_ISPRUNING that will tell the transaction that we
>> do not want to delete this ref from the packed refs.
>
> Interesting.  Since the flag is per ref update, it even would allow
> deleting some refs and pruning others in the same transaction.  Makes
> sense.
>
> Looks like this doesn't batch up multiple ref-prunings into a single
> transaction.  Makes sense (it would just make the period while refs
> are locked longer without having any real benefit).
>
> [...]
>> +#define REF_ISPRUNING        0x0100
>
> Can this conflict with bit values declared elsewhere some day?  It
> would be more comfortable if refs.h also had a note about bits >=
> 0x100 being reserved for private use.
>
> The rest of the patch looks good.
>
> Thanks,
> Jonathan

Re: [PATCH v8 30/44] refs.c: add transaction.status and track OPEN/CLOSED/ERROR

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 3:22 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> Please pull my ref-transactions branch.
>
> I'm at bd5736cb (2014-05-21 13:46) now.
>
>> On Wed, May 21, 2014 at 3:00 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>>> Ronnie Sahlberg wrote:
>
>>>> --- a/refs.c
>>>> +++ b/refs.c
>>>> @@ -3308,6 +3308,12 @@ struct ref_update {
>>>>       const char refname[FLEX_ARRAY];
>>>>  };
>>>>
>>>> +enum ref_transaction_status {
>>>> +     REF_TRANSACTION_OPEN   = 0,
>>>> +     REF_TRANSACTION_CLOSED = 1,
>>>> +     REF_TRANSACTION_ERROR  = 2,
>>>
>>> What is the difference between _TRANSACTION_CLOSED and
>>> _TRANSACTION_ERROR?
>>
>> Closed is a transaction that has been committed successfully, and
>> which we can not do any more updates onto.
>> Error is a transaction that has failed, and which we can not do any
>> more updates onto.
>
> That means that both mean the caller cannot do any more updates,
> right?  Why not call them both _CLOSED?
>
>> The distinction could be useful if in the future we add support to
>> reuse a transaction
>
> If the distinction becomes useful in the future, wouldn't that
> be the moment to add a new state?
>
> I don't mean that there has to be a big useful distinction.  E.g.,
> maybe the idea is that when using a debugger it can be useful to see
> the difference between _ERROR and _CLOSED or something, and I think
> that would be fine as long as the intended meaning is documented
> (e.g., using comments).  My only complaint is that it's hard to
> understand the code and keep track of which status should be used in a
> given place unless there is some distinction between them.

I have documented the transaction states in refs.c

Thanks.
ronnie sahlberg

Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic

[Ronnie Sahlberg]

On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2044,6 +2044,9 @@ static struct ref_lock *lock_ref_sha1_basic(const char *refname,
>>       int missing = 0;
>>       int attempts_remaining = 3;
>>
>> +     if (check_refname_format(refname, REFNAME_ALLOW_ONELEVEL))
>> +             return NULL;
>
> Are we sure that no callers are locking a poorly named ref
> as preparation for repairing it?
>
> To see what works already in that vein, I tried:
>
>         $ git rev-parse HEAD >.git/refs/heads/foo..bar
>         $ git branch -m foo..bar something-saner
>         fatal: Invalid branch name: 'foo..bar'
>
> "git branch -m" has an explicit codepath ("recovery = 1;") to handle
> this case, but it looks like it was not well tested and regressed in
> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has
> the right format, 2011-09-15).
>
> Is what the recovery codepath of branch -m does misguided?  One
> school of thought would be that people with malformed refs are
> responsible for recovering using low-level tools like "mv" and "vi"
> instead of normal git commands since normal git commands should never
> have created such a bad situation.  Another school of thought would
> assert that
>
>  * git commands can have bugs
>  * the format checked by check_refname_format() keeps getting stricter
>    over time
>
> which means it's nice when people can recover with 'update-ref -d'
> or 'branch -m'.  It's not obvious to me what the right thing to do
> here is (maybe a special flag to be attached to a ref update during
> recovery?).
>

I don't think we should allow external caller any other way to
access/modify refs than through
the transaction api. This may make it awkward to handle cases such as
foo/../bar was created  and how would you now delete it ?

The exception to this I think would be 'git fsck'. We could allow fsck
to have low level access to the backing store and allow it to access
the files directly,
or allow fsck to set magic flags that disable various checks.




> Hope that helps,
> Jonathan

Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic

[Jonathan Nieder]

Ronnie Sahlberg wrote:
> On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>>         $ git rev-parse HEAD >.git/refs/heads/foo..bar
>>         $ git branch -m foo..bar something-saner
>>         fatal: Invalid branch name: 'foo..bar'
>>
>> "git branch -m" has an explicit codepath ("recovery = 1;") to handle
>> this case, but it looks like it was not well tested and regressed in
>> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has
>> the right format, 2011-09-15).
>>
>> Is what the recovery codepath of branch -m does misguided?
[...]
> I don't think we should allow external caller any other way to
> access/modify refs than through
> the transaction api. This may make it awkward to handle cases such as
> foo/../bar was created  and how would you now delete it ?
>
> The exception to this I think would be 'git fsck'. We could allow fsck
> to have low level access to the backing store and allow it to access
> the files directly,
> or allow fsck to set magic flags that disable various checks.

Interesting.  Do you mean that 'git fsck' should notice invalid refnames
and have an option to repair them on its own?

That would be a big change from what git fsck currently does --- it's
currently read-only (except for "fsck --lost-found", which is read-only
except in the .git/lost-found/ directory).  If there's an option to
check refnames only and not have to wait for the full check of all
objects then I think it makes sense.

In the meantime, 'git branch -m' and 'git update-ref -d' (which were
the historical ways to do this kind of repair) are already broken, so
at least this patch doesn't seem to make anything worse. :/  (I
haven't checked all callers, though.)

Jonathan

Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref

[Jonathan Nieder]

Ronnie Sahlberg wrote:
> On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Ronnie Sahlberg wrote:

>>> This means that most loose refs will no longer be present after the rename
>>
>> Is this to handle the "git branch -m foo/bar foo" case or for some other
>> purpose?
>
> Yes. That is the main reason.

Thanks.  Please say so in the log message.  Remember that people of
the future debugging and trying to figure out how to fix the latest
regression without breaking something else and whether to just revert
your patch cannot read your mind. ;-)

Hope that helps,
Jonathan

Re: [PATCH v8 40/44] refs.c: call lock_ref_sha1_basic directly from commit

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> Signed-off-by: Ronnie Sahlberg <sahlberg@google.com>
> ---
>  refs.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 39/44] refs.c: move the check for valid refname to lock_ref_sha1_basic

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 10:44 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>> On Wed, May 21, 2014 at 6:42 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>
>>>         $ git rev-parse HEAD >.git/refs/heads/foo..bar
>>>         $ git branch -m foo..bar something-saner
>>>         fatal: Invalid branch name: 'foo..bar'
>>>
>>> "git branch -m" has an explicit codepath ("recovery = 1;") to handle
>>> this case, but it looks like it was not well tested and regressed in
>>> v1.7.8-rc0~19^2~7 (resolve_ref(): verify that the input refname has
>>> the right format, 2011-09-15).
>>>
>>> Is what the recovery codepath of branch -m does misguided?
> [...]
>> I don't think we should allow external caller any other way to
>> access/modify refs than through
>> the transaction api. This may make it awkward to handle cases such as
>> foo/../bar was created  and how would you now delete it ?
>>
>> The exception to this I think would be 'git fsck'. We could allow fsck
>> to have low level access to the backing store and allow it to access
>> the files directly,
>> or allow fsck to set magic flags that disable various checks.
>
> Interesting.  Do you mean that 'git fsck' should notice invalid refnames
> and have an option to repair them on its own?

I think that would be useful.
But it would prompt the user before it does any changes. Even for a
fsck it would be rude to do any changes without asking the user.

'.git/refs/heads/foo^G^G^G.........bar' it not a valid ref.
[s kip] [d elete] [r ename] ?


>
> That would be a big change from what git fsck currently does --- it's
> currently read-only (except for "fsck --lost-found", which is read-only
> except in the .git/lost-found/ directory).  If there's an option to
> check refnames only and not have to wait for the full check of all
> objects then I think it makes sense.
>
> In the meantime, 'git branch -m' and 'git update-ref -d' (which were
> the historical ways to do this kind of repair) are already broken, so
> at least this patch doesn't seem to make anything worse. :/  (I
> haven't checked all callers, though.)
>
> Jonathan

Re: [PATCH v8 38/44] refs.c: pack all refs before we start to rename a ref

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 10:51 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>> On Wed, May 21, 2014 at 4:57 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>>> Ronnie Sahlberg wrote:
>
>>>> This means that most loose refs will no longer be present after the rename
>>>
>>> Is this to handle the "git branch -m foo/bar foo" case or for some other
>>> purpose?
>>
>> Yes. That is the main reason.
>
> Thanks.  Please say so in the log message.  Remember that people of
> the future debugging and trying to figure out how to fix the latest
> regression without breaking something else and whether to just revert
> your patch cannot read your mind. ;-)
>

Done.

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Jonathan Nieder]

Hi,

Ronnie Sahlberg wrote:

> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.
> This flag indicates that the ref does not exist as a loose ref andf only as
> a packed ref. If this is the case we then change the commit code so that
> we skip taking out a lock file and we skip calling delete_ref_loose.

This seems wrong.  Can't someone else create a loose ref which will
shadow the packed ref and break the serializability of updates to this
ref?

The above doesn't explain why we want to avoid locking the loose ref,
but I assume it's for the sake of the "git branch -m foo/bar foo"
case.  For that case, wouldn't the following sequence of filesystem
operations work?

	- create $GIT_DIR/refs/heads/foo/bar.lock
	- create $GIT_DIR/refs/heads/foo.lock
	- if $GIT_DIR/refs/heads/foo/bar exists, add the ref to
	  packed-refs (using the usual lockfile-update mechanism)
	  and unlink $GIT_DIR/refs/heads/foo/bar
	- verify that current foo and foo/bar state are okay.  If
	  not, roll back.
	- unlink $GIT_DIR/refs/heads/foo/bar.lock
	- rmdir $GIT_DIR/refs/heads/foo
	- rename $GIT_DIR/refs/heads/foo.lock into place

Or:

	- create $GIT_DIR/refs/heads/foo/bar.lock
	- create $GIT_DIR/refs/heads/foo.lock
	- verify state of all relevant refs

	- update packed-refs to remove refs/heads/foo/bar and
	  add refs/heads/foo

	- unlink $GIT_DIR/refs/heads/foo/bar
	- unlink $GIT_DIR/refs/heads/foo
	- unlink $GIT_DIR/refs/heads/foo/bar.lock
	- unlink $GIT_DIR/refs/heads/foo.lock

Thanks,
Jonathan

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 11:17 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Ronnie Sahlberg wrote:
>
>> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.
>> This flag indicates that the ref does not exist as a loose ref andf only as
>> a packed ref. If this is the case we then change the commit code so that
>> we skip taking out a lock file and we skip calling delete_ref_loose.
>
> This seems wrong.  Can't someone else create a loose ref which will
> shadow the packed ref and break the serializability of updates to this
> ref?
>
> The above doesn't explain why we want to avoid locking the loose ref,
> but I assume it's for the sake of the "git branch -m foo/bar foo"
> case.  For that case, wouldn't the following sequence of filesystem
> operations work?
>
>         - create $GIT_DIR/refs/heads/foo/bar.lock
>         - create $GIT_DIR/refs/heads/foo.lock
>         - if $GIT_DIR/refs/heads/foo/bar exists, add the ref to
>           packed-refs (using the usual lockfile-update mechanism)
>           and unlink $GIT_DIR/refs/heads/foo/bar
>         - verify that current foo and foo/bar state are okay.  If
>           not, roll back.
>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>         - rmdir $GIT_DIR/refs/heads/foo
>         - rename $GIT_DIR/refs/heads/foo.lock into place
>
> Or:
>
>         - create $GIT_DIR/refs/heads/foo/bar.lock
>         - create $GIT_DIR/refs/heads/foo.lock
>         - verify state of all relevant refs
>
>         - update packed-refs to remove refs/heads/foo/bar and
>           add refs/heads/foo
>
>         - unlink $GIT_DIR/refs/heads/foo/bar
>         - unlink $GIT_DIR/refs/heads/foo
>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>         - unlink $GIT_DIR/refs/heads/foo.lock
>


I removed all the rename_ref related patches for now. rename_ref is
probably best implemented specifically for each backend anyway.

I will still produce a separate patch that will do something like this
you suggested
(since rename_ref is still racy and fragile)

>         - create $GIT_DIR/refs/heads/foo/bar.lock
>         - create $GIT_DIR/refs/heads/foo.lock
>         - verify state of all relevant refs
>
>         - update packed-refs to remove refs/heads/foo/bar and
>           add refs/heads/foo
>
>         - unlink $GIT_DIR/refs/heads/foo/bar
>         - unlink $GIT_DIR/refs/heads/foo
>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>         - unlink $GIT_DIR/refs/heads/foo.lock
>

Thanks
ronnie sahlberg

Re: [PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> --- a/refs.c
> +++ b/refs.c
> @@ -798,11 +798,19 @@ struct name_conflict_cb {
>  	const char *refname;
>  	const char *oldrefname;
>  	const char *conflicting_refname;
> +	const char **skip;
> +	int skipnum;

Would a struct string_list make sense here?  (See
Documentation/technical/api-string-list.txt.)

[...]
>  };
>  
>  static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
>  {
>  	struct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;
> +	int i;
> +	for(i = 0; i < data->skipnum; i++) {

(style nit) missing space after 'for'.

> +		if (!strcmp(entry->name, data->skip[i])) {
> +			return 0;
> +		}

Style: git tends to avoid braces around a single-line if/for/etc body.

[...]
> @@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
>   * conflicting with the name of an existing reference in dir.  If
>   * oldrefname is non-NULL, ignore potential conflicts with oldrefname
>   * (e.g., because oldrefname is scheduled for deletion in the same
> - * operation).
> + * operation). skip contains a list of refs we want to skip checking for
> + * conflicts with. Refs may be skipped due to us knowing that it will
> + * be deleted later during a transaction that deletes one reference and then
> + * creates a new conflicting reference. For example a rename from m to m/m.

This example of "Refs may be skipped due to" seems overly complicated.
Isn't the idea just that skip contains a list of refs scheduled for
deletion in this transaction, since they shouldn't be treated as
conflicts at all (for example when renamining m to m/m)?

I wonder if there's some way to make use of the result of the naive
refname_available check to decide what to do when creating a ref.

E.g.: if a refname would be available except there's a ref being
deleted in the way, we could do one of the following:

 a. delete all relevant loose refs and perform the transaction in
    packed-refs, or

 b. order operations to avoid the D/F conflict, even with loose refs
    (the hardest case is if the ref being deleted uses a directory
    and we want to create a file with the same name.  But that's
    still doable if we're willing to rmdir when needed as part of
    the loop to commit changes)

The packed-refs trick (a) seems much simpler, but either should work.

This could be done e.g. by checking is_refname_available with an empty
list first before doing the real thing with a list of exclusions.

[...]
> @@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
>  	int log = !lstat(git_path("logs/%s", oldrefname), &loginfo);
>  	const char *symref = NULL;
>  
> +	if (!strcmp(oldrefname, newrefname))
> +		return 0;

What is the intended result if I try to rename a nonexistent ref or an
existent symref to its own name?

Sorry to be so fussy about this part.  It's not that I think that this
change is trying to do something bad --- in fact, it's more the
opposite, that I'm excited to see git learning to have a better
understanding and handling of refname D/F conflicts.

That would allow:

 * "git fetch --prune" working as a single transaction even if
   the repository being fetched from removed a refs/heads/topic
   branch and created refs/heads/topic/1 and refs/heads/topic/2

 * "git fast-import" and "git fetch --mirror" learning the same trick

 * fewer code paths having to be touched to be able to (optionally)
   let git actually tolerate D/F conflicts, for people who want to
   have 'topic', 'topic/1', and 'topic/2' branches at the same time.
   This could be turned on by default for remote-tracking refs.  It
   would be especially nice for people on Windows and Mac OS where
   there can be D/F conflicts that people on Linux didn't notice due
   to case-sensitivity.

   Longer term, through a configuration that starts turned off by
   default and has the default flipped as more people have upgraded
   git, this could make D/F conflicts in refnames stop being an error
   altogether.

So it's kind of exciting to see, even though it's fussy to get it
right.

Thanks,
Jonathan

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This version completes the work to convert all ref updates to use transactions.

Finally got through this.  It had thorny bits but generally goes in a
very good direction.  Thanks for a pleasant read.

Feel free to send another iteration if you'd like review for the newer
code.  I expect except for the part about renames that most of what's
left is just nits so it should go faster.

Thanks,
Jonathan

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This patch series can also be found at
> https://github.com/rsahlberg/git/tree/ref-transactions

Thoughts on 65a1cb7b (2014-05-22 12:08):

 01/40 remove ref_transaction_rollback
 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 02/40 constify the sha arguments for ref_transaction_create|delete|update
 still Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 03/40 allow passing NULL to ref_transaction_free
 The third paragraph ("This allows us to write code like") is still
 confusing.  Why not drop that paragraph?

 04/40 --- oh, lunch time.  Will pick up when I get back.

Thanks,
Jonathan

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This patch series can also be found at
> https://github.com/rsahlberg/git/tree/ref-transactions

Thoughts on 65a1cb7b (2014-05-22 12:08):

 04/40 add a strbuf argument to ref_transaction_commit for error logging

 Ideally this would come after the functions it calls so the comment
 "If error is non-NULL we will add an error string to it to explain
 why the transaction failed" would be true already.  But rearranging
 the series for that would be more fuss than it's worth IMHO.

 The sanity check

	int ref_transaction_commit(...)
	{
		int ret = ref_transaction_commit_real(...);
		if (ret && err && !err->len)
			die("BUG: ref_transaction_commit forgot to write an error message");
		return ret;
	}

 shows no instances of forgotten error messages in cases exercised by
 tests, at least.  And it's a step in the right direction.

 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 05/40 add an err argument to repack_without_refs
 unable_to_lock_strbuf could be called unable_to_lock_message (which
 would make its behavior more obvious imho).

 This makes errno meaningful when commit_packed_refs returns an error.
 Probably its API documentation should say so to make it obvious to
 people modifying it in the future to preserve that property or audit
 callers.

 Via the new call to unable_to_lock_..., repack_without_refs cares
 about errno after a failed call to lock_packed_refs.  lock_packed_refs
 can only fail in hold_lock_file_for_update.  hold_lock_file_for_update
 is a thin wrapper around lockfile.c::lock_file.  lock_file can error
 out because

	strlen(path) >= max_path_len
	adjust_shared_perm failed [calls error(), clobbers errno]
	open failed

 So lock_file needs a similar kind of fix, and it's probably worth
 updating API documentation for these calls to make it clear that
 their errno is used (though that's not a new problem since
 repack_without_refs already called unable_to_lock_error).  Could be
 a separate, earlier patch since it's fixing an existing bug.

 06/40 make ref_update_reject_duplicates take a strbuf argument for errors
 still Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 07/40 add an err argument to delete_ref_loose
 The new unlink_or_err has an odd contract when the err argument is passed.
 On error:

  * if errno != ENOENT, it will append a message to err and return -1 (good)
  * if errno == ENOENT, it will not append a message to err but will
    still return -1.

 Perhaps it should return 0 when errno == ENOENT.  After all, in that
 case the file does not exist any more, which is all we wanted.  And it
 would save the caller from having to inspect errno.

 On failure we seem to add our own message to err, too, so the resulting
 message would be something like

	fatal: unable to unlink .git/refs/heads/master: \
	Permission deniedfailed to delete loose ref '.git/refs/heads/master.lock'

 The second strbuf_addf is probably not needed.

 08/40 make update_ref_write update a strbuf on failure
 still Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
 
 09/40 log transaction error from the update_ref
 No actual functional change intended, right?  I'd say something like
 "update-ref: use err argument to get error from ref_transaction_commit"
 or something similar to make it clearer that this is just about
 changing APIs.  Or if there's an intended functional change, then the
 commit message could say something about that.

 10/40 remove the onerr argument to ref_transaction_commit
 still Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Ronnie Sahlberg]

I hate rename_ref :-)


I have reworked the transaction code to special case the deletion of
the old ref for n/n -> n  and n -> n/n renames
so that we can carefully avoid n/n.lock files to exist or prevent the
directory <-> file transition for n during these renames.

This should allow us to have rename_ref becoming reasonably
implementation agnostic, aside for that it wants to lstat the ref to
see if it is a soft link, and re-use the code for other refs backends.


Please see the ref-transaction branch.


On Thu, May 22, 2014 at 12:12 PM, Ronnie Sahlberg <sahlberg@google.com> wrote:
> On Thu, May 22, 2014 at 11:17 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Hi,
>>
>> Ronnie Sahlberg wrote:
>>
>>> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.
>>> This flag indicates that the ref does not exist as a loose ref andf only as
>>> a packed ref. If this is the case we then change the commit code so that
>>> we skip taking out a lock file and we skip calling delete_ref_loose.
>>
>> This seems wrong.  Can't someone else create a loose ref which will
>> shadow the packed ref and break the serializability of updates to this
>> ref?
>>
>> The above doesn't explain why we want to avoid locking the loose ref,
>> but I assume it's for the sake of the "git branch -m foo/bar foo"
>> case.  For that case, wouldn't the following sequence of filesystem
>> operations work?
>>
>>         - create $GIT_DIR/refs/heads/foo/bar.lock
>>         - create $GIT_DIR/refs/heads/foo.lock
>>         - if $GIT_DIR/refs/heads/foo/bar exists, add the ref to
>>           packed-refs (using the usual lockfile-update mechanism)
>>           and unlink $GIT_DIR/refs/heads/foo/bar
>>         - verify that current foo and foo/bar state are okay.  If
>>           not, roll back.
>>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>>         - rmdir $GIT_DIR/refs/heads/foo
>>         - rename $GIT_DIR/refs/heads/foo.lock into place
>>
>> Or:
>>
>>         - create $GIT_DIR/refs/heads/foo/bar.lock
>>         - create $GIT_DIR/refs/heads/foo.lock
>>         - verify state of all relevant refs
>>
>>         - update packed-refs to remove refs/heads/foo/bar and
>>           add refs/heads/foo
>>
>>         - unlink $GIT_DIR/refs/heads/foo/bar
>>         - unlink $GIT_DIR/refs/heads/foo
>>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>>         - unlink $GIT_DIR/refs/heads/foo.lock
>>
>
>
> I removed all the rename_ref related patches for now. rename_ref is
> probably best implemented specifically for each backend anyway.
>
> I will still produce a separate patch that will do something like this
> you suggested
> (since rename_ref is still racy and fragile)
>
>>         - create $GIT_DIR/refs/heads/foo/bar.lock
>>         - create $GIT_DIR/refs/heads/foo.lock
>>         - verify state of all relevant refs
>>
>>         - update packed-refs to remove refs/heads/foo/bar and
>>           add refs/heads/foo
>>
>>         - unlink $GIT_DIR/refs/heads/foo/bar
>>         - unlink $GIT_DIR/refs/heads/foo
>>         - unlink $GIT_DIR/refs/heads/foo/bar.lock
>>         - unlink $GIT_DIR/refs/heads/foo.lock
>>
>
> Thanks
> ronnie sahlberg

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> This patch series can also be found at
> https://github.com/rsahlberg/git/tree/ref-transactions

Continuing with the review of 65a1cb7b (2014-05-22 12:08):

 11/40 change ref_transaction_update() to do error checking and return status
 The "there will be in the future" sounds ominous.  Do you have an
 example in mind?  E.g., I suppose it would be nice if _update could
 notice D/F conflicts or connection to a database server closing early,
 but it's not clear to me whether the kind of errors you're talking
 about are that or something else.

 With or without such a clarification,
 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 12/40 change ref_transaction_create to do error checking and return status
 What does "On failure the err buffer will be updated" mean?  Will
 it clear err and replace it with a message, append to err, or
 something else?  Does the message explain the context or is the
 caller responsible for adding to it?  Does the message end with a
 newline or is the caller responsible for adding one when printing it
 out?

 For cases like this where lots of functions have a similar API,
 API comments start to become potentially repetitive.  It might be
 better to explain conventions at the top of the file or in
 Documentation/technical/api-refs.txt and say "See the top of the
 file for error handling conventions" or "Returns non-zero and
 appends a message to err on error.  See
 Documentation/technical/api-refs.txt for more details on error
 handling".

 13/40 ref_transaction_delete to check for error and return status
 Each successive commit dropped something from its subject. :)
 (First the (), then the verb.)

 Same comments as before about an example being useful for the
 log message and the API documentation on error handling being a
 bit vague.

 14/40 make ref_transaction_begin take an err argument
 I found the "failed to connect to mysql" example instructive while
 doing reviews.  Perhaps it would be worth mentioning in the commit
 message.

 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 15/40 add transaction.status and track OPEN/CLOSED/ERROR
 It says an ERRORed transaction cannot be committed and can be rolled
 back by calling _free.  Can a CLOSED transaction be committed or
 _freed?

 What does "faild" mean in the documentation comments?  (Maybe
 "non-OPEN"?)

 In the previous version of this patch passing a non-OPEN transaction
 would die("BUG: "...) to diagnose the caller's mistake.  Now I'm
 confused about the API: it seems you're allowed to pass a non-OPEN
 transaction but it doesn't append a message to 'err' in that case.
 Is this meant as a way to save the caller some typing, like
 fwrite/fclose do?  (I've found people often make mistakes with the
 fwrite API fwiw but can understand the appeal of it.)

 Maybe with more context I'd like this.  As is, it feels like a step
 in the wrong direction.

 16/40 tag: use ref transactions when doing updates
 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 17/40 replace: use ref transactions when doing updates
 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

 18/40 commit: use ref transactions for updates
 Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
 
 19/40 sequencer: use ref transactions for all ref updates
 This would be a lot simpler if the "ref_transaction_commit should not
 free the transaction" patch came before it (yes, sorry, killing the
 fun).  I can push the result of a rebase doing that somewhere if you
 like.

 20/40 fast-import: change update_branch to use ref transactions
 Likewise.

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Jonathan Nieder]

Ronnie Sahlberg wrote:

> I hate rename_ref :-)
>
> I have reworked the transaction code to special case the deletion of
> the old ref for n/n -> n  and n -> n/n renames
> so that we can carefully avoid n/n.lock files to exist or prevent the
> directory <-> file transition for n during these renames.

Thanks.

I suspect the REF_ISRENAME flag shouldn't be needed.  Wouldn't
something like the following work (in _commit)?

	Allocate work space
	Copy sort, and reject duplicate refs
	Acquire all locks while verifying old values
		This calls is_refname_available.
		If a refname is unavailable, goto slowpath.
	Perform updates first so live commits remain referenced.
	Perform deletes now that updates are safely completed.
	Done.

 slowpath:
	Acquire locks, telling is_refname_available not to worry
		about deleted refs.
	Lock packed-refs.
	Add all relevant refs to packed-refs (pack_if_possible_fn).
	Commit packed-refs.
	Unlink the corresponding loose refs so packed-refs
		becomes authoritative for them.
	Lock packed-refs.
	Perform updates and removals in the packed-refs cache.
	Commit packed-refs.
	Release locks.
	Done.

This wouldn't be any slower for the case without D/F conflicts, and in
the D/F conflict case, it should work for arbitrary transactions
that want to remove one ref to make room for another.

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Jonathan Nieder]

Jonathan Nieder wrote:
> Ronnie Sahlberg wrote:

>> I hate rename_ref :-)
>>
>> I have reworked the transaction code to special case the deletion of
>> the old ref for n/n -> n  and n -> n/n renames
>> so that we can carefully avoid n/n.lock files to exist or prevent the
>> directory <-> file transition for n during these renames.
[...]
> 	Unlink the corresponding loose refs so packed-refs
> 		becomes authoritative for them.
> 	Lock packed-refs.
> 	Perform updates and removals in the packed-refs cache.
> 	Commit packed-refs.

... or is the problem that the reflogs conflict?

How does rename_ref handle propagating the reflog from the old
name to the new name, by the way?

Thanks,
Jonathan

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 4:53 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Jonathan Nieder wrote:
>> Ronnie Sahlberg wrote:
>
>>> I hate rename_ref :-)
>>>
>>> I have reworked the transaction code to special case the deletion of
>>> the old ref for n/n -> n  and n -> n/n renames
>>> so that we can carefully avoid n/n.lock files to exist or prevent the
>>> directory <-> file transition for n during these renames.
> [...]
>>       Unlink the corresponding loose refs so packed-refs
>>               becomes authoritative for them.
>>       Lock packed-refs.
>>       Perform updates and removals in the packed-refs cache.
>>       Commit packed-refs.
>
> ... or is the problem that the reflogs conflict?
>
> How does rename_ref handle propagating the reflog from the old
> name to the new name, by the way?

I haven't touched that yet, but we can fix it after the next series
when we have transaction support for reflogs.

It still renames the reflog via the magic name
#define TMP_RENAMED_LOG  "logs/refs/.tmp-renamed-log"

and thus, if you run two renames at the same time there is a race that
might make you end up with the wrong reflog after the rename.



>
> Thanks,
> Jonathan

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Michael Haggerty]

On 05/15/2014 07:29 PM, Ronnie Sahlberg wrote:
> Add a new flag REF_ISPACKONLY that we can use in ref_transaction_delete.
> This flag indicates that the ref does not exist as a loose ref andf only as
> a packed ref. If this is the case we then change the commit code so that
> we skip taking out a lock file and we skip calling delete_ref_loose.
> Check for this flag and die(BUG:...) if used with _update or _create.
> 
> At the start of the transaction, before we even start locking any refs,
> we add all such REF_ISPACKONLY refs to delnames so that we have a list of
> all pack only refs that we will be deleting during this transaction.

Let me make a few comments about how I think ref packing fits into the
scheme for pluggable reference back ends.  The comments may or may not
be relevant to this particular commit.

We want to have a reference API that can be implemented by various back
ends.  Obviously we need operations to read a reference, enumerate
existing references, update references within a transaction, and
probably do similar things with symbolic references.

The status quo is that we have a single reference back end consisting of
loose references sitting on top of packed references.

But really, loose references and packed references are two relatively
independent reference back ends [1].  We just happen to use them layered
on top of each other.

This suggests to me that our current structure is best modeled as two
independent reference back ends, with a third implementation of the same
reference API whose job it is to compose the first two.  In pseudocode,

    interface ReferenceBackend:
        read_ref(refname)
        __iter__(...)
        begin_transaction(...)
        update_reference(...)
        ...
        commit_transaction(...)

    class LooseReferenceBackend(ReferenceBackend):
        ...

    class PackedReferenceBackend(ReferenceBackend):
        ...

    class StackedReferenceBackend(ReferenceBackend):
        def __init__(self, backend1, backend2):
            self.backend1 = backend1
            self.backend2 = backend2

        def read_ref(refname):
            try:
                return backend1.read_ref(refname)
            except ReferenceNotFound:
                return backend2.read_ref(refname)

        def __iter__(...):
            i1 = self.backend1.iterate()
            i2 = self.backend2.iterate()
            while i1.has_next() and i2.has_next():
                if i1.peek_next() < i2.peek_next():
                    yield i1.next()
                elif i1.peek_next() == i2.peek_next():
                    yield i1.next()
                    i2.next() # discard
                else
                    yield i2.next()
            while i1.has_next():
                yield i1.next()
            while i2.has_next():
                yield i2.next()

        def pack_refs(...):
            ...

et cetera.

>From this point of view it is clear that packing refs is not an
operation that belongs in the ReferenceBackend API, but rather in the
StackedReferenceBackend interface.  The reason is that it doesn't affect
how references appear to the outside world, but rather just reorganizes
them between the two backends to which StackedReferenceBackend
delegates.  The *implementation* of pack_refs() involves adding
references to backend2 and deleting them from backend1.  But since
adding and deleting references *are* part of the ReferenceBackend API,
StackedReferenceBackend can compose *any arbitrary two* reference
backends (including other StackedReferenceBackends).  Pruning references
that have been packed is also not part of the ReferenceBackend API.
Rather it is a plain old deletion of a reference from backend1 of a
StackedReferenceBackend.

I haven't thought through all of the ramifications of this design, but
I'm pretty sure it's where we want to be headed.

Michael

[1] Forget for the sake of this discussion that we can't store symbolic
references as packed refs.

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Jonathan Nieder]

Hi,

Michael Haggerty wrote:

> The status quo is that we have a single reference back end consisting of
> loose references sitting on top of packed references.
>
> But really, loose references and packed references are two relatively
> independent reference back ends [1].  We just happen to use them layered
> on top of each other.
>
> This suggests to me that our current structure is best modeled as two
> independent reference back ends, with a third implementation of the same
> reference API whose job it is to compose the first two.
[...]
> [1] Forget for the sake of this discussion that we can't store symbolic
> references as packed refs.

I find it hard to forget that. ;-)  More to the point, the trouble
with loose refs and packed refs as independent reference backends is
that neither has very good performance characteristics.  Enumerating
many loose refs is slow.  Adding a new packed ref to a large list is
also slow.  Git currently uses both loose and packed refs in a way
that allows each to overcome the limitations of the other, and the
fact that it involves two on-disk data structures seems to me like an
implementation detail of how it achieves that.

So I believe most git code should not have to know about the
difference between loose and packed refs (or the upper and lower
layer) to allow the details of the layering can be tuned in low-level
ref handling code.

On the other hand, from a code structure perspective I can easily
believe that implementing some subset (or maybe even all) of the
reference backend API for loose refs and packed refs separately and
providing a separate file describing how to compose them might be the
cleanest way to write this code.  It's more general layering that
seems to lie in the direction of madness.

Maybe I'm wrong and people will find lots of use for combinations like
 * loose refs shadowing an sqlite database
 * tdb shadowing mysql
 * etc
It's easy to prove a naysayer wrong with code and I don't want to
discourage that.

For the topic at hand it's relevant because packed-refs have
properties that make some operations (certain deletion/ref creation
combinations) much less fussy than loose refs, and it would be nice to
be able to take advantage of that.  In the long term I would like to
see git taking advantage of that when someone tries to fetch refs with
names that would conflict on the filesystem (e.g., topic, topic/a,
topic/b).

Thanks,
Jonathan

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Jonathan Nieder]

Ronnie Sahlberg wrote:
> On Thu, May 22, 2014 at 4:53 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:

>> ... or is the problem that the reflogs conflict?
>>
>> How does rename_ref handle propagating the reflog from the old
>> name to the new name, by the way?
>
> I haven't touched that yet, but we can fix it after the next series
> when we have transaction support for reflogs.
>
> It still renames the reflog via the magic name
> #define TMP_RENAMED_LOG  "logs/refs/.tmp-renamed-log"

Okay, after looking at what "git branch -m" currently does (deletes
the old ref and creates the new one in separate transactions), I'm
convinced that this take-locks-for-a-shorter-period-than-is-necessary
trick is an improvement relative to the status quo.  It gets rid of
the window with objects unreferenced between the ref deletion and ref
creation, which is a nice improvement.

I still haven't looked closely at the details of the code change but
the idea looks sane as a first step.

Later we can try to get the semantics right for this kind of
delete/create pair in general transactions, if someone is interested.
:)  No need for the perfect to be the enemy of the good in the
meantime.

Thanks,
Jonathan

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Michael Haggerty]

On 05/23/2014 05:53 PM, Jonathan Nieder wrote:
> Hi,
> 
> Michael Haggerty wrote:
> 
>> The status quo is that we have a single reference back end consisting of
>> loose references sitting on top of packed references.
>>
>> But really, loose references and packed references are two relatively
>> independent reference back ends [1].  We just happen to use them layered
>> on top of each other.
>>
>> This suggests to me that our current structure is best modeled as two
>> independent reference back ends, with a third implementation of the same
>> reference API whose job it is to compose the first two.
> [...]
>> [1] Forget for the sake of this discussion that we can't store symbolic
>> references as packed refs.
> 
> I find it hard to forget that. ;-)  More to the point, the trouble
> with loose refs and packed refs as independent reference backends is
> that neither has very good performance characteristics.  Enumerating
> many loose refs is slow.  Adding a new packed ref to a large list is
> also slow.  Git currently uses both loose and packed refs in a way
> that allows each to overcome the limitations of the other, and the
> fact that it involves two on-disk data structures seems to me like an
> implementation detail of how it achieves that.

I'm not advocating that we use loose refs or packed refs alone.  But I
like the code decoupling that this implementation would (I predict) yield.

My main point was that pack-refs is not an integral part of the
reference API but rather a tuning feature very specific to the
loose/packed reference storage scheme.

> So I believe most git code should not have to know about the
> difference between loose and packed refs (or the upper and lower
> layer) to allow the details of the layering can be tuned in low-level
> ref handling code.
> 
> On the other hand, from a code structure perspective I can easily
> believe that implementing some subset (or maybe even all) of the
> reference backend API for loose refs and packed refs separately and
> providing a separate file describing how to compose them might be the
> cleanest way to write this code.  It's more general layering that
> seems to lie in the direction of madness.
> 
> Maybe I'm wrong and people will find lots of use for combinations like
>  * loose refs shadowing an sqlite database
>  * tdb shadowing mysql
>  * etc
> It's easy to prove a naysayer wrong with code and I don't want to
> discourage that.

I admit that I don't have any other layered storage schemes in mind.

> For the topic at hand it's relevant because packed-refs have
> properties that make some operations (certain deletion/ref creation
> combinations) much less fussy than loose refs, and it would be nice to
> be able to take advantage of that.  In the long term I would like to
> see git taking advantage of that when someone tries to fetch refs with
> names that would conflict on the filesystem (e.g., topic, topic/a,
> topic/b).

A transition to allowing D/F-conflicting reference names has two very
distinct aspects to it:

1. Changing to how references (and reflogs!) are stored to make it
technically possible to store such references.

2. Removing restrictions on actually creating such references.

We can take step 1 any time because it is a purely local decision.
Though I think you would need a repository format bump to allow it.
Even though you could work around the D/F problem for references by
packing problematic ones, it is a kludge with a potentially significant
performance cost.  And we have the same problem with reflogs, with no
analogous kludge.  I'd rather make a clean break, for example mapping
reference names into filenames using some kind of escaping of special
characters and suffixes like ".d" and ".f" to prevent directories and
files from conflicting.  Maybe (depending on the OS and/or filesystem?)
escape all non-ASCII characters or even all non-lower-case ASCII
characters to prevent problems with case sensitivity, internal vs.
filesystem character encodings, and NFC vs NFD.

But we won't want to take step 2 until Git clients that are 1-capable
are widespread; otherwise people with different client versions will
have trouble collaborating.  Maybe step 2 should be governed by a
configuration option with three settings:

    FORBIDDEN - don't allow references with D/F conflicts to exist
        in this repository

    NO_CREATE - don't allow the creation of such references locally,
        but accept them from remote sources via commands like "fetch".
        This setting could be used to avoid creating problems for
        collaborators.

    ALLOWED - no restriction on the creation of references with D/F
        conflicts.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> This suggests to me that our current structure is best modeled as two
> independent reference back ends, with a third implementation of the same
> reference API whose job it is to compose the first two.  In pseudocode,
> ...

That is an interesting view.

How does reflog fit into the picture?  Is it a completely
independent thing that is called from any implementation of
ReferenceBackend interface?

> From this point of view it is clear that packing refs is not an
> operation that belongs in the ReferenceBackend API, but rather in the
> StackedReferenceBackend interface.

When an implementation of ReferenceBackend has skewed performance
characteristics (e.g. PackedReferenceBackend really prefers to be
modified in bulk), how would that interact with the abstraction?

For example, when the application does:

    begin_transaction()
    for ref in many_refs():
	delete_reference(ref)
    commit_transaction()

StackedReferenceBackend() that consists of Loose on top of Packed
may want to implement the commit phase like so:

    - tell Packed backend to repack without the deleted refs
    - tell Loose backend to delete the deleted refs

But the above would not quite work, as somebody needs to remove logs
for refs that were only in the Packed backend, and "repack without
these refs" API supported by the Packed backend cannot be that
somebody---"repack packed-refs without A B C" cannot unconditionally
remove logs for A B C without checking if A B C exists as Loose.



    

Re: [PATCH v8 42/44] refs.c: pass a skip list to name_conflict_fn

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 12:27 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -798,11 +798,19 @@ struct name_conflict_cb {
>>       const char *refname;
>>       const char *oldrefname;
>>       const char *conflicting_refname;
>> +     const char **skip;
>> +     int skipnum;
>
> Would a struct string_list make sense here?  (See
> Documentation/technical/api-string-list.txt.)

It would. But it is better to do that change later.
Currently we have both repack_without_ref and repack_without_refs that
take the same char ** argument.
After next series we will have removed one of these functions and have
an easier API to modify (once we start tracking the skiplist as part
of the transaction instead).

Instead of doing this change and then redoing it once we move a lot of
the actual work from _commit  to _update
I will do this change towards the end of the next series.

>
> [...]
>>  };
>>
>>  static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
>>  {
>>       struct name_conflict_cb *data = (struct name_conflict_cb *)cb_data;
>> +     int i;
>> +     for(i = 0; i < data->skipnum; i++) {
>

Fixed.

> (style nit) missing space after 'for'.
>
>> +             if (!strcmp(entry->name, data->skip[i])) {
>> +                     return 0;
>> +             }
>
> Style: git tends to avoid braces around a single-line if/for/etc body.
>

Fixed.

> [...]
>> @@ -817,15 +825,21 @@ static int name_conflict_fn(struct ref_entry *entry, void *cb_data)
>>   * conflicting with the name of an existing reference in dir.  If
>>   * oldrefname is non-NULL, ignore potential conflicts with oldrefname
>>   * (e.g., because oldrefname is scheduled for deletion in the same
>> - * operation).
>> + * operation). skip contains a list of refs we want to skip checking for
>> + * conflicts with. Refs may be skipped due to us knowing that it will
>> + * be deleted later during a transaction that deletes one reference and then
>> + * creates a new conflicting reference. For example a rename from m to m/m.
>
> This example of "Refs may be skipped due to" seems overly complicated.
> Isn't the idea just that skip contains a list of refs scheduled for
> deletion in this transaction, since they shouldn't be treated as
> conflicts at all (for example when renamining m to m/m)?
>
> I wonder if there's some way to make use of the result of the naive
> refname_available check to decide what to do when creating a ref.
>
> E.g.: if a refname would be available except there's a ref being
> deleted in the way, we could do one of the following:
>
>  a. delete all relevant loose refs and perform the transaction in
>     packed-refs, or
>
>  b. order operations to avoid the D/F conflict, even with loose refs
>     (the hardest case is if the ref being deleted uses a directory
>     and we want to create a file with the same name.  But that's
>     still doable if we're willing to rmdir when needed as part of
>     the loop to commit changes)
>
> The packed-refs trick (a) seems much simpler, but either should work.
>
> This could be done e.g. by checking is_refname_available with an empty
> list first before doing the real thing with a list of exclusions.
>
> [...]
>> @@ -2592,6 +2609,9 @@ int rename_ref(const char *oldrefname, const char *newrefname, const char *logms
>>       int log = !lstat(git_path("logs/%s", oldrefname), &loginfo);
>>       const char *symref = NULL;
>>
>> +     if (!strcmp(oldrefname, newrefname))
>> +             return 0;
>
> What is the intended result if I try to rename a nonexistent ref or an
> existent symref to its own name?

Yeah, I should get rid of that.

I have renoved the rename_ref patch and moved it to the next series.
I think we can solve this easier/better once we have the other patches in first.

>
> Sorry to be so fussy about this part.  It's not that I think that this
> change is trying to do something bad --- in fact, it's more the
> opposite, that I'm excited to see git learning to have a better
> understanding and handling of refname D/F conflicts.
>
> That would allow:
>
>  * "git fetch --prune" working as a single transaction even if
>    the repository being fetched from removed a refs/heads/topic
>    branch and created refs/heads/topic/1 and refs/heads/topic/2
>
>  * "git fast-import" and "git fetch --mirror" learning the same trick
>
>  * fewer code paths having to be touched to be able to (optionally)
>    let git actually tolerate D/F conflicts, for people who want to
>    have 'topic', 'topic/1', and 'topic/2' branches at the same time.
>    This could be turned on by default for remote-tracking refs.  It
>    would be especially nice for people on Windows and Mac OS where
>    there can be D/F conflicts that people on Linux didn't notice due
>    to case-sensitivity.
>
>    Longer term, through a configuration that starts turned off by
>    default and has the default flipped as more people have upgraded
>    git, this could make D/F conflicts in refnames stop being an error
>    altogether.
>
> So it's kind of exciting to see, even though it's fussy to get it
> right.
>
> Thanks,
> Jonathan

Re: [PATCH v8 00/44] Use ref transactions for all ref updates

[Ronnie Sahlberg]

On Thu, May 22, 2014 at 4:08 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Ronnie Sahlberg wrote:
>
>> This patch series can also be found at
>> https://github.com/rsahlberg/git/tree/ref-transactions
>
> Continuing with the review of 65a1cb7b (2014-05-22 12:08):
>
>  11/40 change ref_transaction_update() to do error checking and return status
>  The "there will be in the future" sounds ominous.  Do you have an
>  example in mind?  E.g., I suppose it would be nice if _update could
>  notice D/F conflicts or connection to a database server closing early,
>  but it's not clear to me whether the kind of errors you're talking
>  about are that or something else.
>

Updated the message.
Next series moves both locking as well as checking for name conflicts
to _update.

>  With or without such a clarification,
>  Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
>  12/40 change ref_transaction_create to do error checking and return status
>  What does "On failure the err buffer will be updated" mean?  Will
>  it clear err and replace it with a message, append to err, or
>  something else?  Does the message explain the context or is the
>  caller responsible for adding to it?  Does the message end with a
>  newline or is the caller responsible for adding one when printing it
>  out?

I have updated the documentation.
Message is appended to the string buffer. Caller is required to
strbuf_reset before calling the transaction if caller wants only most
recent error instead of all errors appended one by one.


>
>  For cases like this where lots of functions have a similar API,
>  API comments start to become potentially repetitive.  It might be
>  better to explain conventions at the top of the file or in
>  Documentation/technical/api-refs.txt and say "See the top of the
>  file for error handling conventions" or "Returns non-zero and
>  appends a message to err on error.  See
>  Documentation/technical/api-refs.txt for more details on error
>  handling".

Done.

>
>  13/40 ref_transaction_delete to check for error and return status
>  Each successive commit dropped something from its subject. :)
>  (First the (), then the verb.)

Done.

>
>  Same comments as before about an example being useful for the
>  log message and the API documentation on error handling being a
>  bit vague.
>
>  14/40 make ref_transaction_begin take an err argument
>  I found the "failed to connect to mysql" example instructive while
>  doing reviews.  Perhaps it would be worth mentioning in the commit
>  message.
>
>  Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
>  15/40 add transaction.status and track OPEN/CLOSED/ERROR
>  It says an ERRORed transaction cannot be committed and can be rolled
>  back by calling _free.  Can a CLOSED transaction be committed or
>  _freed?
>
>  What does "faild" mean in the documentation comments?  (Maybe
>  "non-OPEN"?)
>
>  In the previous version of this patch passing a non-OPEN transaction
>  would die("BUG: "...) to diagnose the caller's mistake.  Now I'm
>  confused about the API: it seems you're allowed to pass a non-OPEN
>  transaction but it doesn't append a message to 'err' in that case.
>  Is this meant as a way to save the caller some typing, like
>  fwrite/fclose do?  (I've found people often make mistakes with the
>  fwrite API fwiw but can understand the appeal of it.)
>
>  Maybe with more context I'd like this.  As is, it feels like a step
>  in the wrong direction.
>
>  16/40 tag: use ref transactions when doing updates
>  Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
>  17/40 replace: use ref transactions when doing updates
>  Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
>  18/40 commit: use ref transactions for updates
>  Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
>  19/40 sequencer: use ref transactions for all ref updates
>  This would be a lot simpler if the "ref_transaction_commit should not
>  free the transaction" patch came before it (yes, sorry, killing the
>  fun).  I can push the result of a rebase doing that somewhere if you
>  like.

Beeing done.

>
>  20/40 fast-import: change update_branch to use ref transactions
>  Likewise.

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Michael Haggerty]

On 05/27/2014 08:27 PM, Junio C Hamano wrote:
> Michael Haggerty <mhagger@alum.mit.edu> writes:
> 
>> This suggests to me that our current structure is best modeled as two
>> independent reference back ends, with a third implementation of the same
>> reference API whose job it is to compose the first two.  In pseudocode,
>> ...
> 
> That is an interesting view.
> 
> How does reflog fit into the picture?  Is it a completely
> independent thing that is called from any implementation of
> ReferenceBackend interface?

That's a good question.  It certainly wouldn't work for each of the
loose and packed backends to try to do logging.  I think the best
solution would be to have a logging wrapper as explained below.

>> From this point of view it is clear that packing refs is not an
>> operation that belongs in the ReferenceBackend API, but rather in the
>> StackedReferenceBackend interface.
> 
> When an implementation of ReferenceBackend has skewed performance
> characteristics (e.g. PackedReferenceBackend really prefers to be
> modified in bulk), how would that interact with the abstraction?
> 
> For example, when the application does:
> 
>     begin_transaction()
>     for ref in many_refs():
> 	delete_reference(ref)
>     commit_transaction()
> 
> StackedReferenceBackend() that consists of Loose on top of Packed
> may want to implement the commit phase like so:
> 
>     - tell Packed backend to repack without the deleted refs
>     - tell Loose backend to delete the deleted refs

I think for any two backends that are stacked, you would need to break
down a transaction as follows (here generalized to include not only
deletions):

    packed->begin_transaction()
    loose->begin_transaction()

    # And this part is done within stacked->commit_transaction():
    for entry in many_ref_updates():
        if have_old:
            stacked->verify_reference(ref, old_sha1)

        if entry is a delete:
            packed->delete_reference(entry)
        loose->update_reference(entry)

    if (!packed->commit_transaction())
        loose->commit_transaction()

Verifying old values is impossible to do batchwise with the current API,
because the old value of the packed ref has to be verified if and only
if there is no corresponding loose ref.

> But the above would not quite work, as somebody needs to remove logs
> for refs that were only in the Packed backend, and "repack without
> these refs" API supported by the Packed backend cannot be that
> somebody---"repack packed-refs without A B C" cannot unconditionally
> remove logs for A B C without checking if A B C exists as Loose.

Correct.  That's another reason that logging has to be the
responsibility of something at the "stacked" level of abstraction or higher.

I think the logging should be done by yet another outer layer of wrapper
that only does the logging, while also passing all updates down 1:1 to
the backend that it wraps (which in our case would be a stacked
backend).  Then the loose and packed backends could remain completely
ignorant of the fact that reference updates can be logged.  I think the
logging layer could implement the same reference backend API as the
other backends.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> I think for any two backends that are stacked, you would need to break
> down a transaction as follows (here generalized to include not only
> deletions):
>
>     packed->begin_transaction()
>     loose->begin_transaction()
>
>     # And this part is done within stacked->commit_transaction():
>     for entry in many_ref_updates():
>         if have_old:
>             stacked->verify_reference(ref, old_sha1)
>
>         if entry is a delete:
>             packed->delete_reference(entry)
>         loose->update_reference(entry)
>
>     if (!packed->commit_transaction())
>         loose->commit_transaction()

Ugggly.  In general you would need to worry about the case where the
first commit succeeds and then the second commit fails, wouldn't
you?

The above happens not to break horribly wrong for the "Loose on top
of Packed" layering, in that the refs you wanted to delete are only
gone from Packed but still are in Loose, and the end result would be
some of them are really gone (because they weren't in Loose) and
some others remain (because they were in Loose), and at least you
did not lose any ref you did not want to discard.  But it still is
not what should happen in a proper "transaction".

>> But the above would not quite work, as somebody needs to remove logs
>> for refs that were only in the Packed backend, and "repack without
>> these refs" API supported by the Packed backend cannot be that
>> somebody---"repack packed-refs without A B C" cannot unconditionally
>> remove logs for A B C without checking if A B C exists as Loose.
>
> Correct.  That's another reason that logging has to be the
> responsibility of something at the "stacked" level of abstraction or higher.
>
> I think the logging should be done by yet another outer layer of
> wrapper that only does the logging, while also passing all updates
> down 1:1 to the backend that it wraps (which in our case would be
> a stacked backend). ... Then the loose and packed backends could
> remain completely ignorant of the fact that reference updates can
> be logged.

That would mean that Loose (or Packed) class cannot be used as-is
and always needs to be wrapped with the layer that does the logging,
no?  In a system with "only packed-refs, no loose", you would want
Packed.deleteRefs() to remove the named refs ref and their reflogs,
but that would mean that the Layered wrapper that uses Loose
overlayed on Packed cannot call that method, because it does not
want reflogs of the refs in Packed covered by the ones in Loose.

Re: [PATCH v8 41/44] refs.c: add a new flag for transaction delete for refs we know are packed only

[Michael Haggerty]

On 05/28/2014 06:58 PM, Junio C Hamano wrote:
> Michael Haggerty <mhagger@alum.mit.edu> writes:
> 
>> I think for any two backends that are stacked, you would need to break
>> down a transaction as follows (here generalized to include not only
>> deletions):
>>
>>     packed->begin_transaction()
>>     loose->begin_transaction()
>>
>>     # And this part is done within stacked->commit_transaction():
>>     for entry in many_ref_updates():
>>         if have_old:
>>             stacked->verify_reference(ref, old_sha1)
>>
>>         if entry is a delete:
>>             packed->delete_reference(entry)
>>         loose->update_reference(entry)
>>
>>     if (!packed->commit_transaction())
>>         loose->commit_transaction()
> 
> Ugggly.

The ugliness would be encapsulated in the stacked reference backend.  It
wouldn't have to appear in client code.

> In general you would need to worry about the case where the
> first commit succeeds and then the second commit fails, wouldn't
> you?
> 
> The above happens not to break horribly wrong for the "Loose on top
> of Packed" layering, in that the refs you wanted to delete are only
> gone from Packed but still are in Loose, and the end result would be
> some of them are really gone (because they weren't in Loose) and
> some others remain (because they were in Loose), and at least you
> did not lose any ref you did not want to discard.  But it still is
> not what should happen in a proper "transaction".

True, but we don't have proper transactions anyway.  If anything goes
wrong when renaming one of the loose reference lockfiles, we have no way
of reliably rolling back the loose references that have already been
changed.  The word "transaction" as we use it really only suggests that
we are doing our best to change the references all-or-nothing even in
the face of concurrent modifications by other processes.  It certainly
doesn't protect against power failures and stuff like that.

I suppose if you wanted to make reference deletions a little bit more
transaction-like, you could split them into two steps to "loosen" any
packed references and a deletion step that deletes the loose references.
 The advantage of this scheme is that each step requires a transaction
on only a single back end at a time, though it requires part or all of
the other back end to be locked simultaneously.

    # The first two iterations "loosen" any packed references
    # corresponding to references that are to be deleted.  It is purely
    # an internal reorganization that doesn't change the externally-
    # visible values of any references and can be done within a
    # separate transaction:
    for each reference to delete:
        if reference exists packed but not loose:
            create a loose ref with the value from the packed ref
    for each reference to delete:
        if reference exists packed:
            delete packed version of reference

    # Now we only have to delete the loose version of the references.
    # This should be done after activating the packed reference file
    # but while continuing to hold the packed-refs lock:
    for each reference to delete:
        delete loose version of reference

I'd have to think more about whether this all really works generically
without requiring lots of extra round-trips to a possibly-remote
backend.  But I'm not sure we really need this composability in its full
generality.  For example, maybe it is impossible to stack a low-latency
and a high-latency backend together while minimizing round-trips to the
latter.

>>> But the above would not quite work, as somebody needs to remove logs
>>> for refs that were only in the Packed backend, and "repack without
>>> these refs" API supported by the Packed backend cannot be that
>>> somebody---"repack packed-refs without A B C" cannot unconditionally
>>> remove logs for A B C without checking if A B C exists as Loose.
>>
>> Correct.  That's another reason that logging has to be the
>> responsibility of something at the "stacked" level of abstraction or higher.
>>
>> I think the logging should be done by yet another outer layer of
>> wrapper that only does the logging, while also passing all updates
>> down 1:1 to the backend that it wraps (which in our case would be
>> a stacked backend). ... Then the loose and packed backends could
>> remain completely ignorant of the fact that reference updates can
>> be logged.
> 
> That would mean that Loose (or Packed) class cannot be used as-is
> and always needs to be wrapped with the layer that does the logging,
> no?

Correct.

> In a system with "only packed-refs, no loose", you would want
> Packed.deleteRefs() to remove the named refs ref and their reflogs,
> but that would mean that the Layered wrapper that uses Loose
> overlayed on Packed cannot call that method, because it does not
> want reflogs of the refs in Packed covered by the ones in Loose.

The two scenarios would be roughly

    refhandler = new LoggingWrapper(new PackedRefs())

and

    refhandler = new LoggingWrapper(
        new StackedRefs(new LooseRefs(), new PackedRefs())
        )

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu