Re: [PATCH] refs: implement reference transaction hooks

["SZEDER Gábor" <szeder.dev@gmail.com>]

On Tue, Jun 02, 2020 at 10:25:44AM +0200, Patrick Steinhardt wrote:
> The low-level reference transactions used to update references are
> currently completely opaque to the user. While certainly desirable in
> most usecases, there are some which might want to hook into the
> transaction to observe all queued reference updates as well as observing
> the abortion or commit of a prepared transaction.
> 
> One such usecase would be to have a set of replicas of a given Git
> repository, where we perform Git operations on all of the repositories
> at once and expect the outcome to be the same in all of them. While
> there exist hooks already for a certain subset of Git commands that
> could be used to implement a voting mechanism for this, many others
> currently don't have any mechanism for this.
> 
> The above scenario is the motivation for a set of three new hooks that
> reach directly into Git's reference transaction. Each of the following
> new hooks (currently) doesn't accept any parameters and receives the set
> of queued reference updates via stdin:
> 
>     - ref-transaction-prepared gets called when all reference updates
>       have been queued. At this stage, the hook may decide to abort the
>       transaction prematurely by returning a non-zero status code.
> 
>     - ref-transaction-committed gets called when a reference transaction
>       was transmitted and all queued updates have been persisted.
> 
>     - ref-transaction-aborted gets called when a reference transaction
>       was aborted and all queued updates have been rolled back.

The point of reference transactions is that they are atomic, and these
hooks must work together to ensure that.  This raises the question how
these hooks can be updated in an actively used repository.

Having multiple hooks means that they can't be updated atomically, and
git might invoke the new abort hook after the transaction was prepared
with the old hook.  Now, if there were a single 'ref-transaction' hook
(which gets the phase of the transaction ('prepared', 'committed' or
'aborted') as a parameter), then it could be updated atomically by
mv-ing it to place, but even that update can happen in between git
invokes 'ref-transaction prepared' and 'ref-transaction aborted'.

I suppose this issue could be addressed by a single hook which runs
during the whole transaction and some back-and-forth communication
through stdin/out between git and the hook.  However, this would, I'm
afraid, complicate both Git's handling of this hook and the hook as
well, so let's take a step back first: is this something we should
worry about in the first place?

> Given the usecase described above, a voting mechanism can now be
> implemented as a "ref-transaction-prepared" hook: as soon as it gets
> called, it will take all of stdin and use it to cast a vote to a central
> service. When all replicas of the repository agree, the hook will exit
> with zero, otherwise it will abort the transaction by returning
> non-zero. The most important upside is that this will catch _all_
> commands writing references at once, allowing to implement strong
> consistency for reference updates via a single mechanism.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  Documentation/githooks.txt       | 51 ++++++++++++++++++
>  refs.c                           | 67 +++++++++++++++++++++++-
>  t/t1416-ref-transaction-hooks.sh | 88 ++++++++++++++++++++++++++++++++
>  3 files changed, 204 insertions(+), 2 deletions(-)
>  create mode 100755 t/t1416-ref-transaction-hooks.sh
> 
> diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt
> index 81f2a87e88..48f8446943 100644
> --- a/Documentation/githooks.txt
> +++ b/Documentation/githooks.txt
> @@ -404,6 +404,57 @@ Both standard output and standard error output are forwarded to
>  `git send-pack` on the other end, so you can simply `echo` messages
>  for the user.
>  
> +ref-transaction-prepared
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This hook is invoked by any Git command that performs reference
> +updates. It executes as soon as all reference updates were queued to
> +the transaction and locked on disk. This hook executes for every
> +reference transaction that is being prepared and may thus get called
> +multiple times.
> +
> +It takes no arguments, but for each ref to be updated it receives on
> +standard input a line of the format:
> +
> +  <old-value> SP <new-value> SP <ref-name> LF
> +
> +If the hook exits with a non-zero status, the transaction is aborted
> +and the command exits immediately. The
> +<<ref-transaction-aborted,'ref-transaction-aborted'>> hook is not
> +executed in that case.
> +
> +[[ref-transaction-aborted]]
> +ref-transaction-aborted
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This hook is invoked by any Git command that performs reference
> +updates. It executes as soon as a reference transaction is aborted and
> +after all reference locks were released and any changes made to
> +references were rolled back. The hook may get called multiple times or
> +never in case no transaction was aborted.
> +
> +The hook takes no arguments, but for each ref to be updated it

Nit: I found it a bit surprising to read about refs "to be updated" in
the description of the 'aborted' hook, because by the time this hook
is called the update has already been refused.  The same applies to
the 'committed' hook below as well.

> +receives on standard input a line of the format:
> +
> +  <old-value> SP <new-value> SP <ref-name> LF
> +
> +The hook's exit code is discarded by Git.
> +
> +ref-transaction-committed
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +This hook is invoked by any Git command that performs reference
> +updates. It executes as soon as a reference transaction is committed,
> +persisting all changes to disk and releasing any locks. The hook may
> +get called multiple times or never in case no transaction was aborted.
> +
> +The hook takes no arguments, but for each ref to be updated it
> +receives on standard input a line of the format:
> +
> +  <old-value> SP <new-value> SP <ref-name> LF
> +
> +The hook's exit code is discarded by Git.
> +
>  push-to-checkout
>  ~~~~~~~~~~~~~~~~
>