[PATCH v4 0/3] fetch: skip unnecessary tasks when using --negotiate-only

From: Glen Choo <chooglen@google.com>
To: git@vger.kernel.org
Cc: Glen Choo <chooglen@google.com>,
	Jonathan Tan <jonathantanmy@google.com>,
	Junio C Hamano <gitster@pobox.com>
Subject: [PATCH v4 0/3] fetch: skip unnecessary tasks when using --negotiate-only
Date: Wed, 12 Jan 2022 16:44:58 -0800	[thread overview]
Message-ID: <20220113004501.78822-1-chooglen@google.com> (raw)
In-Reply-To: <20211222001134.28933-1-chooglen@google.com>

`git fetch --negotiate-only` is used internally by push negotation and
it behaves very differently from other uses of `git fetch`, e.g. it does
not update refs or fetch objects. But because of how cmd_fetch() is
written, `git fetch --negotiate-only` performs tasks that it shouldn't.
This is results in behavior that is unnecessary at best, and incorrect
at worst:

* Submodules are updated if enabled by recurse.submodules=true, but
  negotiation fetch doesn't actually update the repo, so this doesn't
  make sense.
* Commit graphs will be written if enabled by
  fetch.writeCommitGraph=true, but this is unnecessary because no
  objects are fetched [1]. 
* gc is run, but according to the commit message in [2], we only do this
  because we expect `git fetch` to introduce objects.

Make `git fetch --negotiate-only` handle these tasks more rigorously by
doing the following:

* Make cmd_fetch() skip irrelevant tasks if we know for certain that
objects will not be fetched
* Disable submodule recursion and die() if a user explicitly asks for it

[1] This is also confirmed by Documentation/config/fetch.txt, which
  states that Git should only write commit graphs if a pack-file is
  downloaded.
[2] 131b8fcbfb (fetch: run gc --auto after fetching, 2013-01-26)

Changes since v3:
* change commit message subject: builtin/fetch -> fetch --negotiate-only
* move the 'goto cleanup' to _after_ the submodule updating task because
  we may want to update submodules even if objects were not fetched (as
  pointed out by Junio, thanks!)
* disable submodule recursion in the patch that checks for
  --negotiate-only + --recurse-submodules, so we never silently ignore
  --recurse-submodules.
* incorporate some of Jonathan's suggestions (thanks!)

Changes since v2:
* added a prepatory patch that introduces a "goto cleanup"
* drop an unnecessary line move (as suggested by Junio)
* check for user-given --recurse-submodules and die() (as suggested by
  Jonathan and Junio)
* update --negotiate-only's documentation

Changes since v1:
* added more context to commit message
* added a NEEDSWORK comment

Glen Choo (3):
  fetch: use goto cleanup in cmd_fetch()
  fetch: skip tasks related to fetching objects
  fetch --negotiate-only: do not update submodules

 Documentation/fetch-options.txt |  1 +
 builtin/fetch.c                 | 40 ++++++++++++++++++++++++++++++---
 t/t5516-fetch-push.sh           | 12 ++++++++++
 t/t5702-protocol-v2.sh          | 12 ++++++++++
 4 files changed, 62 insertions(+), 3 deletions(-)

Range-diff against v3:
1:  97e2cba633 ! 1:  ffa1a24109 builtin/fetch: use goto cleanup in cmd_fetch()
    @@ Metadata
     Author: Glen Choo <chooglen@google.com>

      ## Commit message ##
    -    builtin/fetch: use goto cleanup in cmd_fetch()
    +    fetch: use goto cleanup in cmd_fetch()

         Replace an early return with 'goto cleanup' in cmd_fetch() so that the
    -    string_list is always cleared.
    -
    -    The string_list_clear() call is purely cleanup; the string_list was not
    -    reused.
    +    string_list is always cleared (the string_list_clear() call is purely
    +    cleanup; the string_list is not reused). This makes cleanup consistent
    +    so that a subsequent commit can use 'goto cleanup' to bail out early.

         Signed-off-by: Glen Choo <chooglen@google.com>

    @@ builtin/fetch.c
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
      			gtransport->smart_options->acked_commits = &acked_commits;
      		} else {
    - 			warning(_("Protocol does not support --negotiate-only, exiting."));
    + 			warning(_("protocol does not support --negotiate-only, exiting"));
     -			return 1;
     +			result = 1;
     +			goto cleanup;
2:  3a3a04b649 < -:  ---------- builtin/fetch: skip unnecessary tasks when using --negotiate-only
-:  ---------- > 2:  b0c73e8135 fetch: skip tasks related to fetching objects
3:  97792b5214 ! 3:  914d30866f builtin/fetch: die on --negotiate-only and --recurse-submodules
    @@ Metadata
     Author: Glen Choo <chooglen@google.com>

      ## Commit message ##
    -    builtin/fetch: die on --negotiate-only and --recurse-submodules
    +    fetch --negotiate-only: do not update submodules

    -    The previous commit ignores the value of --recurse-submodules if
    -    --negotiate-only is given. Since non "no" values of --recurse-submodules
    -    are not supported with --negotiate-only, make cmd_fetch() check for
    -    this invalid combination and die.
    +    `git fetch --negotiate-only` is an implementation detail of push
    +    negotiation and, unlike most `git fetch` invocations, does not actually
    +    update the main repository. Thus it should not update submodules even
    +    if submodule recursion is enabled.

    -    This is unlikely to affect internal usage of --negotiate-only, but it
    -    may be helpful for users. We may also want to discourage users from
    -    using --negotiate-only altogether because it was not intended for them.
    +    This is not just slow, it is wrong e.g. push negotiation with
    +    "submodule.recurse=true" will cause submodules to be updated because it
    +    invokes `git fetch --negotiate-only`.
    +
    +    Fix this by disabling submodule recursion if --negotiate-only was given.
    +    Since this makes --negotiate-only and --recurse-submodules incompatible,
    +    check for this invalid combination and die.
    +
    +    This does not use the "goto cleanup" introduced in the previous commit
    +    because we want to recurse through submodules whenever a ref is fetched,
    +    and this can happen without introducing new objects.

         Signed-off-by: Glen Choo <chooglen@google.com>

    @@ Documentation/fetch-options.txt: configuration variables documented in linkgit:g
      	ancestors of the provided `--negotiation-tip=*` arguments,
      	which we have in common with the server.
      +
    -+This is incompatible with `--recurse-submodules[=yes|on-demand]`.
    ++This is incompatible with `--recurse-submodules=[yes|on-demand]`.
      Internally this is used to implement the `push.negotiate` option, see
      linkgit:git-config[1].

    @@ builtin/fetch.c: static struct transport *gtransport;
      static struct transport *gsecondary;
      static const char *submodule_prefix = "";
      static int recurse_submodules = RECURSE_SUBMODULES_DEFAULT;
    -+static int recurse_submodules_explicit = RECURSE_SUBMODULES_DEFAULT;
    ++static int recurse_submodules_cli = RECURSE_SUBMODULES_DEFAULT;
      static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;
      static int shown_url = 0;
      static struct refspec refmap = REFSPEC_INIT_FETCH;
    @@ builtin/fetch.c: static struct option builtin_fetch_options[] = {
      	OPT_BOOL('P', "prune-tags", &prune_tags,
      		 N_("prune local tags no longer on remote and clobber changed tags")),
     -	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules, N_("on-demand"),
    -+	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_explicit, N_("on-demand"),
    ++	OPT_CALLBACK_F(0, "recurse-submodules", &recurse_submodules_cli, N_("on-demand"),
      		    N_("control recursive fetching of submodules"),
      		    PARSE_OPT_OPTARG, option_fetch_parse_recurse_submodules),
      	OPT_BOOL(0, "dry-run", &dry_run,
     @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
    + 
      	argc = parse_options(argc, argv, prefix,
      			     builtin_fetch_options, builtin_fetch_usage, 0);
    - 
    -+	if (recurse_submodules_explicit != RECURSE_SUBMODULES_DEFAULT)
    -+		recurse_submodules = recurse_submodules_explicit;
     +
    - 	if (negotiate_only) {
    --		/*
    --		 * --negotiate-only should never recurse into
    --		 * submodules, so there is no need to read .gitmodules.
    --		 */
    --		recurse_submodules = RECURSE_SUBMODULES_OFF;
    -+		switch (recurse_submodules_explicit) {
    ++	if (recurse_submodules_cli != RECURSE_SUBMODULES_DEFAULT)
    ++		recurse_submodules = recurse_submodules_cli;
    ++
    ++	if (negotiate_only) {
    ++		switch (recurse_submodules_cli) {
     +		case RECURSE_SUBMODULES_OFF:
     +		case RECURSE_SUBMODULES_DEFAULT: {
     +			/*
     +			 * --negotiate-only should never recurse into
    -+			 * submodules, so there is no need to read .gitmodules.
    ++			 * submodules. Skip it by setting recurse_submodules to
    ++			 * RECURSE_SUBMODULES_OFF.
     +			 */
     +			recurse_submodules = RECURSE_SUBMODULES_OFF;
     +			break;
    @@ builtin/fetch.c: int cmd_fetch(int argc, const char **argv, const char *prefix)
     +		default:
     +			die(_("--negotiate-only and --recurse-submodules cannot be used together"));
     +		}
    - 	}
    - 
    ++	}
    ++
      	if (recurse_submodules != RECURSE_SUBMODULES_OFF) {
    + 		int *sfjc = submodule_fetch_jobs_config == -1
    + 			    ? &submodule_fetch_jobs_config : NULL;
    +
    + ## t/t5516-fetch-push.sh ##
    +@@ t/t5516-fetch-push.sh: test_expect_success 'push with negotiation proceeds anyway even if negotiation f
    + 	test_i18ngrep "push negotiation failed" err
    + '
    + 
    ++test_expect_success 'push with negotiation does not attempt to fetch submodules' '
    ++	mk_empty submodule_upstream &&
    ++	test_commit -C submodule_upstream submodule_commit &&
    ++	git submodule add ./submodule_upstream submodule &&
    ++	mk_empty testrepo &&
    ++	git push testrepo $the_first_commit:refs/remotes/origin/first_commit &&
    ++	test_commit -C testrepo unrelated_commit &&
    ++	git -C testrepo config receive.hideRefs refs/remotes/origin/first_commit &&
    ++	git -c submodule.recurse=true -c protocol.version=2 -c push.negotiate=1 push testrepo refs/heads/main:refs/remotes/origin/main 2>err &&
    ++	! grep "Fetching submodule" err
    ++'
    ++
    + test_expect_success 'push without wildcard' '
    + 	mk_empty testrepo &&
    + 

      ## t/t5702-protocol-v2.sh ##
     @@ t/t5702-protocol-v2.sh: test_expect_success 'usage: --negotiate-only without --negotiation-tip' '
    @@ t/t5702-protocol-v2.sh: test_expect_success 'usage: --negotiate-only without --n
      '

     +test_expect_success 'usage: --negotiate-only with --recurse-submodules' '
    -+	SERVER="server" &&
    -+	URI="file://$(pwd)/server" &&
    -+
    -+	setup_negotiate_only "$SERVER" "$URI" &&
    -+
     +	cat >err.expect <<-\EOF &&
     +	fatal: --negotiate-only and --recurse-submodules cannot be used together
     +	EOF

base-commit: 90d242d36e248acfae0033274b524bfa55a947fd
-- 
2.33.GIT