All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ramsay Jones <ramsay@ramsayjones.plus.com>
To: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>,
	git@vger.kernel.org
Cc: tom.saeger@oracle.com, gitster@pobox.com,
	sunshine@sunshineco.com, Derrick Stolee <stolee@gmail.com>,
	Josh Steadmon <steadmon@google.com>,
	Emily Shaffer <emilyshaffer@google.com>,
	Derrick Stolee <derrickstolee@github.com>,
	Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH v3 2/3] fetch: add --prefetch option
Date: Sun, 11 Apr 2021 22:09:01 +0100	[thread overview]
Message-ID: <9e009c7e-a2bf-1740-ba43-c64a97f1dc28@ramsayjones.plus.com> (raw)
In-Reply-To: <7f488eea6dbdd93ff1a97a2572de7ddb0d65b7e0.1618020225.git.gitgitgadget@gmail.com>



On 10/04/2021 03:03, Derrick Stolee via GitGitGadget wrote:
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> The --prefetch option will be used by the 'prefetch' maintenance task
> instead of sending refspecs explicitly across the command-line. The
> intention is to modify the refspec to place all results in
> refs/prefetch/ instead of anywhere else.
> 
> Create helper method filter_prefetch_refspec() to modify a given refspec
> to fit the rules expected of the prefetch task:
> 
>  * Negative refspecs are preserved.
>  * Refspecs without a destination are removed.
>  * Refspecs whose source starts with "refs/tags/" are removed.
>  * Other refspecs are placed within "refs/prefetch/".
> 
> Finally, we add the 'force' option to ensure that prefetch refs are
> replaced as necessary.
> 
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/fetch-options.txt   |  5 +++
>  builtin/fetch.c                   | 56 +++++++++++++++++++++++++++++++
>  t/t5582-fetch-negative-refspec.sh | 30 +++++++++++++++++
>  3 files changed, 91 insertions(+)
> 
> diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
> index 07783deee309..9e7b4e189ce0 100644
> --- a/Documentation/fetch-options.txt
> +++ b/Documentation/fetch-options.txt
> @@ -110,6 +110,11 @@ ifndef::git-pull[]
>  	setting `fetch.writeCommitGraph`.
>  endif::git-pull[]
>  
> +--prefetch::
> +	Modify the configured refspec to place all refs into the
> +	`refs/prefetch/` namespace. See the `prefetch` task in
> +	linkgit:git-maintenance[1].
> +
>  -p::
>  --prune::
>  	Before fetching, remove any remote-tracking references that no
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 0b90de87c7a2..30856b442b79 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -48,6 +48,7 @@ enum {
>  static int fetch_prune_config = -1; /* unspecified */
>  static int fetch_show_forced_updates = 1;
>  static uint64_t forced_updates_ms = 0;
> +static int prefetch = 0;
>  static int prune = -1; /* unspecified */
>  #define PRUNE_BY_DEFAULT 0 /* do we prune by default? */
>  
> @@ -158,6 +159,8 @@ static struct option builtin_fetch_options[] = {
>  		    N_("do not fetch all tags (--no-tags)"), TAGS_UNSET),
>  	OPT_INTEGER('j', "jobs", &max_jobs,
>  		    N_("number of submodules fetched in parallel")),
> +	OPT_BOOL(0, "prefetch", &prefetch,
> +		 N_("modify the refspec to place all refs within refs/prefetch/")),
>  	OPT_BOOL('p', "prune", &prune,
>  		 N_("prune remote-tracking branches no longer on remote")),
>  	OPT_BOOL('P', "prune-tags", &prune_tags,
> @@ -436,6 +439,55 @@ static void find_non_local_tags(const struct ref *refs,
>  	oidset_clear(&fetch_oids);
>  }
>  
> +static void filter_prefetch_refspec(struct refspec *rs)
> +{
> +	int i;
> +
> +	if (!prefetch)
> +		return;
> +
> +	for (i = 0; i < rs->nr; i++) {
> +		struct strbuf new_dst = STRBUF_INIT;
> +		char *old_dst;
> +		const char *sub = NULL;
> +
> +		if (rs->items[i].negative)
> +			continue;
> +		if (!rs->items[i].dst ||
> +		    (rs->items[i].src &&
> +		     !strncmp(rs->items[i].src, "refs/tags/", 10))) {
> +			int j;
> +
> +			free(rs->items[i].src);
> +			free(rs->items[i].dst);
> +
> +			for (j = i + 1; j < rs->nr; j++) {
> +				rs->items[j - 1] = rs->items[j];
> +				rs->raw[j - 1] = rs->raw[j];
> +			}
> +			rs->nr--;

Hmm, don't you need to do 'i--;' here?

(Sorry in advance if this is nonsense, I am just skimming the
patches without reading the whole series carefully).

Maybe try a test which has an entry, which requires the 'prefetch'
modification, that immediately follows a 'tag' or 'empty dst' entry.
(I can't quite tell, just reading the email, whether that is covered
by the tests below - so please just ignore me if it already works ;)

ATB,
Ramsay Jones


> +			continue;
> +		}
> +
> +		old_dst = rs->items[i].dst;
> +		strbuf_addstr(&new_dst, "refs/prefetch/");
> +
> +		/*
> +		 * If old_dst starts with "refs/", then place
> +		 * sub after that prefix. Otherwise, start at
> +		 * the beginning of the string.
> +		 */
> +		if (!skip_prefix(old_dst, "refs/", &sub))
> +			sub = old_dst;
> +		strbuf_addstr(&new_dst, sub);
> +
> +		rs->items[i].dst = strbuf_detach(&new_dst, NULL);
> +		rs->items[i].force = 1;
> +
> +		free(old_dst);
> +	}
> +}
> +
>  static struct ref *get_ref_map(struct remote *remote,
>  			       const struct ref *remote_refs,
>  			       struct refspec *rs,
> @@ -452,6 +504,10 @@ static struct ref *get_ref_map(struct remote *remote,
>  	struct hashmap existing_refs;
>  	int existing_refs_populated = 0;
>  
> +	filter_prefetch_refspec(rs);
> +	if (remote)
> +		filter_prefetch_refspec(&remote->fetch);
> +
>  	if (rs->nr) {
>  		struct refspec *fetch_refspec;
>  
> diff --git a/t/t5582-fetch-negative-refspec.sh b/t/t5582-fetch-negative-refspec.sh
> index f34509727702..030e6f978c4e 100755
> --- a/t/t5582-fetch-negative-refspec.sh
> +++ b/t/t5582-fetch-negative-refspec.sh
> @@ -240,4 +240,34 @@ test_expect_success "push with matching +: and negative refspec" '
>  	git -C two push -v one
>  '
>  
> +test_expect_success '--prefetch correctly modifies refspecs' '
> +	git -C one config --unset-all remote.origin.fetch &&
> +	git -C one config --add remote.origin.fetch "refs/tags/*:refs/tags/*" &&
> +	git -C one config --add remote.origin.fetch ^refs/heads/bogus/ignore &&
> +	git -C one config --add remote.origin.fetch "refs/heads/bogus/*:bogus/*" &&
> +
> +	git tag -a -m never never-fetch-tag HEAD &&
> +
> +	git branch bogus/fetched HEAD~1 &&
> +	git branch bogus/ignore HEAD &&
> +
> +	git -C one fetch --prefetch --no-tags &&
> +	test_must_fail git -C one rev-parse never-fetch-tag &&
> +	git -C one rev-parse refs/prefetch/bogus/fetched &&
> +	test_must_fail git -C one rev-parse refs/prefetch/bogus/ignore &&
> +
> +	# correctly handle when refspec set becomes empty
> +	# after removing the refs/tags/* refspec.
> +	git -C one config --unset-all remote.origin.fetch &&
> +	git -C one config --add remote.origin.fetch "refs/tags/*:refs/tags/*" &&
> +
> +	git -C one fetch --prefetch --no-tags &&
> +	test_must_fail git -C one rev-parse never-fetch-tag &&
> +
> +	# The refspec for refs that are not fully qualified
> +	# are filtered multiple times.
> +	git -C one rev-parse refs/prefetch/bogus/fetched &&
> +	test_must_fail git -C one rev-parse refs/prefetch/bogus/ignore
> +'
> +
>  test_done
> 

  reply	other threads:[~2021-04-11 21:09 UTC|newest]

Thread overview: 72+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-04-05 13:04 [PATCH 0/5] Maintenance: adapt custom refspecs Derrick Stolee via GitGitGadget
2021-04-05 13:04 ` [PATCH 1/5] maintenance: simplify prefetch logic Derrick Stolee via GitGitGadget
2021-04-05 17:01   ` Tom Saeger
2021-04-05 13:04 ` [PATCH 2/5] test-lib: use exact match for test_subcommand Derrick Stolee via GitGitGadget
2021-04-05 17:31   ` Eric Sunshine
2021-04-05 17:43     ` Junio C Hamano
2021-04-05 13:04 ` [PATCH 3/5] refspec: output a refspec item Derrick Stolee via GitGitGadget
2021-04-05 16:57   ` Tom Saeger
2021-04-05 17:40     ` Eric Sunshine
2021-04-05 17:44       ` Junio C Hamano
2021-04-06 11:21         ` Derrick Stolee
2021-04-06 15:23           ` Eric Sunshine
2021-04-06 16:51             ` Derrick Stolee
2021-04-07  8:46   ` Ævar Arnfjörð Bjarmason
2021-04-07 20:53     ` Derrick Stolee
2021-04-07 22:05       ` Ævar Arnfjörð Bjarmason
2021-04-07 22:49         ` Junio C Hamano
2021-04-07 23:01           ` Ævar Arnfjörð Bjarmason
2021-04-08  7:33             ` Junio C Hamano
2021-04-05 13:04 ` [PATCH 4/5] test-tool: test refspec input/output Derrick Stolee via GitGitGadget
2021-04-05 17:52   ` Eric Sunshine
2021-04-06 11:13     ` Derrick Stolee
2021-04-07  8:54   ` Ævar Arnfjörð Bjarmason
2021-04-05 13:04 ` [PATCH 5/5] maintenance: allow custom refspecs during prefetch Derrick Stolee via GitGitGadget
2021-04-05 17:16   ` Tom Saeger
2021-04-06 11:15     ` Derrick Stolee
2021-04-07  8:53   ` Ævar Arnfjörð Bjarmason
2021-04-07 10:26     ` Ævar Arnfjörð Bjarmason
2021-04-09 11:48       ` Derrick Stolee
2021-04-09 19:28         ` Ævar Arnfjörð Bjarmason
2021-04-10  0:56           ` Derrick Stolee
2021-04-10 11:37             ` Ævar Arnfjörð Bjarmason
2021-04-07 13:47   ` Ævar Arnfjörð Bjarmason
2021-04-06 18:47 ` [PATCH v2 0/5] Maintenance: adapt custom refspecs Derrick Stolee via GitGitGadget
2021-04-06 18:47   ` [PATCH v2 1/5] maintenance: simplify prefetch logic Derrick Stolee via GitGitGadget
2021-04-07 23:23     ` Emily Shaffer
2021-04-09 19:00       ` Derrick Stolee
2021-04-06 18:47   ` [PATCH v2 2/5] test-lib: use exact match for test_subcommand Derrick Stolee via GitGitGadget
2021-04-06 18:47   ` [PATCH v2 3/5] refspec: output a refspec item Derrick Stolee via GitGitGadget
2021-04-06 18:47   ` [PATCH v2 4/5] test-tool: test refspec input/output Derrick Stolee via GitGitGadget
2021-04-07 23:08     ` Josh Steadmon
2021-04-07 23:26     ` Emily Shaffer
2021-04-06 18:47   ` [PATCH v2 5/5] maintenance: allow custom refspecs during prefetch Derrick Stolee via GitGitGadget
2021-04-06 19:36     ` Tom Saeger
2021-04-06 19:45       ` Derrick Stolee
2021-04-07 23:09     ` Josh Steadmon
2021-04-07 23:37     ` Emily Shaffer
2021-04-08  0:23     ` Jonathan Tan
2021-04-10  2:03   ` [PATCH v3 0/3] Maintenance: adapt custom refspecs Derrick Stolee via GitGitGadget
2021-04-10  2:03     ` [PATCH v3 1/3] maintenance: simplify prefetch logic Derrick Stolee via GitGitGadget
2021-04-12 20:13       ` Tom Saeger
2021-04-12 20:27         ` Derrick Stolee
2021-04-10  2:03     ` [PATCH v3 2/3] fetch: add --prefetch option Derrick Stolee via GitGitGadget
2021-04-11 21:09       ` Ramsay Jones [this message]
2021-04-12 20:23         ` Derrick Stolee
2021-04-10  2:03     ` [PATCH v3 3/3] maintenance: use 'git fetch --prefetch' Derrick Stolee via GitGitGadget
2021-04-11  1:35     ` [PATCH v3 0/3] Maintenance: adapt custom refspecs Junio C Hamano
2021-04-12 16:48       ` Tom Saeger
2021-04-12 17:24         ` Tom Saeger
2021-04-12 17:41           ` Tom Saeger
2021-04-12 20:25             ` Derrick Stolee
2021-04-16 12:49     ` [PATCH v4 0/4] " Derrick Stolee via GitGitGadget
2021-04-16 12:49       ` [PATCH v4 1/4] maintenance: simplify prefetch logic Derrick Stolee via GitGitGadget
2021-04-16 18:02         ` Tom Saeger
2021-04-16 12:49       ` [PATCH v4 2/4] fetch: add --prefetch option Derrick Stolee via GitGitGadget
2021-04-16 17:52         ` Tom Saeger
2021-04-16 18:26           ` Tom Saeger
2021-04-16 12:49       ` [PATCH v4 3/4] maintenance: use 'git fetch --prefetch' Derrick Stolee via GitGitGadget
2021-04-16 12:49       ` [PATCH v4 4/4] maintenance: respect remote.*.skipFetchAll Derrick Stolee via GitGitGadget
2021-04-16 13:54         ` Ævar Arnfjörð Bjarmason
2021-04-16 14:33           ` Tom Saeger
2021-04-16 18:31         ` Tom Saeger

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9e009c7e-a2bf-1740-ba43-c64a97f1dc28@ramsayjones.plus.com \
    --to=ramsay@ramsayjones.plus.com \
    --cc=derrickstolee@github.com \
    --cc=dstolee@microsoft.com \
    --cc=emilyshaffer@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitgitgadget@gmail.com \
    --cc=gitster@pobox.com \
    --cc=steadmon@google.com \
    --cc=stolee@gmail.com \
    --cc=sunshine@sunshineco.com \
    --cc=tom.saeger@oracle.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.