Re: [PATCH 2/9] maintenance: add prefetch task

From: Emily Shaffer <emilyshaffer@google.com>
To: Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Cc: git@vger.kernel.org, sandals@crustytoothpaste.net,
	steadmon@google.com, jrnieder@gmail.com, peff@peff.net,
	congdanhqx@gmail.com, phillip.wood123@gmail.com,
	sluongng@gmail.com, jonathantanmy@google.com,
	Derrick Stolee <derrickstolee@github.com>,
	Derrick Stolee <dstolee@microsoft.com>
Subject: Re: [PATCH 2/9] maintenance: add prefetch task
Date: Wed, 12 Aug 2020 16:10:36 -0700	[thread overview]
Message-ID: <20200812231036.GH2965447@google.com> (raw)
In-Reply-To: <85118ed5f19468d5051dd1579e35cae3c3114d24.1596731425.git.gitgitgadget@gmail.com>

On Thu, Aug 06, 2020 at 04:30:17PM +0000, Derrick Stolee via GitGitGadget wrote:
> 
> 3. By adding a new refspec "+refs/heads/*:refs/prefetch/<remote>/*"
>    we can ensure that we actually load the new values somewhere in
>    our refspace while not updating refs/heads or refs/remotes. By
>    storing these refs here, the commit-graph job will update the
>    commit-graph with the commits from these hidden refs.

[emily] How does the content of refs/prefetch/* get used?
[jrnieder] How does the content of refs/prefetch/* get cleaned up?
[jonathantan] refs/prefetch/* will get used in negotiation so it is
useful to keep these.
> 
> 4. --prune will delete the refs/prefetch/<remote> refs that no
>    longer appear on the remote.
[jrnieder] this is what cleans up refs/prefetch/* later :)

> +static int fetch_remote(const char *remote, struct maintenance_opts *opts)
> +{
> +	struct child_process child = CHILD_PROCESS_INIT;
> +
> +	child.git_cmd = 1;
> +	strvec_pushl(&child.args, "fetch", remote, "--prune", "--no-tags",
> +		     "--no-write-fetch-head", "--refmap=", NULL);
[jonathantan] It would be good to pass --recurse-submodules=no.
[jrnieder] Since we are specifying the refmap here, if we were to
recurse into submodules, we'd have to be careful about making sure
refmap gets propagated correctly.

> +static int fill_each_remote(struct remote *remote, void *cbdata)
[jrnieder] Since this is a callback that happens for each remote, maybe
this should be named to indicate it only fills one remote at a time
instead. Nit :)
> +{
> +	struct string_list *remotes = (struct string_list *)cbdata;
> +
> +	string_list_append(remotes, remote->name);
> +	return 0;
> +}
> +
> +static int maintenance_task_prefetch(struct maintenance_opts *opts)
> +{
> +	int result = 0;
> +	struct string_list_item *item;
> +	struct string_list remotes = STRING_LIST_INIT_DUP;
> +
> +	if (for_each_remote(fill_each_remote, &remotes)) {
> +		error(_("failed to fill remotes"));
> +		result = 1;
> +		goto cleanup;
> +	}
> +
> +	for (item = remotes.items;
> +	     item && item < remotes.items + remotes.nr;
> +	     item++)

Is there a reason not to use for_each_string_list_item() instead? This
would be more brief and also easier to read (less thinking about what
the loop is doing).

> +		result |= fetch_remote(item->string, opts);
> +
> +cleanup:
> +	string_list_clear(&remotes, 0);
> +	return result;
> +}
> +

>  enum maintenance_task_label {
> +	TASK_PREFETCH,
[jrnieder] Nit: Is there a sort order for these? Should we establish an order
early on (e.g. alphabetical)?
>  	TASK_GC,
>  	TASK_COMMIT_GRAPH,

> +test_expect_success 'prefetch multiple remotes' '
> +	git clone . clone1 &&
> +	git clone . clone2 &&
> +	git remote add remote1 "file://$(pwd)/clone1" &&
> +	git remote add remote2 "file://$(pwd)/clone2" &&
> +	git -C clone1 switch -c one &&
> +	git -C clone2 switch -c two &&
> +	test_commit -C clone1 one &&
> +	test_commit -C clone2 two &&
> +	GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch 2>/dev/null &&
> +	fetchargs="--prune --no-tags --no-write-fetch-head --refmap= --quiet" &&
> +	test_subcommand git fetch remote1 $fetchargs +refs/heads/\\*:refs/prefetch/remote1/\\* <run-prefetch.txt &&
[jrnieder] In practice, why were all the \\ needed? Trying to figure out
where Git is using a shell that would need the * escaped and finding it
hard to reason about.
> +	test_subcommand git fetch remote2 $fetchargs +refs/heads/\\*:refs/prefetch/remote2/\\* <run-prefetch.txt &&
> +	test_path_is_missing .git/refs/remotes &&
> +	test_cmp clone1/.git/refs/heads/one .git/refs/prefetch/remote1/one &&
> +	test_cmp clone2/.git/refs/heads/two .git/refs/prefetch/remote2/two &&
[jrnieder] Should we use test_cmp_rev instead to make compatible with
packed-refs?
> +	git log prefetch/remote1/one &&
> +	git log prefetch/remote2/two
[jonathantan] Why do we use 'git log' to check? I'm a little confused
about what's going on; if you just want to check that the refs are
present you could use 'git rev-parse' instead?