[PATCH] clone: pass --progress decision to recursive submodules

[PATCH] clone: pass --progress decision to recursive submodules

[Jeff King]

When cloning with "--recursive", we'd generally expect
submodules to show progress reports if the main clone did,
too.

In older versions of git, this mostly worked out of the
box. Since we show progress by default when stderr is a tty,
and since the child clones inherit the parent stderr, then
both processes would come to the same decision by default.
If the parent clone was asked for "--quiet", we passed down
"--quiet" to the child. However, if stderr was not a tty and
the user specified "--progress", we did not propagate this
to the child.

That's a minor bug, but things got much worse when we
switched recently to submodule--helper's update_clone
command. With that change, the stderr of the child clones
are always connected to a pipe, and we never output
progress at all.

This patch teaches git-submodule and git-submodule--helper
how to pass down an explicit "--progress" flag when cloning.
The clone command then decides to propagate that flag based
on the cloning decision made earlier (which takes into
account isatty(2) of the parent process, existing --progress
or --quiet flags, etc). Since the child processes always run
without a tty on stderr, we don't have to worry about
passing an explicit "--no-progress"; it's the default for
them.

This fixes the recent loss of progress during recursive
clones. And as a bonus, it makes:

  git clone --recursive --progress ... 2>&1 | cat

work by triggering progress explicitly in the children.

Signed-off-by: Jeff King <peff@peff.net>
---
I don't usually use submodules, but I happened to be testing something,
and my goto "this has a lot of submodules" repository happens to have
some pretty large submodules. So I had plenty of time to contemplate the
lack of progress reporting. :)

I checked with --jobs, too, and this should do the right thing; it
queues up the progress reports for submodules that aren't the
output_owner, and then dumps them all at once (so the progress meter
will appear to whiz by for those jobs until it catches up to the current
state, which is the only reasonable thing we can do, I think).

I imagine there are other code paths that want similar treatment, but I
didn't look into them. I'd assume "fetch" is one. I'm not sure if we do
parallel checkouts, but that's another potential.

 builtin/clone.c             | 16 ++++++++++++++--
 builtin/submodule--helper.c | 18 +++++++++++++++---
 git-submodule.sh            |  5 +++++
 3 files changed, 34 insertions(+), 5 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 404c5e8..28ce938 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -670,7 +670,7 @@ static void update_head(const struct ref *our, const struct ref *remote,
 	}
 }
 
-static int checkout(void)
+static int checkout(int submodule_progress)
 {
 	unsigned char sha1[20];
 	char *head;
@@ -734,6 +734,9 @@ static int checkout(void)
 		if (max_jobs != -1)
 			argv_array_pushf(&args, "--jobs=%d", max_jobs);
 
+		if (submodule_progress)
+			argv_array_push(&args, "--progress");
+
 		err = run_command_v_opt(args.argv, RUN_GIT_CMD);
 		argv_array_clear(&args);
 	}
@@ -841,6 +844,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	const char *src_ref_prefix = "refs/heads/";
 	struct remote *remote;
 	int err = 0, complete_refs_before_fetch = 1;
+	int submodule_progress;
 
 	struct refspec *refspec;
 	const char *fetch_pattern;
@@ -1099,6 +1103,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 	update_head(our_head_points_at, remote_head, reflog_msg.buf);
 
+	/*
+	 * We want to show progress for recursive submodule clones iff
+	 * we did so for the main clone. But only the transport knows
+	 * the final decision for this flag, so we need to rescue the value
+	 * before we free the transport.
+	 */
+	submodule_progress = transport->progress;
+
 	transport_unlock_pack(transport);
 	transport_disconnect(transport);
 
@@ -1108,7 +1120,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	}
 
 	junk_mode = JUNK_LEAVE_REPO;
-	err = checkout();
+	err = checkout(submodule_progress);
 
 	strbuf_release(&reflog_msg);
 	strbuf_release(&branch_top);
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 7b8ddfe..d2f9d7d 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -443,7 +443,8 @@ static int module_name(int argc, const char **argv, const char *prefix)
 }
 
 static int clone_submodule(const char *path, const char *gitdir, const char *url,
-			   const char *depth, struct string_list *reference, int quiet)
+			   const char *depth, struct string_list *reference,
+			   int quiet, int progress)
 {
 	struct child_process cp = CHILD_PROCESS_INIT;
 
@@ -451,6 +452,8 @@ static int clone_submodule(const char *path, const char *gitdir, const char *url
 	argv_array_push(&cp.args, "--no-checkout");
 	if (quiet)
 		argv_array_push(&cp.args, "--quiet");
+	if (progress)
+		argv_array_push(&cp.args, "--progress");
 	if (depth && *depth)
 		argv_array_pushl(&cp.args, "--depth", depth, NULL);
 	if (reference->nr) {
@@ -575,6 +578,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)
 {
 	const char *name = NULL, *url = NULL, *depth = NULL;
 	int quiet = 0;
+	int progress = 0;
 	FILE *submodule_dot_git;
 	char *p, *path = NULL, *sm_gitdir;
 	struct strbuf rel_path = STRBUF_INIT;
@@ -601,6 +605,8 @@ static int module_clone(int argc, const char **argv, const char *prefix)
 			   N_("string"),
 			   N_("depth for shallow clones")),
 		OPT__QUIET(&quiet, "Suppress output for cloning a submodule"),
+		OPT_BOOL(0, "progress", &progress,
+			   N_("force cloning progress")),
 		OPT_END()
 	};
 
@@ -634,7 +640,8 @@ static int module_clone(int argc, const char **argv, const char *prefix)
 
 		prepare_possible_alternates(name, &reference);
 
-		if (clone_submodule(path, sm_gitdir, url, depth, &reference, quiet))
+		if (clone_submodule(path, sm_gitdir, url, depth, &reference,
+				    quiet, progress))
 			die(_("clone of '%s' into submodule path '%s' failed"),
 			    url, path);
 	} else {
@@ -684,6 +691,7 @@ struct submodule_update_clone {
 	struct submodule_update_strategy update;
 
 	/* configuration parameters which are passed on to the children */
+	int progress;
 	int quiet;
 	int recommend_shallow;
 	struct string_list references;
@@ -702,7 +710,7 @@ struct submodule_update_clone {
 	int failed_clones_nr, failed_clones_alloc;
 };
 #define SUBMODULE_UPDATE_CLONE_INIT {0, MODULE_LIST_INIT, 0, \
-	SUBMODULE_UPDATE_STRATEGY_INIT, 0, -1, STRING_LIST_INIT_DUP, \
+	SUBMODULE_UPDATE_STRATEGY_INIT, 0, 0, -1, STRING_LIST_INIT_DUP, \
 	NULL, NULL, NULL, \
 	STRING_LIST_INIT_DUP, 0, NULL, 0, 0}
 
@@ -804,6 +812,8 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 	child->err = -1;
 	argv_array_push(&child->args, "submodule--helper");
 	argv_array_push(&child->args, "clone");
+	if (suc->progress)
+		argv_array_push(&child->args, "--progress");
 	if (suc->quiet)
 		argv_array_push(&child->args, "--quiet");
 	if (suc->prefix)
@@ -950,6 +960,8 @@ static int update_clone(int argc, const char **argv, const char *prefix)
 		OPT_BOOL(0, "recommend-shallow", &suc.recommend_shallow,
 			    N_("whether the initial clone should follow the shallow recommendation")),
 		OPT__QUIET(&suc.quiet, N_("don't print cloning progress")),
+		OPT_BOOL(0, "progress", &suc.progress,
+			    N_("force cloning progress")),
 		OPT_END()
 	};
 
diff --git a/git-submodule.sh b/git-submodule.sh
index a1cc71b..a024a13 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -44,6 +44,7 @@ update=
 prefix=
 custom_name=
 depth=
+progress=
 
 die_if_unmatched ()
 {
@@ -498,6 +499,9 @@ cmd_update()
 		-q|--quiet)
 			GIT_QUIET=1
 			;;
+		--progress)
+			progress="--progress"
+			;;
 		-i|--init)
 			init=1
 			;;
@@ -573,6 +577,7 @@ cmd_update()
 
 	{
 	git submodule--helper update-clone ${GIT_QUIET:+--quiet} \
+		${progress:+"$progress"} \
 		${wt_prefix:+--prefix "$wt_prefix"} \
 		${prefix:+--recursive-prefix "$prefix"} \
 		${update:+--update "$update"} \
-- 
2.10.0.482.gae5a597

Re: [PATCH] clone: pass --progress decision to recursive submodules

[Stefan Beller]

On Wed, Sep 21, 2016 at 10:24 PM, Jeff King <peff@peff.net> wrote:
> When cloning with "--recursive", we'd generally expect
> submodules to show progress reports if the main clone did,
> too.
>
> In older versions of git, this mostly worked out of the
> box. Since we show progress by default when stderr is a tty,
> and since the child clones inherit the parent stderr, then
> both processes would come to the same decision by default.
> If the parent clone was asked for "--quiet", we passed down
> "--quiet" to the child. However, if stderr was not a tty and
> the user specified "--progress", we did not propagate this
> to the child.
>
> That's a minor bug, but things got much worse when we
> switched recently to submodule--helper's update_clone
> command. With that change, the stderr of the child clones
> are always connected to a pipe, and we never output
> progress at all.

Right, that is an issue.

>
> Signed-off-by: Jeff King <peff@peff.net>

Acked and thanked by Stefan ;)

>
> I imagine there are other code paths that want similar treatment, but I
> didn't look into them. I'd assume "fetch" is one. I'm not sure if we do
> parallel checkouts, but that's another potential.

Looking for run_processes_parallel in the code,
it seems to only be used in fetch_populated_submodules
and the submodule helper.

>
>         update_head(our_head_points_at, remote_head, reflog_msg.buf);
>
> +       /*
> +        * We want to show progress for recursive submodule clones iff
> +        * we did so for the main clone. But only the transport knows
> +        * the final decision for this flag, so we need to rescue the value
> +        * before we free the transport.
> +        */
> +       submodule_progress = transport->progress;
> +

Good point! I was aware of this bug (but I did not consider it to be impactful
or as you put it "much worse"), but I anticipated we would need some refactoring
of the transport code, e.g. have the decision via isatty(2) as a
separate outside
function that we consult before we even setup the transport and then
pass it down
to the submodules as well. This seems to solve this bug elegantly.

>         transport_unlock_pack(transport);
>         transport_disconnect(transport);
>
> @@ -1108,7 +1120,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>         }
>
>         junk_mode = JUNK_LEAVE_REPO;
> -       err = checkout();
> +       err = checkout(submodule_progress);
>
>         strbuf_release(&reflog_msg);
>         strbuf_release(&branch_top);
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index 7b8ddfe..d2f9d7d 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -443,7 +443,8 @@ static int module_name(int argc, const char **argv, const char *prefix)
>  }
>
>  static int clone_submodule(const char *path, const char *gitdir, const char *url,
> -                          const char *depth, struct string_list *reference, int quiet)
> +                          const char *depth, struct string_list *reference,
> +                          int quiet, int progress)

I am not sure if having both quiet and progress is maintainable well,
but it get's the job done here, specifically if we consider this patch a bug
fix that we'd want to merge down to maint.

Re: [PATCH] clone: pass --progress decision to recursive submodules

[Jeff King]

On Thu, Sep 22, 2016 at 08:36:01AM -0700, Stefan Beller wrote:

> > Signed-off-by: Jeff King <peff@peff.net>
> 
> Acked and thanked by Stefan ;)

Thanks.

> > +       /*
> > +        * We want to show progress for recursive submodule clones iff
> > +        * we did so for the main clone. But only the transport knows
> > +        * the final decision for this flag, so we need to rescue the value
> > +        * before we free the transport.
> > +        */
> > +       submodule_progress = transport->progress;
> > +
> 
> Good point! I was aware of this bug (but I did not consider it to be
> impactful or as you put it "much worse"), but I anticipated we would
> need some refactoring

To be fair, "much worse" is relative; it's _just_ a disabled progress
bar. :)

I do think this counts as a real regression, though. The "do not pass
down explicit --progress" bug was something probably nobody ever cared
about. But losing progress with the default settings is a thing people
might actually notice.

> of the transport code, e.g. have the decision via isatty(2) as a
> separate outside function that we consult before we even setup the
> transport and then pass it down to the submodules as well. This seems
> to solve this bug elegantly.

Yeah, I considered passing a "no really, stderr is a tty" environment
variable down (which would have avoided all of the boilerplate
propagation of --progress through the various helpers). But when I
realized that we do not handle explicit "--progress" either, the
correct solution became more obvious.

And hopefully all that propagation boilerplate will eventually go away
(or at least be simplified) as the submodule code consolidates in C.

> >  static int clone_submodule(const char *path, const char *gitdir, const char *url,
> > -                          const char *depth, struct string_list *reference, int quiet)
> > +                          const char *depth, struct string_list *reference,
> > +                          int quiet, int progress)
> 
> I am not sure if having both quiet and progress is maintainable well,
> but it get's the job done here, specifically if we consider this patch a bug
> fix that we'd want to merge down to maint.

Yeah, I had a similar thought that we might need to combine these, or
possibly that we could even drop "quite". But I tried to err on the side
of making the minimal change, as this isn't really a code path I'm very
familiar with.

-Peff