[PATCH] fetch --prune: exit with error if pruning fails

[PATCH] fetch --prune: exit with error if pruning fails

[Thomas Gummerer]

When pruning refs fails, we currently print an error to stderr, but
still exit 0 from 'git fetch'.  Since this is a genuine error fetch
should be exiting with some non-zero exit code.  Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add
--prune option", 2009-11-10).  Unfortunately it's unclear from that
commit whether ignoring the exit code was an oversight or
intentional, but it feels like an oversight.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/fetch.c  | 12 ++++++++----
 t/t5510-fetch.sh | 10 ++++++++++
 2 files changed, 18 insertions(+), 4 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 5f06b21f8e..54545efedd 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1609,11 +1609,15 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
+		}
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 20f7110ec1..df824cc3d0 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 
-- 
2.31.1

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> When pruning refs fails, we currently print an error to stderr, but
> still exit 0 from 'git fetch'.  Since this is a genuine error fetch
> should be exiting with some non-zero exit code.  Make it so.
>
> The --prune option was introduced in f360d844de ("builtin-fetch: add
> --prune option", 2009-11-10).  Unfortunately it's unclear from that
> commit whether ignoring the exit code was an oversight or
> intentional, but it feels like an oversight.

It is a good idea to signal, with a non-zero status, that the user
needs to inspect the situation and possibly take a corrective
action.  I however do not know if it is sufficient to just give the
same exit status as returned by prune_refs(), which may or may not
happen to crash with other possible non-zero exit status to make it
indistinguishable from other kinds of errors.

>  		if (rs->nr) {
> -			prune_refs(rs, ref_map, transport->url);
> +			retcode = prune_refs(rs, ref_map, transport->url);
>  		} else {
> -			prune_refs(&transport->remote->fetch,
> -				   ref_map,
> -				   transport->url);
> +			retcode = prune_refs(&transport->remote->fetch,
> +					     ref_map,
> +					     transport->url);
> +		}

It seems that it is possible for do_fetch() to return a negative
value, even without this patch.  The return value from prune_refs()
comes from refs.c::delete_refs(), which can come from error_errno()
that is -1, so this patch adds more of the same problem to existing
ones, though.

And the value propagates up to cmd_fetch() via fetch_one().  This
will be relayed to exit() without changing via this callchain:

    git.c::cmd_main()
      git.c::run_argv()
        git.c::handle_builtin()
          git.c::run_builtin()

This is not a new problem, which does not want to be corrected as
part of this patch, but let's leave a mental note as #leftoverbits
for now.

> +		if (retcode) {
> +			free_refs(ref_map);
> +			goto cleanup;
>  		}

This part is iffy.  We tried to prune refs, we may have removed some
of the refs missing from the other side but we may still have some
other refs that are missing from the other side due to the failure
we noticed.

Is it sensible to abort the fetching?  I am undecided, but without
further input, my gut reaction is that it is safe and may even be
better to treat this as a soft error and try to go closer to where
the user wanted to go as much as possible by continuing to fetch
from the other side, given that we have already paid for the cost of
discovering the refs from the other side.

> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
> index 20f7110ec1..df824cc3d0 100755
> --- a/t/t5510-fetch.sh
> +++ b/t/t5510-fetch.sh
> @@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
>  	git rev-parse sometag
>  '
>  
> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
> +	cd "$D" &&
> +	git clone . prune-fail &&
> +	cd prune-fail &&
> +	git update-ref refs/remotes/origin/extrabranch main &&
> +	>.git/packed-refs.new &&
> +	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion
of refs but does not block creation and updates to existing refs
that it is an effective test for the "--prune" issue?  If we somehow
"locked" the whole ref updates, then the fetch would fail even
without "--prune", so it may be "effective", but smells like knowing
too much implementation detail.  Yuck, but I do not offhand think of
any better way (it is easier to think of worse ways), so without
further input, I would say that this is the best (or "least bad") we
can do.

Another tangent #leftoverbits is that many tests in this script are
so old-style that chdir around without isolating themselves in
subshells, while some do.  We may want to clean them up when the
file is quiescent.

Thanks.

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Johannes Schindelin]

Hi Junio,

On Thu, 27 Jan 2022, Junio C Hamano wrote:

> Thomas Gummerer <t.gummerer@gmail.com> writes:
>
> > +		if (retcode) {
> > +			free_refs(ref_map);
> > +			goto cleanup;
> >  		}
>
> This part is iffy.  We tried to prune refs, we may have removed some
> of the refs missing from the other side but we may still have some
> other refs that are missing from the other side due to the failure
> we noticed.
>
> Is it sensible to abort the fetching?  I am undecided, but without
> further input, my gut reaction is that it is safe and may even be
> better to treat this as a soft error and try to go closer to where
> the user wanted to go as much as possible by continuing to fetch
> from the other side, given that we have already paid for the cost of
> discovering the refs from the other side.

I am not so sure. When pruning failed, there may very well be directories
or files in the way of fetching the refs as desired. And it might be even
worse if pruning failed _without_ the fetch failing afterwards: the user
specifically asked for stale refs to be cleaned up, the command succeeded,
but did not do what the user asked for.

Maybe Thomas has an even stronger argument in favor of erroring out. In
any case, I don't think that `--prune` should be a "best effort, otherwise
just shrug" option. If we wanted that, we could introduce
`--prune-best-effort` or some such...

Ciao,
Dscho

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Thomas Gummerer]

Johannes Schindelin writes:

> Hi Junio,
>
> On Thu, 27 Jan 2022, Junio C Hamano wrote:
>
>> Thomas Gummerer <t.gummerer@gmail.com> writes:
>>
>> > +		if (retcode) {
>> > +			free_refs(ref_map);
>> > +			goto cleanup;
>> >  		}
>>
>> This part is iffy.  We tried to prune refs, we may have removed some
>> of the refs missing from the other side but we may still have some
>> other refs that are missing from the other side due to the failure
>> we noticed.
>>
>> Is it sensible to abort the fetching?  I am undecided, but without
>> further input, my gut reaction is that it is safe and may even be
>> better to treat this as a soft error and try to go closer to where
>> the user wanted to go as much as possible by continuing to fetch
>> from the other side, given that we have already paid for the cost of
>> discovering the refs from the other side.

> I am not so sure. When pruning failed, there may very well be directories
> or files in the way of fetching the refs as desired. And it might be even
> worse if pruning failed _without_ the fetch failing afterwards: the user
> specifically asked for stale refs to be cleaned up, the command succeeded,
> but did not do what the user asked for.

I was thinking along similar lines here.  I was going back and forth
between letting the fetch continue, and then exiting with a non-zero
exit code, and just erroring out directly.  I ended up with the latter
because it felt like the right thing to do for the user.  The command
failed, so I'd think it's more confusing that it does more work after
that (even if we already did part of that work).

But I don't care too much one way or another, as long as we end up with
a non-zero exit code when the fetch fails.

> Maybe Thomas has an even stronger argument in favor of erroring out. In
> any case, I don't think that `--prune` should be a "best effort, otherwise
> just shrug" option. If we wanted that, we could introduce
> `--prune-best-effort` or some such...

I don't have a strong argument on whether we should error out
immediately after failing to prune the refs, or just erroring out after
doing the rest of the work, other than exiting early feels right to me.

From the comments upthread it seems to me that the argument is more
whether we should continue after failing to prune, and exit with a
non-zero exit code, or if we should just fail early.

Though I'm happy to introduce a '--prune-best-effort' option or
something like that if we still want to preserve that option for users.
I don't really see people wanting to use that though, and it should be
very rare that '--prune' fails, but the rest of the fetch succeeds.

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Johannes Schindelin]

Hi Thomas & Junio,

On Fri, 28 Jan 2022, Thomas Gummerer wrote:

> Johannes Schindelin writes:
>
> > On Thu, 27 Jan 2022, Junio C Hamano wrote:
> >
> >> Thomas Gummerer <t.gummerer@gmail.com> writes:
> >>
> >> > +		if (retcode) {
> >> > +			free_refs(ref_map);
> >> > +			goto cleanup;
> >> >  		}
> >>
> >> This part is iffy.  We tried to prune refs, we may have removed some
> >> of the refs missing from the other side but we may still have some
> >> other refs that are missing from the other side due to the failure
> >> we noticed.
> >>
> >> Is it sensible to abort the fetching?  I am undecided, but without
> >> further input, my gut reaction is that it is safe and may even be
> >> better to treat this as a soft error and try to go closer to where
> >> the user wanted to go as much as possible by continuing to fetch
> >> from the other side, given that we have already paid for the cost of
> >> discovering the refs from the other side.
>
> > I am not so sure. When pruning failed, there may very well be directories
> > or files in the way of fetching the refs as desired. And it might be even
> > worse if pruning failed _without_ the fetch failing afterwards: the user
> > specifically asked for stale refs to be cleaned up, the command succeeded,
> > but did not do what the user asked for.
>
> I was thinking along similar lines here.  I was going back and forth
> between letting the fetch continue, and then exiting with a non-zero
> exit code, and just erroring out directly.

Oh, I think I misunderstood Junio. As long as the failed prune will cause
a non-zero exit code, I am fine with continuing to try to fetch.

Ciao,
Dscho

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> I am not so sure. When pruning failed, there may very well be directories
> or files in the way of fetching the refs as desired. And it might be even
> worse if pruning failed _without_ the fetch failing afterwards: the user
> specifically asked for stale refs to be cleaned up, the command succeeded,
> but did not do what the user asked for.
>
> Maybe Thomas has an even stronger argument in favor of erroring out. In
> any case, I don't think that `--prune` should be a "best effort, otherwise
> just shrug" option. If we wanted that, we could introduce
> `--prune-best-effort` or some such...

I am not opposed to reporting an error by exiting with non-zero exit
code.  I never said it should be best effort, and doing the "fetch" part
after a failed prune does not make it best effort.

What I am questioning is if it makes sense to stop the fetching
part.  When we fetch to update multiple refs, we do not stop at the
first ref-update failure, but try to do as much as possible and then
report an error, no?  It is the same thing.

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Thomas Gummerer]

Junio C Hamano writes:

> Thomas Gummerer <t.gummerer@gmail.com> writes:
>
>> When pruning refs fails, we currently print an error to stderr, but
>> still exit 0 from 'git fetch'.  Since this is a genuine error fetch
>> should be exiting with some non-zero exit code.  Make it so.
>>
>> The --prune option was introduced in f360d844de ("builtin-fetch: add
>> --prune option", 2009-11-10).  Unfortunately it's unclear from that
>> commit whether ignoring the exit code was an oversight or
>> intentional, but it feels like an oversight.
>
> It is a good idea to signal, with a non-zero status, that the user
> needs to inspect the situation and possibly take a corrective
> action.  I however do not know if it is sufficient to just give the
> same exit status as returned by prune_refs(), which may or may not
> happen to crash with other possible non-zero exit status to make it
> indistinguishable from other kinds of errors.

Not sure I understand this correctly.  Are you saying that we should
take the return value from prune refs, and always munge that to the same
exit code for 'git fetch --prune' if it fails?  E.g. so prune_refs()
could return 1 or 2 or 3 or whatever, but we would always want the exit
code for 'fetch' to be 1?

Happy to implement that.

>> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
>> index 20f7110ec1..df824cc3d0 100755
>> --- a/t/t5510-fetch.sh
>> +++ b/t/t5510-fetch.sh
>> @@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
>>  	git rev-parse sometag
>>  '
>>
>> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
>> +	cd "$D" &&
>> +	git clone . prune-fail &&
>> +	cd prune-fail &&
>> +	git update-ref refs/remotes/origin/extrabranch main &&
>> +	>.git/packed-refs.new &&
>> +	test_must_fail git fetch --prune origin
>
> Is it because the lockfile ".new" on packed-refs prevents deletion
> of refs but does not block creation and updates to existing refs
> that it is an effective test for the "--prune" issue?  If we somehow
> "locked" the whole ref updates, then the fetch would fail even
> without "--prune", so it may be "effective", but smells like knowing
> too much implementation detail.  Yuck, but I do not offhand think of
> any better way (it is easier to think of worse ways), so without
> further input, I would say that this is the best (or "least bad") we
> can do.

Yes, that's correct.  New refs will be created as loose refs, so they
don't care about packed-refs.  However deletions can potentially be
happening in packed-refs, and that's why it fails when 'packed-refs.new'
exists.

I don't love the test either, but I also can't think of a better way to
do this.

> Another tangent #leftoverbits is that many tests in this script are
> so old-style that chdir around without isolating themselves in
> subshells, while some do.  We may want to clean them up when the
> file is quiescent.
>
> Thanks.

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Johannes Schindelin]

Hi Thomas,

On Fri, 28 Jan 2022, Thomas Gummerer wrote:

> Junio C Hamano writes:
>
> > Thomas Gummerer <t.gummerer@gmail.com> writes:
> >
> >> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
> >> +	cd "$D" &&
> >> +	git clone . prune-fail &&
> >> +	cd prune-fail &&
> >> +	git update-ref refs/remotes/origin/extrabranch main &&
> >> +	>.git/packed-refs.new &&
> >> +	test_must_fail git fetch --prune origin
> >
> > Is it because the lockfile ".new" on packed-refs prevents deletion
> > of refs but does not block creation and updates to existing refs
> > that it is an effective test for the "--prune" issue?  If we somehow
> > "locked" the whole ref updates, then the fetch would fail even
> > without "--prune", so it may be "effective", but smells like knowing
> > too much implementation detail.  Yuck, but I do not offhand think of
> > any better way (it is easier to think of worse ways), so without
> > further input, I would say that this is the best (or "least bad") we
> > can do.
>
> Yes, that's correct.  New refs will be created as loose refs, so they
> don't care about packed-refs.  However deletions can potentially be
> happening in packed-refs, and that's why it fails when 'packed-refs.new'
> exists.
>
> I don't love the test either, but I also can't think of a better way to
> do this.

Maybe add a code comment about it? Something like:

	[...]
	: this will prevent --prune from locking packed-refs &&
	>.git/packed-refs.new &&
	[...]

Ciao,
Dscho

Re: [PATCH] fetch --prune: exit with error if pruning fails

[Thomas Gummerer]

Johannes Schindelin writes:

> Hi Thomas,
>
> On Fri, 28 Jan 2022, Thomas Gummerer wrote:
>
>> Junio C Hamano writes:
>>
>> > Thomas Gummerer <t.gummerer@gmail.com> writes:
>> >
>> >> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
>> >> +	cd "$D" &&
>> >> +	git clone . prune-fail &&
>> >> +	cd prune-fail &&
>> >> +	git update-ref refs/remotes/origin/extrabranch main &&
>> >> +	>.git/packed-refs.new &&
>> >> +	test_must_fail git fetch --prune origin
>> >
>> > Is it because the lockfile ".new" on packed-refs prevents deletion
>> > of refs but does not block creation and updates to existing refs
>> > that it is an effective test for the "--prune" issue?  If we somehow
>> > "locked" the whole ref updates, then the fetch would fail even
>> > without "--prune", so it may be "effective", but smells like knowing
>> > too much implementation detail.  Yuck, but I do not offhand think of
>> > any better way (it is easier to think of worse ways), so without
>> > further input, I would say that this is the best (or "least bad") we
>> > can do.
>>
>> Yes, that's correct.  New refs will be created as loose refs, so they
>> don't care about packed-refs.  However deletions can potentially be
>> happening in packed-refs, and that's why it fails when 'packed-refs.new'
>> exists.
>>
>> I don't love the test either, but I also can't think of a better way to
>> do this.
>
> Maybe add a code comment about it? Something like:
>
> 	[...]
> 	: this will prevent --prune from locking packed-refs &&
> 	>.git/packed-refs.new &&
> 	[...]

Yeah, I think a comment here is a good idea, thanks!

[PATCH v2] fetch --prune: exit with error if pruning fails

[Thomas Gummerer]

When pruning refs fails, we currently print an error to stderr, but
still exit 0 from 'git fetch'.  Since this is a genuine error fetch
should be exiting with some non-zero exit code.  Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add
--prune option", 2009-11-10).  Unfortunately it's unclear from that
commit whether ignoring the exit code was an oversight or
intentional, but it feels like an oversight.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---

changes in v2:
- retcode will now always be 1 if we get an error from prune_refs,
  no matter what the original error code was.
- we will now continue the fetch, and only exit with a non-zero
  exit code after having finished the rest of the fetch operation
- added a comment in the test explaining why creating a packed-refs.new
  file will make the fetch --prune fail.

 builtin/fetch.c  | 10 ++++++----
 t/t5510-fetch.sh | 11 +++++++++++
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 5f06b21f8e..648f0694f2 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
 		}
+		if (retcode != 0)
+			retcode = 1;
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
 		free_refs(ref_map);
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 20f7110ec1..ef0da0a63b 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -164,6 +164,17 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	: this will prevent --prune from locking packed-refs for deleting refs, but adding loose refs still succeeds  &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 
-- 
2.31.1

Re: [PATCH v2] fetch --prune: exit with error if pruning fails

[Junio C Hamano]

Thomas Gummerer <t.gummerer@gmail.com> writes:

> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 5f06b21f8e..648f0694f2 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
>  		 * don't care whether --tags was specified.
>  		 */
>  		if (rs->nr) {
> -			prune_refs(rs, ref_map, transport->url);
> +			retcode = prune_refs(rs, ref_map, transport->url);
>  		} else {
> -			prune_refs(&transport->remote->fetch,
> -				   ref_map,
> -				   transport->url);
> +			retcode = prune_refs(&transport->remote->fetch,
> +					     ref_map,
> +					     transport->url);
>  		}
> +		if (retcode != 0)
> +			retcode = 1;
>  	}

Looks trivially correct, even though a few style things look a bit
irritating to my eyes ;-).

Thanks, will queue.