All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/3] checkout: fix two bugs on count of updated entries
@ 2022-07-13  4:19 Matheus Tavares
  2022-07-13  4:19 ` [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
                   ` (3 more replies)
  0 siblings, 4 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-13  4:19 UTC (permalink / raw)
  To: git

This fixes two issues at the "Updated %d path from the index" report at
the end of a `git checkout <paths>` operation:

  - Delayed checkout entries being counted twice.
  - Failed entries being included in the count.

The first two patches add tests and the third implements the fix. I came
across this while working at parallel checkout, but only managed to get
back to it now.

Matheus Tavares (3):
  checkout: document bug where delayed checkout counts entries twice
  checkout: show bug about failed entries being included in final report
  checkout: fix two bugs on the final count of updated entries

 builtin/checkout.c                  |  2 +-
 convert.h                           |  6 +++-
 entry.c                             | 34 ++++++++++++--------
 entry.h                             |  3 +-
 parallel-checkout.c                 | 10 ++++--
 parallel-checkout.h                 |  4 ++-
 t/lib-parallel-checkout.sh          |  6 +++-
 t/t0021-conversion.sh               | 22 +++++++++++++
 t/t2080-parallel-checkout-basics.sh | 50 +++++++++++++++++++++++++++++
 unpack-trees.c                      |  2 +-
 10 files changed, 115 insertions(+), 24 deletions(-)

-- 
2.37.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice
  2022-07-13  4:19 [PATCH 0/3] checkout: fix two bugs on count of updated entries Matheus Tavares
@ 2022-07-13  4:19 ` Matheus Tavares
  2022-07-13 17:57   ` Junio C Hamano
  2022-07-13  4:19 ` [PATCH 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 11+ messages in thread
From: Matheus Tavares @ 2022-07-13  4:19 UTC (permalink / raw)
  To: git

At the end of a `git checkout <pathspec>` operation, git reports how
many paths were checked out with a message like "Updated N paths from
the index". However, entries that end up on the delayed checkout queue
(as requested by a long-running process filter) get counted twice,
producing a wrong number in the final report. We will fix this bug in an
upcoming commit. For now, only document/demonstrate it with a
test_expect_failure.

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 t/t0021-conversion.sh | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index bad37abad2..00df9b5c18 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -1132,4 +1132,26 @@ do
 	'
 done
 
+test_expect_failure PERL 'delayed checkout correctly reports the number of updated entries' '
+	rm -rf repo &&
+	git init repo &&
+	(
+		cd repo &&
+		git config filter.delay.process "../rot13-filter.pl delayed.log clean smudge delay" &&
+		git config filter.delay.required true &&
+
+		echo "*.a filter=delay" >.gitattributes &&
+		echo a >test-delay10.a &&
+		echo a >test-delay11.a &&
+		git add . &&
+		git commit -m files &&
+
+		rm *.a &&
+		git checkout . 2>err &&
+		grep "IN: smudge test-delay10.a .* \\[DELAYED\\]" delayed.log &&
+		grep "IN: smudge test-delay11.a .* \\[DELAYED\\]" delayed.log &&
+		grep "Updated 2 paths from the index" err
+	)
+'
+
 test_done
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH 2/3] checkout: show bug about failed entries being included in final report
  2022-07-13  4:19 [PATCH 0/3] checkout: fix two bugs on count of updated entries Matheus Tavares
  2022-07-13  4:19 ` [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
@ 2022-07-13  4:19 ` Matheus Tavares
  2022-07-13 11:14   ` Ævar Arnfjörð Bjarmason
  2022-07-13  4:19 ` [PATCH 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares
  2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
  3 siblings, 1 reply; 11+ messages in thread
From: Matheus Tavares @ 2022-07-13  4:19 UTC (permalink / raw)
  To: git

After checkout, git usually reports how many entries were updated at
that operation. However, because we count the entries too soon during
the checkout process, we may actually include entries that do not get
properly checked out in the end. This can lead to an inaccurate final
report if the user expects it to show only the *successful* updates.
This will be fixed in the next commit, but for now let's document it
with a test that cover all checkout modes.

Note that `test_checkout_workers` have to be slightly adjusted in order
to use the construct `test_checkout_workers ...  test_must_fail git
checkout`. The function runs the command given to it with an assignment
prefix to set the GIT_TRACE2 variable. However, this this assignment has
an undefined behavior when the command is a shell function (like
`test_must_fail`). As POSIX specifies:

  If the command name is a function that is not a standard utility
  implemented as a function, variable assignments shall affect the
  current execution environment during the execution of the function. It
  is unspecified:

    - Whether or not the variable assignments persist after the
      completion of the function

    - Whether or not the variables gain the export attribute during the
      execution of the function

Thus, in order to make sure the GIT_TRACE2 value gets visible to the git
command executed by `test_must_fail`, export the variable and run git in
a subshell.

[1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
     (Section 2.9.1: Simple Commands)

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 t/lib-parallel-checkout.sh          |  6 +++-
 t/t2080-parallel-checkout-basics.sh | 50 +++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh
index 83b279a846..acaee9cbb6 100644
--- a/t/lib-parallel-checkout.sh
+++ b/t/lib-parallel-checkout.sh
@@ -25,7 +25,11 @@ test_checkout_workers () {
 
 	local trace_file=trace-test-checkout-workers &&
 	rm -f "$trace_file" &&
-	GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 &&
+	(
+		GIT_TRACE2="$(pwd)/$trace_file" &&
+		export GIT_TRACE2 &&
+		"$@" 2>&8
+	) &&
 
 	local workers="$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l)" &&
 	test $workers -eq $expected_workers &&
diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
index 3e0f8c675f..6fd7e4c4b2 100755
--- a/t/t2080-parallel-checkout-basics.sh
+++ b/t/t2080-parallel-checkout-basics.sh
@@ -226,4 +226,54 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d
 	)
 '
 
+# This test is here (and not in e.g. t2022-checkout-paths.sh), because we
+# check the final report including sequential, parallel, and delayed entries
+# all at the same time. So we must have finer control of the parallel checkout
+# variables.
+test_expect_failure PERL '"git checkout ." report should not include failed entries' '
+	write_script rot13-filter.pl "$PERL_PATH" \
+		<"$TEST_DIRECTORY"/t0021/rot13-filter.pl &&
+
+	test_config_global filter.delay.process \
+		"\"$(pwd)/rot13-filter.pl\" --always-delay delayed.log clean smudge delay" &&
+	test_config_global filter.delay.required true &&
+	test_config_global filter.cat.clean cat  &&
+	test_config_global filter.cat.smudge cat  &&
+	test_config_global filter.cat.required true  &&
+
+	set_checkout_config 2 0 &&
+	git init failed_entries &&
+	(
+		cd failed_entries &&
+		cat >.gitattributes <<-EOF &&
+		*delay*              filter=delay
+		parallel-ineligible* filter=cat
+		EOF
+		echo a >missing-delay.a &&
+		echo a >parallel-ineligible.a &&
+		echo a >parallel-eligible.a &&
+		echo b >success-delay.b &&
+		echo b >parallel-ineligible.b &&
+		echo b >parallel-eligible.b &&
+		git add -A &&
+		git commit -m files &&
+
+		a_blob="$(git rev-parse :parallel-ineligible.a)" &&
+		dir="$(echo "$a_blob" | cut -c 1-2)" &&
+		file="$(echo "$a_blob" | cut -c 3-)" &&
+		rm ".git/objects/$dir/$file" &&
+		rm *.a *.b &&
+
+		test_checkout_workers 2 test_must_fail git checkout . 2>err &&
+
+		# All *.b entries should succeed and all *.a entries should fail:
+		#  - missing-delay.a: the delay filter will drop this path
+		#  - parallel-*.a: the blob will be missing
+		#
+		grep "Updated 3 paths from the index" err &&
+		test "$(ls *.b | wc -l)" -eq 3 &&
+		test "$(ls *.a | wc -l)" -eq 0
+	)
+'
+
 test_done
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH 3/3] checkout: fix two bugs on the final count of updated entries
  2022-07-13  4:19 [PATCH 0/3] checkout: fix two bugs on count of updated entries Matheus Tavares
  2022-07-13  4:19 ` [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
  2022-07-13  4:19 ` [PATCH 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
@ 2022-07-13  4:19 ` Matheus Tavares
  2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
  3 siblings, 0 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-13  4:19 UTC (permalink / raw)
  To: git

At the end of `git checkout <pathspec>`, we get a message informing how
many entries were updated in the working tree. However, this number can
be inaccurate for two reasons:

1) Delayed entries currently get counted twice.
2) Failed entries are included in the count.

The first problem happens because the counter is first incremented
before inserting the entry in the delayed checkout queue, and once again
when finish_delayed_checkout() calls checkout_entry(). And the second
happens because the counter is incremented too early in
checkout_entry(), before the entry was in fact checked out. Fix that by
moving the count increment further down in the call stack and removing
the duplicate increment on delayed entries. Note that we have to keep
a per-entry reference for the counter (both on parallel checkout and
delayed checkout) because not all entries are always accumulated at the
same counter. See checkout_worktree(), at builtin/checkout.c for an
example.

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 builtin/checkout.c                  |  2 +-
 convert.h                           |  6 ++++-
 entry.c                             | 34 +++++++++++++++++------------
 entry.h                             |  3 +--
 parallel-checkout.c                 | 10 ++++++---
 parallel-checkout.h                 |  4 +++-
 t/t0021-conversion.sh               |  2 +-
 t/t2080-parallel-checkout-basics.sh |  2 +-
 unpack-trees.c                      |  2 +-
 9 files changed, 40 insertions(+), 25 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 2eefda81d8..df3f1663d7 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -417,7 +417,7 @@ static int checkout_worktree(const struct checkout_opts *opts,
 	mem_pool_discard(&ce_mem_pool, should_validate_cache_entries());
 	remove_marked_cache_entries(&the_index, 1);
 	remove_scheduled_dirs();
-	errs |= finish_delayed_checkout(&state, &nr_checkouts, opts->show_progress);
+	errs |= finish_delayed_checkout(&state, opts->show_progress);
 
 	if (opts->count_checkout_paths) {
 		if (nr_unmerged)
diff --git a/convert.h b/convert.h
index 5ee1c32205..0a6e4086b8 100644
--- a/convert.h
+++ b/convert.h
@@ -53,7 +53,11 @@ struct delayed_checkout {
 	enum ce_delay_state state;
 	/* List of filter drivers that signaled delayed blobs. */
 	struct string_list filters;
-	/* List of delayed blobs identified by their path. */
+	/*
+	 * List of delayed blobs identified by their path. The `util` member
+	 * holds a counter pointer which must be incremented when/if the
+	 * associated blob gets checked out.
+	 */
 	struct string_list paths;
 };
 
diff --git a/entry.c b/entry.c
index 1c9df62b30..616e4f073c 100644
--- a/entry.c
+++ b/entry.c
@@ -157,12 +157,11 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)
 
 	available = string_list_lookup(available_paths, item->string);
 	if (available)
-		available->util = (void *)item->string;
+		available->util = item->util;
 	return !available;
 }
 
-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
-			    int show_progress)
+int finish_delayed_checkout(struct checkout *state, int show_progress)
 {
 	int errs = 0;
 	unsigned processed_paths = 0;
@@ -227,7 +226,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
 						       strlen(path->string), 0);
 				if (ce) {
 					display_progress(progress, ++processed_paths);
-					errs |= checkout_entry(ce, state, NULL, nr_checkouts);
+					errs |= checkout_entry(ce, state, NULL, path->util);
 					filtered_bytes += ce->ce_stat_data.sd_size;
 					display_throughput(progress, filtered_bytes);
 				} else
@@ -266,7 +265,8 @@ void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,
 
 /* Note: ca is used (and required) iff the entry refers to a regular file. */
 static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca,
-		       const struct checkout *state, int to_tempfile)
+		       const struct checkout *state, int to_tempfile,
+		       int *nr_checkouts)
 {
 	unsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;
 	struct delayed_checkout *dco = state->delayed_checkout;
@@ -279,6 +279,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 	struct stat st;
 	const struct submodule *sub;
 	struct checkout_metadata meta;
+	static int scratch_nr_checkouts;
 
 	clone_checkout_metadata(&meta, &state->meta, &ce->oid);
 
@@ -333,9 +334,15 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 			ret = async_convert_to_working_tree_ca(ca, ce->name,
 							       new_blob, size,
 							       &buf, &meta, dco);
-			if (ret && string_list_has_string(&dco->paths, ce->name)) {
-				free(new_blob);
-				goto delayed;
+			if (ret) {
+				struct string_list_item *item =
+					string_list_lookup(&dco->paths, ce->name);
+				if (item) {
+					item->util = nr_checkouts ? nr_checkouts
+							: &scratch_nr_checkouts;
+					free(new_blob);
+					goto delayed;
+				}
 			}
 		} else {
 			ret = convert_to_working_tree_ca(ca, ce->name, new_blob,
@@ -392,6 +399,8 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 					   ce->name);
 		update_ce_after_write(state, ce , &st);
 	}
+	if (nr_checkouts)
+		(*nr_checkouts)++;
 delayed:
 	return 0;
 }
@@ -476,7 +485,7 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,
 			convert_attrs(state->istate, &ca_buf, ce->name);
 			ca = &ca_buf;
 		}
-		return write_entry(ce, topath, ca, state, 1);
+		return write_entry(ce, topath, ca, state, 1, nr_checkouts);
 	}
 
 	strbuf_reset(&path);
@@ -540,18 +549,15 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,
 
 	create_directories(path.buf, path.len, state);
 
-	if (nr_checkouts)
-		(*nr_checkouts)++;
-
 	if (S_ISREG(ce->ce_mode) && !ca) {
 		convert_attrs(state->istate, &ca_buf, ce->name);
 		ca = &ca_buf;
 	}
 
-	if (!enqueue_checkout(ce, ca))
+	if (!enqueue_checkout(ce, ca, nr_checkouts))
 		return 0;
 
-	return write_entry(ce, path.buf, ca, state, 0);
+	return write_entry(ce, path.buf, ca, state, 0, nr_checkouts);
 }
 
 void unlink_entry(const struct cache_entry *ce)
diff --git a/entry.h b/entry.h
index 252fd24c2e..9be4659881 100644
--- a/entry.h
+++ b/entry.h
@@ -43,8 +43,7 @@ static inline int checkout_entry(struct cache_entry *ce,
 }
 
 void enable_delayed_checkout(struct checkout *state);
-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
-			    int show_progress);
+int finish_delayed_checkout(struct checkout *state, int show_progress);
 
 /*
  * Unlink the last component and schedule the leading directories for
diff --git a/parallel-checkout.c b/parallel-checkout.c
index 31a3d0ee1b..4f6819f240 100644
--- a/parallel-checkout.c
+++ b/parallel-checkout.c
@@ -143,7 +143,8 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,
 	}
 }
 
-int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)
+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca,
+		     int *checkout_counter)
 {
 	struct parallel_checkout_item *pc_item;
 
@@ -159,6 +160,7 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)
 	memcpy(&pc_item->ca, ca, sizeof(pc_item->ca));
 	pc_item->status = PC_ITEM_PENDING;
 	pc_item->id = parallel_checkout.nr;
+	pc_item->checkout_counter = checkout_counter;
 	parallel_checkout.nr++;
 
 	return 0;
@@ -200,7 +202,8 @@ static int handle_results(struct checkout *state)
 
 		switch(pc_item->status) {
 		case PC_ITEM_WRITTEN:
-			/* Already handled */
+			if (pc_item->checkout_counter)
+				(*pc_item->checkout_counter)++;
 			break;
 		case PC_ITEM_COLLIDED:
 			/*
@@ -225,7 +228,8 @@ static int handle_results(struct checkout *state)
 			 * add any extra overhead.
 			 */
 			ret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,
-						 state, NULL, NULL);
+						 state, NULL,
+						 pc_item->checkout_counter);
 			advance_progress_meter();
 			break;
 		case PC_ITEM_PENDING:
diff --git a/parallel-checkout.h b/parallel-checkout.h
index 80f539bcb7..c575284005 100644
--- a/parallel-checkout.h
+++ b/parallel-checkout.h
@@ -31,7 +31,8 @@ void init_parallel_checkout(void);
  * entry is not eligible for parallel checkout. Otherwise, enqueue the entry
  * for later write and return 0.
  */
-int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);
+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca,
+		     int *checkout_counter);
 size_t pc_queue_size(void);
 
 /*
@@ -68,6 +69,7 @@ struct parallel_checkout_item {
 	struct cache_entry *ce;
 	struct conv_attrs ca;
 	size_t id; /* position in parallel_checkout.items[] of main process */
+	int *checkout_counter;
 
 	/* Output fields, sent from workers. */
 	enum pc_item_status status;
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index 00df9b5c18..1c840348bd 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -1132,7 +1132,7 @@ do
 	'
 done
 
-test_expect_failure PERL 'delayed checkout correctly reports the number of updated entries' '
+test_expect_success PERL 'delayed checkout correctly reports the number of updated entries' '
 	rm -rf repo &&
 	git init repo &&
 	(
diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
index 6fd7e4c4b2..950361d767 100755
--- a/t/t2080-parallel-checkout-basics.sh
+++ b/t/t2080-parallel-checkout-basics.sh
@@ -230,7 +230,7 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d
 # check the final report including sequential, parallel, and delayed entries
 # all at the same time. So we must have finer control of the parallel checkout
 # variables.
-test_expect_failure PERL '"git checkout ." report should not include failed entries' '
+test_expect_success PERL '"git checkout ." report should not include failed entries' '
 	write_script rot13-filter.pl "$PERL_PATH" \
 		<"$TEST_DIRECTORY"/t0021/rot13-filter.pl &&
 
diff --git a/unpack-trees.c b/unpack-trees.c
index d561ca01ed..8a454e03bf 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -487,7 +487,7 @@ static int check_updates(struct unpack_trees_options *o,
 		errs |= run_parallel_checkout(&state, pc_workers, pc_threshold,
 					      progress, &cnt);
 	stop_progress(&progress);
-	errs |= finish_delayed_checkout(&state, NULL, o->verbose_update);
+	errs |= finish_delayed_checkout(&state, o->verbose_update);
 	git_attr_set_direction(GIT_ATTR_CHECKIN);
 
 	if (o->clone)
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [PATCH 2/3] checkout: show bug about failed entries being included in final report
  2022-07-13  4:19 ` [PATCH 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
@ 2022-07-13 11:14   ` Ævar Arnfjörð Bjarmason
       [not found]     ` <CAHd-oW6AeOGv=zQ=9Udtzwau=5XbQkhuctVDa0=4PoMTSU20HQ@mail.gmail.com>
  0 siblings, 1 reply; 11+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2022-07-13 11:14 UTC (permalink / raw)
  To: Matheus Tavares; +Cc: git


On Wed, Jul 13 2022, Matheus Tavares wrote:

> After checkout, git usually reports how many entries were updated at
> that operation. However, because we count the entries too soon during
> the checkout process, we may actually include entries that do not get
> properly checked out in the end. This can lead to an inaccurate final
> report if the user expects it to show only the *successful* updates.
> This will be fixed in the next commit, but for now let's document it
> with a test that cover all checkout modes.
>
> Note that `test_checkout_workers` have to be slightly adjusted in order
> to use the construct `test_checkout_workers ...  test_must_fail git
> checkout`. The function runs the command given to it with an assignment
> prefix to set the GIT_TRACE2 variable. However, this this assignment has
> an undefined behavior when the command is a shell function (like
> `test_must_fail`). As POSIX specifies:
>
>   If the command name is a function that is not a standard utility
>   implemented as a function, variable assignments shall affect the
>   current execution environment during the execution of the function. It
>   is unspecified:
>
>     - Whether or not the variable assignments persist after the
>       completion of the function
>
>     - Whether or not the variables gain the export attribute during the
>       execution of the function
>
> Thus, in order to make sure the GIT_TRACE2 value gets visible to the git
> command executed by `test_must_fail`, export the variable and run git in
> a subshell.
>
> [1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
>      (Section 2.9.1: Simple Commands)
>
> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
> ---
>  t/lib-parallel-checkout.sh          |  6 +++-
>  t/t2080-parallel-checkout-basics.sh | 50 +++++++++++++++++++++++++++++
>  2 files changed, 55 insertions(+), 1 deletion(-)
>
> diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh
> index 83b279a846..acaee9cbb6 100644
> --- a/t/lib-parallel-checkout.sh
> +++ b/t/lib-parallel-checkout.sh
> @@ -25,7 +25,11 @@ test_checkout_workers () {
>  
>  	local trace_file=trace-test-checkout-workers &&
>  	rm -f "$trace_file" &&
> -	GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 &&
> +	(
> +		GIT_TRACE2="$(pwd)/$trace_file" &&
> +		export GIT_TRACE2 &&
> +		"$@" 2>&8
> +	) &&
>  
>  	local workers="$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l)" &&
>  	test $workers -eq $expected_workers &&
> diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
> index 3e0f8c675f..6fd7e4c4b2 100755
> --- a/t/t2080-parallel-checkout-basics.sh
> +++ b/t/t2080-parallel-checkout-basics.sh
> @@ -226,4 +226,54 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d
>  	)
>  '
>  
> +# This test is here (and not in e.g. t2022-checkout-paths.sh), because we
> +# check the final report including sequential, parallel, and delayed entries
> +# all at the same time. So we must have finer control of the parallel checkout
> +# variables.
> +test_expect_failure PERL '"git checkout ." report should not include failed entries' '
> +	write_script rot13-filter.pl "$PERL_PATH" \
> +		<"$TEST_DIRECTORY"/t0021/rot13-filter.pl &&
> +
> +	test_config_global filter.delay.process \
> +		"\"$(pwd)/rot13-filter.pl\" --always-delay delayed.log clean smudge delay" &&
> +	test_config_global filter.delay.required true &&
> +	test_config_global filter.cat.clean cat  &&
> +	test_config_global filter.cat.smudge cat  &&
> +	test_config_global filter.cat.required true  &&
> +
> +	set_checkout_config 2 0 &&
> +	git init failed_entries &&
> +	(
> +		cd failed_entries &&
> +		cat >.gitattributes <<-EOF &&
> +		*delay*              filter=delay
> +		parallel-ineligible* filter=cat
> +		EOF
> +		echo a >missing-delay.a &&
> +		echo a >parallel-ineligible.a &&
> +		echo a >parallel-eligible.a &&
> +		echo b >success-delay.b &&
> +		echo b >parallel-ineligible.b &&
> +		echo b >parallel-eligible.b &&
> +		git add -A &&
> +		git commit -m files &&
> +
> +		a_blob="$(git rev-parse :parallel-ineligible.a)" &&
> +		dir="$(echo "$a_blob" | cut -c 1-2)" &&
> +		file="$(echo "$a_blob" | cut -c 3-)" &&

Can't this use test_oid_to_path?
> +		rm ".git/objects/$dir/$file" &&
> +		rm *.a *.b &&
> +
> +		test_checkout_workers 2 test_must_fail git checkout . 2>err &&
> +
> +		# All *.b entries should succeed and all *.a entries should fail:
> +		#  - missing-delay.a: the delay filter will drop this path
> +		#  - parallel-*.a: the blob will be missing
> +		#
> +		grep "Updated 3 paths from the index" err &&
> +		test "$(ls *.b | wc -l)" -eq 3 &&
> +		test "$(ls *.a | wc -l)" -eq 0

And this test_stdout_line_count?

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 2/3] checkout: show bug about failed entries being included in final report
       [not found]     ` <CAHd-oW6AeOGv=zQ=9Udtzwau=5XbQkhuctVDa0=4PoMTSU20HQ@mail.gmail.com>
@ 2022-07-13 13:00       ` Matheus Tavares
  0 siblings, 0 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-13 13:00 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: git

On Wed, Jul 13, 2022 at 8:15 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> On Wed, Jul 13 2022, Matheus Tavares wrote:
> >
> > +             a_blob="$(git rev-parse :parallel-ineligible.a)" &&
> > +             dir="$(echo "$a_blob" | cut -c 1-2)" &&
> > +             file="$(echo "$a_blob" | cut -c 3-)" &&
>
> Can't this use test_oid_to_path?
[...]
> > +             test "$(ls *.b | wc -l)" -eq 3 &&
> > +             test "$(ls *.a | wc -l)" -eq 0
>
> And this test_stdout_line_count?

Sure! Thanks for pointing that out. Will change.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice
  2022-07-13  4:19 ` [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
@ 2022-07-13 17:57   ` Junio C Hamano
  0 siblings, 0 replies; 11+ messages in thread
From: Junio C Hamano @ 2022-07-13 17:57 UTC (permalink / raw)
  To: Matheus Tavares; +Cc: git

Matheus Tavares <matheus.bernardino@usp.br> writes:

> At the end of a `git checkout <pathspec>` operation, git reports how
> many paths were checked out with a message like "Updated N paths from
> the index". However, entries that end up on the delayed checkout queue
> (as requested by a long-running process filter) get counted twice,
> producing a wrong number in the final report. We will fix this bug in an
> upcoming commit. For now, only document/demonstrate it with a
> test_expect_failure.
>
> Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
> ---
>  t/t0021-conversion.sh | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
>
> diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
> index bad37abad2..00df9b5c18 100755
> --- a/t/t0021-conversion.sh
> +++ b/t/t0021-conversion.sh
> @@ -1132,4 +1132,26 @@ do
>  	'
>  done
>  
> +test_expect_failure PERL 'delayed checkout correctly reports the number of updated entries' '

It is unfortunate that we depend on Perl only to run rot13-filter;
I'll leave a #leftoverbit label here to remind us to write a
"test-tool rot13-filter" someday.  No need to do so in this series.

> +	rm -rf repo &&
> +	git init repo &&
> +	(
> +		cd repo &&
> +		git config filter.delay.process "../rot13-filter.pl delayed.log clean smudge delay" &&
> +		git config filter.delay.required true &&
> +
> +		echo "*.a filter=delay" >.gitattributes &&
> +		echo a >test-delay10.a &&
> +		echo a >test-delay11.a &&
> +		git add . &&
> +		git commit -m files &&
> +
> +		rm *.a &&
> +		git checkout . 2>err &&
> +		grep "IN: smudge test-delay10.a .* \\[DELAYED\\]" delayed.log &&
> +		grep "IN: smudge test-delay11.a .* \\[DELAYED\\]" delayed.log &&
> +		grep "Updated 2 paths from the index" err
> +	)
> +'
> +
>  test_done

^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v2 0/3] checkout: fix two bugs on count of updated entries
  2022-07-13  4:19 [PATCH 0/3] checkout: fix two bugs on count of updated entries Matheus Tavares
                   ` (2 preceding siblings ...)
  2022-07-13  4:19 ` [PATCH 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares
@ 2022-07-14 11:49 ` Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
                     ` (2 more replies)
  3 siblings, 3 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-14 11:49 UTC (permalink / raw)
  To: gitster; +Cc: git, avarab

Changed since v1: small simplification at test t2080, to use
test_oid_to_path and test_stdout_line_count.


v1 cover letter:

This fixes two issues at the "Updated %d path from the index" report at
the end of a `git checkout <paths>` operation:

  - Delayed checkout entries being counted twice.
  - Failed entries being included in the count.

The first two patches add tests and the third implements the fix. I came
across this while working at parallel checkout, but only managed to get
back to it now.

Matheus Tavares (3):
  checkout: document bug where delayed checkout counts entries twice
  checkout: show bug about failed entries being included in final report
  checkout: fix two bugs on the final count of updated entries

 builtin/checkout.c                  |  2 +-
 convert.h                           |  6 +++-
 entry.c                             | 34 +++++++++++---------
 entry.h                             |  3 +-
 parallel-checkout.c                 | 10 ++++--
 parallel-checkout.h                 |  4 ++-
 t/lib-parallel-checkout.sh          |  6 +++-
 t/t0021-conversion.sh               | 22 +++++++++++++
 t/t2080-parallel-checkout-basics.sh | 48 +++++++++++++++++++++++++++++
 unpack-trees.c                      |  2 +-
 10 files changed, 113 insertions(+), 24 deletions(-)

Range-diff against v1:
1:  694aeb19f5 = 1:  694aeb19f5 checkout: document bug where delayed checkout counts entries twice
2:  8da18a0a8c ! 2:  4541e90224 checkout: show bug about failed entries being included in final report
    @@ t/t2080-parallel-checkout-basics.sh: test_expect_success SYMLINKS 'parallel chec
     +		git commit -m files &&
     +
     +		a_blob="$(git rev-parse :parallel-ineligible.a)" &&
    -+		dir="$(echo "$a_blob" | cut -c 1-2)" &&
    -+		file="$(echo "$a_blob" | cut -c 3-)" &&
    -+		rm ".git/objects/$dir/$file" &&
    ++		rm .git/objects/$(test_oid_to_path $a_blob) &&
     +		rm *.a *.b &&
     +
     +		test_checkout_workers 2 test_must_fail git checkout . 2>err &&
    @@ t/t2080-parallel-checkout-basics.sh: test_expect_success SYMLINKS 'parallel chec
     +		#  - parallel-*.a: the blob will be missing
     +		#
     +		grep "Updated 3 paths from the index" err &&
    -+		test "$(ls *.b | wc -l)" -eq 3 &&
    -+		test "$(ls *.a | wc -l)" -eq 0
    ++		test_stdout_line_count = 3 ls *.b &&
    ++		! ls *.a
     +	)
     +'
     +
3:  5e9452be66 = 3:  b182d74e96 checkout: fix two bugs on the final count of updated entries
-- 
2.37.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v2 1/3] checkout: document bug where delayed checkout counts entries twice
  2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
@ 2022-07-14 11:49   ` Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares
  2 siblings, 0 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-14 11:49 UTC (permalink / raw)
  To: gitster; +Cc: git, avarab

At the end of a `git checkout <pathspec>` operation, git reports how
many paths were checked out with a message like "Updated N paths from
the index". However, entries that end up on the delayed checkout queue
(as requested by a long-running process filter) get counted twice,
producing a wrong number in the final report. We will fix this bug in an
upcoming commit. For now, only document/demonstrate it with a
test_expect_failure.

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 t/t0021-conversion.sh | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index bad37abad2..00df9b5c18 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -1132,4 +1132,26 @@ do
 	'
 done
 
+test_expect_failure PERL 'delayed checkout correctly reports the number of updated entries' '
+	rm -rf repo &&
+	git init repo &&
+	(
+		cd repo &&
+		git config filter.delay.process "../rot13-filter.pl delayed.log clean smudge delay" &&
+		git config filter.delay.required true &&
+
+		echo "*.a filter=delay" >.gitattributes &&
+		echo a >test-delay10.a &&
+		echo a >test-delay11.a &&
+		git add . &&
+		git commit -m files &&
+
+		rm *.a &&
+		git checkout . 2>err &&
+		grep "IN: smudge test-delay10.a .* \\[DELAYED\\]" delayed.log &&
+		grep "IN: smudge test-delay11.a .* \\[DELAYED\\]" delayed.log &&
+		grep "Updated 2 paths from the index" err
+	)
+'
+
 test_done
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH v2 2/3] checkout: show bug about failed entries being included in final report
  2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
@ 2022-07-14 11:49   ` Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares
  2 siblings, 0 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-14 11:49 UTC (permalink / raw)
  To: gitster; +Cc: git, avarab

After checkout, git usually reports how many entries were updated at
that operation. However, because we count the entries too soon during
the checkout process, we may actually include entries that do not get
properly checked out in the end. This can lead to an inaccurate final
report if the user expects it to show only the *successful* updates.
This will be fixed in the next commit, but for now let's document it
with a test that cover all checkout modes.

Note that `test_checkout_workers` have to be slightly adjusted in order
to use the construct `test_checkout_workers ...  test_must_fail git
checkout`. The function runs the command given to it with an assignment
prefix to set the GIT_TRACE2 variable. However, this this assignment has
an undefined behavior when the command is a shell function (like
`test_must_fail`). As POSIX specifies:

  If the command name is a function that is not a standard utility
  implemented as a function, variable assignments shall affect the
  current execution environment during the execution of the function. It
  is unspecified:

    - Whether or not the variable assignments persist after the
      completion of the function

    - Whether or not the variables gain the export attribute during the
      execution of the function

Thus, in order to make sure the GIT_TRACE2 value gets visible to the git
command executed by `test_must_fail`, export the variable and run git in
a subshell.

[1]: https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html
     (Vol. 3: Shell and Utilities, Section 2.9.1: Simple Commands)

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 t/lib-parallel-checkout.sh          |  6 +++-
 t/t2080-parallel-checkout-basics.sh | 48 +++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/t/lib-parallel-checkout.sh b/t/lib-parallel-checkout.sh
index 83b279a846..acaee9cbb6 100644
--- a/t/lib-parallel-checkout.sh
+++ b/t/lib-parallel-checkout.sh
@@ -25,7 +25,11 @@ test_checkout_workers () {
 
 	local trace_file=trace-test-checkout-workers &&
 	rm -f "$trace_file" &&
-	GIT_TRACE2="$(pwd)/$trace_file" "$@" 2>&8 &&
+	(
+		GIT_TRACE2="$(pwd)/$trace_file" &&
+		export GIT_TRACE2 &&
+		"$@" 2>&8
+	) &&
 
 	local workers="$(grep "child_start\[..*\] git checkout--worker" "$trace_file" | wc -l)" &&
 	test $workers -eq $expected_workers &&
diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
index 3e0f8c675f..7d6d26e1a4 100755
--- a/t/t2080-parallel-checkout-basics.sh
+++ b/t/t2080-parallel-checkout-basics.sh
@@ -226,4 +226,52 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d
 	)
 '
 
+# This test is here (and not in e.g. t2022-checkout-paths.sh), because we
+# check the final report including sequential, parallel, and delayed entries
+# all at the same time. So we must have finer control of the parallel checkout
+# variables.
+test_expect_failure PERL '"git checkout ." report should not include failed entries' '
+	write_script rot13-filter.pl "$PERL_PATH" \
+		<"$TEST_DIRECTORY"/t0021/rot13-filter.pl &&
+
+	test_config_global filter.delay.process \
+		"\"$(pwd)/rot13-filter.pl\" --always-delay delayed.log clean smudge delay" &&
+	test_config_global filter.delay.required true &&
+	test_config_global filter.cat.clean cat  &&
+	test_config_global filter.cat.smudge cat  &&
+	test_config_global filter.cat.required true  &&
+
+	set_checkout_config 2 0 &&
+	git init failed_entries &&
+	(
+		cd failed_entries &&
+		cat >.gitattributes <<-EOF &&
+		*delay*              filter=delay
+		parallel-ineligible* filter=cat
+		EOF
+		echo a >missing-delay.a &&
+		echo a >parallel-ineligible.a &&
+		echo a >parallel-eligible.a &&
+		echo b >success-delay.b &&
+		echo b >parallel-ineligible.b &&
+		echo b >parallel-eligible.b &&
+		git add -A &&
+		git commit -m files &&
+
+		a_blob="$(git rev-parse :parallel-ineligible.a)" &&
+		rm .git/objects/$(test_oid_to_path $a_blob) &&
+		rm *.a *.b &&
+
+		test_checkout_workers 2 test_must_fail git checkout . 2>err &&
+
+		# All *.b entries should succeed and all *.a entries should fail:
+		#  - missing-delay.a: the delay filter will drop this path
+		#  - parallel-*.a: the blob will be missing
+		#
+		grep "Updated 3 paths from the index" err &&
+		test_stdout_line_count = 3 ls *.b &&
+		! ls *.a
+	)
+'
+
 test_done
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* [PATCH v2 3/3] checkout: fix two bugs on the final count of updated entries
  2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
  2022-07-14 11:49   ` [PATCH v2 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
@ 2022-07-14 11:49   ` Matheus Tavares
  2 siblings, 0 replies; 11+ messages in thread
From: Matheus Tavares @ 2022-07-14 11:49 UTC (permalink / raw)
  To: gitster; +Cc: git, avarab

At the end of `git checkout <pathspec>`, we get a message informing how
many entries were updated in the working tree. However, this number can
be inaccurate for two reasons:

1) Delayed entries currently get counted twice.
2) Failed entries are included in the count.

The first problem happens because the counter is first incremented
before inserting the entry in the delayed checkout queue, and once again
when finish_delayed_checkout() calls checkout_entry(). And the second
happens because the counter is incremented too early in
checkout_entry(), before the entry was in fact checked out. Fix that by
moving the count increment further down in the call stack and removing
the duplicate increment on delayed entries. Note that we have to keep
a per-entry reference for the counter (both on parallel checkout and
delayed checkout) because not all entries are always accumulated at the
same counter. See checkout_worktree(), at builtin/checkout.c for an
example.

Signed-off-by: Matheus Tavares <matheus.bernardino@usp.br>
---
 builtin/checkout.c                  |  2 +-
 convert.h                           |  6 ++++-
 entry.c                             | 34 +++++++++++++++++------------
 entry.h                             |  3 +--
 parallel-checkout.c                 | 10 ++++++---
 parallel-checkout.h                 |  4 +++-
 t/t0021-conversion.sh               |  2 +-
 t/t2080-parallel-checkout-basics.sh |  2 +-
 unpack-trees.c                      |  2 +-
 9 files changed, 40 insertions(+), 25 deletions(-)

diff --git a/builtin/checkout.c b/builtin/checkout.c
index 2eefda81d8..df3f1663d7 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -417,7 +417,7 @@ static int checkout_worktree(const struct checkout_opts *opts,
 	mem_pool_discard(&ce_mem_pool, should_validate_cache_entries());
 	remove_marked_cache_entries(&the_index, 1);
 	remove_scheduled_dirs();
-	errs |= finish_delayed_checkout(&state, &nr_checkouts, opts->show_progress);
+	errs |= finish_delayed_checkout(&state, opts->show_progress);
 
 	if (opts->count_checkout_paths) {
 		if (nr_unmerged)
diff --git a/convert.h b/convert.h
index 5ee1c32205..0a6e4086b8 100644
--- a/convert.h
+++ b/convert.h
@@ -53,7 +53,11 @@ struct delayed_checkout {
 	enum ce_delay_state state;
 	/* List of filter drivers that signaled delayed blobs. */
 	struct string_list filters;
-	/* List of delayed blobs identified by their path. */
+	/*
+	 * List of delayed blobs identified by their path. The `util` member
+	 * holds a counter pointer which must be incremented when/if the
+	 * associated blob gets checked out.
+	 */
 	struct string_list paths;
 };
 
diff --git a/entry.c b/entry.c
index 1c9df62b30..616e4f073c 100644
--- a/entry.c
+++ b/entry.c
@@ -157,12 +157,11 @@ static int remove_available_paths(struct string_list_item *item, void *cb_data)
 
 	available = string_list_lookup(available_paths, item->string);
 	if (available)
-		available->util = (void *)item->string;
+		available->util = item->util;
 	return !available;
 }
 
-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
-			    int show_progress)
+int finish_delayed_checkout(struct checkout *state, int show_progress)
 {
 	int errs = 0;
 	unsigned processed_paths = 0;
@@ -227,7 +226,7 @@ int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
 						       strlen(path->string), 0);
 				if (ce) {
 					display_progress(progress, ++processed_paths);
-					errs |= checkout_entry(ce, state, NULL, nr_checkouts);
+					errs |= checkout_entry(ce, state, NULL, path->util);
 					filtered_bytes += ce->ce_stat_data.sd_size;
 					display_throughput(progress, filtered_bytes);
 				} else
@@ -266,7 +265,8 @@ void update_ce_after_write(const struct checkout *state, struct cache_entry *ce,
 
 /* Note: ca is used (and required) iff the entry refers to a regular file. */
 static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca,
-		       const struct checkout *state, int to_tempfile)
+		       const struct checkout *state, int to_tempfile,
+		       int *nr_checkouts)
 {
 	unsigned int ce_mode_s_ifmt = ce->ce_mode & S_IFMT;
 	struct delayed_checkout *dco = state->delayed_checkout;
@@ -279,6 +279,7 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 	struct stat st;
 	const struct submodule *sub;
 	struct checkout_metadata meta;
+	static int scratch_nr_checkouts;
 
 	clone_checkout_metadata(&meta, &state->meta, &ce->oid);
 
@@ -333,9 +334,15 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 			ret = async_convert_to_working_tree_ca(ca, ce->name,
 							       new_blob, size,
 							       &buf, &meta, dco);
-			if (ret && string_list_has_string(&dco->paths, ce->name)) {
-				free(new_blob);
-				goto delayed;
+			if (ret) {
+				struct string_list_item *item =
+					string_list_lookup(&dco->paths, ce->name);
+				if (item) {
+					item->util = nr_checkouts ? nr_checkouts
+							: &scratch_nr_checkouts;
+					free(new_blob);
+					goto delayed;
+				}
 			}
 		} else {
 			ret = convert_to_working_tree_ca(ca, ce->name, new_blob,
@@ -392,6 +399,8 @@ static int write_entry(struct cache_entry *ce, char *path, struct conv_attrs *ca
 					   ce->name);
 		update_ce_after_write(state, ce , &st);
 	}
+	if (nr_checkouts)
+		(*nr_checkouts)++;
 delayed:
 	return 0;
 }
@@ -476,7 +485,7 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,
 			convert_attrs(state->istate, &ca_buf, ce->name);
 			ca = &ca_buf;
 		}
-		return write_entry(ce, topath, ca, state, 1);
+		return write_entry(ce, topath, ca, state, 1, nr_checkouts);
 	}
 
 	strbuf_reset(&path);
@@ -540,18 +549,15 @@ int checkout_entry_ca(struct cache_entry *ce, struct conv_attrs *ca,
 
 	create_directories(path.buf, path.len, state);
 
-	if (nr_checkouts)
-		(*nr_checkouts)++;
-
 	if (S_ISREG(ce->ce_mode) && !ca) {
 		convert_attrs(state->istate, &ca_buf, ce->name);
 		ca = &ca_buf;
 	}
 
-	if (!enqueue_checkout(ce, ca))
+	if (!enqueue_checkout(ce, ca, nr_checkouts))
 		return 0;
 
-	return write_entry(ce, path.buf, ca, state, 0);
+	return write_entry(ce, path.buf, ca, state, 0, nr_checkouts);
 }
 
 void unlink_entry(const struct cache_entry *ce)
diff --git a/entry.h b/entry.h
index 252fd24c2e..9be4659881 100644
--- a/entry.h
+++ b/entry.h
@@ -43,8 +43,7 @@ static inline int checkout_entry(struct cache_entry *ce,
 }
 
 void enable_delayed_checkout(struct checkout *state);
-int finish_delayed_checkout(struct checkout *state, int *nr_checkouts,
-			    int show_progress);
+int finish_delayed_checkout(struct checkout *state, int show_progress);
 
 /*
  * Unlink the last component and schedule the leading directories for
diff --git a/parallel-checkout.c b/parallel-checkout.c
index 31a3d0ee1b..4f6819f240 100644
--- a/parallel-checkout.c
+++ b/parallel-checkout.c
@@ -143,7 +143,8 @@ static int is_eligible_for_parallel_checkout(const struct cache_entry *ce,
 	}
 }
 
-int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)
+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca,
+		     int *checkout_counter)
 {
 	struct parallel_checkout_item *pc_item;
 
@@ -159,6 +160,7 @@ int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca)
 	memcpy(&pc_item->ca, ca, sizeof(pc_item->ca));
 	pc_item->status = PC_ITEM_PENDING;
 	pc_item->id = parallel_checkout.nr;
+	pc_item->checkout_counter = checkout_counter;
 	parallel_checkout.nr++;
 
 	return 0;
@@ -200,7 +202,8 @@ static int handle_results(struct checkout *state)
 
 		switch(pc_item->status) {
 		case PC_ITEM_WRITTEN:
-			/* Already handled */
+			if (pc_item->checkout_counter)
+				(*pc_item->checkout_counter)++;
 			break;
 		case PC_ITEM_COLLIDED:
 			/*
@@ -225,7 +228,8 @@ static int handle_results(struct checkout *state)
 			 * add any extra overhead.
 			 */
 			ret |= checkout_entry_ca(pc_item->ce, &pc_item->ca,
-						 state, NULL, NULL);
+						 state, NULL,
+						 pc_item->checkout_counter);
 			advance_progress_meter();
 			break;
 		case PC_ITEM_PENDING:
diff --git a/parallel-checkout.h b/parallel-checkout.h
index 80f539bcb7..c575284005 100644
--- a/parallel-checkout.h
+++ b/parallel-checkout.h
@@ -31,7 +31,8 @@ void init_parallel_checkout(void);
  * entry is not eligible for parallel checkout. Otherwise, enqueue the entry
  * for later write and return 0.
  */
-int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca);
+int enqueue_checkout(struct cache_entry *ce, struct conv_attrs *ca,
+		     int *checkout_counter);
 size_t pc_queue_size(void);
 
 /*
@@ -68,6 +69,7 @@ struct parallel_checkout_item {
 	struct cache_entry *ce;
 	struct conv_attrs ca;
 	size_t id; /* position in parallel_checkout.items[] of main process */
+	int *checkout_counter;
 
 	/* Output fields, sent from workers. */
 	enum pc_item_status status;
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index 00df9b5c18..1c840348bd 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -1132,7 +1132,7 @@ do
 	'
 done
 
-test_expect_failure PERL 'delayed checkout correctly reports the number of updated entries' '
+test_expect_success PERL 'delayed checkout correctly reports the number of updated entries' '
 	rm -rf repo &&
 	git init repo &&
 	(
diff --git a/t/t2080-parallel-checkout-basics.sh b/t/t2080-parallel-checkout-basics.sh
index 7d6d26e1a4..c683e60007 100755
--- a/t/t2080-parallel-checkout-basics.sh
+++ b/t/t2080-parallel-checkout-basics.sh
@@ -230,7 +230,7 @@ test_expect_success SYMLINKS 'parallel checkout checks for symlinks in leading d
 # check the final report including sequential, parallel, and delayed entries
 # all at the same time. So we must have finer control of the parallel checkout
 # variables.
-test_expect_failure PERL '"git checkout ." report should not include failed entries' '
+test_expect_success PERL '"git checkout ." report should not include failed entries' '
 	write_script rot13-filter.pl "$PERL_PATH" \
 		<"$TEST_DIRECTORY"/t0021/rot13-filter.pl &&
 
diff --git a/unpack-trees.c b/unpack-trees.c
index d561ca01ed..8a454e03bf 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -487,7 +487,7 @@ static int check_updates(struct unpack_trees_options *o,
 		errs |= run_parallel_checkout(&state, pc_workers, pc_threshold,
 					      progress, &cnt);
 	stop_progress(&progress);
-	errs |= finish_delayed_checkout(&state, NULL, o->verbose_update);
+	errs |= finish_delayed_checkout(&state, o->verbose_update);
 	git_attr_set_direction(GIT_ATTR_CHECKIN);
 
 	if (o->clone)
-- 
2.37.0


^ permalink raw reply related	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2022-07-14 11:49 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-13  4:19 [PATCH 0/3] checkout: fix two bugs on count of updated entries Matheus Tavares
2022-07-13  4:19 ` [PATCH 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
2022-07-13 17:57   ` Junio C Hamano
2022-07-13  4:19 ` [PATCH 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
2022-07-13 11:14   ` Ævar Arnfjörð Bjarmason
     [not found]     ` <CAHd-oW6AeOGv=zQ=9Udtzwau=5XbQkhuctVDa0=4PoMTSU20HQ@mail.gmail.com>
2022-07-13 13:00       ` Matheus Tavares
2022-07-13  4:19 ` [PATCH 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares
2022-07-14 11:49 ` [PATCH v2 0/3] checkout: fix two bugs on " Matheus Tavares
2022-07-14 11:49   ` [PATCH v2 1/3] checkout: document bug where delayed checkout counts entries twice Matheus Tavares
2022-07-14 11:49   ` [PATCH v2 2/3] checkout: show bug about failed entries being included in final report Matheus Tavares
2022-07-14 11:49   ` [PATCH v2 3/3] checkout: fix two bugs on the final count of updated entries Matheus Tavares

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.