* [PATCH] upload-pack: teach deepen-relative in protocol v2
@ 2018-12-18 21:24 Jonathan Tan
2019-01-09 19:35 ` Josh Steadmon
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Tan @ 2018-12-18 21:24 UTC (permalink / raw)
To: git; +Cc: Jonathan Tan, avarab
Commit 685fbd3291 ("fetch-pack: perform a fetch using v2", 2018-03-15)
attempted to teach Git deepen-relative in protocol v2 (among other
things), but it didn't work:
(1) fetch-pack.c needs to emit "deepen-relative".
(2) upload-pack.c needs to ensure that the correct deepen_relative
variable is passed to deepen() (there are two - the static variable
and the one in struct upload_pack_data).
(3) Before deepen() computes the list of reachable shallows, it first
needs to mark all "our" refs as OUR_REF. v2 currently does not do
this, because unlike v0, it is not needed otherwise.
Fix all this and include a test demonstrating that it works now. For
(2), the static variable deepen_relative is also eliminated, removing a
source of confusion.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
Similar to my first fix [1], another fix for an issue discovered by
Aevar's GIT_TEST_PROTOCOL_VERSION patches. This one is more
straightforward.
With this fix and my first fix [1], t5500 no longer reveals any more
bugs. (There might be more in other test files.)
[1] https://public-inbox.org/git/20181218010811.143608-1-jonathantanmy@google.com/
---
fetch-pack.c | 2 ++
t/t5702-protocol-v2.sh | 29 +++++++++++++++++++++++++++++
upload-pack.c | 17 +++++++++++++++--
3 files changed, 46 insertions(+), 2 deletions(-)
diff --git a/fetch-pack.c b/fetch-pack.c
index 9691046e64..c383ea46b3 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1042,6 +1042,8 @@ static void add_shallow_requests(struct strbuf *req_buf,
packet_buf_write(req_buf, "deepen-not %s", s->string);
}
}
+ if (args->deepen_relative)
+ packet_buf_write(req_buf, "deepen-relative\n");
}
static void add_wants(int no_dependents, const struct ref *wants, struct strbuf *req_buf)
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 0f2b09ebb8..340953f01c 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -471,6 +471,35 @@ test_expect_success 'upload-pack respects client shallows' '
grep "fetch< version 2" trace
'
+test_expect_success 'deepen-relative' '
+ rm -rf server client trace &&
+
+ test_create_repo server &&
+ test_commit -C server one &&
+ test_commit -C server two &&
+ test_commit -C server three &&
+ git clone --depth 1 "file://$(pwd)/server" client &&
+ test_commit -C server four &&
+
+ # Sanity check that only "three" is downloaded
+ git -C client log --pretty=tformat:%s master >actual &&
+ echo three >expected &&
+ test_cmp expected actual &&
+
+ GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \
+ fetch --deepen=1 origin &&
+ # Ensure that protocol v2 is used
+ grep "fetch< version 2" trace &&
+
+ git -C client log --pretty=tformat:%s origin/master >actual &&
+ cat >expected <<-\EOF &&
+ four
+ three
+ two
+ EOF
+ test_cmp expected actual
+'
+
# Test protocol v2 with 'http://' transport
#
. "$TEST_DIRECTORY"/lib-httpd.sh
diff --git a/upload-pack.c b/upload-pack.c
index 5e81f1ff24..9df27b55a0 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -43,7 +43,6 @@
static timestamp_t oldest_have;
-static int deepen_relative;
static int multi_ack;
static int no_done;
static int use_thin_pack, use_ofs_delta, use_include_tag;
@@ -662,6 +661,9 @@ static void send_unshallow(const struct object_array *shallows,
}
}
+static int check_ref(const char *refname_full, const struct object_id *oid,
+ int flag, void *cb_data);
+
static void deepen(int depth, int deepen_relative,
struct object_array *shallows, struct object_array *want_obj)
{
@@ -676,6 +678,13 @@ static void deepen(int depth, int deepen_relative,
struct object_array reachable_shallows = OBJECT_ARRAY_INIT;
struct commit_list *result;
+ /*
+ * Checking for reachable shallows requires that our refs be
+ * marked with OUR_REF.
+ */
+ head_ref_namespaced(check_ref, NULL);
+ for_each_namespaced_ref(check_ref, NULL);
+
get_reachable_list(shallows, &reachable_shallows);
result = get_shallow_commits(&reachable_shallows,
depth + 1,
@@ -712,6 +721,7 @@ static void deepen_by_rev_list(int ac, const char **av,
static int send_shallow_list(int depth, int deepen_rev_list,
timestamp_t deepen_since,
struct string_list *deepen_not,
+ int deepen_relative,
struct object_array *shallows,
struct object_array *want_obj)
{
@@ -834,6 +844,7 @@ static void receive_needs(struct object_array *want_obj)
int has_non_tip = 0;
timestamp_t deepen_since = 0;
int deepen_rev_list = 0;
+ int deepen_relative = 0;
shallow_nr = 0;
for (;;) {
@@ -925,7 +936,8 @@ static void receive_needs(struct object_array *want_obj)
return;
if (send_shallow_list(depth, deepen_rev_list, deepen_since,
- &deepen_not, &shallows, want_obj))
+ &deepen_not, deepen_relative, &shallows,
+ want_obj))
packet_flush(1);
object_array_clear(&shallows);
}
@@ -1398,6 +1410,7 @@ static void send_shallow_info(struct upload_pack_data *data,
if (!send_shallow_list(data->depth, data->deepen_rev_list,
data->deepen_since, &data->deepen_not,
+ data->deepen_relative,
&data->shallows, want_obj) &&
is_repository_shallow(the_repository))
deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows,
--
2.19.0.271.gfe8321ec05.dirty
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2018-12-18 21:24 [PATCH] upload-pack: teach deepen-relative in protocol v2 Jonathan Tan
@ 2019-01-09 19:35 ` Josh Steadmon
2019-01-10 18:30 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Josh Steadmon @ 2019-01-09 19:35 UTC (permalink / raw)
To: Jonathan Tan; +Cc: git, avarab
On 2018.12.18 13:24, Jonathan Tan wrote:
> Commit 685fbd3291 ("fetch-pack: perform a fetch using v2", 2018-03-15)
> attempted to teach Git deepen-relative in protocol v2 (among other
> things), but it didn't work:
>
> (1) fetch-pack.c needs to emit "deepen-relative".
> (2) upload-pack.c needs to ensure that the correct deepen_relative
> variable is passed to deepen() (there are two - the static variable
> and the one in struct upload_pack_data).
> (3) Before deepen() computes the list of reachable shallows, it first
> needs to mark all "our" refs as OUR_REF. v2 currently does not do
> this, because unlike v0, it is not needed otherwise.
>
> Fix all this and include a test demonstrating that it works now. For
> (2), the static variable deepen_relative is also eliminated, removing a
> source of confusion.
>
> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
> ---
> Similar to my first fix [1], another fix for an issue discovered by
> Aevar's GIT_TEST_PROTOCOL_VERSION patches. This one is more
> straightforward.
>
> With this fix and my first fix [1], t5500 no longer reveals any more
> bugs. (There might be more in other test files.)
>
> [1] https://public-inbox.org/git/20181218010811.143608-1-jonathantanmy@google.com/
> ---
> fetch-pack.c | 2 ++
> t/t5702-protocol-v2.sh | 29 +++++++++++++++++++++++++++++
> upload-pack.c | 17 +++++++++++++++--
> 3 files changed, 46 insertions(+), 2 deletions(-)
>
> diff --git a/fetch-pack.c b/fetch-pack.c
> index 9691046e64..c383ea46b3 100644
> --- a/fetch-pack.c
> +++ b/fetch-pack.c
> @@ -1042,6 +1042,8 @@ static void add_shallow_requests(struct strbuf *req_buf,
> packet_buf_write(req_buf, "deepen-not %s", s->string);
> }
> }
> + if (args->deepen_relative)
> + packet_buf_write(req_buf, "deepen-relative\n");
> }
>
> static void add_wants(int no_dependents, const struct ref *wants, struct strbuf *req_buf)
> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
> index 0f2b09ebb8..340953f01c 100755
> --- a/t/t5702-protocol-v2.sh
> +++ b/t/t5702-protocol-v2.sh
> @@ -471,6 +471,35 @@ test_expect_success 'upload-pack respects client shallows' '
> grep "fetch< version 2" trace
> '
>
> +test_expect_success 'deepen-relative' '
> + rm -rf server client trace &&
> +
> + test_create_repo server &&
> + test_commit -C server one &&
> + test_commit -C server two &&
> + test_commit -C server three &&
> + git clone --depth 1 "file://$(pwd)/server" client &&
> + test_commit -C server four &&
> +
> + # Sanity check that only "three" is downloaded
> + git -C client log --pretty=tformat:%s master >actual &&
> + echo three >expected &&
> + test_cmp expected actual &&
> +
> + GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \
> + fetch --deepen=1 origin &&
> + # Ensure that protocol v2 is used
> + grep "fetch< version 2" trace &&
> +
> + git -C client log --pretty=tformat:%s origin/master >actual &&
> + cat >expected <<-\EOF &&
> + four
> + three
> + two
> + EOF
> + test_cmp expected actual
> +'
> +
> # Test protocol v2 with 'http://' transport
> #
> . "$TEST_DIRECTORY"/lib-httpd.sh
> diff --git a/upload-pack.c b/upload-pack.c
> index 5e81f1ff24..9df27b55a0 100644
> --- a/upload-pack.c
> +++ b/upload-pack.c
> @@ -43,7 +43,6 @@
>
> static timestamp_t oldest_have;
>
> -static int deepen_relative;
> static int multi_ack;
> static int no_done;
> static int use_thin_pack, use_ofs_delta, use_include_tag;
> @@ -662,6 +661,9 @@ static void send_unshallow(const struct object_array *shallows,
> }
> }
>
> +static int check_ref(const char *refname_full, const struct object_id *oid,
> + int flag, void *cb_data);
> +
> static void deepen(int depth, int deepen_relative,
> struct object_array *shallows, struct object_array *want_obj)
> {
> @@ -676,6 +678,13 @@ static void deepen(int depth, int deepen_relative,
> struct object_array reachable_shallows = OBJECT_ARRAY_INIT;
> struct commit_list *result;
>
> + /*
> + * Checking for reachable shallows requires that our refs be
> + * marked with OUR_REF.
> + */
> + head_ref_namespaced(check_ref, NULL);
> + for_each_namespaced_ref(check_ref, NULL);
> +
> get_reachable_list(shallows, &reachable_shallows);
> result = get_shallow_commits(&reachable_shallows,
> depth + 1,
> @@ -712,6 +721,7 @@ static void deepen_by_rev_list(int ac, const char **av,
> static int send_shallow_list(int depth, int deepen_rev_list,
> timestamp_t deepen_since,
> struct string_list *deepen_not,
> + int deepen_relative,
> struct object_array *shallows,
> struct object_array *want_obj)
> {
> @@ -834,6 +844,7 @@ static void receive_needs(struct object_array *want_obj)
> int has_non_tip = 0;
> timestamp_t deepen_since = 0;
> int deepen_rev_list = 0;
> + int deepen_relative = 0;
>
> shallow_nr = 0;
> for (;;) {
> @@ -925,7 +936,8 @@ static void receive_needs(struct object_array *want_obj)
> return;
>
> if (send_shallow_list(depth, deepen_rev_list, deepen_since,
> - &deepen_not, &shallows, want_obj))
> + &deepen_not, deepen_relative, &shallows,
> + want_obj))
> packet_flush(1);
> object_array_clear(&shallows);
> }
> @@ -1398,6 +1410,7 @@ static void send_shallow_info(struct upload_pack_data *data,
>
> if (!send_shallow_list(data->depth, data->deepen_rev_list,
> data->deepen_since, &data->deepen_not,
> + data->deepen_relative,
> &data->shallows, want_obj) &&
> is_repository_shallow(the_repository))
> deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows,
> --
> 2.19.0.271.gfe8321ec05.dirty
>
Looks good to me.
Reviewed-by: Josh Steadmon <steadmon@google.com>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2019-01-09 19:35 ` Josh Steadmon
@ 2019-01-10 18:30 ` Junio C Hamano
2019-01-10 20:43 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2019-01-10 18:30 UTC (permalink / raw)
To: Josh Steadmon; +Cc: Jonathan Tan, git, avarab
Josh Steadmon <steadmon@google.com> writes:
> Looks good to me.
> Reviewed-by: Josh Steadmon <steadmon@google.com>
Thanks, both.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2019-01-10 18:30 ` Junio C Hamano
@ 2019-01-10 20:43 ` Junio C Hamano
2019-01-10 21:28 ` Jonathan Tan
0 siblings, 1 reply; 7+ messages in thread
From: Junio C Hamano @ 2019-01-10 20:43 UTC (permalink / raw)
To: Josh Steadmon; +Cc: Jonathan Tan, git, avarab
Junio C Hamano <gitster@pobox.com> writes:
> Josh Steadmon <steadmon@google.com> writes:
>
>> Looks good to me.
>> Reviewed-by: Josh Steadmon <steadmon@google.com>
>
> Thanks, both.
This, when applied on top of ecbdaf08991, seems to break t5702.30,
at least for me.
Tip of 'pu' in today's push-out will be broken as it has this patch
merged.
Can somebody take a look?
expecting success:
rm -rf server client trace &&
test_create_repo server &&
test_commit -C server one &&
test_commit -C server two &&
test_commit -C server three &&
git clone --depth 1 "file://$(pwd)/server" client &&
test_commit -C server four &&
# Sanity check that only "three" is downloaded
git -C client log --pretty=tformat:%s master >actual &&
echo three >expected &&
test_cmp expected actual &&
GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \
fetch --deepen=1 origin &&
# Ensure that protocol v2 is used
grep "fetch< version 2" trace &&
git -C client log --pretty=tformat:%s origin/master >actual &&
cat >expected <<-\EOF &&
four
three
two
EOF
test_cmp expected actual
Initialized empty Git repository in /home/gitster/w/git.git/t/trash directory.t5702-protocol-v2/server/.git/
[master (root-commit) 1581e3e] one
Author: A U Thor <author@example.com>
1 file changed, 1 insertion(+)
create mode 100644 one.t
[master 5680d21] two
Author: A U Thor <author@example.com>
1 file changed, 1 insertion(+)
create mode 100644 two.t
[master 94705d7] three
Author: A U Thor <author@example.com>
1 file changed, 1 insertion(+)
create mode 100644 three.t
Cloning into 'client'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (5/5), done.
[master 0b417c7] four
Author: A U Thor <author@example.com>
1 file changed, 1 insertion(+)
create mode 100644 four.t
remote: Enumerating objects: 6, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From file:///home/gitster/w/git.git/t/trash directory.t5702-protocol-v2/server
94705d7..0b417c7 master -> origin/master
* [new tag] four -> four
fatal: shallow file has changed since we read it
not ok 30 - deepen-relative
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2019-01-10 20:43 ` Junio C Hamano
@ 2019-01-10 21:28 ` Jonathan Tan
2019-01-10 22:11 ` Jonathan Tan
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Tan @ 2019-01-10 21:28 UTC (permalink / raw)
To: gitster; +Cc: steadmon, jonathantanmy, git, avarab
> Junio C Hamano <gitster@pobox.com> writes:
>
> > Josh Steadmon <steadmon@google.com> writes:
> >
> >> Looks good to me.
> >> Reviewed-by: Josh Steadmon <steadmon@google.com>
> >
> > Thanks, both.
>
> This, when applied on top of ecbdaf08991, seems to break t5702.30,
> at least for me.
>
> Tip of 'pu' in today's push-out will be broken as it has this patch
> merged.
>
> Can somebody take a look?
I can reproduce this. I'll investigate this further. In the meantime,
feel free to remove this from pu.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2019-01-10 21:28 ` Jonathan Tan
@ 2019-01-10 22:11 ` Jonathan Tan
2019-01-10 22:54 ` Junio C Hamano
0 siblings, 1 reply; 7+ messages in thread
From: Jonathan Tan @ 2019-01-10 22:11 UTC (permalink / raw)
To: jonathantanmy; +Cc: gitster, steadmon, git, avarab
> > This, when applied on top of ecbdaf08991, seems to break t5702.30,
> > at least for me.
> >
> > Tip of 'pu' in today's push-out will be broken as it has this patch
> > merged.
> >
> > Can somebody take a look?
>
> I can reproduce this. I'll investigate this further. In the meantime,
> feel free to remove this from pu.
The test failure is caused by a bug I fixed in [1]. I've verified that
if I apply [1] first, then this, all tests pass. If you do include this
patch in some branch, it should be dependent on [1], but I understand if
you'd like reviewers to look at both this and [1] first.
I thought I tested this without something that fixes the issue fixed by
[1] and it passed, but apparently that's not true.
[1] https://public-inbox.org/git/20190110193645.34080-1-jonathantanmy@google.com/
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] upload-pack: teach deepen-relative in protocol v2
2019-01-10 22:11 ` Jonathan Tan
@ 2019-01-10 22:54 ` Junio C Hamano
0 siblings, 0 replies; 7+ messages in thread
From: Junio C Hamano @ 2019-01-10 22:54 UTC (permalink / raw)
To: Jonathan Tan; +Cc: steadmon, git, avarab
Jonathan Tan <jonathantanmy@google.com> writes:
>> > This, when applied on top of ecbdaf08991, seems to break t5702.30,
>> > at least for me.
>> >
>> > Tip of 'pu' in today's push-out will be broken as it has this patch
>> > merged.
>> >
>> > Can somebody take a look?
>>
>> I can reproduce this. I'll investigate this further. In the meantime,
>> feel free to remove this from pu.
>
> The test failure is caused by a bug I fixed in [1]. I've verified that
> if I apply [1] first, then this, all tests pass. If you do include this
> patch in some branch, it should be dependent on [1], but I understand if
> you'd like reviewers to look at both this and [1] first.
>
> I thought I tested this without something that fixes the issue fixed by
> [1] and it passed, but apparently that's not true.
>
> [1] https://public-inbox.org/git/20190110193645.34080-1-jonathantanmy@google.com/
OK, that means that it would be the cleanest to make them into
two-patch series, I guess.
Let's requeue these two and see what happens.
Thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-01-10 22:54 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-18 21:24 [PATCH] upload-pack: teach deepen-relative in protocol v2 Jonathan Tan
2019-01-09 19:35 ` Josh Steadmon
2019-01-10 18:30 ` Junio C Hamano
2019-01-10 20:43 ` Junio C Hamano
2019-01-10 21:28 ` Jonathan Tan
2019-01-10 22:11 ` Jonathan Tan
2019-01-10 22:54 ` Junio C Hamano
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.