reducing object store size with remote alternates or shallow clone?

reducing object store size with remote alternates or shallow clone?

[Kumar Gala]

I'm trying to package a linux kernel source tree and would like to just tar up a tree with .git/.  However this is a bit problematic as the size of tgz is about 500M which exceeds the limit of the image I'm trying to produce.

I was wondering if anyone had a means to reduce the size drastically.  I'm ok if a post processing step is required to get full git support.  One idea I had was doing a shallow clone and if there was some means to "reconnect" it to a full work state after the fact.

thanks

- k

Re: reducing object store size with remote alternates or shallow clone?

[Junio C Hamano]

Kumar Gala <galak@kernel.crashing.org> writes:

> I'm trying to package a linux kernel source tree and would like to just
> tar up a tree with .git/.  However this is a bit problematic as the size
> of tgz is about 500M which exceeds the limit of the image I'm trying to
> produce.
>
> I was wondering if anyone had a means to reduce the size drastically.
> I'm ok if a post processing step is required to get full git support.
> One idea I had was doing a shallow clone and if there was some means to
> "reconnect" it to a full work state after the fact.

Although your message body does not state your design constraints clearly,
your subject line hints that you are fine with a solution that involves
use of remote alternates by the recipient of your tarball.

I am _guessing_ that you have a fork from some well known tree and want to
sneakernet it to your recipients (iow, they do not "git pull" from your
repository).

How about doing

    $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    $ git fetch $LINUS
    $ git bundle create myfork.bundle HEAD..master

and sending the thing over?  The recipient would then do something like:

    $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
    $ git clone $LINUS linux
    $ cd linux
    $ git pull /tmp/myfork.bundle master

Re: reducing object store size with remote alternates or shallow clone?

[Brandon Casey]

On 08/24/2010 11:45 AM, Junio C Hamano wrote:

> How about doing
> 
>     $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

>     $ git fetch $LINUS
>     $ git bundle create myfork.bundle HEAD..master

I think you mean

      $ git fetch $LINUS master
      $ git bundle create myfork.bundle FETCH_HEAD..master

-Brandon

Re: reducing object store size with remote alternates or shallow clone?

[Junio C Hamano]

Brandon Casey <brandon.casey.ctr@nrlssc.navy.mil> writes:

> On 08/24/2010 11:45 AM, Junio C Hamano wrote:
>
>> How about doing
>> 
>>     $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>
>>     $ git fetch $LINUS
>>     $ git bundle create myfork.bundle HEAD..master
>
> I think you mean
>
>       $ git fetch $LINUS master
>       $ git bundle create myfork.bundle FETCH_HEAD..master

Thanks, of course you are right.

Strictly speaking, as I know there is only one branch in the repository of
Linus, there is no need to say "master" when fetching, but it would be a
good discipline to explicitly specify what you mean on the command line.

Re: reducing object store size with remote alternates or shallow clone?

[Brandon Casey]

On 08/24/2010 01:59 PM, Junio C Hamano wrote:
> Brandon Casey <brandon.casey.ctr@nrlssc.navy.mil> writes:
> 
>> On 08/24/2010 11:45 AM, Junio C Hamano wrote:
>>
>>> How about doing
>>>
>>>     $ LINUS=git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>>
>>>     $ git fetch $LINUS
>>>     $ git bundle create myfork.bundle HEAD..master
>>
>> I think you mean
>>
>>       $ git fetch $LINUS master
>>       $ git bundle create myfork.bundle FETCH_HEAD..master
> 
> Thanks, of course you are right.
> 
> Strictly speaking, as I know there is only one branch in the repository of
> Linus, there is no need to say "master" when fetching

Hmm.  It appears that if the current checked-out branch has a configured
merge ref, then a fetch that supplies a repository url (not a remote name)
and no fetch refspec, will not fall back to fetch HEAD from the remote
repository.

i.e. the following fetch does not retrieve any objects nor update FETCH_HEAD

   $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux
   $ cd linux
   $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

but, if you create a new branch, that has no merge ref configuration, then
git behaves as expected:

   $ git branch foo
   $ git checkout foo
   $ git fetch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

Namely we retrieve new objects and update FETCH_HEAD:

   From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6
   * branch            HEAD       -> FETCH_HEAD


I think the problem is in builtin/fetch.c: get_ref_map().

When fetch is called as above, with a repository url but no refspec,
we get this call sequence:

   cmd_fetch -> fetch_one
     fetch_one -> do_fetch(argc = 0)
       do_fetch -> get_ref_map(ref_count = 0)
         line 148: has_merge is assigned 1 since the current checked out
                   branch has a merge ref configured
         The 'if' branch is entered, the 'for' loop is not entered,
         ref_map retains its NULL initialization value and
         get_ref_map() returns NULL
       do_fetch -> fetch_refs(ref_map = NULL)
         and the transports do nothing since no refs have been requested

Perhaps the fix should look something like this (warning copy/paste):


diff --git a/builtin/fetch.c b/builtin/fetch.c
index fab3fce..218e71d 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -146,7 +146,8 @@ static struct ref *get_ref_map(struct transport *transport,
                struct remote *remote = transport->remote;
                struct branch *branch = branch_get(NULL);
                int has_merge = branch_has_merge_config(branch);
-               if (remote && (remote->fetch_refspec_nr || has_merge)) {
+               if (remote && (remote->fetch_refspec_nr || (has_merge &&
+                               !strcmp(branch->remote_name, remote->name)))) {
                        for (i = 0; i < remote->fetch_refspec_nr; i++) {
                                get_fetch_map(remote_refs, &remote->fetch[i], &tail, 0);
                                if (remote->fetch[i].dst &&


-Brandon

[PATCH 1/2] t/t5510: demonstrate failure to fetch when current branch has merge ref

[Brandon Casey]

From: Brandon Casey <drafnel@gmail.com>

When 'git fetch' is supplied just a repository URL (not a remote name),
and without a fetch refspec, it should fetch from the remote HEAD branch
and update FETCH_HEAD with the fetched ref.  Currently, when 'git fetch'
is called like this, it fails to retrieve anything, and does not update
FETCH_HEAD, if the current checked-out branch has a configured merge ref.

i.e. this fetch fails to retrieve anything nor update FETCH_HEAD:

   git checkout master
   git config branch.master.merge refs/heads/master
   git fetch git://git.kernel.org/pub/scm/git/git.git

but this one does:

   git config --unset branch.master.merge
   git fetch git://git.kernel.org/pub/scm/git/git.git

Add a test to demonstrate this flaw.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 t/t5510-fetch.sh |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 4eb10f6..3c7569c 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -240,6 +240,12 @@ test_expect_success 'fetch with a non-applying branch.<name>.merge' '
 	git fetch blub
 '
 
+test_expect_failure 'fetch from GIT URL with a non-applying branch.<name>.merge' '
+	git update-ref -d FETCH_HEAD &&
+	git fetch one &&
+	git rev-parse --verify FETCH_HEAD
+'
+
 # the strange name is: a\!'b
 test_expect_success 'quoting of a strangely named repo' '
 	test_must_fail git fetch "a\\!'\''b" > result 2>&1 &&
-- 
1.7.2.1

[PATCH 2/2] builtin/fetch.c: ignore merge config when not fetching from branch's remote

[Brandon Casey]

From: Brandon Casey <drafnel@gmail.com>

When 'git fetch' is supplied a single argument, it tries to match it
against a configured remote and then fetch the refs specified by the
named remote's fetchspec.  Additionally, or alternatively, if the current
branch has a merge ref configured, and if the name of the remote supplied
to fetch matches the one in the branch's configuration, then git also adds
the merge ref to the list of refs to update.

If the argument to fetch does not specify a named remote, or if the name
supplied does not match the remote configured for the current branch, then
the current branch's merge configuration should not be considered.

git currently mishandles the case when the argument to fetch specifies a
GIT URL(i.e. not a named remote) and the current branch has a configured
merge ref.  In this case, fetch should ignore the branch's merge ref and
attempt to fetch from the remote repository's HEAD branch.  But, since
fetch only checks _whether_ the current branch has a merge ref configured,
and does _not_ check whether the branch's configured remote matches the
command line argument (until later), it will mistakenly enter the wrong
branch of an 'if' statement and will not fall back to fetch the HEAD branch.
The fetch ends up doing nothing and returns with a successful zero status.

Fix this by comparing the remote repository's name to the branch's remote
name, in addition to whether it has a configured merge ref, sooner, so that
fetch can correctly decide whether the branch's configuration is interesting
or not, and fall back to fetching from the remote's HEAD branch when
appropriate.

This fixes the test in t5510.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin/fetch.c  |    3 ++-
 t/t5510-fetch.sh |    2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index ea14d5d..be6c27a 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -146,7 +146,8 @@ static struct ref *get_ref_map(struct transport *transport,
 		struct remote *remote = transport->remote;
 		struct branch *branch = branch_get(NULL);
 		int has_merge = branch_has_merge_config(branch);
-		if (remote && (remote->fetch_refspec_nr || has_merge)) {
+		if (remote && (remote->fetch_refspec_nr || (has_merge &&
+				!strcmp(branch->remote_name, remote->name)))) {
 			for (i = 0; i < remote->fetch_refspec_nr; i++) {
 				get_fetch_map(remote_refs, &remote->fetch[i], &tail, 0);
 				if (remote->fetch[i].dst &&
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 3c7569c..8fbd894 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -240,7 +240,7 @@ test_expect_success 'fetch with a non-applying branch.<name>.merge' '
 	git fetch blub
 '
 
-test_expect_failure 'fetch from GIT URL with a non-applying branch.<name>.merge' '
+test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge' '
 	git update-ref -d FETCH_HEAD &&
 	git fetch one &&
 	git rev-parse --verify FETCH_HEAD
-- 
1.7.2.1

Re: [PATCH 2/2] builtin/fetch.c: ignore merge config when not fetching from branch's remote

[Jonathan Nieder]

Brandon Casey wrote:

> If the argument to fetch does not specify a named remote, or if the name
> supplied does not match the remote configured for the current branch, then
> the current branch's merge configuration should not be considered.

Thanks for a fix.

> +++ b/builtin/fetch.c
> @@ -146,7 +146,8 @@ static struct ref *get_ref_map(struct transport *transport,
>  		struct remote *remote = transport->remote;
>  		struct branch *branch = branch_get(NULL);
>  		int has_merge = branch_has_merge_config(branch);
> -		if (remote && (remote->fetch_refspec_nr || has_merge)) {
> +		if (remote && (remote->fetch_refspec_nr || (has_merge &&
> +				!strcmp(branch->remote_name, remote->name)))) {

What will happen with this (invalid) branch?

	[branch "tmp"]
		merge = refs/heads/tmp

Re: [PATCH 1/2] t/t5510: demonstrate failure to fetch when current branch has merge ref

[Junio C Hamano]

Brandon Casey <casey@nrlssc.navy.mil> writes:

> From: Brandon Casey <drafnel@gmail.com>
>
> When 'git fetch' is supplied just a repository URL (not a remote name),
> and without a fetch refspec, it should fetch from the remote HEAD branch
> and update FETCH_HEAD with the fetched ref.  Currently, when 'git fetch'
> is called like this, it fails to retrieve anything, and does not update
> FETCH_HEAD, if the current checked-out branch has a configured merge ref.
>
> i.e. this fetch fails to retrieve anything nor update FETCH_HEAD:
>
>    git checkout master
>    git config branch.master.merge refs/heads/master
>    git fetch git://git.kernel.org/pub/scm/git/git.git

Hmph, we can call it a regression, since we certainly won't see this
failure with versions of git that is unaware of branch.*.merge.

But what should we be expecting?

Just as a datapoint, an ancient git (e.g. v1.4.0), the above command line
would have fetched the HEAD from the remote side, no matter what that
symref is pointing at.  Your [2/2] patch replicates this behaviour, which
is fine by me [*1*].

Your test only checks if we leave _anything_ in FETCH_HEAD, and does not
check if we only fetch one, if we fetch all the refs, or if we fetch what
the configuration branch.*.merge asks for (but without the corresponding
branch.*.remote configuration, doing so is pointless).

I think it would be better to have two tests.  One arranges the current
branch to track the same branch the HEAD at the remote points at, and the
other arranges the current branch to track a branch different from the
HEAD at the remote points at.  In both cases, as "fetch" should ignore the
configuration, we should get the object pointed by the HEAD on the remote
side.

Thanks.


[Footnote]

*1* A plausible alternative is to match the given URL against list of
existing remote.<name>.url (make sure there is only one), and behave as if
that the remote name is given.  I can be persuaded either way, but not
looking at the configuration feels a lot simpler to explain and understand
(i.e. "with name, we use the set of configuration variable given to that
name; without name, there is no configuration for us to look up").

Re: [PATCH 2/2] builtin/fetch.c: ignore merge config when not fetching from branch's remote

[Brandon Casey]

On 08/25/2010 04:16 PM, Jonathan Nieder wrote:
> Brandon Casey wrote:
> 
>> If the argument to fetch does not specify a named remote, or if the name
>> supplied does not match the remote configured for the current branch, then
>> the current branch's merge configuration should not be considered.
> 
> Thanks for a fix.
> 
>> +++ b/builtin/fetch.c
>> @@ -146,7 +146,8 @@ static struct ref *get_ref_map(struct transport *transport,
>>  		struct remote *remote = transport->remote;
>>  		struct branch *branch = branch_get(NULL);
>>  		int has_merge = branch_has_merge_config(branch);
>> -		if (remote && (remote->fetch_refspec_nr || has_merge)) {
>> +		if (remote && (remote->fetch_refspec_nr || (has_merge &&
>> +				!strcmp(branch->remote_name, remote->name)))) {
> 
> What will happen with this (invalid) branch?
> 
> 	[branch "tmp"]
> 		merge = refs/heads/tmp

The same thing that would have happened before, since a few lines
further down there is this:

   if (has_merge &&
       !strcmp(branch->remote_name, remote->name))
           add_merge_config(&ref_map, remote_refs, branch, &tail);

I didn't trace branch_get() to check whether it returns an object
with remote_name initialized in all cases.  I relied on the form
of the existing code.  Perhaps it's worth investigating.  If something
needs to be fixed, then it was already broken and deserves a separate
patch anyway.

-Brandon

Re: [PATCH 2/2] builtin/fetch.c: ignore merge config when not fetching from branch's remote

[Junio C Hamano]

Brandon Casey <casey@nrlssc.navy.mil> writes:

> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index ea14d5d..be6c27a 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -146,7 +146,8 @@ static struct ref *get_ref_map(struct transport *transport,
>  		struct remote *remote = transport->remote;
>  		struct branch *branch = branch_get(NULL);
>  		int has_merge = branch_has_merge_config(branch);
> -		if (remote && (remote->fetch_refspec_nr || has_merge)) {
> +		if (remote && (remote->fetch_refspec_nr || (has_merge &&
> +				!strcmp(branch->remote_name, remote->name)))) {

Couldn't branch->remote_name or remote->name be NULL here?

I think remote->name would be the same as the URL given on the command
line (i.e. no risk of being NULL), and has_merge would be false even when
branch.<name>.merge is specified if branch.<name>.remote is missing
(i.e. no risk of running this strcmp() to begin with), but the latter
safety guarantee is a bit too subtle for my taste to go without a in-code
comment.

Thanks.

[PATCH 1/2] builtin/fetch.c: comment that branch->remote_name is usable when has_merge

[Brandon Casey]

From: Brandon Casey <drafnel@gmail.com>

Save future readers the trouble of tracing code to determine that the two
uses of branch->remote_name are safe when has_merge is set, by adding a
comment explaining that it is so.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 builtin/fetch.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/builtin/fetch.c b/builtin/fetch.c
index fccc9cb..6fc5047 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -148,6 +148,7 @@ static struct ref *get_ref_map(struct transport *transport,
 		int has_merge = branch_has_merge_config(branch);
 		if (remote &&
 		    (remote->fetch_refspec_nr ||
+		     /* Note: has_merge implies non-NULL branch->remote_name */
 		     (has_merge && !strcmp(branch->remote_name, remote->name)))) {
 			for (i = 0; i < remote->fetch_refspec_nr; i++) {
 				get_fetch_map(remote_refs, &remote->fetch[i], &tail, 0);
@@ -162,6 +163,8 @@ static struct ref *get_ref_map(struct transport *transport,
 			 * if the remote we're fetching from is the same
 			 * as given in branch.<name>.remote, we add the
 			 * ref given in branch.<name>.merge, too.
+			 *
+			 * Note: has_merge implies non-NULL branch->remote_name
 			 */
 			if (has_merge &&
 			    !strcmp(branch->remote_name, remote->name))
-- 
1.7.2.1

[PATCH 2/2] t/t5510-fetch.sh: improve testing with explicit URL and merge spec

[Brandon Casey]

From: Brandon Casey <drafnel@gmail.com>

Commit 6106ce46 introduced a test to demonstrate fetch's failure to
retrieve any objects or update FETCH_HEAD when it was supplied a repository
URL and the current branch had a configured merge spec.  This commit
expands the original test based on comments from Junio Hamano.  In addition
to actually verifying that the fetch updates FETCH_HEAD correctly, and does
not update the current branch, two more tests are added to ensure that the
merge configuration is ignored even when the supplied URL matches the URL
of the remote configured for the branch.

Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil>
---
 t/t5510-fetch.sh |   30 ++++++++++++++++++++++++++++--
 1 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 8fbd894..efb42d1 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -240,10 +240,36 @@ test_expect_success 'fetch with a non-applying branch.<name>.merge' '
 	git fetch blub
 '
 
-test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge' '
+# URL supplied to fetch does not match the url of the configured branch's remote
+test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [1]' '
+	one_head=$(cd one && git rev-parse HEAD) &&
+	this_head=$(git rev-parse HEAD) &&
 	git update-ref -d FETCH_HEAD &&
 	git fetch one &&
-	git rev-parse --verify FETCH_HEAD
+	test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
+	test $this_head = "$(git rev-parse --verify HEAD)"
+'
+
+# URL supplied to fetch matches the url of the configured branch's remote and
+# the merge spec matches the branch the remote HEAD points to
+test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [2]' '
+	one_ref=$(cd one && git symbolic-ref HEAD) &&
+	git config branch.master.remote blub &&
+	git config branch.master.merge "$one_ref" &&
+	git update-ref -d FETCH_HEAD &&
+	git fetch one &&
+	test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
+	test $this_head = "$(git rev-parse --verify HEAD)"
+'
+
+# URL supplied to fetch matches the url of the configured branch's remote, but
+# the merge spec does not match the branch the remote HEAD points to
+test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [3]' '
+	git config branch.master.merge "${one_ref}_not" &&
+	git update-ref -d FETCH_HEAD &&
+	git fetch one &&
+	test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
+	test $this_head = "$(git rev-parse --verify HEAD)"
 '
 
 # the strange name is: a\!'b
-- 
1.7.2.1