Re: git gc expanding packed data?

Re: git gc expanding packed data?

[Andreas Schwab]

Nicolas Pitre <nico@cam.org> writes:

> It appears that the git installation serving clone requests for
> git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I
> just cloned it and the pack I was sent contains 1383356 objects (can be
> determined with 'git show-index < .git/objects/pack/*.idx | wc -l').
> However, there are only 978501 actually referenced objects in that
> cloned repository ( 'git rev-list --all --objects | wc -l').  That makes
> for 404855 useless objects in the cloned repository.

Those objects are not useless.  They are referenced by the remote refs
on the remote side, which are not fetched by default.  If you clone a
mirror of the repository you'll see no unreferenced objects.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git gc expanding packed data?

[Hin-Tak Leung]

On Sat, Aug 8, 2009 at 2:11 AM, Andreas Schwab<schwab@linux-m68k.org> wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
>> It appears that the git installation serving clone requests for
>> git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I
>> just cloned it and the pack I was sent contains 1383356 objects (can be
>> determined with 'git show-index < .git/objects/pack/*.idx | wc -l').
>> However, there are only 978501 actually referenced objects in that
>> cloned repository ( 'git rev-list --all --objects | wc -l').  That makes
>> for 404855 useless objects in the cloned repository.
>
> Those objects are not useless.  They are referenced by the remote refs
> on the remote side, which are not fetched by default.  If you clone a
> mirror of the repository you'll see no unreferenced objects.
>
> Andreas.
>
> --
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>

Thanks... It is a difference between svn and git mentality probably -
one only pushes reasonably reliable code to a public git repository,
whereas anything transient is recorded in svn - I think many of the
unreferenced objects are svn user-branches (which are probably of use
to people who intend to work on gcc for fairly extended periods,
rather than casual users like me).
The case with gcc is probably quite extreme - many user branches, and
very large code base - but is there anything on the git side with git
gc which can lessen this kind of pathological behavior (expanding
packs)?

Thanks a lot for the explanation and the discussion.

Hin-Tak

Re: git gc expanding packed data?

[Andreas Schwab]

Hin-Tak Leung <hintak.leung@gmail.com> writes:

> Thanks... It is a difference between svn and git mentality probably -

It is just that the remote is a git-svn tree, with only a few branches
created as local branches.

> The case with gcc is probably quite extreme - many user branches, and
> very large code base - but is there anything on the git side with git
> gc which can lessen this kind of pathological behavior (expanding
> packs)?

If you fetch all refs, not only refs/heads/*, all objects will be
referenced.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git gc expanding packed data?

[Nicolas Pitre]

On Sat, 8 Aug 2009, Andreas Schwab wrote:

> Nicolas Pitre <nico@cam.org> writes:
> 
> > It appears that the git installation serving clone requests for
> > git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I
> > just cloned it and the pack I was sent contains 1383356 objects (can be
> > determined with 'git show-index < .git/objects/pack/*.idx | wc -l').
> > However, there are only 978501 actually referenced objects in that
> > cloned repository ( 'git rev-list --all --objects | wc -l').  That makes
> > for 404855 useless objects in the cloned repository.
> 
> Those objects are not useless.  They are referenced by the remote refs
> on the remote side, which are not fetched by default.  If you clone a
> mirror of the repository you'll see no unreferenced objects.

If you do a clone using the git:// protocol and the server sends you 
only the ref for the trunk branch, then it should send you only objects 
reachable from that branch.  Any extra objects sent by the server are 
useless to me and wastes my and everyone else's bandwidth, and on my 
next repack those objects are pruned anyway.  The point of the git 
protocol is _not_ necessarily to send a copy of the remote pack file 
over, even during a clone.


Nicolas

Re: git gc expanding packed data?

[Andreas Schwab]

Nicolas Pitre <nico@cam.org> writes:

> If you do a clone using the git:// protocol and the server sends you 
> only the ref for the trunk branch,

A clone will fetch all branches from refs/heads/*.

> then it should send you only objects reachable from that branch.

Apparantly this does not work.  I'd guess the extra objects are needed
due to the delta compression.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: git clone sending unneeded objects (was : git gc expanding packed data?)

[Jason Merrill]

On 08/09/2009 03:43 AM, Andreas Schwab wrote:
> Nicolas Pitre<nico@cam.org>  writes:
>
>> If you do a clone using the git:// protocol and the server sends you
>> only the ref for the trunk branch,
>
> A clone will fetch all branches from refs/heads/*.
>
>> then it should send you only objects reachable from that branch.
>
> Apparantly this does not work.  I'd guess the extra objects are needed
> due to the delta compression.

I just tried doing a clone of the GCC repository, then git gc 
--prune=now, and another clone specifying --reference to the first, and 
it wanted to download all the unreachable objects again.  So it doesn't 
seem to be a compression issue.

This is with git 1.6.4 on both ends.

Jason

Re: git clone sending unneeded objects

[Matthieu Moy]

Jason Merrill <jason@redhat.com> writes:

> On 08/09/2009 03:43 AM, Andreas Schwab wrote:
>> Nicolas Pitre<nico@cam.org>  writes:
>>
>>> If you do a clone using the git:// protocol and the server sends you
>>> only the ref for the trunk branch,
>>
>> A clone will fetch all branches from refs/heads/*.
>>
>>> then it should send you only objects reachable from that branch.
>>
>> Apparantly this does not work.  I'd guess the extra objects are needed
>> due to the delta compression.
>
> I just tried doing a clone of the GCC repository, then git gc
> --prune=now, and another clone specifying --reference to the first,
> and it wanted to download all the unreachable objects again.  So it
> doesn't seem to be a compression issue.
>
> This is with git 1.6.4 on both ends.

Which protocol did you use?

If you use git:// or ssh://, it's normally a security feature that Git
sends you only reachable objects. If it doesn't, it's a serious bug.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: git clone sending unneeded objects

[Jason Merrill]

On 09/25/2009 03:34 PM, Matthieu Moy wrote:
> Which protocol did you use?

git://

Jason

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Fri, 25 Sep 2009, Matthieu Moy wrote:

> Jason Merrill <jason@redhat.com> writes:
> 
> > On 08/09/2009 03:43 AM, Andreas Schwab wrote:
> >> Nicolas Pitre<nico@cam.org>  writes:
> >>
> >>> If you do a clone using the git:// protocol and the server sends you
> >>> only the ref for the trunk branch,
> >>
> >> A clone will fetch all branches from refs/heads/*.
> >>
> >>> then it should send you only objects reachable from that branch.
> >>
> >> Apparantly this does not work.  I'd guess the extra objects are needed
> >> due to the delta compression.
> >
> > I just tried doing a clone of the GCC repository, then git gc
> > --prune=now, and another clone specifying --reference to the first,
> > and it wanted to download all the unreachable objects again.  So it
> > doesn't seem to be a compression issue.
> >
> > This is with git 1.6.4 on both ends.
> 
> Which protocol did you use?
> 
> If you use git:// or ssh://, it's normally a security feature that Git
> sends you only reachable objects. If it doesn't, it's a serious bug.

I did reproduce the issue with git:// back when this discussion started. 
I also asked for more information about the remote which didn't come 
forth.


Nicolas

Re: git clone sending unneeded objects

[Jason Merrill]

On 09/25/2009 03:53 PM, Nicolas Pitre wrote:
> I did reproduce the issue with git:// back when this discussion started.
> I also asked for more information about the remote which didn't come
> forth.

Looking back, I only see you asking about the git version on the server, 
which is 1.6.4.

So again:

git clone git://gcc.gnu.org/git/gcc.git
  (1399509 objects, ~600MB .git dir)
git gc --prune=now (988906 objects, ~450MB .git dir)

...then

git clone git://gcc.gnu.org/git/gcc.git --reference $firstclone
  (573401 objects, ~550MB .git dir)
git fsck (clean)
git gc --prune=now (5 objects, ~7MB .git dir)

What's going on here?

Jason

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Fri, 25 Sep 2009, Jason Merrill wrote:

> On 09/25/2009 03:53 PM, Nicolas Pitre wrote:
> > I did reproduce the issue with git:// back when this discussion started.
> > I also asked for more information about the remote which didn't come
> > forth.
> 
> Looking back, I only see you asking about the git version on the server, which
> is 1.6.4.
> 
> So again:
> 
> git clone git://gcc.gnu.org/git/gcc.git
>  (1399509 objects, ~600MB .git dir)
> git gc --prune=now (988906 objects, ~450MB .git dir)
> 
> ...then
> 
> git clone git://gcc.gnu.org/git/gcc.git --reference $firstclone
>  (573401 objects, ~550MB .git dir)
> git fsck (clean)
> git gc --prune=now (5 objects, ~7MB .git dir)
> 
> What's going on here?

Some screw up.

Do you have access to the remote machine?  Is it possible to have a 
tarball of the gcc.git directory from there?


Nicolas

Re: git clone sending unneeded objects

[Jason Merrill]

On 09/25/2009 04:47 PM, Nicolas Pitre wrote:
> Do you have access to the remote machine?  Is it possible to have a
> tarball of the gcc.git directory from there?

http://gcc.gnu.org/gcc-git.tar.gz

I'll leave it there for a few days.

Jason

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Fri, 25 Sep 2009, Jason Merrill wrote:

> On 09/25/2009 04:47 PM, Nicolas Pitre wrote:
> > Do you have access to the remote machine?  Is it possible to have a
> > tarball of the gcc.git directory from there?
> 
> http://gcc.gnu.org/gcc-git.tar.gz
> 
> I'll leave it there for a few days.

Thanks, I got it now.  And I was able to reproduce the issue locally.

Cloning the original repository does transfer objects which become 
unreferenced in the clone.  But cloning that cloned repository (before 
pruning the unreferenced objects) does not transfer those objects again.  

Just need to find out why.


Nicolas

[PATCH] make 'git clone' ask the remote only for objects it cares about

[Nicolas Pitre]

Current behavior of 'git clone' when not using --mirror is to fetch 
everything from the peer, and then filter out unwanted refs just before 
writing them out to the cloned repository.  This may become highly 
inefficient if the peer has an unusual ref namespace, or if it simply 
has "remotes" refs of its own, and those locally unwanted refs are 
connecting to a large set of objects which becomes unreferenced as soon 
as they are fetched.

Let's filter out those unwanted refs from the peer _before_ asking it 
what refs we want to fetch instead, which is the most logical thing to 
do anyway.

Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
---

On Fri, 25 Sep 2009, Nicolas Pitre wrote:

> On Fri, 25 Sep 2009, Jason Merrill wrote:
> 
> > On 09/25/2009 04:47 PM, Nicolas Pitre wrote:
> > > Do you have access to the remote machine?  Is it possible to have a
> > > tarball of the gcc.git directory from there?
> > 
> > http://gcc.gnu.org/gcc-git.tar.gz
> > 
> > I'll leave it there for a few days.
> 
> Thanks, I got it now.  And I was able to reproduce the issue locally.
> 
> Cloning the original repository does transfer objects which become 
> unreferenced in the clone.  But cloning that cloned repository (before 
> pruning the unreferenced objects) does not transfer those objects again.  
> 
> Just need to find out why.

And the "why" is described above.  The problem was actually on the 
client side and was affecting clones of any repository containing 
anything outside refs/heads and refs/tags.

The fact that the git repository on gcc.gnu.org has lots of stuff in 
"remote" branches that don't get cloned by default is a separate 
configuration/policy issue on that server which might need (or not) to 
be looked into.  For instance at least, as a bare repository, it should 
have all the git files in gcc.git/ directly instead of gcc.git/.git/.

diff --git a/builtin-clone.c b/builtin-clone.c
index bab2d84..edf7c7f 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -329,24 +329,28 @@ static void remove_junk_on_signal(int signo)
 	raise(signo);
 }
 
-static struct ref *write_remote_refs(const struct ref *refs,
-		struct refspec *refspec, const char *reflog)
+static struct ref *wanted_peer_refs(const struct ref *refs,
+		struct refspec *refspec)
 {
 	struct ref *local_refs = NULL;
 	struct ref **tail = &local_refs;
-	struct ref *r;
 
 	get_fetch_map(refs, refspec, &tail, 0);
 	if (!option_mirror)
 		get_fetch_map(refs, tag_refspec, &tail, 0);
 
+	return local_refs;
+}
+
+static void write_remote_refs(const struct ref *local_refs, const char *reflog)
+{
+	const struct ref *r;
+
 	for (r = local_refs; r; r = r->next)
 		add_extra_ref(r->peer_ref->name, r->old_sha1, 0);
 
 	pack_refs(PACK_REFS_ALL);
 	clear_extra_refs();
-
-	return local_refs;
 }
 
 int cmd_clone(int argc, const char **argv, const char *prefix)
@@ -495,9 +499,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 	strbuf_reset(&value);
 
-	if (path && !is_bundle)
+	if (path && !is_bundle) {
 		refs = clone_local(path, git_dir);
-	else {
+		mapped_refs = wanted_peer_refs(refs, refspec);
+	} else {
 		struct remote *remote = remote_get(argv[0]);
 		transport = transport_get(remote, remote->url[0]);
 
@@ -520,14 +525,16 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 					     option_upload_pack);
 
 		refs = transport_get_remote_refs(transport);
-		if (refs)
-			transport_fetch_refs(transport, refs);
+		if (refs) {
+			mapped_refs = wanted_peer_refs(refs, refspec);
+			transport_fetch_refs(transport, mapped_refs);
+		}
 	}
 
 	if (refs) {
 		clear_extra_refs();
 
-		mapped_refs = write_remote_refs(refs, refspec, reflog_msg.buf);
+		write_remote_refs(mapped_refs, reflog_msg.buf);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =

Re: [PATCH] make 'git clone' ask the remote only for objects it cares about

[Andreas Schwab]

Nicolas Pitre <nico@fluxnic.net> writes:

> The fact that the git repository on gcc.gnu.org has lots of stuff in 
> "remote" branches that don't get cloned by default is a separate 
> configuration/policy issue on that server which might need (or not) to 
> be looked into.  For instance at least, as a bare repository, it should 
> have all the git files in gcc.git/ directly instead of gcc.git/.git/.

The remote is just a git-svn tree.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: [PATCH] make 'git clone' ask the remote only for objects it cares about

[Shawn O. Pearce]

Nicolas Pitre <nico@fluxnic.net> wrote:
> Current behavior of 'git clone' when not using --mirror is to fetch 
> everything from the peer, and then filter out unwanted refs just before 
> writing them out to the cloned repository.  This may become highly 
> inefficient if the peer has an unusual ref namespace, or if it simply 
> has "remotes" refs of its own, and those locally unwanted refs are 
> connecting to a large set of objects which becomes unreferenced as soon 
> as they are fetched.
...
> +static void write_remote_refs(const struct ref *local_refs, const char *reflog)

Here reflog is now unused.  I'm going to squash this in.

diff --git a/builtin-clone.c b/builtin-clone.c
index edf7c7f..4992c25 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -342,7 +342,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	return local_refs;
 }
 
-static void write_remote_refs(const struct ref *local_refs, const char *reflog)
+static void write_remote_refs(const struct ref *local_refs)
 {
 	const struct ref *r;
 
@@ -534,7 +534,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	if (refs) {
 		clear_extra_refs();
 
-		write_remote_refs(mapped_refs, reflog_msg.buf);
+		write_remote_refs(mapped_refs);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =

-- 
Shawn.

Re: [PATCH] make 'git clone' ask the remote only for objects it cares about

[Nicolas Pitre]

On Sat, 26 Sep 2009, Shawn O. Pearce wrote:

> Nicolas Pitre <nico@fluxnic.net> wrote:
> > Current behavior of 'git clone' when not using --mirror is to fetch 
> > everything from the peer, and then filter out unwanted refs just before 
> > writing them out to the cloned repository.  This may become highly 
> > inefficient if the peer has an unusual ref namespace, or if it simply 
> > has "remotes" refs of its own, and those locally unwanted refs are 
> > connecting to a large set of objects which becomes unreferenced as soon 
> > as they are fetched.
> ...
> > +static void write_remote_refs(const struct ref *local_refs, const char *reflog)
> 
> Here reflog is now unused.  I'm going to squash this in.

Yeah, I noticed.  Since I didn't know what was the original intent for 
it, I just left it there.


Nicolas

Re: git clone sending unneeded objects

[Jason Merrill]

Incidentally, somewhat related to this issue, I've noticed that if I 
fetch a branch which I don't currently have in my repository, and I have 
most of the commits on that branch in my object store (or in an 
alternate repository) but not the most recent commit, git fetch isn't 
smart enough to only grab the commits I'm actually missing, it wants to 
fetch much more.

I would expect that since the clone pulled down everything in the 
gcc.git repository, I could then do

git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'
git fetch

and have all the branches, not just the ones in refs/heads.  But when I 
do this git fetch wants to fetch some 500k redundant objects.

Jason

Re: git clone sending unneeded objects

[Jason Merrill]

On 09/26/2009 12:44 AM, Jason Merrill wrote:
> git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'
> git fetch

git count-objects -v before:

count: 44
size: 1768
in-pack: 1399509
packs: 1
size-pack: 600456
prune-packable: 0
garbage: 0

and after (transferred 278MB):

count: 44
size: 1768
in-pack: 1947339
packs: 2
size-pack: 1178408
prune-packable: 8
garbage: 0

and then after git gc --prune=now:

count: 0
size: 0
in-pack: 1399613
packs: 1
size-pack: 839900
prune-packable: 0
garbage: 0

So I only actually needed 104 more objects, but fetch wasn't clever 
enough to see that, and my new pack is much less efficient.

I've run into the same issue using alternates to set up multiple working 
directories for different branches; if the alternate directory isn't 
completely up-to-date, fetch wants to pull down lots of data again 
rather than use what I have and only fetch the last one or two commits.

Jason

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Sat, 26 Sep 2009, Jason Merrill wrote:

> On 09/26/2009 12:44 AM, Jason Merrill wrote:
> > git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'
> > git fetch
> 
> git count-objects -v before:
> 
> count: 44
> size: 1768
> in-pack: 1399509
> packs: 1
> size-pack: 600456
> prune-packable: 0
> garbage: 0

I'm sure if you had done 'git rev-list --all --objects | wc -l' at that 
point, the result would have been something around 900000.  That's the 
actual number of objects git had a reference to, compared to the total 
objects contained in the object store.

> and after (transferred 278MB):
> 
> count: 44
> size: 1768
> in-pack: 1947339
> packs: 2
> size-pack: 1178408
> prune-packable: 8
> garbage: 0

And those 500000 extra objects or so (minus a couple dozens which were 
probably used to "complete" the fetched thin pack and are duplicates of 
local objects -- the fetch progress message gave the exact number) were 
obtained from the remote repository because git has no way to tell the 
remote it already had them.  That's what I was explaining in my previous 
email.

> and then after git gc --prune=now:
> 
> count: 0
> size: 0
> in-pack: 1399613
> packs: 1
> size-pack: 839900
> prune-packable: 0
> garbage: 0
> 
> So I only actually needed 104 more objects, but fetch wasn't clever enough to
> see that, and my new pack is much less efficient.

Like I said, it's not that the fetch wasn't clever enough.  Rather that 
your initial clone asked for way too many objects in the first place.  
That's what my patch fixed.

Now the pack efficiency can be explained as well.  A single pack is 
always going to be more efficient than 2 packs.  Problem is when you do 
a gc, by default git does the least costly operation which consists of 
copying as much data from existing packs without extra processing.  
That means that many objects were copied from the second (newly 
received) pack although a better delta representation was most probably 
available in the other larger pack (remember that most objects from that 
second pack already existed in the first pack).  Git do select the 
second pack in preference to the other pack because it is more recent, 
and normally more recent packs contains more recent objects which is a 
good heuristic to optimizes the object enumeration.  In this case this 
didn't produce a good result, but again we're talking about a scenario 
which is bogus from the start and shouldn't be.

So if you do a 'git gc --aggressive' and let it run for a while, you 
should get back a smaller pack, possibly even much smaller than the 
original 
one.


Nicolas

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Sat, 26 Sep 2009, Jason Merrill wrote:

> Incidentally, somewhat related to this issue, I've noticed that if I fetch a
> branch which I don't currently have in my repository, and I have most of the
> commits on that branch in my object store (or in an alternate repository) but
> not the most recent commit, git fetch isn't smart enough to only grab the
> commits I'm actually missing, it wants to fetch much more.
> 
> I would expect that since the clone pulled down everything in the gcc.git
> repository, I could then do
> 
> git config remote.origin.fetch 'refs/remotes/*:refs/remotes/origin/*'
> git fetch
> 
> and have all the branches, not just the ones in refs/heads.  But when I do
> this git fetch wants to fetch some 500k redundant objects.

Well...  Assuming a fixed git using the patch I posted yesterday, my 
clone of gcc.git has 988941 objects.  The source repository used for the 
clone has 1399551 objects.  Of course the source repo has more objects 
because it has extra branches in the refs/remotes/ namespace that the 
clone didn't fetch.  If you wish to also fetch those branches as you 
illustrated above then you'll get the difference i.e. 410610 additional 
objects.

And even if the broken clone (before my patch) did pull everything from 
gcc.git, in the cloned repository those 410610 extra objects are 
considered as garbage because nothing actually reference them.  So even 
if you decide to fetch the extra branches that the initial clone didn't 
pick up, or if you do reference that repository with "garbage" objects 
for another clone to which you want to add those extra branches, git has 
no way to know that it already had access to those objects locally and 
"ungarbage" them as they aren't referenced.  Result is a useless fetch 
of 410610 objects that you already have, but that you weren't supposed 
to have in the first place.


Nicolas

Re: git clone sending unneeded objects

[Shawn O. Pearce]

Nicolas Pitre <nico@fluxnic.net> wrote:
> And even if the broken clone (before my patch) did pull everything from 
> gcc.git, in the cloned repository those 410610 extra objects are 
> considered as garbage because nothing actually reference them.  So even 
> if you decide to fetch the extra branches that the initial clone didn't 
> pick up, or if you do reference that repository with "garbage" objects 
> for another clone to which you want to add those extra branches, git has 
> no way to know that it already had access to those objects locally and 
> "ungarbage" them as they aren't referenced.  Result is a useless fetch 
> of 410610 objects that you already have, but that you weren't supposed 
> to have in the first place.

Just to clarify a minor nit:

Actually, if those refs have not changed, quickfetch should kick in
and realize that all 410610 objects are reachable locally without
errors, permitting the client to avoid the object transfer.

However, if *ANY* of those refs were to change to something you
don't actually have, quickfetch would fail, and we would need to
fetch all 410610 objects.

-- 
Shawn.

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Sat, 26 Sep 2009, Shawn O. Pearce wrote:

> Nicolas Pitre <nico@fluxnic.net> wrote:
> > And even if the broken clone (before my patch) did pull everything from 
> > gcc.git, in the cloned repository those 410610 extra objects are 
> > considered as garbage because nothing actually reference them.  So even 
> > if you decide to fetch the extra branches that the initial clone didn't 
> > pick up, or if you do reference that repository with "garbage" objects 
> > for another clone to which you want to add those extra branches, git has 
> > no way to know that it already had access to those objects locally and 
> > "ungarbage" them as they aren't referenced.  Result is a useless fetch 
> > of 410610 objects that you already have, but that you weren't supposed 
> > to have in the first place.
> 
> Just to clarify a minor nit:
> 
> Actually, if those refs have not changed, quickfetch should kick in
> and realize that all 410610 objects are reachable locally without
> errors, permitting the client to avoid the object transfer.
> 
> However, if *ANY* of those refs were to change to something you
> don't actually have, quickfetch would fail, and we would need to
> fetch all 410610 objects.

Right.  But since we're talking about a git mirror for the gcc svn repo 
and gcc is a rather active project, the likelyhood of any ref to change 
at any time is rather high.


Nicolas

Re: git clone sending unneeded objects

[Jason Merrill]

On 09/26/2009 10:04 PM, Shawn O. Pearce wrote:
> Actually, if those refs have not changed, quickfetch should kick in
> and realize that all 410610 objects are reachable locally without
> errors, permitting the client to avoid the object transfer.
>
> However, if *ANY* of those refs were to change to something you
> don't actually have, quickfetch would fail, and we would need to
> fetch all 410610 objects.

Right.  That seems unfortunate to me; couldn't fetch do a bit more 
checking before it decides to download the whole world again?

Jason

Re: git clone sending unneeded objects

[Nicolas Pitre]

On Sun, 27 Sep 2009, Jason Merrill wrote:

> On 09/26/2009 10:04 PM, Shawn O. Pearce wrote:
> > Actually, if those refs have not changed, quickfetch should kick in
> > and realize that all 410610 objects are reachable locally without
> > errors, permitting the client to avoid the object transfer.
> > 
> > However, if *ANY* of those refs were to change to something you
> > don't actually have, quickfetch would fail, and we would need to
> > fetch all 410610 objects.
> 
> Right.  That seems unfortunate to me; couldn't fetch do a bit more checking
> before it decides to download the whole world again?

The quickfetch test could be turned into a filter so refs that are 
already available locally could simply not be fetched on a per ref 
basis.  But that would be a rather expensive test which couldn't keep 
its "quick" qualifier anymore, and so for a case that shouldn't have 
happened normally anyway if git didn't have a bug with its clone 
operation as I've explained already.


Nicolas

Re: git clone sending unneeded objects

[Hin-Tak Leung]

On Fri, Sep 25, 2009 at 9:20 PM, Jason Merrill <jason@redhat.com> wrote:
> On 09/25/2009 03:53 PM, Nicolas Pitre wrote:
>>
>> I did reproduce the issue with git:// back when this discussion started.
>> I also asked for more information about the remote which didn't come
>> forth.
>
> Looking back, I only see you asking about the git version on the server,
> which is 1.6.4.

Hmm, I was under the impression from the previous thread that the
server is a bit older and/or have more backward compatible settings to
cater for older git clients?

>
> So again:
>
> git clone git://gcc.gnu.org/git/gcc.git
>  (1399509 objects, ~600MB .git dir)
> git gc --prune=now (988906 objects, ~450MB .git dir)
>
> ...then
>
> git clone git://gcc.gnu.org/git/gcc.git --reference $firstclone
>  (573401 objects, ~550MB .git dir)
> git fsck (clean)
> git gc --prune=now (5 objects, ~7MB .git dir)
>
> What's going on here?

FWIW, I still have my clone (git://) and do my periodic 'git fetch'
and 'git gc prune=now' (learned my lessons!) and it is currently .git
dir is about 350MB. (from previous discussion the optimal at the time
was about 300MB, so it has grown a bit in the last couple of months).

And thanks everybody for all the discussion and advice. git is a great
tool. (and I have essentially stopped using svn, prefering git-svn!).

git gc expanding packed data?

[Hin-Tak Leung]

I cloned gcc's git about a week ago to work on some problems I have
with gcc on minor platforms, just plain 'git clone
git://gcc.gnu.org/git/gcc.git gcc' .and ran gcc fetch about daily, and
'git rebase origin' from time to time. I don't have local changes,
just following and monitoring what's going on in gcc. So after a week,
I thought I'd do a git gc . Then it goes very bizarre.

Before I start 'git gc', .The whole of .git was about 700MB and
git/objects/pack was a bit under 600MB, with a few other directories
under .git/objects at 10's of K's and a few 30000-40000K's, and the
checkout was, well, the size of gcc source code. But after I started
git gc, the message stays in the 'counting objects' at about 900,000
for a long time, while a lot of directories under .git/objects/ gets a
bit large, and .git blows up to at least 7GB with a lot of small files
under .git/objects/*/, before seeing as I will run out of disk space,
I kill the whole lot and ran git clone again, since I don't have any
local change and there is nothing to lose.

I am running git version 1.6.2.5 (fedora 11). Is there any reason why
'git gc' does that?

Re: git gc expanding packed data?

[Nicolas Pitre]

On Tue, 4 Aug 2009, Hin-Tak Leung wrote:

> I cloned gcc's git about a week ago to work on some problems I have
> with gcc on minor platforms, just plain 'git clone
> git://gcc.gnu.org/git/gcc.git gcc' .and ran gcc fetch about daily, and
> 'git rebase origin' from time to time. I don't have local changes,
> just following and monitoring what's going on in gcc. So after a week,
> I thought I'd do a git gc . Then it goes very bizarre.
> 
> Before I start 'git gc', .The whole of .git was about 700MB and
> git/objects/pack was a bit under 600MB, with a few other directories
> under .git/objects at 10's of K's and a few 30000-40000K's, and the
> checkout was, well, the size of gcc source code. But after I started
> git gc, the message stays in the 'counting objects' at about 900,000
> for a long time, while a lot of directories under .git/objects/ gets a
> bit large, and .git blows up to at least 7GB with a lot of small files
> under .git/objects/*/, before seeing as I will run out of disk space,
> I kill the whole lot and ran git clone again, since I don't have any
> local change and there is nothing to lose.
> 
> I am running git version 1.6.2.5 (fedora 11). Is there any reason why
> 'git gc' does that?

There is probably a reason, although a bad one for sure.

Well... OK.

It appears that the git installation serving clone requests for 
git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I 
just cloned it and the pack I was sent contains 1383356 objects (can be 
determined with 'git show-index < .git/objects/pack/*.idx | wc -l').  
However, there are only 978501 actually referenced objects in that 
cloned repository ( 'git rev-list --all --objects | wc -l').  That makes 
for 404855 useless objects in the cloned repository.

Now git has a safety mechanism to _not_ delete unreferenced objects 
right away when running 'git gc'.  By default unreferenced objects are 
kept around for a period of 2 weeks.  This is to make it easy for you to 
recover accidentally deleted branches or commits, or to avoid a race 
where a just-created object in the process of being but not yet 
referenced could be deleted by a 'git gc' process running in parallel.

So to give that grace period to packed but unreferenced objects, the 
repack process pushes those unreferenced objects out of the pack into 
their loose form so they can be aged and eventually pruned.  Objects 
becoming unreferenced are usually not that many though.  Having 404855 
unreferenced objects is quite a lot, and being sent those objects in the 
first place via a clone is stupid and a complete waste of network 
bandwidth.

Anyone has an idea of the git version running on gcc.gnu.org?  It is 
certainly buggy and needs fixing.

Anyway... To solve your problem, you simply need to run 'git gc' with 
the --prune=now argument to disable that grace period and get rid of 
those unreferenced objects right away (safe only if no other git 
activities are taking place at the same time which should be easy to 
ensure on a workstation).  The resulting .git/objects directory size 
will shrink to about 441 MB.  If the gcc.gnu.org git server was doing 
its job properly, the size of the clone transfer would also be 
significantly smaller, meaning around 414 MB instead of the current 600+ 
MB.

And BTW, using 'git gc --aggressive' with a later git version (or
'git repack -a -f -d --window=250 --depth=250') gives me a .git/objects 
directory size of 310 MB, meaning that the actual repository with all 
the trunk history is _smaller_ than the actual source checkout.  If that 
repository was properly repacked on the server, the clone data transfer 
would be 283 MB.  This is less than half the current clone transfer 
size.


Nicolas

Re: git gc expanding packed data?

[Hin-Tak Leung]

On Wed, Aug 5, 2009 at 11:39 PM, Nicolas Pitre<nico@cam.org> wrote:
> On Tue, 4 Aug 2009, Hin-Tak Leung wrote:
>
>> I cloned gcc's git about a week ago to work on some problems I have
>> with gcc on minor platforms, just plain 'git clone
>> git://gcc.gnu.org/git/gcc.git gcc' .and ran gcc fetch about daily, and
>> 'git rebase origin' from time to time. I don't have local changes,
>> just following and monitoring what's going on in gcc. So after a week,
>> I thought I'd do a git gc . Then it goes very bizarre.
>>
>> Before I start 'git gc', .The whole of .git was about 700MB and
>> git/objects/pack was a bit under 600MB, with a few other directories
>> under .git/objects at 10's of K's and a few 30000-40000K's, and the
>> checkout was, well, the size of gcc source code. But after I started
>> git gc, the message stays in the 'counting objects' at about 900,000
>> for a long time, while a lot of directories under .git/objects/ gets a
>> bit large, and .git blows up to at least 7GB with a lot of small files
>> under .git/objects/*/, before seeing as I will run out of disk space,
>> I kill the whole lot and ran git clone again, since I don't have any
>> local change and there is nothing to lose.
>>
>> I am running git version 1.6.2.5 (fedora 11). Is there any reason why
>> 'git gc' does that?
>
> There is probably a reason, although a bad one for sure.
>
> Well... OK.
>
> It appears that the git installation serving clone requests for
> git://gcc.gnu.org/git/gcc.git generates lots of unreferenced objects. I
> just cloned it and the pack I was sent contains 1383356 objects (can be
> determined with 'git show-index < .git/objects/pack/*.idx | wc -l').
> However, there are only 978501 actually referenced objects in that
> cloned repository ( 'git rev-list --all --objects | wc -l').  That makes
> for 404855 useless objects in the cloned repository.
>
> Now git has a safety mechanism to _not_ delete unreferenced objects
> right away when running 'git gc'.  By default unreferenced objects are
> kept around for a period of 2 weeks.  This is to make it easy for you to
> recover accidentally deleted branches or commits, or to avoid a race
> where a just-created object in the process of being but not yet
> referenced could be deleted by a 'git gc' process running in parallel.
>
> So to give that grace period to packed but unreferenced objects, the
> repack process pushes those unreferenced objects out of the pack into
> their loose form so they can be aged and eventually pruned.  Objects
> becoming unreferenced are usually not that many though.  Having 404855
> unreferenced objects is quite a lot, and being sent those objects in the
> first place via a clone is stupid and a complete waste of network
> bandwidth.
>
> Anyone has an idea of the git version running on gcc.gnu.org?  It is
> certainly buggy and needs fixing.
>
> Anyway... To solve your problem, you simply need to run 'git gc' with
> the --prune=now argument to disable that grace period and get rid of
> those unreferenced objects right away (safe only if no other git
> activities are taking place at the same time which should be easy to
> ensure on a workstation).  The resulting .git/objects directory size
> will shrink to about 441 MB.  If the gcc.gnu.org git server was doing
> its job properly, the size of the clone transfer would also be
> significantly smaller, meaning around 414 MB instead of the current 600+
> MB.
>
> And BTW, using 'git gc --aggressive' with a later git version (or
> 'git repack -a -f -d --window=250 --depth=250') gives me a .git/objects
> directory size of 310 MB, meaning that the actual repository with all
> the trunk history is _smaller_ than the actual source checkout.  If that
> repository was properly repacked on the server, the clone data transfer
> would be 283 MB.  This is less than half the current clone transfer
> size.
>
>
> Nicolas
>

'git gc --prune=now' does work, but 'git gc --prune=now --aggressive'
(before) and 'git gc --aggressive' (after) both create very large
(>2GB; I stopped it) packs from the ~400MB-600MB packed objects. I
noted that you specifically wrote 'with a later git version' -
presumably there is a some sort of a known and fixed issue there? Just
curious.

I guess --aggressive doesn't always save space...

Hin-Tak

Re: git gc expanding packed data?

[Nicolas Pitre]

On Tue, 11 Aug 2009, Hin-Tak Leung wrote:

> On Wed, Aug 5, 2009 at 11:39 PM, Nicolas Pitre<nico@cam.org> wrote:
> > Anyway... To solve your problem, you simply need to run 'git gc' with
> > the --prune=now argument to disable that grace period and get rid of
> > those unreferenced objects right away (safe only if no other git
> > activities are taking place at the same time which should be easy to
> > ensure on a workstation).  The resulting .git/objects directory size
> > will shrink to about 441 MB.  If the gcc.gnu.org git server was doing
> > its job properly, the size of the clone transfer would also be
> > significantly smaller, meaning around 414 MB instead of the current 600+
> > MB.
> >
> > And BTW, using 'git gc --aggressive' with a later git version (or
> > 'git repack -a -f -d --window=250 --depth=250') gives me a .git/objects
> > directory size of 310 MB, meaning that the actual repository with all
> > the trunk history is _smaller_ than the actual source checkout.  If that
> > repository was properly repacked on the server, the clone data transfer
> > would be 283 MB.  This is less than half the current clone transfer
> > size.
> >
> >
> > Nicolas
> >
> 
> 'git gc --prune=now' does work, but 'git gc --prune=now --aggressive'
> (before) and 'git gc --aggressive' (after) both create very large
> (>2GB; I stopped it) packs from the ~400MB-600MB packed objects. I
> noted that you specifically wrote 'with a later git version' -
> presumably there is a some sort of a known and fixed issue there? Just
> curious.

>From git v1.6.3 the --aggressive switch makes for 'git repack' to be 
called with --window=250 --depth=250, meaning the equivalent of:

	git repack -a -d -f --window=250 --depth=250

Do you still get a huge pack with the above?

> I guess --aggressive doesn't always save space...

If so that is (and was) a bug.


Nicolas

Re: git gc expanding packed data?

[Hin-Tak Leung]

On Tue, Aug 11, 2009 at 10:33 PM, Nicolas Pitre<nico@cam.org> wrote:
<snipped>

> From git v1.6.3 the --aggressive switch makes for 'git repack' to be
> called with --window=250 --depth=250, meaning the equivalent of:
>
>        git repack -a -d -f --window=250 --depth=250
>
> Do you still get a huge pack with the above?
>
>> I guess --aggressive doesn't always save space...
>
> If so that is (and was) a bug.

I tried 'git repack -a -d -f --window=250 --depth=250' with 1.6.2.5
(fc11.x86_64) and it took half a day, swallowed up all the memory -
3GB virtual & 1.3GB resident - and finally the kernel oom killer
killed it at a last message of (601460/957910). Left no temp files.
Would git 1.6.3 use less memory? :-(

Hin-Tak

Re: git gc expanding packed data?

[Nicolas Pitre]

On Wed, 12 Aug 2009, Hin-Tak Leung wrote:

> On Tue, Aug 11, 2009 at 10:33 PM, Nicolas Pitre<nico@cam.org> wrote:
> <snipped>
> 
> > From git v1.6.3 the --aggressive switch makes for 'git repack' to be
> > called with --window=250 --depth=250, meaning the equivalent of:
> >
> >        git repack -a -d -f --window=250 --depth=250
> >
> > Do you still get a huge pack with the above?
> >
> >> I guess --aggressive doesn't always save space...
> >
> > If so that is (and was) a bug.
> 
> I tried 'git repack -a -d -f --window=250 --depth=250' with 1.6.2.5
> (fc11.x86_64) and it took half a day, swallowed up all the memory -
> 3GB virtual & 1.3GB resident - and finally the kernel oom killer
> killed it at a last message of (601460/957910). Left no temp files.
> Would git 1.6.3 use less memory? :-(

Probably not.  However you should try:

	git config pack.deltaCacheSize 1

That limits the delta cache size to one byte (effectively disabling it) 
instead of the default of 0 which means unlimited.  With that I'm able 
to repack that repository using the above git repack command on an 
x86-64 system with 4GB of RAM and using 4 threads (this is a quad core).  
Resident memory usage grows to nearly 3.3GB though.

If your machine is SMP and you don't have sufficient RAM then you can 
reduce the number of threads to only one:

	git config pack.threads 1

Additionally, you can further limit memory usage with the 
--window-memory argument to 'git repack'.  For example, using 
--window-memory=128M should keep a reasonable upper bound on the delta 
search memory usage although this can result in less optimal delta match 
if the repo contains lots of large files (and I think this is the case 
for the gcc repo).


Nicolas

Re: git gc expanding packed data?

[Hin-Tak Leung]

On Wed, Aug 12, 2009 at 4:35 PM, Nicolas Pitre<nico@cam.org> wrote:
> On Wed, 12 Aug 2009, Hin-Tak Leung wrote:
>
>> On Tue, Aug 11, 2009 at 10:33 PM, Nicolas Pitre<nico@cam.org> wrote:
>> <snipped>
>>
>> > From git v1.6.3 the --aggressive switch makes for 'git repack' to be
>> > called with --window=250 --depth=250, meaning the equivalent of:
>> >
>> >        git repack -a -d -f --window=250 --depth=250
>> >
>> > Do you still get a huge pack with the above?
>> >
>> >> I guess --aggressive doesn't always save space...
>> >
>> > If so that is (and was) a bug.
>>
>> I tried 'git repack -a -d -f --window=250 --depth=250' with 1.6.2.5
>> (fc11.x86_64) and it took half a day, swallowed up all the memory -
>> 3GB virtual & 1.3GB resident - and finally the kernel oom killer
>> killed it at a last message of (601460/957910). Left no temp files.
>> Would git 1.6.3 use less memory? :-(
>
> Probably not.  However you should try:
>
>        git config pack.deltaCacheSize 1
>
> That limits the delta cache size to one byte (effectively disabling it)
> instead of the default of 0 which means unlimited.  With that I'm able
> to repack that repository using the above git repack command on an
> x86-64 system with 4GB of RAM and using 4 threads (this is a quad core).
> Resident memory usage grows to nearly 3.3GB though.
>
> If your machine is SMP and you don't have sufficient RAM then you can
> reduce the number of threads to only one:
>
>        git config pack.threads 1
>
> Additionally, you can further limit memory usage with the
> --window-memory argument to 'git repack'.  For example, using
> --window-memory=128M should keep a reasonable upper bound on the delta
> search memory usage although this can result in less optimal delta match
> if the repo contains lots of large files (and I think this is the case
> for the gcc repo).
>
>
> Nicolas
>

Thanks.  I used the two git config pack.* commands, and
   git repack -a -d -f --window=250 --depth=250
finished after 8 hours (dual core Turion, 2GB RAM + 2GB swap). The
pack directory went from 457MB to 308MB.

Thanks a lot for the advice - learned a few interesting things about
git on the way :-).

Hin-Tak