[JGIT PATCH 0/5] Add jgit init, clone, receive-pack; transport fixes

[JGIT PATCH 0/5] Add jgit init, clone, receive-pack; transport fixes

[Shawn O. Pearce]

Define a "jgit init", "jgit clone" commands from the command line.

Add "jgit receive-pack".  This now works from C git:

  git push --receive-pack='jgit receive-pack' dst.git

Fix issue #22, causing jgit to fetch too many objects during an
initial clone/fetch of a repository.

Change "jgit daemon" so it can run outside of a repository, making
it slightly more useful.


Shawn O. Pearce (5):
  Add "jgit receive-pack" and permit commands to start not in a
    repository
  Add "jgit init" command to create a new repository
  Modify "jgit daemon" so it can run outside of a repository
  Add "jgit clone" to support cloning off URLs that are JGit specific
  Fix "fetch pulled too many objects" when auto-following tags

 .../services/org.spearce.jgit.pgm.TextBuiltin      |    3 +
 .../org/spearce/jgit/pgm/AbstractFetchCommand.java |  126 ++++++++++++++
 .../src/org/spearce/jgit/pgm/Clone.java            |  177 ++++++++++++++++++++
 .../src/org/spearce/jgit/pgm/Daemon.java           |   33 +---
 .../src/org/spearce/jgit/pgm/Fetch.java            |   83 +---------
 .../src/org/spearce/jgit/pgm/Init.java             |   60 +++++++
 .../src/org/spearce/jgit/pgm/Main.java             |   19 ++-
 .../src/org/spearce/jgit/pgm/ReceivePack.java      |   67 ++++++++
 .../src/org/spearce/jgit/pgm/TextBuiltin.java      |   24 +++-
 .../jgit/transport/BaseFetchConnection.java        |   26 ++-
 .../jgit/transport/BasePackFetchConnection.java    |   25 +++-
 .../spearce/jgit/transport/FetchConnection.java    |   21 ++-
 .../org/spearce/jgit/transport/FetchProcess.java   |    6 +-
 .../spearce/jgit/transport/TransportBundle.java    |    3 +-
 .../jgit/transport/WalkFetchConnection.java        |   14 ++-
 15 files changed, 544 insertions(+), 143 deletions(-)
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java

[JGIT PATCH 1/5] Add "jgit receive-pack" and permit commands to start not in a repository

[Shawn O. Pearce]

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../services/org.spearce.jgit.pgm.TextBuiltin      |    1 +
 .../src/org/spearce/jgit/pgm/Main.java             |   19 +++---
 .../src/org/spearce/jgit/pgm/ReceivePack.java      |   67 ++++++++++++++++++++
 .../src/org/spearce/jgit/pgm/TextBuiltin.java      |   24 ++++++-
 4 files changed, 99 insertions(+), 12 deletions(-)
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java

diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
index 5fb0953..0e7e7d8 100644
--- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
+++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
@@ -9,6 +9,7 @@ org.spearce.jgit.pgm.LsRemote
 org.spearce.jgit.pgm.LsTree
 org.spearce.jgit.pgm.MergeBase
 org.spearce.jgit.pgm.Push
+org.spearce.jgit.pgm.ReceivePack
 org.spearce.jgit.pgm.RevList
 org.spearce.jgit.pgm.Rm
 org.spearce.jgit.pgm.ShowRev
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java
index c8bade8..8de4ae9 100644
--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java
@@ -143,15 +143,18 @@ private void execute(final String[] argv) throws Exception {
 			System.exit(1);
 		}
 
-		if (gitdir == null)
-			gitdir = findGitDir();
-		if (gitdir == null || !gitdir.isDirectory()) {
-			System.err.println("error: can't find git directory");
-			System.exit(1);
-		}
-
 		final TextBuiltin cmd = subcommand;
-		cmd.init(new Repository(gitdir));
+		if (cmd.requiresRepository()) {
+			if (gitdir == null)
+				gitdir = findGitDir();
+			if (gitdir == null || !gitdir.isDirectory()) {
+				System.err.println("error: can't find git directory");
+				System.exit(1);
+			}
+			cmd.init(new Repository(gitdir), gitdir);
+		} else {
+			cmd.init(null, gitdir);
+		}
 		try {
 			cmd.execute(arguments.toArray(new String[arguments.size()]));
 		} finally {
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java
new file mode 100644
index 0000000..579f893
--- /dev/null
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java
@@ -0,0 +1,67 @@
+/*
+ * Copyright (C) 2008, Google Inc.
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.pgm;
+
+import java.io.File;
+
+import org.kohsuke.args4j.Argument;
+import org.spearce.jgit.lib.Repository;
+
+@Command(common = false, usage = "Server side backend for 'jgit push'")
+class ReceivePack extends TextBuiltin {
+	@Argument(index = 0, required = true, metaVar = "DIRECTORY", usage = "Repository to receive into")
+	File gitdir;
+
+	@Override
+	protected final boolean requiresRepository() {
+		return false;
+	}
+
+	@Override
+	protected void run() throws Exception {
+		final org.spearce.jgit.transport.ReceivePack rp;
+
+		if (new File(gitdir, ".git").isDirectory())
+			gitdir = new File(gitdir, ".git");
+		db = new Repository(gitdir);
+		if (!db.getObjectsDirectory().isDirectory())
+			throw die("'" + gitdir.getPath() + "' not a git repository");
+		rp = new org.spearce.jgit.transport.ReceivePack(db);
+		rp.receive(System.in, System.out, System.err);
+	}
+}
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java
index 1f3a136..d3e32b3 100644
--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java
@@ -43,6 +43,7 @@
 import static org.spearce.jgit.lib.Constants.R_TAGS;
 
 import java.io.BufferedWriter;
+import java.io.File;
 import java.io.IOException;
 import java.io.OutputStreamWriter;
 import java.io.PrintWriter;
@@ -77,6 +78,9 @@
 	/** Git repository the command was invoked within. */
 	protected Repository db;
 
+	/** Directory supplied via --git-dir command line option. */
+	protected File gitdir;
+
 	/** RevWalk used during command line parsing, if it was required. */
 	protected RevWalk argWalk;
 
@@ -84,10 +88,15 @@ final void setCommandName(final String name) {
 		commandName = name;
 	}
 
-	void init(final Repository repo) {
+	/** @return true if {@link #db}/{@link #getRepository()} is required. */
+	protected boolean requiresRepository() {
+		return true;
+	}
+
+	void init(final Repository repo, final File gd) {
 		try {
-			String outputEncoding = repo.getConfig().getString("i18n", null,
-					"logOutputEncoding");
+			final String outputEncoding = repo != null ? repo.getConfig()
+					.getString("i18n", null, "logOutputEncoding") : null;
 			if (outputEncoding != null)
 				out = new PrintWriter(new BufferedWriter(
 						new OutputStreamWriter(System.out, outputEncoding)));
@@ -97,7 +106,14 @@ void init(final Repository repo) {
 		} catch (IOException e) {
 			throw die("cannot create output stream");
 		}
-		db = repo;
+
+		if (repo != null) {
+			db = repo;
+			gitdir = repo.getDirectory();
+		} else {
+			db = null;
+			gitdir = gd;
+		}
 	}
 
 	/**
-- 
1.6.1.rc4.301.g5497a

[JGIT PATCH 2/5] Add "jgit init" command to create a new repository

[Shawn O. Pearce]

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../services/org.spearce.jgit.pgm.TextBuiltin      |    1 +
 .../src/org/spearce/jgit/pgm/Init.java             |   60 ++++++++++++++++++++
 2 files changed, 61 insertions(+), 0 deletions(-)
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java

diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
index 0e7e7d8..4a2b605 100644
--- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
+++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
@@ -4,6 +4,7 @@ org.spearce.jgit.pgm.DiffTree
 org.spearce.jgit.pgm.Fetch
 org.spearce.jgit.pgm.Glog
 org.spearce.jgit.pgm.IndexPack
+org.spearce.jgit.pgm.Init
 org.spearce.jgit.pgm.Log
 org.spearce.jgit.pgm.LsRemote
 org.spearce.jgit.pgm.LsTree
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java
new file mode 100644
index 0000000..197864d
--- /dev/null
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java
@@ -0,0 +1,60 @@
+/*
+ * Copyright (C) 2008, Google Inc.
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.pgm;
+
+import java.io.File;
+
+import org.spearce.jgit.lib.Repository;
+
+@Command(common = true, usage = "Create an empty git repository")
+class Init extends TextBuiltin {
+	@Override
+	protected final boolean requiresRepository() {
+		return false;
+	}
+
+	@Override
+	protected void run() throws Exception {
+		if (gitdir == null)
+			gitdir = new File(".git");
+		db = new Repository(gitdir);
+		db.create();
+		out.println("Initialized empty Git repository in "
+				+ gitdir.getAbsolutePath());
+	}
+}
-- 
1.6.1.rc4.301.g5497a

[JGIT PATCH 3/5] Modify "jgit daemon" so it can run outside of a repository

[Shawn O. Pearce]

We no longer export the current repository, but instead export the
directory trees given.  This behavior is more closely matches that
of "git daemon".

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../src/org/spearce/jgit/pgm/Daemon.java           |   33 +++++--------------
 1 files changed, 9 insertions(+), 24 deletions(-)

diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java
index aafc82e..39b43b2 100644
--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java
@@ -44,7 +44,6 @@
 
 import org.kohsuke.args4j.Argument;
 import org.kohsuke.args4j.Option;
-import org.spearce.jgit.lib.Repository;
 import org.spearce.jgit.transport.DaemonService;
 
 @Command(common = true, usage = "Export repositories over git://")
@@ -67,10 +66,15 @@
 	@Option(name = "--forbid-override", metaVar = "SERVICE", usage = "configure the service in daemon.servicename", multiValued = true)
 	final List<String> forbidOverride = new ArrayList<String>();
 
-	@Argument(metaVar = "DIRECTORY", usage = "directories to export")
+	@Argument(required = true, metaVar = "DIRECTORY", usage = "directories to export")
 	final List<File> directory = new ArrayList<File>();
 
 	@Override
+	protected boolean requiresRepository() {
+		return false;
+	}
+
+	@Override
 	protected void run() throws Exception {
 		final org.spearce.jgit.transport.Daemon d;
 
@@ -88,13 +92,9 @@ protected void run() throws Exception {
 		for (final String n : forbidOverride)
 			service(d, n).setOverridable(false);
 
-		if (directory.isEmpty()) {
-			export(d, db);
-		} else {
-			for (final File f : directory) {
-				out.println("Exporting " + f.getAbsolutePath());
-				d.exportDirectory(f);
-			}
+		for (final File f : directory) {
+			out.println("Exporting " + f.getAbsolutePath());
+			d.exportDirectory(f);
 		}
 		d.start();
 		out.println("Listening on " + d.getAddress());
@@ -107,19 +107,4 @@ private DaemonService service(final org.spearce.jgit.transport.Daemon d,
 			throw die("Service '" + n + "' not supported");
 		return svc;
 	}
-
-	private void export(final org.spearce.jgit.transport.Daemon daemon,
-			final Repository repo) {
-		File d = repo.getDirectory();
-		String name = d.getName();
-		while (name.equals(".git") || name.equals(".")) {
-			d = d.getParentFile();
-			name = d.getName();
-		}
-		if (!name.endsWith(".git"))
-			name += ".git";
-
-		out.println("Exporting current repository as \"" + name + "\"");
-		daemon.exportRepository(name, repo);
-	}
 }
-- 
1.6.1.rc4.301.g5497a

[JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific

[Shawn O. Pearce]

The amazon-s3:// protocol is unique to JGit, and is not supported
by any other Git implementation.  The easiest way to clone off of
one of those URLs is to use "jgit clone".  This program also acts
as a simple example for any IDE developers who want to implement
a clone feature, as it is the minimum effort required to make a
user reasonably happy with the resulting repository.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../services/org.spearce.jgit.pgm.TextBuiltin      |    1 +
 .../org/spearce/jgit/pgm/AbstractFetchCommand.java |  126 ++++++++++++++
 .../src/org/spearce/jgit/pgm/Clone.java            |  177 ++++++++++++++++++++
 .../src/org/spearce/jgit/pgm/Fetch.java            |   83 +---------
 4 files changed, 306 insertions(+), 81 deletions(-)
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java
 create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java

diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
index 4a2b605..40177f9 100644
--- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
+++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin
@@ -1,4 +1,5 @@
 org.spearce.jgit.pgm.Branch
+org.spearce.jgit.pgm.Clone
 org.spearce.jgit.pgm.Daemon
 org.spearce.jgit.pgm.DiffTree
 org.spearce.jgit.pgm.Fetch
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java
new file mode 100644
index 0000000..ea6f277
--- /dev/null
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java
@@ -0,0 +1,126 @@
+/*
+ * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.pgm;
+
+import org.kohsuke.args4j.Option;
+import org.spearce.jgit.lib.Constants;
+import org.spearce.jgit.lib.RefUpdate;
+import org.spearce.jgit.transport.FetchResult;
+import org.spearce.jgit.transport.TrackingRefUpdate;
+import org.spearce.jgit.transport.Transport;
+
+abstract class AbstractFetchCommand extends TextBuiltin {
+	@Option(name = "--verbose", aliases = { "-v" }, usage = "be more verbose")
+	private boolean verbose;
+
+	protected void showFetchResult(final Transport tn, final FetchResult r) {
+		boolean shownURI = false;
+		for (final TrackingRefUpdate u : r.getTrackingRefUpdates()) {
+			if (!verbose && u.getResult() == RefUpdate.Result.NO_CHANGE)
+				continue;
+
+			final char type = shortTypeOf(u.getResult());
+			final String longType = longTypeOf(u);
+			final String src = abbreviateRef(u.getRemoteName(), false);
+			final String dst = abbreviateRef(u.getLocalName(), true);
+
+			if (!shownURI) {
+				out.print("From ");
+				out.print(tn.getURI());
+				out.println();
+				shownURI = true;
+			}
+
+			out.format(" %c %-17s %-10s -> %s", type, longType, src, dst);
+			out.println();
+		}
+	}
+
+	private String longTypeOf(final TrackingRefUpdate u) {
+		final RefUpdate.Result r = u.getResult();
+		if (r == RefUpdate.Result.LOCK_FAILURE)
+			return "[lock fail]";
+
+		if (r == RefUpdate.Result.IO_FAILURE)
+			return "[i/o error]";
+
+		if (r == RefUpdate.Result.NEW) {
+			if (u.getRemoteName().startsWith(Constants.R_HEADS))
+				return "[new branch]";
+			else if (u.getLocalName().startsWith(Constants.R_TAGS))
+				return "[new tag]";
+			return "[new]";
+		}
+
+		if (r == RefUpdate.Result.FORCED) {
+			final String aOld = u.getOldObjectId().abbreviate(db).name();
+			final String aNew = u.getNewObjectId().abbreviate(db).name();
+			return aOld + "..." + aNew;
+		}
+
+		if (r == RefUpdate.Result.FAST_FORWARD) {
+			final String aOld = u.getOldObjectId().abbreviate(db).name();
+			final String aNew = u.getNewObjectId().abbreviate(db).name();
+			return aOld + ".." + aNew;
+		}
+
+		if (r == RefUpdate.Result.REJECTED)
+			return "[rejected]";
+		if (r == RefUpdate.Result.NO_CHANGE)
+			return "[up to date]";
+		return "[" + r.name() + "]";
+	}
+
+	private static char shortTypeOf(final RefUpdate.Result r) {
+		if (r == RefUpdate.Result.LOCK_FAILURE)
+			return '!';
+		if (r == RefUpdate.Result.IO_FAILURE)
+			return '!';
+		if (r == RefUpdate.Result.NEW)
+			return '*';
+		if (r == RefUpdate.Result.FORCED)
+			return '+';
+		if (r == RefUpdate.Result.FAST_FORWARD)
+			return ' ';
+		if (r == RefUpdate.Result.REJECTED)
+			return '!';
+		if (r == RefUpdate.Result.NO_CHANGE)
+			return '=';
+		return ' ';
+	}
+}
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java
new file mode 100644
index 0000000..51dd95d
--- /dev/null
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java
@@ -0,0 +1,177 @@
+/*
+ * Copyright (C) 2008, Google Inc.
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.pgm;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+import org.kohsuke.args4j.Argument;
+import org.kohsuke.args4j.Option;
+import org.spearce.jgit.errors.NotSupportedException;
+import org.spearce.jgit.errors.TransportException;
+import org.spearce.jgit.lib.Commit;
+import org.spearce.jgit.lib.Constants;
+import org.spearce.jgit.lib.GitIndex;
+import org.spearce.jgit.lib.Ref;
+import org.spearce.jgit.lib.RefComparator;
+import org.spearce.jgit.lib.RefUpdate;
+import org.spearce.jgit.lib.Repository;
+import org.spearce.jgit.lib.TextProgressMonitor;
+import org.spearce.jgit.lib.Tree;
+import org.spearce.jgit.lib.WorkDirCheckout;
+import org.spearce.jgit.transport.FetchResult;
+import org.spearce.jgit.transport.RefSpec;
+import org.spearce.jgit.transport.RemoteConfig;
+import org.spearce.jgit.transport.Transport;
+import org.spearce.jgit.transport.URIish;
+
+@Command(common = true, usage = "Clone a repository into a new directory")
+class Clone extends AbstractFetchCommand {
+	@Option(name = "--origin", aliases = { "-o" }, metaVar = "name", usage = "use <name> instead of 'origin' to track upstream")
+	private String remoteName = "origin";
+
+	@Argument(index = 0, required = true, metaVar = "uri-ish")
+	private String sourceUri;
+
+	@Argument(index = 1, metaVar = "directory")
+	private String localName;
+
+	@Override
+	protected final boolean requiresRepository() {
+		return false;
+	}
+
+	@Override
+	protected void run() throws Exception {
+		if (localName != null && gitdir != null)
+			throw die("conflicting usage of --git-dir and arguments");
+
+		final URIish uri = new URIish(sourceUri);
+		if (localName == null) {
+			String p = uri.getPath();
+			while (p.endsWith("/"))
+				p = p.substring(0, p.length() - 1);
+			final int s = p.lastIndexOf('/');
+			if (s < 0)
+				throw die("cannot guess local name from " + sourceUri);
+			localName = p.substring(s + 1);
+			if (localName.endsWith(".git"))
+				localName = localName.substring(0, localName.length() - 4);
+		}
+		if (gitdir == null)
+			gitdir = new File(localName, ".git");
+		db = new Repository(gitdir);
+		db.create();
+		out.println("Initialized empty Git repository in "
+				+ gitdir.getAbsolutePath());
+		out.flush();
+
+		saveRemote(uri);
+		final FetchResult r = runFetch();
+		final Ref branch = guessHEAD(r);
+		doCheckout(branch);
+	}
+
+	private void saveRemote(final URIish uri) throws URISyntaxException,
+			IOException {
+		final RemoteConfig rc = new RemoteConfig(db.getConfig(), remoteName);
+		rc.addURI(uri);
+		rc.addFetchRefSpec(new RefSpec().setForceUpdate(true)
+				.setSourceDestination(Constants.R_HEADS + "*",
+						Constants.R_REMOTES + remoteName + "/*"));
+		rc.update(db.getConfig());
+		db.getConfig().save();
+	}
+
+	private FetchResult runFetch() throws NotSupportedException,
+			URISyntaxException, TransportException {
+		final Transport tn = Transport.open(db, remoteName);
+		final FetchResult r;
+		try {
+			r = tn.fetch(new TextProgressMonitor(), null);
+		} finally {
+			tn.close();
+		}
+		showFetchResult(tn, r);
+		return r;
+	}
+
+	private Ref guessHEAD(final FetchResult result) {
+		final Ref idHEAD = result.getAdvertisedRef(Constants.HEAD);
+		final List<Ref> availableRefs = new ArrayList<Ref>();
+		Ref head = null;
+		for (final Ref r : result.getAdvertisedRefs()) {
+			final String n = r.getName();
+			if (!n.startsWith(Constants.R_HEADS))
+				continue;
+			availableRefs.add(r);
+			if (idHEAD == null || head != null)
+				continue;
+			if (r.getObjectId().equals(idHEAD.getObjectId()))
+				head = r;
+		}
+		Collections.sort(availableRefs, RefComparator.INSTANCE);
+		if (idHEAD != null && head == null)
+			head = idHEAD;
+		return head;
+	}
+
+	private void doCheckout(final Ref branch) throws IOException {
+		if (branch == null)
+			throw die("cannot checkout; no HEAD advertised by remote");
+		if (!Constants.HEAD.equals(branch.getName()))
+			db.writeSymref(Constants.HEAD, branch.getName());
+
+		final Commit commit = db.mapCommit(branch.getObjectId());
+		final RefUpdate u = db.updateRef(Constants.HEAD);
+		u.setNewObjectId(commit.getCommitId());
+		u.forceUpdate();
+
+		final GitIndex index = new GitIndex(db);
+		final Tree tree = commit.getTree();
+		final WorkDirCheckout co;
+
+		co = new WorkDirCheckout(db, db.getWorkDir(), index, tree);
+		co.checkout();
+		index.write();
+	}
+}
diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java
index e9d3260..8f3f7d5 100644
--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java
+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java
@@ -41,19 +41,13 @@
 
 import org.kohsuke.args4j.Argument;
 import org.kohsuke.args4j.Option;
-import org.spearce.jgit.lib.Constants;
-import org.spearce.jgit.lib.RefUpdate;
 import org.spearce.jgit.lib.TextProgressMonitor;
 import org.spearce.jgit.transport.FetchResult;
 import org.spearce.jgit.transport.RefSpec;
-import org.spearce.jgit.transport.TrackingRefUpdate;
 import org.spearce.jgit.transport.Transport;
 
 @Command(common = true, usage = "Update remote refs from another repository")
-class Fetch extends TextBuiltin {
-	@Option(name = "--verbose", aliases = { "-v" }, usage = "be more verbose")
-	private boolean verbose;
-
+class Fetch extends AbstractFetchCommand {
 	@Option(name = "--fsck", usage = "perform fsck style checks on receive")
 	private Boolean fsck;
 
@@ -91,79 +85,6 @@ protected void run() throws Exception {
 		} finally {
 			tn.close();
 		}
-
-		boolean shownURI = false;
-		for (final TrackingRefUpdate u : r.getTrackingRefUpdates()) {
-			if (!verbose && u.getResult() == RefUpdate.Result.NO_CHANGE)
-				continue;
-
-			final char type = shortTypeOf(u.getResult());
-			final String longType = longTypeOf(u);
-			final String src = abbreviateRef(u.getRemoteName(), false);
-			final String dst = abbreviateRef(u.getLocalName(), true);
-
-			if (!shownURI) {
-				out.print("From ");
-				out.print(tn.getURI());
-				out.println();
-				shownURI = true;
-			}
-
-			out.format(" %c %-17s %-10s -> %s", type, longType, src, dst);
-			out.println();
-		}
-	}
-
-	private String longTypeOf(final TrackingRefUpdate u) {
-		final RefUpdate.Result r = u.getResult();
-		if (r == RefUpdate.Result.LOCK_FAILURE)
-			return "[lock fail]";
-
-		if (r == RefUpdate.Result.IO_FAILURE)
-			return "[i/o error]";
-
-		if (r == RefUpdate.Result.NEW) {
-			if (u.getRemoteName().startsWith(Constants.R_HEADS))
-				return "[new branch]";
-			else if (u.getLocalName().startsWith(Constants.R_TAGS))
-				return "[new tag]";
-			return "[new]";
-		}
-
-		if (r == RefUpdate.Result.FORCED) {
-			final String aOld = u.getOldObjectId().abbreviate(db).name();
-			final String aNew = u.getNewObjectId().abbreviate(db).name();
-			return aOld + "..." + aNew;
-		}
-
-		if (r == RefUpdate.Result.FAST_FORWARD) {
-			final String aOld = u.getOldObjectId().abbreviate(db).name();
-			final String aNew = u.getNewObjectId().abbreviate(db).name();
-			return aOld + ".." + aNew;
-		}
-
-		if (r == RefUpdate.Result.REJECTED)
-			return "[rejected]";
-		if (r == RefUpdate.Result.NO_CHANGE)
-			return "[up to date]";
-		return "[" + r.name() + "]";
-	}
-
-	private static char shortTypeOf(final RefUpdate.Result r) {
-		if (r == RefUpdate.Result.LOCK_FAILURE)
-			return '!';
-		if (r == RefUpdate.Result.IO_FAILURE)
-			return '!';
-		if (r == RefUpdate.Result.NEW)
-			return '*';
-		if (r == RefUpdate.Result.FORCED)
-			return '+';
-		if (r == RefUpdate.Result.FAST_FORWARD)
-			return ' ';
-		if (r == RefUpdate.Result.REJECTED)
-			return '!';
-		if (r == RefUpdate.Result.NO_CHANGE)
-			return '=';
-		return ' ';
+		showFetchResult(tn, r);
 	}
 }
-- 
1.6.1.rc4.301.g5497a

[JGIT PATCH 5/5] Fix "fetch pulled too many objects" when auto-following tags

[Shawn O. Pearce]

If we don't take into consideration the objects obtained during
the first connection when we open a second to auto-follow tags
we will download a large chunk of the repository a second time.
This is very wasteful of network bandwidth, and is an abuse of
the server.

Because we delay all ref updates until the very end of the fetch
process we need to hold onto the set of objects we requested in
the first connection, and pass that set into the subsequent one
so it can be considered reachable.

Issue: http://code.google.com/p/egit/issues/detail?id=22
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .../jgit/transport/BaseFetchConnection.java        |   26 ++++++++++++-------
 .../jgit/transport/BasePackFetchConnection.java    |   25 +++++++++++++++----
 .../spearce/jgit/transport/FetchConnection.java    |   21 +++++++++++----
 .../org/spearce/jgit/transport/FetchProcess.java   |    6 ++++-
 .../spearce/jgit/transport/TransportBundle.java    |    3 +-
 .../jgit/transport/WalkFetchConnection.java        |   14 ++++++++--
 6 files changed, 69 insertions(+), 26 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java
index 6709bfc..bb81296 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java
@@ -38,8 +38,10 @@
 package org.spearce.jgit.transport;
 
 import java.util.Collection;
+import java.util.Set;
 
 import org.spearce.jgit.errors.TransportException;
+import org.spearce.jgit.lib.ObjectId;
 import org.spearce.jgit.lib.ProgressMonitor;
 import org.spearce.jgit.lib.Ref;
 
@@ -54,9 +56,10 @@
 abstract class BaseFetchConnection extends BaseConnection implements
 		FetchConnection {
 	public final void fetch(final ProgressMonitor monitor,
-			final Collection<Ref> want) throws TransportException {
+			final Collection<Ref> want, final Set<ObjectId> have)
+			throws TransportException {
 		markStartedOperation();
-		doFetch(monitor, want);
+		doFetch(monitor, want, have);
 	}
 
 	/**
@@ -68,19 +71,22 @@ public boolean didFetchIncludeTags() {
 	}
 
 	/**
-	 * Implementation of {@link #fetch(ProgressMonitor, Collection)} without
-	 * checking for multiple fetch.
+	 * Implementation of {@link #fetch(ProgressMonitor, Collection, Set)}
+	 * without checking for multiple fetch.
 	 *
 	 * @param monitor
-	 *            as in {@link #fetch(ProgressMonitor, Collection)}
+	 *            as in {@link #fetch(ProgressMonitor, Collection, Set)}
 	 * @param want
-	 *            as in {@link #fetch(ProgressMonitor, Collection)}
+	 *            as in {@link #fetch(ProgressMonitor, Collection, Set)}
+	 * @param have
+	 *            as in {@link #fetch(ProgressMonitor, Collection, Set)}
 	 * @throws TransportException
-	 *             as in {@link #fetch(ProgressMonitor, Collection)}, but
+	 *             as in {@link #fetch(ProgressMonitor, Collection, Set)}, but
 	 *             implementation doesn't have to care about multiple
-	 *             {@link #fetch(ProgressMonitor, Collection)} calls, as it is
-	 *             checked in this class.
+	 *             {@link #fetch(ProgressMonitor, Collection, Set)} calls, as it
+	 *             is checked in this class.
 	 */
 	protected abstract void doFetch(final ProgressMonitor monitor,
-			final Collection<Ref> want) throws TransportException;
+			final Collection<Ref> want, final Set<ObjectId> have)
+			throws TransportException;
 }
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java
index 542a8a9..2cb9b64 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java
@@ -41,10 +41,12 @@
 import java.io.IOException;
 import java.util.Collection;
 import java.util.Date;
+import java.util.Set;
 
 import org.spearce.jgit.errors.TransportException;
 import org.spearce.jgit.lib.AnyObjectId;
 import org.spearce.jgit.lib.MutableObjectId;
+import org.spearce.jgit.lib.ObjectId;
 import org.spearce.jgit.lib.ProgressMonitor;
 import org.spearce.jgit.lib.Ref;
 import org.spearce.jgit.revwalk.RevCommit;
@@ -137,9 +139,10 @@ BasePackFetchConnection(final PackTransport packTransport) {
 	}
 
 	public final void fetch(final ProgressMonitor monitor,
-			final Collection<Ref> want) throws TransportException {
+			final Collection<Ref> want, final Set<ObjectId> have)
+			throws TransportException {
 		markStartedOperation();
-		doFetch(monitor, want);
+		doFetch(monitor, want, have);
 	}
 
 	public boolean didFetchIncludeTags() {
@@ -151,10 +154,11 @@ public boolean didFetchTestConnectivity() {
 	}
 
 	protected void doFetch(final ProgressMonitor monitor,
-			final Collection<Ref> want) throws TransportException {
+			final Collection<Ref> want, final Set<ObjectId> have)
+			throws TransportException {
 		try {
 			markRefsAdvertised();
-			markReachable(maxTimeWanted(want));
+			markReachable(have, maxTimeWanted(want));
 
 			if (sendWants(want)) {
 				negotiate(monitor);
@@ -193,7 +197,8 @@ private int maxTimeWanted(final Collection<Ref> wants) {
 		return maxTime;
 	}
 
-	private void markReachable(final int maxTime) throws IOException {
+	private void markReachable(final Set<ObjectId> have, final int maxTime)
+			throws IOException {
 		for (final Ref r : local.getAllRefs().values()) {
 			try {
 				final RevCommit o = walk.parseCommit(r.getObjectId());
@@ -204,6 +209,16 @@ private void markReachable(final int maxTime) throws IOException {
 			}
 		}
 
+		for (final ObjectId id : have) {
+			try {
+				final RevCommit o = walk.parseCommit(id);
+				o.add(REACHABLE);
+				reachableCommits.add(o);
+			} catch (IOException readError) {
+				// If we cannot read the value of the ref skip it.
+			}
+		}
+
 		if (maxTime > 0) {
 			// Mark reachable commits until we reach maxTime. These may
 			// wind up later matching up against things we want and we
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java
index a56ca6c..61ef219 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java
@@ -38,8 +38,10 @@
 package org.spearce.jgit.transport;
 
 import java.util.Collection;
+import java.util.Set;
 
 import org.spearce.jgit.errors.TransportException;
+import org.spearce.jgit.lib.ObjectId;
 import org.spearce.jgit.lib.ProgressMonitor;
 import org.spearce.jgit.lib.Ref;
 
@@ -85,23 +87,29 @@
 	 * @param want
 	 *            one or more refs advertised by this connection that the caller
 	 *            wants to store locally.
+	 * @param have
+	 *            additional objects known to exist in the destination
+	 *            repository, especially if they aren't yet reachable by the ref
+	 *            database. Connections should take this set as an addition to
+	 *            what is reachable through all Refs, not in replace of it.
 	 * @throws TransportException
 	 *             objects could not be copied due to a network failure,
 	 *             protocol error, or error on remote side, or connection was
 	 *             already used for fetch.
 	 */
-	public void fetch(final ProgressMonitor monitor, final Collection<Ref> want)
+	public void fetch(final ProgressMonitor monitor,
+			final Collection<Ref> want, final Set<ObjectId> have)
 			throws TransportException;
 
 	/**
-	 * Did the last {@link #fetch(ProgressMonitor, Collection)} get tags?
+	 * Did the last {@link #fetch(ProgressMonitor, Collection, Set)} get tags?
 	 * <p>
 	 * Some Git aware transports are able to implicitly grab an annotated tag if
 	 * {@link TagOpt#AUTO_FOLLOW} or {@link TagOpt#FETCH_TAGS} was selected and
 	 * the object the tag peels to (references) was transferred as part of the
-	 * last {@link #fetch(ProgressMonitor, Collection)} call. If it is possible
-	 * for such tags to have been included in the transfer this method returns
-	 * true, allowing the caller to attempt tag discovery.
+	 * last {@link #fetch(ProgressMonitor, Collection, Set)} call. If it is
+	 * possible for such tags to have been included in the transfer this method
+	 * returns true, allowing the caller to attempt tag discovery.
 	 * <p>
 	 * By returning only true/false (and not the actual list of tags obtained)
 	 * the transport itself does not need to be aware of whether or not tags
@@ -113,7 +121,8 @@ public void fetch(final ProgressMonitor monitor, final Collection<Ref> want)
 	public boolean didFetchIncludeTags();
 
 	/**
-	 * Did the last {@link #fetch(ProgressMonitor, Collection)} validate graph?
+	 * Did the last {@link #fetch(ProgressMonitor, Collection, Set)} validate
+	 * graph?
 	 * <p>
 	 * Some transports walk the object graph on the client side, with the client
 	 * looking for what objects it is missing and requesting them individually
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java
index bb2d051..09718eb 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java
@@ -75,6 +75,9 @@
 	/** Set of refs we will actually wind up asking to obtain. */
 	private final HashMap<ObjectId, Ref> askFor = new HashMap<ObjectId, Ref>();
 
+	/** Objects we know we have locally. */
+	private final HashSet<ObjectId> have = new HashSet<ObjectId>();
+
 	/** Updates to local tracking branches (if any). */
 	private final ArrayList<TrackingRefUpdate> localUpdates = new ArrayList<TrackingRefUpdate>();
 
@@ -133,6 +136,7 @@ else if (tagopt == TagOpt.FETCH_TAGS)
 				// There are more tags that we want to follow, but
 				// not all were asked for on the initial request.
 				//
+				have.addAll(askFor.keySet());
 				askFor.clear();
 				for (final Ref r : additionalTags) {
 					final ObjectId id = r.getPeeledObjectId();
@@ -173,7 +177,7 @@ else if (tagopt == TagOpt.FETCH_TAGS)
 
 	private void fetchObjects(final ProgressMonitor monitor)
 			throws TransportException {
-		conn.fetch(monitor, askFor.values());
+		conn.fetch(monitor, askFor.values(), have);
 		if (transport.isCheckFetchedObjects()
 				&& !conn.didFetchTestConnectivity() && !askForIsComplete())
 			throw new TransportException(transport.getURI(),
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java
index 7d38b02..1734d94 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java
@@ -171,7 +171,8 @@ public boolean didFetchTestConnectivity() {
 
 		@Override
 		protected void doFetch(final ProgressMonitor monitor,
-				final Collection<Ref> want) throws TransportException {
+				final Collection<Ref> want, final Set<ObjectId> have)
+				throws TransportException {
 			verifyPrerequisites();
 			try {
 				final IndexPack ip = newIndexPack();
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java
index d089f7b..91c5ea8 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java
@@ -195,8 +195,9 @@ public boolean didFetchTestConnectivity() {
 
 	@Override
 	protected void doFetch(final ProgressMonitor monitor,
-			final Collection<Ref> want) throws TransportException {
-		markLocalRefsComplete();
+			final Collection<Ref> want, final Set<ObjectId> have)
+			throws TransportException {
+		markLocalRefsComplete(have);
 		queueWants(want);
 
 		while (!monitor.isCancelled() && !workQueue.isEmpty()) {
@@ -642,7 +643,7 @@ private void saveLooseObject(final AnyObjectId id, final byte[] compressed)
 		return null;
 	}
 
-	private void markLocalRefsComplete() throws TransportException {
+	private void markLocalRefsComplete(final Set<ObjectId> have) throws TransportException {
 		for (final Ref r : local.getAllRefs().values()) {
 			try {
 				markLocalObjComplete(revWalk.parseAny(r.getObjectId()));
@@ -651,6 +652,13 @@ private void markLocalRefsComplete() throws TransportException {
 						+ " is missing object(s).", readError);
 			}
 		}
+		for (final ObjectId id : have) {
+			try {
+				markLocalObjComplete(revWalk.parseAny(id));
+			} catch (IOException readError) {
+				throw new TransportException("Missing assumed "+id.name(), readError);
+			}
+		}
 	}
 
 	private void markLocalObjComplete(RevObject obj) throws IOException {
-- 
1.6.1.rc4.301.g5497a

Re: [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific

[Robin Rosenberg]

Would it be hard to get the progress look better?

-- robin

EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2
Initialized empty Git repository in /home/me/SW/LIBGIT2/.git                  
Counting objects:       547                                                   
Compressing objects:    100% (192/192)                                        
ts:                     100% (192/192)                                        
Compressing objects:    100% (192/192)                                        
ng objects:             100% (192/192)                                        
Compressing objects:    100% (192/192)                                        
bjects:                 100% (192/192)                                        
Compressing objects:    100% (192/192)
pressing objects:       100% (192/192)
ts:                     100% (192/192)
Compressing objects:    100% (192/192)
Receiving objects:      100% (547/547)
jects:                  100% (192/192)
Compressing objects:    100% (192/192)
ressing objects:        100% (192/192)
s:                      100% (192/192)
Compressing objects:    100% (192/192)
ssing objects:          100% (192/192)
ts:                     100% (192/192)
Compressing objects:    100% (192/192)
 objects:               100% (192/192)
Compressing objects:    100% (192/192)
essing objects:         100% (192/192)
cts:                    100% (192/192)
Compressing objects:    100% (192/192)
g objects:              100% (192/192)
Compressing objects:    100% (192/192)
ressing objects:        100% (192/192)
Resolving deltas:       100% (358/358)
From git://repo.or.cz/libgit2.git
 * [new branch]      master     -> origin/master

Re: [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific

[Shawn O. Pearce]

Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
> Would it be hard to get the progress look better?

This happens because we are scanning the progress channel wrong in
some cases.  Its random; most of the time we can get it right, but
every once in a while we get out of sync and get these short names.

This is a general problem with the progress client side code, not
the new server side code I've introduced.  We also see it going
against a standard C git daemon.  I haven't been able to figure
out a way to smooth out the messages.

FWIW, git-gui also gets this sort of progress corruption sometimes
when it does a clone.  So I've reproduced the same parsing bug in
two different languages and two different implementations... ;-)

 
> EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2
> Initialized empty Git repository in /home/me/SW/LIBGIT2/.git                  
> Counting objects:       547                                                   
> Compressing objects:    100% (192/192)                                        
> ts:                     100% (192/192)                                        
> Compressing objects:    100% (192/192)                                        
> ng objects:             100% (192/192)                                        
> Compressing objects:    100% (192/192)                                        
> bjects:                 100% (192/192)                                        
> Compressing objects:    100% (192/192)
> pressing objects:       100% (192/192)
> ts:                     100% (192/192)
> Compressing objects:    100% (192/192)
> Receiving objects:      100% (547/547)
> jects:                  100% (192/192)
> Compressing objects:    100% (192/192)
> ressing objects:        100% (192/192)
> s:                      100% (192/192)
> Compressing objects:    100% (192/192)
> ssing objects:          100% (192/192)
> ts:                     100% (192/192)
> Compressing objects:    100% (192/192)
>  objects:               100% (192/192)
> Compressing objects:    100% (192/192)
> essing objects:         100% (192/192)
> cts:                    100% (192/192)
> Compressing objects:    100% (192/192)
> g objects:              100% (192/192)
> Compressing objects:    100% (192/192)
> ressing objects:        100% (192/192)
> Resolving deltas:       100% (358/358)
> From git://repo.or.cz/libgit2.git
>  * [new branch]      master     -> origin/master

-- 
Shawn.

[JGIT PATCH] Improve the sideband progress scaper to be more accurate

[Shawn O. Pearce]

By matching only whole lines we should be able to improve the
progress scaper so we avoid ugly output like we had been seeing:

  EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2
  Initialized empty Git repository in /home/me/SW/LIBGIT2/.git
  Counting objects:       547
  Compressing objects:    100% (192/192)
  ts:                     100% (192/192)
  Compressing objects:    100% (192/192)
  ng objects:             100% (192/192)

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
 > Would it be hard to get the progress look better?

 Maybe this does the trick.  Its hard to reproduce so its hard to
 come up with the condition that was giving us the problem before.
 I suspect its because we were getting line fragments on the sideband
 channel, but I'm not sure that was really the case.

 .../jgit/transport/SideBandInputStream.java        |   37 +++++++++++++++++--
 1 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java b/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java
index 3ec9bff..f0ba3d3 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java
@@ -73,10 +73,10 @@
 	private static final int CH_ERROR = 3;
 
 	private static Pattern P_UNBOUNDED = Pattern.compile(
-			".*?([\\w ]+): (\\d+)(, done)?.*", Pattern.DOTALL);
+			"^([\\w ]+): (\\d+)( |, done)?.*", Pattern.DOTALL);
 
 	private static Pattern P_BOUNDED = Pattern.compile(
-			".*?([\\w ]+):.*\\((\\d+)/(\\d+)\\).*", Pattern.DOTALL);
+			"^([\\w ]+):.*\\((\\d+)/(\\d+)\\).*", Pattern.DOTALL);
 
 	private final PacketLineIn pckIn;
 
@@ -84,6 +84,8 @@
 
 	private final ProgressMonitor monitor;
 
+	private String progressBuffer = "";
+
 	private String currentTask;
 
 	private int lastCnt;
@@ -160,7 +162,31 @@ private void needDataPacket() throws IOException {
 		}
 	}
 
-	private void progress(final String msg) {
+	private void progress(String pkt) {
+		pkt = progressBuffer + pkt;
+		for (;;) {
+			final int lf = pkt.indexOf('\n');
+			final int cr = pkt.indexOf('\r');
+			final int s;
+			if (0 <= lf && 0 <= cr)
+				s = Math.min(lf, cr);
+			else if (0 <= lf)
+				s = lf;
+			else if (0 <= cr)
+				s = cr;
+			else
+				break;
+
+			final String msg = pkt.substring(0, s);
+			if (doProgressLine(msg))
+				pkt = pkt.substring(s + 1);
+			else
+				break;
+		}
+		progressBuffer = pkt;
+	}
+
+	private boolean doProgressLine(final String msg) {
 		Matcher matcher;
 
 		matcher = P_BOUNDED.matcher(msg);
@@ -175,7 +201,7 @@ private void progress(final String msg) {
 			final int cnt = Integer.parseInt(matcher.group(2));
 			monitor.update(cnt - lastCnt);
 			lastCnt = cnt;
-			return;
+			return true;
 		}
 
 		matcher = P_UNBOUNDED.matcher(msg);
@@ -189,7 +215,10 @@ private void progress(final String msg) {
 			final int cnt = Integer.parseInt(matcher.group(2));
 			monitor.update(cnt - lastCnt);
 			lastCnt = cnt;
+			return true;
 		}
+
+		return false;
 	}
 
 	private String readString(final int len) throws IOException {
-- 
1.6.1.59.g6f6746


-- 
Shawn.

Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate

[Robin Rosenberg]

onsdag 31 december 2008 20:04:01 skrev Shawn O. Pearce:
> By matching only whole lines we should be able to improve the
> progress scaper so we avoid ugly output like we had been seeing:
> 
>   EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2
>   Initialized empty Git repository in /home/me/SW/LIBGIT2/.git
>   Counting objects:       547
>   Compressing objects:    100% (192/192)
>   ts:                     100% (192/192)
>   Compressing objects:    100% (192/192)
>   ng objects:             100% (192/192)
> 
> Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
> ---
>  Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
>  > Would it be hard to get the progress look better?
> 
>  Maybe this does the trick.  Its hard to reproduce so its hard to
>  come up with the condition that was giving us the problem before.
>  I suspect its because we were getting line fragments on the sideband
>  channel, but I'm not sure that was really the case.

Nasty. I couldn't reproduce it myself. I'll hold onto this one for a while and
see if I'll get the opportunity to test it live with this problem.

-- robin

Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate

[Nicolas Pitre]

On Sat, 3 Jan 2009, Robin Rosenberg wrote:

> onsdag 31 december 2008 20:04:01 skrev Shawn O. Pearce:
> > By matching only whole lines we should be able to improve the
> > progress scaper so we avoid ugly output like we had been seeing:
> > 
> >   EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2
> >   Initialized empty Git repository in /home/me/SW/LIBGIT2/.git
> >   Counting objects:       547
> >   Compressing objects:    100% (192/192)
> >   ts:                     100% (192/192)
> >   Compressing objects:    100% (192/192)
> >   ng objects:             100% (192/192)
> > 
> > Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
> > ---
> >  Robin Rosenberg <robin.rosenberg@dewire.com> wrote:
> >  > Would it be hard to get the progress look better?
> > 
> >  Maybe this does the trick.  Its hard to reproduce so its hard to
> >  come up with the condition that was giving us the problem before.
> >  I suspect its because we were getting line fragments on the sideband
> >  channel, but I'm not sure that was really the case.
> 
> Nasty. I couldn't reproduce it myself. I'll hold onto this one for a while and
> see if I'll get the opportunity to test it live with this problem.

There may indeed be line fragments sent over the sideband channel, as 
well as the opposite which is multiple lines sent at once in a single 
packet.  If you look at sideband.c you'll find about all those cases.

In general, what you have to do is:

 - for each packet:
   - split into multiple chunks on line breaks ('\r' or '\n')
   - for each chunk:
     - if last chunk didn't end with a line break, or if current 
       chunk is empty or only contains a line break, then skip printing 
       the "remote:" prefix.  Otherwise print it.
     - print the current chunk up to any line break
     - if current chunk contains a line break and other characters then
       print a sequence to clear the remaining of the screen line
     - print the line break if any

That should cover all cases.  Note that the case where a previous chunk 
didn't end with a line break is always seen when moving from one packet 
to another, so that state cannot be local to the inner loop.


Nicolas

Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate

[Shawn O. Pearce]

Nicolas Pitre <nico@cam.org> wrote:
> There may indeed be line fragments sent over the sideband channel, as 
> well as the opposite which is multiple lines sent at once in a single 
> packet.  If you look at sideband.c you'll find about all those cases.

Thanks, that's what I thought...
 
> In general, what you have to do is:
> 
>  - for each packet:
>    - split into multiple chunks on line breaks ('\r' or '\n')
>    - for each chunk:
>      - if last chunk didn't end with a line break, or if current 
>        chunk is empty or only contains a line break, then skip printing 
>        the "remote:" prefix.  Otherwise print it.
>      - print the current chunk up to any line break
>      - if current chunk contains a line break and other characters then
>        print a sequence to clear the remaining of the screen line
>      - print the line break if any

Hmm.  I should note that C Git still screws this up sometimes.  I've
seen 1.6.1 git fetch mess up the output from repo.or.cz's sideband.
I'm sure Pasky isn't running JGit's daemon, its too damn fast. :-)

I don't have a spew of it handy, but late last week I saw it screw
up while doing a clone off repo.or.cz.

Robin, maybe we should apply this patch?  Its along the lines
Nico is describing above, but because our progress code doesn't do
direct display its a little less complex, we don't have to worry
about clearing the current line.  But I think I'm doing the chunk
splitting and filtering he's talking about.

-- 
Shawn.

Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate

[Nicolas Pitre]

On Tue, 6 Jan 2009, Shawn O. Pearce wrote:

> Nicolas Pitre <nico@cam.org> wrote:
> > There may indeed be line fragments sent over the sideband channel, as 
> > well as the opposite which is multiple lines sent at once in a single 
> > packet.  If you look at sideband.c you'll find about all those cases.
> 
> Thanks, that's what I thought...
>  
> > In general, what you have to do is:
> > 
> >  - for each packet:
> >    - split into multiple chunks on line breaks ('\r' or '\n')
> >    - for each chunk:
> >      - if last chunk didn't end with a line break, or if current 
> >        chunk is empty or only contains a line break, then skip printing 
> >        the "remote:" prefix.  Otherwise print it.
> >      - print the current chunk up to any line break
> >      - if current chunk contains a line break and other characters then
> >        print a sequence to clear the remaining of the screen line
> >      - print the line break if any
> 
> Hmm.  I should note that C Git still screws this up sometimes.  I've
> seen 1.6.1 git fetch mess up the output from repo.or.cz's sideband.
> I'm sure Pasky isn't running JGit's daemon, its too damn fast. :-)
> 
> I don't have a spew of it handy, but late last week I saw it screw
> up while doing a clone off repo.or.cz.

Next time you see such things please send me a snapshot.

What is still possible and IMHO not worth the effort to fix is some 
interaction between local and remote displays which, if intermixed in 
some unlucky way, do screw up the final appearance.  For example, if the 
remote has sent a partial line, and before the remaining of it is 
received/printed you actually have some local process also displaying a 
line of its own.  If that was to become a real issue, then the fix would 
involve caching any partial line sent from the remote until the 
associated line break is received.


Nicolas