* [JGIT PATCH 0/5] Add jgit init, clone, receive-pack; transport fixes
@ 2008-12-23 18:03 Shawn O. Pearce
2008-12-23 18:03 ` [JGIT PATCH 1/5] Add "jgit receive-pack" and permit commands to start not in a repository Shawn O. Pearce
0 siblings, 1 reply; 13+ messages in thread
From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw)
To: Robin Rosenberg; +Cc: git
Define a "jgit init", "jgit clone" commands from the command line.
Add "jgit receive-pack". This now works from C git:
git push --receive-pack='jgit receive-pack' dst.git
Fix issue #22, causing jgit to fetch too many objects during an
initial clone/fetch of a repository.
Change "jgit daemon" so it can run outside of a repository, making
it slightly more useful.
Shawn O. Pearce (5):
Add "jgit receive-pack" and permit commands to start not in a
repository
Add "jgit init" command to create a new repository
Modify "jgit daemon" so it can run outside of a repository
Add "jgit clone" to support cloning off URLs that are JGit specific
Fix "fetch pulled too many objects" when auto-following tags
.../services/org.spearce.jgit.pgm.TextBuiltin | 3 +
.../org/spearce/jgit/pgm/AbstractFetchCommand.java | 126 ++++++++++++++
.../src/org/spearce/jgit/pgm/Clone.java | 177 ++++++++++++++++++++
.../src/org/spearce/jgit/pgm/Daemon.java | 33 +---
.../src/org/spearce/jgit/pgm/Fetch.java | 83 +---------
.../src/org/spearce/jgit/pgm/Init.java | 60 +++++++
.../src/org/spearce/jgit/pgm/Main.java | 19 ++-
.../src/org/spearce/jgit/pgm/ReceivePack.java | 67 ++++++++
.../src/org/spearce/jgit/pgm/TextBuiltin.java | 24 +++-
.../jgit/transport/BaseFetchConnection.java | 26 ++-
.../jgit/transport/BasePackFetchConnection.java | 25 +++-
.../spearce/jgit/transport/FetchConnection.java | 21 ++-
.../org/spearce/jgit/transport/FetchProcess.java | 6 +-
.../spearce/jgit/transport/TransportBundle.java | 3 +-
.../jgit/transport/WalkFetchConnection.java | 14 ++-
15 files changed, 544 insertions(+), 143 deletions(-)
create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java
create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java
create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java
create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java
^ permalink raw reply [flat|nested] 13+ messages in thread* [JGIT PATCH 1/5] Add "jgit receive-pack" and permit commands to start not in a repository 2008-12-23 18:03 [JGIT PATCH 0/5] Add jgit init, clone, receive-pack; transport fixes Shawn O. Pearce @ 2008-12-23 18:03 ` Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 2/5] Add "jgit init" command to create a new repository Shawn O. Pearce 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../src/org/spearce/jgit/pgm/Main.java | 19 +++--- .../src/org/spearce/jgit/pgm/ReceivePack.java | 67 ++++++++++++++++++++ .../src/org/spearce/jgit/pgm/TextBuiltin.java | 24 ++++++- 4 files changed, 99 insertions(+), 12 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin index 5fb0953..0e7e7d8 100644 --- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin +++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin @@ -9,6 +9,7 @@ org.spearce.jgit.pgm.LsRemote org.spearce.jgit.pgm.LsTree org.spearce.jgit.pgm.MergeBase org.spearce.jgit.pgm.Push +org.spearce.jgit.pgm.ReceivePack org.spearce.jgit.pgm.RevList org.spearce.jgit.pgm.Rm org.spearce.jgit.pgm.ShowRev diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java index c8bade8..8de4ae9 100644 --- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Main.java @@ -143,15 +143,18 @@ private void execute(final String[] argv) throws Exception { System.exit(1); } - if (gitdir == null) - gitdir = findGitDir(); - if (gitdir == null || !gitdir.isDirectory()) { - System.err.println("error: can't find git directory"); - System.exit(1); - } - final TextBuiltin cmd = subcommand; - cmd.init(new Repository(gitdir)); + if (cmd.requiresRepository()) { + if (gitdir == null) + gitdir = findGitDir(); + if (gitdir == null || !gitdir.isDirectory()) { + System.err.println("error: can't find git directory"); + System.exit(1); + } + cmd.init(new Repository(gitdir), gitdir); + } else { + cmd.init(null, gitdir); + } try { cmd.execute(arguments.toArray(new String[arguments.size()])); } finally { diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java new file mode 100644 index 0000000..579f893 --- /dev/null +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/ReceivePack.java @@ -0,0 +1,67 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.pgm; + +import java.io.File; + +import org.kohsuke.args4j.Argument; +import org.spearce.jgit.lib.Repository; + +@Command(common = false, usage = "Server side backend for 'jgit push'") +class ReceivePack extends TextBuiltin { + @Argument(index = 0, required = true, metaVar = "DIRECTORY", usage = "Repository to receive into") + File gitdir; + + @Override + protected final boolean requiresRepository() { + return false; + } + + @Override + protected void run() throws Exception { + final org.spearce.jgit.transport.ReceivePack rp; + + if (new File(gitdir, ".git").isDirectory()) + gitdir = new File(gitdir, ".git"); + db = new Repository(gitdir); + if (!db.getObjectsDirectory().isDirectory()) + throw die("'" + gitdir.getPath() + "' not a git repository"); + rp = new org.spearce.jgit.transport.ReceivePack(db); + rp.receive(System.in, System.out, System.err); + } +} diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java index 1f3a136..d3e32b3 100644 --- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java @@ -43,6 +43,7 @@ import static org.spearce.jgit.lib.Constants.R_TAGS; import java.io.BufferedWriter; +import java.io.File; import java.io.IOException; import java.io.OutputStreamWriter; import java.io.PrintWriter; @@ -77,6 +78,9 @@ /** Git repository the command was invoked within. */ protected Repository db; + /** Directory supplied via --git-dir command line option. */ + protected File gitdir; + /** RevWalk used during command line parsing, if it was required. */ protected RevWalk argWalk; @@ -84,10 +88,15 @@ final void setCommandName(final String name) { commandName = name; } - void init(final Repository repo) { + /** @return true if {@link #db}/{@link #getRepository()} is required. */ + protected boolean requiresRepository() { + return true; + } + + void init(final Repository repo, final File gd) { try { - String outputEncoding = repo.getConfig().getString("i18n", null, - "logOutputEncoding"); + final String outputEncoding = repo != null ? repo.getConfig() + .getString("i18n", null, "logOutputEncoding") : null; if (outputEncoding != null) out = new PrintWriter(new BufferedWriter( new OutputStreamWriter(System.out, outputEncoding))); @@ -97,7 +106,14 @@ void init(final Repository repo) { } catch (IOException e) { throw die("cannot create output stream"); } - db = repo; + + if (repo != null) { + db = repo; + gitdir = repo.getDirectory(); + } else { + db = null; + gitdir = gd; + } } /** -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [JGIT PATCH 2/5] Add "jgit init" command to create a new repository 2008-12-23 18:03 ` [JGIT PATCH 1/5] Add "jgit receive-pack" and permit commands to start not in a repository Shawn O. Pearce @ 2008-12-23 18:03 ` Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 3/5] Modify "jgit daemon" so it can run outside of a repository Shawn O. Pearce 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../src/org/spearce/jgit/pgm/Init.java | 60 ++++++++++++++++++++ 2 files changed, 61 insertions(+), 0 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin index 0e7e7d8..4a2b605 100644 --- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin +++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin @@ -4,6 +4,7 @@ org.spearce.jgit.pgm.DiffTree org.spearce.jgit.pgm.Fetch org.spearce.jgit.pgm.Glog org.spearce.jgit.pgm.IndexPack +org.spearce.jgit.pgm.Init org.spearce.jgit.pgm.Log org.spearce.jgit.pgm.LsRemote org.spearce.jgit.pgm.LsTree diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java new file mode 100644 index 0000000..197864d --- /dev/null +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Init.java @@ -0,0 +1,60 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.pgm; + +import java.io.File; + +import org.spearce.jgit.lib.Repository; + +@Command(common = true, usage = "Create an empty git repository") +class Init extends TextBuiltin { + @Override + protected final boolean requiresRepository() { + return false; + } + + @Override + protected void run() throws Exception { + if (gitdir == null) + gitdir = new File(".git"); + db = new Repository(gitdir); + db.create(); + out.println("Initialized empty Git repository in " + + gitdir.getAbsolutePath()); + } +} -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [JGIT PATCH 3/5] Modify "jgit daemon" so it can run outside of a repository 2008-12-23 18:03 ` [JGIT PATCH 2/5] Add "jgit init" command to create a new repository Shawn O. Pearce @ 2008-12-23 18:03 ` Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Shawn O. Pearce 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git We no longer export the current repository, but instead export the directory trees given. This behavior is more closely matches that of "git daemon". Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../src/org/spearce/jgit/pgm/Daemon.java | 33 +++++-------------- 1 files changed, 9 insertions(+), 24 deletions(-) diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java index aafc82e..39b43b2 100644 --- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Daemon.java @@ -44,7 +44,6 @@ import org.kohsuke.args4j.Argument; import org.kohsuke.args4j.Option; -import org.spearce.jgit.lib.Repository; import org.spearce.jgit.transport.DaemonService; @Command(common = true, usage = "Export repositories over git://") @@ -67,10 +66,15 @@ @Option(name = "--forbid-override", metaVar = "SERVICE", usage = "configure the service in daemon.servicename", multiValued = true) final List<String> forbidOverride = new ArrayList<String>(); - @Argument(metaVar = "DIRECTORY", usage = "directories to export") + @Argument(required = true, metaVar = "DIRECTORY", usage = "directories to export") final List<File> directory = new ArrayList<File>(); @Override + protected boolean requiresRepository() { + return false; + } + + @Override protected void run() throws Exception { final org.spearce.jgit.transport.Daemon d; @@ -88,13 +92,9 @@ protected void run() throws Exception { for (final String n : forbidOverride) service(d, n).setOverridable(false); - if (directory.isEmpty()) { - export(d, db); - } else { - for (final File f : directory) { - out.println("Exporting " + f.getAbsolutePath()); - d.exportDirectory(f); - } + for (final File f : directory) { + out.println("Exporting " + f.getAbsolutePath()); + d.exportDirectory(f); } d.start(); out.println("Listening on " + d.getAddress()); @@ -107,19 +107,4 @@ private DaemonService service(final org.spearce.jgit.transport.Daemon d, throw die("Service '" + n + "' not supported"); return svc; } - - private void export(final org.spearce.jgit.transport.Daemon daemon, - final Repository repo) { - File d = repo.getDirectory(); - String name = d.getName(); - while (name.equals(".git") || name.equals(".")) { - d = d.getParentFile(); - name = d.getName(); - } - if (!name.endsWith(".git")) - name += ".git"; - - out.println("Exporting current repository as \"" + name + "\""); - daemon.exportRepository(name, repo); - } } -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific 2008-12-23 18:03 ` [JGIT PATCH 3/5] Modify "jgit daemon" so it can run outside of a repository Shawn O. Pearce @ 2008-12-23 18:03 ` Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 5/5] Fix "fetch pulled too many objects" when auto-following tags Shawn O. Pearce 2008-12-31 7:12 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Robin Rosenberg 0 siblings, 2 replies; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git The amazon-s3:// protocol is unique to JGit, and is not supported by any other Git implementation. The easiest way to clone off of one of those URLs is to use "jgit clone". This program also acts as a simple example for any IDE developers who want to implement a clone feature, as it is the minimum effort required to make a user reasonably happy with the resulting repository. Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../services/org.spearce.jgit.pgm.TextBuiltin | 1 + .../org/spearce/jgit/pgm/AbstractFetchCommand.java | 126 ++++++++++++++ .../src/org/spearce/jgit/pgm/Clone.java | 177 ++++++++++++++++++++ .../src/org/spearce/jgit/pgm/Fetch.java | 83 +--------- 4 files changed, 306 insertions(+), 81 deletions(-) create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java create mode 100644 org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java diff --git a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin index 4a2b605..40177f9 100644 --- a/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin +++ b/org.spearce.jgit.pgm/src/META-INF/services/org.spearce.jgit.pgm.TextBuiltin @@ -1,4 +1,5 @@ org.spearce.jgit.pgm.Branch +org.spearce.jgit.pgm.Clone org.spearce.jgit.pgm.Daemon org.spearce.jgit.pgm.DiffTree org.spearce.jgit.pgm.Fetch diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java new file mode 100644 index 0000000..ea6f277 --- /dev/null +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/AbstractFetchCommand.java @@ -0,0 +1,126 @@ +/* + * Copyright (C) 2008, Shawn O. Pearce <spearce@spearce.org> + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.pgm; + +import org.kohsuke.args4j.Option; +import org.spearce.jgit.lib.Constants; +import org.spearce.jgit.lib.RefUpdate; +import org.spearce.jgit.transport.FetchResult; +import org.spearce.jgit.transport.TrackingRefUpdate; +import org.spearce.jgit.transport.Transport; + +abstract class AbstractFetchCommand extends TextBuiltin { + @Option(name = "--verbose", aliases = { "-v" }, usage = "be more verbose") + private boolean verbose; + + protected void showFetchResult(final Transport tn, final FetchResult r) { + boolean shownURI = false; + for (final TrackingRefUpdate u : r.getTrackingRefUpdates()) { + if (!verbose && u.getResult() == RefUpdate.Result.NO_CHANGE) + continue; + + final char type = shortTypeOf(u.getResult()); + final String longType = longTypeOf(u); + final String src = abbreviateRef(u.getRemoteName(), false); + final String dst = abbreviateRef(u.getLocalName(), true); + + if (!shownURI) { + out.print("From "); + out.print(tn.getURI()); + out.println(); + shownURI = true; + } + + out.format(" %c %-17s %-10s -> %s", type, longType, src, dst); + out.println(); + } + } + + private String longTypeOf(final TrackingRefUpdate u) { + final RefUpdate.Result r = u.getResult(); + if (r == RefUpdate.Result.LOCK_FAILURE) + return "[lock fail]"; + + if (r == RefUpdate.Result.IO_FAILURE) + return "[i/o error]"; + + if (r == RefUpdate.Result.NEW) { + if (u.getRemoteName().startsWith(Constants.R_HEADS)) + return "[new branch]"; + else if (u.getLocalName().startsWith(Constants.R_TAGS)) + return "[new tag]"; + return "[new]"; + } + + if (r == RefUpdate.Result.FORCED) { + final String aOld = u.getOldObjectId().abbreviate(db).name(); + final String aNew = u.getNewObjectId().abbreviate(db).name(); + return aOld + "..." + aNew; + } + + if (r == RefUpdate.Result.FAST_FORWARD) { + final String aOld = u.getOldObjectId().abbreviate(db).name(); + final String aNew = u.getNewObjectId().abbreviate(db).name(); + return aOld + ".." + aNew; + } + + if (r == RefUpdate.Result.REJECTED) + return "[rejected]"; + if (r == RefUpdate.Result.NO_CHANGE) + return "[up to date]"; + return "[" + r.name() + "]"; + } + + private static char shortTypeOf(final RefUpdate.Result r) { + if (r == RefUpdate.Result.LOCK_FAILURE) + return '!'; + if (r == RefUpdate.Result.IO_FAILURE) + return '!'; + if (r == RefUpdate.Result.NEW) + return '*'; + if (r == RefUpdate.Result.FORCED) + return '+'; + if (r == RefUpdate.Result.FAST_FORWARD) + return ' '; + if (r == RefUpdate.Result.REJECTED) + return '!'; + if (r == RefUpdate.Result.NO_CHANGE) + return '='; + return ' '; + } +} diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java new file mode 100644 index 0000000..51dd95d --- /dev/null +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Clone.java @@ -0,0 +1,177 @@ +/* + * Copyright (C) 2008, Google Inc. + * + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials provided + * with the distribution. + * + * - Neither the name of the Git Development Community nor the + * names of its contributors may be used to endorse or promote + * products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND + * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; + * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER + * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF + * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +package org.spearce.jgit.pgm; + +import java.io.File; +import java.io.IOException; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +import org.kohsuke.args4j.Argument; +import org.kohsuke.args4j.Option; +import org.spearce.jgit.errors.NotSupportedException; +import org.spearce.jgit.errors.TransportException; +import org.spearce.jgit.lib.Commit; +import org.spearce.jgit.lib.Constants; +import org.spearce.jgit.lib.GitIndex; +import org.spearce.jgit.lib.Ref; +import org.spearce.jgit.lib.RefComparator; +import org.spearce.jgit.lib.RefUpdate; +import org.spearce.jgit.lib.Repository; +import org.spearce.jgit.lib.TextProgressMonitor; +import org.spearce.jgit.lib.Tree; +import org.spearce.jgit.lib.WorkDirCheckout; +import org.spearce.jgit.transport.FetchResult; +import org.spearce.jgit.transport.RefSpec; +import org.spearce.jgit.transport.RemoteConfig; +import org.spearce.jgit.transport.Transport; +import org.spearce.jgit.transport.URIish; + +@Command(common = true, usage = "Clone a repository into a new directory") +class Clone extends AbstractFetchCommand { + @Option(name = "--origin", aliases = { "-o" }, metaVar = "name", usage = "use <name> instead of 'origin' to track upstream") + private String remoteName = "origin"; + + @Argument(index = 0, required = true, metaVar = "uri-ish") + private String sourceUri; + + @Argument(index = 1, metaVar = "directory") + private String localName; + + @Override + protected final boolean requiresRepository() { + return false; + } + + @Override + protected void run() throws Exception { + if (localName != null && gitdir != null) + throw die("conflicting usage of --git-dir and arguments"); + + final URIish uri = new URIish(sourceUri); + if (localName == null) { + String p = uri.getPath(); + while (p.endsWith("/")) + p = p.substring(0, p.length() - 1); + final int s = p.lastIndexOf('/'); + if (s < 0) + throw die("cannot guess local name from " + sourceUri); + localName = p.substring(s + 1); + if (localName.endsWith(".git")) + localName = localName.substring(0, localName.length() - 4); + } + if (gitdir == null) + gitdir = new File(localName, ".git"); + db = new Repository(gitdir); + db.create(); + out.println("Initialized empty Git repository in " + + gitdir.getAbsolutePath()); + out.flush(); + + saveRemote(uri); + final FetchResult r = runFetch(); + final Ref branch = guessHEAD(r); + doCheckout(branch); + } + + private void saveRemote(final URIish uri) throws URISyntaxException, + IOException { + final RemoteConfig rc = new RemoteConfig(db.getConfig(), remoteName); + rc.addURI(uri); + rc.addFetchRefSpec(new RefSpec().setForceUpdate(true) + .setSourceDestination(Constants.R_HEADS + "*", + Constants.R_REMOTES + remoteName + "/*")); + rc.update(db.getConfig()); + db.getConfig().save(); + } + + private FetchResult runFetch() throws NotSupportedException, + URISyntaxException, TransportException { + final Transport tn = Transport.open(db, remoteName); + final FetchResult r; + try { + r = tn.fetch(new TextProgressMonitor(), null); + } finally { + tn.close(); + } + showFetchResult(tn, r); + return r; + } + + private Ref guessHEAD(final FetchResult result) { + final Ref idHEAD = result.getAdvertisedRef(Constants.HEAD); + final List<Ref> availableRefs = new ArrayList<Ref>(); + Ref head = null; + for (final Ref r : result.getAdvertisedRefs()) { + final String n = r.getName(); + if (!n.startsWith(Constants.R_HEADS)) + continue; + availableRefs.add(r); + if (idHEAD == null || head != null) + continue; + if (r.getObjectId().equals(idHEAD.getObjectId())) + head = r; + } + Collections.sort(availableRefs, RefComparator.INSTANCE); + if (idHEAD != null && head == null) + head = idHEAD; + return head; + } + + private void doCheckout(final Ref branch) throws IOException { + if (branch == null) + throw die("cannot checkout; no HEAD advertised by remote"); + if (!Constants.HEAD.equals(branch.getName())) + db.writeSymref(Constants.HEAD, branch.getName()); + + final Commit commit = db.mapCommit(branch.getObjectId()); + final RefUpdate u = db.updateRef(Constants.HEAD); + u.setNewObjectId(commit.getCommitId()); + u.forceUpdate(); + + final GitIndex index = new GitIndex(db); + final Tree tree = commit.getTree(); + final WorkDirCheckout co; + + co = new WorkDirCheckout(db, db.getWorkDir(), index, tree); + co.checkout(); + index.write(); + } +} diff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java index e9d3260..8f3f7d5 100644 --- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java +++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Fetch.java @@ -41,19 +41,13 @@ import org.kohsuke.args4j.Argument; import org.kohsuke.args4j.Option; -import org.spearce.jgit.lib.Constants; -import org.spearce.jgit.lib.RefUpdate; import org.spearce.jgit.lib.TextProgressMonitor; import org.spearce.jgit.transport.FetchResult; import org.spearce.jgit.transport.RefSpec; -import org.spearce.jgit.transport.TrackingRefUpdate; import org.spearce.jgit.transport.Transport; @Command(common = true, usage = "Update remote refs from another repository") -class Fetch extends TextBuiltin { - @Option(name = "--verbose", aliases = { "-v" }, usage = "be more verbose") - private boolean verbose; - +class Fetch extends AbstractFetchCommand { @Option(name = "--fsck", usage = "perform fsck style checks on receive") private Boolean fsck; @@ -91,79 +85,6 @@ protected void run() throws Exception { } finally { tn.close(); } - - boolean shownURI = false; - for (final TrackingRefUpdate u : r.getTrackingRefUpdates()) { - if (!verbose && u.getResult() == RefUpdate.Result.NO_CHANGE) - continue; - - final char type = shortTypeOf(u.getResult()); - final String longType = longTypeOf(u); - final String src = abbreviateRef(u.getRemoteName(), false); - final String dst = abbreviateRef(u.getLocalName(), true); - - if (!shownURI) { - out.print("From "); - out.print(tn.getURI()); - out.println(); - shownURI = true; - } - - out.format(" %c %-17s %-10s -> %s", type, longType, src, dst); - out.println(); - } - } - - private String longTypeOf(final TrackingRefUpdate u) { - final RefUpdate.Result r = u.getResult(); - if (r == RefUpdate.Result.LOCK_FAILURE) - return "[lock fail]"; - - if (r == RefUpdate.Result.IO_FAILURE) - return "[i/o error]"; - - if (r == RefUpdate.Result.NEW) { - if (u.getRemoteName().startsWith(Constants.R_HEADS)) - return "[new branch]"; - else if (u.getLocalName().startsWith(Constants.R_TAGS)) - return "[new tag]"; - return "[new]"; - } - - if (r == RefUpdate.Result.FORCED) { - final String aOld = u.getOldObjectId().abbreviate(db).name(); - final String aNew = u.getNewObjectId().abbreviate(db).name(); - return aOld + "..." + aNew; - } - - if (r == RefUpdate.Result.FAST_FORWARD) { - final String aOld = u.getOldObjectId().abbreviate(db).name(); - final String aNew = u.getNewObjectId().abbreviate(db).name(); - return aOld + ".." + aNew; - } - - if (r == RefUpdate.Result.REJECTED) - return "[rejected]"; - if (r == RefUpdate.Result.NO_CHANGE) - return "[up to date]"; - return "[" + r.name() + "]"; - } - - private static char shortTypeOf(final RefUpdate.Result r) { - if (r == RefUpdate.Result.LOCK_FAILURE) - return '!'; - if (r == RefUpdate.Result.IO_FAILURE) - return '!'; - if (r == RefUpdate.Result.NEW) - return '*'; - if (r == RefUpdate.Result.FORCED) - return '+'; - if (r == RefUpdate.Result.FAST_FORWARD) - return ' '; - if (r == RefUpdate.Result.REJECTED) - return '!'; - if (r == RefUpdate.Result.NO_CHANGE) - return '='; - return ' '; + showFetchResult(tn, r); } } -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 13+ messages in thread
* [JGIT PATCH 5/5] Fix "fetch pulled too many objects" when auto-following tags 2008-12-23 18:03 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Shawn O. Pearce @ 2008-12-23 18:03 ` Shawn O. Pearce 2008-12-31 7:12 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Robin Rosenberg 1 sibling, 0 replies; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-23 18:03 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git If we don't take into consideration the objects obtained during the first connection when we open a second to auto-follow tags we will download a large chunk of the repository a second time. This is very wasteful of network bandwidth, and is an abuse of the server. Because we delay all ref updates until the very end of the fetch process we need to hold onto the set of objects we requested in the first connection, and pass that set into the subsequent one so it can be considered reachable. Issue: http://code.google.com/p/egit/issues/detail?id=22 Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- .../jgit/transport/BaseFetchConnection.java | 26 ++++++++++++------- .../jgit/transport/BasePackFetchConnection.java | 25 +++++++++++++++---- .../spearce/jgit/transport/FetchConnection.java | 21 +++++++++++---- .../org/spearce/jgit/transport/FetchProcess.java | 6 ++++- .../spearce/jgit/transport/TransportBundle.java | 3 +- .../jgit/transport/WalkFetchConnection.java | 14 ++++++++-- 6 files changed, 69 insertions(+), 26 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java index 6709bfc..bb81296 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BaseFetchConnection.java @@ -38,8 +38,10 @@ package org.spearce.jgit.transport; import java.util.Collection; +import java.util.Set; import org.spearce.jgit.errors.TransportException; +import org.spearce.jgit.lib.ObjectId; import org.spearce.jgit.lib.ProgressMonitor; import org.spearce.jgit.lib.Ref; @@ -54,9 +56,10 @@ abstract class BaseFetchConnection extends BaseConnection implements FetchConnection { public final void fetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException { + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException { markStartedOperation(); - doFetch(monitor, want); + doFetch(monitor, want, have); } /** @@ -68,19 +71,22 @@ public boolean didFetchIncludeTags() { } /** - * Implementation of {@link #fetch(ProgressMonitor, Collection)} without - * checking for multiple fetch. + * Implementation of {@link #fetch(ProgressMonitor, Collection, Set)} + * without checking for multiple fetch. * * @param monitor - * as in {@link #fetch(ProgressMonitor, Collection)} + * as in {@link #fetch(ProgressMonitor, Collection, Set)} * @param want - * as in {@link #fetch(ProgressMonitor, Collection)} + * as in {@link #fetch(ProgressMonitor, Collection, Set)} + * @param have + * as in {@link #fetch(ProgressMonitor, Collection, Set)} * @throws TransportException - * as in {@link #fetch(ProgressMonitor, Collection)}, but + * as in {@link #fetch(ProgressMonitor, Collection, Set)}, but * implementation doesn't have to care about multiple - * {@link #fetch(ProgressMonitor, Collection)} calls, as it is - * checked in this class. + * {@link #fetch(ProgressMonitor, Collection, Set)} calls, as it + * is checked in this class. */ protected abstract void doFetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException; + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException; } diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java index 542a8a9..2cb9b64 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/BasePackFetchConnection.java @@ -41,10 +41,12 @@ import java.io.IOException; import java.util.Collection; import java.util.Date; +import java.util.Set; import org.spearce.jgit.errors.TransportException; import org.spearce.jgit.lib.AnyObjectId; import org.spearce.jgit.lib.MutableObjectId; +import org.spearce.jgit.lib.ObjectId; import org.spearce.jgit.lib.ProgressMonitor; import org.spearce.jgit.lib.Ref; import org.spearce.jgit.revwalk.RevCommit; @@ -137,9 +139,10 @@ BasePackFetchConnection(final PackTransport packTransport) { } public final void fetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException { + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException { markStartedOperation(); - doFetch(monitor, want); + doFetch(monitor, want, have); } public boolean didFetchIncludeTags() { @@ -151,10 +154,11 @@ public boolean didFetchTestConnectivity() { } protected void doFetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException { + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException { try { markRefsAdvertised(); - markReachable(maxTimeWanted(want)); + markReachable(have, maxTimeWanted(want)); if (sendWants(want)) { negotiate(monitor); @@ -193,7 +197,8 @@ private int maxTimeWanted(final Collection<Ref> wants) { return maxTime; } - private void markReachable(final int maxTime) throws IOException { + private void markReachable(final Set<ObjectId> have, final int maxTime) + throws IOException { for (final Ref r : local.getAllRefs().values()) { try { final RevCommit o = walk.parseCommit(r.getObjectId()); @@ -204,6 +209,16 @@ private void markReachable(final int maxTime) throws IOException { } } + for (final ObjectId id : have) { + try { + final RevCommit o = walk.parseCommit(id); + o.add(REACHABLE); + reachableCommits.add(o); + } catch (IOException readError) { + // If we cannot read the value of the ref skip it. + } + } + if (maxTime > 0) { // Mark reachable commits until we reach maxTime. These may // wind up later matching up against things we want and we diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java index a56ca6c..61ef219 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchConnection.java @@ -38,8 +38,10 @@ package org.spearce.jgit.transport; import java.util.Collection; +import java.util.Set; import org.spearce.jgit.errors.TransportException; +import org.spearce.jgit.lib.ObjectId; import org.spearce.jgit.lib.ProgressMonitor; import org.spearce.jgit.lib.Ref; @@ -85,23 +87,29 @@ * @param want * one or more refs advertised by this connection that the caller * wants to store locally. + * @param have + * additional objects known to exist in the destination + * repository, especially if they aren't yet reachable by the ref + * database. Connections should take this set as an addition to + * what is reachable through all Refs, not in replace of it. * @throws TransportException * objects could not be copied due to a network failure, * protocol error, or error on remote side, or connection was * already used for fetch. */ - public void fetch(final ProgressMonitor monitor, final Collection<Ref> want) + public void fetch(final ProgressMonitor monitor, + final Collection<Ref> want, final Set<ObjectId> have) throws TransportException; /** - * Did the last {@link #fetch(ProgressMonitor, Collection)} get tags? + * Did the last {@link #fetch(ProgressMonitor, Collection, Set)} get tags? * <p> * Some Git aware transports are able to implicitly grab an annotated tag if * {@link TagOpt#AUTO_FOLLOW} or {@link TagOpt#FETCH_TAGS} was selected and * the object the tag peels to (references) was transferred as part of the - * last {@link #fetch(ProgressMonitor, Collection)} call. If it is possible - * for such tags to have been included in the transfer this method returns - * true, allowing the caller to attempt tag discovery. + * last {@link #fetch(ProgressMonitor, Collection, Set)} call. If it is + * possible for such tags to have been included in the transfer this method + * returns true, allowing the caller to attempt tag discovery. * <p> * By returning only true/false (and not the actual list of tags obtained) * the transport itself does not need to be aware of whether or not tags @@ -113,7 +121,8 @@ public void fetch(final ProgressMonitor monitor, final Collection<Ref> want) public boolean didFetchIncludeTags(); /** - * Did the last {@link #fetch(ProgressMonitor, Collection)} validate graph? + * Did the last {@link #fetch(ProgressMonitor, Collection, Set)} validate + * graph? * <p> * Some transports walk the object graph on the client side, with the client * looking for what objects it is missing and requesting them individually diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java index bb2d051..09718eb 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/FetchProcess.java @@ -75,6 +75,9 @@ /** Set of refs we will actually wind up asking to obtain. */ private final HashMap<ObjectId, Ref> askFor = new HashMap<ObjectId, Ref>(); + /** Objects we know we have locally. */ + private final HashSet<ObjectId> have = new HashSet<ObjectId>(); + /** Updates to local tracking branches (if any). */ private final ArrayList<TrackingRefUpdate> localUpdates = new ArrayList<TrackingRefUpdate>(); @@ -133,6 +136,7 @@ else if (tagopt == TagOpt.FETCH_TAGS) // There are more tags that we want to follow, but // not all were asked for on the initial request. // + have.addAll(askFor.keySet()); askFor.clear(); for (final Ref r : additionalTags) { final ObjectId id = r.getPeeledObjectId(); @@ -173,7 +177,7 @@ else if (tagopt == TagOpt.FETCH_TAGS) private void fetchObjects(final ProgressMonitor monitor) throws TransportException { - conn.fetch(monitor, askFor.values()); + conn.fetch(monitor, askFor.values(), have); if (transport.isCheckFetchedObjects() && !conn.didFetchTestConnectivity() && !askForIsComplete()) throw new TransportException(transport.getURI(), diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java index 7d38b02..1734d94 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportBundle.java @@ -171,7 +171,8 @@ public boolean didFetchTestConnectivity() { @Override protected void doFetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException { + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException { verifyPrerequisites(); try { final IndexPack ip = newIndexPack(); diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java index d089f7b..91c5ea8 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/WalkFetchConnection.java @@ -195,8 +195,9 @@ public boolean didFetchTestConnectivity() { @Override protected void doFetch(final ProgressMonitor monitor, - final Collection<Ref> want) throws TransportException { - markLocalRefsComplete(); + final Collection<Ref> want, final Set<ObjectId> have) + throws TransportException { + markLocalRefsComplete(have); queueWants(want); while (!monitor.isCancelled() && !workQueue.isEmpty()) { @@ -642,7 +643,7 @@ private void saveLooseObject(final AnyObjectId id, final byte[] compressed) return null; } - private void markLocalRefsComplete() throws TransportException { + private void markLocalRefsComplete(final Set<ObjectId> have) throws TransportException { for (final Ref r : local.getAllRefs().values()) { try { markLocalObjComplete(revWalk.parseAny(r.getObjectId())); @@ -651,6 +652,13 @@ private void markLocalRefsComplete() throws TransportException { + " is missing object(s).", readError); } } + for (final ObjectId id : have) { + try { + markLocalObjComplete(revWalk.parseAny(id)); + } catch (IOException readError) { + throw new TransportException("Missing assumed "+id.name(), readError); + } + } } private void markLocalObjComplete(RevObject obj) throws IOException { -- 1.6.1.rc4.301.g5497a ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific 2008-12-23 18:03 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 5/5] Fix "fetch pulled too many objects" when auto-following tags Shawn O. Pearce @ 2008-12-31 7:12 ` Robin Rosenberg 2008-12-31 7:35 ` Shawn O. Pearce 1 sibling, 1 reply; 13+ messages in thread From: Robin Rosenberg @ 2008-12-31 7:12 UTC (permalink / raw) To: Shawn O. Pearce; +Cc: git Would it be hard to get the progress look better? -- robin EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2 Initialized empty Git repository in /home/me/SW/LIBGIT2/.git Counting objects: 547 Compressing objects: 100% (192/192) ts: 100% (192/192) Compressing objects: 100% (192/192) ng objects: 100% (192/192) Compressing objects: 100% (192/192) bjects: 100% (192/192) Compressing objects: 100% (192/192) pressing objects: 100% (192/192) ts: 100% (192/192) Compressing objects: 100% (192/192) Receiving objects: 100% (547/547) jects: 100% (192/192) Compressing objects: 100% (192/192) ressing objects: 100% (192/192) s: 100% (192/192) Compressing objects: 100% (192/192) ssing objects: 100% (192/192) ts: 100% (192/192) Compressing objects: 100% (192/192) objects: 100% (192/192) Compressing objects: 100% (192/192) essing objects: 100% (192/192) cts: 100% (192/192) Compressing objects: 100% (192/192) g objects: 100% (192/192) Compressing objects: 100% (192/192) ressing objects: 100% (192/192) Resolving deltas: 100% (358/358) From git://repo.or.cz/libgit2.git * [new branch] master -> origin/master ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific 2008-12-31 7:12 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Robin Rosenberg @ 2008-12-31 7:35 ` Shawn O. Pearce 2008-12-31 19:04 ` [JGIT PATCH] Improve the sideband progress scaper to be more accurate Shawn O. Pearce 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-31 7:35 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git Robin Rosenberg <robin.rosenberg@dewire.com> wrote: > Would it be hard to get the progress look better? This happens because we are scanning the progress channel wrong in some cases. Its random; most of the time we can get it right, but every once in a while we get out of sync and get these short names. This is a general problem with the progress client side code, not the new server side code I've introduced. We also see it going against a standard C git daemon. I haven't been able to figure out a way to smooth out the messages. FWIW, git-gui also gets this sort of progress corruption sometimes when it does a clone. So I've reproduced the same parsing bug in two different languages and two different implementations... ;-) > EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2 > Initialized empty Git repository in /home/me/SW/LIBGIT2/.git > Counting objects: 547 > Compressing objects: 100% (192/192) > ts: 100% (192/192) > Compressing objects: 100% (192/192) > ng objects: 100% (192/192) > Compressing objects: 100% (192/192) > bjects: 100% (192/192) > Compressing objects: 100% (192/192) > pressing objects: 100% (192/192) > ts: 100% (192/192) > Compressing objects: 100% (192/192) > Receiving objects: 100% (547/547) > jects: 100% (192/192) > Compressing objects: 100% (192/192) > ressing objects: 100% (192/192) > s: 100% (192/192) > Compressing objects: 100% (192/192) > ssing objects: 100% (192/192) > ts: 100% (192/192) > Compressing objects: 100% (192/192) > objects: 100% (192/192) > Compressing objects: 100% (192/192) > essing objects: 100% (192/192) > cts: 100% (192/192) > Compressing objects: 100% (192/192) > g objects: 100% (192/192) > Compressing objects: 100% (192/192) > ressing objects: 100% (192/192) > Resolving deltas: 100% (358/358) > From git://repo.or.cz/libgit2.git > * [new branch] master -> origin/master -- Shawn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* [JGIT PATCH] Improve the sideband progress scaper to be more accurate 2008-12-31 7:35 ` Shawn O. Pearce @ 2008-12-31 19:04 ` Shawn O. Pearce 2009-01-03 22:12 ` Robin Rosenberg 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2008-12-31 19:04 UTC (permalink / raw) To: Robin Rosenberg; +Cc: git By matching only whole lines we should be able to improve the progress scaper so we avoid ugly output like we had been seeing: EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2 Initialized empty Git repository in /home/me/SW/LIBGIT2/.git Counting objects: 547 Compressing objects: 100% (192/192) ts: 100% (192/192) Compressing objects: 100% (192/192) ng objects: 100% (192/192) Signed-off-by: Shawn O. Pearce <spearce@spearce.org> --- Robin Rosenberg <robin.rosenberg@dewire.com> wrote: > Would it be hard to get the progress look better? Maybe this does the trick. Its hard to reproduce so its hard to come up with the condition that was giving us the problem before. I suspect its because we were getting line fragments on the sideband channel, but I'm not sure that was really the case. .../jgit/transport/SideBandInputStream.java | 37 +++++++++++++++++-- 1 files changed, 33 insertions(+), 4 deletions(-) diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java b/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java index 3ec9bff..f0ba3d3 100644 --- a/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java +++ b/org.spearce.jgit/src/org/spearce/jgit/transport/SideBandInputStream.java @@ -73,10 +73,10 @@ private static final int CH_ERROR = 3; private static Pattern P_UNBOUNDED = Pattern.compile( - ".*?([\\w ]+): (\\d+)(, done)?.*", Pattern.DOTALL); + "^([\\w ]+): (\\d+)( |, done)?.*", Pattern.DOTALL); private static Pattern P_BOUNDED = Pattern.compile( - ".*?([\\w ]+):.*\\((\\d+)/(\\d+)\\).*", Pattern.DOTALL); + "^([\\w ]+):.*\\((\\d+)/(\\d+)\\).*", Pattern.DOTALL); private final PacketLineIn pckIn; @@ -84,6 +84,8 @@ private final ProgressMonitor monitor; + private String progressBuffer = ""; + private String currentTask; private int lastCnt; @@ -160,7 +162,31 @@ private void needDataPacket() throws IOException { } } - private void progress(final String msg) { + private void progress(String pkt) { + pkt = progressBuffer + pkt; + for (;;) { + final int lf = pkt.indexOf('\n'); + final int cr = pkt.indexOf('\r'); + final int s; + if (0 <= lf && 0 <= cr) + s = Math.min(lf, cr); + else if (0 <= lf) + s = lf; + else if (0 <= cr) + s = cr; + else + break; + + final String msg = pkt.substring(0, s); + if (doProgressLine(msg)) + pkt = pkt.substring(s + 1); + else + break; + } + progressBuffer = pkt; + } + + private boolean doProgressLine(final String msg) { Matcher matcher; matcher = P_BOUNDED.matcher(msg); @@ -175,7 +201,7 @@ private void progress(final String msg) { final int cnt = Integer.parseInt(matcher.group(2)); monitor.update(cnt - lastCnt); lastCnt = cnt; - return; + return true; } matcher = P_UNBOUNDED.matcher(msg); @@ -189,7 +215,10 @@ private void progress(final String msg) { final int cnt = Integer.parseInt(matcher.group(2)); monitor.update(cnt - lastCnt); lastCnt = cnt; + return true; } + + return false; } private String readString(final int len) throws IOException { -- 1.6.1.59.g6f6746 -- Shawn. ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate 2008-12-31 19:04 ` [JGIT PATCH] Improve the sideband progress scaper to be more accurate Shawn O. Pearce @ 2009-01-03 22:12 ` Robin Rosenberg 2009-01-06 19:23 ` Nicolas Pitre 0 siblings, 1 reply; 13+ messages in thread From: Robin Rosenberg @ 2009-01-03 22:12 UTC (permalink / raw) To: Shawn O. Pearce; +Cc: git onsdag 31 december 2008 20:04:01 skrev Shawn O. Pearce: > By matching only whole lines we should be able to improve the > progress scaper so we avoid ugly output like we had been seeing: > > EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2 > Initialized empty Git repository in /home/me/SW/LIBGIT2/.git > Counting objects: 547 > Compressing objects: 100% (192/192) > ts: 100% (192/192) > Compressing objects: 100% (192/192) > ng objects: 100% (192/192) > > Signed-off-by: Shawn O. Pearce <spearce@spearce.org> > --- > Robin Rosenberg <robin.rosenberg@dewire.com> wrote: > > Would it be hard to get the progress look better? > > Maybe this does the trick. Its hard to reproduce so its hard to > come up with the condition that was giving us the problem before. > I suspect its because we were getting line fragments on the sideband > channel, but I'm not sure that was really the case. Nasty. I couldn't reproduce it myself. I'll hold onto this one for a while and see if I'll get the opportunity to test it live with this problem. -- robin ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate 2009-01-03 22:12 ` Robin Rosenberg @ 2009-01-06 19:23 ` Nicolas Pitre 2009-01-06 19:27 ` Shawn O. Pearce 0 siblings, 1 reply; 13+ messages in thread From: Nicolas Pitre @ 2009-01-06 19:23 UTC (permalink / raw) To: Robin Rosenberg; +Cc: Shawn O. Pearce, git On Sat, 3 Jan 2009, Robin Rosenberg wrote: > onsdag 31 december 2008 20:04:01 skrev Shawn O. Pearce: > > By matching only whole lines we should be able to improve the > > progress scaper so we avoid ugly output like we had been seeing: > > > > EGIT.contrib/jgit clone git://repo.or.cz/libgit2.git LIBGIT2 > > Initialized empty Git repository in /home/me/SW/LIBGIT2/.git > > Counting objects: 547 > > Compressing objects: 100% (192/192) > > ts: 100% (192/192) > > Compressing objects: 100% (192/192) > > ng objects: 100% (192/192) > > > > Signed-off-by: Shawn O. Pearce <spearce@spearce.org> > > --- > > Robin Rosenberg <robin.rosenberg@dewire.com> wrote: > > > Would it be hard to get the progress look better? > > > > Maybe this does the trick. Its hard to reproduce so its hard to > > come up with the condition that was giving us the problem before. > > I suspect its because we were getting line fragments on the sideband > > channel, but I'm not sure that was really the case. > > Nasty. I couldn't reproduce it myself. I'll hold onto this one for a while and > see if I'll get the opportunity to test it live with this problem. There may indeed be line fragments sent over the sideband channel, as well as the opposite which is multiple lines sent at once in a single packet. If you look at sideband.c you'll find about all those cases. In general, what you have to do is: - for each packet: - split into multiple chunks on line breaks ('\r' or '\n') - for each chunk: - if last chunk didn't end with a line break, or if current chunk is empty or only contains a line break, then skip printing the "remote:" prefix. Otherwise print it. - print the current chunk up to any line break - if current chunk contains a line break and other characters then print a sequence to clear the remaining of the screen line - print the line break if any That should cover all cases. Note that the case where a previous chunk didn't end with a line break is always seen when moving from one packet to another, so that state cannot be local to the inner loop. Nicolas ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate 2009-01-06 19:23 ` Nicolas Pitre @ 2009-01-06 19:27 ` Shawn O. Pearce 2009-01-06 20:55 ` Nicolas Pitre 0 siblings, 1 reply; 13+ messages in thread From: Shawn O. Pearce @ 2009-01-06 19:27 UTC (permalink / raw) To: Nicolas Pitre; +Cc: Robin Rosenberg, git Nicolas Pitre <nico@cam.org> wrote: > There may indeed be line fragments sent over the sideband channel, as > well as the opposite which is multiple lines sent at once in a single > packet. If you look at sideband.c you'll find about all those cases. Thanks, that's what I thought... > In general, what you have to do is: > > - for each packet: > - split into multiple chunks on line breaks ('\r' or '\n') > - for each chunk: > - if last chunk didn't end with a line break, or if current > chunk is empty or only contains a line break, then skip printing > the "remote:" prefix. Otherwise print it. > - print the current chunk up to any line break > - if current chunk contains a line break and other characters then > print a sequence to clear the remaining of the screen line > - print the line break if any Hmm. I should note that C Git still screws this up sometimes. I've seen 1.6.1 git fetch mess up the output from repo.or.cz's sideband. I'm sure Pasky isn't running JGit's daemon, its too damn fast. :-) I don't have a spew of it handy, but late last week I saw it screw up while doing a clone off repo.or.cz. Robin, maybe we should apply this patch? Its along the lines Nico is describing above, but because our progress code doesn't do direct display its a little less complex, we don't have to worry about clearing the current line. But I think I'm doing the chunk splitting and filtering he's talking about. -- Shawn. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [JGIT PATCH] Improve the sideband progress scaper to be more accurate 2009-01-06 19:27 ` Shawn O. Pearce @ 2009-01-06 20:55 ` Nicolas Pitre 0 siblings, 0 replies; 13+ messages in thread From: Nicolas Pitre @ 2009-01-06 20:55 UTC (permalink / raw) To: Shawn O. Pearce; +Cc: Robin Rosenberg, git On Tue, 6 Jan 2009, Shawn O. Pearce wrote: > Nicolas Pitre <nico@cam.org> wrote: > > There may indeed be line fragments sent over the sideband channel, as > > well as the opposite which is multiple lines sent at once in a single > > packet. If you look at sideband.c you'll find about all those cases. > > Thanks, that's what I thought... > > > In general, what you have to do is: > > > > - for each packet: > > - split into multiple chunks on line breaks ('\r' or '\n') > > - for each chunk: > > - if last chunk didn't end with a line break, or if current > > chunk is empty or only contains a line break, then skip printing > > the "remote:" prefix. Otherwise print it. > > - print the current chunk up to any line break > > - if current chunk contains a line break and other characters then > > print a sequence to clear the remaining of the screen line > > - print the line break if any > > Hmm. I should note that C Git still screws this up sometimes. I've > seen 1.6.1 git fetch mess up the output from repo.or.cz's sideband. > I'm sure Pasky isn't running JGit's daemon, its too damn fast. :-) > > I don't have a spew of it handy, but late last week I saw it screw > up while doing a clone off repo.or.cz. Next time you see such things please send me a snapshot. What is still possible and IMHO not worth the effort to fix is some interaction between local and remote displays which, if intermixed in some unlucky way, do screw up the final appearance. For example, if the remote has sent a partial line, and before the remaining of it is received/printed you actually have some local process also displaying a line of its own. If that was to become a real issue, then the fix would involve caching any partial line sent from the remote until the associated line break is received. Nicolas ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2009-01-06 20:56 UTC | newest] Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2008-12-23 18:03 [JGIT PATCH 0/5] Add jgit init, clone, receive-pack; transport fixes Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 1/5] Add "jgit receive-pack" and permit commands to start not in a repository Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 2/5] Add "jgit init" command to create a new repository Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 3/5] Modify "jgit daemon" so it can run outside of a repository Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Shawn O. Pearce 2008-12-23 18:03 ` [JGIT PATCH 5/5] Fix "fetch pulled too many objects" when auto-following tags Shawn O. Pearce 2008-12-31 7:12 ` [JGIT PATCH 4/5] Add "jgit clone" to support cloning off URLs that are JGit specific Robin Rosenberg 2008-12-31 7:35 ` Shawn O. Pearce 2008-12-31 19:04 ` [JGIT PATCH] Improve the sideband progress scaper to be more accurate Shawn O. Pearce 2009-01-03 22:12 ` Robin Rosenberg 2009-01-06 19:23 ` Nicolas Pitre 2009-01-06 19:27 ` Shawn O. Pearce 2009-01-06 20:55 ` Nicolas Pitre
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.