[PoC PATCH JGIT 0/2] Proof of concept code for Git Freenet transport

[PoC PATCH JGIT 0/2] Proof of concept code for Git Freenet transport

[Daniel Cheng (aka SDiZ)]

Hi JGit / Freenet community,

Here is some proof-of-concept code for Git-over-Freenet.
I am sending this to see the feedback from communities.

The code need some more cleanups, so it it is not ready for apply (yet).

This is a real-life example

Push:
   $ git remote add fcp fcp://SSK@[my public key]^[my private key]/test.git
   $ ./jgit push fcp refs/remotes/origin/stable:refs/heads/master
 
 /ALTERNATIVLY/

   Insert a bare repository under USK@<.....>/test.git/-1/

Pull:
 $ ./jgit clone fcp://SSK@[my public key]^[my private key]/test.git


To workaround the metadata update problem, this client translate the
path seperator to "-", that means:

On push:
   objects/aa/bbbbbbbb   --> USK@..../test.git-objects-aa-bbbbbbb/-1/
   refs/heads/xxx        --> USK@..../test.git-objects-ref-heads-xxx/-1/

On pull:
   To support uploading from jSite, 
    when we load the info/refs we first check USK@..../test.git-info-refs/-1/
    if it is unavailiable, we would use USK@..../test.git/-1/info/refs

   The "traditional" type (USK@..../test.git/-1/objects) repository is
   always added as an alternative objects database. No other info/alternatives 
   are supported

FIXME:
 - How to store the private key of repository?
   Currently, we use URI of form fcp://SSK@<public key>^<private key>/some-id
   This is quite ugly. Could we use a per remote Config ? How can I get remote
   name from transport?

 - Make pushing async, could we?

[PoC PATCH JGIT 1/2] Freenet Freenet Client Protocol (FCP) 2.0 Client

[Daniel Cheng (aka SDiZ)]

Signed-off-by: Daniel Cheng (aka SDiZ) <j16sdiz+freenet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
---
 .../jgit/transport/fcpv2/FCPConnection.java        |  427 ++++++++++++++++++++
 1 files changed, 427 insertions(+), 0 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/fcpv2/FCPConnection.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/fcpv2/FCPConnection.java b/org.spearce.jgit/src/org/spearce/jgit/transport/fcpv2/FCPConnection.java
new file mode 100644
index 0000000..ce2ca90
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/fcpv2/FCPConnection.java
@@ -0,0 +1,427 @@
+/*
+ * Copyright (C) 2008, CHENG Yuk-Pong, Daniel <j16sdiz+freenet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.transport.fcpv2;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.spearce.jgit.lib.ProgressMonitor;
+
+/**
+ * Freenet Client Protocol 2.0 Connection
+ * 
+ * A simplified interface to Freenet 0.7 node using FCPv2. This is not a
+ * full-blown interface to Freenet node, but rather some "quick and dirty" hack
+ * just enough to upload/download simple file from the network.
+ * <p>
+ * See <a href="http://wiki.freenetproject.org/FreenetFCPSpec2Point0">Freenet
+ * Client Protocol 2.0 Specification</a> for detail on this protocol.
+ */
+public class FCPConnection {
+	/** Default FCP port */
+	public static final int DEFAULT_FCP_PORT = 9481;
+
+	protected InetAddress addr;
+
+	protected int port;
+
+	protected Socket socket;
+
+	protected InputStream is;
+
+	protected OutputStream os;
+
+	/**
+	 * Create a new FCP Connection to default host and port (
+	 * <code>localhost:9481</code>)
+	 * 
+	 * @throws UnknownHostException
+	 *             if no IP address for the <code>localhost</code> could be
+	 *             found.
+	 */
+	public FCPConnection() throws UnknownHostException {
+		this(InetAddress.getAllByName("127.0.0.1")[0], DEFAULT_FCP_PORT);
+	}
+
+	/**
+	 * Create a new FCP Connection to the specified IP address and port
+	 * 
+	 * @param address
+	 *            the node IP address
+	 * @param port
+	 *            the port number
+	 */
+	public FCPConnection(InetAddress address, int port) {
+		this.addr = address;
+		this.port = port;
+	}
+
+	/**
+	 * Connect to the node
+	 * 
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 */
+	public void connect() throws IOException {
+		socket = new Socket(addr, port);
+
+		is = new BufferedInputStream(socket.getInputStream());
+		os = new BufferedOutputStream(socket.getOutputStream());
+	}
+
+	/**
+	 * Hello Message
+	 * 
+	 * Send handshake <code>ClientHello</code> message to node. Block until
+	 * <code>NodeHello</code> is received.
+	 * 
+	 * @param clientName
+	 *            Client name, must be unique in the freenet node
+	 * 
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 */
+	public void hello(String clientName) throws IOException {
+		FCPMessage msg = new FCPMessage();
+		msg.messageType = "ClientHello";
+		msg.put("ExpectedVersion", "2.0");
+		msg.put("Name", clientName);
+
+		msg.writeTo(os);
+
+		while (true) {
+			FCPMessage reply = FCPMessage.parse(is);
+			if ("NodeHello".equals(reply.messageType))
+				return;
+			if ("ProtocolError".equals(reply.messageType))
+				throw new IOException("Protocol error");
+		}
+	}
+
+	/**
+	 * Simple FCP Put
+	 * 
+	 * This method block until the data are fetchable or <code>PutFailed</code>
+	 * received.
+	 * 
+	 * @param freenetURI
+	 *            the freenet URI
+	 * @param data
+	 *            the data to be put
+	 * @param monitor
+	 *            progress monitor to post write completion to during the
+	 *            stream's close method.
+	 * @param monitorTask
+	 *            task name to display during the close method.
+	 * @return <code>true</code> iff <code>PutFetchable</code>,
+	 *         <code>PutSuccessful</code> or <code>PutFetchable</code> with
+	 *         insert collision (data already inserted)
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 */
+	public boolean simplePut(String freenetURI, byte[] data,
+			ProgressMonitor monitor, String monitorTask) throws IOException {
+		System.err.println(freenetURI);
+
+		FCPMessage msg = new FCPMessage();
+		msg.messageType = "ClientPut";
+		msg.put("URI", freenetURI);
+		msg.put("Identifier", freenetURI);
+		msg.put("Verbosity", "1");
+		msg.put("Global", "false");
+		msg.put("UploadFrom", "direct");
+		msg.put("DataLength", "" + data.length);
+		msg.extraData = data;
+
+		msg.writeTo(os);
+
+		int totalBlocks = -1;
+		int completedBlocks = 0;
+
+		try {
+			while (true) {
+				FCPMessage reply = FCPMessage.parse(is);
+
+				if ("ProtocolError".equals(reply.messageType))
+					throw new IOException("Protocol error");
+				if (!freenetURI.equals(reply.get("Identifier"))) // identifier don't match, ignore it
+					continue;
+				if ("IdentifierCollision".equals(reply.messageType))
+					throw new IOException("IdentifierCollision");
+
+				if ("PutFailed".equals(reply.messageType)) {
+					if ("9".equals(reply.get("Code"))) // collision, data already inserted
+						return true;
+					return false;
+				}
+
+				if ("SimpleProgress".equals(reply.messageType)) {
+					if (totalBlocks == -1) {
+						totalBlocks = Integer.parseInt(reply.get("Total"));
+						monitor.beginTask(monitorTask, totalBlocks);
+					}
+					int tmp = Integer.parseInt(reply.get("Succeeded"));
+					if (tmp < totalBlocks)
+						monitor.update(tmp - completedBlocks);
+					completedBlocks = tmp;
+				}
+
+				if ("PutSuccessful".equals(reply.messageType))
+					return true;
+				if ("PutFetchable".equals(reply.messageType))
+					return true;
+			}
+		} finally {
+			if (totalBlocks == -1)
+				monitor.beginTask(monitorTask, 1);
+			monitor.endTask();
+		}
+	}
+
+	/**
+	 * Simple FCP Get
+	 * 
+	 * @param freenetURI
+	 *            the freenet URI
+	 * @return the data
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 * @throws FileNotFoundException
+	 *             if data not found
+	 */
+	public byte[] simpleGet(String freenetURI) throws IOException,
+			FileNotFoundException {
+		FCPMessage msg = new FCPMessage();
+		msg.messageType = "ClientGet";
+		msg.put("URI", freenetURI);
+		msg.put("Identifier", freenetURI);
+		msg.put("Verbosity", "1");
+		msg.put("MaxSize", "2147483647");
+		msg.put("Global", "false");
+		msg.put("ClientToken", freenetURI);
+		msg.put("ReturnType", "direct");
+
+		msg.writeTo(os);
+
+		while (true) {
+			FCPMessage reply = FCPMessage.parse(is);
+			if ("ProtocolError".equals(reply.messageType))
+				throw new IOException("Protocol error");
+			if (!freenetURI.equals(reply.get("Identifier"))) // identifier don't match, ignore it
+				continue;
+			if ("IdentifierCollision".equals(reply.messageType))
+				throw new IOException("IdentifierCollision");
+
+			if ("GetFailed".equals(reply.messageType)) {
+				String redirectURI = reply.get("RedirectURI");
+				if (redirectURI != null)
+					return simpleGet(redirectURI);
+
+				System.err.println(" -> nf");
+				throw new FileNotFoundException();
+			}
+
+			if ("AllData".equals(reply.messageType))
+				return reply.extraData;
+		}
+	}
+
+	/**
+	 * Generate SSK Key Pair
+	 * 
+	 * @return the generated SSK key pair. <code>key[0]</code> is the public
+	 *         key, <code>key[1]</code> is the private key.
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 */
+	public String[] generateSSK() throws IOException {
+		FCPMessage msg = new FCPMessage();
+		msg.messageType = "GenerateSSK";
+		msg.put("Identifier", "GenerateSSK");
+
+		msg.writeTo(os);
+
+		while (true) {
+			FCPMessage reply = FCPMessage.parse(is);
+			if ("ProtocolError".equals(reply.messageType))
+				throw new IOException("Protocol error");
+
+			if ("SSKKeypair".equals(reply.messageType)) {
+				String[] keys = new String[2];
+				keys[0] = reply.get("RequestURI");
+				keys[1] = reply.get("InsertURI");
+				return keys;
+			}
+		}
+	}
+
+	/**
+	 * Close the connection
+	 * 
+	 * @throws IOException
+	 *             if any I/O error occurred
+	 */
+	public void close() throws IOException {
+		os.close();
+		is.close();
+		socket.close();
+	}
+
+	private static class FCPMessage {
+		private String messageType;
+
+		private Map<String, String> parameter = new HashMap<String, String>();
+
+		byte[] extraData;
+
+		FCPMessage() {
+			// default constructor
+		}
+
+		private static FCPMessage parse(InputStream in) throws IOException {
+			FCPMessage ret = new FCPMessage();
+			String line = readLine(in);
+			ret.messageType = line;
+
+			while (true) {
+				line = readLine(in);
+
+				if (line == null)
+					throw new IOException("Malformed FCP message");
+				if ("EndMessage".equals(line))
+					break;
+				if ("Data".equals(line)) {
+					String strLen = ret.parameter.get("DataLength");
+					if (strLen == null)
+						throw new IOException("DataLength not found");
+					int len;
+					try {
+						len = Integer.parseInt(strLen);
+					} catch (NumberFormatException e) {
+						throw new IOException("DataLength malformed");
+					}
+					ret.extraData = readData(in, len);
+					break;
+				}
+
+				String[] v = line.split("=", 2);
+				if (v.length != 2)
+					throw new IOException("No '=' found in: " + line);
+				ret.parameter.put(v[0], v[1]);
+			}
+
+			return ret;
+		}
+
+		void put(String key, String value) {
+			parameter.put(key, value);
+		}
+
+		String get(String key) {
+			return parameter.get(key);
+		}
+
+		void writeTo(OutputStream os) throws IOException {
+			os.write(messageType.getBytes("UTF-8"));
+			os.write('\n');
+
+			for (Map.Entry<String, String> e : parameter.entrySet()) {
+				String l = e.getKey() + '=' + e.getValue();
+				os.write(l.getBytes("UTF-8"));
+				os.write('\n');
+			}
+
+			os.write("EndMessage".getBytes("UTF-8"));
+			os.write('\n');
+
+			if (extraData != null)
+				os.write(extraData);
+			os.flush();
+		}
+	}
+
+	private static String readLine(InputStream in) throws IOException {
+		byte[] buf = new byte[256];
+		int offset = 0;
+
+		while (true) {
+			int b = in.read();
+			if (b == -1)
+				return null;
+			if (b == '\n') {
+				if (offset == 0)
+					continue; // skip empty line
+				break;
+			}
+
+			if (offset == buf.length) {
+				if (offset >= 4096)
+					throw new IOException("line too long");
+
+				byte[] buf2 = new byte[buf.length * 2];
+				System.arraycopy(buf, 0, buf2, 0, buf.length);
+				buf = buf2;
+			}
+
+			buf[offset++] = (byte) b;
+		}
+
+		return new String(buf, 0, offset, "UTF-8");
+	}
+
+	private static byte[] readData(InputStream in, int len) throws IOException {
+		byte[] buf = new byte[len];
+		int read = 0;
+		while (read < len) {
+			int r = in.read(buf, read, len - read);
+			if (r == -1)
+				throw new IOException("Not enough data");
+			read += r;
+		}
+		return buf;
+	}
+}
-- 
1.6.2.rc2

[PoC PATCH JGIT 2/2] PoC Freenet transport

[Daniel Cheng (aka SDiZ)]

Signed-off-by: Daniel Cheng (aka SDiZ) <j16sdiz+freenet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
---
 .../src/org/spearce/jgit/transport/Transport.java  |    3 +
 .../org/spearce/jgit/transport/TransportFcp2.java  |  436 ++++++++++++++++++++
 2 files changed, 439 insertions(+), 0 deletions(-)
 create mode 100644 org.spearce.jgit/src/org/spearce/jgit/transport/TransportFcp2.java

diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java b/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java
index a0a2575..3fc1735 100644
--- a/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/Transport.java
@@ -206,6 +206,9 @@ else if (TransportGitAnon.canHandle(remote))
 		else if (TransportAmazonS3.canHandle(remote))
 			return new TransportAmazonS3(local, remote);
 
+		else if (TransportFcp2.canHandle(remote))
+			return new TransportFcp2(local, remote);
+		
 		else if (TransportBundleFile.canHandle(remote))
 			return new TransportBundleFile(local, remote);
 
diff --git a/org.spearce.jgit/src/org/spearce/jgit/transport/TransportFcp2.java b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportFcp2.java
new file mode 100644
index 0000000..ed12995
--- /dev/null
+++ b/org.spearce.jgit/src/org/spearce/jgit/transport/TransportFcp2.java
@@ -0,0 +1,436 @@
+/*
+ * Copyright (C) 2009, CHENG Yuk-Pong, Daniel <j16sdiz+freenet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
+ * Copyright (C) 2008, Shawn O. Pearce <spearce-o8xcbExO1WpAfugRpC6u6w@public.gmane.org>
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above copyright
+ *   notice, this list of conditions and the following disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *   copyright notice, this list of conditions and the following
+ *   disclaimer in the documentation and/or other materials provided
+ *   with the distribution.
+ *
+ * - Neither the name of the Git Development Community nor the
+ *   names of its contributors may be used to endorse or promote
+ *   products derived from this software without specific prior
+ *   written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+ * CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+ * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
+ * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+ * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+ * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+ * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF
+ * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ */
+
+package org.spearce.jgit.transport;
+
+import java.io.BufferedReader;
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.URISyntaxException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.spearce.jgit.errors.NotSupportedException;
+import org.spearce.jgit.errors.PackProtocolException;
+import org.spearce.jgit.errors.TransportException;
+import org.spearce.jgit.lib.Constants;
+import org.spearce.jgit.lib.NullProgressMonitor;
+import org.spearce.jgit.lib.ObjectId;
+import org.spearce.jgit.lib.ProgressMonitor;
+import org.spearce.jgit.lib.Ref;
+import org.spearce.jgit.lib.Repository;
+import org.spearce.jgit.lib.Ref.Storage;
+import org.spearce.jgit.transport.fcpv2.FCPConnection;
+
+/**
+ * Transport over Freenet Client Protocol 2.0
+ * <p>
+ * The Freenet transport is similar to HTTP Transport, but more restrictive. To
+ * workaround the freenet restrictions, path separator ('/') are replaced with
+ * dash ('-').
+ *<p>
+ * URI are in forms of <code>fcp://USK@PUBLIC_KEY/identifier</code> (read-only)
+ * or <code>fcp://USK@PUBLIC_KEY^PRIVATE_KEY/identifier</code>. The identifier
+ * must not contain any slash. The Freenet URI should begin with
+ * <code>SSK@</code>, all updatable component are converted to <code>USK@</code>
+ * internally.
+ * 
+ * @see WalkFetchConnection
+ */
+class TransportFcp2 extends WalkTransport {
+	static boolean canHandle(final URIish uri) {
+		if (!uri.isRemote())
+			return false;
+		final String s = uri.getScheme();
+		return "fcp".equals(s);
+	}
+
+	private final String publicKey;
+
+	private final String privateKey;
+
+	private final String identifier;
+
+	private FCPConnection fcp;
+
+	TransportFcp2(final Repository local, final URIish uri)
+			throws NotSupportedException {
+		super(local, uri);
+
+		String uriString = uri.toString();
+		if (!uriString.startsWith("fcp://SSK@"))
+			throw new NotSupportedException("URL not SSK@ " + uri);
+
+		String[] uriSegment = uriString.split("\\/");
+		if (uriSegment.length != 4)
+			throw new NotSupportedException("Invalid URL " + uri);
+
+		String[] keys = uriSegment[2].split("\\^");
+		if (keys.length > 2)
+			throw new NotSupportedException("Invalid URL " + uri);
+		publicKey = keys[0].substring(4); // Remove 'SSK@' prefix
+		privateKey = (keys.length == 2) ? keys[1] : null;
+		System.out.println(privateKey);
+		identifier = uriSegment[3];
+	}
+
+	@Override
+	public FetchConnection openFetch() throws TransportException {
+		try {
+			fcp = new FCPConnection();
+			fcp.connect();
+			fcp.hello("JGit-" + toString());
+
+			final Fcp2ObjectDB c = new Fcp2ObjectDB("/objects", false);
+			final WalkFetchConnection r = new WalkFetchConnection(this, c);
+			r.available(c.readAdvertisedRefs());
+			return r;
+		} catch (IOException e) {
+			throw new TransportException("IO Error" + e, e);
+		}
+	}
+
+	@Override
+	public PushConnection openPush() throws TransportException {
+		try {
+			fcp = new FCPConnection();
+			fcp.connect();
+			fcp.hello("JGit-" + toString());
+
+			final Fcp2ObjectDB c = new Fcp2ObjectDB("/objects", false);
+			final WalkPushConnection r = new WalkPushConnection(this, c);
+			r.available(c.readAdvertisedRefs());
+			return r;
+		} catch (IOException e) {
+			throw new TransportException("IO Error" + e, e);
+		}
+	}
+
+	@Override
+	public void close() {
+		try {
+			if (fcp != null)
+				fcp.close();
+		} catch (IOException e) {
+			// Fall through.
+		}
+		fcp = null;
+	}
+
+	class Fcp2ObjectDB extends WalkRemoteObjectDatabase {
+		private final String path;
+		private boolean useArchive;
+
+		@Override
+		void deleteFile(String path) throws IOException {
+			// TODO Auto-generated method stub
+			super.deleteFile(path);
+		}
+
+		@Override
+		OutputStream writeFile(String path, ProgressMonitor monitor,
+				String monitorTask) throws IOException {
+			System.err.println(" write -> " + path);
+			return new FcpOutputStream(path, monitor, monitorTask);
+		}
+
+		class FcpOutputStream extends OutputStream {
+			ByteArrayOutputStream os = new ByteArrayOutputStream();
+
+			private String path;
+
+			private ProgressMonitor monitor;
+
+			private String monitorTask;
+
+			public FcpOutputStream(String path, ProgressMonitor monitor,
+					String monitorTask) {
+				if (monitor == null)
+					monitor = NullProgressMonitor.INSTANCE;
+				if (monitorTask == null)
+					monitorTask = "Uploading " + path;
+
+				this.path = path;
+				this.monitor = monitor;
+				this.monitorTask = monitorTask;
+			}
+
+			@Override
+			public void write(int b) throws IOException {
+				os.write(b);
+			}
+
+			@Override
+			public void close() throws IOException {
+				byte[] b = os.toByteArray();
+				os = null;
+
+				fcp.simplePut(getFreenetURI(path, true), b, monitor,
+						monitorTask);
+			}
+		}
+
+		Fcp2ObjectDB(String path, boolean useArchive) {
+			if (path.endsWith("/"))
+				path = path.substring(0, path.length() - 1);
+			if (!path.startsWith("/"))
+				path = "/" + path;
+			this.path = path;
+			this.useArchive = useArchive;
+		}
+
+		private String resolveKey(String subpath) {
+			if (subpath.endsWith("/"))
+				subpath = subpath.substring(0, subpath.length() - 1);
+			String k = path;
+			while (subpath.startsWith(ROOT_DIR)) {
+				k = k.substring(0, k.lastIndexOf('/'));
+				subpath = subpath.substring(3);
+			}
+			return k + "/" + subpath;
+		}
+
+		@Override
+		URIish getURI() {
+			try {
+				// just a pseudo-URIish for logging, not really supported
+				return new URIish("fcp://SSK@" + publicKey + "/" + identifier
+						+ path);
+			} catch (URISyntaxException e) {
+				return null;
+			}
+		}
+
+		@Override
+		Collection<WalkRemoteObjectDatabase> getAlternates() throws IOException {
+			if (!useArchive)	// try to use archive
+				return java.util.Arrays.asList( 
+				 new WalkRemoteObjectDatabase[] { new  Fcp2ObjectDB(path, true) } );
+			return null;
+		}
+
+		@Override
+		WalkRemoteObjectDatabase openAlternate(final String location)
+				throws IOException {
+			return null;
+		}
+
+		@Override
+		Collection<String> getPackNames() throws IOException {
+			final Collection<String> packs = new ArrayList<String>();
+			try {
+				final BufferedReader br = openReader(INFO_PACKS);
+				try {
+					for (;;) {
+						final String s = br.readLine();
+						if (s == null || s.length() == 0)
+							break;
+						if (!s.startsWith("P pack-") || !s.endsWith(".pack"))
+							throw invalidAdvertisement(s);
+						packs.add(s.substring(2));
+					}
+					return packs;
+				} finally {
+					br.close();
+				}
+			} catch (FileNotFoundException err) {
+				return packs;
+			}
+		}
+
+		@Override
+		FileStream open(final String path) throws IOException {
+			System.err.println(" read -> " + path);
+			byte[] b = fcp.simpleGet(getFreenetURI(path, false));
+			if (b == null)
+				throw new FileNotFoundException();
+
+			ByteArrayInputStream is = new ByteArrayInputStream(b);
+			return new FileStream(is);
+		}
+
+		private String getFreenetURI(final String path, boolean push) {
+			String key = push ? privateKey : publicKey;
+			String fPath = resolveKey(path);
+			String ret;
+
+			if (useArchive)
+				ret = "USK@" + key + "/" + identifier + "/1/" + fPath;
+			else
+				ret = "USK@" + key + "/" + identifier + fPath.replaceAll("\\/", "-") + "/1/";
+
+			System.err.println(ret);
+			return ret;
+		}
+
+		Map<String, Ref> readAdvertisedRefs() throws TransportException {
+			final TreeMap<String, Ref> avail = new TreeMap<String, Ref>();
+			try {
+				readPackedRefs(avail);
+			} catch (TransportException t) {}
+			try {
+				readInfoRefs(avail);
+			} catch (TransportException t) {}
+			try {
+				readRef(avail, Constants.HEAD);
+			} catch (TransportException t) {}
+
+			if (avail.isEmpty() && !useArchive) {
+				return  new Fcp2ObjectDB(path, true).readAdvertisedRefs();
+			}
+
+			return avail;
+		}
+
+		private Map<String, Ref> readInfoRefs(TreeMap<String, Ref> avail)
+				throws TransportException {
+			try {
+				final BufferedReader br = openReader(INFO_REFS);
+				for (;;) {
+					String line = br.readLine();
+					if (line == null)
+						break;
+
+					final int tab = line.indexOf('\t');
+					if (tab < 0)
+						throw invalidAdvertisement(line);
+
+					String name;
+					final ObjectId id;
+
+					name = line.substring(tab + 1);
+					id = ObjectId.fromString(line.substring(0, tab));
+					if (name.endsWith("^{}")) {
+						name = name.substring(0, name.length() - 3);
+						final Ref prior = avail.get(name);
+						if (prior == null)
+							throw outOfOrderAdvertisement(name);
+
+						if (prior.getPeeledObjectId() != null)
+							throw duplicateAdvertisement(name + "^{}");
+
+						avail.put(name, new Ref(Ref.Storage.NETWORK, name,
+								prior.getObjectId(), id, true));
+					} else {
+						final Ref prior = avail.put(name, new Ref(
+								Ref.Storage.NETWORK, name, id));
+						if (prior != null)
+							throw duplicateAdvertisement(name);
+					}
+				}
+				return avail;
+			} catch (IOException err) {
+				throw new TransportException(INFO_REFS
+						+ ": cannot read available refs", err);
+			}
+		}
+
+		private PackProtocolException outOfOrderAdvertisement(final String n) {
+			return new PackProtocolException("advertisement of " + n
+					+ "^{} came before " + n);
+		}
+
+		private PackProtocolException duplicateAdvertisement(final String n) {
+			return new PackProtocolException("duplicate advertisements of " + n);
+		}
+
+		private Ref readRef(final TreeMap<String, Ref> avail, final String rn)
+				throws TransportException {
+			final String s;
+			String ref = ROOT_DIR + rn;
+			try {
+				final BufferedReader br = openReader(ref);
+				try {
+					s = br.readLine();
+				} finally {
+					br.close();
+				}
+			} catch (FileNotFoundException noRef) {
+				return null;
+			} catch (IOException err) {
+				throw new TransportException(getURI(), "read " + ref, err);
+			}
+
+			if (s == null)
+				throw new TransportException(getURI(), "Empty ref: " + rn);
+
+			if (s.startsWith("ref: ")) {
+				final String target = s.substring("ref: ".length());
+				Ref r = avail.get(target);
+				if (r == null)
+					r = readRef(avail, target);
+				if (r == null)
+					return null;
+				r = new Ref(r.getStorage(), rn, r.getObjectId(), r
+						.getPeeledObjectId(), r.isPeeled());
+				avail.put(r.getName(), r);
+				return r;
+			}
+
+			if (ObjectId.isId(s)) {
+				// FIXME this does not really works
+				final Ref r = new Ref(loose(avail.get(rn)), rn, ObjectId
+						.fromString(s));
+				avail.put(r.getName(), r);
+				return r;
+			}
+
+			throw new TransportException(getURI(), "Bad ref: " + rn + ": " + s);
+		}
+
+		private Storage loose(final Ref r) {
+			if (r != null && r.getStorage() == Storage.PACKED)
+				return Storage.LOOSE_PACKED;
+			return Storage.LOOSE;
+		}
+
+		private PackProtocolException invalidAdvertisement(final String n) {
+			return new PackProtocolException("invalid advertisement of " + n);
+		}
+
+		@Override
+		void close() {
+			// We do not maintain persistent connections.
+		}
+	}
+}
-- 
1.6.2.rc2

Re: [PoC PATCH JGIT 0/2] Proof of concept code for Git Freenet transport

[Daniel Cheng]

On Mon, Mar 9, 2009 at 11:36 AM, Daniel Cheng (aka SDiZ)
<j16sdiz+freenet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi JGit / Freenet community,
>
> Here is some proof-of-concept code for Git-over-Freenet.
> I am sending this to see the feedback from communities.

Sorry, all. The code I have posted wasn't work.
(the pack file is missing on push, only idx files are pushed)

The new, fixed code is now available at http://github.com/j16sdiz/egit-freenet/
Please use this repository if you want to actually try it.

>
> The code need some more cleanups, so it it is not ready for apply (yet).
>
> This is a real-life example
>
> Push:
>   $ git remote add fcp fcp://SSK@[my public key]^[my private key]/test.git
>   $ ./jgit push fcp refs/remotes/origin/stable:refs/heads/master
>
>  /ALTERNATIVLY/
>
>   Insert a bare repository under USK@<.....>/test.git/-1/
>
> Pull:
>  $ ./jgit clone fcp://SSK@[my public key]^[my private key]/test.git
>
>
> To workaround the metadata update problem, this client translate the
> path seperator to "-", that means:
>
> On push:
>   objects/aa/bbbbbbbb   --> USK@..../test.git-objects-aa-bbbbbbb/-1/
>   refs/heads/xxx        --> USK@..../test.git-objects-ref-heads-xxx/-1/
>
> On pull:
>   To support uploading from jSite,
>    when we load the info/refs we first check USK@..../test.git-info-refs/-1/
>    if it is unavailiable, we would use USK@..../test.git/-1/info/refs
>
>   The "traditional" type (USK@..../test.git/-1/objects) repository is
>   always added as an alternative objects database. No other info/alternatives
>   are supported
>
> FIXME:
>  - How to store the private key of repository?
>   Currently, we use URI of form fcp://SSK@<public key>^<private key>/some-id
>   This is quite ugly. Could we use a per remote Config ? How can I get remote
>   name from transport?
>
>  - Make pushing async, could we?
>
>
>

Re: [PoC PATCH JGIT 0/2] Proof of concept code for Git Freenet transport

[Johannes Schindelin]

Hi,

On Mon, 9 Mar 2009, Daniel Cheng (aka SDiZ) wrote:

> Hi JGit / Freenet community,
> 
> Here is some proof-of-concept code for Git-over-Freenet.

Thanks!

I was talking to some Freenet guys at the GSoC mentor summit 2007, and 
always wanted to give it a try (albeit using C Git, not JGit), but that 
item just got pushed off my TODO list...

Ciao,
Dscho

Re: [PoC PATCH JGIT 0/2] Proof of concept code for Git Freenet transport

[Shawn O. Pearce]

"Daniel Cheng (aka SDiZ)" <j16sdiz+freenet@gmail.com> wrote:
> 
> Here is some proof-of-concept code for Git-over-Freenet.
> I am sending this to see the feedback from communities.

Interesting!

I'm quite open to bringing this into JGit itself, so long as it
doesn't cause us to import 42 other libraries that we don't need.

:-)
 
> FIXME:
>  - How to store the private key of repository?
>    Currently, we use URI of form fcp://SSK@<public key>^<private key>/some-id
>    This is quite ugly. Could we use a per remote Config ? How can I get remote
>    name from transport?

Use something like amazon-s3 does, where there is a file path under
~/ specified by the host part of the URI, and store the data there?

I did that rather than storing into ~/.gitconfig or GIT_DIR/config
as the private key really should remain private.  Chmod'ing your
config file to be private is a pain, and "git config" last I checked
wouldn't preserve the permissions when it modified the file for you.

That said, the Transport API in JGit doesn't get access to the
RemoteConfig it was created with, because it isn't always made
from a remote (e.g. you can pass the URIish on the command line).
We could however optionally expose it, but in general I prefer
to make everything that can be obtained via the RemoteConfig be
settable without one, so it can also come through from the CLI
or some higher-level GUI.
 
>  - Make pushing async, could we?

I'm not sure I understand that.  Do you want to background the push
task?  Isn't that what your shell is for?  Or do you want to perform
the writes in an async fashion in parallel, to reduce the latency?

-- 
Shawn.