Re: [RFC/PATCH 0/3] protocol v2

From: Shawn Pearce <spearce@spearce.org>
To: Duy Nguyen <pclouds@gmail.com>
Cc: Junio C Hamano <gitster@pobox.com>,
	Stefan Beller <sbeller@google.com>,
	Git Mailing List <git@vger.kernel.org>
Subject: Re: [RFC/PATCH 0/3] protocol v2
Date: Tue, 3 Mar 2015 20:27:33 -0800	[thread overview]
Message-ID: <CAJo=hJtgda4ATtPeLWbhgDPiTm9h-gzBkpknxsOdRddueUgTbw@mail.gmail.com> (raw)
In-Reply-To: <CACsJy8B_r_0nP9NyKFBnr9bXgwjx8dJkSVkHbZw+Mxin_YpZZw@mail.gmail.com>

On Tue, Mar 3, 2015 at 5:54 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Wed, Mar 4, 2015 at 12:13 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> My recollection is that the consensus from the last time we
>> discussed protocol revamping was to list one capability per packet
>> so that packet length limit does not matter, but you may want to
>> check with the list archive yourself.
>
> I couldn't find that consensus mail, but this one [1] is good enough
> evidence that we can hit packet length limit in capability line
> easily.
> With an escape hatch to allow maximum packet length up to  uint_max, I

The symbolic ref thing was done badly. There isn't an escape hatch in
current v1 protocol sufficient to allow this but each ref should be
its own pkt-line, or should be a small batch of refs per pkt-line, or
the ref advertisement should be a data stream in a side-band-64k sort
of format inside the pkt-line framing.

At 64k per frame of side-band there is plenty of data to header ratio
that we don't need  to escape to uint_max.

> Looks like one cap per pkt-line is winning..

Yes.

> [1] http://thread.gmane.org/gmane.comp.version-control.git/237929

Let me go on a different tangent a bit from the current protocol.

http://www.grpc.io/ was recently released and is built on the HTTP/2
standard. It uses protobuf as a proven extensibility mechanism.
Including a full C based grpc stack just to speak the Git wire
protocol is quite likely overkill, but I think the embedding of a
proven extensible format inside of a bi-directional framed streaming
system like HTTP/2 offers some good guidance.

Network protocol parsing is hard. Especially in languages like C where
buffer overflows are possible. Or where a client could trivially DoS a
server by sending a packet of size uint_max and the server naively
trying to malloc() that buffer. Defining the network protocol in an
IDL like protobuf 3 and being machine generated from stable well
maintained code has its advantages.