Re: [PATCH v6 06/13] pkt-line: add functions to read/write flush terminated packet streams

[Stefan Beller <sbeller@google.com>]

On Thu, Aug 25, 2016 at 5:55 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> On Thu, Aug 25, 2016 at 3:31 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> What is wrong about that?  4*80k = 320kB overhead for length fields
>> to transfer 5GB worth of data?  I do not think it is worth worrying
>> about it.
>>
>> But I am more surprised by seeing that "why not a single huge
>> packet" suggestion immediately after you talked about "without the
>> possibility to intervene".  They do not seem to be remotely related;
>> in fact, they are going into opposite directions.
>>
>> Puzzled.
>
> Stefan's argument to me is thus "If we're already going to ignore
> sideband packets here, why not go all the way and make variable length
> packets and send a single packet of a maximum length? Doing thus will
> solve some set of future problems nicely and makes this code easier."
>
> I'm not sure I agree myself, but that's the logic as I understand it.

Yeah. To me it seems this design explicitly makes it hard for side bands.
As we do not need sidebands for local transfers, this is fine for sure.

(If we wanted to make it sideband friendly, I'd expect you could register
callbacks for either all packets or for a given sideband until the next
flush comes.)

So as hinted by this design, we want a protocol that
* doesn't care about sidebands
* cares about large data (hence maybe throughput)
* has easy/clean interface

And one large packet would suffice for these three points as well
and additionally has benefits for the network stuff.

The 320kB additional transmission are negligible overhead, so I was not
concerned about the size, but rather the code being bloated, i.e. we need
one layer of additional code to cope with the repetitive packets.

---
My background is mostly submodule related, and whenever I come up
with a shiny novel idea that would help submodules tremendously, someone
(mostly Peff) comes along and suggests a broader more generic thing, that
works just as well for submodules but is applicable to all of Git.

So I picked up that way of thinking: If we write code here, that helps with a
very special niche of things, can we make so, that the rest also benefits?

I may not understand all the requirements in this case, but to me it looks like
the "one packet" approach covers all requirements, but has the huge potential
to make other parts better in the long run.
-- 

Thanks,
Stefan