Thoughts on refactoring the transport (+helper) code

Thoughts on refactoring the transport (+helper) code

[Dave Borowitz]

I spent a day trying to understand what the heck is going on in the
transport code, with the intent of adding an option like
--sign-if-possible to git push. This has come up twice on the mailing
list in the past couple weeks, and I also think it's important for
$DAY_JOB.

I'm giving up in despair, but I decided to leave some comments that
will hopefully help a future, more enterprising refactorer. Please
don't read this (entirely) as a rant; I understand that the transport
code is old and hairy and grew organically to get to this point. I
really do just hope this makes the next guy's job easier in
understanding the code. (But then again, this hope may be in vain:
it's not like I searched the mailing list myself before diving in :)

One of the biggest issues IMO is that the idea of "options" for
transports is grossly overloaded:

-struct git_transport_options contains mostly boolean options that are
read in fetch.c to enable certain options.

-In push.c, we for the most part ignore the smart_options field in
transport, and instead rely on the flags bitfield. However, some (not
all) flags in this bitfield have seemingly-equivalent fields in
git_transport_options. In some cases (particularly push_cert), the
field in git_transport_options is ignored.

-Similarly, one might think the executable arg to the connect callback
might bear some relation to the uploadpack/receivepack field in
smart_options. It does not.

-Some (but by no means all) options are set via transport_set_option
by passing in one of the TRANS_OPT_* string constants. This a)
switches on these values and populates the appropriate smart_options
field, and b) calls the set_option callback.

-The end result is that the TRANS_OPT_* constants are a mixture of
things that can be set on the command line and things that are
documented in the remote helper protocol. But there are also options
used in the remote helper protocol that are hard-coded and have no
constant, or can't be set on the command line, etc.

-The helper protocol's set_helper_option synchronously sends the
option to the helper process and reads the response. Naturally, the
return code from transport_set_option is almost always ignored.

In my ideal world:
-smart_options would never be NULL, and would instead be called
"options" with a "smart" bit which is unset for dumb protocols.
-Command line option processing code in {fetch,clone,push}.c would set
fields in options (formerly known as smart_options) rather than
passing around string constants.
-TRANS_OPT_* string constants would only be used for remote helper
protocol option names, and no more hard-coding these names.
-The flags arg to the push* callbacks would go away, and callbacks
would respect options instead.
-The helper code would not send options immediately, but instead send
just the relevant options immediately before a particular command
requires them. Hopefully we could then eliminate the set_option
callback entirely. (Two things I ran into that complicated this: 1)
backfill_tags mutates just a couple of options before reusing the
transport, and 2) the handling of push_cas_option is very
special-cased.)

There are other confusing things but I think that would make option
handling in particular less of a head-scratcher.

Re: Thoughts on refactoring the transport (+helper) code

[Ilari Liusvaara]

On Thu, Aug 13, 2015 at 11:42:50AM -0400, Dave Borowitz wrote:
> 
> In my ideal world:
> -smart_options would never be NULL, and would instead be called
> "options" with a "smart" bit which is unset for dumb protocols.
> -Command line option processing code in {fetch,clone,push}.c would set
> fields in options (formerly known as smart_options) rather than
> passing around string constants.
> -TRANS_OPT_* string constants would only be used for remote helper
> protocol option names, and no more hard-coding these names.
> -The flags arg to the push* callbacks would go away, and callbacks
> would respect options instead.
> -The helper code would not send options immediately, but instead send
> just the relevant options immediately before a particular command
> requires them. Hopefully we could then eliminate the set_option
> callback entirely. (Two things I ran into that complicated this: 1)
> backfill_tags mutates just a couple of options before reusing the
> transport, and 2) the handling of push_cas_option is very
> special-cased.)

AFAIK, here are what one can encounter with remote helpers:

Some remote helpers are always smart, some are always dumb, and some
may be smart or dumb, depending on the URL.

I don't know how useful the last one is (smart or dumb depending on
URL) is. I have never seen such remote helper (HTTP doesn't count,
from Git PoV it is always dumb).

All smart helpers take common options associated with git smart
transport (e.g. thin, follow tags, push certs, etc..).

Dumb transports may take some of these kind of of options (esp.
smart HTTP), but it is highly dependent on the helper.

Then transports can have connection-level options (e.g. HTTPS
cert options). These can be present regardless of wheither
transport is smart or dumb.

The receive-pack / send-pack paths fall into connection-level
options, even if those are presently in smart transport common
options. Since those things make sense for some smart transports
(e.g. ssh://), but not others (e.g. git://).


-Ilari

Re: Thoughts on refactoring the transport (+helper) code

[Dave Borowitz]

On Thu, Aug 13, 2015 at 12:45 PM, Ilari Liusvaara
<ilari.liusvaara@elisanet.fi> wrote:
> On Thu, Aug 13, 2015 at 11:42:50AM -0400, Dave Borowitz wrote:
>>
>> In my ideal world:
>> -smart_options would never be NULL, and would instead be called
>> "options" with a "smart" bit which is unset for dumb protocols.
>> -Command line option processing code in {fetch,clone,push}.c would set
>> fields in options (formerly known as smart_options) rather than
>> passing around string constants.
>> -TRANS_OPT_* string constants would only be used for remote helper
>> protocol option names, and no more hard-coding these names.
>> -The flags arg to the push* callbacks would go away, and callbacks
>> would respect options instead.
>> -The helper code would not send options immediately, but instead send
>> just the relevant options immediately before a particular command
>> requires them. Hopefully we could then eliminate the set_option
>> callback entirely. (Two things I ran into that complicated this: 1)
>> backfill_tags mutates just a couple of options before reusing the
>> transport, and 2) the handling of push_cas_option is very
>> special-cased.)
>
> AFAIK, here are what one can encounter with remote helpers:
>
> Some remote helpers are always smart, some are always dumb, and some
> may be smart or dumb, depending on the URL.
>
> I don't know how useful the last one is (smart or dumb depending on
> URL) is. I have never seen such remote helper (HTTP doesn't count,
> from Git PoV it is always dumb).
>
> All smart helpers take common options associated with git smart
> transport (e.g. thin, follow tags, push certs, etc..).
>
> Dumb transports may take some of these kind of of options (esp.
> smart HTTP), but it is highly dependent on the helper.
>
> Then transports can have connection-level options (e.g. HTTPS
> cert options). These can be present regardless of wheither
> transport is smart or dumb.
>
> The receive-pack / send-pack paths fall into connection-level
> options, even if those are presently in smart transport common
> options. Since those things make sense for some smart transports
> (e.g. ssh://), but not others (e.g. git://).

Yeah, thanks for summarizing some of the differences between options.
The really confusing thing when looking at the code, though, is that
the various different ways of specifying options in the code don't
actually align with those distinctions. Maybe they once did, but they
certainly don't today.

I wouldn't be opposed to splitting up git_transport_options into
different structs for connection options, smart fetch options, smart
push options, etc., rather than putting everything in one kitchen-sink
struct.

>
> -Ilari