wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

Dear Git Folks,

Originally this issue was mentioned in previous thread [1], and I have decided
to bring it into a separate thread.  ATM there is a dichotomy in git behavior
between cloning non-bare repos:  if I clone over ssh or just locally by
providing url without trailing /.git, git senses for /.git and works just fine
with ssh or local repositories, but fails for "dummy" http ones, the demo
script is here [2] which produces output listed below.  From which you can see
that  cloning using http URL to the repository without /.git fails (git version
2.8.1, Debian).  As it was noted in [1], concern could have been to not
traverse website since could lead to dangerous places.  But .git is under
originating url directory, as well as info/ or HEAD or any other object
accessed by git, so IMHO this concern is not a concern.

So do you think that cloning from  http urls could be adjusted so git senses
for presence of .git/ subdirectory if originating url is missing necessary
/info/refs?.... ?  That would make behavior uniform and help us and users in
many use-cases IMHO (in particular with relative paths to submodules as in
[1])

Thank you in advance, and please maintain CC

$> /tmp/democlone
Initiating repo
cloning via ssh: localhost:/tmp/repo1
Cloning into 'repo1-ssh'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.
SUCCESS
cloning locally: /tmp/repo1
Cloning into 'repo1-local'...
done.
SUCCESS
starting local http server
Serving HTTP on 0.0.0.0 port 8080 ...
cloning via http: http://localhost:8080/repo1
Cloning into 'repo1-http-failed'...
127.0.0.1 - - [06/May/2016 09:13:41] code 404, message File not found
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/info/refs?service=git-upload-pack HTTP/1.1" 404 -
fatal: repository 'http://localhost:8080/repo1/' not found
FAILED
doing with /.git: http://localhost:8080/repo1/.git
Cloning into 'repo1-http'...
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/info/refs?service=git-upload-pack HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/HEAD HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/36/31fa81eb6422349035c915a5a11b177688f491 HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/12/ddeb87cc045cd67063e95125fbeb014bd1d9b1 HTTP/1.1" 200 -
127.0.0.1 - - [06/May/2016 09:13:41] "GET /repo1/.git/objects/e6/9de29bb2d1d6434b8b29ae775ad8c2e48c5391 HTTP/1.1" 200 -
Checking connectivity... done.
SUCCESS



[1] problems serving non-bare repos with submodules over http
http://thread.gmane.org/gmane.comp.version-control.git/292032
[2] http://www.onerussian.com/tmp/democlone
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Fri, 06 May 2016, Yaroslav Halchenko wrote:

> Dear Git Folks,

> Originally this issue was mentioned in previous thread [1], and I have decided
> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
> between cloning non-bare repos:  if I clone over ssh or just locally by
> providing url without trailing /.git, git senses for /.git and works just fine
> with ssh or local repositories, but fails for "dummy" http ones, the demo
> script is here [2] which produces output listed below.  From which you can see
> that  cloning using http URL to the repository without /.git fails (git version
> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
> traverse website since could lead to dangerous places.  But .git is under
> originating url directory, as well as info/ or HEAD or any other object
> accessed by git, so IMHO this concern is not a concern.
> ...

If there is a better venue (bug tracker?) to spark the interest
and discussion, please let me know ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Junio C Hamano]

Yaroslav Halchenko <yoh@onerussian.com> writes:

> On Fri, 06 May 2016, Yaroslav Halchenko wrote:
>
>> Dear Git Folks,
>
>> Originally this issue was mentioned in previous thread [1], and I have decided
>> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
>> between cloning non-bare repos:  if I clone over ssh or just locally by
>> providing url without trailing /.git, git senses for /.git and works just fine
>> with ssh or local repositories, but fails for "dummy" http ones, the demo
>> script is here [2] which produces output listed below.  From which you can see
>> that  cloning using http URL to the repository without /.git fails (git version
>> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
>> traverse website since could lead to dangerous places.  But .git is under
>> originating url directory, as well as info/ or HEAD or any other object
>> accessed by git, so IMHO this concern is not a concern.

I am afraid that the reason why you saw no response is primarily
because nobody is interested in extending dumb commit-walker HTTP
transport after the world has largely moved on and abandoned it.

The necessary update to the client might as simple as using
$GIVEN_URL/.git/ and attempting the request again after seeing the
probe for $GIVEN_URL/info/refs fails.

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Jacob Keller]

On Tue, May 10, 2016 at 1:11 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Yaroslav Halchenko <yoh@onerussian.com> writes:
>
>> On Fri, 06 May 2016, Yaroslav Halchenko wrote:
>>
>>> Dear Git Folks,
>>
>>> Originally this issue was mentioned in previous thread [1], and I have decided
>>> to bring it into a separate thread.  ATM there is a dichotomy in git behavior
>>> between cloning non-bare repos:  if I clone over ssh or just locally by
>>> providing url without trailing /.git, git senses for /.git and works just fine
>>> with ssh or local repositories, but fails for "dummy" http ones, the demo
>>> script is here [2] which produces output listed below.  From which you can see
>>> that  cloning using http URL to the repository without /.git fails (git version
>>> 2.8.1, Debian).  As it was noted in [1], concern could have been to not
>>> traverse website since could lead to dangerous places.  But .git is under
>>> originating url directory, as well as info/ or HEAD or any other object
>>> accessed by git, so IMHO this concern is not a concern.
>
> I am afraid that the reason why you saw no response is primarily
> because nobody is interested in extending dumb commit-walker HTTP
> transport after the world has largely moved on and abandoned it.
>
> The necessary update to the client might as simple as using
> $GIVEN_URL/.git/ and attempting the request again after seeing the
> probe for $GIVEN_URL/info/refs fails.
> --

I know at least Jenkin's Git plugin has a workaround to solve this
issue that is quite similar.

Thanks,
Jake

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Tue, 10 May 2016, Jacob Keller wrote:
> > The necessary update to the client might as simple as using
> > $GIVEN_URL/.git/ and attempting the request again after seeing the
> > probe for $GIVEN_URL/info/refs fails.
> I know at least Jenkin's Git plugin has a workaround to solve this
> issue that is quite similar.

On Tue, 10 May 2016, Junio C Hamano wrote:

> >> traverse website since could lead to dangerous places.  But .git is under
> >> originating url directory, as well as info/ or HEAD or any other object
> >> accessed by git, so IMHO this concern is not a concern.

> I am afraid that the reason why you saw no response is primarily
> because nobody is interested in extending dumb commit-walker HTTP
> transport after the world has largely moved on and abandoned it.

> The necessary update to the client might as simple as using
> $GIVEN_URL/.git/ and attempting the request again after seeing the
> probe for $GIVEN_URL/info/refs fails.

Sure -- workarounds are possible, and we are at the state that many dependent
projects seems are doing that already (as above noted Jenkin's Git plugin, smth
along the lines probably done by github for https as well).  In my case I even
have managed to erect a lovely apache rewrite rule which seems to work, so I
can just 'git clone --recursive' a collection of 30 submodules without a hiccup
(we are also interested in .git/annex part here ;) ).  Citing here if it
comes handy for anyone

    # To overcome http://thread.gmane.org/gmane.comp.version-control.git/293777
    # we need to rewrite urls so that there is no need for explicit .git/
    RewriteEngine On
    RewriteCond "!.*/\.git/.*"
    RewriteRule "(.*?/)((?<!\.git/)(HEAD|config|annex/objects/.*|objects/(info/[^/]+|[0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}.(pack|idx))|git-(upload|receive)-pack)|info/refs)$" "$1.git/$2" [PT]

and sure thing in our project now we can add such sensing for .git/ and/or
reattempts to request again.  But that is the point -- it seems to be a
relatively common hiccup which could be fixed in the "root".  So I (and
possibly others) would just very much appreciate if it was (thus it was a
[wishlist] ;) )

Cheers,
-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Junio C Hamano]

Yaroslav Halchenko <yoh@onerussian.com> writes:

>> The necessary update to the client might be as simple as using
>> $GIVEN_URL/.git/ and attempting the request again after seeing the
>> probe for $GIVEN_URL/info/refs fails.
>
> Sure -- workarounds are possible,...

Just so that there is no misunderstanding, the above was not a
workaround but is an outline of an implementation of updated http
client shipped with Git.

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Yaroslav Halchenko]

On Tue, 10 May 2016, Junio C Hamano wrote:
> >> The necessary update to the client might be as simple as using
> >> $GIVEN_URL/.git/ and attempting the request again after seeing the
> >> probe for $GIVEN_URL/info/refs fails.

> > Sure -- workarounds are possible,...

> Just so that there is no misunderstanding, the above was not a
> workaround but is an outline of an implementation of updated http
> client shipped with Git.

ah, sorry, I have indeed might have misread it.  So we are on the same
page -- that is I saw also as the tentative implementation ;)

-- 
Yaroslav O. Halchenko
Center for Open Neuroscience     http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Re: wishlist; unify behavior while cloning non-bare repos over http to be in line with ssh/local

[Junio C Hamano]

Yaroslav Halchenko <yoh@onerussian.com> writes:

> On Tue, 10 May 2016, Junio C Hamano wrote:
>> >> The necessary update to the client might be as simple as using
>> >> $GIVEN_URL/.git/ and attempting the request again after seeing the
>> >> probe for $GIVEN_URL/info/refs fails.
>
>> > Sure -- workarounds are possible,...
>
>> Just so that there is no misunderstanding, the above was not a
>> workaround but is an outline of an implementation of updated http
>> client shipped with Git.
>
> ah, sorry, I have indeed might have misread it.  So we are on the same
> page -- that is I saw also as the tentative implementation ;)

Good.  Now somebody who is interested in seeing that happen (when I
said "shipped with Git" above, I meant "shipped with possible future
Git") needs to find (or be) somebody to code that change ;-)

Thanks.