Partial clone fails with file:/// URI

Partial clone fails with file:/// URI

[Nipunn Koorapati]

Hi friends,

When I run a git clone with --filter and a file:/// URI - it fails.

Repro steps:

I have the `git` repo cloned and built into the CWD at v2.28.0
git/git clone --filter=blob:none file:///home/nipunn/src/git git2

nipunn@nipunn-dbx:~/src$ git/git clone --filter=blob:none
file:///home/nipunn/src/git git2
Cloning into 'git2'...
warning: templates not found in /home/nipunn/share/git-core/templates
remote: Enumerating objects: 177110, done.
remote: Counting objects: 100% (177110/177110), done.
remote: Compressing objects: 100% (61436/61436), done.
Receiving objects: 100% (177110/177110), 62.58 MiB | 43.95 MiB/s, done.
remote: Total 177110 (delta 114305), reused 176904 (delta 114102), pack-reused 0
Resolving deltas: 100% (114305/114305), done.
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

It seems to work ok with a remote URI or w/o a --filter arg.
I was able to confirm that the "bad pack header" is a "-1" response
code PH_ERROR_EOF
Perhaps the `fatal: protocol error: bad pack header` - should include
the error code as well.
Thanks
--Nipunn

Re: Partial clone fails with file:/// URI

[Taylor Blau]

Hi Nipunn,

On Wed, Oct 14, 2020 at 05:38:15PM +0100, Nipunn Koorapati wrote:
> Hi friends,
>
> When I run a git clone with --filter and a file:/// URI - it fails.
>
> Repro steps:
>
> I have the `git` repo cloned and built into the CWD at v2.28.0
> git/git clone --filter=blob:none file:///home/nipunn/src/git git2

I couldn't quite reproduce this. Here's my reproduction:

  #!/bin/sh

  rm -fr repo clone
  git init repo

  (
    cd repo

    git config uploadpack.allowfilter true
    git config uploadpack.allowanysha1inwant true

    for f in a b; do
      echo $f > $f
      git add $f
      git commit -m "$f: initial commit"
    done
  )

  git clone --filter=blob:none file://$(pwd)/repo clone

That works on v2.28.0, and v2.29.0 for me. Is git installed properly on
your system?

Thanks,
Taylor

Re: Partial clone fails with file:/// URI

[Nipunn Koorapati]

Hi,

I've been building git from source and directly using the executable. Roughly
```
cd git
make
cd ..
git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git
```

This is what I was missing:
git config uploadpack.allowanysha1inwant true

If this is not set, then the checkout fails w/ a

fatal: protocol error: bad pack header

Perhaps the error message and/or docs could be improved here to
something with more detail than "bad pack header"?
Not seeing anything to suggest this in the docs
https://git-scm.com/docs/partial-clone

Thanks for the prompt response. Appreciated!
--Nipunn

On Wed, Oct 14, 2020 at 5:56 PM Taylor Blau <me@ttaylorr.com> wrote:
>
> Hi Nipunn,
>
> On Wed, Oct 14, 2020 at 05:38:15PM +0100, Nipunn Koorapati wrote:
> > Hi friends,
> >
> > When I run a git clone with --filter and a file:/// URI - it fails.
> >
> > Repro steps:
> >
> > I have the `git` repo cloned and built into the CWD at v2.28.0
> > git/git clone --filter=blob:none file:///home/nipunn/src/git git2
>
> I couldn't quite reproduce this. Here's my reproduction:
>
>   #!/bin/sh
>
>   rm -fr repo clone
>   git init repo
>
>   (
>     cd repo
>
>     git config uploadpack.allowfilter true
>     git config uploadpack.allowanysha1inwant true
>
>     for f in a b; do
>       echo $f > $f
>       git add $f
>       git commit -m "$f: initial commit"
>     done
>   )
>
>   git clone --filter=blob:none file://$(pwd)/repo clone
>
> That works on v2.28.0, and v2.29.0 for me. Is git installed properly on
> your system?
>
> Thanks,
> Taylor

Re: Partial clone fails with file:/// URI

[Taylor Blau]

On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:
> Hi,
>
> I've been building git from source and directly using the executable. Roughly
> ```
> cd git
> make
> cd ..
> git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git
> ```
>
> This is what I was missing:
> git config uploadpack.allowanysha1inwant true
>
> If this is not set, then the checkout fails w/ a
>
> fatal: protocol error: bad pack header
>
> Perhaps the error message and/or docs could be improved here to
> something with more detail than "bad pack header"?
> Not seeing anything to suggest this in the docs
> https://git-scm.com/docs/partial-clone

Ah. OK; I feel much better knowing what the difference between our two
setups is. This is a long-standing issue (I remember seeing it when I
was experimenting with partial clones several releases ago) so I'm glad
that we _don't_ have to treat this as a regression in 2.29.

> Thanks for the prompt response. Appreciated!
> --Nipunn

Thanks,
Taylor

Re: Partial clone fails with file:/// URI

[Nipunn Koorapati]

I improved the "bad pack header" error message in a patch off of
v2.28.0 (https://github.com/gitgitgadget/git/pull/755)
However, rebasing onto v2.29.0, it seems like the error no longer
occurs during checkout.
Was there some recent improvement? Looking through the changelog, I'm
not finding anything obvious.

It appears that regardless of
git config uploadpack.allowanysha1inwant true

we find `git checkout` succeeds on v2.29.0

There is a test case in the PR above
Also a manual repro:

cd git
git checkout v2.29.0-rc1
make
cd ..
git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git

Thanks
--Nipunn

On Wed, Oct 14, 2020 at 6:27 PM Taylor Blau <me@ttaylorr.com> wrote:
>
> On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:
> > Hi,
> >
> > I've been building git from source and directly using the executable. Roughly
> > ```
> > cd git
> > make
> > cd ..
> > git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git
> > ```
> >
> > This is what I was missing:
> > git config uploadpack.allowanysha1inwant true
> >
> > If this is not set, then the checkout fails w/ a
> >
> > fatal: protocol error: bad pack header
> >
> > Perhaps the error message and/or docs could be improved here to
> > something with more detail than "bad pack header"?
> > Not seeing anything to suggest this in the docs
> > https://git-scm.com/docs/partial-clone
>
> Ah. OK; I feel much better knowing what the difference between our two
> setups is. This is a long-standing issue (I remember seeing it when I
> was experimenting with partial clones several releases ago) so I'm glad
> that we _don't_ have to treat this as a regression in 2.29.
>
> > Thanks for the prompt response. Appreciated!
> > --Nipunn
>
> Thanks,
> Taylor

Re: Partial clone fails with file:/// URI

[Jeff King]

On Wed, Oct 14, 2020 at 06:04:18PM +0100, Nipunn Koorapati wrote:

> I've been building git from source and directly using the executable. Roughly
> ```
> cd git
> make
> cd ..
> git/git clone --filter=blob:none file://$(pwd)/git a_clone_of_git
> ```

Note that running a built copy directly without installing may end up
running a different version of Git under the hood. Your initial
"git/git" will run the newly built version, but it will put the
to-be-installed exec-path at the front of $PATH (probably
~/libexec/git-core if you haven't configured anything). But since you
haven't installed anything there, that won't do anything, and instead
any git subprocesses it runs (including the server-side upload-pack and
any child fetches it runs to get the blobs) will be done with whatever
version of git is in your $PATH already.

Try:

  git/bin-wrappers/git clone ...

which is a shell script that sets up the exec-path properly for running
out of the build directory.

When I run your commands, I get:

  error: Server does not allow request for unadvertised object d16e5f041f5d3ad91129c4fe6713c71727895e9c
  error: unable to read sha1 file of Documentation/RelNotes/2.1.4.txt (d16e5f041f5d3ad91129c4fe6713c71727895e9c)

and so on for each object. Some older versions of Git did not correctly
handle this situation (they asked for the object even though the server
did not claim to allow it). It was fixed in 2b69d3116f (Merge branch
'mm/fetch-show-error-message-on-unadvertised-object' into maint,
2017-03-28), I think, in v2.21.3. If your system Git is older than that,
this might be the culprit.

> This is what I was missing:
> git config uploadpack.allowanysha1inwant true

Note that you shouldn't need to do that starting with v2.29, because the
v2 protocol is the default (which always allows any sha1 to be
requested).

> If this is not set, then the checkout fails w/ a
> 
> fatal: protocol error: bad pack header
> 
> Perhaps the error message and/or docs could be improved here to
> something with more detail than "bad pack header"?

The problem is that the client is hanging up when we are trying to read
the pack header. I think in newer versions of Git this is better, as
well (the server is better about sending an over-the-wire ERR packet,
and the client is better about reading and relaying it, though there are
still some races around write() errors here).

-Peff