Argument list too long when fetching many missing objects from partial clone

Argument list too long when fetching many missing objects from partial clone

[Bagas Sanjaya]

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

I'm poking around with partial clone. On my local server on my computer
I have full clone of Git (git.git) repo.

I do partial clone from that server as remote (which is actually on the same
computer):

    git clone https://my-local-server.git/myself/git.git --filter=blob:none

Inside the partial clone, I tried to convert it to full clone. First, I gather
list of missing objects:

    git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list

Then I fetched those:

    git fetch origin $(cat .git/missing.list)

What did you expect to happen? (Expected behavior)

All missing objects fetched successfully

What happened instead? (Actual behavior)

Git returned:

     -bash: /opt/git/bin/git: Argument list too long

What's different between what you expected and what actually happened?

(nothing)

Anything else you want to add:

I think the problem lies on how to feed list of 110K+ objects to `git fetch`.

Note: Both the local server and the computer use Git 2.30.1

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.

[System Info]
git version:
git version 2.30.1
cpu: x86_64
built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
compiler info: gnuc: 9.3
libc info: glibc: 2.31
$SHELL (typically, interactive shell): /bin/bash

[Enabled Hooks]
(none)

-- 
An old man doll... just what I always wanted! - Clara

Re: Argument list too long when fetching many missing objects from partial clone

[Jeff Hostetler]

On 3/1/21 8:18 AM, Bagas Sanjaya wrote:
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
> 
> What did you do before the bug happened? (Steps to reproduce your issue)
> 
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
> 
> I do partial clone from that server as remote (which is actually on the 
> same
> computer):
> 
>     git clone https://my-local-server.git/myself/git.git --filter=blob:none
> 
> Inside the partial clone, I tried to convert it to full clone. First, I 
> gather
> list of missing objects:
> 
>     git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > 
> .git/missing.list
> 
> Then I fetched those:
> 
>     git fetch origin $(cat .git/missing.list)
> 

That command will build a very very long command line and cause the
shell to complain.  Instead, try passing the list of objects via stdin.

     $ cat .git/missing.list | git fetch --stdin origin



> What did you expect to happen? (Expected behavior)
> 
> All missing objects fetched successfully
> 
> What happened instead? (Actual behavior)
> 
> Git returned:
> 
>      -bash: /opt/git/bin/git: Argument list too long
> 
> What's different between what you expected and what actually happened?
> 
> (nothing)
> 
> Anything else you want to add:
> 
> I think the problem lies on how to feed list of 110K+ objects to `git 
> fetch`.
> 
> Note: Both the local server and the computer use Git 2.30.1
> 
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
> 
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 
> 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
> 
> [Enabled Hooks]
> (none)
> 

Re: Argument list too long when fetching many missing objects from partial clone

[Bryan Turner]

On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>
> Thank you for filling out a Git bug report!
> Please answer the following questions to help us understand your issue.
>
> What did you do before the bug happened? (Steps to reproduce your issue)
>
> I'm poking around with partial clone. On my local server on my computer
> I have full clone of Git (git.git) repo.
>
> I do partial clone from that server as remote (which is actually on the same
> computer):
>
>     git clone https://my-local-server.git/myself/git.git --filter=blob:none
>
> Inside the partial clone, I tried to convert it to full clone. First, I gather
> list of missing objects:
>
>     git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>
> Then I fetched those:
>
>     git fetch origin $(cat .git/missing.list)

Since you're on Git 2.30, instead of trying to place all the object
IDs on the `git fetch` command line, have you tried adding `--stdin`
and having it read from your `missing.list` directly? Something like
`git fetch --stdin origin <.git/missing.list` might do what you need.

>
> What did you expect to happen? (Expected behavior)
>
> All missing objects fetched successfully
>
> What happened instead? (Actual behavior)
>
> Git returned:
>
>      -bash: /opt/git/bin/git: Argument list too long
>
> What's different between what you expected and what actually happened?
>
> (nothing)
>
> Anything else you want to add:
>
> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>
> Note: Both the local server and the computer use Git 2.30.1
>
> Please review the rest of the bug report below.
> You can delete any lines you don't wish to share.
>
> [System Info]
> git version:
> git version 2.30.1
> cpu: x86_64
> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
> sizeof-long: 8
> sizeof-size_t: 8
> shell-path: /bin/sh
> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
> compiler info: gnuc: 9.3
> libc info: glibc: 2.31
> $SHELL (typically, interactive shell): /bin/bash
>
> [Enabled Hooks]
> (none)
>
> --
> An old man doll... just what I always wanted! - Clara

Re: Argument list too long when fetching many missing objects from partial clone

[Bagas Sanjaya]

After trying your suggestion, I now get:

error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
fatal: unable to write request to remote: Broken pipe

On 02/03/21 02.22, Bryan Turner wrote:
> On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>>
>> Thank you for filling out a Git bug report!
>> Please answer the following questions to help us understand your issue.
>>
>> What did you do before the bug happened? (Steps to reproduce your issue)
>>
>> I'm poking around with partial clone. On my local server on my computer
>> I have full clone of Git (git.git) repo.
>>
>> I do partial clone from that server as remote (which is actually on the same
>> computer):
>>
>>      git clone https://my-local-server.git/myself/git.git --filter=blob:none
>>
>> Inside the partial clone, I tried to convert it to full clone. First, I gather
>> list of missing objects:
>>
>>      git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
>>
>> Then I fetched those:
>>
>>      git fetch origin $(cat .git/missing.list)
> 
> Since you're on Git 2.30, instead of trying to place all the object
> IDs on the `git fetch` command line, have you tried adding `--stdin`
> and having it read from your `missing.list` directly? Something like
> `git fetch --stdin origin <.git/missing.list` might do what you need.
> 
>>
>> What did you expect to happen? (Expected behavior)
>>
>> All missing objects fetched successfully
>>
>> What happened instead? (Actual behavior)
>>
>> Git returned:
>>
>>       -bash: /opt/git/bin/git: Argument list too long
>>
>> What's different between what you expected and what actually happened?
>>
>> (nothing)
>>
>> Anything else you want to add:
>>
>> I think the problem lies on how to feed list of 110K+ objects to `git fetch`.
>>
>> Note: Both the local server and the computer use Git 2.30.1
>>
>> Please review the rest of the bug report below.
>> You can delete any lines you don't wish to share.
>>
>> [System Info]
>> git version:
>> git version 2.30.1
>> cpu: x86_64
>> built from commit: 773e25afc41b1b6533fa9ae2cd825d0b4a697fad
>> sizeof-long: 8
>> sizeof-size_t: 8
>> shell-path: /bin/sh
>> uname: Linux 5.10.11-kernelorg-upstream-generic #1 SMP Fri Jan 29 12:56:19 WIB 2021 x86_64
>> compiler info: gnuc: 9.3
>> libc info: glibc: 2.31
>> $SHELL (typically, interactive shell): /bin/bash
>>
>> [Enabled Hooks]
>> (none)
>>
>> --
>> An old man doll... just what I always wanted! - Clara

-- 
An old man doll... just what I always wanted! - Clara

Re: Argument list too long when fetching many missing objects from partial clone

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1552 bytes --]

[Note: we generally prefer inline responses.]

On 2021-03-02 at 06:55:27, Bagas Sanjaya wrote:
> After trying your suggestion, I now get:
> 
> error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
> fatal: unable to write request to remote: Broken pipe

Hmmm, this means that your local Git server is unhappy with the size of
the request.  That means your request is probably very large, and it's
likely that it is kindly asking you not to DoS it.

> On 02/03/21 02.22, Bryan Turner wrote:
> > On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > > Inside the partial clone, I tried to convert it to full clone. First, I gather
> > > list of missing objects:
> > > 
> > >      git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
> > > 
> > > Then I fetched those:
> > > 
> > >      git fetch origin $(cat .git/missing.list)
> > 
> > Since you're on Git 2.30, instead of trying to place all the object
> > IDs on the `git fetch` command line, have you tried adding `--stdin`
> > and having it read from your `missing.list` directly? Something like
> > `git fetch --stdin origin <.git/missing.list` might do what you need.

You may want to try this:

  xargs git fetch origin < .git/missing.list

That will batch the requests to fit on the command line, which will
probably also have the pleasant side effect that it makes each
individual request smaller to avoid the 413.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]